IBM DB2 specific SAP NetWeaver Business Warehouse Near-Line Storage Solution Karl Fleckenstein (karl.fleckenstein@de.ibm.com) IBM Deutschland Research & Development GmbH June 22, 2011 © 2011 IBM Corporation Important Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: – CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR – ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. Trademarks IBM, the IBM logo, ibm.com, AIX and DB2 are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Windows is a trademark of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. SAP, SAP NetWeaver, SAP Business Information Warehouse, SAP BW, SAP NetWeaver BW, SAP NetWeaver Business Warehouse Accelerator and other SAP products and services mentioned herein are trademarks or registered trademarks of SAP AG in Germany and in several other countries. Other company, product and service names may be trademarks or service marks of others. 2 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Agenda • SAP NetWeaver BW Near-Line Storage (NLS) • IBM DB2 specific SAP NetWeaver BW NLS solution – DB2 solution – Integration – Performance • Summary and additional information 3 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Challenges for Business Applications and Databases Amount of data increases – More users – More data sources – More and new types of data (e.g. Point of Sales (POS), Radio Frequency Identification (RFID)) t – New business transactions System performance – Increasing query response times – Increasing time for data processing, e.g. ETL – Increasing time for database administration & operation e.g. performing backups, table/index reorganization Increasing costs – DB Server – Storage – High Availability (HA) & $€£ Disaster Recovery (DR) solution 4 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Address the Challenges with Near-Line Storage Reduce amount of data in online database – – – – – – Faster query processing Online Faster and smaller backups Storage Reduced data volume on HA and DR site Reduced administration effort Retain existing hardware infrastructure Reduce costs Near-Line Storage (NLS) = Near (On)Line Storage – Move aged data to NLS – NLS data can be stored on less expensive Near-Line infrastructure Storage – Lower HA requirements, less frequent backups 5 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Information Lifecycle Management – Types of Data Storage Frequent data access • Actual data • Online data access • Read/Write/Update/Delete Online Storage Near-Line Storage Data Archiving Less frequent data access • Aged data • Online data access • Read-Only Very rare access to data • Long term storage • No online data access • Read-Only 6 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – General • • • SAP BW NLS solution is integrated into SAP data archiving stragegy (SAP Archiving Development Kit = SAP ADK) SAP BW InfoProviders can be archived with a Data Archiving Process (DAP) Options to archive SAP BW data: – Archive to Data Archive only (SAP ADK) – Transfer to Near-line Storage only – Archive / transfer to both, SAP ADK and NLS Online Database Near-Line Storage = “Near Online” Storage Infrequently read data (aged) Very rarely read data (aged) Data Archiving Process Online BW Data Frequently read/changed data (actual) Classic Archive Data location: optimal tolerable unacceptable Data-Archive (SAP ADK) Near-Line Storage 7 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – Data Transfer • • • Request based data transfer from SAP BW objects to NLS objects Process steps of the NLS data transfer – Read data from BW InfoProvider (BW database) in data packages – Write data into Near-Line Storage object – Data verification: compare NLS data with BW data – Delete transfered data in the BW InfoProvider (BW database) Reload of NLS data into BW database for modification purpose TransparentAccess Access Transparent SAP NetWeaver BW BI OLAP BW OLAP BI Data Manager DBInterface Interface DB Layer Layer DBMS Relational DB DB Relational Interface Interface BW Online Data General NLS General Near-Line Interface Interface TREX NLS /Partner DB2 LUW Near-Line Interface Interface Near-Line Storage 2011 2010 2009 2008 8 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – Data Archiving Process (DAP) • DAP with NLS archiving 9 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – Data Archiving Process (DAP) • Definition of the NLS archiving criteria: time and optional additional criteria 10 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – NLS Request • Create and monitor an NLS data archiving request 11 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS Solution – Data Access SAP BW Query / Data extraction • • • • • Online access to both BW and NLS data BW OLAP processor splits query into two parts and sends it to the BW Online Database and to NLS BW OLAP processor calculates final result SAP BW 7.0x query access specification – On query level with transaction rsrt – On InfoProvider level (DB2 LUW specific) SAP BW 7.3 query access specification – In query selection screen – In BEx Query Designer – On Multi-Provider level – On InfoProvider level TransparentAccess Access Transparent SAP NetWeaver BW BI OLAP BW OLAP BI Data Manager DBInterface Interface DB Layer Layer DBMS Relational DB DB Relational Interface Interface BW Online Data General NLS General Near-Line Interface Interface TREX NLS /Partner DB2 LUW Near-Line Interface Interface Near-Line Storage 2011 2010 2009 2008 12 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation NLS Positioning within the SAP NetWeaver BW Solution The NLS concept complements with the SAP NetWeaver Business Warehouse Accelerator (BWA), keeping aged data away from the valuable SAP NetWeaver BWA and BW Online Database TransparentAccess Access Transparent SAP NetWeaver BW BW OLAP Business Warehouse Accelerator DB Interface BWA Interface Database Interface Near-Line Storage Interface Near-Line Storage BW Online Database Data Access: very frequently frequently 13 Karl Fleckenstein, IBM Deutschland R&D not frequently © 2011 IBM Corporation Agenda • SAP NetWeaver BW Near-Line Storage (NLS) • IBM DB2 specific SAP NetWeaver BW NLS solution – DB2 solution – Integration – Performance • Summary and additional information 14 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP BW NLS Solution with IBM DB2 for LUW • Separate DB2 database used to store NLS data • IBM DB2 specific NLS partner interface implemented • • • • SAP NetWeaver BW Access via SAP ABAP secondary connection SAP BW Unicode systems only Supported DB2 release for NLS: DB2 V9.7 BW OLAP BI Data Manager DBInterface Interface DB Layer Layer DBMS General NLS General Near-Line Interface Interface TREX NLSNear-Line / DB2 LUW DB2 Interface Interface Relational DB Relational DB InterfaceInterface Proven IBM DB2 features are used – – – – – BI OLAP Automatic Storage Self Tuning Memory Management Compression Multi-Dimensional Clustering (MDC) Database Partitioning Feature (DPF) is optional BW database InfoCube 2009 2008 2007 DSO 2009 2008 2007 2006 15 Karl Fleckenstein, IBM Deutschland R&D NLS / DB2 database InfoCube 2006 2005 2004 2003 DSO 2005 2004 2003 © 2011 IBM Corporation SAP BW NLS on DB2 – Delivery & Support • Delivery with SAP NetWeaver BW 7.01 SP6; DB2/NLS solution is part of SAP BW • SAP NetWeaver BW Included in SAP IBM DB2 OEM license at no extra cost • Same SAP support channel as SAP BW on IBM DB2 solution • NLS / IBM DB2 database can be connected to SAP BW systems running on other (nonIBM DB2 for LUW) databases BI OLAP BW OLAP BI Data Manager DBInterface Interface DB Layer Layer DBMS General NLS General Near-Line Interface Interface TREX NLSNear-Line / DB2 LUW DB2 Interface Interface Relational DB Relational DB InterfaceInterface BW database InfoCube 2009 2008 2007 DSO 2009 2008 2007 2006 16 Karl Fleckenstein, IBM Deutschland R&D NLS / DB2 database InfoCube 2006 2005 2004 2003 DSO 2005 2004 2003 © 2011 IBM Corporation SAP BW NLS on DB2 – Set-Up Alternatives • Separate NLS database for each SAP BW system – Pros: NLS performance on DB level, independent database recovery – Cons: More administration effort with more NLS databases, e.g. db backups Test System BW1 database • NL1 BW2 database Production System NL2 BW3 database NL3 One NLS database for several SAP BW systems Seperate database schema on the NLS database for each SAP BW system – Pros: Less administration effort – Cons: Other SAP BW systems impacts NLS performance, no access to all NLS systems during NLS database recovery Test System BW1 database • Quality System Quality System BW2 database Production System BW3 database NLS Database Schema db2nl3 Schema db2nl2 Schema db2nl1 Meaningful NLS database set-up – Separate NLS database for production system – Shared NLS database for test & quality systems 17 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP BW NLS on DB2 – Operation (1) • Operation of the NLS / IBM DB2 solution with existing DB2 DBA skill (no new technology introduced) • Administration & Monitoring with SAP DBACockpit • Database backups of the NLS system can be performed in less frequent intervals using different infrastructure (e.g. Flash copy vs. tape) BW database NLS / DB2 database tape tape tape tape tape Flash Copy Flash Copy Flash Copy Flash Copy Flash Copy tape tape 18 Karl Fleckenstein, IBM Deutschland R&D t © 2011 IBM Corporation SAP BW NLS on DB2 – Operation (2) • Synchronized NLS recovery process – Advised process for NLS database recovery • Status information of an NLS request is stored in the SAP BW datatabase – Automatic corrections of data inconsistency (if appropriate) – Automatic data consistency check tape Point in Time 1 BW database Restore Restore NLS / DB2 database Roll forward time Roll forward Point in Time 2 tape 19 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP BW NLS on DB2 – SAP Operations • Data Archiving Processes (DAPs) can be transported within the SAP system landscape – Prerequisite: Same NLS connection name on source and target system Test System DAP1, DAP2, db6nls BW database NL1 Production System DAP3 db6nls BW database NL2 • SAP System Copy – Target system has no NLS access after system copy (per default) – SAP DBA Cockpit support for simple verification and reconnect to NLS DB 20 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS – Supported SAP BW Objects Supported SAP BW objects • InfoCubes – Standard InfoCubes – Real-time Infocubes – SAP BW 7.0x (x = 1,2,..): SAP BW compression is needed before data can be transferred to NLS (data transfer from E fact table) SAP BW 7.3: SAP BW compression is no longer a prerequisite (data transfer from F fact table is possible) • • • – DataStore Objects (DSO) – Standard DSOs – Write-optimized DSOs – DSOs for direct update MultiProvider support – SAP BW 7.0x (x = 1,2,..): Modeling solution with VirtualProviders is available supported by an ABAP tool – SAP BW 7.3: Native support Semantically Partitioned Objects (with 7.3) Not supported SAP BW objects • Non-cumulative InfoCubes • PSA tables, but – SAP recommendation: Do not use PSA tables for long-term SAP BW data, instead use write-optimized DSOs (which are supported for NLS) • InfoSets 21 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – InfoCube Table Layout NLS InfoCube SAP BW InfoCube M1 M2 M3 T1 MDkey MDkey MDkey MDkey MDkey MDkey MDkey MDkey MDkey MDkey MDkey MDkey S7 S8 SID SID SID SID S1 S2 S3 S4 S5 S6 X1 SID SID SID SID SID SID SID SID SID SID SID SID SID Product Area Customer DIMID DIMID DIMID Simple Star-Schema – One fact table with DB2 Multi-Dimensional Clustering (MDC) • On Arch-Request column • Other MDC columns are derived from SAP BW fact table – Arch-Request is an additional dimension MDKey MDKey MDKey MDKey MDKey MDKey MDKey Product Area Customer PKey PKey PKey DIMD DIMID DIMID DIMD DIMID DIMID Key E Key DIMID DIMID DIMID Package Time Unit SID SID SID SID SID SID Key Req-ID State Key Fact DIMID DIMID DIMID DIMID DIMID DIMID SID Key Time Arch-Request Key F PKey Key PKey PKey PKey Package Time Unit MDKey MDKey MDKey MDKey MDKey MDKey MDKey 22 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – DataStore Table Layout BW DataStore Object (DSO) NLS DataStore Object (DSO) Standard DataStore Object • Archiving request information is stored in the addtitional Arch-Request table Change Log • NLS DSO table with MDC Activation Queue Active Table – On Arch-Request column – Other MDC columns are derived from SAP BW DSO table Req-ID State Time Arch-Request PKey Write-optimized DataStore Object Key WriteOptimized DSO NLS DSO Table 23 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – MultiProvider Support SAP BW 7.0x SAP BW 7.3 • • Modeling solution with VirtualProviders One VirtualProvider for all InfoProviders • Native MultiProvider support without additional modeling of VirtualProviders • One VirtualProviders for each InfoProvider; tool support to create VirtualProviders (DB2 specificSAP note 1566019) • New property to specify NLS access 24 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Agenda • SAP NetWeaver BW Near-Line Storage (NLS) • IBM DB2 specific SAP NetWeaver BW NLS solution – DB2 solution – Integration – Performance • Summary and additional information 25 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Integration • NLS database installation fully integrated into SAP Installation Tool SAPInst • NLS Enhancements in SAP‘s DBA Cockpit DBACockpit – Configure connection to NLS/DB2 database – Administration and monitoring of User db2nl1 NLS/DB2 database – Backup of NLS database – Space consumption of NLS objects BW database 26 Karl Fleckenstein, IBM Deutschland R&D NLS / DB2 database © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Installation of NLS DB • NLS / DB2 database installation with SAP’s installation tool SAPInst • SAPInst creates separate DB2 Instance and DB2 database for Near-Line Storage 27 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Installation of NLS DB • Connect to NLS database • Supported within SAP’s database administration tool DBACockpit NLS Database Schema: DB2NL1 User db2nl1 InfoCube DSO 2004 2003 28 Karl Fleckenstein, IBM Deutschland R&D 2003 © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – NLS Administration • NLS / DB2 administration and monitoring is integrated into SAP’s database administration tool DBACockpit • Use of existing DBACockpit functionality for administration and monitoring of remote databases (NLS database): – Administration of Remote Tablespaces – Backup of remote databases 29 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – NLS Administration • NLS / DB2 administration and monitoring is integrated into SAP’s database administration tool DBACockpit • Use of existing DBACockpit functionality for administration and monitoring of remote databases (NLS database): – Administration of Remote Tablespaces – Backup of remote databases • NLS specific extensions of the DBACockpit – NLS Configuration – NLS Objects Overview – NLS Synchronisation Check 30 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Monitoring of NLS DB • Analyse SAP BW InfoProviders and the corresponding NLS objects 31 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Monitoring of NLS DB • Analyse SAP BW InfoProviders and the corresponding NLS objects 32 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Monitoring of NLS DB • Analyse SAP BW InfoProviders and the corresponding NLS objects 33 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Agenda • SAP NetWeaver BW Near-Line Storage (NLS) • IBM DB2 specific SAP NetWeaver BW NLS solution – DB2 solution – Integration – Performance • Summary and additional information 34 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP BW NLS on DB2 – Test Environment System configuration SAP BW NLS Ratio DB Server DB Server NLS / BW Amount of disk spindles 192 96 RAM size, GB 64 32 CPU architecture POWER 7 POWER 6 Threads per core 4 2 CPU cores, total 6 4 CPU threads, total 24 8 35 Karl Fleckenstein, IBM Deutschland R&D 50% 50% --50% 67% 33% © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Data Transfer Performance Test scenario • InfoCube has 745 Million records • Transfer 500 Million records from BW to NLS SAP NetWeaver BW 10 CPU Cores, SMT=4 64 GB Memory (POWER7, 3.3 GHz) Results • Copy + verification performance, records per hour: – with 8 parallel processes / archiving runs : 114 Million records / hour – with 15 parallel processes / archiving runs: 185 Million records / hour • Delete performance (with MDC on the Time BW DB Server NLS DB Server - Data on 7 DB2 partitions - MDC on time - Data on 8 DB2 partitions - MDC on time 6 CPU Cores, SMT=4 64 GB Memory (POWER7, 3.3 GHz) Dimension) : – 1 process is running: 546 Million records / hour Overall performance for the complete data transfer to NLS: Copy + verification + deletion 135 Million records / hour Parameter name Average user CPU load 4 CPU Cores, SMT=2 64 GB Memory (POWER6, 3.5 GHz) SAP SAP Application database server server NLS server 10 cores, 40 threads 6 cores, 24 threads 4 cores, 8 threads 56% 14% 48% 36 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – BW Query Performance (1) Test Description pSeries P570, Power 6, 3.5 GHz • Fact table has 350 Million rows • SAP BW Queries – Aggregation of 65 Million rows – 2000 rows returned LPAR1: 4 Cores, 32 GByte DB Partition 0 DB Partition 1 DB Partition 2 DB Partition 3 DB Partition 4 DB Partition 5 DB Partition 6 • Test Scenarios – 1 and 2 Queries read data from BW database – 1 and 2 Queries read data from NLS database DB Partition 7 pSeries P570, Power 6, 3.5 GHz LPAR1: 4 Cores, 32 GByte with 8 database partitions – 1 and 2 Queries read data from single-partitioned NLS database DB Partition 0 – Each query read different time intervals 37 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – BW Query Performance (4) • Effect of DB2 database partitioning on NLS System – Scalability • In general: SAP BW query performance scales with number of DB2 db partitons Query Run Time 363 Seconds 296 50 65 58 33 BW only NLS only NLSonly (7 db partitions, 6 cores, 24 threads) (8 db partitions 4 Cores, 8 threads) (1 db partitions 4 Cores, 8 threads) 38 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Scalability Flexibility of the NLS hardware environment, according to individual needs • • • IBM DB2 database partitioning feature (DPF) allows to grow the environment Flexible - From Cost Sensitive to Maximum Throughput Possibility to grow with amount of NLS data 39 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Agenda • SAP NetWeaver BW Near-Line Storage (NLS) • IBM DB2 specific SAP NetWeaver BW NLS solution • Summary and additional information 40 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Summary Why DB2 specific NLS for SAP NetWeaver BW? Criteria SAP BW NLS on DB2 Delivery With SAP BW Releases; part of standard SAP BW Integration Fully integrated into SAP tools (SAPInst, DBACockpit), which are delivered with standard SAP BW and SAP Basis releases Operation: Monitoring Administration, Backup, Recovery - With SAP standard tools or DB2 tools Compression Data volume reduced typically by factor 3 to 5 compared to uncompressed data Performance - Very fast data transfer to NLS - Synchronized recovery process - Parallel query processing System Layout Flexibel; depends on Performance & TCO requirements Support Use existing SAP/DB2 support channel Needed Skill Existing DB2 skill TCO Reduced TCO 41 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation SAP NetWeaver BW NLS on DB2 – Additional Information News, Blogs, Backgrounds, Docs: https://www.sdn.sap.com/irj/sdn/db6 Documentation “Enabling SAP NW BW to Use IBM DB2 for Linux, UNIX, and Windows as Near-Line Storage“ available at: http://service.sap.com/instguidesnw/ SAP NetWeaver 7.0 (2004s) -> Operations SAP NetWeaver 7.3 -> Operations SAP Note 1405664: DB6 Near-Line Storage Impl. (supplementary note) available at SAP Service Market Place: http://service.sap.com/notes E-Learning Expert Session on IBM DB2 Near-Line Storage Solution for SAP NetWeaver BW available at: http://www.sdn.sap.com/irj/sdn/db6?rid=/webcontent/uuid/400c5e34-c601-2c10-6cb4-9aff67ca31c3 42 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation Thank you for attending! 43 Karl Fleckenstein, IBM Deutschland R&D © 2011 IBM Corporation