SAP SD Performance Tuning - Xbox and the Need for Speed Christoph Schuler, Cindi McCutchen SAP Application Support Microsoft Corporation Agenda SAP at Microsoft – Quick Facts • Why Performance Tuning • SAP Service Offerings • Project Approach & Best Practices • Tuning for Xbox – Changes Implemented – Summary of Results – Lessons Learned SAP at Microsoft Quick Facts • Modules implemented: SD, MM, FI, CO, AM, HR, BW, APO • 875 GB in a single, central Database; 6 GB/wk growth • 2,000 named users; 50,000+ total users • > 600 concurrent users; heavy EDI usage • 0.5 - 2 sec response time globally (incl. WAN connectivity) • 99.9+% availability • Platform: SQL Server 2000, Windows .NET server Why Performance Tuning ? • Business need for data throughput – Xbox Launch North America, Nov. ’01: Peak volumes Sales orders / day Deliveries / day Sales orders / hour * Deliveries / hour * * Each order & delivery with 10 line items Existing ~ 5,000 ~ 5,000 ~ 250 ~ 1,000 Xbox + 7,600 + 11,400 + 2,500 + 2,500 Change + 152 % + 228 % + 1,000 % + 250 % Why Performance Tuning ? • System availability & supportability – Prevention of: • • • • • Cross-module increase in response time Business disruptions Penalties End user frustration Increased support costs • System scalability – Ability to handle future growth SAP Service Offerings Technical Solution Optimization (TSO) Now: Solutions Management Optimization (SMO) Interface Management Jun ‘00: • EDI Interface • GBI Interface Business Process Optimization Sep ‘00: • Sales Orders • Backorder Resch. • Deliveries • Billing Data Mgmt. & Archiving Oct ‘00: • FI Archiving Focus on critical SD business processes 50+ recommendations to improve performance Project Approach • Project team – – – – – Analysis Functional Basis/Technical Development Stress Test Consulting (BUIT & SAP) • Project timeline – Driven by business need • Repetitive cycle – One item at a time • Project metrics – e.g. seconds/document Test Change Test Best Practices Analysis Test • Have an original baseline test • Automate tests and standardize your data & configuration • Include anticipated new functionality (e.g. routing, allocations) • Consider systems external to SAP • Make test as realistic as possible ! Change Analysis Best Practices Analysis Test • Define your metrics (e.g. sec/doc, response time, etc..) • Carefully document results and improvements with each cycle • Review results and plan for next changes to be implemented and tested Change Analysis Analysis Change Useful Tools & Transactions Test • • • • • • • • • • • Systemwide work process overview – SM66 Workload monitor – ST03 SQL trace - ST05 SQL trace interpreter - SQLR Application analysis – ST14 Table access statistics – ST10 Memory configuration monitor – ST02 R/3 OS monitor – ST06 IDoc monitor – WE05 Gateway Monitor - SMGW Windows PerfMon – CPU utilization Analysis Analysis Process Overview - SM66 Test • Quick overview of all work processes • Look for: – – – – Long-running processes “Stopped” processes Highly utilized servers Processes accessing the same tables Change Analysis Analysis Database Trace - ST05 Test • Important tool to analyze database performance problems • Look for: – Most “expensive” database accesses (Trace Summary) – Repetitive database accesses Change Change Analysis Best Practices • Changes may include: – OSS notes (> 2,600 performance notes for 4.5B) – ABAP code optimization – Configuration – Technical settings – Hardware – Business process or logic • Carefully document changes made Change Agenda • Why Performance Tuning • SAP Service Offerings • Project Approach & Best Practices Tuning for Xbox – Changes Implemented – Summary of Results – Lessons Learned Tuning for Xbox Retailer 1 Sales Order (EDI 850) Microsoft (SAP System) Sales Order Distribution Warehouse Tuning for Xbox 1 – Sales Order Processing • Switch to parallel processing of inbound IDocs • • Partner profile change: ’Immediate’ to ‘Background’ (via WE20) Server group configuration with resource parameters (via RZ12) –OSS 384971 Schedule program RBDAPP01 Specify packet size and ‘Parallel’ posting option Deactivate IDoc syntax check Increase of program buffer Tuning for Xbox 1 – Sales Order Processing • SD number range buffering • ABAP code improvements • Table NRIV - Object RV_BELEG (via SNRO) Local buffering by work process ID Exclusion of sales orgs (OSS note 424486) VA01 user exits Elimination of repetitive table reads Table buffering Monitor access statistics (via ST10) Tuning for Xbox Retailer 1 Sales Order (EDI 850) Microsoft (SAP System) Sales Order Rescheduling 2 Distribution Warehouse Tuning for Xbox 2 - Backorder Rescheduling • • Cleanup of old orders Program SDV03V02 by plant – daily • Modification (repair) of SDV03V02 • Evaluation of program SDV03V01 Due to SD number buffering Take order creation date & time into account Custom pre-processing program Dynamic job builder and distribute workload over several processes Utilize standard SAP programs Tuning for Xbox Retailer 1 Sales Order (EDI 850) Microsoft (SAP System) Sales Order Rescheduling Delivery 2 3 Ship Order (EDI 940) Distribution Warehouse Tuning for Xbox 3 – Delivery (EDI Ship Order) • Custom pre-processing program: • • Dynamic job builder and distribute workload over several processes Utilize standard SAP program (RV50SBT1) Additional index on table VBBE Separation of output from document creation Schedule RSNAST00 to process output Schedule RSEOUT00 to process IDocs and combine into files Tuning for Xbox 3 – Delivery (EDI Ship Order) • • • Deactivate IDoc Syntax Check SD number range buffering Activate shared locking • For availability checking group level for ATP (via OVZ2) Evaluate program RVV50R1T (VL10) instead of current DDL Program was evaluated but not implemented Server group functionality is available in SAP version 4.6c Tuning for Xbox Retailer 1 Sales Order (EDI 850) Microsoft (SAP System) Sales Order Rescheduling Delivery 2 5 3 Ship Order (EDI 940) 4 Shipment Goods Issue Ship Confirm (EDI 945) Distribution Warehouse Tuning for Xbox 5 – Post Goods Issue (EDI Ship Confirm) • Switch to parallel processing of inbound IDocs via RBDAPP01 • Prevent plant locking • Allow IDOC to be reprocessed (from a status 64) instead of going into an error (status 51) Activate ‘Late exclusive enqueue’ • Same as sales order processing Table Locking – MVER (OSS Note 318581) For material movements via OMJI (OSS 192423) Deactivate IDoc syntax check Tuning for Xbox Retailer 1 Invoice (EDI 810) Sales Order (EDI 850) 6 Microsoft (SAP System) Sales Order Rescheduling Delivery 2 5 3 Ship Order (EDI 940) 4 Shipment Goods Issue Ship Confirm (EDI 945) Distribution Warehouse Billing Tuning for Xbox 6- Billing • SD number range buffering • Parallel processing of billing due list • Exclude certain countries due to legal requirements (OSS Note 23835, 363901) Program RV60SBAT, specify ‘Number of jobs’ Separation of output from document creation Schedule RSNAST00 to process output Schedule RSEOUT00 to process IDocs Tuning for Xbox Additional Changes Implemented • ATP server (Available to Promise) • Tax solution (Vertex) upgrade • Information in OSS note 99999 Installed on central instance Use ‘Registered RFC mode’ (OSS 63930) APO RFC calls (Advanced Planning Optimizer) RFC queue in R/3 spawning ARFC* jobs Tuning for Xbox Summary of Testing Results • SD Core Transactions: Billing 2 4 PGI 12 7 Deliveries 1 Sales Orders 1 0 Before After 4 20 5 10 15 20 Seconds / Document (10 lines) 25 Tuning for Xbox Summary of Results 30,000 Xbox Transaction Volume 20,000 10,000 0 Oct-01 Xbox Transaction Performance Orders Deliveries 4.00 3.00 2.00 1.00 0.00 Nov-01 Dec-01 Jan-02 s/Order s/Delivery Tuning for Xbox Summary of Lessons Learned • • • • • Utilize SAP/OSS resources Assemble a project team with a wide range of knowledge Only test ONE change at a time Be prepared to spend about 80% of your time testing And learn to hide from the Basis team References • “SAP R/3 Performance Optimization” • by Thomas Schneider, Sybex “ALE, EDI & IDoc Technologies for SAP” by Arvind Nagpal, Prima Tech Questions?? • Cindi McCutchen & Christoph Schuler • Microsoft Corporation • Email: cynmc@microsoft.com or christos@microsoft.com