Further. Forward. Faster. 1. Microsoft Private Cloud Solution for SAP: Hardware, Network & SAN Authors Cameron Gardiner, Microsoft Senior Program Manager – SAP Technical Reviewers Juergen Thomas, Principal Lead Program Manager Summary This document provides guidance for customers and partners to deploy SAP on a Microsoft Private Cloud Solution for SAP. This chapter discusses Hardware, Network & SAN configurations Hardware, Network & SAN Table of Contents Table of Contents ........................................................................................................ 2 1 Summary ................................................................................................................ 4 2 Microsoft Private Cloud Solution for SAP Hardware Infrastructure .............. 5 2.1 SERVER PROCESSOR & RAM ................................................................................................................... 5 2.1.1 NUMA .............................................................................................................................................. 6 2.1.2 Hyper-Threading ......................................................................................................................... 6 2.2 RACK VERSUS BLADE ................................................................................................................................. 6 2.3 NETWORK ................................................................................................................................................... 7 2.3.1 How Many Networks are Required? .................................................................................... 7 2.3.2 802.1q VLAN Tagging vs. Many Physical Network Cards ............................................ 7 2.4 FIBRE CHANNEL, FCOE, SMB 3.0 OR ISCSI .......................................................................................... 9 2.5 SAN & LUN/DISK DESIGN ..................................................................................................................... 9 2.6 FUSIONIO CARDS ...................................................................................................................................... 9 2.7 NETWORK CABLING................................................................................................................................... 9 2.8 NATIVE 4K DISK FORMATS....................................................................................................................... 9 3 Windows 2012 Hyper-V 3.0 Installation & Configuration .............................10 3.1 WINDOWS 2012 INSTALLATION .......................................................................................................... 10 3.1.1 Windows Core versus Windows Full GUI ........................................................................ 10 3.2 ADD HYPER-V & CLUSTER FEATURES ................................................................................................. 10 3.3 CONFIGURE VIRTUAL NETWORK SWITCH ............................................................................................ 11 3.3.1 Private, Internal and External Networks .......................................................................... 11 4 Microsoft Private Cloud Solution for SAP – VM Configuration ....................12 4.1 SIZING ...................................................................................................................................................... 12 4.2 HIGH AVAILABILITY & DISASTER RECOVERY ....................................................................................... 12 4.3 SUPPORTED GUEST OPERATING SYSTEMS........................................................................................... 12 4.4 SUPPORT FOR GUEST OPERATING SYSTEM CLUSTERING................................................................... 12 5 Network Configuration ......................................................................................13 5.1 PHYSICAL SERVER NETWORK CONFIGURATION (HYPERVISOR HOST) ............................................ 13 5.1.1 Team 1.......................................................................................................................................... 14 5.1.2 Team 2.......................................................................................................................................... 14 5.1.2.1 iSCSI Network (if used) ...................................................................................................... 14 5.1.2.2 LiveMigration Network...................................................................................................... 15 5.2 VIRTUAL SWITCHES ON PHYSICAL HYPERVISOR HOST...................................................................... 16 5.3 VIRTUAL NETWORK CARDS (VNIC) ON VIRTUAL MACHINE ............................................................. 17 SAP on Microsoft Private Cloud Page 2 of 37 Created: 19-Mar-16 Hardware, Network & SAN 5.3.1 5.3.2 5.3.3 5.3.4 Public Network .......................................................................................................................... 18 SAP Internal Network ............................................................................................................. 18 Heartbeat Network .................................................................................................................. 18 Management Network (for Guest VM) ............................................................................ 19 5.4 CLUSTER NETWORK CONFIGURATION FOR HIGHLY AVAILABLE VMS ............................................. 20 6 Network Tuning ..................................................................................................21 6.1 SRIOV ...................................................................................................................................................... 21 6.2 VMQ ........................................................................................................................................................ 22 6.3 RSS........................................................................................................................................................... 22 6.4 IP SEC ...................................................................................................................................................... 23 6.5 JUMBO FRAMES (FOR ISCSI ONLY) ...................................................................................................... 23 6.6 DISABLING NETBIOS & DNS CLIENT REGISTRATION ........................................................................ 23 7 Energy & Power Savings ....................................................................................24 8 Software Releases Required ..............................................................................25 8.1 WINDOWS RELEASE................................................................................................................................ 25 8.2 DATABASE RELEASE ................................................................................................................................ 25 8.3 SAP MONITORING AGENTS .................................................................................................................. 25 8.4 SAP PRODUCT AVAILABILITY MATRIX ................................................................................................. 25 9 Appendix 1 – FCoE, iSCSI & Fibre Channel ......................................................26 10 Appendix 2 – Screenshots Showing Sample Configuration ...........................27 11 Appendix 3 – Screenshots Showing SRIOv Configuration .............................28 11.1 STEP 1: COLLECT NETWORK CONFIGURATION INFORMATION.......................................................... 28 11.2 CHECKING THE USAGE OF NETWORK ACCELERATORS ON THE HYPER-V HOST .............................. 28 12 Appendix 4 - Comparison of Hyper-V versions ..............................................36 13 Appendix 5 – Links Page ....................................................................................37 SAP on Microsoft Private Cloud Page 3 of 37 Created: 19-Mar-16 Hardware, Network & SAN 1 Summary Windows 2012 Hyper-V 3.0 is a powerful integrated Virtualization platform. In order to leverage the capabilities of Virtualization platform appropriately configured Hardware must be deployed and configured The most important topics are discussed in detail in this document. In summary: 1. 2. 3. 4. 5. Deploy 2 processor servers with 384GB RAM Use 10 Gigabit Ethernet Cards Network cards must support SRIOv, VMQ, RSS and other modern features If iSCSI is used configure Jumbo Frames on all devices including the switch Separate network traffic into “Public” and “SAP Internal” networks Hardware & Network Checklist Item Network Cards, Server BIOS, Firmware & Drivers confirmed by H/W vendor to support SRIOv with Windows 2012 Network Cards, Firmware & Drivers confirmed by H/W vendor to support VMQ with Windows 2012 Network Cards, Firmware & Drivers confirmed by H/W vendor to support RSS with Windows 2012 If iSCSI is used, there are two 10 Gigabit network cards set with Jumbo Frames. All networking devices and iSCSI source are configured with a uniform MTU size = 9014 Spreadsheet built with IP Address Hardware CPU exactly matches or exceeds the specification in this document RAM in each physical server is 384GB or greater Windows 2012 Datacenter Edition Installed on cluster nodes Windows 2012 GUI removed and Hyper-V cluster nodes are running Core SQL 2012 Enterprise Edition Service Pack 1 installed as a Named Instance SAP Internal, Public, Management and Heartbeat Networks defined Disable Netbios on all interfaces, Disable DNS registration on any NIC with a Virtual Network Switch assigned to it Security Hardening Procedure Implemented Do not proceed with the installation of SAP on Microsoft Private Cloud Solution unless all of the tick boxes are confirmed SAP on Microsoft Private Cloud Page 4 of 37 Created: 19-Mar-16 Hardware, Network & SAN 2 Microsoft Private Cloud Solution for SAP Hardware Infrastructure Microsoft Private Cloud Solution for SAP reference deployment is based on a specific hardware configuration. The Hardware configuration is tested and validated by deployment with pilot customers. During lab testing and pilot deployment three factors were observed to be of very high importance (1) sufficient RAM (2) 10 Gigabit networking and (3) use of Windows 2012 or higher. It is therefore strongly recommended to follow the guidance within this document and to ensure the minimum requirements are met. One of the most important factors that determines the performance of Microsoft Hyper-V 3.0 solutions is RAM. It is therefore very important to follow the RAM guidance in this document. The cost and impact of a serious performance problem is much lower than the cost1 of RAM. The use of obsolete operating systems such as Windows 2003, 32 bit or obsolete versions of SQL Server has never been tested and is not recommended. 2.1 Server Processor & RAM Review SAP Note 1612283 - Hardware Configuration Standards and Guidance. This document should be read in its entirety as many crucial concepts are explained. This documentation series assumes the reader is familiar with this Note and does not duplicate the content. Intel SandyBridge E5 2600, AMD equivalent or later processors such as “IvyBridge” are supported. Older processors do not have the virtualization features or performance required for this reference architecture. Older processors, power saving processors or processors with fewer cores will not produce reliable performance when deployed with this reference architecture. The following Intel SandyBrdige (or AMD equivalent) are recommended: Cores (Threads) 8 (16) Processor Model 2687W 2690 2680 2670 2665 2660 2658 2650 2650L 2648L CPU Clock Standard 3.1 GHz 2.9 GHz 2.7 GHz 2.6 GHz 2.4 GHz 2.2 GHz 2.1 GHz 2.0 GHz 1.8 GHz 1.8 GHz Turbo Cores (Threads) Processor Model CPU Clock Standard Turbo 3.8 GHz 3.8 GHz 3.5 GHz 3.3 GHz 3.1 GHz 3.0 GHz 2.4 GHz 2.8 GHz 2.3 GHz 2.1 GHz L3 Cache 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB 20 MB Supported Memory 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 4x DDR3-1600 TDP L3 Cache Supported Memory TDP 150 W 135 W 130 W 115 W 115 W 95 W 95 W 95 W 70 W 70 W 1600MHz 16GB DIMM should cost ~$300USD or less at the end of 2012. A 32GB DIMM may cost more than $1300USD. Source: www.dell.com and www.ibm.com 1 SAP on Microsoft Private Cloud Page 5 of 37 Created: 19-Mar-16 Hardware, Network & SAN 16/16 Opteron 6376 Opteron 6378 Opteron 6380 Opteron 6386 SE 2.3 GHz 3.2 GHz 8 MB 4x PC3-15000 115 W 2.4 GHz 3.3 GHz 8 MB 4x PC3-15000 115 W 2.5 GHz 3.5 GHz 8 MB 4x PC3-15000 115 W 2.8 GHz 3.5 GHz 8 MB 4x PC3-15000 140 W 2.1.1 NUMA In order to reduce remote memory access and increase the amount of local memory 2 processor systems are only recommended. 4 processor systems can be built by customers with expert knowledge. See Appendix 1 of Chapter 9. Review SAP Note 1612283 - Hardware Configuration Standards and Guidance carefully before deploying SAP on Hyper-V 3.0. 2.1.2 Hyper-Threading Hyperthreading (SMT)2 should be switched on in the Server BIOS unless SAP or Microsoft support advise to disable this (in very rare cases). 2.2 Rack versus Blade Integrated Blade Systems such as Cisco UCS can simplify the networking and storage configuration required for building a SAP on Microsoft Private Cloud Infrastructure. Both Blade and Rack based deployments are supported. There are some differences between Rack and Blade servers: 1. Rack based servers are typically released first when new processors are available 2. Rack based servers typically have more DIMM slots (typically 24) allowing use of cheaper 16GB DIMMs 3. Blade based systems have network and/or storage aggregation technologies – these should be tested for performance and reliability in conjunction with technologies such as RSS, SRIOv, VMQ 4. Blade based systems may have “intelligent” network functions that are able to route network traffic from one VM and/or Physical Server to another VM and/or Physical Server within the same Blade Chassis. This may improve network performance and stability by bypassing an additional network hop to the rack or core switch 5. Blade based systems with converged SAN and Network infrastructure will usually allocate either 4Gbit/sec or 8Gbit/sec of bandwidth to a FCoE adapter for SAN connectivity. This bandwidth is removed from the pool available for network connectivity When purchasing new Hardware ensure the vendor guarentees that the Blade or Rack based networking is fully compatible with SRIOv, RSS, VMQ in combination with Windows 2012 Hyper-V 3.0. Firmware and/or Driver updates may be required 2 SAP NetWeaver and Hyper-Threading on Windows Servers : to be or not to be SAP on Microsoft Private Cloud Page 6 of 37 Created: 19-Mar-16 Hardware, Network & SAN 2.3 Network 10 Gigabit network is mandatory. Microsoft does not support the use of 1 Gigabit network infrastructure for SAP on Windows Private Cloud. The network between the SAP application servers and the database server (SAP Internal Network), LiveMigration Network and iSCSI Network (if used) must be 10 Gigabit. The use of 1 Gigabit Network for these connections invalidates the design and is strongly advised against. 1 Gigabit Network cards may be used for purposes such as a Management Network. 2.3.1 How Many Networks are Required? The following networks may be required: SAP Internal Network Public Network LiveMigration Network iSCSI Network (if used) Management Network Heartbeat (physical host and/or VM guest cluster) 2.3.2 802.1q VLAN Tagging vs. Many Physical Network Cards 802.1q is a technology to “partition” a physical network card into multiple logical network cards. 802.1q3 requires configuration on the network switch 4 and configuration on the network card or Virtual Network Card. Most modern network cards support 802.1q VLAN Tagging and “Priority”. Priority is a Quality of Service (QoS) concept whereby a specific VLAN can configured in the network settings properties for each VM. It is generally recommended to use 802.1q VLAN Tagging thereby reducing the number of physical network cards required. 3 http://en.wikipedia.org/wiki/IEEE_802.1Q 4 http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1 a/AccessTrunk.html#wp1206629 SAP on Microsoft Private Cloud Page 7 of 37 Created: 19-Mar-16 Hardware, Network & SAN iSCSI Source Onboard NIC 5 2 x 1G Team 1: NIC 1/Port 2 and NIC 2/Port 2 are Teamed with LACP load balancing on the physical Hyper-V Host. No IP address is assigned to this Team VLANs running on this team are: SAP Internal Network VLAN = 40 Public Network VLAN = 50 Heartbeat Network for Guest VLAN = 60 Management Network for Guest VLAN = 70 Onboard NIC 6 2 x 1G Dual Port NIC 2 2 x 10G Dual Port NIC 3 2 x 10G Dual Port NIC 1 2 x 10G Dual Port NIC 4 2 x 10G Team 2: NIC 1/Port 1 and NIC 2/Port 1 are Teamed with LACP load balancing on the physical Hyper-V Host. Jumboframes set to 9014 Static IP address is assigned to this Team VLANs running on this team are: iSCSI_1 VLAN = 80 iSCSI_2 VLAN = 90 Live_Migration VLAN = 100 Rack Switch 1 Rack Switch 2 Rack Switches in HA Configuration Core Switch(es) Teamed trunks to Core Switch Network teaming in Windows 2012 is explained in detail in this MSDN Blog SAP on Microsoft Private Cloud Page 8 of 37 Created: 19-Mar-16 Management Interface: Dedicated Network Card for Management recommended. Recommend against teaming. This interface is to be used for Hyper-V cluster Heartbeat Static IP address is assigned to this Interface. RSS and SRIOv disabled on this network card. 1Gigabit Ethernet sufficient Hardware, Network & SAN 2.4 Fibre Channel, FCoE, SMB 3.0 or iSCSI Windows 2012 Hyper-V 3.0 supports vHBA for Fibre Channel, iSCSI, SMB 3.0 and FCoE Each Hyper-V cluster server must have adequate number of connections (HBA, iSCSI etc) and effective MPIO software to balance the IO requests proportionately across all available ports. SMB 3.0 configurations will be covered in a later document after successful customer pilot deployments of this technology. Correct network configuration and high performance network cards and drivers become critical for SMB 3.0. It is recommended to use RDMA for SMB 3.0 systems. 2.5 SAN & LUN/Disk Design See Chapter 6 for the details on how many LUNs, Cluster Shared Volumes, Virtual Hard Disks and SQL Data files to configure 2.6 FusionIO Cards FusionIO cards can be used to place the VHDX holding the SQL Server transaction log. Large configurations or where performance is more critical should consider using internal server SSD. 2.7 Network Cabling CAT6a or CAT7 Twinax SFP cables or Fibre Optical cables are supported. CAT6 or earlier is not permitted. 2.8 Native 4K Disk Formats SQL Server requires a 64k NTFS Allocation Unit Size. VHDX or disks presented directly via the Virtual Fibre Channel Adaptor (vHBA) 5 should be formatted in Disk Management with a 64k NTFS AUS regardless of the disk type (512k, 512e or Native 4k). The file system holding the VHDX should be formatted with the default NTFS Allocation Unit Size of 4k. There performance benefits from using more modern disks that support native 4K alignment. It is highly recommend to request H/W vendors to supply Native 4K disks when they become mainstream. Further information: Microsoft support policy for 4K sector hard drives in Windows SQL Server - New Drives Use 4K Sector Size Transition to Advanced Format 4K Sector Hard Drives Why Windows Server 2012 Hyper-V VHDX 4K Alignment Is So Important AlwaysOn is the recommended HA technology. Support of traditional SQL Server cluster is for reverse compatibility reasons only. New installations should use AlwaysOn 5 SAP on Microsoft Private Cloud Page 9 of 37 Created: 19-Mar-16 Hardware, Network & SAN 3 Windows 2012 Hyper-V 3.0 Installation & Configuration After the SAP on Microsoft Private Cloud Instructure has been commissioned the installation and configuration of the Windows 2012 Hyper-V 3.0 can begin. 3.1 Windows 2012 Installation Windows 2012 x64 Datacenter Edition or later should be installed onto the physical hosts. Windows 2012 Standard Edition does not offer the virtualization features required such as a large number of active VMs. Windows 2012 Standard Edition is suitable as the Guest OS. 3.1.1 Windows Core versus Windows Full GUI Windows 2012 can be installed in Full GUI mode, Hybrid Mode or Core mode. Core mode has no GUI, Internet Explorer or other features. The benefit of this is that patching requirements is dramatically lowered. Windows 2012 is able to install and remove the GUI interface as required. All Windows administration tasks can be performed remotely or via command line. However some customers may be more familiar with Windows Full GUI, therefore it may be useful to install the full GUI and then remove the GUI once the Microsoft Private Cloud is setup, tested and ready for go live. It is recommended to use Windows Core on the Hyper-V cluster nodes 3.2 Add Hyper-V & Cluster Features SAP on Microsoft Private Cloud requires Hyper-V feature and Windows Cluster feature. Hyper-V feature is required to create VMs and the cluster feature is required for the Cluster File System Systems using iSCSI will require the File Services Role -> iSCSI Target Server SAP on Microsoft Private Cloud Page 10 of 37 Created: 19-Mar-16 Hardware, Network & SAN 3.3 Configure Virtual Network Switch 3.3.1 Private, Internal and External Networks In the Hyper-V architecture, guest operating systems do not have direct access to hardware resources. In Hyper-V the management OS controls the I/O (input / output) of all physical devices. Guest OSs have high speed “hardware” devices installed on the VM, which are presented to the guest OS in the same manner as if it was physical hardware. These high speed devices run in a Virtualization Service Client (VSC). When hardware requests are made within the guest OS, they are redirected by the VSC via a high-speed interconnect, unique to each VM, known as a Virtual Machine Bus (VMBus), which then sends it to the VSP (Virtualization Service Provider) in the management OS. The management OS then manages the requests to the physical hardware. The correlating physical hardware response is then redirected back to the guest OS in the same manner. If only communication between virtual machines is needed and not with the physical machine or the external network, create a private virtual network. If only communication between virtual machines and the physical machine is needed, create an internal virtual network. If the virtual machines need to communicate with the entire network or the Internet, create an external network. SAP on Microsoft Private Cloud Page 11 of 37 Created: 19-Mar-16 Hardware, Network & SAN 4 Microsoft Private Cloud Solution for SAP – VM Configuration Microsoft Private Cloud Solution for SAP systems follows standard best practices for the Guest Virtual Machines. More details are available in the HA/DR chapters. In general Windows 2012 is the recommended guest operation system unless a specific SAP component is not available on Windows 2012. Windows 2008 R2 has only been tested on a limited number of SAP components that were not supported on Windows 2012. Windows 2003 is strongly advised against as this product is obsolete. 4.1 Sizing See Chapter 9 on Landscape Design 4.2 High Availability & Disaster Recovery See HA/DR Chapter 4.3 Supported Guest Operating Systems Windows 2012 x64 Standard Edition or higher is the only Operating System tested thoroughly as a guest. Windows 2008 R2 should only be used for SAP on Microsoft Private Cloud reference acrchitecture if the SAP component is not supported on Windows 2012. There are a few rare cases where some SAP components must run on 32 bit operating systems and/or Windows 2008. SAP does not support 32 bit virtual environments. If a SAP component requires this please open an OSS message. Standard SAP ABAP, Java and Business Objects systems are all supported on and should run on Windows 2012 guests operating system. 4.4 Support for Guest Operating System Clustering SAP supports creating a cluster between two or more Virtual Machines that are running on a Windows Hyper-V 3.0 Cluster. This is documented in SAP Note 1374671 - High Availability in Virtual Environment on Windows As discussed in the note the Virtual Machines must be configured such that the cluster nodes never run on the same physical host. http://msdn.microsoft.com/en-us/library/aa369651(VS.85).aspx http://support.microsoft.com/kb/296799 See HA/DR Chapter for more information on Host and Guest clustering options SAP on Microsoft Private Cloud Page 12 of 37 Created: 19-Mar-16 Hardware, Network & SAN 5 Network Configuration Network configuration is critical to ensuring the performance and stability of a Microsoft Private Cloud Solution for SAP. This section deals with the different networks at the different layers of the solution and their optimal configuration. There are at least four key topics: 1. Configuration of the physical networks on the HyperVisor host 2. Configuration of the Virtual Switches in the Hyper-V Administration tool 3. Configuration of the Virtual Network Cards (vNIC) inside the VM 4. Special configuration for clustered and iSCSI based systems The network configurations and naming standards must be followed to allow Microsoft support resources to quickly and easily troubleshoot network problems. To rename the networks right click and select “Rename”. 5.1 Physical Server Network Configuration (HyperVisor Host) The physical servers running Windows 2012, Hyper-V and Windows Cluster software must have at least 5 network interfaces (at least 4 ports that are 10G and one 1G). Additional network interfaces may be needed of systems with very high IO on using iSCSI based storage6. Note: Quad port and Dual Port network cards are common as are Converged Network solutions. Generally two or three physical network cards is sufficient to provide enough interfaces and ports even after teaming. The diagram on page 8 details the network teams and physical network configuration. Network Team 1.Public 2.SAP_Internal 3.Heartbeat 4.Management 5.iSCSI 6.Live_Migration 1 1 1 1 2 2 RSS. Rings/Queue Yes Yes No No Yes Yes Jumbo Frames No No No No Yes* Yes* Binding Order 1st 2nd 3rd 4th - Cluster Traffic No No Yes, no client No No No Band width 1G+ 4-10G 1G or less 1G or less 2 x 10G+ 4G+ * Jumbo Frame Size should be 9014 Care must be taken when setting network bindings order. Refer to Note 1841151 - Windows Server 2012 running in a VM - different NIC order A typical deployment on a Hyper-V Physical Host would resemble the below (iSCSI based system). Typically the 802.1q VLAN tagging and QoS would be sufficient for almost all customers. Customers with very high IO requirements are recommended to use FusionIO cards inside the server for the transaction log. This has proven to reduce the IOPS significantly. 6 SAP on Microsoft Private Cloud Page 13 of 37 Created: 19-Mar-16 Hardware, Network & SAN Note: After the vSwitch is bound to the Team the name will be changed. The screenshot below shows the physical host network configuration after the vSwitch has been bound to Team1. It is strictly mandatory that the physical network cards, configuration and naming standard is identical on all Hyper-V hosts. LiveMigration and other functionality will not function correctly without this. Note: Creating a team prevents Windows 2012 Hyper-V from providing an SRIOv virtual function through to the Virtual Machine. Very large systems will benefit from SRIOv and can consider teaming within the guest VM. Most small and medium systems will perform well without SRIOv 5.1.1 Team 1 Do not assign an IP to Team 1. All that is required is to create a vSwitch (see section 5.2). The networks that exist on Team 1 are all 802.1q VLANs and are assigned later in section 5.3 5.1.2 Team 2 Assign an IP to Team 1. If iSCSI is required within a VM (shared disk clustering using iSCSI is used) then create a vSwitch (section 5.2). Ensure that “allow management OS to share this network adapter” is ticked. If iSCSI based clustering within a guest VM is not used it is not required to create a vSwitch 5.1.2.1 iSCSI Network (if used) Bandwidth required = 2 x 10 Gigabit or more. Extremely low latency. Network Name = “iSCSI_1” and “iSCSI_2” Binding on Physical Host = 2 VLAN Tagging = 80 and 90 (if iSCSI is used for shared disk storage within a Guest VM cluster add VLAN tags) SAP on Microsoft Private Cloud Page 14 of 37 Created: 19-Mar-16 Hardware, Network & SAN iSCSI is already a popular solution for small and medium sized virtual environments. The performance7 of correctly configured iSCSI solutions is more than sufficient for almost all SAP customers. Correctly configured network cards, firmware, drivers and Windows configuration is critical for achieving good iSCSI performance. Un-bind Microsoft network client from these NIC. RSS must be configured correctly and Jumbo Frames must be set to 9000 on the network card, switch and iSCSI target. 5.1.2.2 LiveMigration Network Bandwidth required = 4 Gigabit or more. Network Name = “Live_Migration” Binding on Physical Host = 2 VLAN Tagging = none A high speed network needs to be provisioned to handle the replication of memory contents during a Live Migration. Live Migration performance is directly linked to the performance of the network used to replicate memory contents. Jumbo Frames will improve throughput This network should be dedicated and assigned in: Hyper-V Manager -> Hyper-V Settings… -> Live Migrations -> Incoming live migrations 7 https://msevents.microsoft.com/CUI/EventDetail.aspx?culture=en-US&EventId=1032432957 SAP on Microsoft Private Cloud Page 15 of 37 Created: 19-Mar-16 Hardware, Network & SAN 5.2 Virtual Switches on Physical HyperVisor Host The Virtual Switch “abstracts” the physical network card and copies the physical network card. In the screenshot below the physical network card is a HP NC362i and has been copied to Hyper-V Virtual Ethernet Adapter. The original physical network adapter no longer has direct network connectivity. The table below shows the settings and configuration required for each Hyper-V Switch. All virtual switches should be created as External for the Microsoft Private Cloud Solution for SAP. Network 1.Team1 2.Team2 Allow Management OS to Communcate No Yes Connection Type External (Team 2 x 10G) External (Team 2 x 10G) In Hyper-V Manager -> Virtual Switch Manager, add Team1 and Team2. Enable SRIOv on both Team1 and Team2 vSwitch. Select the “Multiplexor Driver”. This configuration must be identical on all physical Hyper-V hosts. Do not allow management operating system to use either Team1. The management operating system must be able to access the vSwitch on Team2. Note: SRIOv will not work on teamed connections. If network is a bottleneck then it is recommended to utilize teaming within a VM. SRIOv must be selected whilst the vSwitch is being created and cannot be configured after the vSwitch has been created. SAP on Microsoft Private Cloud Page 16 of 37 Created: 19-Mar-16 Hardware, Network & SAN 5.3 Virtual Network Cards (vNIC) on Virtual Machine The final layer in the Microsoft Private Cloud Solution for SAP network solution is the presentation of the Virtual Network Switches defined in section 5.2 to the Virtual Machine. The table below states the name and properties for each network switch. Network VMQ 1.Public 2.SAP_Internal 3.Heartbeat 4.Management Yes Yes No No RSS. Rings/Queue Yes Yes Yes Yes Jumbo Frames No No No No Binding Order 1st 2nd 3rd 4th Cluster Traffic VLAN Tag No No Yes, no client No 40 50 60 70 To add a vNIC to a Virtual Machine Failover Cluster Manager must be used and not Hyper-V Manager. This is because all the VMs in the Microsoft Private Cloud Solution for SAP can run on any of the hosts. Open Failover Cluster Manager -> right click on each VM and select Settings -> Add Hardware -> Network Adapter. Configure the VLAN Tag and other details as per the details below SAP on Microsoft Private Cloud Page 17 of 37 Created: 19-Mar-16 Hardware, Network & SAN Note: SRIOv will not function with host based network teaming. If SRIOv is required utilize teaming inside the VM Guest OS 5.3.1 Public Network Bandwidth required = 1 Gigabit sufficient in most cases. Extremely low latency between application servers and message server. SAPGUI and RFC client traffic may be WAN link. Physical Network Name = “1.Public” Binding order in VM = 1 VLAN Tagging = 40 The Public Network is the interface that all clients communicate to the SAP application servers. This interface must be the highest in the binding order 5.3.2 SAP Internal Network Bandwidth required = 10 Gigabit or more. Extremely low latency. Network Name = “2.SAP_Internal” Binding order in VM = 2 VLAN Tagging = 50 Microsoft already document8 how to setup and configure a dedicated network to handle traffic between SQL Server and SAP application servers. A dedicated 10 Gigabit network for SAP Internal traffic is mandatory. 1 Gigabit network is not supported for this purpose 5.3.3 Heartbeat Network 8 How to Setup a Dedicated SAP Application Server to DB Server Network SAP on Microsoft Private Cloud Page 18 of 37 Created: 19-Mar-16 Hardware, Network & SAN Bandwidth required = 1 Gigabit sufficient. Extremely low latency Network Name = “3.Heartbeat” Binding order in VM = 3 VLAN Tagging = 60 The heartbeat network is used by Hyper-V guest clusters to communicate between the cluster nodes. This network is used by the ASCS cluster between the two VMs. 5.3.4 Management Network (for Guest VM) Bandwidth required = 1 Gigabit sufficient. Latency not critical Network Name = “4.Management” Binding order in VM = 4 VLAN Tagging = 70 The Management network is used to administer and to Terminal Services to the VM. This physical Hyper-V clusters do not use this network and have a dedicated network card for Management and Physical Hyper-V cluster heartbeat The screenshot below shows the networks presented to a VM. The VM below is a Database server running SQL 2012 AlwaysOn and therefore requires a Heartbeat network. A VM that was a standalone server (no clustering) does not require a Heartbeat network within the VM. SAP on Microsoft Private Cloud Page 19 of 37 Created: 19-Mar-16 Hardware, Network & SAN 5.4 Cluster Network Configuration for Highly Available VMs Configuration and naming standards on each physical cluster node must be identical to ensure that Live Migration and other cluster features work correctly. If network configuration is asymmetric errors such as the below will be observed SAP on Microsoft Private Cloud Page 20 of 37 Created: 19-Mar-16 Hardware, Network & SAN 6 Network Tuning SAP applications require high performance and stable networks. In recent years new technologies have evolved to offload specific activities from the operating system and/or hypervisor. Some of these are discussed below. Hardware vendors are responsible for ensuring that all the technologies listed as mandatory in this documentation are supported by the network cards. 6.1 SRIOv SRIOv is a technology that allows a single PCI card such as a Network Card present as multiple virtual “devices” to virtual machines. SRIOv dramatically improves performance. SAP and Microsoft only support Network Cards with drivers and firmware that function correctly with Hyper-V 3.0 and SRIOv. Customers must ensure Hardware vendors provide SRIOv network cards. SRIOv must be enabled in BIOS, on the network card and inside the settings for each VM. Diagram 1. shows a Network Card and VM without SRIOv. SAP on Microsoft Private Cloud Page 21 of 37 Created: 19-Mar-16 Hardware, Network & SAN Diagram 2. shows a Network Card and VM with SRIOv. The VM is able to directly access the Network Card. Advanced Topic: Windows 2012 integrated network teaming (LBFO) can be configured. 6.2 VMQ VMQ enabled devices can route frames more efficiently using DMA. SAP and Microsoft only support Network Cards with drivers and firmware that function correctly with Hyper-V 3.0 and VMQ. Customers must ensure Hardware vendors provide RSS network cards. 6.3 RSS Receive Side Scaling9 is a technology to parallelize the network processing onto multiple Logical Processors within a VM and on the host operating system. Correct functioning of RSS is essential for good performance. SAP and Microsoft only support Network Cards with drivers and firmware that function correctly with Hyper-V 3.0 and RSS. Customers must ensure Hardware vendors provide RSS network cards. http://blogs.msdn.com/b/saponsqlserver/archive/2012/01/12/network-settings-networkteaming-receive-side-scaling-rss-amp-unbalanced-cpu-load.aspx 9 SAP on Microsoft Private Cloud Page 22 of 37 Created: 19-Mar-16 Hardware, Network & SAN 6.4 IP SEC Customers wishing to deploy IPSEC should purchase IPSEC PAS offload network cards. Contact the author of this whitepaper before deploying IPSEC 6.5 Jumbo Frames (for iSCSI only) Standard Ethernet frame size is 1500 bytes. This is too small for iSCSI implementations and larger values should be configured to reduce the number of frames transmitted and received but increasing the size of each frame. Typical values are 9000 bytes. At high IO rates CPU decrease of 30% has been seen with Jumbo Frames. This needs to be configured on any Network Card that is used for iSCSI traffic, any switch in between the Network Card and the iSCSI source and the iSCSI device itself. The setting must be identical on all interfaces. Jumbo Frames does not benefit SAP workloads other than iSCSI traffic 6.6 Disabling Netbios & DNS Client Registration ◦ On the guest NIC disable NetBIOS and disable DNS client name registration in the IPv4 networking settings. ◦ In the Network Connections, Advanced Settings, set the host NIC as the primary connection, and disable the File and Printer Sharing and Client for Microsoft Networks bindings on the guest NIC. SAP on Microsoft Private Cloud Page 23 of 37 Created: 19-Mar-16 Hardware, Network & SAN 7 Energy & Power Savings Power Savings settings and configuration in an SAP environment is discussed in detail in an existing Microsoft blog10. During deployment and testing it may be useful to set the Power Savings feature to “High Performance”. All network interfaces should have the following setting disabled. iSCSI interfaces must have this setting disabled or the iSCSI interface may be unstable 10 Effect of Windows Power Mode on SAP Netweaver Applications SAP on Microsoft Private Cloud Page 24 of 37 Created: 19-Mar-16 Hardware, Network & SAN 8 Software Releases Required The following software components are required in the Microsoft Private Cloud Solution for SAP. 8.1 Windows Release Windows 2012 Datacenter Edition is required on the physical hosts to support many active VMs. The Virtual Machines themselves may run Windows 2012 Standard Edition. Datacenter Edition and Standard Edition are identical in the amount of RAM and processors supported, the principal difference is the virtualization capability11. The latest Cumulative Update for Windows 201212 must be installed. It is strongly recommended to regularly review the KB on this link containing Hyper-V fixes. 8.2 Database Release SQL Server 2012 Service Pack 1 or higher is documented and tested in this whitepaper13. Standalone components such as Content Server and LiveCache are supported on Hyper-V and are documented in Note 1492000 - General Support Statement for Virtual Environments 8.3 SAP Monitoring Agents Enhanced Monitoring agents must be installed and configured on all Virtual Machines including dedicated SQL Server database servers. The following note must be fully implemented Note 1409604 14 - Virtualization on Windows Enhanced monitoring. It is recommended to use this PowerShell script 15 to implement the steps rather than manually configuring each VM. 8.4 SAP Product Availability Matrix SAP publishes release compatibility information in the Product Availability Matrix (PAM) 16 . The Microsoft Private Cloud Solution for SAP is based upon Windows 2012 and SQL 2012, both of which are supported by SAP. There are no separate PAM entries for the Microsoft Private Cloud Solution for SAP as the versions and releases of SQL Server and Windows are already Generally Available and documented in the PAM. Support of virtualized systems is documented in SAP Note 1492000 http://download.microsoft.com/download/0/4/B/04BD0EB1-42FE-488B-919F3981EF9B2101/WS2012_Licensing-Pricing_Datasheet.pdf 11 12 http://support.microsoft.com/kb/2756872 13 Check Note 1492000 for other products/vendors. 14 Note 1409604 - Virtualization on Windows: Enhanced monitoring http://blogs.msdn.com/b/saponsqlserver/archive/2012/07/31/how-to-enable-sap-quotenhanced-monitoring-quot-via-powershell.aspx 15 16 https://service.sap.com/pam SAP on Microsoft Private Cloud Page 25 of 37 Created: 19-Mar-16 Hardware, Network & SAN 9 Appendix 1 – FCoE, iSCSI & Fibre Channel Diagram shows the different protocols for storage systems17. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11495142_ns1060_Networking_Solutions_White_Paper.html 17 SAP on Microsoft Private Cloud Page 26 of 37 Created: 19-Mar-16 Hardware, Network & SAN 10 Appendix 2 – Screenshots Showing Sample Configuration Screenshot showing configuring Jumbo frames on the physical Hyper-V host or on a Guest VM cluster. SAP on Microsoft Private Cloud Page 27 of 37 Created: 19-Mar-16 Hardware, Network & SAN 11 Appendix 3 – Screenshots Showing SRIOv Configuration Version 0.5 Juergen Thomas 11/06/2012 The goal of this chapter is to help you get an understanding of performance of your Hyper-V deployment as well as make an initial triage of the situation when a performance issue is reported. Typically, this process can be described in three steps as follows: 11.1 Step 1: Collect Network configuration information The goal in this step is to collect data such that one can describe the deployment in detail (without missing any aspects). It is imperative to collect information on Hyper-V host such as: a. Make/Model of the hardware (Ex: OEM name, processor number, RAM) b. Storage hardware and configuration (Ex: RAID, Size, MPIO) c. Network configuration (Ex: Number of network cards, 1G or 10G, number of virtual switches, Virtual Machine Queue (VMQ) or SR-IOV enabled, number of VMQ processors). d. VM configuration (Ex: Number of Virtual Processors, Memory, Static or Dynamic Memory, Operating System, Integration Component version) e. VHD (Ex: Size, Type, location) f. Application or software running on management partition (besides Hyper-V server) g. Hyper-V Server Operating System version 11.2 Checking the usage of Network accelerators on the Hyper-V host There are two different technologies which got introduced in Windows Server 2008R2 and Windows Server 2012 to speed up network handling and to gain network throughput between host machine and VMs running on the host server. The first technology is called VMQ and is explained a bit in this article: http://technet.microsoft.com/en-us/library/gg162704(v=ws.10).aspx The other one is called SR-IOV and is explained in more detail in this article: http://technet.microsoft.com/en-us/library/hh831389.aspx or even in more details in this series of Blogs: http://blogs.technet.com/b/jhoward/archive/2012/03/12/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-1.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/13/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-2.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/14/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-3.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/15/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-4.aspx SAP on Microsoft Private Cloud Page 28 of 37 Created: 19-Mar-16 Hardware, Network & SAN http://blogs.technet.com/b/jhoward/archive/2012/03/16/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-5.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/19/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-6.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/20/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-7.aspx http://blogs.technet.com/b/jhoward/archive/2012/03/21/everything-you-wanted-to-knowabout-sr-iov-in-hyper-v-part-8.aspx Let’s first check the abilities of the NICs to run VMQ or SR-IOV In order to do so run the following commands in a PowerShell which got opened as administrator: Get-NetAdapterVmq The result expected could look like: BaseVmqProce NumberOfReceiv Name InterfaceDescription Enabled ssor MaxProcessors eQueues HP Ethernet 10Gb 2Ethernet port 560FLB... TRUE 0:00 16 63 HP Ethernet 10Gb 2Ethernet2 port 560FLB... TRUE 0:00 16 In this case the second adapter was not connected to the network. Therefore no Receive Queues are shown. Means the adapter is able to deal with VMQ. Now let’s check whether the adapter is able to deal with SR-IOV. In order to get that status, you need to issue this command in PowerShell (again started ‘as administrator’) Get-NetAdapterSriov The result could look like: Name : Ethernet InterfaceDescription : HP Ethernet 10Gb 2-port 560FLB Adapter Enabled : True SriovSupport : NoOscSupport SwitchName : DefaultSwitchName NumVFs : 62 Important: If the NIC could leverage SR-IOV, we would expect it state ‘Support’ in the column ‘SeiovSupport’ like this one: PS C:\Users\Administrator> Get-NetAdapterSriov Name : Ethernet 4 InterfaceDescription : HP Ethernet 10Gb 2-port 560FLB Adapter Enabled : True SriovSupport : Supported SwitchName : DefaultSwitchName NumVFs : 62 SAP on Microsoft Private Cloud Page 29 of 37 Created: 19-Mar-16 Hardware, Network & SAN Reading in this article: http://technet.microsoft.com/en-us/library/jj130915.aspx, the message given in the first output, means that the BIOS of the server seem not to have SR-IOV enabled. Ideally it gets enabled from its default state of disabled in the BIOS like shown below: The first step would be to enable it in the BIOS since like above, the default setting might be ‘disabled’. In order to know whether you need a BIOS upgrade, please check with your hardware vendor. VMQ does not need settings in the BIOS. As a next step it has to be made sure that SR-IOV and/or VMQ is enabled for the particular NIC. To check go into the properties of the NIC and check whether SR-IOV or VMQ respectively is enabled SAP on Microsoft Private Cloud Page 30 of 37 Created: 19-Mar-16 Hardware, Network & SAN When you create the virtual switch, you want to enable both optimizations. Please be aware that SR-IOV needs to be enabled when creating the switch. It can’t be changed after the Virtual Switch had been created. After you created the Virtual Switch or switches on the host server, you want to check which type of optimization is taken and in place. This can be done with Power Shell (Power Shell needs to be started as Administrator) using this command: get-vmswitch | fl * If multiple switches are configured, multiple will be shown. A typical output for a system which is NOT able to use SR-IOV could look like: PS C:\Windows\system32> get-vmswitch | fl * ComputerName : saphypervtest1 Name : saphypervtest1 Adapter1 Id : xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx Notes : SwitchType : External AllowManagementOS : True NetAdapterInterfaceDescription : HP Ethernet 10Gb 2-port 560FLB Adapter AvailableVMQueues : 0 NumberVmqAllocated : 0 IovEnabled : True IovVirtualFunctionCount : 0 IovVirtualFunctionsInUse : 0 IovQueuePairCount : 0 IovQueuePairsInUse : 0 AvailableIPSecSA : 512 NumberIPSecSAAllocated : 0 BandwidthPercentage : 0 BandwidthReservationMode : None DefaultFlowMinimumBandwidthAbsolute : 0 DefaultFlowMinimumBandwidthWeight : 0 Extensions : {Microsoft NDIS Capture, Microsoft Windows Filtering Platform} IovSupport : False IovSupportReasons : {To use SR-IOV on this system, the system BIOS must be updated to allow Windows SAP on Microsoft Private Cloud Page 31 of 37 Created: 19-Mar-16 Hardware, Network & SAN to control PCI Express. Contact your system manufacturer for an update., SR-IOV cannot be used on this system as the PCI Express hardware does not support Access Control Services (ACS) at any root port. Contact your system vendor for support IsDeleted further information., This network adapter does not SR-IOV.} : False The output of a System which is able to use SR-IOV would look like: PS C:\Users\Administrator> get-vmswitch | fl ComputerName : saphypervtest1 Name : saphypervtest1 Adapter1 Id : xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx Notes : SwitchType : External AllowManagementOS : False NetAdapterInterfaceDescription : HP Ethernet 10Gb 2-port 560FLB Adapter AvailableVMQueues : 0 NumberVmqAllocated : 0 IovEnabled : True IovVirtualFunctionCount : 0 IovVirtualFunctionsInUse : 0 IovQueuePairCount : 0 IovQueuePairsInUse : 0 AvailableIPSecSA : 0 NumberIPSecSAAllocated : 0 BandwidthPercentage : 0 BandwidthReservationMode : None DefaultFlowMinimumBandwidthAbsolute : 0 DefaultFlowMinimumBandwidthWeight : 0 Extensions : {Microsoft NDIS Capture, Microsoft Windows Filtering Platform} IovSupport : True IovSupportReasons : {OK} IsDeleted : False When you configure a VM running on this host where we know that SR-IOV is supported by the BIOS of the hardware, by the NIC and configured to be leveraged by the Hyper-V Virtual network Switch, it also needs to be enabled on a VM basis. As soon as a VM is connected to a Hyper-V Virtual Switch using the VM Settings dialog, one needs to check a box in the ‘Hardware Acceleration’ section of the NIC as shown below: SAP on Microsoft Private Cloud Page 32 of 37 Created: 19-Mar-16 Hardware, Network & SAN Enabling/Disabling of SR-IOV can be done online and does not interrupt the network connectivity to the VM. After this step we now want to check the settings of the virtual network adapters of the different VMs started on the host. We can do that with this PS command: get-vmnetworkadapter <VM-Name> | fl * The output of a VM where SR-IOV is not used, but VMQ is enabled would look like: PS C:\Windows\system32> get-vmnetworkadapter saphekadb | fl * IovWeight IovQueuePairsRequested IovQueuePairsAssigned IovInterruptModeration IovUsage VirtualFunction IsLegacy IsManagementOs IsExternalAdapter Id AdapterId DynamicMacAddressEnabled MacAddress MacAddressSpoofing SwitchId Connected PoolName SAP on Microsoft Private Cloud : : : : : : : : : : : : : : : : : 100 1 0 Default 1 False False False Microsoft:848093EE-…………………………………………………………. fcb42e6c-…………………………………………. True xxxxxxxxxxxx Off 9ab9b331-……………………………………… True Page 33 of 37 Created: 19-Mar-16 Hardware, Network & SAN SwitchName : saphypervtest1 Adapter1 AclList : {} DhcpGuard : Off RouterGuard : Off PortMirroringMode : None IeeePriorityTag : Off VirtualSubnetId : 0 AllowTeaming : Off VMQWeight : 100 IPsecOffloadMaxSA : 512 VmqUsage : 1 IPsecOffloadSAUsage : 0 VFDataPathActive : False VMQueue : MSFT_NetAdapterVmqQueueSettingData: …………………………………………… MandatoryFeatureId : {} MandatoryFeatureName : {} VlanSetting : Microsoft.HyperV.PowerShell.VMNetworkAdapterVlanSetting BandwidthSetting : BandwidthPercentage : 0 TestReplicaPoolName : TestReplicaSwitchName : StatusDescription : {OK} Status : {Ok} IPAddresses : {xxx.xxx.xxx.xxx} ComputerName : saphypervtest1 Name : Network Adapter IsDeleted : False VMId : 848………………………………………………. VMName : saphypervvmt2 VMSnapshotId : 00000000-0000-0000-0000-000000000000 VMSnapshotName : If IovWeight and IovUsage are remaining at values of 0, the reason usually is that the VF (Virtual Function) Driver in the VM is not installed. This is done by the step of enabling SRIOV for the specific VM in the Hardware Acceleration Setting. In regards to details on the VF Driver, please check this article: http://blogs.technet.com/b/jhoward/archive/2012/03/14/everything-you-wanted-to-know-aboutsr-iov-in-hyper-v-part-3.aspx You also can cross check the fact on whether the VF driver is enabled or installed in the VM by opening the Device manager in the OS running in the VM. The normal picture w/o support of SR-IOV within the VM would look like: SAP on Microsoft Private Cloud Page 34 of 37 Created: 19-Mar-16 Hardware, Network & SAN Having the VF driver for SR-IOV support enabled in the VM, we would look at the following picture: You realize that there is a third Network Adapter Entry which is ‘Intel® 82599 Virtual Function. Basically the SR-IOV counterpart of the SR-IOV Physical Device on the Hyper-V host. SAP on Microsoft Private Cloud Page 35 of 37 Created: 19-Mar-16 Hardware, Network & SAN 12 Appendix 4 - Comparison of Hyper-V versions Windows Server 2008 Processor 16 LPs HW Logical Support Physical Memory Support Cluster Scale Virtual Machine Processor Support VM Memory Live Migration Live Storage Migration Servers in a Cluster VP:LP Ratio SAP on Microsoft Private Cloud Windows Server 2008 R2 64 LPs Windows Server 2012 320 LPs 1 TB 16 Nodes up to 1000 VMs Up to 4 VPs 1 TB 16 Nodes up to 1000 VMs Up to 4 VPs 4 TB 64 Nodes up to 4000 VMs Up to 64 VPs Up to 64 GB Yes, one at a time Up to 64 GB Yes, one at a time Up to 1 TB Yes, with no limits. As many as hardware will allow. No. Quick Storage Migration via No. Quick Storage Migration via Yes, with no limits. As many as SCVMM SCVMM hardware will allow. 16 16 64 8:1 8:1 for Server No limits. As many as hardware will 12:1 for Client (VDI) allow. Page 36 of 37 Created: 19-Mar-16 Hardware, Network & SAN 13 Appendix 5 – Links Page Required OSS Notes: 1732161 - SAP Systems on Windows Server 2012 1811298 - SAP NetWeaver Upgrade/Update on Windows Server 2012 1753578 - Windows Server 2012 in Virtual Environments Blog Articles: Windows Server 2012 Scale-Out File Server for SQL Server 2012 - Step-by-step Installation Channel9: Whitepapers: Windows Server 2012 Server Virtualization http://download.microsoft.com/download/5/D/B/5DB1C7BF-6286-4431-A244438D4605DB1D/WS%202012%20White%20Paper_Hyper-V.pdf Understanding Networking with Hyper-V http://www.microsoft.com/en-us/download/confirmation.aspx?id=9843 Hyper-V: Virtual Networking Survival Guide (en-US) http://social.technet.microsoft.com/wiki/contents/articles/151.hyper-v-virtual-networking-survivalguide-en-us.aspx Configuring Hyper-V Virtual Networking in Windows Server 2012 - "Early Experts" Challenge - Exam 70-410 and 70-417 http://blogs.technet.com/b/keithmayer/archive/2012/11/26/configuring-hyper-v-virtualnetworking-in-windows-server-2012-quot-early-experts-quot-challenge-exam-70-410-and-70417.aspx#.UPSW9-b28ck Forums: http://social.technet.microsoft.com/forums/en/winserverhyperv/threads/ Dell: Dell™ Solutions Overview Guide for Microsoft® Hyper-V™ http://support.dell.com/support/edocs/software/HyperV/en/sov/sova00.pdf Deploying Microsoft® Hyper-V™ with Dell EqualLogic™ PS Series Arrays http://media.community.dell.com/en/dtc/attach/deploying%20microsoft%20hyperv%20with%20ps%20series%20arrays.pdf SAP on Microsoft Private Cloud Page 37 of 37 Created: 19-Mar-16