VSP3868
VMware vStorage
Best Practices
Name, Title, Company
Disclaimer
 This session may contain product features that are
currently under development.
 This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
 Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
 Technical feasibility and market demand will affect final delivery.
 Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
2
Agenda
VMware Storage Best Practices
Storage Environment
Performance Improvements and Tuning
Performance Monitoring
Common Storage Issues
3
VMware Storage Best Practices
Overview
Storage Environment
Business Continuity and Disaster Recovery
Vendor Knowledge and Partnership
Hard Disk Performance
4
Business Continuity and Disaster Recovery
 Backups
• Number and depth based on application
RPO/RTO
• Test your backups
 Support considerations
• Virtual Machine Snapshots are not backups!
• Available backups allow for a wider variety of
options in data loss scenarios
5
Vendor Knowledge and Partnership
 Know your SAN environment
• Open communication channel to your internal storage team
• Follow storage vendor best practice guidelines for ESX(i)
 Know your storage vendor
• Support relationship – proactive versus reactive
• Software/Firmware updates for storage hardware
 Know your storage network or fabric
• HBA and switch software/firmware updates
6
Hard Disk Performance
 Areas of Concern
• Disk subsystem bottlenecks
• Spikes and sustained latency
 Performance versus Capacity
• Disk performance does not scale with drive size
• Larger drives generally equate lower performance
 For example: 1 TB of space is required
for an app
• 2 x 500GB 15K RPM SAS drives = ~300 IOPS
• Capacity needs satisfied, Performance low
• 8 x 146GB 15K RPM SAS drives = ~1,240 IOPS
• Capacity needs satisfied, Performance high
 More spindles generally equals greater performance - check with
your SAN array vendor for technology details
7
VMware Storage Best Practices
Overview
Storage Environment
Performance Improvements and Tuning
vStorage APIs for Array Integration (VAAI)
Multipathing
Storage I/O Control (SIOC)
8
vStorage APIs for Array Integration (VAAI)
 VAAI
• Enables your host to offload specific virtual machine and storage management
operations to compliant storage hardware.
 Performance Benefits
• ESX(i) hosts consume less CPU and memory resources, and less storage
fabric and network bandwidth.
• Improved Migration of virtual machines with Storage vMotion
• Faster deployment and cloning of virtual machines
• Reduces SCSI Reservation conflicts
• Faster zeroing operations (creation of Fault Tolerant VMs)
 Benefits derived through new SCSI primitives
 Full Copy
 Block Zeroing
9
 Atomic Test and Set (ATS) Locking
vStorage APIs for Array Integration (VAAI) – Requirements





ESX(i) version 4.1 or later.
VAAI compatible storage solution
VAAI hardware acceleration is enabled by default on ESX(i)
ESX(i) 4.1 – block devices only (NAS not yet supported)
ESXi 5.0 – NFS and block device support available for vendor
implementation
 Consult storage vendor for array configuration requirements
10
vStorage APIs for Array Integration (VAAI) - Summary
 VAAI
• Is configured by default, can be disabled through host Advanced Settings,
if required by storage vendor
• Storage array must support VAAI SCSI primitives
• Tested VAAI functionality can be reviewed via the vSphere GUI
11
Multipathing - Overview
 Pluggable Storage Architecture - PSA
 Native Multipathing Plugin – NMP
• Storage Array Type Plugin – SATP
• Path Selection Policy – PSP
 Fixed Path
 Most Recently Used
12
 Round Robin
Multipathing – Path Selection Plugins
 Most Recently Used – MRU
• Active/Passive arrays
 Fixed
• Active/Active arrays
• Asymmetrical LUN Unit Access – ALUA
 Round Robin
• Load balance I/O for improved performance
• Uses Active or Optimized paths, not Standby paths
 Vendor/Partner Multipathing Plugin solutions
13
Multipathing – Best Practices
 Consistency
• One PSP per LUN across all ESX(i) hosts
 Configuration
• Match to storage array type (AP, AA, AA-ALUA)
• All LUNs on AP arrays to all ESX(i) hosts through same storage processor
 Multipathing – iSCSI
• One vmkernel port group per physical NIC
• Follow VMware and/or vendor best practice specifics
• VMware iSCSI Configuration Guide (pages 29-51)
• https://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf
14
Storage I/O Control - Overview
 Storage I/O Control
• Monitors I/O latency to datastores at each ESX host sharing a physical device.
• When the average normalized latency exceeds a set threshold (30ms by
default), the datastore is considered to be congested.
 If congested, SIOC distributes
available storage resources to virtual
machines in proportion to their
configured shares.
 Used to determine migration needs
with Storage DRS in ESXi 5.0
15
Storage I/O Control – Usage Scenario
 Without SIOC, Disk shares do not provide a fair distributed load on
a shared VMFS.
16
VMware Storage Best Practices
Overview
Storage Environment
Performance Improvements and Tuning
Performance Monitoring
vCenter/vSphere Client
esxtop/resxtop
Latency Values
Partner Vendor Tools
17
Performance Monitoring – vCenter/vSphere Client
 Performance Charts
• Performance Advanced Chart Options  Datastore/Disk
• Latency and IOPS statistic counters for disks
• Historical availability depends on vCenter Statistics level (vCenter Advanced
Settings)
• Statistics available for ESX(i) host objects
• Datastore
• Read/Write Latency = device average in milliseconds
• Disk
• Physical Device Command/Write/Read Latency = device average in milliseconds
 Additional detail on vCenter Performance Metrics
• vSphere Datacenter Administration Guide, Appendix B, page 265:
• http://www.vmware.com/pdf/vsphere4/r41/vsp_41_dc_admin_guide.pdf
18
Performance Monitoring – vCenter/vSphere Client
19
Performance Monitoring – esxtop/resxtop
 esxtop/resxtop
• local/remote
• More depth than vSphere Client
 Storage statistics
• CMDS/s – I/Ops
• DAVG, KAVG, GAVG – device/kernel/guest latency in milliseconds
• NFS latency, vSphere 4.1 or later
• Statistics for disk device, disk adapter, virtual machine
 Using esxtop to identify storage performance issues
• http://kb.vmware.com/kb/1008205
 Additional detail on esxtop/resxtop
• vSphere Resource Management Guide, Appendix A, page 89
• http://www.vmware.com/pdf/vsphere4/r41/vsp_41_resource_mgmt.pdf
20
Performance Monitoring – esxtop/resxtop
21
Performance Monitoring – Latency Values
 Good latency values are subjective based on application
performance needs and storage environment capabilities
 Lower is better!
 General rules of thumb:
• 10 milliseconds or less is adequate
• 20 milliseconds or higher, sustained, should be investigated
• Sustained spikes in latency may need to be investigated
 Commands not acknowledged by the SAN within 5000 ms will be
aborted
• This may lead to performance issues as the commands are retried
• Multiple aborts should be investigated
22
Performance Monitoring – Partner Vendor Tools
 Partner Tools
• vSphere APIs
 vCenter Plugins
• Consult with storage vendor
23
VMware Storage Best Practices
Storage
Storage Environment
Performance Improvements and Tuning
Performance Monitoring
Common Storage Issues
Snapshots
Misconfiguration
Improper Device Removal
24
Common Storage Issues – VM Snapshots
 Best Practices
• Snapshots are not backups
• Regular monitoring
• Configure vCenter Snapshot alarms – KB 1018029
• Limit to 2-3 snapshots in a chain to prevent performance degradation
• Delete all snapshots before Virtual Machine disk changes
• Confirm via command line if uncertainty of snapshot state exists
• As a general rule, use Snapshots for no more than 24-72 hours
 Improvements
• ESX(i) 4.0 U2 – Snapshot deletion takes up less space on disk
• ESXi 5.0 – New functionality to monitor snapshots and provide warning if
snapshots need consolidation.
 Snapshot Best Practices – KB 1025279
 Understanding Virtual Machine Snapshots in ESX – KB 1015180
25
Common Storage Issues - Misconfiguration
 Firmware and Driver issues
• SAN array firmware does not match to ESX(i) version
• HBA, FC switch, physical NIC firmware and/or drivers out of date
• VMware Hardware Compatibility Guide
http://www.vmware.com/resources/guides.html
 Inconsistent Path Selection Policy
• (example 1) Host A, LUN A – MRU; Host B, LUN A – Fixed
• (example 2) Host A, LUN A – vendor MMP; Host B, LUN A - Fixed
 LUN detected as a snapshot
• Missing datastores resolved via force-mount
• Resignature for long-term solution
• Consistent LUN presentation to all hosts from SAN
26
Common Storage Issues – Improper Device Removal
 Improper removal of a physical device containing a VMFS
datastore or RDM
• APD (All Paths Dead) state in ESX(i) 4.x
• Upgrade to 4.0 U2 / 4.1 U1 or more recent
• Follow steps outlined in KB 1015084 when removing a datastore
 Rescan after making changes to the storage environment!
VMFS
VMFS
27
RDM
VMware Storage Best Practices
Summary
Storage Environment
Performance Improvements and Tuning
Performance Monitoring
Common Storage Issues
28
Appendix - VMware Storage Best Practices
 ESX FC SAN configuration guide
• https://www.vmware.com/pdf/vsphere4/r41/vsp_41_san_cfg.pdf
 ESX iSCSI Configuration Guide
• https://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf
 MSCS Configuration Guide
• https://www.vmware.com/pdf/vsphere4/r41/vsp_41_mscs.pdf
 VMware Performance Best Practices 4.1
• http://www.vmware.com/pdf/Perf_Best_Practices_vSphere4.1.pdf
29
COMING DECEMBER 2011
By Mike Laverick
Complete guide to using SRM 5.0 with
both VMware and third-party products.
Available in print and eBook formats and
online through Safari Books Online.
VMWORLD 2011 EXCLUSIVE
Buy the Pre-Publication Edition, Receive the
eBook edition for FREE – a $39.99 Value.
Visit VMworld Bookstore for details.
ISBN: 9780321799920
COMING DECEMBER 2011
By Cody Bunch
Complete with real-world case studies
and code.
Available in print and eBook formats and
online through Safari Books Online.
VMWORLD 2011 EXCLUSIVE
Get a FREE preview chapter from this and
two other VMware Press releases.
ISBN: 9780321799913
PLUS: PRE-ORDER and Save 30%
Visit VMworld Bookstore for details
COMING FEBRUARY 2012
By Mostafa Khalil
Demystifies the "black box" of vSphere
storage.
Available in print and eBook formats and
online through Safari Books Online.
VMWORLD 2011 EXCLUSIVE
Get a FREE preview chapter from this and
two other VMware Press releases.
ISBN: 9780321799937
PLUS: PRE-ORDER and Save 30%
Visit VMworld Bookstore for details