Optimize Oracle RDBMS on
VMware
Guy Harrison
Director, R&D Melbourne
www.guyharrison.net
Guy.harrison@quest.com
@guyharrison
Introductions
Star trek shirt fatality analysis
Red
Yellow
Blue
0
10
20
30
40
Pct
50
60
70
80
Agenda
• Motivations for Virtualization
• VMware ESX resource management:
• Memory
• CPU
• IO
• Paravirtualization (OVM) vs Hardware Assisted
Virtualization (ESX)
• RAC on VMware
10
©2011 Quest Software, Inc. All rights reserved..
Motivations for Virtualization
Server
Consolidation
• Better utilization of server resources
• Reduced power consumption
Manageability
• Fewer physical machines
• Backup, cloning, rapid provisioning
Elastic computing
• Adjust resources on demand
• A complement to the physical “grid”
vision
11
©2011 Quest Software, Inc. All rights reserved..
Resistance to Database virtualization
Performance
• Virtual CPU & IO
• Sharing of virtual resources
Scale
• Large databases too big for a
single VM
• RAC-style clustering problematic
Support
• Oracle’s stance often
misunderstood
• See MyOracleSupport 249212.1
12
©2011 Quest Software, Inc. All rights reserved..
DB virtualization is happening
13
©2011 Quest Software, Inc. All rights reserved..
Oracle virtualization is lagging....
Which of the following do you run in VMs?
None of the Above
Oracle
Java
Sharepoint
Exchange
Apache
Active Directory
IIS
File and Print Servers
SQL Server
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Quest survey of vFoglight users , 2010
14
©2011 Quest Software, Inc. All rights reserved..
ESX Memory management
15
©2011 Quest Software, Inc. All rights reserved..
Managing ESX memory
• ESX can “overcommit” memory
• Sum of all VM physical memory allocations > actual ESX physical memory
• Memory is critical to Oracle server performance
• SGA memory to reduce datafile IO
• PGA memory to reduce sort & hash IO
• ESX uses four methods to share memory:
• Memory Page Sharing
• Memory compression
• “Ballooning”
• ESX swapping
• DBA needs to carefully configure to avoid disaster
16
©2011 Quest Software, Inc. All rights reserved..
Configuring VM memory
VMs Compete for memory in this
range
Relative Memory Priority for this
VM
Minimum Memory for
this VM
Maximum memory for
the VM (dynamic)
Monitoring VM memory
18
©2011 Quest Software, Inc. All rights reserved..
ESX and VM memory
ESX Swap
ESX swap
ESX physical
memory
Effective VM
physical
memory
ESX virtual
memory
VM
VM virtual
memory
ESX Ballooning
ESX Swap
ESX swap
Vmmemctl
“balloon”
ESX physical
memory
Effective VM
physical
memory
VM Swap
Apparent VM
physical
memory
VM
VM Swap
ESX virtual
memory
ESX Ballooning
As memory grows, ESX balloon driver (vmmemctl) forces VM to page out
memory to VM swapfile
ESX Ballooning
• Inside the VM, paging to the
swapfile is observed.
• The guest OS will determine
which pages are paged out
• If LOCK_SGA=TRUE, then the
SGA should not be paged.
ESX Swapping
ESX swap
ESX Swap
Effective VM
physical
memory
ESX physical
memory
VM
VM virtual
memory
ESX virtual
memory
ESX Swapping
ESX swap
ESX Swap
VM
Effective VM
physical
memory
ESX physical
memory
Apparent VM
physical
memory
ESX virtual
memory
ESX Swapping
ESX swaps out VM memory to ESX swapfile
ESX Swapping
• Within the VM, swapping cannot
be detected.
• ESX will determine which
memory pages go to disk
• Usually occurs when VMware
tools are not installed
• Even if LOCK_SGA=TRUE,
SGA memory might be on disk
Avoiding Ballooning and swapping
memory reservations help avoid ballooning or ESX
swapping
Ballooning vs. Swapping
Swingbench workload running on Oracle database – from VMWare whitepaper:
http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
28
©2011 Quest Software, Inc. All rights reserved..
VMware memory recommendations
• Paging or swapping of PGA or SGA is almost always a Very Bad
ThingTM.
• Use memory reservations to avoid swapping or ballooning
• Install VMware tools to allow ballooning instead of swapping
• Set Memory reservation = PGA+SGA+process Overhead
• Be realistic about memory requirements:
• In physical machines, we are used to using all available memory
• In VM, use only the memory you need, freeing up memory for other VMs
• Oracle advisories (or Spotlight) can show you how much memory is
needed
• Reduce VM reservation and Oracle memory targets in tandem to
release memory
29
©2011 Quest Software, Inc. All rights reserved..
ESX CPU management
31
©2011 Quest Software, Inc. All rights reserved..
ESX CPU management
• If more virtual CPUs than ESX CPUs, then vCPUs will
sometimes wait for physical CPU
• Time “stops” inside the VM when this occurs
• For multi-CPU VMs, it’s (nearly) all or nothing.
• A vCPU can be in one of three states:
• Associated with an ESX CPU but idle
• Associated with an ESX CPU and executing instructions
• Waiting for ESX CPU to become available
• Shares and reservations determine which VM wins access
to the ESX CPUs
32
©2011 Quest Software, Inc. All rights reserved..
Configuring VM CPU
VMs compete for CPU in this
range
Shares determine relative CPU
allocated when competing
CPU utilization VM
“CPU Ready” is the
amount of time VM
spends waiting on
ESX for CPU
Inside the VM, CPU
stats can be
misleading
SMP for vCPUs
• ESX usually has to
schedule all vCPUs for a
VM simultaneously
• The more CPUs the
harder this is
• Some CPU is also
needed for ESX
• More is therefore not
always better
(Thanks to Carl Bradshaw for letting me reprint this diagram
from his Oracle on VMWare whitepaper)
ESX CPU performance comparisons
VT enabled
Vs 2 core 1.8 GHz physical machine
36
©2011 Quest Software, Inc. All rights reserved..
Programmatic performance
Standalone Java
Simple C program
ESX 2 VCPU 2.26 GHz VT enabled
Java Stored Proc
ESX 2 CPU 3.5 GHz no-VT
Physical 2 CPU 1.8 GHz
PLSQL compiled
PLSQL
0
10
20
30
Elapsed Time (s)
40
50
NB: Not a benchmark! Just some informal measurements!!
37
©2011 Quest Software, Inc. All rights reserved..
Programmatic performance (2)
ESX 2 VCPU 2.26 GHz VT enabled
72
ESX 2 CPU 3.5 GHz no-VT
333
0
50
100
150
200
250
300
350
Elapsed time Pct relative to Physical CPU adjusted for GHz
38
©2011 Quest Software, Inc. All rights reserved..
ESX CPU recommendations
• Use up to date chipsets and ESX software
• Allocate as few VCPUs as possible to each VM
• Use reservations and shares to prioritise access to ESX CPU
• Monitor ESX Ready time to determine the “penalty” of competing
with other virtual machines
39
©2011 Quest Software, Inc. All rights reserved..
ESX IO management
40
©2011 Quest Software, Inc. All rights reserved..
Typical VMWare disk configuration
IO Resource Allocation
• Disk shares can be used to prioritize IO bandwidth.
• This is poorly implemented prior to vSphere 4.1
Storage IO Control
• Prior to vSphere 4.1:
• disk shares could be used only at the VM level, and only within a single
ESX host
• vSphere 4.1 Storage IO Control (SIOC):
• Manages disk share priorities for all VMs attaching to the same
datastore
• Is triggered by high (“congested”) latency
• Can be enabled globally at the datastore level
• Enables equitable distribution even when set to defaults
43
©2011 Quest Software, Inc. All rights reserved..
Storage IO Control
vSphere 4.1 SIOC
45
©2011 Quest Software, Inc. All rights reserved..
SOIC won’t make up for
a poorly configured IO layout
46
©2011 Quest Software, Inc. All rights reserved..
Performant VMware disk configuration
Optimal configuration
• See “Oracle Database Scalability in VMware® ESX” at
www.vmware.com/oracle
• Each virtual disk directly mapped via RDM to dedicated
RAID 0 (+1) group
41 Spindles!
48
©2011 Quest Software, Inc. All rights reserved..
ESX IO recommendations
• Follow normal best practice for physical disks
• Avoid sharing disk workloads
• Dedicated datastores using VMFS
• Align virtual disks to physical disks?
• Consider Raw Device Mapping (RDM)
• Consider SIOC in vSphere 4.1
• If you can’t optimize IO, avoid IO:
• Tune, tune, tune SQL
• Prefer indexed paths
• Memory configuration
• Don’t forget about temp IO (sorts, hash joins)
49
©2011 Quest Software, Inc. All rights reserved..
Shameless plugs
50
©2011 Quest Software, Inc. All rights reserved..
52
©2011 Quest Software, Inc. All rights reserved..
53
©2011 Quest Software, Inc. All rights reserved..
Paravirtualization vs Hardware Virtualization
54
©2011 Quest Software, Inc. All rights reserved..
Paravirtualization vs “Hardware Virtualization”
• Virtualization is not emulation....
• Where-ever possible, Hypervisor runs native code from
OS against underlying hardware
• Because a virtualized operating system is running
outside privileged x86 “ring 0”, direct calls to hardware
need special handling.
• The three main approaches are:
• Full Virtualization (VMWare on older hardware)
• ParaVirtualization (Xen, Oracle VM)
• Hardware Assisted Virtualization (Intel VT, AMD-V)
55
©2011 Quest Software, Inc. All rights reserved..
Full virtualization
• Hardware calls from the VM
are handled by the hypervisor
by:
VM
• Catching the calls as they occur at
run time
• Re-writing the VM image at load
time (binary translation)
Ring 0
Hypervisor
• Requires no special
hardware
• Supports any guest OS
• Relatively Poor performance
Hardware
• Used by ESX on older chipsets
Hardware Assisted virtualization
Root Mode
Non-Root
Mode
VM
• Intel VT and AMD-V chips
add a non-root mode Ring 0.
• VM can issue instructions
from non-root Ring 0.
• CPU can divert these to
hypervisor
Ring 0
Hypervisor
• No changes to OS required
• Good performance
• Requires modern chipsets
Hardware
Paravirtualization
VM
(domU)
Ring 0
Hypervisor
Hardware
VM
(dom0)
• VM operating system is
rewritten to translate
device calls to “hypercalls”
• Hypercalls are handled by
a special VM (dom0 in
Xen/OVM)
• Good performance but
requires modified VM OS
• Xen can use either
paravirtualization or
hardware assist
RAC and ESX
59
©2011 Quest Software, Inc. All rights reserved..
Paravirtualization, ESX and RAC
• Prior to 11.2.0.2, Oracle relied on paravirtualized kernels to
maintain time synchronization for RAC clusters.
• From 11.2.0.2 Oracle uses Cluster Time Synchronization
Service (CTSS) to maintain clock sync, and this works on
ESX
• Therefore, Oracle supports RAC on Vmware ESX only
from 11.2.0.2 onwards
• See Oracle MySupport Note 249212.1
60
©2011 Quest Software, Inc. All rights reserved..
References
• Latest version of this presentation:
• http://www.slideshare.net/gharriso/optimize-oracle-on-vmware-5271530
• My blog (www.guyharrison.net ):
• http://guyharrison.squarespace.com/blog/2010/2/22/memorymanagement-for-oracle-databases-on-vmware-esx.html
• http://guyharrison.squarespace.com/blog/2010/4/9/esx-cpu-optimizationfor-oracle-databases.html
• http://guyharrison.squarespace.com/blog/2010/7/12/stolen-cpu-on-xenbased-virtual-machines.html
• http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
• http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf
• http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf
61
©2011 Quest Software, Inc. All rights reserved..
COLLABORATE 12
April 22-26, 2012
Mandalay Bay Convention Center
Las Vegas, Nevada, USA
www.collaborate12.org
www.collaborate12.ioug.org