Managing the Capacity and Performance of a
VMware Cluster environment
Presented by: Pete Weilnau
CTO
PERFMAN
pete.weilnau@perfman.com
Agenda
• Quick Architecture Overview
• Cluster Level Performance
• Sources of additional information
2
VMware Architecture
From VMware VI3 brochure – © 2008 VMware, Inc.
3
VMware Clusters
• Must have VirtualCenter to create and manage a “Cluster”
• Without VirtualCenter there are only ESX Servers (Hosts)
• When VirtualCenter is down, cluster benefits don’t exist
• Cluster Benefits:
• Host failover recovery via HA (High Availability)
• Workload balancing via DRS (Dynamic Resource
Scheduling)
4
VMware Cluster
Cluster
From VMware VI3 brochure – © 2008 VMware, Inc.
5
Resource Pools
• A logical abstraction for hierarchically managing CPU and
memory resources
• Used on a standalone host or VMware DRS-enabled
clusters
• Provides resources for virtual machines and child pools
6
VMware Resource Pools
Resource
Pools
From VMware VI3 brochure – © 2008 VMware, Inc.
7
VMware DRS
• Managed by VirtualCenter
• Balances virtual machine load across hosts in a cluster
• Enforces resource policies accurately
• (Reservations, limits, shares)
• Respects placement constraints
• Affinity and anti-affinity rules
• VMotion compatibility
8
VMware HA (High Availability)
• Provides automatic restart of virtual machines in case of a
physical server failure.
• A feature of VirtualCenter
9
VirtualCenter Cluster - Hosts
10
VirtualCenter Cluster - VMs
11
VirtualCenter – Cluster Level Performance (CPU)
12
VirtualCenter – Cluster Level Performance (Memory)
13
About VI Performance Counters
• VirtualCenter statistics levels:
• Higher settings increase the amount of data collected.
• Level 1 – collects resource use averages (excludes devices)
• uptime, heartbeat and DRS data
• Level 2 – adds usage summation and rollup types to level 1
• Level 3 – adds device metrics
• Level 4 – adds maximum and minimum rollup types
14
Cluster Resource Considerations
•
•
•
•
•
•
15
CPU
Memory
Network IO
Disk IO
Disk Space
Cluster Services
CPU Measurement Examples – VI SDK
Cluster
Resource
Pool
Usage (%)
Usagemhz (MHz)
1
VM
1
1
1
1
System (ms)
3
Wait (ms)
3
Ready (ms)
3
Extra (ms)
3
Used (ms)
effectiveCPU (MHz)
(capacity rating)
3
X
Note – this list is not comprehensive
16
1
Host
3
resCPU Measurements – VI SDK
Cluster
Resource
Pool
Host
VM
3
3
3
3
3
3
3
3
Actav1/5/15 (%)
Average active time for the CPU over the past minute/5 min/15 min
Actpk1/5/15 (%)
The peak active time for the CPU over the past minute
Runav1/5/15 (%)
The average runtime for the CPU over the past minute/5 min/15 min
Runpk1/5/15 (%)
The peak runtime for the CPU over the past minute/5 min/15 min
Note – this list is not comprehensive
17
Cluster CPU statistics
18
Cluster CPU statistics – Derived Utilization
19
Cluster CPU statistics – Derived CPU % Ready
20
Cluster CPU statistics – Derived VM Usage Summary
21
Cluster CPU statistics – Derived MultiHost Comparisons
22
Cluster CPU statistics – Derived MultiHost Comparisons
23
Cluster CPU statistics – Derived MultiHost Comparisons
24
Resource Pool CPU MHz Usage
25
VM Level CPU Usage
26
VM Level CPU Usage
27
VM Level CPU Usage
28
VM Consumption of Virtual CPU
29
VM Level - % Ready Time
30
VM Level – 1 Min Peak CPU
31
VM - Number of Virtual CPUs
32
VM Impact on Cluster - Derived
33
Memory Measurement Examples – VI SDK
34
Cluster
Resource
Pool
Host
VM
Usage (%)
Portion of memory in use
(configured / available memory)
1
1
1
1
Granted (KB)
Memory granted to VMs
2
2
2
2
Active (KB)
Active = recently touched pages
2
2
2
2
Comsumed (KB)
Phys mem used by VMs, excluding
shared and overhead
2
2
2
2
Shared (KB)
Shared between VMs
2
2
2
2
Swapused (KB)
Memory used by swap
2
Sharedcommon
(KB)
Memory shared in common
between VMs
Swapped (KB)
Amount of memory swapped
Reservedcapacity
(MB)
Memory reserved for VMs
2
Totalmem (KB)
DRS Effective memory resources
1
Note – this list is not comprehensive
2
2
2
2
2
2
Cluster Memory statistics
35
Cluster Memory statistics - Derived
36
Cluster Memory statistics - Derived
37
Cluster Memory statistics - Derived
38
Cluster Memory Heap statistics - Derived
39
Cluster Memory – MultiHost Comparisons
40
Cluster Memory – MultiHost Comparisons
41
Cluster Memory – MultiHost Comparisons
42
Resource Pool Memory Stats
43
Network Measurement Examples – VI SDK
Cluster
44
Resource
Pool
Host
VM
Usage (KBps)
Sum of data transmitted
1
1
packetRx
Packets received (by nic)
3
3
packetTx
Packets transmitted (by nic)
3
3
Received
Rate data is received (by nic)
3
3
Transmitted
Rate data is transmitted (by nic)
3
3
Note – this list is not comprehensive
Cluster Network Activity - Derived
45
Cluster Network Activity – MultiHost Comparisons
46
Disk Measurement Examples – VI SDK
Cluster
Resource
Pool
Host
VM
Usage (KB)
Total data reads + writes
1
1
Read (KBps)
Data read (by disk)
3
3
Write (KBps)
Data Written (by disk)
3
3
totalReadLatency
Avg time for a read by a guest OS
3
3
kernelReadLatency
Time spent in the ESX Server
VMkernel
3
3
deviceReadLatency
Avg time taken to read from the
physical device
3
3
queueReadLatency
Avg time spent in VMkernel queue
per read
3
3
3
3
Etc.
Note – this list is not comprehensive
47
Cluster Disk Activity – Derived
48
Cluster Disk Activity – Derived
49
Cluster Disk Activity – Derived
50
Cluster Disk Activity – MultiHost Comparisons
51
Cluster Disk Activity – MultiHost Comparisons
52
Cluster Disk Activity – MultiHost Comparisons
53
Disk Space
• VI reports disk space at:
• Virtual Machine -> Disk
• Host -> Datastore
54
Cluster Disk Space Information by VM – Derived
55
Cluster Disk Space by Datastore – Host Level
56
Summary
• VMware Clusters provide a powerful virtualization platform
• Limited direct instrumentation is available from
VirtualCenter and the SDK
• But with careful thought it is possible to derive powerful
views of cluster activity
57
Thank You for your time.
Presented by: Pete Weilnau
Chief Architect
PERFMAN
pete.weilnau@perfman.com