HelgeKlein.com
Home of SetACL
A scientific approach to
XenApp farm sizing
Helge Klein
HelgeKlein.com
Home of SetACL
Who is
Helge Klein?
CTP, MVP
Author of SetACL and Delprof2
Independent consultant and developer
Architect of what later became Citrix Profile Management
HelgeKlein.com
Home of SetACL
What is he
talking about?
Scientifically sound farm sizing methodology
How to calculate farm capacity
HelgeKlein.com
Home of SetACL
Methodology
1. Determine capacity of existing farm
2. Measure load and identify bottlenecks
3. Calculate capacity of new farm
HelgeKlein.com
Home of SetACL
Example
Description
Farm capacity Farm capacity
old (total)
old (per user)
Factor
Farm capacity
new (per user)
Farm capacity
new (total)
RAM for user sessions [MB]
630,000
315
2.0
630
1,575,000
CPU [normalized cores]
1,250
0.63
0.25
0.16
391
Hard disk [write IOPS]
7,300
3.65
1.5
5.48
13,688
Network [MB/s]
24,000
12.00
0.1
1.20
3,000
HelgeKlein.com
Home of SetACL
Where to get the
numbers?
1. Data collection
2. Observation
3. Measurements
4. Calculation
HelgeKlein.com
Home of SetACL
Data collection
HelgeKlein.com
Home of SetACL
for /f %i in (AllFarmServers.txt) do wmic
/node:%i cpu get name, maxclockspeed,
systemname, description, manufacturer,
revision /format:csv >> CPUs.txt
Collect CPU data
Create AllFarmServers.txt with qfarm
Use resulting list to determine server model
HelgeKlein.com
Home of SetACL
Srv001,x86
Srv001,x86
Srv001,x86
Srv001,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv002,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv003,x86
Srv004,x86
Srv004,x86
Srv004,x86
...
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
Family
15 Model 4
15 Model 4
15 Model 4
15 Model 4
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 44
6 Model 26
6 Model 26
6 Model 26
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
Stepping
10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001
10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001
10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001
10,GenuineIntel,3400,Intel(R) Xeon(TM) CPU 3.40GHz,1034,Srv001
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv002
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
2,GenuineIntel,2666,Intel(R) Xeon(R) CPU E5640 @ 2.67GHz,11266,Srv003
5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004
5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004
5,GenuineIntel,2666,Intel(R) Pentium(R) III Xeon-Prozessor,6661,Srv004
The result
HelgeKlein.com
Home of SetACL
RAM, NICs
hard disks
Could be determined via WMI, too
Often knowing the server model is sufficient
Components per model often identical
HelgeKlein.com
Home of SetACL
Spec sheet
Component
HP DL360 G4
CPU
2x Xeon dual core 3,0 GHz, no Hyper Threading
4 Cores, CPUID: x86 Family 15 Model 4 Stepping 1
RAM
4 GB
NICs
2x Gigabit, 1 connected
RAID
Smart Array 6i
Hard disks
2x 70 GB configured as RAID-1, Compaq BF0728A4CB, 3,5“, 15.000
R/Min, Ultra320 SCSI
HelgeKlein.com
Home of SetACL
Resulting data
Server model
DL360 G7
DL360 G6
DL360 G5
DL360 G4p
DL360 G4
Number
26
52
70
60
35
HelgeKlein.com
Home of SetACL
Observation
HelgeKlein.com
Home of SetACL
How many users are logged on?
Load of CPU, RAM, NICs
Individual processes with a lot of RAM or CPU?
HelgeKlein.com
Home of SetACL
Physical Disk\% Disk Time
Physical Disk\Avg. Disk Queue
Length
„Time“ is equivalent to flickering of hard disk LED
Disk queue length: Number of waiting IOs
HelgeKlein.com
Home of SetACL
Hypothesis: farm is
memory limited
Limiting factor will differ between farms
HelgeKlein.com
Home of SetACL
Measurements
Tool: Perfmon
Next slides: relevant counters
HelgeKlein.com
Home of SetACL
Terminal Services\Active Sessions
Terminal Services\Inactive Sessions
Terminal Services\Total Sessions
System\Processes
General system information
HelgeKlein.com
Home of SetACL
PhysicalDisk(_Total)\% Disk Time
PhysicalDisk(_Total)\Avg. Disk Queue Length
PhysicalDisk(_Total)\Disk Reads/sec
PhysicalDisk(_Total)\Disk Writes/sec
PhysicalDisk(_Total)\Avg. Disk sec/Transfer
Hard disk activity
Load, queue length, operations per second, latency
HelgeKlein.com
Home of SetACL
Processor(_Total)\% Processor Time
Memory\Available MBytes
Network Interface(*)\Bytes Total/sec
CPU, RAM and network
RAM: total amount must be known!
HelgeKlein.com
Home of SetACL
logman create counter TSPerf -f csv -cf
C:\PerfLogs\Counters.txt -o
C:\PerfLogs\Server13.csv -si 60 -rf 24:00:00
Automation
Create and start data collector set. Format CSV, performance counters are
read from C:\PerfLogs\Counters.txt, output file is C:\PerfLogs\Server013.csv,
60 second sampling interval, duration 24 hours.
HelgeKlein.com
Home of SetACL
logman create counter TSPerf -f csv -cf
C:\PerfLogs\Counters.txt –o
C:\PerfLogs\Server13.csv -si 60 -rf 24:00:00
-s Server13
Execution on remote computer Server13
HelgeKlein.com
Home of SetACL
for /f %i in (Servers.txt) do logman create
counter TSPerf -f csv -cf
C:\PerfLogs\Counters.txt -o C:\PerfLogs\%i.csv
-si 60 -rf 24:00:00 -s %i
Many servers
One computer name per line in Servers.txt
HelgeKlein.com
Home of SetACL
Analyzing
the measured data
HelgeKlein.com
Home of SetACL
CPU and
sessions
Server 37
Moderate load during
logon phase,
afterwards even less
A lot of overcapacity
HelgeKlein.com
Home of SetACL
CPU and
sessions
Server 89
HelgeKlein.com
Home of SetACL
Overlaying the
CPU load of
many servers
Easily verify the analyis
HelgeKlein.com
Home of SetACL
HDD and
sessions
Server 37
Moderate load, peaks
during logon phase
Full load at aprox. 200
HelgeKlein.com
Home of SetACL
HDD and
sessions
Server 89
HelgeKlein.com
Home of SetACL
Overlaying the
HDD load of
many servers
Easily verify the analyis
HelgeKlein.com
Home of SetACL
RAM and
sessions
Server 37
Continually increasing
load, maximum in the
afternoon
Available RAM must
not go near zero
(because of disk
cache)
High load
HelgeKlein.com
Home of SetACL
RAM and
sessions
Server 89
HelgeKlein.com
Home of SetACL
Overlaying the
memory load
of many
servers
Easily verify the analyis
HelgeKlein.com
Home of SetACL
Network and
sessions
Server 37
200 = 2 MB/s
Average rate < 200
KB/s
Very low load, a lot of
overcapacity
HelgeKlein.com
Home of SetACL
Network and
sessions
Server 89
HelgeKlein.com
Home of SetACL
Overlaying the
network load
of many
servers
Easily verify the analyis
HelgeKlein.com
Home of SetACL
Hypothesis confirmed:
farm is limited by
available memory
CPU load: low, network: negligible
hard disk: moderate
HelgeKlein.com
Home of SetACL
Calculating
farm capacity
Description
Farm capacity Farm capacity
old (total)
old (per user)
Factor
Farm capacity
new (per user)
Farm capacity
new (total)
RAM for user sessions [MB]
630,000
315
2.0
630
1,575,000
CPU [normalized cores]
1,250
0.63
0.25
0.16
391
Hard disk [write IOPS]
7,300
3.65
1.5
5.48
13,688
Network [MB/s]
24,000
12.00
0.1
1.20
3,000
HelgeKlein.com
Home of SetACL
Normalizing
CPU performance
How to compare performance of different CPUs?
Benchmarking is difficult
Better: Moore’s law (doubling of performance every 18-24 months)
Surprisingly accurate (amongst other things because it is a self-fulfilling prophecy)
HelgeKlein.com
Home of SetACL
𝐿
𝑡
=𝑒
Performance after time (in months)
Assumed performance doubling every 21 months
Oldest CPU in farm = 1,0
0,033𝑡
HelgeKlein.com
Home of SetACL
CPU
Cores
First sold
Xeon 7100
„Tulsa“ 3,0 GHz
2
08/2006
Performance index Performance index
per core
1,0
1,0
Xeon 7100
„Tulsa“ 3,4 / 3,6
GHz
Xeon E5430 /
E5440
2
03/2007
1,26
1,26
4
11/2007
1,64
0,82
Xeon X5550
4
03/2009
2,78
1,39
Xeon E5640
4
03/2010
4,13
2,06
HelgeKlein.com
Home of SetACL
CPU
Number of Number of
CPUs
cores
Number of
normalized CPUs
Number of
normalized cores
Xeon 3 GHz
70
140
70
140
Xeon 3,4/3,6
GHz
119
238
150
300
Xeon
E5430/E5440
70
280
115
230
Xeon X5550
67
268
186
370
Xeon E5640
26
104
107
210
Total
352
1030
628
1250
HelgeKlein.com
Home of SetACL
Farm capacity:
1250 normalized CPUs
0.63 CPUs / user
HelgeKlein.com
Home of SetACL
Hard disk performance =
IOPS
With many concurrent accesses transfer rate is mostly irrelevant
More important: IOPS (operations per second)
Exact number depends on measurement method  do not believe vendors
HelgeKlein.com
Home of SetACL
PhysicalDisk(_Total)\Disk Reads/sec
PhysicalDisk(_Total)\Disk Writes/sec
IOPS measurement with Perfmon
Read and write IOPS may be very different
HelgeKlein.com
Home of SetACL
Read and
write IOPS
Average: ~15
Including spikes: 30
HelgeKlein.com
Home of SetACL
Farm capacity:
7300 IOPS
3,7 IOPS / user
HelgeKlein.com
Home of SetACL
RAM – how much
do we have?
We need total RAM that is available for user sessions
HelgeKlein.com
Home of SetACL
Server model
RAM / server [GB] #Servers
RAM / server total [GB]
DL360 G4
3,5
35
122,5
DL360 G4p
3,5
60
210
DL360 G5
3,25
70
227,5
DL360 G6
3,49
52
181,48
DL360 G7
3,49
26
90,74
243
832
Total
HelgeKlein.com
Home of SetACL
Description
Memory [MB]
Session 0 (OS + Presentation Server + antivirus)
325
Kernel (pooled + non-pooled)
175
File system cache
400
Total
900
Calculate RAM used by the OS (Server 2003)
HelgeKlein.com
Home of SetACL
Farm capacity:
630 GB
315 MB / user
25% Overhead because 900 MB used by OS
HelgeKlein.com
Home of SetACL
Network –
too much for HDX?
1 Gigabit/second per server
~ 100 MB/s
HelgeKlein.com
Home of SetACL
Farm capacity:
24 GB/s
12 MB/s / user
Extreme overcapacity
HelgeKlein.com
Home of SetACL
Future
capacity
Description
Farm capacity Farm capacity
old (total)
old (per user)
Factor
Farm capacity
new (per user)
Farm capacity
new (total)
RAM for user sessions [MB]
630,000
315
2.0
630
1,575,000
CPU [normalized cores]
1,250
0.63
0.25
0.16
391
Hard disk [write IOPS]
7,300
3.65
1.5
5.48
13,688
Network [MB/s]
24,000
12.00
0.1
1.20
3,000
HelgeKlein.com
Home of SetACL
Parameter
Value
Farm users old (CCU)
2,000
Farm users new (CCU)
2,500
RAM for OS new [MB]
1,300
CPU performance new [normalized
cores]
1.9
IOPS per disk new
300
Reserve for disaster recovery
1.5
Parameters. Calculation includes:
Changed user count, different memory requirements of new OS,
server replacement, capacity increase for disaster recovery
HelgeKlein.com
Home of SetACL
Virtual server
RAM [MB]
32.768
User sessions
49.9
CPU cores (physical)
4.1
IOPS
273
NICs
0.6
Servers are memory limited  after setting the amount of RAM,
the other parameters can be calculated
HelgeKlein.com
Home of SetACL
Physical servers
VMs per host
3
#Virtual
servers
50
RAM [MB]
101,376
#Hosts
25
CPU cores
12
Users /
host
150
Hard disks
3
NICs
2
After setting the number of VMs per host the other parameters
can be calculated.
Excel sheet available upon request.
HelgeKlein.com
Home of SetACL
Thank you!
Helge Klein