Performance Concepts
Mark A. Magumba
Introduction
• Research done on 1058 correspondents in 2006 found that
75% OF them would not return to a website that took
longer than 4 seconds to load, the same research five years
earlier showed people were willing to wait 8 seconds for a
page to load
• People tend to value predictability of a system. For instance
if a system slows down only once a week to a user this may
seem like the system overall is slow
• Perceived performance is how well a system appears to
perform
• Perceived performance can be improved by communicating
with the user by giving estimates of how long a task will
take e.g. visually through a progress bar
Determining Performance During
Design
• When designing a system performance must
be considered when the system works well
but also when it is in a special state for
instance what performance is required
– When some component failure has occurred
– When the system is in maintenance
– When patches are being installed
– When batch jobs are being run
Ways of determining performance in
the design phase
• Generally it is difficult to determine the
performance of a system at the design phase
however some methods can be employed like
– Using the experience of vendors
– Prototyping
– Ensuring system scalability
Managing Bottlenecks
• The performance of the system as a whole is
determined by certain components which may be
overloaded known as bottlenecks
• For instance a system may have very fast
networks but very slow hard disks, the fast
network speeds are immaterial then as
performance will be degraded by the hard disks
• The hard disks in this case would be a
performance bottleneck
Benchmarking
• Benchmarking measures the comparative
performance of test component against an
industry standard via a test program.
• The following table shows some common
bench mark tests
Common Bench Mark tests
Performing performance Tests
• There are three general methods for doing
performance tests of whole systems
– Load testing: it’s a test to see how well a system
operates under a particular load
– Stress testing: Is a test to see how well a system
operates under extreme load, it is a test to see at
where (the bottle necks) and when (the load) a
particular system breaks
– Endurance testing: Is a measure to see how a system
behaves when subjected to the expected load for a
sustained period of time
Graph of response time against
number of users for typical system
Performance testing methodology
• Performance testing usually employs a group of
servers to act as injectors. These simulate users
• Another server group known as conductors
evaluate the test performance
• It is important that the test environment matches
the production environment as much as possible
otherwise results will be unreliable
• Where the cost of setting up a complete test
environment is prohibitive it is possible to use a
temporary test environment for instance by
renting servers from your vendors
Sources of Performance Metrics
• Kernel Counters
– Most operating systems contain counters that store
performance information. Tools are provided to
retrieve this information for instance on windows you
have the perfmon tool and on UNIX systems that sar
tool
• SNMP counters
– SNMP is the standard network management protocol
and accumulates information about devices in a
Management Information Base. This information can
be accessed via a Network Management System.
SNMP also provides for tarps which is a special
message type sent when a device counter exceeds its
normal limit
Sources of Performance Metrics
• Logging. Stores system event information of
infrastructure components sometimes on a
dedicated log server. Logging usually
generates a lot of information but the level of
logging is usually configurable
• Analyzing logs is fundamentally different from
monitoring using tools like SNMP as these are
able to react in real time, log files are analysed
after some time has elapsed
Performance Patterns
• Performance patterns
– There are several ways to improve performance at
the infrastructure level but it is important to note
that 80% of performance issues do not originate
from the infrastructure but from poorly
configured applications
• Increasing performance in upper layers
– Database and application tuning can provide
higher performance boosts than infrastructure
upgrades
Infrastructure Optimizations
• Caching
– Improves infrastructure performance by keeping
frequently accessed instructions and data in
memory to improve access times examples
include
• Disk caching
• Web proxies
• Operational Data Store
– Is a smaller database that is a subset of an organization’s
larger database whose purpose is to improve access times to a
particular subset of frequently queried data
• In memory databases
Infrastructure Optimizations
• Scalability
– Implies the ease with which configuration changes
can be made to a system in response to changes in
number of users. A system whose performance
improves proportionally to increase in capacity is
said to scale well
Vertical VS Horizontal Scaling
• Vertical scaling (scaling up) involves adding capacity to a single
system component for instance increasing the number of
CPUs
• The problem is that this usually has an upper limit also the
cost of adding a single component can quickly become very
high
• Horizontal scaling (scaling out) which involves adding more
servers to a system. This usually leads to higher management
complexity and may increase latency and affect throughput
between nodes
• Horizontal scaling works best when a system is partitioned in
layers
• In general scaling is best achieved by using more components
(horizontally) rather than using very fast components
(vertically) and this approach also suffers some performance
penalty due to increased overhead
Scaling in partitioned system
Load Balancing
• To take advantage of scaling usually some load
balancing is applied
• Load balancing is the spreading of the
application load across multiple identical
servers operating in parallel
Load Balancing
• High performance clusters
– In this arrangement several standard computers
are grouped together via high speed connections
to emulate an infinitely powerful super computer
• Grids
– Grids are high performance clusters spread
geographically.
• They rely on the idle processor time of the participating
computers.
• The computers communicate over the internet and in
this case the network is a possible bottleneck
• Security is also an important consideration and in some
cases computations must be done twice
Design for use
• In the case of special purpose performance critical systems
some guarantees must be made and these will call for
specific design decisions
• Designing for an interactive system for instance will call for
a different set of requirements than for a batch system
• In some cases special products may be required like custom
operating systems and hardware
• Most vendors will provide architects with standard system
implementation plans and in most cases it is advisable that
these are followed, it is also a good idea to have vendors
verify system plans
• When possible try to spread system load over time to avoid
unnecessary spikes
• Some systems have offline copies of data available to
improve performance of I/O expensive operations