Load Testing 101: Essential Tips
for Testers and Developers
Load Testing 101: Essential Tips for Testers and Developers | 1
Content
3 | Preface
4 | Load Testing Basics
6 | Goals of Load Testing
8 | Preparing Your Load Tests
13 | Making It Real: Emulating Real Life Conditions
In Your Load Tests
16 | The Dos and Don’ts of Load Testing
18 | About LoadNinja
Load Testing 101: Essential Tips for Testers and Developers | 2
By the time any software development
a practice has been around for years, we still see
project nears completion, it likely will have
well known mobile and web applications getting
gone through numerous tests, particularly
overwhelmed during peak traffic hours.
in an Agile environment where testing and
As a result, load testing needs to be done more
development happen concurrently. But no
frequently, more effectively, and more efficiently.
matter how many tests you’ve run, once your
It’s also created a need to simply train more people
application is nearly complete, there’s really
in the basics of load testing. Given that our lives
only one way to know whether or not your
software can handle the actual demands
your army of end users will soon be placing
on it — load testing.
Load testing is the process of putting simulated
demand on software, an application, or website in
a way that demonstrates its behavior under various
conditions. Over the last decade, the importance
of load testing has skyrocketed. What was once a
simple pre-deployment exercise to ensure a web
increasingly rest on software functioning properly
— whether it’s in medical devices, transportation,
communications, defense, or entertainment — software performance has never been more important.
That’s what this eBook is ultimately about — a “Load
Testing 101” manual to get the new & aspiring testers started with load testing.
We’ll cover:
|| Load testing basics
application could handle the load of multiple users
|| How to prepare for load testing
is now an integral part of software development
|| Emulating real life conditions in your load tests
and continuous performance improvement. As
websites and web applications become more inno-
|| Load testing dos and don’ts
vative and complex, load testing poses a significant
challenge for teams to 1. Execute properly and 2.
Cover all the bases. Though load testing as
Load Testing 101: Essential Tips for Testers and Developers | 3
Load Testing
Basics
Load testing is a type of performance testing.
peak load conditions. This allows you to verify that
Performance testing is a series of testing methods
your application can meet the desired performance
employed to understand how a system performs
objectives; which are often specified in a service
in terms of responsiveness and stability under a
level agreement (SLA). Load testing also enables
specific set of strains. Oftentimes, teams begin per-
you to measure response times, throughput rates,
formance testing in conjunction with or after func-
resource utilization levels, and to identify your
tional testing within the test phase of the software
application’s breaking point, assuming that breaking
development lifecycle. Each type of performance
point occurs below the peak load condition. Load
testing method helps answer a specific question
testing helps you check your web server’s perfor-
about your application’s behavior that helps carve a
mance under a massive load, determine its robust-
path for issue identification & iterative performance
ness, and estimate its scalability.
improvement.
Stress testing enables you to evaluate your ap-
For instance, load testing helps you answer, how
plication’s behavior when it is pushed beyond the
will my application behave in production with the
normal or peak load conditions, and helps you un-
typical traffic that we see? Load testing is used to
derstand ‘what is my application’s breaking point?’
verify your application’s behavior under normal and
The purpose of web server stress testing is to find
Load Testing
Stress Testing
Capacity Testing
Load Testing 101: Essential Tips for Testers and Developers | 4
the target application’s crash point. The crash point
and you’d like it to be able to handle potentially up
is not always an error message or access violation.
to 10,000 user submissions per minute during peak
It can be a perceptible slowdown in the request
load times. While developing the software, you may
processing. The goal of stress testing is to unearth
have performed unit tests as the code was being
application bugs that surface only under high
written, plus periodic regression tests to make sure
load conditions. Stress testing helps you find your
you didn’t break existing functionality with each new
application’s weak points, and how it behaves under
modification as development progressed, but at
extreme load conditions.
what point did you begin testing for multiple users?
Capacity testing is complementary to load testing
At what point did you begin testing the program to
and determines your server’s ultimate failure point,
accept hundreds or even thousands of overlapping
whereas load testing monitors results at various
field entries, form submissions, and other commands?
levels of load and traffic patterns. For example, let’s
This is when load testing comes into the picture.
say you’re developing a new online voting platform,
Load testing usually involves the following steps:
Iterate
Identify key
scenarios
Identify
workload
Identify
metrics
Create test
cases
Simulate
load
Analyze
results
Load Testing 101: Essential Tips for Testers and Developers | 5
Goals of Load
Testing
The main goals of load testing are to:
1. Identify bottlenecks and their causes
2. Optimize the application configuration (both the hardware and software)
for maximum performance.
3. Verify the reliability of your application under stress
Load/stress testing helps you identify the following characteristics:
|| Response time
|| Throughput
|| Maximum concurrent users supported
|| Resource utilization in terms of the amount of CPU, RAM, network I/O, and disk I/O
resources your application consumes during the test
|| Behavior under various workload patterns including normal load conditions, excessive
load conditions, and conditions in between
|| Application breaking point
|| Symptoms and causes of application failure under stress conditions
|| Weak points in your application
|| What is required to support a projected increase in load
Load Testing 101: Essential Tips for Testers and Developers | 6
This eBook keeps the abovementioned steps in mind and uses our load testing tool LoadNinja
to help you understand basics of load testing.
We’ll approach load testing with a four-step process:
|| Prepare: to prepare your web application for load testing
|| Record: How to record load testing scenarios
|| Test: Creating tests that match real-life circumstances
|| Analyze: Understanding and using load testing data
Load Testing 101: Essential Tips for Testers and Developers | 7
Preparing Your
With shift-left and DevOps motion, software
For example:
teams are pressured to test more and test often
Load Tests
|| If you hope to discover how your application
Testing should be performed at each step in the
performs with little or no load in order to get a
development cycle and should continue after the
baseline, you will run a single user test.
application is live. While it can be frustrating that a
tester’s job is never done, it’s important to take into
consideration that with each testing and remediation cycle, the application improves.
Applications go down under load for two reasons:
either developers didn’t load test or, worse, they
took the time to load test but didn’t prepare properly. Without adequate prep work, a load test can’t
find all the issues that it should.
So, how do you best prepare for a load test? Here
|| If you hope to determine how your system will
perform under normal expected load, you will
run a load test.
|| If you hope to determine the breaking point,
the point where your application either stops
responding or responds so slowly that it is unusable, you need to run a stress test.
|| If you want to know if your application has memory
leaks, you will want to run endurance or soak test.
are our 10 steps to help you prepare:
2.
1.
Decide on a number of users
What do you really need to know?
If you are going to load test, how many virtual users
Determine what you want to learn about your application or system. Each type of test is run differently,
and looks at your application in a different light. So,
do you want to simulate? In order to answer this, you
will want to approximate how many concurrent users
may visit your site, depending on the time of day.
you’ll need to run different types of tests based on
Don’t guess. Instead, leverage some of the data you
what you hope to find out.
already have. Talk to your marketing team and take
Load Testing 101: Essential Tips for Testers and Developers | 8
real traffic patterns from tools like Google Analytics,
You can find peak hourly visits and visit duration
or engage your Operations team and use data from
from an analytics tool, such as Google Analytics.
their Real User Monitoring (RUM) tools, which can
Note that this formula should be used to esti-
provide insight into realistic scenarios to test. If
mate the number of concurrent users required to
you want to know some concrete statistics from a
achieve a specific page-view rate at fully-ramped
historical perspective, go directly to your analytics
load. It should not be used to estimate the number
reports. You may even want to ask your engineers
of pages that will load within a specific time interval.
how many concurrent users they designed the application for, and your product owner for projected
numbers, based on promotional activities. Plan to
test that number and some percentage above it.
3.
Study your analytics
4.
Performance is a team goal
gather your team You need to involve a number of
people in the testing effort, including: developer,
network engineer, DBA, and business owner. All of
these individuals have a vested interest in making
the application successful, and each will approach
the problem from a different angle.
The correct solution will not fall directly into one of
Don’t pretend to know how your customers use
There are numerous ways you can find the number
these buckets, but will be a combination of two or
your application. The only way to truly understand
of virtual users needed to run a load test. We rec-
more. Make sure each is aligned with the perfor-
your users is to study history (i..e. analytics). By
ommend the following formula to find the number
mance expectations and is available during testing to:
studying your analytics, you will be able to create
of virtual users needed to run a load test.
tests that are representative of your actual users,
Concurrent users =
as opposed to tests that you think are represen-
(peak hourly visits * visit duration in seconds)
tative of your users. In this regard, analytics are a
3600
tester’s best friend.
|| Monitor their area of expertise
|| Provide balanced feedback
|| Gain a sense of ownership for the health and
performance of the application
Load Testing 101: Essential Tips for Testers and Developers | 9
5.
Prepare your browsers
Use testing software that brings you as close to
your actual users’ experience as possible. You
should be able to record your scenario in the
browsers of your choice, but you also need to
anticipate the browsers your users will most likely
use. Consider the countries and regions where
you anticipate high usage, and research the most
used browsers.
You’ll need to have these installed on your machine
to begin testing. Then you need to make sure your
load testing software emulates as closely as possible actual user behavior.
This includes:
|| Parallel thread processing
|| Think time
In LoadNinja, you can leverage a real browser to
those people that were present during testing?).
run a test, so you know you are getting the most
Results need to be looked at to ensure bottlenecks/
accurate test results as possible.
errors/weaknesses are understood and remedia-
6.
Be prepared to test your
production application
While it is valuable to test your application when
it is in a staging environment, this can leave some
holes in your testing. There are a few reasons for
this, including:
|| Staging environments are not often exact
duplicates of production.
|| Staging environments are often accessible
from only inside the firewall.
|| There is something to be said for testing
the same system that you are gathering
information about.
tion is effective. Reach out to everyone involved and
schedule adequate time.
8.
Set aside time to make changes
Different remediation will have differing costs in
terms of time. Remediation such as implementing
a caching strategy, refactoring code, database
optimization and hardware upgrades have a wide
range of costs to implement in terms of both time
and money. As an example, adding more hardware
will require time to order the hardware, receive the
shipment, test the new hardware, install software
and data, test, install into the network, and test
some more. This can be weeks or months of work.
This is less of a problem if you are in the cloud. In
|| Multiple concurrent scenarios
7.
|| Complex scenarios
Set aside time to analyze results
nies are opting to move to cloud infrastructures,
You should be prepared to spend some time ana-
without the need for additional hardware costs and
lyzing test results as a group (remember all of
time constraints.
|| Parameterization
|| Generating load from multiple agents
(network/cloud)
which case, it takes less than a day. Many compaoffering an unlimited number of environments
Load Testing 101: Essential Tips for Testers and Developers | 10
However, it’s always advisable to load test appli-
Now that you’ve taken the time to really prepare,
cations that are accessible within your firewall as
load testing your application will help you continually
well. Some load testing tools, like LoadNinja, allow
improve your product and your business.
you to do both.
9.
Plan an Agile testing methodology
Once you remediate, it is time to test again. The
saying, “testing is a process, not a destination” is
very true. Each time a bottleneck is uncovered and
corrected, another one rises to take its place. It is
important to plan an Agile testing methodology,
whereby performance testing is baked into each
How role playing games can save your app:
It’s easy to get swept up in role-playing video games.
Who can resist the temptation to be anyone you
want in a fictional world filled with unending excitement? Believe it or not, role playing games also
offer valuable lessons for testing your application.
In the next section, we look at how you can create
scenarios in order to perform load tests, and why
this needs to be a major aspect of both deploying
step of the development cycle. Additional testing
and improving web applications.
should be performed:
10.
|| When code is modified or updated
|| When environment/infrastructure changes are
introduced
|| When changes are made to the application
server or DB server
|| When traffic spikes are anticipated
How to record scenarios
The first thing to do is to determine the roles you
will define for use in your test. A role is equivalent to
a certain type of user that will visit the tested website, and the steps they will take while visiting.
Load Testing 101: Essential Tips for Testers and Developers | 11
If the tested site is a retail site, for example, you
retail site may have 95% of users browsing and leaving,
might have the following roles:
and 5% (or less!) actually making a purchase (Fig. 1).
|| Browse and leave
The combination of these two roles, or scenarios,
|| Browse, add something to the cart, and check out
will represent actual site traffic.
|| If your tested site is a restaurant site, your roles
After each scenario is recorded, you need to verify
might look something like this:
• Browse menu and find directions
• Look at hours of operation and make
a reservation
it individually. This involves running a single virtual
user for a single pass through the scenario. This
step should never be forgotten.
Now you’re ready to start testing. By recording sce-
It’s best to choose at least three of the most com-
narios that imitate actual user traffic, you’re setting
mon pathways through your site, and add a few
the stage to greatly improve customer experience
uncommon routes as well. Next, you need to break
and, if you’re into e-commerce, get that percentage
these roles down by percentage of traffic. A typical
of purchases above 5%.
Fig.1
Percentage
Role
95%
Browe and Leave
5%
Browse, add something to the cart and checkout
Load Testing 101: Essential Tips for Testers and Developers | 12
Making It Real:
Whether it’s an elementary school math quiz, a
Ninja, you can configure the ramp up time and
college history exam, or a software development
delay between user sessions, so you can increase
Emulating Real Life
team’s load test, we always want our tests to
the user number over time. For example: you start
emulate real-life conditions. Otherwise, what’s
with a single virtual user and add one virtual user
Conditions In Your
the point of testing?
every two seconds until you reach a certain number
Load Tests
In this chapter, we’ll discuss ways you can ensure
for the duration of the test.
your load tests match reality. Some of the best
of simultaneous users, and then hold that number
perks about our load testing tool, LoadNinja, are the
Setting load duration
settings that aid the process of generating a realistic
In order for you to run a test with large numbers of
load test. Of course, like all load testing tools, you
virtual users, you will need to set duration for your
can specify the number of virtual users to be sim-
test. By setting duration, each virtual user will exe-
ulated, but you can also set certain conditions that
cute its scenario and when it reaches the end it will
easily create more powerful and reliable load tests.
start over – thereby maintaining the level of load. For
Plus, one of the biggest advantages is running your
example, if your scenario takes two minutes to be ex-
load tests in real browsers and getting test results
ecuted but you run a test for 10 minutes, the scenar-
with browser-based data. No matter which load
io might be executed five times by each virtual user.
testing tool you choose, make sure it allows you to
This function is part of the test set-up in LoadNinja,
set some version of these basic conditions.
where you can select either a duration based test
Ramping up virtual users
When running performance tests, it’s not desirable
(and realistic) to start all virtual users at the same
time. Starting all virtual users at the same time can
or a iterations based test. LoadNinja will simulate
each virtual user for the number of times set by total
iterations or for the duration that is selected.
Parameterizing tests
create artificial bottlenecks in certain parts of the
While recording a scenario, you may need to specify
application — such as the login process. In Load
some parameters that will be used for further test
Load Testing 101: Essential Tips for Testers and Developers | 13
runs. For example, you can enter some search
rately measuring your application performance.
terms, user names and so on in the application’s
With this advantage, your team can spend more time
fields. However, it is not a good practice to play
testing & analyzing and less time reconfiguring the
back a test with the same recorded data for each
original test scripts to ensure they playback properly.
user as it does not simulate the real-life conditions.
To solve the problem, LoadNinja allows you to parameterize your load test using real data. Data-driven
tests empower your team to create tests that reflect
realistic scenarios, and by leveraging this feature, the
requests can use the provided data during test runs.
Replicating Browsers, network
connection bandwidth and
You can also specify think time for each tested
page (we call this think time, as it simulates the time
when a user is viewing the page and thinking). You
can easily randomize the think time for simulated
scenarios to better emulate real user activity.
Analyzing data in Load testing:
What you need to know
browsing speed
More often than not, developers and QA managers
Real users visit a website using different browsers,
little more than the number of users that will crash
the bandwidth of user’s Internet connections can
their system. Unless they have a professional load
vary significantly, and they spend different amounts
tester on staff, most development teams don’t have
of time on each page. It’s important to keep all of
the resources, or knowledge to garner all they could
these in mind when you’re configuring your tests,
from their load tests.
so you know that the tests you run will be helpful
in gauging how your application will perform in
production.
come away from a trial of load testing software with
That data is wasted because the person running the
tests is unable to apply it to an application’s performance. Luckily, improvements in graphics and UIs
In LoadNinja, you record, replay, and run every test
have made interpreting data much easier — if you
in a real browser. This way you know you’re accu
know what to look for.
Load Testing 101: Essential Tips for Testers and Developers | 14
Here’s a list of the most important results in load testing and how you
should be working with them.
Page load time
is important to see which errors and warnings are
cially important in a case where load is increasing,
generated and be able to see how that changes as
but bandwidth reaches and maintains a plateau. In
You need to know the average page load time for
load increases. A common error at high levels of
this case, it becomes apparent that bandwidth is
each page in your scenario. You might have a strict
load is “Server Error 500s.”
being throttled at some point in the process, possi-
Service Level Agreement (SLA) that mandates how
quickly pages must load, or may just want to know
what this number is. It is also important to know if
one page takes longer than others to load— this
Navigation Timings
Understanding what your end user experience and
bly at the firewall.
Hosts
why is key in identifying performance issues. Some
Because so many of today’s websites call out to a
of these metrics include redirect time, connect time,
plethora of additional hosts for things like content
Response load time
first byte time, response time, DOM load time, and
delivery networks, ad servers, analytics servers,
event time. It’s important to understand what your
social media and syndicated content, it is important
Just knowing page load time is not enough. If a page
performance benchmarks (or competitive bench-
for these sites to be enumerated in your reports.
is slow, you need to know why. Being able to look at
marks are) for these metrics to make sure that the
It’s equally important to be able to view all of the
average response times for each response gives you
experience you’re delivering through the browser
calls to a particular host. If a host is called from your
a detailed look into where the time is being spent.
meets your standards.
pages, the response time for those requests will
Errors and warnings
Request and response
throughput
add to the time it takes your pages to render.
indicates a bottleneck in your application.
You need to know which errors and warnings were
generated and at what level of load. This is especial-
It’s important to see the amount of data going to
ly important information to see in chart format. It
and coming from the tested system. This is espe-
Load Testing 101: Essential Tips for Testers and Developers | 15
The Dos & Don’ts
As you get ready to implement load testing to
not required, as we test in real browsers. However
your performance strategy, there are a few dos
if you’re relying on an emulator this may be
of Load Testing
and don’ts you’ll want to keep in mind.
important).
Dos:
Parameterize scenarios to simulate more realistic
Record tests with an end-user in mind. Map out
user journeys that are realistic, comprehensive,
and critical to business functions. Understanding
that your application can handle what a user
would realistically do is key in ensuring it will stand
up in production.
Generate load from different servers than those
that host your application. Serve them from different
environments so you can get the best view of
performance without accidentally altering results.
load on the server. Parameterizing scenarios
involves replacing recorded parameters in the
requests with variable values. This way you can
make virtual users send user-specific data to the
server. These data-driven tests will give you a more
realistic view of how your application will perform
when a unique group of users interact with it.
Verify user scenarios. Before creating tests on
the basis of a recorded scenario, make sure that
the scenario is executed successfully for one
virtual user. This can help you identify bottlenecks
Start recording a new scenario from the web
of the scenario and eliminate problems which are
browser’s start. If you start recording a scenario
not related to the number of virtual users and
after you connect to the tested web server and
additional testing conditions.
open a few web pages, the playback of the scenario
will fail. This will happen because the recorded
traffic will not reproduce the authentication
Arrange user scenarios in your tests so that
critical functionality is tested first.
procedure, and the tested web server will ignore
the simulated requests. (Note: In LoadNinja, this is
Load Testing 101: Essential Tips for Testers and Developers | 16
Don’ts:
Do not run tests in real environments. A real
environment can have other network traffic, and
this may affect test results. To avoid excess data
transfer, use a test environment that behaves in the
same way as the real environment except that there
is no other traffic usage. (Refer details on ‘prepare’
stage to perform load test in production).
Do not try to crash the tested server. The goal of
web server performance testing is not to break the
server, but to identify web application performance
bottlenecks under various loads.
Do not overly stress the client test systems.
Do not use zero think time. Make sure that think
time in your test is based on real-life conditions.
Using zero think time does not provide realistic user
simulation and puts an abnormal load on the tested
server. However, omitting think time can help you
determine bottleneck issues.
Load Testing 101: Essential Tips for Testers and Developers | 17
About LoadNinja
LoadNinja by SmartBear allows you to quickly create
scriptless sophisticated load tests, reduce testing time by
50%, replace load emulators with real browsers, and get
actionable, brower-based metrics, all at ninja speed. You
can easily capture client-side interactions, debug in real
LoadNinja
time, and identify performance problems immediately.
LoadNinja empowers teams to increase their test coverage
without sacrificing quality by removing the tedious efforts
Start Your Free Trial
of dynamic correlation, script translation, and script
scrubbing. With LoadNinja, engineers, testers and product
teams can focus more on building apps that scale and less
on building load testing scripts.
Load Testing 101: Essential Tips for Testers and Developers | 18
Load Testing 101: Essential Tips for Testers and Developers | 19