Load Testing with Visual Studio Team Edition for Tester

advertisement
Understanding VSTS Load Testing Framework
Understanding the Visual Studio Team System
Load Testing Framework
Introduction ............................................................................................................................................... 2
Overview ................................................................................................................................................... 2
Test Client ................................................................................................................................................. 3
Load Test................................................................................................................................................... 4
Scenario ................................................................................................................................................. 4
Counter Sets .......................................................................................................................................... 5
Run Settings .......................................................................................................................................... 5
Test Execution Environment ..................................................................................................................... 6
Test Controller & Test Agent................................................................................................................ 6
Run Configurations ............................................................................................................................... 6
Test Rig ................................................................................................................................................. 7
Test Management Repository ................................................................................................................... 8
References ............................................................................................................................................... 10
© 2006 Chris Birmele, Microsoft Corporation
1
Understanding VSTS Load Testing Framework
Introduction
The purpose of this paper is to provide an overview of the Load Testing Framework that is part of
Microsoft’s Visual Studio Team System.
This paper is not a step-by-step tool guide - refer to the comprehensive Visual Studio Team System
Online Help, neither is it a best practices guide - consult with Testing Consultants for this kind of
advice.
Overview
Before delving into the details, let’s start with some basics.
The purpose of a load test is to understand a system’s behaviour when used by a large number of
concurrent users. Is the system still reliable when 500 users access it simultaneously? What about
response times? How many users can the system handle? This and many other questions can be
answered by performing load tests.
One way to find out is to invite all your friends and family members and have them use the system; the
other is to use a software testing solution like Visual Studio Team System. A much cheaper and more
reliable alternative!
In order to perform a load test on a system using software tools:

We need a test client (a user interface and/or API’s) to define tests we want to run against the
system, to launch tests and monitor and analyse results.

We need to create load tests that accurately simulate the behaviour of a large number of users.

We need a test execution environment responsible for running tests and collecting data for
analysis.

We need a test management repository to store test results and manage test asset versions.

We need a system to test!
Visual Studio Team System
e
c
at
re
Test Client
ma
n
ag
e
results
Tests
Tests
Tests
Tests
run
Test Execution
Environment
Test Asset
Management
Repository
System Under Test
Figure 1
© 2006 Chris Birmele, Microsoft Corporation
2
Understanding VSTS Load Testing Framework
Test Client
Visual Studio Team System1 provides two test clients:
Test Client


Visual Studio
A graphical user interface
A command line interface (not covered in this paper!)
Command Line
Test Manager Interface
The test client allows us to create and manage tests, launch
tests and monitor and analyse test results.
The two interfaces communicate via the Test Manager
Interface (a managed API) with other components of the
testing framework, namely the Test Execution Environment
and the Test Management Repository.
.NET Remoting
.NET Remoting
Test Assets
Management
Repository
Test Execution
Environment
(Test Controller)
(Team Foundation Server)
Figure 2
Tests
The graphical user interface in Visual
Studio Team System consists of a Test
Manager view allowing you to create
and manage tests, a Test Run View to
see current and past test runs as well
as a Test Results view (Figure 3).
Test Results
Test Runs
Figure 3
I use the term Visual Studio Team System in this paper, which is the name used to describe an entire product family. The testing
capabilities described in this paper are really only part of two products:
1


Visual Studio Team Suite
Visual Studio Team Edition for Testers
Check this for an overview of the various product version: http://msdn.microsoft.com/vstudio/teamsystem/products/compare/default.aspx
© 2006 Chris Birmele, Microsoft Corporation
3
Understanding VSTS Load Testing Framework
Load Test
Designing a load test to simulate the behaviour of a large number of users is not an easy task. The
challenge is to understand how users will use the system, what data we need to capture, where and how
we collect the required data and what does the data mean.
In Visual Studio Team System this information is captured in a load test. A load test consists of the
following parts:

Scenarios specifying the tests to run to
simulate a specific user interaction.

Counter Sets specifying the data a test run
will collect and their thresholds.

Run Settings specifying the computers that
will collect the data and other runtime
parameters such as the test duration and the
data collection sampling rate.
Load Test
Scenarios
Scenarios
Scenarios
Counter
Sets
Counter
CounterSets
Sets
Run
Settings
Run
RunSettings
Settings
Figure 4
Scenario
A scenario captures all the information required to simulate a specific user interaction with the
system as realistically as possible:

Test Mix describes the mix of tests
that constitute a test scenario.

Browser Mix describes the browser
mix simulated during execution.

Network Mix describes the network
mix simulated during execution.

Load Pattern describes how test
instances are generated during
execution.
Scenario
Test Mix
Browser Mix
• Web Test
• Unit Test
• IE
• Netscape
•…
© 2006 Chris Birmele, Microsoft Corporation
Network Mix
• LAN
• WAN
•…
Load Pattern
• Constant
• Stepped
• Goal Seeking
Figure 5
4
Understanding VSTS Load Testing Framework
Counter Sets
A counter set specifies the data collected during the load test execution. Visual Studio Team
System provides a large number of counter sets that group individual counters around
technologies.
Counters can have thresholds associated with them to indicate values that need to be reported as
warnings or as errors during the execution of a load test.
Run Settings
The run settings specify important runtime parameters such as:




The load test duration
The data collection sampling rate
The maximum number of errors to report
The computers that collect the data (includes test controllers, agents and any other computer
you want to collect data from)
A Load Test can be associated with one or more run settings, but only one can be active at
anyone time.
Figure 6 is an example of how scenarios, counter sets and run settings appear in Visual Studio
Team System:
Scenarios
Counter Sets
Run Settings
Properties
Run Settings
Figure 6
© 2006 Chris Birmele, Microsoft Corporation
5
Understanding VSTS Load Testing Framework
Test Execution Environment
The test execution environment simulates concurrent users
by executing a number of test instances – one for each
simulated user – against the system under test.
At the heart of the test execution environment is the test
controller and test agents:
Test Client
Test Management Interface
.NET Remoting
Test Execution
Environment
Controller
Test Controllers coordinate test agents, collect test results
and communicate with test clients.
Agent(s)
Agent(s)
Agent(s)
Test Agents execute test code, collect performance statistics
and return the information to the controller.
System under Test
Figure 7
Test Controller & Test Agent
A test client initiates a load test run by
communicating to the test controller the test(s) to
execute.
Test Client
Test Controller
•Test Code
•Tests
•Test Results
The controller communicates performance data
back to the test client for analysis.
4. results
Agent Service
Agent Process
Test Code
3. runs
2. starts
Test Agent
1. deploys
The test agent executes the test code, collects
performance data and returns it to the controller,
which logs it in a data store.
Test Results
Store
Controller Service
The controller then delegates the task of executing
a test instance to one or more agents.
Figure 8
The interactions between test client, test controller and test agents are governed by the Run
Settings and the Run Configurations settings.
Run Configurations
Run Settings are specific to load tests; Run Configurations define more generic settings
applicable to most test types (Figure 9):


Test run naming conventions
Whether tests are run locally or on a remote Test Rig
© 2006 Chris Birmele, Microsoft Corporation
6
Understanding VSTS Load Testing Framework





Assemblies that need to be instrumented to collect Code Coverage information
Additional files and directories to be deployed for tests to run successfully (the test code is
automatically deployed)
Generic setup and cleanup scripts that must be executed as part of a test run
Test abortion and test failure timeouts
Web Test specific settings that need to be persisted across multiple test runs
A Visual Studio Team System solution can have multiple Run Configurations, but only one can
be active at anyone time (Figure 9).
Figure 9
Test Rig
Tests - load tests and others - run either locally or on a
remote Test Rig.
For a local load test run, all required components are
located on the test client machine (i.e. Visual Studio
Team System).
Generating test instances to simulate virtual users
requires memory and CPU cycles. There is a limit of
how many test instances can be generated on a single
computer.
Test Client
Test Agent
Test Controller
System Under Test
Figure 11
If you need to simulate a large number of test instances that can not be generated from a single
computer you need to set up a Test Rig. A Test Rig consists of one test controller and a
collection of test agents that generate the desired user load (Figure 12).
© 2006 Chris Birmele, Microsoft Corporation
7
Understanding VSTS Load Testing Framework
In this scenario, the test client communicates
with a remote test controller to initiate a test
run.
The test controller collaborates with all
registered test agents each of which executes
test instances against the system under test.
Test Client
Test Rig
Test Controller
The Run Configurations settings determine
how the test controller distributes the test
instances among the test agents.
Test controllers and test agents have default
counter sets assigned, determining the
performance data to collect.
Test Agent
Test Agent
Visual Studio Team System automatically
maps these counter sets to the test controller
and test agents that make up the test rig.
Test Agent
System Under Test
Figure 12
Test Management Repository
One of the most important components of Visual Studio
Team System is the Team Foundation Server or TFS.
It is beyond the scope of this paper to describe all features of
TFS, but suffice to say that here it serves as the Test
Management Repository (see Figure 13).
Test Client
Test Management Interface
Web Service
Team Foundation Server
The test assets, i.e. the tests (load test as well as other tests)
and the Run Configurations are stored in the version control
system of TFS. They can be viewed, compared, branched
and merged like any other file.
The results of individual test runs are stored in a separate
database. The summary of these runs is recorded in the data
warehouse.
Source
Control
Build / Test
Results
Data
Warehouse
Work Items
Test runs can be associated with any work items. For
example a failed test can be associated with a new defect, or
a successful test can be associated with a Quality of Service
requirement to prove the requirement has been satisfied.
© 2006 Chris Birmele, Microsoft Corporation
Figure 13
8
Understanding VSTS Load Testing Framework
Figure 14 shows how these scenarios appear in Visual Studio Team System.
Work Item Associations
Test Asset Version Control
Test Run Summaries
Team
Foundation
Server
Figure 14
© 2006 Chris Birmele, Microsoft Corporation
9
Understanding VSTS Load Testing Framework
References
The content of this paper is based on a variety of information sources, such as blogs, Microsoft
presentations, the Visual Studio Team System online help and others.
To the best of my knowledge the explanations given in this document are correct, but I would
appreciate any corrections and constructive suggestions of how this paper could be improved.
Sydney, June 06
Chris Birmele
Developer Tools Technical Specialist
Australia & New Zealand
cbirmele@microsoft.com
© 2006 Chris Birmele, Microsoft Corporation
10
Download