Understanding VSTS Load Testing Framework Understanding the Visual Studio Team System Load Testing Framework Introduction ............................................................................................................................................... 2 Overview ................................................................................................................................................... 2 Test Client ................................................................................................................................................. 3 Load Test................................................................................................................................................... 4 Scenario ................................................................................................................................................. 4 Counter Sets .......................................................................................................................................... 5 Run Settings .......................................................................................................................................... 5 Test Execution Environment ..................................................................................................................... 6 Test Controller & Test Agent................................................................................................................ 6 Run Configurations ............................................................................................................................... 6 Test Rig ................................................................................................................................................. 7 Test Management Repository ................................................................................................................... 8 References ............................................................................................................................................... 10 © 2006 Chris Birmele, Microsoft Corporation 1 Understanding VSTS Load Testing Framework Introduction The purpose of this paper is to provide an overview of the Load Testing Framework that is part of Microsoft’s Visual Studio Team System. This paper is not a step-by-step tool guide - refer to the comprehensive Visual Studio Team System Online Help, neither is it a best practices guide - consult with Testing Consultants for this kind of advice. Overview Before delving into the details, let’s start with some basics. The purpose of a load test is to understand a system’s behaviour when used by a large number of concurrent users. Is the system still reliable when 500 users access it simultaneously? What about response times? How many users can the system handle? This and many other questions can be answered by performing load tests. One way to find out is to invite all your friends and family members and have them use the system; the other is to use a software testing solution like Visual Studio Team System. A much cheaper and more reliable alternative! In order to perform a load test on a system using software tools: We need a test client (a user interface and/or API’s) to define tests we want to run against the system, to launch tests and monitor and analyse results. We need to create load tests that accurately simulate the behaviour of a large number of users. We need a test execution environment responsible for running tests and collecting data for analysis. We need a test management repository to store test results and manage test asset versions. We need a system to test! Visual Studio Team System e c at re Test Client ma n ag e results Tests Tests Tests Tests run Test Execution Environment Test Asset Management Repository System Under Test Figure 1 © 2006 Chris Birmele, Microsoft Corporation 2 Understanding VSTS Load Testing Framework Test Client Visual Studio Team System1 provides two test clients: Test Client Visual Studio A graphical user interface A command line interface (not covered in this paper!) Command Line Test Manager Interface The test client allows us to create and manage tests, launch tests and monitor and analyse test results. The two interfaces communicate via the Test Manager Interface (a managed API) with other components of the testing framework, namely the Test Execution Environment and the Test Management Repository. .NET Remoting .NET Remoting Test Assets Management Repository Test Execution Environment (Test Controller) (Team Foundation Server) Figure 2 Tests The graphical user interface in Visual Studio Team System consists of a Test Manager view allowing you to create and manage tests, a Test Run View to see current and past test runs as well as a Test Results view (Figure 3). Test Results Test Runs Figure 3 I use the term Visual Studio Team System in this paper, which is the name used to describe an entire product family. The testing capabilities described in this paper are really only part of two products: 1 Visual Studio Team Suite Visual Studio Team Edition for Testers Check this for an overview of the various product version: http://msdn.microsoft.com/vstudio/teamsystem/products/compare/default.aspx © 2006 Chris Birmele, Microsoft Corporation 3 Understanding VSTS Load Testing Framework Load Test Designing a load test to simulate the behaviour of a large number of users is not an easy task. The challenge is to understand how users will use the system, what data we need to capture, where and how we collect the required data and what does the data mean. In Visual Studio Team System this information is captured in a load test. A load test consists of the following parts: Scenarios specifying the tests to run to simulate a specific user interaction. Counter Sets specifying the data a test run will collect and their thresholds. Run Settings specifying the computers that will collect the data and other runtime parameters such as the test duration and the data collection sampling rate. Load Test Scenarios Scenarios Scenarios Counter Sets Counter CounterSets Sets Run Settings Run RunSettings Settings Figure 4 Scenario A scenario captures all the information required to simulate a specific user interaction with the system as realistically as possible: Test Mix describes the mix of tests that constitute a test scenario. Browser Mix describes the browser mix simulated during execution. Network Mix describes the network mix simulated during execution. Load Pattern describes how test instances are generated during execution. Scenario Test Mix Browser Mix • Web Test • Unit Test • IE • Netscape •… © 2006 Chris Birmele, Microsoft Corporation Network Mix • LAN • WAN •… Load Pattern • Constant • Stepped • Goal Seeking Figure 5 4 Understanding VSTS Load Testing Framework Counter Sets A counter set specifies the data collected during the load test execution. Visual Studio Team System provides a large number of counter sets that group individual counters around technologies. Counters can have thresholds associated with them to indicate values that need to be reported as warnings or as errors during the execution of a load test. Run Settings The run settings specify important runtime parameters such as: The load test duration The data collection sampling rate The maximum number of errors to report The computers that collect the data (includes test controllers, agents and any other computer you want to collect data from) A Load Test can be associated with one or more run settings, but only one can be active at anyone time. Figure 6 is an example of how scenarios, counter sets and run settings appear in Visual Studio Team System: Scenarios Counter Sets Run Settings Properties Run Settings Figure 6 © 2006 Chris Birmele, Microsoft Corporation 5 Understanding VSTS Load Testing Framework Test Execution Environment The test execution environment simulates concurrent users by executing a number of test instances – one for each simulated user – against the system under test. At the heart of the test execution environment is the test controller and test agents: Test Client Test Management Interface .NET Remoting Test Execution Environment Controller Test Controllers coordinate test agents, collect test results and communicate with test clients. Agent(s) Agent(s) Agent(s) Test Agents execute test code, collect performance statistics and return the information to the controller. System under Test Figure 7 Test Controller & Test Agent A test client initiates a load test run by communicating to the test controller the test(s) to execute. Test Client Test Controller •Test Code •Tests •Test Results The controller communicates performance data back to the test client for analysis. 4. results Agent Service Agent Process Test Code 3. runs 2. starts Test Agent 1. deploys The test agent executes the test code, collects performance data and returns it to the controller, which logs it in a data store. Test Results Store Controller Service The controller then delegates the task of executing a test instance to one or more agents. Figure 8 The interactions between test client, test controller and test agents are governed by the Run Settings and the Run Configurations settings. Run Configurations Run Settings are specific to load tests; Run Configurations define more generic settings applicable to most test types (Figure 9): Test run naming conventions Whether tests are run locally or on a remote Test Rig © 2006 Chris Birmele, Microsoft Corporation 6 Understanding VSTS Load Testing Framework Assemblies that need to be instrumented to collect Code Coverage information Additional files and directories to be deployed for tests to run successfully (the test code is automatically deployed) Generic setup and cleanup scripts that must be executed as part of a test run Test abortion and test failure timeouts Web Test specific settings that need to be persisted across multiple test runs A Visual Studio Team System solution can have multiple Run Configurations, but only one can be active at anyone time (Figure 9). Figure 9 Test Rig Tests - load tests and others - run either locally or on a remote Test Rig. For a local load test run, all required components are located on the test client machine (i.e. Visual Studio Team System). Generating test instances to simulate virtual users requires memory and CPU cycles. There is a limit of how many test instances can be generated on a single computer. Test Client Test Agent Test Controller System Under Test Figure 11 If you need to simulate a large number of test instances that can not be generated from a single computer you need to set up a Test Rig. A Test Rig consists of one test controller and a collection of test agents that generate the desired user load (Figure 12). © 2006 Chris Birmele, Microsoft Corporation 7 Understanding VSTS Load Testing Framework In this scenario, the test client communicates with a remote test controller to initiate a test run. The test controller collaborates with all registered test agents each of which executes test instances against the system under test. Test Client Test Rig Test Controller The Run Configurations settings determine how the test controller distributes the test instances among the test agents. Test controllers and test agents have default counter sets assigned, determining the performance data to collect. Test Agent Test Agent Visual Studio Team System automatically maps these counter sets to the test controller and test agents that make up the test rig. Test Agent System Under Test Figure 12 Test Management Repository One of the most important components of Visual Studio Team System is the Team Foundation Server or TFS. It is beyond the scope of this paper to describe all features of TFS, but suffice to say that here it serves as the Test Management Repository (see Figure 13). Test Client Test Management Interface Web Service Team Foundation Server The test assets, i.e. the tests (load test as well as other tests) and the Run Configurations are stored in the version control system of TFS. They can be viewed, compared, branched and merged like any other file. The results of individual test runs are stored in a separate database. The summary of these runs is recorded in the data warehouse. Source Control Build / Test Results Data Warehouse Work Items Test runs can be associated with any work items. For example a failed test can be associated with a new defect, or a successful test can be associated with a Quality of Service requirement to prove the requirement has been satisfied. © 2006 Chris Birmele, Microsoft Corporation Figure 13 8 Understanding VSTS Load Testing Framework Figure 14 shows how these scenarios appear in Visual Studio Team System. Work Item Associations Test Asset Version Control Test Run Summaries Team Foundation Server Figure 14 © 2006 Chris Birmele, Microsoft Corporation 9 Understanding VSTS Load Testing Framework References The content of this paper is based on a variety of information sources, such as blogs, Microsoft presentations, the Visual Studio Team System online help and others. To the best of my knowledge the explanations given in this document are correct, but I would appreciate any corrections and constructive suggestions of how this paper could be improved. Sydney, June 06 Chris Birmele Developer Tools Technical Specialist Australia & New Zealand cbirmele@microsoft.com © 2006 Chris Birmele, Microsoft Corporation 10