System Testing
and Test Plan
CS 4311
Hans Van Vliet, Software Engineering, Principles and Practice,
3rd edition, John Wiley & Sons, 2008. Chapter 13.
I. Burnstein. Practical Software Testing, Springer-Verlag, 2003.
1
Hierarchy of Testing
Testing
Ad Hoc
Program Testing
Unit Testing
Integration Testing
Black Box
Top Down
Equivalence
State Transition
Acceptance
Testing
Function
Properties
Benchmark
Performance
Pilot
Reliability
Alpha
Bottom Up
Boundary
Decision Table
System Testing
Big Bang
Sandwich
Availability
Beta
Security
Use Case
Usability
Domain Analysis
Documentation
Portability
White Box
Control Flow
Data Flow
Capacity
2
Outline

System testing
 Function
testing
 Properties testing


Acceptance testing
V&V documents
 Test
plan
3
Describe the following types of
system testing

Function testing

Property testing

Acceptance testing

Installation testing
Pairs
4
Types of System Testing

Function testing
 Integrated

Property testing
 Integrated

system performs specified function
system meets non-functional requirements
Acceptance testing
 Customers

test system (beta tests)
Installation testing
 If
acceptance tests are in different environment
5
Function Testing

General strategy
 Identify
threads, sequences of actions associated with
a function of the system under test
 Test one thread at a time

Most of the black box techniques discussed
apply here, esp. use cases
6
Test Cases from Use Cases

Use case


High level description of interactions or actions that must be
performed by actor and system
Strategy

Draft a test case for each use case success path and each use
case alternate path


Test case steps from the use case sequence of actions
Refine a draft test case to include sufficient details to either write
code or execute tests


Do iteratively as the development continues (e.g., UI and design)
Details such as menus, data entry web pages, data search web
pages, report web pages, printouts, and so forth
=> Possibly more test cases, e.g., testing individually before testing
together
7
Example
Use case
•
Customer logs on by providing a login name
and password
•
System checks the login name and password
•
System shows the main menu consisting of …
•
Customer browse catalog by selecting …
…
Test case from use case
Step
No.
Step
Expected Result
1.
Log on
Main menu appeared
2.
Browse catalog
A “blue widget” found
3.
Browse product detail
The blue widget description displayed
4.
Update the shopping cart
The blue widget in the shopping cart
5.
Purchase shopping cart products
The blue widget bought
6.
Purchase order completion
A blue widget purchase confirmation received
7.
Log off
The application exited successfully
8
Example
Use case
•
Customer logs on by providing a login name
and password
•
System checks the login name and password
•
System shows the main menu consisting of …
•
Customer browse catalog by selecting …
…
Test case from use case
Step
No.
Step
Expected Result
1.
Log on
Main menu appeared
2.
Browse catalog
A “blue widget” found
3.
Browse product detail
The blue widget description displayed
4.
Update the shopping cart
5.
6.
7.
blue widget added
InThe
general,
Purchase shopping cart products
The blue widget bought
actor action
=> test step
Purchase order completion
blue widget purchase confirmation received
systemAaction
=> result
Log off
The application exited successfully
9
Example: Refine
Include sufficient details to write code or execute tests
(a) Detailed interactions, e.g., screens and menus
(b) Detailed and sample data
Step
No.
Step
Expected Result
1.
Launch the purchase order screen (select “Purchase”)
Screen appeared
2.
Enter purchase name (“Diana N.”)
Valid name accepted
3.
Enter purchase address street (“500 W. Univ., El Paso”)
Street address accepted
e
Enter purchase address state (“Texas”)
State selected
5.
Enter purchase address zip (“79968”)
Zip code accepted
6.
Select method of payment (credit)
Check/credit card selected
7.
Exit purchase order screen (click “exit” button)
Screen stopped successfully
10
Property (Performance) Testing











Stress tests: Maximum throughput on all subsystems simultaneously
Volume tests: Can system handle maximum amount of data?
Configuration tests: Can system handle specified configurations?
Compatibility tests: Does software work with other software?
Security tests: Does software pass basic security screen?
Timing tests: Response and throughput times?
Recovery tests: Does system recover from known faults?
Overload testing: Exceed specifications
Robustness testing: Test things not in specifications
Human factors tests: Usability
Documentation tests
11
Acceptance Testing

Benchmark tests
 Typical

Pilot tests
 Install

operation tests prepared by customer
and let users use system
Alpha tests
 In-house

pilot
Beta tests
 On-site
pilot
12
Outline
 System testing
 Function testing
 Properties testing
 Acceptance testing
 V&V documents
 Test
plan
13
Test Plan

Objectives



Product: may need to sell test plan (Military MIL spec, FDA
specs, commitment for support, etc.)
Tool: helps manage testing project and discover and fix
faults
Testing documentation



Facilitates task of testing
Improves communication about testing
Provides structure for



Organizing
Scheduling
Managing
14
Things to Include in Test Plan

Lists/tables


Outlines


E.g., function list, top-level/user-visible functions, sub-functions (options
or submenus), entry and exit conditions on fully parameterized methods
Matrices


Files, features and functions, inputs and outputs, error messages
List function/operation vs. test conditions, e.g., the save operation with
the following conditions: disk full, almost full, write-protected, …
Notes

How to run test, expected results, special instructions, one-shot or
regression test, what test is looking for, assumptions in the test
15
Test Plan Report







Table of contents
Document control
Approval
Document change control
Distribution list
Change summary
1. Introduction

1.1 Purpose
 1.2 Scope
 1.3 System overview
 1.4 Test approach overview




2. Applicable references
3. Testing approach
4. Testing schedule
5. Test cases

5.1 System testing




Test case XXX
…
5.2 Integration testing
5.3 Unit testing
16
Testing Approach (Section 3)




Specify types of tests to be performed
List specific tests
Test descriptions are in Section 5
May include:





Test management requirements
Personnel requirements
Hardware requirements
Software requirements
Cost
17
Approach




Who does it
Techniques and activities
When is a group of features adequately tested
Constraints: deadlines, availability of people, etc.
18
Section 5 and later


Include planned/performed test cases
Each test case should include





Unique test case ID (or test number)
Unique test title
Brief description
Current status (passed/failed/pending)
Testing approach including




Step, operator action, purpose, expected results
Concluding remarks
Testing team
Date completed
19
Test Case Template
Test No.
<unique test ID>
Current status
 Passed
Title
<unique test tile>
Description
<short description of this test>
Approach
<strategy that is being used on this test>
Step No.
Action
 Failed
Purpose
 Pending
Expected result Comment
Concluding remark
Testing team
<lead and members of testing team>
Date completed
20