CS411W NOTES
Product Testing and Acceptance
Objectives:
1.
2.
3.
Provide an overview of types of testing typically used during the development
process
Discuss the concept of how testing approaches relate to system or productrequirements (requirements traceability)
Review methods for identifying, developing, and documenting test cases.
Development Lifecycle Revisited
Requirements
Design
Development
Test and Acceptance
Integrity Over Time
CS411W NOTES
Product Testing and Acceptance
“Program testing can be used to show the presence of bugs, but never to
show their absence!”
Esdger Dijkstra (Dutch Computer Scientist, Turing Award 1972. 1930-2002)
•
Testing – the process of verifying that a design, system, or product
conforms to specification
•
•
•
•
Verifies that all performance requirements are met
Validates user interface and it’s relationship to functional capabilities and product
performance
Verifies external interface protocols and information exchange are conducted in
accordance with interface specifications
Exercises event sequences which may identify performance deficiencies or out-ofspec/unacceptable product behavior (lockups, database integrity issues, race
conditions, memory leaks, etc.)
–
–
Race Condition - a flaw in a system or process whereby the output of the process is
unexpectedly and critically dependent on the sequence or timing of other events.
Memory Leak - unintentional memory consumption by a computer program where the
program fails to release memory when no longer needed
CS411W NOTES
Product Testing and Acceptance
Types of Testing:
Unit
Unit
Integration
Unit
Unit
System
Acceptance
CS411W NOTES
Product Testing and Acceptance
Types of Testing:
–
Unit Testing
•
•
–
The goal of unit testing is to isolate each component of a product, system, or
program and show that the performance of these individual components is correct.
Typically performed by developers of the individual unit.
Integration Testing
•
•
•
•
The goal of integration testing is to verify functional, performance and reliability
requirements of individual components operating together.
Test cases are constructed to verify that all components interact correctly.
Tests cases establish the set of conditions or variables under which a tester will
determine if a requirement are partially or fully satisfied.
Integration testing follows unit testing and may be conducted by developers or an
independent test team.
CS411W NOTES
Product Testing and Acceptance
Types of Testing:
–
System/Product Testing
•
System testing is conducted on a complete, integrated system to
evaluate the system's compliance with its specified requirements.
•
System testing is an investigatory testing phase
•
•
–
test not only the design but also the behavior
–
intended to test up to and beyond the bounds defined in the software/hardware
requirements specification(s)
Testing typically conducted during System Test phase
–
Functional Testing: Testing that ignores the internal mechanism of a system and focuses
solely on the outputs generated in response to selected inputs and execution conditions.
Functional testing verifies that product features and capabilities perform as required.
–
Performance Testing: Demonstrates that the system/product meets specified performance
criteria such as response times, managing external interfaces, system capacities, etc.
–
User Interface Testing: Verifies that the user interface performs in accordance with
specifications. Validates data entry formats, limits, action/response, and display of
information.
Normally conducted by independent system/product test team
CS411W NOTES
Product Testing and Acceptance
Types of Testing (continued):
–
Acceptance Testing: acceptance testing is performed by the customer on a
system prior to the customer accepting delivery or accepting transfer of
ownership of that system
•
Tests are conducted against supplied input data using an acceptance test
script to direct the testers.
•
Results obtained are compared with the expected results. If there is a correct
results for every test case, the test suite is said to pass. If not, the system
may either be rejected or accepted on conditions previously agreed between
the customer and the manufacturer.
•
once completed successfully, and provided certain additional (contractually
agreed) acceptance criteria are met, the sponsors will then sign off on the
system as satisfying the contract (previously agreed between sponsor and
manufacturer), and deliver final payment.
CS411W NOTES
Product Testing and Acceptance
Types of Testing (continued):
–
Regression Testing: Seeks to uncover problems introduced in
functional areas that previously worked as a result of changes
introduced during the development or product improvement lifecycle.
•
Regression testing is typically performed after any significant
modification or improvement.
•
Common methods of regression testing include re-running
previously run tests and checking whether previously fixed faults
have reemerged or new faults have been introduced.
CS411W NOTES
Product Testing and Acceptance
•
Establishing Test Requirements - “If you are not going to
identify the defects, the customer certainly will!”
–
Test Objective(s): demonstrate, validate, and measure functional and
performance characteristics of a system or product
–
Comprehensive testing: traceability between requirements and testing
is required to ensure test and validation process is complete and
comprehensive
–
Requirements tracing: the process of documenting the links between
the requirements for the system you’re building and the test cases and
procedures developed to implement and verify those requirements.
CS411W NOTES
Product Testing and Acceptance
•
Requirements Traceability
–
–
Requirements tracing helps a development team to understand which
parts of the design implement the user’s requirements, and which tests
are necessary to verify that the user’s requirements have been
implemented correctly.
•
Traceability ensures completeness, that all lower level requirements come
from higher level requirements, and that all higher level requirements are
allocated to lower level requirements
•
Traceability is also used to manage change and provides the basis for test
planning.
A Traceability Matrix is used to establish the cross-reference between a
test case document and the Functional/design specification
documents. It is a key tool for planning and conducting the testing
process.
CS411W NOTES
Product Testing and Acceptance
•
Traceability Matrix
–
Traceability:
•
•
the origin of each requirement is clear
the test cases intended to verify each requirement is clear
–
The Traceability Matrix is used to establish the cross-reference
between a test case document and the Functional/design specification
documents
–
It is a key tool for planning and conducting the testing process
Test Case 1
Requirement 1
Test Case 2
Requirement 2
Test Case 3
Requirement 3
Test Case 4
Test Case 5
Requirements to/from Case Traceability Concept
CS411W NOTES
Product Testing and Acceptance
•
Traceability Matrix (Example)
Specification to Test Cross Reference
(Backward from Requirements to Test Traceability)
Specification
Reference
Requirement
Description
Test ID
Test Case Description
1.3.2
Authorization for database access
2.1
Case 2.1-1: User Entry
Case 2.1-3: Access Authorization
Case 2.1-6: Authorization Validation
1.4.2
User removal
2.1
Case 2.1-2: User Removal
1.5.5
Authorization change
2.1
Case 2.1-5: Authorization Change
1.6.1
New user setup
2.1
Case 2.1-1: User Entry
3.2.1
Display financial profile
2.2
Case 2.5-4: Enter User ID
Case 2.2-2: Retrieve Personal Profile
CS411W NOTES
Product Testing and Acceptance
•
Traceability Matrix (Example)
Test to Specification Cross Reference
(Forward from Test to Requirements Traceability)
Test ID
Test Description
Test Cases
Specification
Reference(s)
2.1
User Setup
Case 2.1-1: User Entry
Case 2.1-2 User Removal
Case 2.1-3: Access Authorization
Case 2.1-4: Access Removal
Case 2.1-5: Authorization Change
Case 2.1-6: Authorization Validation
1.3.2, 1.6.1
1.4.2
1.3.2
1.5.3
1.5.5
1.3.2
2.2
Data Retrieval
Case 2.2-1: Retrieve Personal Profile
Case 2.2-2: Retrieve Financial Profile
TBD
3.2.1
CS411W NOTES
Product Testing and Acceptance
•
Developing Test Cases
–
–
–
–
Test case – a set of conditions or variables under which a tester will
determine if a requirement is partially or fully satisfied.
It may take many test cases to determine that a requirement is fully satisfied.
Must be at least one test case for each requirement unless a requirement
has sub requirements.
A Traceability Matrix is typically used to trace test cases back to
requirements as well as to cross-reference requirements to test events.
CS411W NOTES
Product Testing and Acceptance
Structuring Test Cases
Information (contains general information about Test case).
•
Identifier is unique identifier of test case for further references, for example, while describing found defect.
•
Test case owner/creator is name of tester or test designer, who created test or is responsible for its
development
•
Version of current Test case definition
•
Name of test case should be human-oriented title which allows to quickly understand test case purpose and
scope.
•
Identifier of requirement which is covered by test case. Also here could be identifier of use case or
functional specification item.
•
Purpose contains short description of test purpose, what functionality it checks.
•
Dependencies
Test case activity
•
Testing environment/configuration contains information about configuration of hardware or software
which must be met while executing test case
•
Initialization describes actions, which must be performed before test case execution is started. For
example, we should open some file.
•
Finalization describes actions to be done after test case is performed. For example if test case crashes
database, tester should restore it before other test cases will be performed.
•
Actions step by step to be done to complete test.
•
Input data description
Results
•
Expected results contains description of what tester should see after all test steps has been completed
•
Actual results contains a brief description of what the tester saw after the test steps has been completed.
This is often replaced with a Pass/Fail. Quite often if a test case fails, reference to the defect involved
should be listed in this column
CS411W NOTES
Product Testing and Acceptance
•
Example Test Case
Test ID: 2.1
Test Case: 1
1
2
3
4
Description: User Setup
Case Name: User Entry Version: 1.0 Written by: J. Doe
Purpose: Verify entry of new users
Setup Conditions: Database on line, administrator access authorized
Test Case Activity
Pass/Fail
Comment/Observations
Bring up New User panel
Enter User Name, Personal Data
Observe User ID displayed
Use User Name, ID to verify
Personal Data correctly entered
CS411W NOTES
Product Testing and Acceptance
•
Testing Documents
–
Test Plan: a management planning document that gives detailed testing
information regarding an upcoming testing effort, including:
•
Scope of testing
–
–
–
–
•
Schedule
–
–
•
Sequence of test events
dependencies between test events
Test Deliverables
–
–
–
•
•
what will be tested
how the testing will be done
Resources required
who will perform the tests
Test report
Data analysis
Error reports
Release/Acceptance/Pass Criteria
Risks and Contingencies
CS411W NOTES
Product Testing and Acceptance
•
Testing Documents
–
Test Specification/Procedure: defines the test setup, testing environment, and detailed
test cases to be exercised to demonstrate requirements compliance. Key components
include:
•
Test Environment (software, hardware)
•
Preparations
–
–
•
Identification of Tests
–
•
Traceability Matrix
Test Requirements
–
–
•
Equipment setup
Preparatory actions
Test Case 1
»
Goals
»
Initialization
»
Assumptions and Restrictions
»
Pass Criteria
»
Test Steps
Test Case 2 …
Special Instructions
–
–
–
Special equipment
Safety procedures
Reporting Forms
CS411W NOTES
Product Testing and Acceptance
•
Testing Documents
–
Test Report: provides summary report of exercised test procedures and
analysis of test results. Key elements include:
•
•
•
•
•
•
Management Summary
Test logs/results of exercising test procedures
Analysis of test results
Summaries of the results of each test failure
The details of any changes made during testing
Recommendations of any action to be taken as a result of the acceptance tests