CEN 5076 Class 3 – 09/19
Testing Terminology
•
•
•
•
•
•
Review Class 2
Test Process Overview
Risk Analysis
The Testing Process
Planning Activities
Team Meeting
Laws of Testing Handbook of SSE
•
•
Online debugging is more efficient than offline
debugging (Sackman’s).
Testing can show the presence but not
the absence of errors (Dijkstra).
•
A developer is unsuited to test his or her code
(Weinberg’s).
•
Approximately 80 percent of defects come
from 20 percent of modules (Paret-Zipf-type)
•
Performance testing benefits significantly
from system-level benchmarks (Gray-Serlin)
CEN 5076 Class 3 - 09/19
2
Laws of Testing Handbook of SSE
•
•
•
•
•
Usability is quantifiable (Nielsen-Norman law).
Partition testing is more effective than random
testing (Gutjahr’s hypothesis).
The adequacy of a coverage criterion can
only be intuitively defined (Weyuker’s
hypothesis).
The test suite needed to verify an arithmetic
path expression can be determined (EndresGlatthaar hypothesis).
Suspicion-based testing can be more
effective than most other approaches
(Hamlet’s hypothesis).
CEN 5076 Class 3 - 09/19
3
Taxonomy of OO Classes
•
•
•
Class Abstraction Technique (CAT) that
support implementation-based testing.
Catalogs classes based on the characteristics
of a class.
“Class characteristics for a given class C are the
properties of the features in C and the relationships C
has with other classes in the implementation. The
properties of the features in C describe how criteria
such as types, accessibility, shared class data,
polymorphism, dynamic binding, deferred features,
exception handling, and concurrency are represented
in the attributes and routines of C.”
[Clarke and Malloy SEA ’04]
CEN 5076 Class 3 - 09/19
4
Taxonomy of OO Classes cont
Descriptors
Nomenclature
Attributes
(Nested)
New
(Multi-Parents)
Recursive
(Friend)
Concurrent
(Has-Friend)
Polymorphic
Generic
Private
Concurrent
Protected
Abstract
Public
Inheritance-free
Constant
Parent
Static
External Child
Internal Child
-
Type
Routines
(Constant)
New
Recursive
Redefined
Concurrent
Synchronized
Exception-R
Exception-H
Has-Polymorphic
Non-Virtual
Virtual
Deferred
Private
Protected
Public
Static
CEN 5076 Class 3 - 09/19
Families
NA
no type
P
primitive type
P*
reference to P
U
user-defined type
U*
reference to U
L
library
L*
reference to L
A
any type (generics)
A*
reference to A
m <n>
parameterized type
m <n *> reference to
parameterized type
where m ε {U, L}
n is any combination of
{ P, P*, U, U*, L, L*, A, A* }
-
5
Taxonomy of OO Classes cont
CEN 5076 Class 3 - 09/19
6
Types of Inheritance
•
Model Inheritance – is-a relation between
abstractions in the model.
-
•
Subtype
View
Restriction
Extension
Variation inheritance – expresses relations within the
software itself rather than the model.
-
Functional
Type
Uneffecting
CEN 5076 Class 3 - 09/19
7
Types of Inheritance
•
Software inheritance – describes a class by how it
differs from another class.
-
Reification
Structure
Implementation
Facility – constant, machine
[Bertrand Meyer ’96]
CEN 5076 Class 3 - 09/19
8
Planning for Testing
•
Testing requires considerable resources.
•
Good planning and good management is
required for the effective utilization of these
resources.
•
Want to plan a test process that complements
your development process.
•
Want to analyze the risks associated with
verifying the required functionality.
•
Need to develop test plans for the different
levels and types of testing required for a
comprehensive test process.
CEN 5076 Class 3 - 09/19
9
Testing Process Overview
•
Testing is applied at various points during
development.
•
Development and testing processes have
different goals and different measures of
success.
•
Development strives to build a product that
meets a need.
•
Testing strives to answer questions about
the product, including whether the product
meets the need that it is intended to meet.
CEN 5076 Class 3 - 09/19
10
Testing Process Overview
•
Example:
-
-
The lower the defect rate (ratio of test cases that
fail to the total number used), the more
successful the development process is
considered to be.
The higher the defect rate, the more successful
the testing process is considered to be.
•
The roles of development and testing are
usually assigned to different people.
•
Developers are responsible for some testing,
e.g., unit testing and integration testing.
CEN 5076 Class 3 - 09/19
11
Testing Process Overview
•
To ensure testing is carried out with the same
vigor when it is done by developers a buddy
system is used.
•
Buddy system – functionality is partitioned
and 2 developers take turns in writing code
and testing.
•
The development and testing processes are
in a feedback loop, recall testing model USDP
diagram.
•
Recall in the USDP increments of the system
are usually developed.
CEN 5076 Class 3 - 09/19
12
Testing Process Overview
Incremental approach:
Increment 1 – Analysis , Design, Implementation,
Testing.
Increment 2 – Analysis , Design, Implementation,
Testing.
…
Increment n – Analysis , Design, Implementation,
Testing.
CEN 5076 Class 3 - 09/19
13
Testing Process Overview
•
The testing perspective must be considered,
preferably by professional testers, when
development methods and tools are selected.
•
The form and quality of the requirements
specification also affects the testing process.
•
Product requirements comprise the source of
test cases in system and acceptance testing.
•
System testers should participate in the
gathering and validation of the reqs. – need to
understand reqs., assess risks, and check for
testability.
CEN 5076 Class 3 - 09/19
14
Testing Process Overview
•
Testability:
1. The degree to which a system or component
facilitates the establishment of test criteria and
the performance of tests to determine whether
those criteria have been met. [IEEE 610]
2. The degree to which a requirement is stated in
terms that permit establishment of test criteria
and performance of tests to determine whether
those criteria have been met. [IEEE 610]
•
Test criteria
-
The criteria that a system or component must
meet in order to pass a given test. [IEEE 610]
CEN 5076 Class 3 - 09/19
15
Testing Process Overview
There are two types of testing criteria:
1. Test data selection criterion – represents
a rule used to determine which test case to
select.
2. Test data adequacy criterion – a rule used
determine whether or not sufficient testing has
been performed.
• Test data selection criterion serves as the
basis for picking a test set to satisfy some
goal, while a test data adequacy criterion
checks to see whether a previously selected
test set satisfies the goal. [Weyuker 93]
CEN 5076 Class 3 - 09/19
16
Testing Process Overview
STEP Testing technique [Hetzel 84]
1. Analysis – product to be tested is examined
to identify any special features that must
receive particular attention and to determine
test cases that should be constructed.
2. Construction – artifacts that are needed for
testing are created. Test cases are translated
into programming languages and scripting
languages, or they are entered in a toolspecific language. Data sets required for
testing are built.
CEN 5076 Class 3 - 09/19
17
Testing Process Overview
STEP Testing technique [Hetzel 84] cont
3. Execution and Evaluation – most visible and
recognized part of the test effort. Test cases
are executed and the results examined to
determine if the software passed or failed the
test suite.
•
Test suites must be maintained, i.e.,
1. as reqs. changed so does the test suite,
2. as problems are found by users test cases will be
added to catch those problems.
CEN 5076 Class 3 - 09/19
18
Risk Analysis – A Tool for Testing
•
Risk
1. Anything that threatens the successful
achievement of a project’s goal.
2. An event that has some probability of occurring,
and if it does there will be some lost.
•
Risk-based testing principle:
-
•
Test most heavily those portions of the system
that pose the highest risk to the project to ensure
that the most harmful faults are identified.
Risk analysis – a procedure for identifying
risks and for identifying ways to prevent
potential problems from becoming real.
CEN 5076 Class 3 - 09/19
19
Risk Analysis – A Tool for Testing
•
Each project requires it own individual
analysis, therefore it is important to apply
testing strategies that make sense.
[“Effective software testing: 50 specific ways to
improve your testing”. E. Dustin 2003 – see
updated online course syllabus]
Some of these strategies include:
•
Select test-design techniques – numerous
techniques are available.
CEN 5076 Class 3 - 09/19
20
Risk Analysis – A Tool for Testing
•
Select testing tools – decide on the vendorprovided tools; how the tools will be used,
which team member will use them.
•
Develop in-house test harness or scripts.
•
Determine test personnel and expertise
required.
-
To write test harness and scripts a developer
must be included in the testing team.
Automation skills required for capture/playback
skills.
Domain expertise usually required.
CEN 5076 Class 3 - 09/19
21
Risk Analysis – A Tool for Testing
•
Determine testing coverage
-
-
-
-
It is essential that testers understand the
coverage required e.g., may be a contractual
agreement in SRD.
There might be a code coverage requirement
e.g., DOD.
Determine test coverage given the resources,
schedules, tools, task at hand, and risks of not
testing an item.
Beizer estimates that from 2% to 80% of the
application size can be code to support testing.
CEN 5076 Class 3 - 09/19
22
Risk Analysis – A Tool for Testing
•
Establish release criteria
-
Indicate when testing can be considered
complete.
-
Try to state in a quantifiable manner e.g., use
cases 1, 3, and 4 need to be defect free.
•
Set the testing schedule
•
Consider the testing phases
-
Different test strategies apply to different test
phases. (see V-model for testing phases in
traditional model)
CEN 5076 Class 3 - 09/19
23
Risk Analysis – A Tool for Testing
High risk factors:
• Short time-to-market
-
•
New design process
-
•
•
Can prevent adequate testing.
Test strategies adapted for the time available.
Intro of new design tools, techniques.
Example change from USDP to eXtreme
programming approach or vise versa.
New technology
Complexity
-
Identify high risk uses case during reqs. analysis.
CEN 5076 Class 3 - 09/19
24
Risk Analysis – A Tool for Testing
•
Frequency of use
-
•
Untestable functional and non-functional reqs.
-
•
This functionality usually represents the core of
the application.
A reason to quantify non-functional requirements.
Must make all requirements testable.
Other sources of risk:
-
Prog. Languages: permits certain classes of
errors and inhibit others e.g., strong typing vs.
weak typing.
CEN 5076 Class 3 - 09/19
25
Risk Analysis – A Tool for Testing
Class Project:
•
Each use case should have a risk, frequency,
and criticality associated with it.
•
Review each of the use cases to ensure the
above properties are quantified and valid.
•
Produce a ranked list of use cases. Focus on
the implemented use cases.
•
Use the risk, frequency, and criticality to identify
the testing effort (time and personnel) required
for each use case during system testing.
CEN 5076 Class 3 - 09/19
26
A Testing Process
Dimensions of software testing (think of the
attributes in italics as a continuum):
1. Who performs the testing?
-
Developer, tester, independent tester.
2. Which pieces will be tested?
-
Test nothing, test a sample, test everything.
-
Use a systematic approach. P. 81
CEN 5076 Class 3 - 09/19
27
A Testing Process
Dimensions of software testing cont:
3. When will testing be performed?
-
When components are developed, when all
components are developed and integrated.
4. How will testing be performed?
-
Specification, implementation.
5. How much testing is adequate?
-
Not testing, exhaustive testing.
CEN 5076 Class 3 - 09/19
28
Roles in Testing Process
Test plan should:
•
Identify the roles each person will be
assigned.
•
For each role allocate time and effort.
•
Schedule time allocated for each part of the
testing effort.
•
Development schedule drives much of the
testing schedule.
•
Identify the resources needed for the testing
effort e.g., h/w, s/w, expertise
CEN 5076 Class 3 - 09/19
29
Roles in Testing Process
Roles:
•
Unit tester – responsibility is to test the
individual classes (cluster of classes) as they
are produced.
•
Integration tester – responsibility testing a
set of objects that are being brought together
from different development sources e.g.,
individuals or teams.
•
System tester – has domain knowledge and
is responsible for independently verifying that
the completed application satisfies the reqs.
CEN 5076 Class 3 - 09/19
30
Roles in Testing Process
•
Test manager – responsible for managing the
test process i.e., requesting, coordinating, and
making effective use of the resources allocated.
Other Roles see handout [Dustin]:
•
Team lead – technical leadership for the test
program, including test approach.
•
Test engineers (usability, manual, automated,
network, security) – specialist testers in each of
the areas.
•
Test environment specialist – installs test tools
and establishes test-tool environment
CEN 5076 Class 3 - 09/19
31
Test Plan
Class Project:
•
Roles for testing each model: test manager,
test engineer (system, integration, class)
tester, and minute keeper.
•
Keep a diary of all activities performed at
each meeting.
•
Use information in handout to identify the
duties/skills of each person on the team e.g.,
P. 73 first entry in table 13.2 .
CEN 5076 Class 3 - 09/19
32
Roles in Testing Process
•
•
•
Each diary entry should contain: start time of
meeting, end time of meeting, persons
present, location of meeting, description of
topics discuss (bullet form), tasks assigned to
team members.
Roles in the team will be rotated as each
USDP model is tested. Initially students that
developed the applications will be domain
experts.
Use an automated tool to generate a testing
schedule e.g., MS project. Identify tasks,
milestones, and deliverables for the testing
activity.
CEN 5076 Class 3 - 09/19
33
A Detailed Set of Test Activities
•
Fig 3.11 Synopsis of testing activities, P. 88
•
Domain analysis
•
Application analysis
•
Architectural design
•
Detailed design
•
Class implementation
•
Application implementation
CEN 5076 Class 3 - 09/19
34
Documentation
IEEE 829 Standard Test Plan outline:
1.0 Introduction – high level view of the testing, includes
type of testing e.g., class, subsystem, system,
acceptance, release.
2.0 Test Items – defines the scope, h/w and s/w to be
tested.
3.0 Tested Features – parts of the s/w spec to be tested.
4.0 Features Not Tested – includes features already
tested
5.0 Testing Strategy and Approach
5.1 Syntax
CEN 5076 Class 3 - 09/19
35
Documentation
5.2 Description of Functionality
5.3 Arguments for Test – includes preconditions
5.4 Expected Output
5.5 Specific Exclusions
5.6 Dependencies
5.7 Test Case Success/Failure Criteria
6.0 Pass/Fail Criteria for the Complete Test Cycle
7.0 Entrance Criteria/Exit Criteria
8.0 Test-Suspension Criteria and Resumption Criteria
CEN 5076 Class 3 - 09/19
36
Documentation
9.0 Test Deliverables/Status Communication Vehicles
10.0 Testing Tasks
11.0 H/w and S/w reqs.
12.0 Problem Determination and Correction
Responsibilities
13.0 Staffing and Training Needs/Assignments
14.0 Test Schedules
15.0 Risks and Contingencies
16.0 Approvals
CEN 5076 Class 3 - 09/19
37
Documentation
Visit the following site for more information:
http://www.coleyconsulting.co.uk/IEEE829.htm
http://www.evolutif.co.uk/tkb/guidelines/ieee829/example.h
tml
CEN 5076 Class 3 - 09/19
38
Project Test Plan
•
Summarizes testing strategy to be employed
for the project.
•
Fig. 3.15 P. 97 summarizes the activities that
are required, the frequency with which each
activity will be employed, and the entity
responsible for this testing phase. Required
for class project!
•
Fig. 3.16 P. 97 associates each of the testing
phases with the specific strategy for that
phase. Required for class project!
CEN 5076 Class 3 - 09/19
39
Component Test Plan
•
Defines the overall strategy and specific test
cases that will be used to test a certain
component.
•
One plan per significant component (see uses
cases).
•
Fig. 3.17 P. 100 shows a template for the test
plan. Required for class project!
•
Each section of the plan contains two types of
guided information:
CEN 5076 Class 3 - 09/19
40
Component Test Plan
1. Project criteria – standards that have been
agreed upon and how thoroughly each
component will be tested,
e.g., 100% of the postconditions on modifier methods
should be tested.
2. Project procedures – techniques that have been
agreed upon as the best way to handle a task.
Provides the details of the test strategies that
were identified in the project plan.
e.g., constructing a Parallel Architecture for Class Testing
(PACT) class for each component that will be tested.
CEN 5076 Class 3 - 09/19
41
Component Test Plan
Sections of template:
•
Objectives for the class - prioritized list of
objectives for the component.
•
Guided Inspection Requirements – (see P.99).
•
Building and Retaining Test Suites
-
Process of creating test driver classes.
-
Scheduled deadline for delivery of test cases.
-
The specification of the test driver
-
Relative number of test cases (prioritized ) in each of
the following categories.
CEN 5076 Class 3 - 09/19
42
Component Test Plan
•
•
Functional Test Cases
-
Approach used to develop test cases from the
specification.
-
The class invariant method
-
Identify the “types” of objects being tested. Based on
the initial state of the object.
Structural Test Cases
-
Info about test cases developed for code coverage
and code-review process.
-
How to use the required test-coverage tool
CEN 5076 Class 3 - 09/19
43
Component Test Plan
•
•
State-Based Test Cases
-
State representation for object.
-
Approach used to generate test cases.
Interaction Test Cases
-
How dependencies will be handled.
-
Creation of stubs required to handle cycles.
CEN 5076 Class 3 - 09/19
44
Use Case Test Plan
•
Describes the system level test to be derived
from a single use case.
•
Incorporated by reference into both the
integration and systems test plan.
•
Types of use cases:

High-level – are abstract use cases that are the basis
for being extended to end-to-end use cases.

Functional sub-use cases – are aggregated into endto-end system-level use cases.

End-to-end – represents a complete transaction in
the operation of the system.
CEN 5076 Class 3 - 09/19
45
Use Case Test Plan
•
•
Other uses cases:

Report – access information in the system,
summarize it, and format it for presentation to the
user.

Boundary – describe startup, shutdown, and
exceptional conditions.
Figs 3.18, 3.19, and 3.20, Pgs 102, 103, and
104.
CEN 5076 Class 3 - 09/19
46
Integration Test Plan
•
Very important in the incremental development
environment.
•
Integrates individual classes into a cluster
(component), components into subsystems,
subsystems into the system.
•
In most cases there is a need for test drivers and
stubs.
•
Integration test provides information on the order
of testing the individual classes, components,
subsystems.
CEN 5076 Class 3 - 09/19
47
Integration Test Plan
•
Test cases span the parts of the system being
integrated.
•
More complex and comprehensive than the
typical unit test.
•
Format follows that of the system test plan
(Fig. 3.21 P. 105)
CEN 5076 Class 3 - 09/19
48
System Test Plan
•
Summarizes the individual use case test
plans and provides information on additional
types of testing .
•
Note that the most of the information
provided by the IEEE test plan format have
already been captured in the individual test
plans.
CEN 5076 Class 3 - 09/19
49