MTAT.03.159: Software Testing
Lecture 05: Test Lifecycle,
Documentation and
Organisation
(Textbook Ch. 6, 7, 8)
Spring 2014
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Dietmar Pfahl
email: dietmar.pfahl@ut.ee
Structure of Lecture 6
•
•
•
•
Test Lifecycle
Test Levels (Ch. 6)
Test Planning and Documentation (Ch. 7)
Test Organisation (Ch. 8)
– Organization Approaches (Kit’s book Ch. 13)
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Management
•
•
•
•
Goal
Policy
Plan
Follow-up
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
•
•
•
•
Reach India
Go west
Get three ships
…
Test Planning
•
•
•
•
•
•
Objectives
What to test
Who will test
When to test
How to test
When to stop
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
IEEE 829-2008:
Standard for
Software and
System Test
Documentation
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Software Project Management Plan
Goals
• Business
• Technical
• Business/technical
• Political
Quantitative/qualitative
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Policies
• High-level statements
of principle or courses
of action
• Govern the activities
implemented to
achieve stated goals
Hierarchy of test plans
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Planning – different levels
Test
Policy
Number of needed levels
depends on context
Company or
product level
Test
Strategy
(product, project, company,
domain, development process,
regulations, …)
Master test plan
Acceptance test plan
Master
Master
Test
Test Plan
Plan
System test plan
Project level
Unit and integration test plan
Evaluation plan
Detailed
Detailed
Detailed
Detailed
Test
Plan
Test
Test
Plan
TestPlan
Plan
Test levels (one for each within
a project, e.g. unit, system, ...)
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Purpose of test case (planning)
• Organization
– All testers and other project team members can review and use test
cases effectively
• Repeatability
– Know what test cases were last run and how so that you could
repeat the same tests
• Tracking
– What requirements or features are tested?
– Tracking information’s value depends on the quality of the test
cases
• Evidence of testing
– Confidence (quality)
– Detect failures
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Defect Report
(Test incident report)
• Summary
• Incident Description
• Impact
Lab 1
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Defect Report (Test incident report)
Summary
• This is a summation/description
of the actual incident.
– Provides enough details to
enable others to understand
how the incident was
discovered and any relevant
supporting information
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
• References to:
– Test Procedure used to
discover the incident
– Test Case Specifications that
will provide the information to
repeat the incident
– Test logs showing the actual
execution of the test cases and
procedures
– Any other supporting
materials, trace logs, memory
dumps/maps etc.
Defect Report (Test incident report)
Incident Description
• Provides as much details on the
incident as possible.
– Especially if there are no other
references to describe the
incident.
• Includes all relevant information
that has not already been
included in the incident summary
information or any additional
supporting information
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
• Information:
– Inputs
– Expected Results
– Actual Results
– Anomalies
– Date and Time
– Procedure Step
– Attempts to Repeat
– Testers
– Observers
Defect Report
(Test incident report)
Impact
• Describe the actual/potential
damage caused by the incident.
– Severity
– Priority
• Severity and Priority need to be
defined so as to ensure
consistent use and
interpretation, for example:
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
• Severity – The potential impact to
the system
– Mission Critical - Application will not
function or system fails
– Major - Severe problems but possible
to work around
– Minor – Does not impact the
functionality or usability of the process
but is not according to
requirements/design specifications
• Priority – The order in which the
incidents are to be addressed
– Immediate – Must be fixed as soon as
possible
– Delayed – System is usable but
incident must be fixed prior to next
level of test or shipment
– Deferred – Defect can be left in if
necessary doe to time or costs
Test results report
• Test cases executed
• Versions tested
• Defects found and reported
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Standards
• IEEE 829-1998 (revised: 2008)
Standard for Software Test Documentation
• IEEE 1012-1998
Standard for Software Verification and Validation
• IEEE 1008-1993
Standard for Software Unit Testing
--• ISO 29119
Software Testing Concepts
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test plan according to
IEEE Std 829-2008 (Appendix II)
a) Test plan identifier
b) Introduction
c) Test items
d) Features to be tested
e) Features not to be tested
f) Approach
g) Item pass/fail criteria
h) Suspension criteria and
resumption requirements
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
i) Test deliverables
j) Testing tasks
k) Environmental needs
l) Responsibilities
m) Staffing and training needs
n) Schedule
o) Risks and contingencies
p) Approvals
Test Plan (1)
Slide not shown in lecture;
Only illustrative background info
a) Test plan identifier
b) Introduction
– Product to be tested, objectives, scope of the test plan
– Software items and features to be tested
– References to project authorization, project plan, QA
plan, CM plan, relevant policies & standards
c) Test items
– Test items including version/revision level
– Items include end-user documentation
– Defect fixes
– How transmitted to testing
– References to software documentation
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Plan (2)
Slide not shown in lecture;
Only illustrative background info
d) Features to be tested
– Identify test design / specification techniques
– Reference requirements or other specs
e) Features not to be tested
– Deferred features, environment combinations, …
– Reasons for exclusion
f) Approach
– How you are going to test this system
• Activities, techniques and tools
– Detailed enough to estimate
– Completion criteria (e.g. coverage, reliability)
– Identify constraints (environment, staff, deadlines)
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Plan (3)
Slide not shown in lecture;
Only illustrative background info
g) Item pass/fail criteria
– What constitutes success of the testing
– Coverage, failure count, failure rate, number of
executed tests, …
– Is NOT product release criteria
h) Suspension and resumption criteria
– For all or parts of testing activities
– Which activities must be repeated on resumption
i) Test deliverables
– Test plan
– Test design specification, Test case specification
– Test procedure specification, Test item transmittal
report
– Test logs, Test incident reports, Test summary reports
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Plan (4)
Slide not shown in lecture;
Only illustrative background info
j) Testing tasks
– Including inter-task dependencies & special skills
– Estimates
k) Environment
– Physical, hardware, software, tools
– Mode of usage, security, office space
– Test environment set-up
l) Responsibilities
– To manage, design, prepare, execute, witness, check,
resolve issues, providing environment, providing the
software to test
m) Staffing and Training needs
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test Plan (5)
Slide not shown in lecture;
Only illustrative background info
n) Schedule
– Test milestones in project schedule
– Item transmittal milestones
– Additional test milestones (environment ready)
– What resources are needed when
o) Risks and Contingencies
– Testing project risks
– Contingency and mitigation plan for each identified risk
p) Approvals
– Names and when approved
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test plan quality criteria
Usefulness – Will the test plan effectively serve its intended functions?
Accuracy – Is the test plan document accurate with respect to any statements of
fact?
Efficiency – Does it make efficient use of available resources?
Adaptability – Will it tolerate reasonable change and unpredictability in the
project?
Clarity – Is the test plan self-consistent and sufficiently unambiguous?
Usability – Is the test plan document concise, maintainable, and helpfully
organized?
Compliance – Does the test plan meet externally imposed requirements?
Foundation – Is the test plan the product of an effective test planning process?
Feasibility – Is the test plan within the capability of the organization that must use
it?
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Structure of Lecture 6
•
•
•
•
Test Lifecycle
Test Levels (Ch. 6)
Test Planning and Documentation (Ch. 7)
Test Organisation (Ch. 8)
– Organization Approaches (Kit’s book Ch. 13)
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
7 approaches to test organisation
1.
2.
3.
4.
5.
6.
7.
Each person’s responsibility
Each unit’s responsibility
Dedicated resource
Test organisation in QA
Test organisation in development
Centralized test organisation
Test technology centre
[Kit, Software Testing in the Real World Ch 13, 1995]
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
1. Each person’s responsibility
M: Manager
P: Programmer
T: Tester
M
T
Product
developers
P
P P P P
+ Natural solution
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
- Testing own software
2. Each unit’s responsibility
M
T
Product
developers
P
M: Manager
P: Programmer
T: Tester
T
P
P P P P
+ Solves dependency
problem
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
- Two tasks
- Double competency?
3a. Dedicated resource
M: Manager
P: Programmer
T: Tester
M
Product
developers
P
T
P P P T
+ Solves multiple task problem
+ Single team
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
- Management of two types
- Competency provision
3b. Dedicated resource on a large scale
M
M
TM
Product
developers
Product
developers
Test developers
P P P P
P P P P
T T T T
+ Solves mgmt problem
of 3a
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
- Where to put the test
organiztion?
4. Test organisation in QA
PDO
QAO
Product
Development
Group
Test
Development
Group
+ Solves mgmt problem of 3b
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
- Teamwork problems?
- TDG lost in QAO
- PDG not responsible for final
product
5. Test organisation in development
PDO
Product
Development
Group
+ Solves mgmt problem of 4
+ May solve teamwork
problem of 4
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test
Development
Group
- Dependent on management
communication
6. Centralized test organisation
VP
Product
Development
Group
+ Solves mgmt problem of 5
+ Career path for test mgmt
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test
Development
Group
- VP key for test
- Teamwork at low level?
- Consistency of methods?
7. Test technology centre
VP
Product
Development
Group
SE
Test
Development
Group
+ Solves consistency of
methods problem of 6
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Test
Technology
Group
- VP key for test
- Teamwork at low
level?
Which organization
should we choose?
• Depending on
– size
– maturity
– focus
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
• The solution is often a
mix of different
approaches
Criteria for choice of
organization
A.
B.
C.
D.
E.
F.
G.
H.
Support for decision making
Enhance teamwork
Independence / Autonomy
Balance testing – quality
Assist test management
Ownership of test technology
Resources utilization
Career path
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
Decision matrix
A
B
C
D
1
2
3
4
5
6
7
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
E
F
G
H
Sum
Test Specialist Skills – Technical
• education in software
engineering principles, and
methodologies
• strong coding skills and an
understanding of code structure
• understanding of testing
principles and practices
• understanding of basic testing
strategies, methods, and
techniques
• experience to plan, design, and
execute test cases and test
procedures on multiple levels
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
• networks, databases, and
operating systems
• configuration management
• test-related documents
• define, collect, and analyze testrelated measurement data
• testing tools and equipment
• quality issues
• process issues
Test Specialist Skills
– personal and managerial
•
•
•
•
•
•
•
•
•
organizational, and planning skills
keep track of, and pay attention to, details
discover and solve problems
be creative
work in a variety of environments
work with users and clients
be able to resolve conflicts
mentor and train others
strong written and oral communication
skills
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014
MTAT.03.159 / Lecture 05 / © Dietmar Pfahl 2014