Tor Stålhane
Requirements Specification and Testing
Requirements testability
Testability definition
According to ISO 9126, testability is defined
as:
Testability: The capability of the software
product to enable modified software to be
validated.
NOTE - Values of this sub-characteristic
may be altered by the modifications under
consideration.
Testability concerns
Testability touches upon two areas of concern:
• How easy is it to test the implementation?
• How test-friendly is the requirement?
These two concerns are not independent and
need to be considered together. We will first
look at it from the requirements side.
Testability
Three basic ways to check that we have achieved our
goals:
• Executing a test. Give input, observe and check
output. A test can be a
– Black box test
– White box test
– Grey box test
• Run experiments
• Inspect the code and other artifacts
Usually, we will include all of these activities in the
term testing
When to use what
The diagram on the next slide is a high level
overview of when to use
• T – Tests. Input / output. Involves the
computer system and peripherals.
• E – Experiments. Input / output but
involves also the users.
• I – Inspections. Evaluation based on
documents.
Concrete requirements from high level goals
T
E
T
E
I
T
I
E
E
E
TDT 4242
Testability
In order to be testable, a requirement needs
to be stated in a precise way. For some
requirements this is in place right from the
start:
When the ACC system is turned on, the
“Active” light on the dashboard shall be
turned on.
In other cases we need to change a
requirement to get a testable version.
The system shall be easy to use.
Testability challenges - 1
Some requirements are more difficult to test
than others. Problems might rise due to:
• Volume of tests needed, e.g. response
time or storage capacity.
• Type of event to be tested, e.g. error
handling or safety mechanisms.
• The required state of the system before
testing, e.g. a rare failure state or a certain
transaction history.
Testability challenges – 2
We can test the requirements at any level. The
formulation of the test will depend on the level
Making a requirement testable – 1
One way to make requirements testable is
the “Design by Objective” method
introduced by Tom Gilb.
The method is simple in principle but is in
some cases difficult to use. There are two
problems
• The resulting tests can in some be rather
extensive and thus quite costly.
• The method requires access to the
system’s end users.
Making a requirement testable – 2
1. What do you mean by <requirement>?
This will give us either (a) a testable
requirement or (b) a set of testable and
non-testable sub-requirements.
2. In case (a) we are finished. In case (b) we
will repeat question 1 for each nontestable sub-requirement
Making a requirement testable – 3
Requirement: Reverse thrust may only be
used, when the airplane is landed.
The important questions are
• “How do you define landed?”
• Who should you ask – e.g. pilots, airplane
construction engineers, or airplane
designers?
Requirements for testability – 1
First and foremost:
The customer needs to know what he wants
and why he wants it. In some cases it is
easier to test if the user actually has
achieved his goal than to test that the
system implements the requirement.
Unfortunately, the “why”-part is usually not
stated as part of a requirement.
Requirements for testability – 2
Each requirement needs to be
• Correct, i.e. without errors
• Complete, i.e. has all possible situations
been covered?
• Consistent, i.e. not in disagreement with
other requirements.
• Clear, i.e. stated in a way that is easy to
read and understand – e.g. using a
commonly known notation.
Requirements for testability – 3
Each requirement needs to be
• Relevant, i.e. pertinent to the system’s
purpose and at the right level of
restrictiveness.
• Feasible, i.e. possible to realize. If it is
difficult to implement, is might also be
difficult to test.
• Traceable, i.e. it must be possible to relate
it to one or more
– Software components
– Process steps
Completeness
All possible situations must be covered.
“If X then….”, “If Y then….” Must also
consider what will happen “If neither X nor
Y…”
Automatic door opener – what is missing?
If the door is closed and a person is
detected then send signal Open_Door. If
no person is detected after 10 sec., send
signal Close_Door.
Consistency
Consistency is a challenge since we, at least
in the general case, need a complete
overview of all requirements.
In most cases, we can make do with
checking all requirements that are related
to the same event, function or parameter.
Clear – 1
This is mainly a question of representation
such as choice of
• Diagram notation
• Description language
• Level of details
Who shall understand the requirement?
• Customers
• Developers, including hired-in consultants
• Testers
Clear – 2
Simple example:
Print the “accounts ledger” for all accounts
This requirement is perfectly clear for
developers who are working in the banking
business.
Other developers might experience some
problems.
Relevant
Two questions are important:
• Do we really need this requirement?
• Is it at the right level of strictness – i.e. not
too strict and not too lax.
Only the second question is important for
the tester.
• Too strict means more work for the
developers
• Too lax means more work for the tester.
Feasible
This question is really related to the contract but
we should also consider it here – can we really
do this?
Testers can contribute to the feasibility question by
asking how it should be tested. This will help /
force everybody involved to make the
requirement more clear and thus improve on the
requirement.
Requirements that are difficult to tests are also
usually difficult to implement – mainly because
they are badly defined.
Some sound advice
The following set of advices on requirements and
testability are quoted from Ludwig Consulting
Services, LLC.
They are not a definition and not “the final words”
on requirements testability. Instead, they should
be used as a checklist.
That one of the following rules are not obeyed does
not mean that the requirement is wrong. It
should, however, be reviewed for potential
problems.
Modifying Phrases
Words and phrases that include:
• as appropriate
• if practical
• as required
• to the extent necessary / practical.
Their meaning
• is subject to interpretation
• make the requirement optional
Phrases like "at a minimum" only ensure the minimum,
while "shall be considered" only requires the
contractor to think about it.
Vague Words
Vague words inject confusion. Examples of frequently
used vague verbs are:
• manage
• track
• handle
• flag
Information systems receive, store, calculate, report, and
transmit data. Use words that express what the system
must do.
Requirement: The system shall process ABC data to the
extent necessary to store it in an appropriate form for
future access.
Correction: The system shall edit ABC data.
Pronouns With No Reference
Example: It shall be displayed.
When this occurs, the writer is usually relying on
a nearby requirement in the requirements
document for the meaning of "it."
As requirements are assigned for
implementation, they are often reordered and
regrouped, and the defining requirement is no
longer nearby.
Passive Voice
Requirements should be written in active voice,
which clearly shows X does or provides Y.
Passive voice: Z shall be calculated.
Active voice: the system shall calculate Z.
Negative Requirements
Everything outside the system is what the system
does not do.
Testing would have to continue forever to prove
that the system does not do something.
State what the system does. Substitute an active
verb that expresses what the system must do.
• Change "the system shall not allow X," to "the
system shall prevent Y."
• Use the prefix "un," such as: The system shall
reject unauthorized users.
Assumptions and Comparisons – 1
The requirement "the system shall increase
throughput by 15%" sounds testable, but isn't.
The assumption is "over current system
throughput." By comparing to another system,
the meaning of the requirement changes
when the other system changes
Assumptions and Comparisons – 2
An example, sometimes found in requests for
proposals, is:
"The system shall address the future needs of
users."
The writer is probably thinking ahead to after the
contract is awarded. The requirement is
meaningless because whenever it is read, it will
point to the future.
A requirement on change management included in
the project management processes, would make
more sense than making it a requirement for the
system.
Indefinite Pronouns
Indefinite pronouns are “stand in” for unnamed people or
things, which makes their meaning subject to interpretation.
Some of these may find their way into requirements:
•
•
•
•
•
•
•
•
All
Another
Any
Anybody
Anything
Each
Either
Every
•
•
•
•
•
•
•
Everybody
Everyone
Everything
Few
Many
Most
Much
The implementation
We will shortly look at four concerns related to
implementation and testability:
• Autonomy of the system under test
• Observability of the testing progress
• Re-test efficiency
• Test restartability
Autonomy – 1
How many other systems are needed to test this
requirement?
It is best if it can be tested using only the SUT
and the autonomy and testability is
successively reduced as the number of other,
necessary systems increase.
If the other systems needed are difficult to
include at the present we will have to write
more or less complex stubs.
Example – 1
Requirement: “If the door is closed and a
person is detected then send signal
Open_Door”
• Sensors and actuators can be tested in
the lab.
• The system with a simulated actuator:
simulate a “person detected” signal on the
sensor and check if a Open_Door signal is
sent to the actuator.
Example – 2
We can build a complete system – door, sensor,
door motor and software system and test by
• Letting persons approach the sensor
• Check if the door opens
– Early enough
– Fast enough
Observability
How easy is it to observe the
• Progress of the test execution?
This is important for tests that do not produce
output – e.g. the requirement is only
concerned with an internal state change or
update of a database.
• Results of the test?
Important for tests where the output is
dependent on an internal state or database
content.
Re-test effiencey
Retest efficiency is concerned with how easy it is
to perform the “test – check – change – re-test”
cycle.
This includes
• Observability – observe the test result
• Traceability – identify all tests related to the
change
Test restartability
This is mostly a question of check points in the
code. How easy is it to
• Stop the test temporarily
• Study current state and output
• Start the test again from
– The point where it was stopped
– Start
Final comments
That a requirement is testable does not
necessarily mean that it is easy to test.
In order to have testable requirements it is
important that
• The testers are involved right from the
start of the project. It is difficult to add
testability later.
• The tests are an integrated part of the
requirement