System Testing
• Earlier we have stated the 2 views of testing:
– To show that the software works (according to specification)
– To detect as many defects (and have them fixed prior to releasing to
the users) as possible
• In system testing we are more interested in showing that the
system, as a whole with all the “major” specifications – especially
cross-component, is working.
• Still Interested in knowing which “major” function is not working.
(But this should be rare if we completed functional and
component test properly.)
– We are not focused on all types of exception during systems test, but
want to make sure that when multiple paths (threads) are executed, it
still works according to the requirements.
• System testing is a blend of functional (black-box) and structural
(white-box) testing - - - testing mostly “main” threads of system
behavior.
Thread
•
A thread is a sequence of activities that are
observable.
•
There are many levels of granularity for a thread:
1.
2.
3.
•
A sequence of instructions as in a DU-path for unit test
A flow of control among a set of related modules as in a
MM-path in an integration path.
A sequence of “atomic” functions that contributes to the
accomplishment of a major functionality as described in a
requirements specification.
In system testing, we are interested in the
–
–
a) high-level (cross-functional) threads and
b) “multiple” threads.
Examples of threads
from the ATM Problem (p27 of the text)
Unit
Testing
1.
Entry of a digit: (port entry of a numerical digit keystroke and a port
out of a digit echo on the screen.
–
2.
Entry of a PIN number: (this entails a set of activities from screen
request of a PIN with interleaved sequence of digit entry until
completion - - - possibly including cancellation and up to three re-tries).
–
3.
This is a sequence of atomic system functions that also qualifies as a
thread, probably fit for integration testing or unit testing levels.
A simple ATM transaction: (this includes a set of functions such as 1)
Card entry, 2) PIN entry, 3) choice of transaction type (deposit,
withdraw, account detail processing, etc.)
–
4.
This is a minimal “atomic system function” and qualifies as a thread of
interest at the unit test level.
This is a sequence of atomic system functions that is a thread &
accomplishes a major function, fit for integration testing or system testing
levels .
Multiple ATM transactions: (this includes a complete user session
where several different transactions- - - query, withdraw, deposit of
different accounts - - - may transpire.)
This is a sequence of different/multiple threads and is definitely in the
System –
domain of system testing.
Testing
Some definitions related to thread
• An atomic system function (ASF) is an action that is observable at
the system level in terms of port entry and port output.
• An ASF graph is a directed graph of a ASF represented system
where the nodes are the ASF’s and the edges represent the
sequential flow.
• A source ASF is an ASF that is a source node in an ASF graph.
• A sink ASF is an ASF that is a sink node in an ASF graph.
• A system thread is a path from a source ASF to a sink ASF in the
ASF graph of the system.
• The thread graph of a system is a directed graph where the
– a) nodes are the threads of the system and
– b) the edges represent the sequential execution flow from one thread
to another.
Requirements Specification for System Testing
• Requirements specification served as a tool (input)
for, mainly black-box testing, but also white-box
testing even at the unit testing level; it is even more
important as a source of developing test cases for
system testing.
• There are many ways to express requirements and
many ways to view the specification. As a whole the
requirements specification may be viewed a set of
inter-related elements (“basis” for requirements):
–
–
–
–
–
Data (information and information structure)
Actions (functional tasks)
Devices (for input, output, and storage)
Events (triggers and combination of action and data)
Threads (business flow or process flow, involving events)
Only thing missing is the non-functional attribute such as quality, performance, etc.
Multiple Ways of Viewing
Specification “Basis” elements
Used mostly for
early spec and
development of
system devices
Contextual
model
Data
Actions
Devices
Events
Structural
model
Used mostly for
development
Behavior
model
Often used for
requirements
specifications
Threads
There are many tools for expressing these models: (e.g.)
- state transition diagram ( or Finite State Machine)
- E-R diagram
- Petri net
Using Finite State Machine or State Transition
Diagram
State
Welcome
Card entry
(state A)
wrong card
Display screen 1;
Eject card
Correct card
Display screen 2
Pin
Entry
(state B)
Successful
Pin Process
Input
Next State
A
correct card
B
screen 2
A
bad card
A
screen 1; card
B
good pin #
C
screen 5
B
failed pin #
A
screen 4
Failed Pin Process
Display screen 4
Tabular form
Display screen 5
Choosing
Transaction
(state C)
Output
Graphical form
Finite State Machine at a “deeper” level for PIN
processing
Welcome
Card entry
(state A)
Incorrect Pin/
cancel
wrong card
Display screen 4, 1
Display screen 1;
Eject card
Correct card
Display screen 2
1st Pin
Entry
(state B)
Incorrect Pin/
cancel
2nd Pin
Entry
Display screen 3, 2 (state D)
Successful
Pin Process
Display screen 5
Choosing
Transaction
(state C)
Incorrect Pin/
cancel
Display screen 3, 2
Correct Pin
Display screen 5
Correct Pin
Display screen 5
3rd Pin
Entry
(state E)
Threads with Finite State Machine
• If we can model the requirements specifications with the finite
state machine model, then:
– A set of transitions (with inputs and outputs) can be a source of
threads for system test because it covers:
•
•
•
•
Events
Action
Data
Thread
( state and input)
(state transition and output)
(inputs and outputs)
(series of state transitions)
• The drawbacks are:
– that not all specifications “easily” fit this model
– that the determination of what depth level should the finite state
machine be for system test; very low a level is more appropriate for
unit or integration test
Test Coverage Metrics with Finite State
Machine
• Every node (state) is covered.
– This is a very minimal coverage, much like every
statement is covered (recall this from structural
path testing)
– In our 1st state transition chart - - - this is
equivalent to covering states A, B, and C with
correct card and successful PIN processing.
• Every node and edge (transition) is covered
– This is a more complete coverage if the finite state
machine is carried to deep enough level.
Strategies for Thread Testing Finite State Machine
modeled systems
• Event based (state-input based):
–
–
–
–
–
Each input event occurs
A “typical” or “common” sequence of input events occur
Each input event occurs in every “relevant” data context
For a given context, all “inappropriate” input events occur
For a given context, all possible (both appropriate &
inappropriate) inputs events occur
• Event based (state-output based):
– Each output event occurs
– Each output event occurs for every “cause”
• Port-based (both input and output):
– Cover all input and out ports
Strategies for thread Testing
Data Represented Systems
• Some software system requirements are better
represented with E-R diagram.
• For E-R modeling used for specifications, the
threads may be identified via:
– Looking at the cardinality of every relationship
•
•
•
•
One-one
One-many
Many-one
Many - many
– Looking at the participation factor of every relationship
– Looking at functional dependencies among relationships
• e.g. loaning a book that’s not available
More importantly --- we want to make sure that the DB “access” is tested,
along with threads such as a) shutdown -restart b) backup-recovery
c) multiple access threads
Using Operational Profile when interested in
“Effectiveness”
• The effectiveness of testing is often measured in
terms of problems found versus effort spent or
– (# of defects found) / person hours spent
– (# of defects found) / number of test cases
Not just the number of
problems found
• Historically, most of the failures also tend to fall in
small parts of the system (80% failures occur in 20%
of the system - - - e.g. the most heavily traversed
threads).
– Thus if we can collect the operational profiles of the users,
we can identify the most heavily used threads.
– This strategy of system testing with operational profile can
improve our test efficiency.
User-Operational Profile Example
(naïve user versus experienced user)
(20/20)
(20/20)
naïve users
(20/80)
default f1
f4
(5/5)
(5/10)
Start
(80 users)
f4
f1
(5/5)
f6
(40/40)
(40/60)
Choice of (10/60)
f1,f2,f3
f3
Note that 2 usage cases take
up 75% of operational profile !
term
(40/40)
f7
f2
prob = .063
term
(5/10)
(10/60)
experienced
users (60/80)
term prob = .25 √
(3/10)
(3/3)
f4
(7/10)
term prob = .50√√
term
(7/7)
f5
prob = .063
prob = .037
prob = .087
term
Software changes and Regression Test
• In the life span of a software, we can anticipate changes
that result in multiple cycles of fixes and release. Thus it
needs to be retested to ensure that what worked before
still works (did not get regressed).
• A common strategy is to re-run the complete system test
over as the regression test
• A more economical strategy for regression test is to
develop (1) new threads for the new areas and (2) only
include those old threads that are affected by the new
release. (Yes, there may be an element of risk)
Think about the
“neighborhood”
testing strategy-last lecture