bEgInSlIdE
Software Testing
1
bEgInSlIdE
Software Life Cycle
Sommerville , 1992: Development efforts are typically distributed
as follows:
Specifications / Design
30% - 40%
Implementation
15% - 30%
Testing
25% - 50%
2
bEgInSlIdE
Remarks by Bill Gates
17th Annual ACM Conference on Object-Oriented Programming,
Seattle, Washington, November 8, 2002
“… When you look at a big commercial software company like
Microsoft, there's actually as much testing that goes in as
development. We have as many testers as we have
developers. Testers basically test all the time, and
developers basically are involved in the testing process
about half the time…
… We've probably changed the industry we're in. We're not
in the software industry; we're in the testing industry, and
writing the software is the thing that keeps us busy doing all
that testing.”
3
bEgInSlIdE
Remarks by Bill Gates (cont’d)
“…The test cases are unbelievably expensive; in fact,
there's more lines of code in the test harness than there is
in the program itself. Often that's a ratio of about three to
one.”
“… Well, one of the interesting questions is, when you
change a program, … what portion of these test cases do you
need to run?“
4
bEgInSlIdE
The V-model of development
Requir ements
specification
System
specification
System
integration
test plan
Acceptance
test plan
Service
System
design
Acceptance
test
Detailed
design
Sub-system
integration
test plan
System
integration test
Module and
Implementation
unit code
and unit
testing
and tess
Sub-system
integration test
5
bEgInSlIdE
Some popular testing categories
Black box
/
white box
Static
/
dynamic
6
bEgInSlIdE
Testing methodologies
Specificationbased testing
Regression
testing
 Reflects true
intention of testing
 Does not need a
specification
 Does not propagate
errors from previous
versions
 Easy to implement
 Finds subtle errors
7
bEgInSlIdE
Black Box Testing
(Behavioral testing)
Input
System under test
Output
How shall we check the I/O relation ?
 Manually (specification-based)
 Table of expected results (specification-based)
 Compare results to previous version (Regression testing)
8
bEgInSlIdE
Input
Black Box Testing
Output
(Behavioral testing)
Testing Input-Output relationships only
 Pros
• This is what the product is about.
• Implementation independent.
 Cons
• For complicated products it is hard to identify erroneous output.
• It is hard to estimate whether the product is error-free.
• Practically: Choosing input with high probability of error
detection is very difficult (e.g. division of two numbers).
9
bEgInSlIdE
White Box Testing
(Operational Testing)
Testing how input becomes output (including
algorithms)
 Pros
• Easier to detect errors.
• Enables to find better tests (direct the tests)
• The only way to check coverage.
 Cons
• Implementation weaknesses are not necessarily those of the
product.
• Code is not always available
10
bEgInSlIdE
Static and Dynamic Testing
Dynamic testing (Run your program)
 Predefined tests
• Good for Regression Testing (comparing an old version against a
new one)
• Testing the product under extreme conditions
 Random tests
 “real life” tests
Static testing (Inspect your code)
 Code analyzers (e.g., tools like lint and purify)
 Inspection (code review)
 Proofs (by tools, or by mathematical arguments)
11
bEgInSlIdE
Dynamic Testing
In combination with black-box testing
In white-box testing:
 Preprocessor controlled code
• The only way for digging into the heart of the code
• Code usually outputs the status of some objects.
• Requires modification whenever the code is modified +
compilation.
Specification-based monitoring
Other methods …
12
bEgInSlIdE
Special Testing Methods
.
Stress Testing
A product that will work under heavy load (e.g, on-line
banking system) should be tested under increasing load
- much heavier than expected.
13
bEgInSlIdE
Static Testing
Code analysis
 Unreachable code
 Objects declared and never used
 Parameters type/number mismatch
 Variable used before initialization
 Variable is assigned to twice, without using the first value
 Function results not used
 Possible array bound violations
14
bEgInSlIdE
Static Testing
Code inspection
 Self - The default choice.
• Subtle errors and micro-flaws may be overlooked.
• Wrong conceptions propagate to review…
 Code review by others - Very efficient !
15
bEgInSlIdE
One more quote…
Dijkstra:
“Testing can only prove the existence of
bugs, not their absence…”
16
bEgInSlIdE
… So why not try to prove correctness?
 In general – it is undecidable, i.e. can’t be done.
 In most cases it is possible, but with manual assistance –
the same way we would prove a math theorem.
 In some cases properties of software can be proved
automatically.
 Chances for errors increase with length of text
• Write short code (e.g, divide into more functions).
• Provide short proofs for correctness (even if they are informal).
17
bEgInSlIdE
Estimate how clean is your
software
Error Implantation (For measuring the effectiveness
of testing)
 Introduce errors.
 See how many of them are detected.
 This gives us an “educated guess” about testing quality.
18
bEgInSlIdE
Estimate how much of the
software’s behavior is covered
Coverage is a mean to estimate how rigorous is the
testing effort
We can use coverage information in order to guide the
process of test generation (some times even
automatically)
19
bEgInSlIdE
Statement Coverage
Example 1
int a, b, sum;
int list1[10] = {00, 11, 22, 33, 44, 55, 66, 77, 88, 99};
int list2[10] = {99, 88, 77, 66, 55, 44, 33, 22, 11, 00};
cin >> a >> b;
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
cout << sum << "\n";
20
bEgInSlIdE
Statement Coverage
Example 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
Statement coverage may be achieved by __ test
case(s):
21
bEgInSlIdE
Statement Coverage
Example 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
But statement coverage may not cater for all
conditions

such as when a and b are beyond the array size.
22
bEgInSlIdE
Branch Coverage
Same Example 1
if (a >= 0 && a <= 9)
sum = list1[a];
if (b >= 0 && b <= 9)
sum = sum + list2[b];
Branch coverage may be achieved by __ test
cases:
23
bEgInSlIdE
branch coverage
Example
switch (x){
case 1: x = 1; break;
case 2: switch (y){
case 1: x = 3; break;
case 2: x = 2; break;
otherwise: x = 1; }
otherwise: 4;}

branch coverage may be achieved by __ test cases
24
bEgInSlIdE
Path Coverage
Same Example 2
if (a >= 0 && a <= 9)
sum =
else sum
if (b >=
sum =
else sum
list1[a];
= 0;
0 && b <= 9)
sum + list2[b];
= 0;
Path coverage may be achieved by __ test cases:
26
bEgInSlIdE
Subsumption Relationships

Path coverage
subsumes

Branch coverage
subsumes

Statement coverage
27
bEgInSlIdE
Branch Coverage
Example 3
if (a >= 0 && a <= 9)
sum = list1[a];
else sum = 0;
if (b >= 0 && b <= 9)
sum = sum + list2[b];
else sum = 0;
...
if (z >= 0 && z <= 9)
sum = sum + list26[b];
else sum = 0;
Branch coverage may be achieved by __ test
cases
28
bEgInSlIdE
Path Coverage
Same Example 3
if (a >= 0 && a <= 9)
sum = list1[a];
else sum = 0;
if (b >= 0 && b <= 9)
sum = sum + list2[b];
else sum = 0;
...
if (z >= 0 && z <= 9)
sum = sum + list26[b];
else sum = 0;
Path coverage may be achieved by __ test cases 29
bEgInSlIdE
Statement Coverage
sum = 0;
Example 4
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Statement coverage may be achieved by __ test
cases:
30
bEgInSlIdE
Branch Coverage
sum = 0;
Same Example 4
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Branch coverage may be achieved by __ test cases:
31
bEgInSlIdE
Loop coverage
Skip the loop entirely
Only 1 pass through the loop
2 passes through the loop
n–1, n and n+1 passes through the loop,
where n is the maximum number of allowable
passes
32
bEgInSlIdE
Loop Coverage
sum = 0;
Same Example 4
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Loop coverage may be achieved by __ test cases:
33
bEgInSlIdE
Path Coverage
sum = 0;
Same Example 4
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
if (b >= 0 && b <= 9)
sum = list2[b];
else sum = 0;
Path coverage may be achieved by __ test cases
34
bEgInSlIdE
Subsumption Relationships

Path coverage
subsumes

Branch coverage
subsumes

Statement coverage

subsumes

Path coverage
Loop coverage
35
bEgInSlIdE
Path Coverage
sum = 0;
Example 5
while (a >= 0 && a <= 9) {
sum = list1[a];
a = a + 1;
};
while (b >= 0 && b <= 9) {
sum = list2[a];
b = b + 1;
};
...
while (z >= 0 && z <= 9) {
sum = list26[z];
z = z + 1;
};
Path coverage may be achieved by __ test cases
36
bEgInSlIdE
Path Coverage is not Everything
Example 1
z = x + y;
Path coverage may
be achieved by 1
test case
x = 8, y = 0
 Cannot detect
z = x - y;
x = 2, y = 2
 Cannot detect
z = x * y;
x = 8, y = 9
 Cannot detect
z = 8 + y;
 Cannot detect
z = x + 9;
We need 2 test cases:
 x = 8, y = 9
 x = 29, y = 18.
37
bEgInSlIdE
Path Coverage is not Everything
Example 2
int a[10];
if (b > 0)
a[b] = 1;
else …
38
bEgInSlIdE
Condition coverage
Example
if (b1 ||(b2 && b3))
a = 1;
else …
Every sub-expression in a predicate should be evaluated
to TRUE and FALSE

Sub-expression coverage may be achieved by __ test cases
39
bEgInSlIdE
Multiple condition coverage
Example
if (b1 ||(b2 && b3))
a = 1;
else …
Every Boolean combination of sub-expressions in a predicate
should be evaluated to TRUE and FALSE

Multiple condition coverage may be achieved by __ test cases
40
bEgInSlIdE
Condition/Branch coverage (MC / DC)
if (!b1 || b2)
a = 1;
else …
Union of Condition coverage and Branch coverage
This is an industry standard

MC/DC coverage may be achieved by __ test cases
41
bEgInSlIdE
Specification-based testing
We will see an example of a system for specificationbased testing of real-time applications.
The testing system is called “Logic Assurance”
It monitors the specification, and can also intervene in
the execution of the program.
42
bEgInSlIdE
Overview
Monitoring
control
Report state
Logic
Assurance
Event
Reporting
System
Under
Development
Report State
Environment
control
Optional
=
43
bEgInSlIdE
A typical specification language: The LA Language (LAL)
Derived from Temporal Logic and C, LAL enables
specification of:
Event order.
Relative frequency ("fairness").
Timing demands.
Logical and mathematical operators.
More...
45
bEgInSlIdE
Examples (1/4):
1. Using event names, time directives and messages:
OPEN_DOOR follows CLOSE_DOOR after not more
than 10 sec ?: message("Open door is
late");
46
bEgInSlIdE
...Examples (2/4)
2. Logical operators and functions:
when time>20: time([CLOSE_DOOR])>
time([OPEN_DOOR])+10?:
message("CLOSE_DOOR is early");
47
bEgInSlIdE
...Examples (3/4)
3. User-defined functions are used to enhance the language
and enable external control:
if JOB_DONE(10) then HEAT(3,5) < 30?:
REDUCE_HEAT(5);
48
bEgInSlIdE
...Examples (4/4)
4. Locators are used to scan the log and retrieve event index:
[2nd SEND s.t. Time>=10, packet=5] > 0 ?:
REDUCE_HEAT(5);
49
bEgInSlIdE
2. Report events:
From inside the SUD by using the LA Report Facility.
From outside the SUD by using black-box techniques
(e.g. OS events)
From the environment (Sensors, etc.)
50
bEgInSlIdE
The LA Report Facility:
The following directive should be inserted where the
event occurs in the program:
LA_report_event (
int
identifier,
float
time-stamp,
user additional data...
)
51
bEgInSlIdE
LA will:
1. Keep an event log.
2. Analyze the rules in real time* (during execution)
using the LA Analyzer.
* Off-line analysis is also possible
52
bEgInSlIdE
3. When a rule is violated, do one of the
following:

Report the exact place and time the
rule was violated.

Invoke a user-defined function.
53
bEgInSlIdE
Demo
A basic communication protocol ("Stop & Wait"):
2
2
2
3
time-out
time-out
Rules:
1. Resend packet after time-out…
2. Do not resend packet that was acknowledged…
…
54
bEgInSlIdE
Incoming events:
# event time
messages:
message index arguments
-- ------ ------- ---------- ------- -----------1: send 10
A
1
7, 10, 21..
2: Tout 15
A
1
8, 9 ,10..
3: send 16
B
2
20,30,21..
:
Rule 1, event 3: failed to resend packet
user screen:
‘A’ sent at 10, Ack expected at 13..
Tout for A activated at 15, must send A again..
‘B’ sent at 16, Ack expected at 19..
.
55
bEgInSlIdE
Advantages of real time monitoring:
Tests can be planned, maintained, expanded
and applied throughout the development
process.
Problems can be detected sooner.
The product is ‘tied’ to the specification.
56
bEgInSlIdE
Advantages of real time monitoring:
A generic tool and methodology.
By receiving input from different sources, it
enables testing of:
 Multiprocessor systems
 Dynamic environment systems.
57
bEgInSlIdE
Advantages of real time monitoring:
Enables temporal tests.
Enables smarter tests by using
information from inside the program.
58