Testing and Inspecting to Ensure High
Quality
Defects in Ordinary Algorithms
Defects in Handling Stress or Unusual
Situations
Defects in Ordinary Algorithms
Incorrect logical conditions
• Defect:
—The logical conditions that govern looping and ifthen-else statements are wrongly formulated.
• Testing strategy:
—Use equivalence class and boundary testing.
—Consider as an input each variable used in a rule or
logical condition.
—Break into smaller, individual logical tests.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
2
Example of incorrect logical conditions defect
What is the hard-to-find defect in the following code?
• The landing gear must be deployed whenever the plane is within two
minutes from landing or takeoff, or within 2000 feet from the ground.
If visibility is less than 1000 feet, then the landing gear must be
deployed whenever the plane is within three minutes from landing or
lower than 2500 feet
If (!landingGearDeployed &&
(min(now-takeoffTime , estLandTime-now))<
(visibility < 1000 ? 180 :120) ||
relativeAltitude <
(visibility < 1000 ? 2500 :2000)
)
{
throw
new LandingGearException();
}
Class Exercise! What is the order of evaluation – for starters!!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
3
Defects in Ordinary Algorithms
Performing a calculation in the wrong part of a control construct
• Defect:
—The program performs an action when it should not, or does not
perform an action when it should.
—Typically caused by inappropriately excluding or including the
action from a loop or a if construct.
• Testing strategies:
—Design tests that execute each loop zero times, exactly once,
and more than once.
- Ensure a Do…While (or While…) is false to begin with
- Ensure a Perform … Until is true to begin with
- Remember: Pre-tests and Post Tests
—Anything that could happen while looping is made to occur on
the first, an intermediate, and the last iteration.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
4
Example of Performing a Calculation in the Wrong
Part of a Control Construct
What is wrong with the following loop?:
while( j < maximum)
{
k=someOperation(j);
j++;
}
If (k == -1) signalAnError();
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
5
What is wrong with the following loop:
while( j < maximum)
{
k=someOperation(j);
j++;
}
If(k == -1) signalAnError();
Must assume j has an acceptable integral value
prior to entering loop.
Could say:
while (j++ <
would affect argument (j)!
© Lethbridge/Laganière 2001
maximum) but this
Chapter 10: Testing and Inspecting for High Quality
6
Defects in Ordinary Algorithms 1
Not terminating a loop or recursion
• Defect:
—A loop or a recursion does not always terminate, i.e. it is ‘infinite’.
—Remember in recursion, you have the ‘recursive case’ and the
‘base case’
—What do we mean by the ‘recursive case?’ Properties??
—What do we mean by the ‘base case?’
• Testing strategies:
—Analyze what causes a repetitive action to be stopped.
—Run test cases that you anticipate might not be handled correctly.
Count on these happening during operations!
Does recursive case converge to base case?
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
7
Defects in Ordinary Algorithms 3
Not setting up the correct preconditions for an algorithm
• Defect:
—Preconditions state what must be true before the algorithm is executed.
—A defect would exist if a program proceeds to do its work, even when
the preconditions are not satisfied. Do we really want to ensure a < 14 is
true prior to going into the algorithm?
while (a<14)
{ ….
……} such that even if a >= 14, this loop (and perhaps other
contents) continues to run appropriately.
• Testing strategy:
—Run test cases in which each precondition is not satisfied.
—See what happens beyond the while-loop.
Absolutely! Insert probes (Displays, printf, system.out.println…)
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
9
Defects in Ordinary Algorithms 4
Not handling null conditions
• Defect:
—A null condition is a situation where there normally are one or more data
items to process, but sometimes there are none.
—It is a defect when a program behaves abnormally when a null condition is
encountered.
—How do you test null (or non-numeric) condition in, say, COBOL, for a
numeric field? How can it happen?
• Testing strategy:
—Brainstorm: determine unusual conditions and run appropriate tests.
—Always test ‘no inputs’ interactively or via an OPENed file.
—Always test to see if the Input or Read or … retrieved data was non-null!
—Always test with only one input;
- Oftentimes code runs perfect with a single input;
—Always test with two or more.
- If code runs correctly with two inputs, it’ll probably run with n inputs (if we
are dependent upon checking null or EOF conditions, etc.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
10
Defects in Ordinary Algorithms 5
Not handling singleton or non-singleton conditions
• Defect:
—A singleton condition occurs when there is normally more than one of
something, but sometimes there is only one. (this is a little different
than the last case)
—A non-singleton condition is the inverse.
—Defects occur when the unusual case is not properly handled.
• Testing strategy:
—Brainstorm: determine unusual conditions and run appropriate tests.
—Can occur with incorrect number of parameters (in some
languages). - Can abend or continue, providing null values to
unmatched parameters…
—Think argc and argv. Can easily test.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
11
Defects in Ordinary Algorithms 6
Off-by-one errors
• Defect:
—A program inappropriately adds or subtracts one.
—Or loops one too many times or one too few times.
—This is a particularly common type of defect.
• Thoughts:
—Results: often EOF reached; executing same instructions twice;
—Very easy in C and Java: side effects of a++ and ++a; careful in using a++ or
a++ within predicate (condition) of loop as in while (a++ < b) is quite different
than while (++a < b) !
—Test: if a = 1 and b = 2, which of the two preceding, if any, were true?
COBOL: Perform …. Until X >= 25 or ….Until X=25!
Java, C++ for (int i = 1; i < 25; i++) ….. ++i or i++ no difference here.
Always able to give correct value if ‘i’ within loop”!
What is value of loop control variable when we terminate the loop?
• Testing strategy:
—Develop tests in which you verify that the program:
-
computes the correct numerical answer.
performs the correct number of iterations.
Always display value of loop control variables upon exiting loop!!!
Insert probes for counting!!!!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
12
Example of off-by-one defect 7
for (i=1; i<arrayname.length; i++)
{
/* do something */
}
Difference between length and size????
What if arrayname.length = 1?
Use Iterators to help eliminate these defects
while (iterator.hasNext())
{
anOperation(++val);
}
Nice boolean checks are almost always benign!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
13
Defects in Ordinary Algorithms 8
Operator precedence errors
• Defect:
—An operator precedence error occurs when a programmer omits
needed parentheses, or puts parentheses in the wrong place.
—Operator precedence errors are often extremely obvious...
- but can occasionally lie hidden until special conditions arise.
—E.g. If x*y+z should be x*(y+z) this would be hidden if z =0
and perhaps is normally zero.
• Testing:
—In software that computes formulae, run tests that anticipate such
defects.
—Know the operator precedence rules!
—Don’t parenthesize to death! It shows you don’t know what you
are doing!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
14
Defects in Ordinary Algorithms 9
Use of inappropriate standard algorithms
• Defect:
—An inappropriate standard algorithm is one that is unnecessarily
inefficient or has some other property that is widely recognized
as being bad.
—Can you think of any?
- How about Order n**2 sorts?
- How about QuickSort?
- How about Address Calculation sorts (various kinds)?
• Testing strategies:
—The tester has to know the properties of algorithms and then
design tests that will determine whether any undesirable
algorithms have been implemented.
—There is no ‘one size fits all.’
—Knowing some of the data characteristics is a big help!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
15
Defects in Handling Stress and Unusual
Situations
Incompatibility with specific configurations of hardware or software
• Defect:
—The system fails if it is run using particular configurations of
hardware, operating systems, and external libraries.
— (How about browsers too?)
• Testing strategy:
—Extensively execute the system with all possible configurations that
might be encountered by users.
—For standard software, must anticipate (or restrict) platforms!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
16
Defects in Handling Stress and Unusual Situations
2
Defects in handling peak loads or missing resources
• Defects:
—The system does not gracefully handle resource shortage.
—Resources that might be in short supply include:
- memory, disk space or network bandwidth, permissions, processors,
other peripherals, ….
- Honeywell: PAT Table Overflow (< 32 files)
—The program being tested should report problems in a way the user
will understand.
• Testing strategies:
—Devise a method of denying resources (test missing resources, such
as disk drives not available, etc.).
—Run a very large number of copies of the program being tested, all at
the same time.
—May require some redesign or rethinking of the process!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
17
Defects in Handling Stress and Unusual Situations
3
Inappropriate management of resources
• Defect:
—A program uses certain resources but does not make them
unavailable when no longer needed.
—Memory leakage? C – free? Java? Always good? No.
• Testing strategy:
—Run the program intensively in such a way that it uses
many resources, relinquishes them and then uses them
again repeatedly.
— Can require some sophisticated testing!
— Can require heavy loading!
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
18
Defects in Handling Stress and Unusual Situations
4
Defects in the process of recovering from a crash
• Defects:
—Any system will undergo a sudden failure if its hardware fails, or
if its power is turned off.
—It is a defect if the system is left in an unstable state and hence
is unable to fully recover.
—It is also a defect if a system does not correctly deal with the
crashes of related systems.
• Testing strategies:
—Kill a program at various times during execution.
—Try turning the power off; however, operating systems
themselves are often intolerant of doing that.
—Abruptly pre-empt key processes.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
19
Disaster Recovery
Earthquakes, Hurricanes, Tornadoes, Floods.
What do we do???
Discussion of Disaster Recovery approaches
Data – the real valuable corporate asset
Backup computing platforms
Renting / leasing resources, CPU cycles,
disk space, etc.
Procedures for
evacuation
fire, emergency generators, travel.
etc.
© Lethbridge/Laganière 2001
Chapter 10: Testing and Inspecting for High Quality
20