Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

1 Testing


Lewis “Software Testing and Continuous Quality Improvement”
Kaner, Falk and Nguyen, “Testing Computer Software”
1.1 White/Black/Gray Box Testing
This is one dimension to classify testing methods.
 White or Clear or Crystal Box Testing – Testing that is based on full knowledge of the code,
architecture, or implementation.
 Black Box Testing – Testing that assumes NO knowledge of code, architecture or
implementation.
 Gray Box Testing – Testing with only partial knowledge of implementation. This knowledge
may include possible algorithms that were used to implement but not the specific code, etc.

Generally, on all but trivial systems, the development of test cases can be a infinite process.
How much time we spend developing test cases is based on an economic decision. We try
to test:
o the most important functionality,
o reduce the risk of shipping defects, and
o reduce the impact of dangerous defects.
2 Levels at which we can test
2.1 Unit Testing







Unit tests are often time part of detailed design documents.
The tests often are reviewed by a Testing Organization,
Running the tests are often the responsibility of the development team,
Sometime test harnesses or simulators are necessary perform unit testing,
Test Stubs – minimal function implementations to check the logic of calling functions and
interfaces between calling and called functions.
Test Harness – code is written to simulate parts of a system that are not yet functional or to
stress code harder than can be could normally be done by hand.
When proof of passing a test is available, the unit software can be released for integrating
into Subsystems.
©2011 Mike Rowe

Page 1
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


Since some knowledge of code may be necessary, this is generally White Box testing.
Sometimes, low-level requirements (specifications) are detailed enough that these tests can
be run without knowledge of the code – this would then be Black Box testing.
2.2 Subsystem Testing




Rather than dealing with individual classes we are dealing with groups of classes. It is
smaller than the entire system.
At a subsystem level we are often testing Abstract Data Types (ADT) and classes. ADTs are
non-classes, but like classes encapsulate and hide the details of the implementation; thus,
limiting the level of detail with which testing that can be done. Often must treat a
subsystem as a Black Box. Encapsulating code limits what program elements we have
access to for testing.
We may require Test Harnesses to drive an entire subsystem or simulate other subsystems
on which the tested subsystem is dependent.
Black Box testing if the test harnesses can be developed without knowledge of the code, if
we need knowledge of the code to develop these we are dealing with white box testing.
2.3 Integration Testing




The incremental testing of a system as subsystems are sequentially added. It is more than
subsystem testing and less than System Level Testing.
We generally do have implementation knowledge of the subsystems. Integration testing
determines how well subsystems play together.
Tests the interfaces among subsystems.
Need knowledge of the architecture to drive subsystems and sets of subsystems, thus White
Box testing
2.4 System Level Testing





System Level testing involves testing only after the system is completely integrated. Some
tests require all/most parts of a system to be in place before they can be implemented.
System tests are generally done at the user’s level. Due to the OO and/or ADT construction
of subsystems, system level testing is hard to do anything other than functional testing.
Tests should cover all User Cases and/or requirements.
Tests should cover non-specified events. If a user might possible hit a keyboard with a fly
swatter we should see what happens, (This is type of testing is technically known as “Fly
swatter testing methods” – just kidding). The system should handle unexpected events.
See Exception Test Cases below.
Often defects found at the system level are the hard to isolate. This is because we dealing
with a large body of code rather than a smaller body.
©2011 Mike Rowe

Page 2
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


Black Box testing.
2.5 Normal v. Exception Test Cases



Defining and testing the “sunny-day” of a system is often very easy and can sometimes be
done exhaustively.
Testing all possible exceptions can be a more formidable task and is often a combinatorially
impossible to test all possible sets of “rainy-day” inputs. We may not be able to test all
combinations “before the sun burns out”.
Generally Black Box testing.
2.6 Regression Testing






Rerunning suites of generally automated tests.
Primarily used on released systems that are undergoing maintenance. Regression testing
helps ensure that working software has not been significantly broken.
“The average bug needs to be fixed twice in the lifespan of a system”, Robin (USDATA
1996).
Regression testing is often automated, some places there is an automatic nightly build,
install, regression test, and defect reports sent ready for the software teams each morning.
This strategy makes fixing defects easy, as, if it worked one day but didn’t the next you can
easily isolate what has changed.
Regression test suites are often augmented as new ways to break a system are discovered.
In verifying that you have fixed a defect, you generate a test. When done, add this test to
your regression suite.
Generally Black Box testing.
2.7 Inspections and Walkthroughs





Inspections, reviews and walkthroughs can start at the very beginning of a project with
review of requirements and use cases and continue through developed code and test cases.
They can range from informal individual “desk reviews” to formal “peer reviews” where
people meet around a table and go through an artifact line by line. Architects and
Designers review designs, domain experts review requirements and use cases,
programmers review code.
Although the process varies, it is are generally effective at spotting defects and weaknesses
in documents, designs and code before testing is possible.
Common problem involves keeping the review process constructive and not offending the
egos of the author(s).
Side benefit is that reviewers can learn new tricks from seeing other peoples’ tricks.
©2011 Mike Rowe

Page 3
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


White Box testing.
3 Input Coverage Testing


Often it is combinatorially impossible to test all possible input values for even one variable.
AMD’s AM3 socket has over 941 pins. If only 200 of these pins can be used for input we
have 200 binary input variables, or 2 to the 200th input combinations (1.6E+60). If we can
do a test every nanosecond it will take only 5E+43 years to do all tests.
o

The age of the universe, in physics, is the time elapsed between the Big Bang and the
present day. Current observations suggest that this is about 13.7 billion years (1.37E+10),
with an uncertainty of about +/-200 million years.
Systematic testing methods are different ways of breaking down impossible large domains
into manageable chunks.
3.1 Equivalence Classes based input testing






Often input values can be segregated into subsets, based on assumed expected outcomes.
o This helps cut down on the total number of tests needed to “cover” this input.
o Weakness is that we are making an assumption on equivalence of values.
We can look at each input variable and group its inputs into sets called equivalence classes,
and make the assumption that all members of a class will behave identically in a system.
o For instance if we are reading voltages (an infinite domain) and the nominal range is
between +4.5 to + 6.0 volts, inclusive.
Nominal subclasses
o Any value between +4.5v to + 6.00v should function identically, thus we can test
 Exactly 4.5 v
 Exactly 6.0 v
Exception subclasses (these should fail), test
o Exactly +4.49v
o Exactly +6.01v
Other cases of interest for which we might have special processing include:
o +0.01 to +4.49v (low voltage processing)
o +6.01 to +20.0 volts (high voltage processing)
o +20.01 to +130v (very high voltage processing)
o >+200v -- (really really high voltage processing)
o 0.00 v -- (total loss in continuity processing)
o -0.01 to –20.00 v – (reverse polarity processing)
We can test one or two values in each of the subclasses and assume that all other values of
that subclass will behave the same. This cuts down on the number of values that we need
to test for one input variable, without HOPEFULLY sacrificing coverage.
©2011 Mike Rowe

Page 4
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


We may have 10, 100 or 1000’s of such input variables. Even if we produce equivalence
subclasses for each of input variables, we may not be able to test all combinations of all
input variables subclasses.
3.1.1 Boundary Testing


This is a means of limiting the number of test cases by selecting only the edges of each
equivalent class. We demonstrated this above.
o We select two values as close to an edge as we can get, one above and one below
the edge.
In the above example if one of the edges is at 4.5v (the lowest value accepted), we want to
test 4.500v and 4.499v. 4.500 v should pass and 4.499v should fail. We don’t need to test
other values above or below these values.
3.1.2 All-Pairs methods



This is a strategy for maximizing coverage of input combinations while minimizing the
number of tests performed.
Let us assume each input variable has 5 equivalence classes of inputs.
o Input variable A contains the set of inputs {a1, a2, a3, a4 and a5}
o Input variable B contains the set of inputs {b1, b2, b3, b4 and b6}
o And so on for as many input variable as we have.
If we have 2 input variables A and B, and each input has 5 subclasses. For this we will need
25 test cases to cover all combinations.
o Compute total combinations of v variables with i inputs each is i to the v power
o iv
Variable A
Variable B
b1
b2
b3
b4
b5

a1
a1
a1
a1
a1
a1
a2
x
x
x
x
x
b1
b2
b3
b4
b5
a2
a2
a2
a2
a2
a3
x
x
x
x
x
b1
b2
b3
b4
b5
a3
a3
a3
a3
a3
a4
x
x
x
x
x
b1
b2
b3
b4
b5
a4
a4
a4
a4
a4
a5
x
x
x
x
x
b1
b2
b3
b4
b5
a5
a5
a5
a5
a5
x
x
x
x
x
b1
b2
b3
b4
b5
If we have 3 input variables (A, B, and C) we have a total of (5 to the 3rd ) 125 combinations.
o But, if we just test all combinations of pairs: A – B (25 combinations), A – C (25
combinations), and B – C (25 combinations) we only have 75 combinations.
o For the 25 Combinations of A-B we randomly select values of C. Likewise, for A-C we
select random values for B, and for B-C we select random values of A.
o This strategy provides a savings of 50 test cases.
©2011 Mike Rowe

Page 5
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

o It is missing the complete coverage of three-way interactions of A-B-C and we
assume that by randomizing on non-paired variable we minimize the impact of
missing the three-way interactions.

If we have 5 input variables (A, B, C, D and E) we have (5 to the 5th) or 3125 combination,
o But if we test all pairs we only have 250 cases to test (25 for each of A—B, A—C, A—
D, A—E, B—C, B—D, B—E, C—D, C—E, and D—E).

If we have 10 input variables we have a total of (5 to the 10th) 9,765,625 combinations.
o But, we have p! / (q! * (p – q)!) This means p items taken q at a time.
o With 10 items taken 2 at a time we have 10! / (2! * 8!) = 45 pairs, each pair has a
25 combinations, or 45 * 25 tests = 1,125 total tests.
o This is a savings of 9,764,200 test cases.
If we have 20 input variables (5 to the 20th) or 95,367,431,640,625 combinations
o 20! / (2! * 18!) = 190 pairs, each pair has 25 combinations, 190 * 25 = 4,750 tests.


The larger the number of input variables, the larger the savings in using All Pairs Testing.
Number
of Input
Variables
2
3
5
10
20
Equivalence
Classes per
input
5
5
5
5
5
Total Combinations
All Pairs
Combinations
% of tests savings
25
125
3125
9,765,625
95,367,431,640,625
25
3 * 25 = 75
10 * 25 = 250
45 * 25 = 1,125
190 * 25 = 4,750
0%
40%
92%
98.848%
99.999999%
3.1.3 Computing probability of catching a failure


Let’s say we have 10 variables A, B, C, D, E, F, G, H, I and J each with 5 inputs.
o We have 5 to the 10th combinations , iv , or 9,765,625 total test combination.
Let say there is one defect in the system and it occurs when we have both i5 and j5. What is
the probability of detecting this defect if we only have time to run one random combination
of inputs?
o Total combinations of inputs: 5 to the 10th = 9,765,625 total test combination.
o Each combination of A, B, C, D, E, F, G, and H with the combination of i5 and j5
will fail:
 There are 5 to the 8th combinations of A, B, C, D, E, F, G and H or 390,625
o The probability of randomly drawing the defective test case is 390,625 / 9,765,625 =
0.0399693824
©2011 Mike Rowe

Page 6
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality



What would be the probability of detecting a defect if we didn’t hit a defect the first time and
could randomly select a second test case to run.
o The first would be 0.0399693824 (see above).
o The second would be 390,625 / 9,765,624 = 0.040000004….
This is as far as we will go with this topic
3.2 Broken Box Testing

This is a form of “gray-box” testing. With knowledge of algorithms, but not necessarily
knowledge of the exact implementation, we select input values that will exercise different
conditions of the algorithms that were possibly used in the code.

Some algorithms have known weaknesses, for instance polynomial methods of estimating
probabilities “wiggles” at the tails. We use data values at the tails to see if algorithms
remain stable.
o Example: some easily implemented algorithms for estimating probability given zscores use polynomials. These may give very inaccurate values at the tails.
3.3 Alpha and Beta testing or some call it “Let the customer test it “

Alpha testing generally occurs at a point when all of the major features, but perhaps not all
of the minor features are working. At least one system-level testing pass has been made.
There may be known critical defects in the system. The code is sent to a very limited
number of “friendly” users who test it under real-world conditions, but not maybe truly
production-level environments.

Beta testing generally occurs at a point when all major and minor features have been
implemented and few if any major defects (but possibly some minor defects) exist.

Major benefit of Alpha and Beta testing is that your system will be tested by many more
people than you could test in house and in many different real-world ways that you cannot
simulate in your test lab.

It is important that Alpha and Beta testing users are carefully. We need to be certain that
they will
o actually install and use the software
o represent a diverse set of users

Apple and Microsoft currently have two very different philosophies on Beta testing. Apple
does very little and very private Beta testing. Microsoft has many, huge and very public Beta
tests.
©2011 Mike Rowe

Page 7
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

o Apple prefers the marketing hype associated with releasing a product that has been
kept secret. In some products this has resulted in major flaws not being discovered
till well after the product was released.
o Microsoft prefers getting information out and also having defects discovered before a
production release occurs. Sometimes a buggy or sluggish product gets bad press
before it is released. But at least MS has a chance to respond to this press before the
final release.
3.4 Structural Testing and Structural Coverage Analysis

From A Practical Tutorial on Modified Condition/ Decision Coverage, By Kelly J. Hayhurst,
Dan S. Veerhusen (from Cedar Rapids), John J. Chilenski, Leanna K. Rierson; NASA/TM-2001210876

Structural Coverage Analysis has the purpose of determining which code structures were or
were not exercised by requirements-based testing (RBT) procedures.

Structural Testing is the process of exercising software with test cases written, which are
based on knowledge of the source code (and NOT the requirements).

Both methods require absolute knowledge of the implemented code, thus they are Clear
Box testing techniques.

We do Structural Testing to identify errors in implementation. For instance if we have the
statement:
o If (z == 2) OR (y > 1) Then . . .
What tests can be run to determine if we screwed up and rather than using and OR we used
an AND.


Generally, your code needs to be instrumented to show coverage
3.4.1 Function Level Coverage

Testing to make sure that each function is executed. Easy to “do it yourself”
instrumentation.
// put at function entry point(s).
cout << “Entering function foo” << endl;
// put at function exit point(s)
cout << “Exiting function foo” << endl;

Can use macros ( #define ... ) to implement these.
©2011 Mike Rowe

Page 8
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

3.4.2 Statement Coverage


Determines which statements are executed by test cases.
If we have the following code:
If (x > 1) AND (y == 0) Then
z=z/x
If (z == 2) OR (y > 1) Then
z=z+1
o If we select the input values x = 2 and y = 0 and z = 4 then every code statement is
covered.
o If we select x = 0 and y = 0 then inner parts of each of the Ifs are not covered.

Statement Coverage requires more sophisticated tools to instrument and analyze. Rational
PureCoverage is one such tool.
3.4.3 Decision Coverage





Used to exercise conditionals (IF, Switches, etc.)
Determines whether all decision options/paths in the code have been exercised. For
instance an IF statement makes two decisions (True or False). To have Decision coverage
we must execute both the True and False Decisions branches.
If we have the following code:
If (z == 2) OR (y > 1) Then
z=z+1
We will need two sets of inputs to have to cover the two decisions
o Case 1: (z == 2) OR (y > 1) True when z = 2 and y = {anything} results in the True
Decision
o Case 2: (z == 2) OR (y > 1) False when z = 3 and y = 0 results in the False Decision
The above results in full Decision coverage.
3.4.4 Condition Coverage




Used to exercise conditionals (IF, Switches, etc.)
Condition coverage tests to see if each condition of the inputs is covered but not
necessarily each decision is covered.
If we have the following code:
If (z == 2) OR (y > 1) Then
z=z+1
We have two Boolean Conditions,
o Condition A := (z == 2)
o Condition B := (y > 1)
o We will need two sets of inputs to cover the set Conditions of A and B, namely
©2011 Mike Rowe

Page 9
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


 A is both (True and False) and
 B is both (True and False),
o This is satisfied by A,B of {T,F} and A,B of {F,T}. For instance,
 Case 1: A := True when z = 2 and B := False when y = 0
 Case 2: A := False when z = 3 and B := True when y = 42
o The above results in full Condition coverage.
We do not need to cover all combinations of conditions of A,B of {F,F}, {F,T}, {T,F} and {T,T}.
We just need to cover A being both T and F and B being both T and F.
3.4.5 Condition/Decision Coverage






Condition/Decision Coverage combines the requirements of both Condition and Decision
coverages, in that we must test
If we have the following code:
If (z == 2) OR (y > 1) Then
z=z+1
both Decisions of
o IF  True and
o IF  False; and
each Condition we must test for both sets of conditions
o A  {True and False} and
o B  {True and False}.
The Decision can be tested with the input of z and y such that AB is TT and AB is FF, for
instance
o Case 1: (z = 2; y = 2)  Condition (AB = TT) and Decision IFT
o Case 2: (z = 0; y = 0)  Condition (AB = FF) and Decision IF F
We do not need to cover all combinations of conditions of A,B of {F,F}, {F,T}, {T,F} and {T,T}.
We just need to cover the conditions of A and B, A being both T and F and B being both T
and F, and the decision of IF -> T and F.
3.4.6 Modified Condition / Multiple Decision Coverage (MC/DC)
MC/DC enhances the Condition/Decision Coverage testing by requiring each Condition (A,
B) to independently influence the Decision (True or False). There are eight conditions that
must be meet to ensure MC/DC
1
2
3
4
For ORed condition
A v B
A v B == True
// true decisions
A v B == False // false decisions
A == True
// condition
A == False
// condition
©2011 Mike Rowe

Page 10
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

5
6
7
8
1
2
3
4
5
6
7
8
B == True
// condition
B == False
// condition
Show that each condition independently affect the decision
A v False == A v B
B v False == A v B
For ANDed Condition
A ^ B
A ^ B == True // true decisions
A ^ B == False // false decisions
A == True
// condition
A == False
// condition
B == True
// condition
B == False
// condition
Show that each condition independently affect the decision
A ^ True == A ^ B
B ^ True == A ^ B


This can be shown to require n + 1 test cases for a Decision of n-inputs.
If we have the following code:
If (z == 2) OR (y > 1) Then
z=z+1

IF DECISION TRUE
o If A = T and B = F  Then IF Decision  True // See # 3, 6, 1 and 7 above
o If A = F and B = T  Then IF Decision  True // See # 4, 5, 1 and 8 above
o The above two sets of conditions verify that conditions A and B independently
influence the IF Decision when the Decision is True
 In more detail, The True IF Decision is not dependent only on Condition A
being True and Condition B being False; and the True IF Decision is not
dependent only on Condition B being True and Condition A is False.
IF DECISION FALSE
o We also must exercise the False Decisions of the IF;
o If A = F and B = F  Then IF Decision  False // see # 2 above
o Note we do not need to test Condition A = True and Condition B = True as we have
already established that if either Condition A is True or Condition B is True we
produce a True IF Decision.
Here we have three test cases, two that are needed to exercise conditions for the True
Decision and one case (the only case possible) to exercise the False Decision.



This level of coverage is required for the Avionics DO 178-B Standard for Level A software
testing.
©2011 Mike Rowe

Page 11
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

o Level A is for the most critical systems – systems that if they were to fail would
result in a high probability of an airplane crashing.
3.4.7 Multiple Condition Coverage

This requires testing each possible combination of inputs of Conditions.
o If there are n binary conditions then this requires 2n input combinations, for AB we
need conditions of TT, TF, FT, and FF.
o If there are three condition we need input combinations: FFF, FFT, FTF, FTT, TFF, TFT,
TTF, and TTT.
3.5 CRUD Testing

Object: Create, Read, Update and Delete

Matrix indicating how processes or modules use specific objects
Object
A
B
C
D
Process
1
CRUD
R
U
C
2
RU
R
R
3
RU
R
4
D
CU
C
RU
5
D
6
D
D

Matrix to track the testing of objects usage.
Create
Read
Object
A
1
B
C
D
Update
Delete
3.6 Oracles – Truth tellers


Sometimes expected results are hard to define and you need to have an expert or some
other system be an oracle for expected results.
Oracles could be a previous version of a system, for instance if we are doing version 2.0 of a
word processor, the fonts should look just like version 1.0 unless otherwise specified. Or, it
could be some industry leading Word Processor – if it works like MS Word it must be
correct!
©2011 Mike Rowe

Page 12
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


Sometimes there are domain experts that need to look at something, for instance an
airplane simulation must be “flown” by some pilots that are experienced with the aircraft
that it is simulating to determine if the necessary degree of realism is there.
3.7 Random Testing
Profile the inputs that users actually make. Randomly generate input based on this profile.
Oracles are difficult to identify in this type of testing.
Benefits are that the inputs are not biased by the tester or developer.
4 More testing
4.1 Usability




Gather people similar to your target end users
Study how they use the product.
Perhaps conduct focus groups to discuss the usability of a product.
Some usability data can be gathered from Alpha and Beta testing, but it is generally never as
good as that data gotten from actually watching a user.
4.2 Mainstream Usage Tests


Use the system the way that you would expect your users will – do some real work with it.
See what happens.
If you don’t know how your users will be using a product, collect some scenarios. Don’t get
too detailed in the scenarios as this will provide some “wandering your way through” the
system as you use it.
4.3 Load: volume, soak, stress, and storage tests
4.3.1 Volume tests:


Study the largest tasks the program can deal with. Feed a compiler a huge program, a word
processor a huge file to manipulate, long sustained rates of I/O, etc.
Also study the smallest tasks, look at what happens when empty files are sent to process.
Look for overflows of storage, memory leaks.
©2011 Mike Rowe

Page 13
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

4.3.2 Soak Test:

Run the system at normal load for a long time.

Monitor system resources, is the system leaking memory, disk, queue length, overheating,
data bandwidth, etc.
4.3.3 Stress tests:


Test a programs response to burst rate activity.
o Start with a small load.
o Ramp it up until the program fails.
o How fast can a word processor accept input before it loses data, how many calls per
second can a telephone switch process before it starts dropping service (the Mother
Day scenario),
Run the system till the wings fall off. Hit a system with increasing levels of input until it fails.
Monitor resources to see what the failure points are (CPU, disk throughput, memory,
communications, etc.)

It is important to understand maximum capacity of a real-world system before it is
deployed. Go to Dick’s on a Friday late afternoon or Saturday morning when they are really
busy. You notice that the P.O.S. system does not to a good job of keeping up with the load.

Storage tests – This can be done as part of Stress testing.
o How does a program function with varying amounts of main memory and disk?
o Will a program function well enough on a minimally configured machine. Monitor
swap or paging space.
4.4 Background Tests

Programs that run in on multitasking environments need to be tested while other typical
programs are active. Need to make sure program is not starved for memory, cpu, disk or
communications.
4.5 Error Recover tests


Test to make sure the system handles and recovers from common failures.
Make sure the system is tested for each type of error listed in the user manual.
©2011 Mike Rowe

Page 14
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality

4.6 Security Tests




This will be based on the type of system and environment in which it runs.
How easy can an unauthorized user gain access to the system?
If some unauthorized person gains access and/or control, how much damage can be done?
See examples about hacking Pacemakers and Spoofing GPS receivers.
TheLadders.com [jobs@TechnologyLadder.com]
4.7 Compatibility and Conversion Tests



Compatibility – checks the way one product works with another.
They may both use the same file,
May communicate with each other,

Forward compatibility – old products can handle newer files/program versions; this is
generally NOT expected.

Backward compatibility – newer products can work with older files/program versions; this
is very desirable.

Multiplatform – programs that run on multiple OS, DBs, hardware, etc. should be
compatible to the degree advertised.
There are many kinds of compatibility that may be claimed, and these should be stated
explicitly in requirements.

©2011 Mike Rowe

Page 15
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality


Conversion – if a program is not directly compatible is there an intermediate program
available to convert data into a form that can work. This happens when database scheme
or file formats change between versions.
4.8 Configuration Tests

A program must run on a wide range of computers. Even on the same brand of computer
there may be different configurations of memory, printers, bios, dlls, or other peripherals.
A program should work well on common configurations of computers.
4.9 Installability and Serviceability






Installation is generally the user’s first exposure to a software product. If it is not a good
experience it may taint the user’s opinion forever.
Installation – does the install program successfully help configure the program to match the
h/w and s/w environment.
Is the install easy (Grandma test). How long does it take an inexperienced user to install the
program.
Can an experienced user customize an install to optimize it for special needs. How long
does it take to customize.
What problems can a user get into.
Check for necessary, h/w, s/w, configurations before the install gets going. It is better to fail
early than to fail late in a process.
4.10 Port Testing







After a system has been ported to another platform there are several areas to test (this is a
non-exhaustive list).
Overall functionality – use a regression test suite
Keyboard handling – press every key sequence to make sure that they behave the same.
Terminal/monitor handling – does the program look the same on various monitors
available for the different system. What about different resolutions? What can we expect
for minimum or maximum resolutions. Example: X-windows v. Win
Sign-on screens, version and system identification – has the program id version number,
etc been changed in ALL places to represent the port.
Disks – does it handle different file systems that are available on the ported system. Check
all media, hard disk, floppies, ZIPs, memory sticks, shared file servers, etc.
Operating system error handling – what happens in normal operating system program
situations, for example disk full. Blues screens are not created equally.
©2011 Mike Rowe

Page 16
02/10/2016 8:03 AM
Notes_006_Types of Testing
SE 3730 / CS 5730 – Software Quality







Installation procedure – generally the install for a different environment/OS will be radically
different. Need to make sure that it configures all needed resources for the new
environment.
Compatibility – if your program was compatible with program_X on its originally
environment and program_X has been ported, is your ported program still compatible with
program_X. May need to document around this type of a situation.
Interface style – platforms have widely different interface environments (Mac, Windows
NT, XP, …, HP-UX, Solaris, Linux, etc.). The program should look like it was designed for that
environment and not just an after thought.
Security – are there known weaknesses that should be checked.
License Management / anti-piracy – Protection of code from piracy is system dependent.
Ask the programmers what they needed to change.
©2011 Mike Rowe

Page 17
02/10/2016 8:03 AM