Software Engineering,
CPSC-4360-01, CPSC-5360-01,
Lecture 11
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
1
Review of Last Lecture


Implementing a Statechart
Review of Java programming concepts




Inheritance
Polymorphism
Type Casting
Interface
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
2
Overview of This Lecture

Software Testing


Overview
Test Phases




Unit Testing
Integration Testing
OO Specific Testing
Automated Test Driver


4/13/2015
Overview
Example
CPSC-4360-01, CPSC-5360-01, Lecture 11
3
Where are we now?
Requirement
Analysis
Design

Evaluating the System
Implement
Test
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
4
Testing – Famous Quotes

“Testing is the process of comparing the invisible to
the ambiguous, so as to avoid the unthinkable
happening to the anonymous.”, James Bach

"Testing is organised skepticism.“, James Bach

"Program testing can be used to show the presence
of bugs, but never to show their absence!“, Edgar
Dijkstra

"Beware of bugs in the above code; I have only
proved it correct, not tried it.“, Donald Knuth
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
5
Testing: Definitions
1.
2.
3.
“Testing is the process of establishing
confidence that a program or system does
what it is supposed to.”
“Testing is the process of executing a
program or system with the intent of finding
errors.”
“Testing is any activity aimed at evaluating
an attribute or capability of a program or
system and determining that it meets its
required results.”
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
6
Testing: Overview
1.
2.
3.
A software test process is based on welldefined software quality control and testing
standards, testing methods, strategy, test
criteria, and tools.
Engineers perform all types of software
testing activities to perform a software test
process.
The last quality checking point for software
is on its production line.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
7
Testing: Objectives
1.
2.
3.
Uncover as many errors (or bugs) as
possible in a given time.
Demonstrate that a given software product
matches its requirement specifications.
Generate high quality test cases, perform
effective tests, and issue correct and helpful
problem reports.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
8
Testing: Objectives
Uncover the errors (defects) in the software,
including errors in:
4.
1.
2.
3.
4.
5.
4/13/2015
requirements from requirement analysis.
design documented in design specifications.
coding (implementation).
system resources and system environment.
hardware problems and their interfaces to
software.
CPSC-4360-01, CPSC-5360-01, Lecture 11
9
Testing: Personnel
Test Manager:
1.
Manage, supervise, and control a software test project.
Define and specify a test plan.
1.
2.
Software Test Engineers and Testers:
2.
Define test cases, write test specifications, run tests.
1.
Development Engineers:
3.
Only perform unit tests and integration tests.
1.
Quality Assurance Group and Engineers:
4.
Perform system testing.
Define software testing standards and quality control
process.
1.
2.
5.
Independent Test Group.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
10
Testing: Activities
Test Planning:
1.
a test schedule for a test process and its activities,
as well as assignments.
test requirements and items.
test strategy and supporting tools.
1.
2.
3.
Test Design and Specification:
2.
1.
2.
4/13/2015
conduct software design based on well-defined
test generation methods.
specify test cases to achieve a targeted test
coverage.
CPSC-4360-01, CPSC-5360-01, Lecture 11
11
Testing: Activities
3. Test Set up:
1. Testing Tools and Environment Set-up.
2. Test Suite Set-up.
4. Test Operation and Execution:
1. Run test cases manually or automatically.
5. Test Result Analysis and Reporting:
1. Report software testing results and conduct test result
analysis.
6. Problem Reporting:
1. Report program errors using a systematic solution.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
12
Testing: Activities
7. Test Management and Measurement:
1. Manage software testing activities, control testing schedule,
measure testing complexity and cost.
8. Test Automation:
1. Define and develop software test tools.
2. Adopt and use software test tools.
3. Write software test scripts.
9. Test Configuration Management:
1. Manage and maintain different versions of software test
suites, test environment and tools, and documents for
various product versions.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
13
Testing: Cycle of Activities


Testing usually involves repetition of testing activities.
Example:
Test
Cases
Test
Results
Specification
Perform
Test
Locate
Error
Design
Error
Repair
Repair
Error
retest
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
14
Test Organization



In a large software system, testing has to be
carried out in multiple stages.
The test stages are linked to the various
software development stages.
Example:


The Acceptance Test is linked to the Requirement
and the Specification stages.
Also known as the V-Model, it refers to the
next slide to see the alphabet ‘V’ in the
diagram.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
15
Testing Organization (V-Model)
Requirements
specification
System
Specification
System
Design
Module
Design
Produce
Produce
Produce
Produce
User
Specification
Validate
System
Specification
Module
Specification
Software
Specification
Validate
Validate
Validate
Acceptance
Testing
System
Testing
Integration
Testing
Unit
Testing
Coding
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
16
Test Stage: Unit Test





Testing individual modules:
 Methods or Functions.
 Classes.
 Sometimes class clusters.
It is based on the information about the structure
of a code fragment.
Carried out by the Development Engineer.
The objective is to test that the unit performs the
function for which it is designed.
Unit tests can be designed before coding begins,
or just after the source code is generated.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
17
Unit Test: Check List

Interface:


Data structure:


Module operates properly at boundaries.
Independent paths:


To ensure the stored data maintains integrity.
Boundary conditions:


To ensure information properly flows in and out.
All paths through the control structure are exercised.
Error handling paths:

4/13/2015
paths that handle errors are exercised.
CPSC-4360-01, CPSC-5360-01, Lecture 11
18
Unit Test: Test Setup

Test case:



Driver:


Test data; should be coupled with expected results.
Often lists Test identifier; Test objectives; Test
resources; Test procedure.
A module that has the test case data, passes it to
component under testing, and logs the results.
Stub:

A dummy module that is called by the component
under testing.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
19
Example (Stub)
Module A
Stub B
Stub C
Stub D
Module A
Module
Under Test
Call Stub B
Call Stub C
Call Stub D
Stub B
Print “in B”
Return
4/13/2015
Stub C
Print “in C”
Return
Stub D
Print “in D”
Return
CPSC-4360-01, CPSC-5360-01, Lecture 11
Temporary
Implementation
20
Generating Unit Test Cases


Statement coverage.
Graph based:




Branch coverage.
Condition coverage.
Path coverage.
These are also applicable to testing methods
within a class.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
21
Unit Testing: Object Oriented Code

Class level testing:




Operations in the class are smallest testable units.
Testing single operation in isolation is difficult.
Unit test is generally driven by structure of
methods of the class and the state behavior of
class.
Recall - the state-transition model and the
state of an object.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
22
Unit Testing: Object Oriented Code

A public method in a class can be tested using a
black-box approach.


Consider each parameter in the method signature,
and identify its possible data range(s):




Start from the specification - class interface.
Equivalence Classes - a black box test approach.
Incorporate pre-conditions and post-conditions in the
test of a method.
Test exceptions.
For complex logic, also use white-box testing or static
testing.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
23
Unit Testing: Object Oriented Code

For private methods,

either modify the class (temporarily) so that it can
be tested externally:



4/13/2015
change the access to public.
or incorporate a test driver within the class.
or use static test methods, such as program
tracing.
CPSC-4360-01, CPSC-5360-01, Lecture 11
24
Using Statechart


Create test cases corresponding to each
transition path that represent a full object life
cycle.
Make sure each transition is exercised at
least once.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
25
Example (Account from Lecture 10)
deposit [amt < -bal]
/bal += amt
statechart
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
26
Example (cont)

Possible Test Cases:

init, deposit(500), withdraw(500):


init, deposit(10), withdraw(20), deposit(5):


Should be in InCredit state.
Should be in OverDrawn state.
init, deposit(10), withDraw(20), deposit(50),
suspend(), unsuspend():


Should be in InCredit state.
The above test cases exercised each transition
at least once.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
27
Test Stage: Integration Testing




The entire system is viewed as a collection of
subsystems (sets of classes) determined during the
system and object design.
The order in which the subsystems are selected for
testing and integration determines the testing
strategy.
It is carried out by the Development Engineer.
A test that leads to construction of the complete
software architecture.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
28
Integration Testing

Big bang integration:


Bottom up integration:



from lower levels.
no test stubs needed.
Top down integration:



all components together.
from higher levels.
no test drivers needed.
Sandwich testing:


4/13/2015
combination of bottom-up and top-down.
top layer with stubs and bottom layer with drivers.
CPSC-4360-01, CPSC-5360-01, Lecture 11
29
Example
A
C
B
E
4/13/2015
Layer I
F
CPSC-4360-01, CPSC-5360-01, Lecture 11
D
Layer II
G
Layer III
30
General Observations

Low level components usually perform more
common tasks:




High level components usually direct the system
activities:


Input/Output.
Repetitive calculations.
Object Oriented: usually entity classes.
Controller.
Summary on functionality:


4/13/2015
Low Level : More specific (Utility Modules).
High Level: More general (Controller Modules).
CPSC-4360-01, CPSC-5360-01, Lecture 11
31
Big-Bang Integration
Unit Test
A
Unit Test
B
Unit Test
C
System Test
Unit Test
D
Unit Test
E
Unit Test
F
4/13/2015
Unit Test
G
CPSC-4360-01, CPSC-5360-01, Lecture 11
32
Bottom-Up Integration
A
Test E
C
B
Test B, E, F
E
Layer I
F
D
G
Layer II
Layer III
Test F
Test C
Test
A, B, C, D,
E, F, G
Test D,G
Test G
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
33
Top-Down Integration
A
C
B
E
4/13/2015
Layer I
F
D
G
Layer II
Layer III
Test A
Test A, B, C, D
Test
A, B, C, D,
E, F, G
Layer I
Layer I + II
All Layers
CPSC-4360-01, CPSC-5360-01, Lecture 11
34
Sandwich Integration
Test B
A
Test E
C
B
Test B, E, F
E
F
Layer I
D
G
Layer II
Layer III
Test F
Test D
Test D,G
Test
A, B, C, D,
E, F, G
Test G
Test A,C
Test A
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
35
Sandwich Integration


Combination of Top-Down and Bottom-Up
approaches.
Select a target level (e.g., Layer II in slide 35):



For level above the target level, perform Top-Down
integration.
For level below the target level, perform Bottom-Up
Integration.
Exercise both Controller modules and Utility
modules at the same time.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
36
Integration Test: Object Oriented Code



Class Clusters:
 Classes that are tightly coupled are good candidates for
an increment integration.
Candidate class clusters:
 Classes in a package.
 Classes in a class hierarchy.
 Classes associated with the interaction diagram for a
use case.
Use based testing:
 group the classes – independent / dependent.


4/13/2015
Begin by testing independent classes.
Then, test dependent classes which use independent classes.
CPSC-4360-01, CPSC-5360-01, Lecture 11
37
Class Cluster: Using Interaction Diagram
:UI
:Controller
request(r)
:X
:Y
methodX()
methodY()
report
4/13/2015
Method to test: request ( )
Input: r
Expected output: a report with
specific information.
CPSC-4360-01, CPSC-5360-01, Lecture 11
38
Class Cluster: Using Use Cases
ATM:
Use Cases:





Withdraw
Money
Withdraw Money.
Deposit funds.
Transfer funds.
Change PIN.
Deposit
Funds
Bank
Customer
Transfer
Funds
Actor:

Bank customer.
Change
PIN
Use Case Diagram
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
39
Summary for Object Oriented Testing
Use Case
Analysis
Use Case
Testing
Cluster
Testing
Class
Design
Integration
Testing
Class
Testing
Method
Design
Method
Testing
Unit
Testing
Method Coding
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
40
Test Stage: System Testing



Performed exclusively in terms of inputs and
outputs of the system.
Performed mostly on the target platform.
Thread-based:



4/13/2015
The behavior that results from a system level input.
An interleaved sequence of system inputs (stimuli)
and outputs (responses).
Depicts a possible scenario of using the system.
CPSC-4360-01, CPSC-5360-01, Lecture 11
41
Test Stage: Testing that Involves Users

Alpha testing:



Beta testing:




In-house testing.
By a test team or end users.
By users or selected subset of actual customers in a
normal work environment.
Product is very close to completion.
Open beta release: Let public carry out the beta testing.
Acceptance testing:

4/13/2015
By users to check that system satisfies requirements to
decide whether to accept the system based on the test
result.
CPSC-4360-01, CPSC-5360-01, Lecture 11
42
Test Case Design: Overview


Black box versus White box approach:
Black box testing:



Knowing the specified function a component has
been designed for.
Tests conducted at the interface of the
component.
White box testing:


4/13/2015
Knowing the internal workings of a component.
Test cases exercise specific sets of condition,
loops, etc.
CPSC-4360-01, CPSC-5360-01, Lecture 11
43
Test Case Design: Overview


Another way to classify the test cases:
Test to specifications:



Test to code:



Also known as black-box, data-driven, functional, or
input/output driven testing.
Ignore the code — use the specifications to select test cases.
Also known as glass-box, logic-driven, structured, or pathoriented testing.
Ignore the specifications — use the code to select test cases.
Neither exhaustive testing to specifications nor
exhaustive testing to code is feasible.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
44
Project: Test Phase
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
45
Requirements for Test Phase

Automated Test Driver:




Read test cases from a file.
Put them through the system.
Capture and validate output.
Test Case Documentation:


Based on Use Cases (Black Box).
Three Tests per Use Case:


4/13/2015
One Valid, Two Invalid.
Validity should be based on functionality.
CPSC-4360-01, CPSC-5360-01, Lecture 11
46
Automated Test Driver

In essence, an Automated Test Driver replaces
the Boundary Classes in your system:


Read directly from a file instead of reading input from
an actual user.
Get output directly from the system and place them
into a file instead of showing on screen.
Input
File
Output
File
4/13/2015
Automated
Test
Driver
CPSC-4360-01, CPSC-5360-01, Lecture 11
Application
Layer
47
Automated Test Driver: Input File

The input file:



Text base: Facilitate modification of test cases.
Contains a number of test cases.
Each of the test cases contains:




4/13/2015
The method to be tested.
Parameter(s) for the method.
The expected output.
Comments (optional).
CPSC-4360-01, CPSC-5360-01, Lecture 11
48
Automated Test Driver: Output File

The output file:



Text base: Easy for human inspection.
Test case result.
Each test case result contains:


Output of method (if any).
Common Categories of Test Result:



4/13/2015
Pass: Test result == Expected result.
Failed: Test result != Expected result.
Human Check: For more complicated cases where the
expected result is hard/impossible to be defined.
CPSC-4360-01, CPSC-5360-01, Lecture 11
49
Test Case Design

The Automated Test Driver can be used to
test in different test phases:




Unit Testing.
Integration Testing.
Etc.
For your project, test cases should be derived
from Use Cases.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
50
Example (Use Case Diagram)

A very simple system to calculate and provide
comments for students’ CAP (Cumulative
Average Percentile on a scale 1-5) points:
Calculate CAP
User
Give Comment
CAP System
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
51
Example (Use Case Descriptions)
Calculate CAP: Basic Course of Events
 User enters a number of grades.
 System displays CAP.
Give Comment: Basic Course of Events
 User enters CAP.
 System displays Comments based on CAP:




4/13/2015
If CAP > 4.0 → "Excellent“;
else if CAP >3.0 → "Good“;
else if CAP >2.0 → "Work hard";
otherwise, → "Work extremely hard“.
CPSC-4360-01, CPSC-5360-01, Lecture 11
52
Example (Class Diagram)
CAP_UI
displayMenu( )
etc...
CAP
computeCAP(grades: double[*])
comment(CAP: double): String
mySystem
4/13/2015
etc...
CPSC-4360-01, CPSC-5360-01, Lecture 11
53
Example (Class Design)


Automated Test Driver takes the place of CAP_UI
class:
Two Classes:

CAPTD: main body of the test driver:



TDIO: utility class:




4/13/2015
A simple loop to go through all test cases.
Makes use of TDIO for reading and writing.
Handles input/output.
Keeps track of pass/fail/check cases.
Contains some useful methods for the test driver in general.
Can be reused for your own Test Driver.
CPSC-4360-01, CPSC-5360-01, Lecture 11
54
Example (Class Design)

The TDIO class:
TDIO
TDIO(ifile: String, ofile: String)
getInput(delim: String): String[*]
pass()
Open “ifile” for reading.
Open “ofile” for writing.
Read a single line from
input file, tokenized using
delim as delimiter.
Write the test result to
output file. Keep track of
the number of each
cases.
fail(s: String)
check(s: String)
endTest()
Write the accumulated
statistics to a file.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
55
Example (Input File)

The test cases (input file):
// The format of the test data for each
//
method is shown first.
// comment,cap[,comment-expected]
comment,4.2,Excellent
comment,1,too bad
comment,3.5
comment,5.0,too good to be true
// computeCAP,grade points separated by comma,
//
cap-expected
computeCAP,1,2,3,4,5,3
// End of test data file
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
56
Example (Test Driver)

The CAPTD class (just a main() method):
public static void main (String[] args) throws IOException {
TDIO t = new TDIO("CAPTDin.txt", "CAPTDout.txt");
CAP c = new CAP();
while ((data = t.getInput(",")) != null) {
if (data[0].equals ("comment")) {
returnedString = c.comment(Double.parseDouble(data[1]));
if (data.length == 3)
if (returnedString.equals(data[2]))
t.pass();
else
t.fail( returnedString );
else t.check( returnedString );
else if (data[0].equals ("computeCAP"))
}
...... }
t.endTest();
}
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
57
Example (Output File)

The output file:
comment,4.2,Excellent
//
pass
comment,1,too bad
//
FAIL! Result expected -->Work extremely hard
comment,3.5
//
Check result! -->Good
... OMITTED ...
comment,5.0,too good to be true
//
FAIL! Result expected -->Excellent
computeCAP,1,2,3,4,5,3
//
pass
pass = 2
fail = 2
Manual Check = 1
total (pass+fail) = 4
Passing rate (pass/total) = 0.5
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
58
Example (Wrapping Up)


Test Case Documentation can be done
directly in the input file.
A good test driver coupled with good test
cases will significantly reduce the
demonstration time.
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
59
Where are we now?
Requirement
Analysis
Design
Implement

Evaluating the System
Test
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
60
Summary

Software Testing


Overview
Test Phases




Unit Testing
Integration Testing
OO Specific Testing
Automated Test Driver


4/13/2015
Overview
Example
CPSC-4360-01, CPSC-5360-01, Lecture 11
61
Reading Suggestions



Chapter 9 of [Bimlesh, Andrei, Soo; 2007]
Chapter 23 of [Somerville; 2007] - Testing
Chapter 9 of [Pfleeger, Atlee; 2006]
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
62
Coming up next



Chapter 9 of [Bimlesh, Andrei, Soo; 2007]
Chapter 23 of [Somerville; 2007] - Testing
Chapters 8 and 9 of [Pfleeger, Atlee; 2006]
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
63
Thank you for your attention!
Questions?
4/13/2015
CPSC-4360-01, CPSC-5360-01, Lecture 11
64