CISC/CMPE320
• Assignment 4 is due Nov. 22 (this Sunday).
• Presentation stuff.
• Today, More Software Engineering:
– System Failure and A Movie!
– Testing, Cont.
Fall 2015
CISC/CMPE320 - Prof. McLeod
1
Week 12 Team Presentations
• Demonstrate of the fruit of your efforts!
• And, summarize (in any order you wish):
– How work ended up being distributed and when it was
completed.
– Difficulties faced and overcome.
– Difficulties not overcome.
– Good/bad library use.
– Work left to do, if any.
– Team techniques that worked and those that did not.
– What you would do differently if you had to do this
again…
Fall 2015
CISC/CMPE320 - Prof. McLeod
2
Presentation Schedule – Week 12
• Tuesday, Dec. 1 – Basswood, Beech, Cherry,
Tamarack.
• Thursday, Dec. 3, Lecture Time – Hickory, Maple,
Oak, Birch.
• Thursday, Dec. 3, Tutorial Time (JEF 128) –
Poplar, BalsamFir, Spruce, Cedar.
• Friday, Dec. 4 – Hemlock, JackPine, Walnut,
WhitePine.
Fall 2015
CISC/CMPE320 - Prof. McLeod
3
Peer Grading, Again.
• Sign-up sheets will be circulated again during
presentations.
• Project Wikis will be make public on Nov. 25.
• As for last time – two Moodle surveys:
– The Wiki: Is the SDD up-to-date and does it accurately
describe the architecture? Is the Wiki looking better
than the last time you saw it?
– The Presentation: Organized, effective, met initial
goals, good discussion of problems overcome?
Fall 2015
CISC/CMPE320 - Prof. McLeod
4
System Failure
• Often the result of a combination of events that is
nearly impossible to predict or test for.
• See presentation from Dr. Terry Shepard linked to
the lecture notes page – in particular the slides on
the FBI VCF (“Virtual Case Files”) failure.
Possibly the most highly publicized software
failure.
• A few more examples ranging from just annoying
to deadly:
Fall 2015
CISC/CMPE320 - Prof. McLeod
5
Microsoft Zune 30 Meltdown
• Apparently 2006 models of this MP3
player just froze up at midnight, Dec. 31,
2008.
http://slashdot.org/article.pl?sid=09/01/04/2034248
• Eventually Microsoft acknowledged the
problem.
• Here is the offending C code:
• Can you spot the problem?
Fall 2015
CISC/CMPE320 - Prof. McLeod
6
#define ORIGINYEAR 1980
BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime) {
int dayofweek, month, year;
UINT8 *month_tab;
//Calculate current day of the week
dayofweek = GetDayOfWeek(days);
year = ORIGINYEAR;
while (days > 365) {
if (IsLeapYear(year)) {
if (days > 366) {
days -= 366;
year += 1;
}
}
else {
days -= 365;
year += 1;
}
}
…
Fall 2015
CISC/CMPE320 - Prof. McLeod
7
Communications Problems
• The designers of the New England branch of the
ARPAnet decided on seven-fold redundancy.
Unfortunately all seven lines were sent through
the same fibre optic trunk line, which was severed
on Dec. 12, 1986.
• (Not all systems problems are a result of software
failure!!)
Fall 2015
CISC/CMPE320 - Prof. McLeod
8
More Communications Problems
• In late 1989, AT&T installed faulty software in 114
electronic switching systems. The problem was
traced to a break statement inside an if
statement inside a switch statement in a C
program.
• A cascading problem caused all 114 switches to
break down.
• AT&T was shut down nationwide for 9 hours, with
an estimated 5 million calls blocked as a result.
Fall 2015
CISC/CMPE320 - Prof. McLeod
9
Nuclear Power
• Bad software contributed to the accident at Three
Mile Island 2 in 1979
• Core temperatures were being measured with
thermocouples, but the software would not report
temperatures over 700 degrees – “??????” was
reported instead, even though the thermocouples
were capable of measuring higher temps.
• Also intended valve settings were displayed by
the control software rather than actual valve
settings…
Fall 2015
CISC/CMPE320 - Prof. McLeod
10
Defense and Warfare Problems
• Patriot Missiles during the Gulf War
– Reported to the media that they were
95% effective.
– Turns out that they were really only about
13% effective due to a clock drift
problem.
– This caused a Patriot missile to miss the
Scud missile that hit an American Army
barrack in Dhahran, killing 29 and injuring
97.
– This is a famous and classic example of a
failure of software engineering.
Fall 2015
CISC/CMPE320 - Prof. McLeod
11
Defense and Warfare Problems, Cont.
• Iran Air Flight 655 was shot down by USS
Vincennes’ missile in 1988
– 290 People were killed.
– The error was blamed mostly on the poor interface
used by the Aegis radar tracking system used aboard
the Vincennes.
Fall 2015
CISC/CMPE320 - Prof. McLeod
12
The F.I.R.S.T. Rules
• Fast tests are more likely to be run as often as
they should be.
• Independent tests will not depend on each other.
You should be able to run tests in any order.
• Repeatable tests can be run in any hardware/OS
environment.
• Self-validating tests either pass or fail.
• Timely tests follow the three laws of TDD.
Fall 2015
CISC/CMPE320 - Prof. McLeod
13
Overview of the Testing Process
Fall 2015
Process
Testing
Coding
Unit Testing
Design
Integration
Testing
Requirements
Analysis
Validation Testing
System
Engineering
System Testing
CISC/CMPE320 - Prof. McLeod
14
Overview, Cont.
• Test Planning – Allocate and schedule resources.
Do this early on – as soon as models stabilize.
• Usability Testing – Test user interface. (See
“Paper Prototyping”, for example.)
• Unit Testing – Test individual methods and
classes.
• Integration Testing –
– First separate classes by threads.
– Start by identifying independent classes and test them
first, then test dependent classes, layer by layer.
– Cluster testing – test a bunch of collaborating classes
together.
– Use regression testing, stub and driver code as needed
(more about these later!).
15
Overview, Cont.
• System Testing – Test all the classes together.
– Functional Testing tests the requirements listed in the
RAD and user manual.
– Performance Testing checks non-functional
requirements and additional design goals from the
SDD. (Done by developers).
– Acceptance and Installation Testing – Check the
system against any agreement made. (Done by clients
with help from developers, if needed).
Fall 2015
CISC/CMPE320 - Prof. McLeod
16
More Definitions…
• A Component is the class, the cluster of classes,
the package or one or more subsystems that is
being tested.
• A Test Case is a set of inputs and expected
results that exercises a component with the
purpose of causing failures and detecting faults.
• A Test Stub is partial implementation of a
component upon which the component being
tested depends.
• A Test Driver is a partial component that depends
on the component being tested.
– Stubs and Drivers allow the component to be
harnessed – isolated from the rest of the system.
Fall 2015
CISC/CMPE320 - Prof. McLeod
17
More Definitions, Cont.
• A Correction is a change to a component made to
repair a fault. A correction can introduce new
faults…
“Round and round we go – where we’ll stop nobody knows!”
Fall 2015
CISC/CMPE320 - Prof. McLeod
18
More Definitions, Cont.
• A system that fails after a correction has
regressed.
• Carrying out the previous set of integration tests
again is called Regression Testing.
Fall 2015
CISC/CMPE320 - Prof. McLeod
19
Unit Testing
• Common types of unit testing:
–
–
–
–
Fall 2015
Equivalence testing
Boundary testing
Path testing
State-based testing
CISC/CMPE320 - Prof. McLeod
20
Equivalence Testing
• A blackbox technique.
– (Deal with the input/output of the component, leave the
internal behaviour alone.)
• Minimal number of test cases.
• First, determine a set of equivalence classes.
• You assume that the component behaves the
same for all members of the equivalence class.
Fall 2015
CISC/CMPE320 - Prof. McLeod
21
Equivalence Testing, Cont.
• Criteria for equivalence classes:
– Coverage – every possible input belongs to one of the
equivalence classes.
– Disjointedness – no input belongs to more than one
equivalence class.
– Representation – if one member of an equivalence
class causes an erroneous state then the same state
can be caused by any other member of that class.
• For each class, select at least two test inputs – a
valid input and an input that should exercise your
exception handlers.
Fall 2015
CISC/CMPE320 - Prof. McLeod
22
Equivalence Testing, Cont.
• Example – suppose you are testing a class that
stores dates, as given by a year and month.
• You would need six equivalence classes:
–
–
–
–
–
–
31 day months, non-leap years
31 day months, leap years
30 day months, non-leap years
30 day months, leap years
28 day month, non-leap years
29 day month, leap year
• Minimum of 12 tests!
Fall 2015
CISC/CMPE320 - Prof. McLeod
23
Boundary Testing
• Look at the “edges” of the equivalence classes.
• For example:
–
–
–
–
Fall 2015
Leap year divisible by 400 (year 2000, for example)
Non-leap year divisible by 100 (year 1900, for example)
Invalid month (at negative edge of legal months): 0
Invalid month (at positive edge of legality): 13
CISC/CMPE320 - Prof. McLeod
24
Equivalence and Boundary Testing
• The problem with these techniques is that they do
not explore many combinations of input data.
• Often a strange combination will cause an error,
and this will take many more tests to locate.
• (My instincts say that there are just not enough
tests!)
Fall 2015
CISC/CMPE320 - Prof. McLeod
25
Path Testing
• A whitebox technique.
– (Focus on the internal structure of the component. Test
every state of the component and every possible
interaction of objects.)
• Exercise every possible path through the code at
least once.
– Exercise every branch of every if statement.
– Run each loop – can you cause a loop not to run?
– Throw each exception.
Fall 2015
CISC/CMPE320 - Prof. McLeod
26
Path Testing, Cont.
• The problem with this technique is that it cannot
show where code is missing – it only tests existing
code.
Fall 2015
CISC/CMPE320 - Prof. McLeod
27
State – Based Testing
• Developed for GUI systems.
• A new technique that is not fully fleshed out.
• Based on the idea that objects (like a button) can
have different states under different conditions
(like active, inactive, pressed, mouseover, etc.)
• See: WindowTester from Google, for example:
https://developers.google.com/java-devtools/wintester/html/index
Fall 2015
CISC/CMPE320 - Prof. McLeod
28
Polymorphism Testing
• OOP provides additional challenges to testing
when a program takes advantage of
polymorphism.
• You must make sure that every state of an object
is tested.
• This can lead to many more tests especially when
polymorphism is used by several objects in a
component.
Fall 2015
CISC/CMPE320 - Prof. McLeod
29