CS 501: Software Engineering
Lecture 21
Reliability 3
1
CS 501 Spring 2005
Administration
2
CS 501 Spring 2005
Validation and Verification
Validation: Are we building the right product?
Verification: Are we building the product right?
In practice, it is sometimes difficult to distinguish between
the two.
That's not a bug. That's a feature!
3
CS 501 Spring 2005
The Testing Process
Unit, System and Acceptance Testing are major parts of a
software project
•
It requires time on the schedule
• It may require substantial investment in test data,
equipment, and test software.
•
Good testing requires good people!
• Documentation, including management and client reports,
are important parts of testing.
What is the definition of "done"?
4
CS 501 Spring 2005
Test Design
Testing can never prove that a system is correct.
It can only show that (a) a system is correct in a
special case, or (b) that it has a fault.
• The objective of testing is to find faults.
• Testing is never comprehensive.
• Testing is expensive.
5
CS 501 Spring 2005
Testing Strategies
• Bottom-up testing. Each unit is tested with its own test
environment.
6
•
Top-down testing. Large components are tested with
dummy stubs.
user interfaces
work-flow
client and management demonstrations
•
Stress testing. Tests the system at and beyond its limits.
real-time systems
transaction processing
CS 501 Spring 2005
Methods of Testing
Closed box testing
Testing is carried out by people who do not know the
internals of what they are testing.
Open box testing
Testing is carried out by people who know the internals of
what they are testing.
(a) What is the advantage of each approach?
(b) In each case, how do you set about selecting test cases?
7
CS 501 Spring 2005
Stages of Testing
Testing is most effective if divided into stages
Unit testing
unit test
System testing
integration test
function test
performance test
installation test
Acceptance testing
8
CS 501 Spring 2005
Testing: Unit Testing
• Tests on small sections of a system,
e.g., a single class
• Emphasis is on accuracy of actual code
• Test data is chosen by developer(s) based on their understanding
of specification and knowledge of the unit
• Can be at various levels of granularity
• Open box: by the developer(s) of the unit
If unit testing is not thorough, system testing becomes almost
impossible. If your are working on a project that is behind
schedule, do not rush the unit testing.
9
CS 501 Spring 2005
Testing: System and Sub-System
Testing
• Tests on components or complete system, combining units
that have already been thoroughly tested
• Emphasis is on integration and interfaces
• Uses trial data that is typical of the actual data, and/or stresses
the boundaries of the system, e.g., failures, restart
• Is carried out systematically, adding components until the
entire system is assembled
• Open or closed box: by development team or by special
testers
System testing is finished fastest if each component is
completely debugged before assembling the next
10
CS 501 Spring 2005
Testing:Acceptance Testing
• Closed box: by the client
• The entire system is tested as a whole
• The emphasis is on whether the system meets the
requirements
• Uses real data in realistic situations, with actual users,
administrators, and operators
The acceptance test must be successfully completed before
the new system can go live or replace a legacy system
Completion of the acceptance test may be a contractual
requirement before the system is paid for
11
CS 501 Spring 2005
Variants of Acceptance Testing
Alpha Testing: Clients operate the system in a realistic
but non-production environment
Beta Testing: Clients operate the system in a carefully
monitored production environment
Parallel Testing: Clients operate new system alongside
old production system with same data and compare
results
12
CS 501 Spring 2005
Test Cases
Test cases are specific tests that are chosen because they are
likely to find faults.
Test cases are chosen to balance expense against chance of
finding serious faults.
13
•
Cases chosen by the development team are effective in
testing known vulnerable areas.
•
Cases chosen by experienced outsiders and clients will be
effective in finding gaps left by the developers.
•
Cases chosen by inexperienced users will find other
faults.
CS 501 Spring 2005
Test Case Selection: Coverage of Inputs
Objective is to test all classes of input
•
Classes of data -- major categories of transaction and
data inputs.
Cornell example: (undergraduate, graduate, transfer, ...)
by (college, school, program, ...) by (standing) by (...)
14
•
Ranges of data -- typical values, extremes
•
Invalid data, reversals, and special cases.
CS 501 Spring 2005
Test Case Selection: Program
Objective is to test all functions of each computer program
•
Paths through the computer programs
Program flow graph
Check that every path is executed at least once
•
Dynamic program analyzers
Count number of times each path is executed
Highlight or color source code
Can not be used with time critical software
15
CS 501 Spring 2005
Test Strategies: Program
(a) Statement analysis
(b) Branch testing
If every statement and every branch is tested is the
program correct?
16
CS 501 Spring 2005
Statistical Testing
•
Determine the operational profile of the software
•
Select or generate a profile of test data
• Apply test data to system, record failure patterns
•
17
Compute statistical values of metrics under test conditions
CS 501 Spring 2005
Statistical Testing
Advantages:
•
Can test with very large numbers of transactions
•
Can test with extreme cases (high loads, restarts, disruptions)
• Can repeat after system modifications
Disadvantages:
•
Uncertainty in operational profile (unlikely inputs)
• Expensive
•
18
Can never prove high reliability
CS 501 Spring 2005
Regression Testing
REGRESSION TESTING IS ONE OF THE KEY
TECHNIQUES OF SOFTWARE ENGINEERING
Applied to modified software to provide confidence that
modifications behave as intended and do not adversely
affect the behavior of unmodified code.
• Basic technique is to repeat entire testing process
after every change, however small.
19
CS 501 Spring 2005
Regression Testing: Program Testing
1. Collect a suite of test cases, each with its expected
behavior.
2. Create scripts to run all test cases and compare with
expected behavior. (Scripts may be automatic or have
human interaction.)
3. When a change is made, however small (e.g., a bug is
fixed), add a new test case that illustrates the change (e.g.,
a test case that revealed the bug).
4. Before releasing the changed code, rerun the entire test
suite.
20
CS 501 Spring 2005
Documentation of Testing
Testing should be documented for
thoroughness, visibility and for maintenance
(a) Test plan
(b) Test specification and evaluation
(c) Test description
(d) Test analysis report
21
CS 501 Spring 2005
A Note on User Interface Testing
User interfaces need two categories of testing.
• During the design phase, user interface testing is carried out
with trial users to ensure that the design is usable. This
design testing is also used to develop graphical elements and
to validate the requirements.
• During the implementation phase, the user interface goes
through the standard steps of unit and system testing to check
the reliability of the implementation.
Acceptance testing is then carried out on the complete system.
22
CS 501 Spring 2005
Fixing Bugs
Isolate the bug
Intermittent --> repeatable
Complex example --> simple example
Understand the bug
Root cause
Dependencies
Structural interactions
Fix the bug
Design changes
Documentation changes
Code changes
23
CS 501 Spring 2005
Moving the Bugs Around
Fixing bugs is an error-prone process!
• When you fix a bug, fix its environment
• Bug fixes need static and dynamic testing
• Repeat all tests that have the slightest relevance
(regression testing)
Bugs have a habit of returning!
• When a bug is fixed, add the failure case to the test suite
for the future.
24
CS 501 Spring 2005
Security in the Software Development
Process
The security goal
The security goal is to make sure that the agents (people or
external systems) who interact with a computer system, its
data, and its resources, are those that the owner of the system
would wish to have such interactions.
Security considerations need to be part of the entire software
development process. They may have a major impact on the
architecture chosen.
Example. Integration of Internet Explorer into Windows
25
CS 501 Spring 2005
Agents and Components
A large system will have many agents and components:
•
each is potentially unreliable and insecure
•
components acquired from third parties may have unknown
security problems
The software development challenge:
•
develop secure and reliable components
• protect whole system from security problems in parts of it
26
CS 501 Spring 2005
Techniques: Barriers
Place barriers that separate parts of a complex system:
• Isolate components, e.g., do not connect a computer to a
network
• Firewalls
• Require authentication to access certain systems or parts
of systems
Every barrier imposes restrictions on permitted uses of the
system
Barriers are most effective when the system can be divided
into subsystems with simple boundaries
27
CS 501 Spring 2005
Techniques: Authentication &
Authorization
Authentication establishes the identity of an agent:
• What the agent knows (e.g., password)
• What the agent possess (e.g., smart card)
• Where does the agent have access to (e.g., controller)
• What are the physical properties of the agent (e.g., fingerprint)
Authorization establishes what an authenticated agent may
do:
• Access control lists
• Group membership
28
CS 501 Spring 2005
Example: An Access Model for Digital
Content
User
Roles
Actions
Digital material
Access
Attributes
29
Operations
Policies
CS 501 Spring 2005
Techniques: Encryption
Allows data to be stored and transmitted securely, even
when the bits are viewed by unauthorized agents
Encryption
Y
X
Decryption
Y
• Private key and public key
X
• Digital signatures
30
CS 501 Spring 2005
Security and People
People are intrinsically insecure:
• Careless (e.g, leave computers logged on, use simple passwords,
leave passwords where others can read them)
• Dishonest (e.g., stealing from financial systems)
• Malicious (e.g., denial of service attack)
Many security problems come from inside the organization:
• In a large organization, there will be some disgruntled and
dishonest employees
31
• Security relies on trusted individuals. What if they are
dishonest?
CS 501 Spring 2005
Design for Security: People
• Make it easy for responsible people to use the system
• Make it hard for dishonest or careless people (e.g., password
management)
• Train people in responsible behavior
• Test the security of the system
• Do not hide violations
32
CS 501 Spring 2005
Suggested Reading
Trust in Cyberspace, Committee on Information Systems
Trustworthiness, National Research Council (1999)
http://www.nap.edu/readingroom/books/trust/
Fred Schneider, Cornell Computer Science, was the chair of
this study.
33
CS 501 Spring 2005