CS 5150
Software Engineering
Lecture 21
Reliability 2
Administrivia
•
Presentation Feedback
•
•
•
•
•
•
•
Transitions
Demos
Code
Jargon
Conclusion
Quiz 3
•
•
•
CS 5150
Generally a big improvement
Design patterns
Security/privacy
Reliability
2
Final Milestone
•
•
CS 5150
Presentation
•
Demo
Report
•
Cumulative
3
SE in the News
CS 5150
4
Reliability Analysis - Consequences
CS 5150
5
Iron Law of Software Reliability
•
CS 5150
For most non-trivial applications, 100%
reliability is effectively impossible
6
Verification and Validation
•
•
•
•
CS 5150
Does the software do what it is intended to
do?
Validation
•
Does the design match the intended
behavior?
Verification
•
Does the implementation match the design?
Informal and formal
7
Verification
•
•
•
CS 5150
Does the implementation exhibit any particular
bad behaviors?
Can we guarantee that the implementation
does what it is designed to do?
“How did software get so good without proof?”
•
Dijkstra
8
Static and Dynamic
•
•
CS 5150
Static
•
•
Examination of source code without actually
running it
Can be manual or tool-assisted
Dynamic
•
•
•
Testing
Debugging
Profiling
9
Informal Static Verification
•
•
Code reviews
Can be light or heavy
•
•
•
Meetings with multiple participants
Benefits:
•
•
•
•
CS 5150
Colleague reads over patch before committing to
repository
“given enough eyeballs, all bugs are shallow”
Opportunity to share knowledge
Opportunity to do housekeeping
Exposes inconsistent assumptions between developers
10
Getting the Most from Code Reviews
•
•
•
CS 5150
Senior team members must show leadership
heavy-weight reviews require preparation
The tone must remain constructive
•
Code reviews are not mini personnel
reviews
11
Review Checklists
•
•
•
•
•
•
CS 5150
Data faults: Initialization, constants, array bounds,
character strings
Control faults: Conditions, loop termination,
compound statements, case statements
Input/output faults: All inputs used; all outputs
assigned a value
Interface faults: Parameter numbers, types, and
order; structures and shared memory
Storage management faults: Modification of links,
allocation and de-allocation of memory
Exceptions: Possible errors, error handlers
12
Working in Pairs
•
•
•
•
Idea: achieve benefits of continuous review by
doing all work in pairs
Hard for management to swallow because it
looks like doubling the personnel cost of
development
Some evidence that it works well
Generally highest value with a senior/junior
pair
•
CS 5150
Junior person should ‘drive’
13
Static Analysis Tools
•
•
•
•
•
•
•
CS 5150
Scan source code for possible problems
Compiler warnings on steroids
Superficial syntactic warnings
Deeper analysis based on symbolic execution
void foo( int *p ) {
printf( “%i”, *p );
}
14
Scaling of Static Analysis
•
In general, static analyses cannot be both
complete and sound
•
•
•
Sound - no false negatives
Static analysis tools have become more
common in the last 10 years
•
CS 5150
Complete - no false positives
Careful engineering trade-offs between false
positives and false negatives
15
Heavy-Weight Formal Verification
•
•
•
CS 5150
Hoare logic
•
•
Pre-conditions/post-conditions
Invariants
Model checking
Assertions
16
Testing Testing Testing
•
Testing comes in many flavors
•
•
•
•
•
•
•
•
•
CS 5150
User testing
Unit testing
Integrations testing
Performance testing
Stress testing
Regression testing
Acceptance testing
Manual/automated testing
Close/open-box (black/white-box) testing
17
Fundamental Facts About Testing
•
•
Testing is indispensable for software
verification, but can never prove that non-trivial
software is defect-free
Each test proves that the system works in one
particular circumstance
•
•
CS 5150
Most non-trivial software has infinitely many
possible execution contexts (i.e., input,
environment, etc)
Every project should have a verification plan
that includes testing strategies
18
Unit Testing
•
•
•
•
•
CS 5150
Each unit test should verify that a single ‘unit’
of software works in isolation
Can be challenging to write unit tests for code
buried deep in a project
Overlap with type systems
Basic sanity checks
Should run automatically
•
e.g. overnight
19
Regression Testing
•
Every time you find a bug in your project,
make a test case that reproduces the bug
•
•
A good set of regression tests provide a
‘ratcheting’ mechanism
•
CS 5150
The test case should be committed to your
repository no later than the bug fix itself
Helps prevent the recurrence of bugs
20
Determinism => Reproducibility
•
•
It is very hard to fix bugs that cannot be
reproduced reliably
Do not introduce unnecessary nondeterminism into your programs
•
•
•
CS 5150
Random numbers
Address-based data structures
Multithreading
21