Test Strategies for Business
Critical Software
Testing business-critical
software
Mike Bartley
Founder and CEO,
Test and Verification Solutions
Content
1.
2.
3.
4.
What is Business-critical Software?
Verification and Testing Requirements
The Cost of High Integrity Software
Safety-critical techniques in business
–
–
–
–
–
Requirements Driven Testing
Unit level testing
Test coverage criteria
Static Analysis
Independence
Critical Systems
• Safety-critical systems
– A system whose failure may result in injury, loss of life or serious
environmental damage.
– An example of a safety-critical system is a control system for a
chemical manufacturing plant.
• Mission-critical systems
– A system whose failure may result in the failure of some goal-directed
activity.
– An example of a mission-critical system is a navigational system for a
spacecraft.
• Business-critical systems
– A system whose failure may result in very high costs for the business
using that system.
– An example of a business-critical system is the customer accounting
system in a bank.
(c) Ian Sommerville 2008
Some Example Failures
The Business Cost of Failure
• Direct costs
–
–
–
–
Cost to fix the bug
Re-release costs
Recall costs
Compensation
• Loss of revenue
• Opportunity costs
– Engineers tied up fixing bugs
• Reputation costs
– Loss of customers
– Brand damage
It is all about risk
• Rising Complexity
– Causes increased likelihood
• Rising cost
– Rising cost of failure when all aspects considered
• Risk = Likelihood * Cost
What are the major pressures?
Safety
Compliance
Audit trail
Coverage
Schedule
Development Costs
Flexibility
Others?
Business
What are the major pressures?
Safety
Compliance
1
Audit trail
1
Coverage
2
Schedule
3
Development Costs
3
Flexibility
4
Others?
Business
Safety-Critical Quality Characteristics
•
•
•
•
•
•
•
Defensive
Resilient to errors
Fail safe?
Necessary functionality only
Extensible and maintainable – long lifetimes
Reliable
Very low residual fault rates
Quality Characteristics in Business
Business
Defensive
Reliable
Resilient to errors
Fail safe?
Necessary functionality only
Extensible and maintainable –
long lifetimes
Very low residual fault rates
Safety-Critical:
Verification and Testing
•
•
•
•
Depend on integrity level/class
Software Verification Plan
Reviews of Specifications and Code
Testing (against specifications)
– Unit
– Software Integration
– Software System
•
•
•
•
Test Coverage Criteria
Requirements and Test Traceability
Independence
Audit trails
IEC61508
• Dynamic analysis and testing
Technique
SIL 1 SIL 2 SIL 3 SIL 4
Structural test coverage (entry points) 100%
HR
HR
HR
HR
Structural test coverage (statements) 100%
R
HR
HR
HR
Structural test coverage (branches) 100%
R
R
HR
HR
Structural test coverage (conditions, MC/DC) 100%
R
R
R
HR
Test case execution from boundary value analysis
R
HR
HR
HR
Test case execution from error guessing
R
R
R
R
Test case execution from error seeding
-
R
R
R
Test case execution from model-based test case
generation
R
R
HR
HR
Performance modelling
R
R
R
HR
Equivalence classes and input partition testing
R
R
R
HR
Is High Integrity Software
Expensive?
• It can be but ….
40-50% project effort on avoidable
rework
Extra 50% cost to develop high
integrity software
Extra 75% cost to maintain low
integrity software
Source: Boehm – Software Management Article 2001
And …
Programmers <50% efficient at finding own
defects
Reviews/Inspections often >65% efficient
Static analysis often >65% efficient
Inspections, Static analysis, Testing:
>97% efficient
Lower costs & schedules by >20%
Lower TCO by >45%
Source: Capers Jones – Software Quality in 2012: A Survey of the State of the Art
All the Right Stages but Not
Necessarily in the Right Order
Software
Requirements
Software System
Testing
High Level
Design
Integration Testing
Unit Level
Design
Unit Testing
Coding
… But We Could Shift ‘Testing’
Left
Requirements
High Level
Design
Low Level
Design
Code
Dev Test
Accept Test
Defects
Requirements
High Level
Design
Low Level
Design
Code
Defect Detection Point
Dev Test
Accept Test
Agile Development
Sprint 1
0
1
Sprint 2
3
2
Sprint
1 Dev
Sprint 3
Sprint
2 Dev
Rwk
TD
Sprint 4
5
4
Sprint
3 Dev
Rwk
TD
Sprint 5
Sprint
3 Dev
Rwk
TD
TD = Technical Debt
Things to consider
•
•
•
•
•
Requirements Driven Testing
Unit level testing
Test coverage criteria
Static Analysis
Independence
Requirements Driven Testing
Bidirectional Map
requirements ↔ tests
Product A
• Advantages
– Identify test holes and test
orphans
– Reporting of requirements
signoff status
– Support for
• Risk-based testing
• Filtering Verification on
Customers and releases
• Impact analysis
– Support for regulatorybased requirements signoff
– Easier variant management
Req B
Req C
Feature E
Req D
Feature F
Feature G
Impl H
Impl I
Test 1
Test 2
Test 3
Unit Testing
• Verification mandatory for higher integrity
levels
• Needs unit specifications
• Typically dynamic testing with coverage
analysis
• Coverage criteria increase with integrity level:
–
–
–
–
Statement
Decision
Condition
MC/DC
The Unit Test Foundation
Integration of Pre-tested
Components
Unit Testing
•
•
•
•
•
Provides confidence in units
Provides integration predictability
Provides basis for regression testing
Provides visibility of progress
Can be expensive so:
– Efficient/fast
– Repeatable
– Automated
Unit Test Environment
Called
Functions
Global data
Input parameters:
Output parameters
Return
SUT
Static
Called
Functions
Functions
Static data
Static Analysis
• IEC 61508
• “defensive programming, modular approach, coding standards,
structured programming” would all be prudent
• How to show these have been considered?
• IEC 60880
• More specific. Examples:
• GOTO statements as well as label variables should be avoided
• Data structures and naming conventions shall be used uniformly throughout
the system
• Subroutines should have only a predefined maximum number of parameters
• Subroutines should be organised as simply as possible
• IEC 62304, ISO 26262, DO-178 etc, etc
• Similar again, with varying levels of the specifics
What They Have in Common
• Define some rules that can be checked e.g.:
–
–
–
–
–
–
–
McCabe Cyclomatic Complexity < 10
Max nesting level < 4
No GOTO statements
All global variables prefixed with glb_
No dead code
No unused variables
etc
• Justify why these are sufficient
• Provide evidence that your development meets
these rules
MISRA
• Coding standard, not a Business standard
– MISRA C 2004
– MISRA C++ 2008
– MISRA C 3 / 2012
• Approach is to write safer software from the
beginning be using a very restrictive subset of
the language
• Using a pre-defined and well know coding
standard reduces the amount of justification
required for auditors
– We’re using MISRA, except for... – great! 
– We’re using our own awesome coding standard – justify yourself! 
Static Analysis Tools
• If you are using tool, rather than code
inspections to verify your coding rules for a
functional Business standard certified project
then:
– You just need to record and justify any deviations to the
planned rules
– Generate a report with any ignored defects and the
justifications why each is accepted and with what
controls
And importantly …
More advanced static analysis tools also weed out critical bugs,
early!
Unit/integration testing then focuses on finding design issues
Independence
• Independence avoids author bias and is often
more effective at finding defects and failures.
• There are various levels of independence:
– The person who wrote the item
– Another person within the same team
– Another person from a different group
• such as an independent test team
– Another person from a different organization or
company
• such as outsourced testing
• or certification by an external body
Benefits of independent testing
•
•
•
•
•
Use of professional testers
A different set of assumptions
A separate reporting route
A separate budget and resource plan
A separate career path
Risks of independent testing
•
•
•
•
•
Lack of deep knowledge of product and code
Need for documentation
“Less efficient” than programmer testing
“Over the wall”
Isolation
– Communication problems
– Them-and-us … open hostility
Summary
• Do we need to consider new approaches for
business critical software?
–
–
–
–
–
Requirements Driven Testing
Unit level testing
Test coverage criteria
Static Analysis
Independence
• Is the cost of failure now high enough?