abc
Logic vs. Magic in
Critical Systems
or: why wizards are not enough
Ian O’Neill
Hampshire BCS – February 2002
1
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
2
abc
A Historical Magic
• Go to statement considered harmful
• Structured programming
• “De-spag” tools
Hampshire BCS – February 2002
3
A Specification
abc
Add up the first N (positive) values of the array
traverse the array until N positive values have been found
and sum them
or
traverse N elements of the array and sum any positive
ones (ignoring any negative ones) ?
Hampshire BCS – February 2002
4
function Sum(A : Atype; Count : Aindex) return Integer
is
Accumulator, Value : Integer;
Index
: Aindex;
abc
begin
Accumulator := 0;
Index := 1;
<<Label1>> Value := A(Index);
if Value <= 0 then
goto Label2;
end if;
Accumulator := Accumulator + Value;
<<Label2>> if Index = Count then
goto Label3;
end if;
Index := Index + 1;
goto Label1;
<<Label3>> return Accumulator;
end Sum;
Hampshire BCS – February 2002
5
abc
function Sum(A
: Atype;
Count : Aindex) return Integer
is
IC
: Counter;
Accumulator, Value : Integer;
Index
: Aindex;
begin
IC := 1;
Accumulator := 0;
Index := 1;
while IC < 6 loop
case IC is
when 1 => Value := A(Index); IC := 2;
when 2 => if Value <= 0 then IC := 4; else IC := 3; end if;
when 3 => Accumulator := Accumulator + Value; IC := 4;
when 4 => if Index = Count then IC := 6; else IC := 5; end if;
when 5 => Index := Index + 1; IC := 1;
when 6 => Null;
end case;
end loop;
return Accumulator;
end Sum;
Hampshire BCS – February 2002
6
abc
Engineering:
The use of science and
mathematics to solve practical
problems
Hampshire BCS – February 2002
7
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
8
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
9
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
10
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
11
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
12
A Specification
abc
Add up the first N (positive) values of the array
nonNegSum : seq Z  N
nonNegSum  0
x : N; s : seq Z  nonNegSum  x ^ s  x  nonNegSum s
x : Z \ N; s : seq Z  nonNegSum  x ^ s  nonNegSum s
sum : seq Z  Z  N
s : seq Z; n : Z  sum s n  nonNegSum( 1..n)  s
Hampshire BCS – February 2002
13
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
14
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
15
abc
Language Insecurity
a tiny example
procedure Init2(X, Y : out Integer)
is
begin
X := 1;
Y := 2;
end Init2;
What is the meaning of:
Init2(A, A);
Hampshire BCS – February 2002
16
abc
The software process
• Progressive refinement:
– a need
– requirements
– specification
– design
– source code
– object code
Hampshire BCS – February 2002
17
abc
Formality at last!
10100110100100001000100010010001
10101000111101010100001010101010
10101010100101001010101010000101
01010010010101001101010101010101
00000101000101010010101001010010
01010101010101010101011110100101
0010101010101010010.......
Hampshire BCS – February 2002
18
abc
A typical development process
Informal, opaque
black box
Precise object
code
Hampshire BCS – February 2002
19
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
20
abc
What are critical systems?
• Some important objectives:
– efficiency
– cost
– time to market
– functionality
Hampshire BCS – February 2002
21
abc
What are critical systems?
• Systems where reliability is more important than:
– efficiency
– cost
– time to market
– functionality
Robert Dewar - Ada Core Technologies
Hampshire BCS – February 2002
22
abc
Contributors to Lifecycle Costs
100
90
80
70
60
50
40
30
20
10
0
Cost of error correction
design
Time spent:
Hampshire BCS – February 2002
code
<50%
<20%
integrate
IV&V
>50%
>80%
23
abc
Producing Safety-Critical Software
• Not just a question of “being more careful”
• The need to be able to show, before there is any
service experience, that a system will be safe enough
requires a qualitatively different approach
• We can only achieve this by logical reasoning.
Hampshire BCS – February 2002
24
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
25
abc
Foundations of Reasoning
•
•
•
•
•
We reason about information
But what about “information hiding”?
We need to hide detail not information
We do this through abstraction
We can’t just magic complexity away
Example: Program “State”
Hampshire BCS – February 2002
26
abc
Swap Algorithm
Temp : T;
procedure Swap(X, Y : in out T) is
begin
Temp := X;
X := Y;
Y := Temp;
end Swap;
Hampshire BCS – February 2002
27
abc
Swap Algorithm
procedure Swap(X, Y : in out T) is
Temp : T;
begin
Temp := X;
X := Y;
Y := Temp;
end Swap;
Hampshire BCS – February 2002
28
abc
A Store Object
package Store
is
procedure Put(X : in T);
function Get return T;
end Store;
Hampshire BCS – February 2002
29
abc
Swap Algorithm
procedure Swap(X, Y : in out T) is
begin
Store.Put(X);
X := Y;
Y := Store.Get;
end Swap;
Hampshire BCS – February 2002
30
abc
Heap
Store
Swap
Hampshire BCS – February 2002
31
abc
Store
Swap
Hampshire BCS – February 2002
????
32
abc
• The location of state is the single biggest
influence on coupling and cohesion.
• It is probably the most important design decision
we must make
yet
• OOP regards it as an “implementation detail”; and
• UML does not even have a notation to express it!
Hampshire BCS – February 2002
33
abc
OOP - The Biggest Magic of them All
• We hide all state
• We disguise the hierarchical relationships between
objects
• We hide the control flow by use of messages and
dynamic dispatch
• In extreme cases, we even deny there is a software
design process at all (Shlaer Mellor)
Of course all this hiding makes things easier - at least until all
the hidden complexity emerges during integration testing!
Hampshire BCS – February 2002
34
abc
An OOP Array Sum
State Handler
Count
Handler
Hampshire BCS – February 2002
Total Handler
Data Handler
35
abc
Hampshire BCS – February 2002
36
abc
“There are two ways of constructing a
software design. One way is to make
it so simple that there are obviously no
deficiencies. And the other way is to
make it so complicated that there are
no obvious deficiencies.”
Professor C.A.R. Hoare
The 1980 Turing award lecture
Hampshire BCS – February 2002
37
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
38
abc
Some obstacles
•
•
•
•
•
•
Fashion
Low expectations
Poor contract writing
The difficulty of saying “no”
Gurus
Belief that:
– Software engineering is the same as knowing tool
or product X
– Knowing tool or product X is more important than
domain or engineering knowledge
Hampshire BCS – February 2002
39
abc
Fashion
• We must use X, everyone else is
• The staff want to use X, we won’t be able to recruit or
keep them if we don’t
• All the tool vendors are supporting X
• Y, which we know is better, “is dead”
• Progress is the same as the number of lines of code
written
• The most important thing is to get to test quickly
Hampshire BCS – February 2002
40
abc
Low Expectations
• Software is always buggy
• Even Microsoft “the best software company in the
world” produces buggy software
• Vendors never offer warranties
• Since software is always a load of rubbish we might
as well buy the cheapest load of rubbish
Hampshire BCS – February 2002
41
abc
Hampshire BCS – February 2002
42
abc
Hampshire BCS – February 2002
43
abc
Poor Contract Writing
• Fixed-price contracting
– sets producer and customer in opposition
– even mutually beneficial changes are difficult
• Time and material contracting
– removes any incentive to do a more efficient
job
Hampshire BCS – February 2002
44
abc
The Difficulty of Saying No
• Engineers have a duty to say “No” where safety is
being compromised; however,
• There is often someone else who can be persuaded
to say “Yes” (and charge you less!)
• The Chinook FADEC saga:
– the safety assessors have been cast as the villains
unreasonably holding up entry to service
– the software is being treated as safe because it is
so badly written it is not possible to prove that it’s
dangerous
Hampshire BCS – February 2002
45
abc
Gurus
• Close associates of wizards and magicians
• Know all 123 operator precedence levels of C
• Don’t make the kinds of mistakes that mere mortals
make so don’t see the benefit of techniques that help
avoid them
• Have a strong incentive to maintain the status quo
because they fear losing their guru status
• Don’t often make good engineers
Hampshire BCS – February 2002
46
abc
Misunderstanding the Nature of
Knowledge
• C programmers are easy to hire so we should use C
– riveters are easy to hire but it hasn’t stopped
aircraft companies adopting composites
• Learning a new tool or programming language is hard
but learning a new problem domain is easy
– so people experienced in banking systems should
write flight control systems
• We should use techniques that are widely known
rather than ones that are appropriate
– like driving Ford Mondeos to the moon!
Hampshire BCS – February 2002
47
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
48
abc
Contributors to Lifecycle Costs
100
90
80
70
60
50
40
30
20
10
0
Cost of error correction
design
Time spent:
Hampshire BCS – February 2002
code
<50%
integrate
IV&V
>50%
49
abc
The Need to Reason
• To bring error detection forward we must be able to
reason about source code
• We can’t reason about source code unless it has a
precise meaning
• Too often the meaning of code is defined by its test
results!
Hampshire BCS – February 2002
50
abc
“… one could communicate with these machines in
any language provided it was an exact language …”
“… the system should resemble normal mathematical
procedure closely, but at the same time should be as
unambiguous as possible.”
Alan Turing, 1948
Hampshire BCS – February 2002
51
abc
What is SPARK?
• A sub-language of Ada with particular properties that
make it ideally suited to the most critical of
applications:
– Completely unambiguous
– All rule violations are detectable
– Formally defined
– Tool supported
• SPARK facilitates Correctness by Construction
Hampshire BCS – February 2002
52
abc
A disclaimer
• SPARK is not the world’s only logic
• SPARK is not a superior magic
Hampshire BCS – February 2002
53
abc
A Store Object
package Store
is
procedure Put(X : in T);
function Get return T;
end Store;
Hampshire BCS – February 2002
54
abc
A Store Object
package Store
--# own State;
is
procedure Put(X : in T);
--# global out State;
function Get return T;
--# global State;
end Store;
Hampshire BCS – February 2002
55
abc
procedure Swap(X, Y : in out T)
--# global out Store.State;
is
begin
Store.Put(X);
X := Y;
Y := Store.Get;
end Swap;
Hampshire BCS – February 2002
56
abc
SHOLIS - Ship Helicopter Operating
Limits Instrumentation System
• MOD’s first Def Stan 00-55 project
• 27,000 lines of SIL4 code
• Z specification
• SPARK code and formal proof
• Information flow analysis used to show separation of
critical and non-critical functions
• Zero defects in acceptance and trials
Hampshire BCS – February 2002
57
abc
Hampshire BCS – February 2002
58
abc
Defence Standard 00-55: Example
Requirements
• Formal specification
– must be possible to verify formally that S/W meets spec
• Formal design
– small on SHOLIS: most code proved directly against spec
• Proof obligations at all stages
– e.g. formal design against spec, code against formal design
• Resource modelling
– to show S/W meets spec of available hardware resources
• Static analysis of code
– including: control, data and information flow analysis
Hampshire BCS – February 2002
59
abc
Z Proof
• Consistency of global variables and constants
• Existence of initial states
• Checking of preconditions
– above proofs show specification is well-defined
• Key safety properties
– e.g.
In ; Calc ; Out
gives the correct warning
• Proofs constructed by “rigorous argument”
– largely manual (tool for mechanical schema expansion
etc.)
– arguments contain the main proof steps, but omit detailed
symbol manipulation a tool requires for complete proofs
Hampshire BCS – February 2002
60
abc
SHOLIS on test...
Hampshire BCS – February 2002
61
abc
Code Proof
• Full partial correctness - formally specified
“SIL4” units
• Proof of safety invariants at “main loop” level
• Freedom from exceptions - all code
• 9000 VCs from code
– 3100 for partial correctness and safety invariants
– 5900 for exception freedom
• 6800 discharged by Simplifier
Hampshire BCS – February 2002
62
Hampshire BCS – February 2002
Other
Acceptance
System Validation
Code Proof
Integration Test
Unit Test
Code
High-level Design
Z Proof
Specification
No. of Faults Found
100
80
70
60
50
0.3
40
30
20
0
Efficiency
abc
Faults found vs. Effort
0.6
90
0.5
0.4
0.2
0.1
10
0
63
Lockheed C130J
Hampshire BCS – February 2002
abc
64
abc
C130J
•
•
130,000 lines of safety related code in mission computer
Process designed to
– reduce V&V costs (and consequent delays)
– meet certification requirements, UK MoD and FAA
•
Based on rigorous specification and design
– CoRE
– Parnas tables
– SPARK
•
Lockheed benefits
– ‘Correctness by construction’
– Errors trapped early in process
– V & V during coding
– Accelerated validation & verification
– Cost reduction
Hampshire BCS – February 2002
65
Lockheed on SPARK
abc
•
Some errors immediately uncovered by formal analysis, such as conditional
initialization errors may only emerge after very extensive testing.
•
The technology for generating and discharging the proof obligations, based on the
SPARK components of Ada, was crucial, in binding the code to the initial requirements.
•
SPARK provides an extremely robust and efficient basis for formal verification.
•
The process has proven effective with typical software developers and did not
necessitate and inordinate amount of additional training.
•
Experience has shown that SPARK coding occurs at near typical Ada rates.
•
Code written in SPARK is deterministic and inherently statically analysable.
•
Very few errors have been found in the software during even the most rigorous levels of
FAA testing, which is being successfully conducted for less than a fifth of the normal
cost in industry.
•
Correctness by construction is no longer a theoretical abstraction; it is now a practical
way to develop software that exceeds its technical goals while delivering sterling
business performance.
Hampshire BCS – February 2002
66
Lockheed on SPARK
abc
•
Some errors immediately uncovered by formal analysis, such as conditional
initialization errors may only emerge after very extensive testing.
•
The technology for generating and discharging the proof obligations, based on the
SPARK components of Ada, was crucial, in binding the code to the initial requirements.
•
SPARK provides an extremely robust and efficient basis for formal verification.
•
The process has proven effective with typical software developers and did not
necessitate and inordinate amount of additional training.
•
Experience has shown that SPARK coding occurs at near typical Ada rates.
•Very
Codefew
writtenerrors
in SPARK have
is deterministic
inherentlyin
statically
analysable.
beenandfound
the software
•
Very few errors have been found in the software during even the most rigorous levels of
during
even the most rigorous levels of FAA testing,
FAA testing, which is being successfully conducted for less than a fifth of the normal
cost in industry.
which
is being successfully conducted for less than
•a fifth
Correctness
by construction
no longer
theoretical abstraction; it is now a practical
of the
normalis cost
ina industry.
way to develop software that exceeds its technical goals while delivering sterling
business performance.
Hampshire BCS – February 2002
67
abc
Aerosystems’ IV&V Conclusions
• Significant, safety-critical errors were found by static
analysis in code developed to DO-178B Level A
• Proof of SPARK code was shown to be cheaper than
other forms of semantic analysis performed
• SPARK code was found to have only 10% of the
residual errors of full Ada and Ada was found to have
only 10% of the residual errors of C
• No statistically significant difference in residual error
rate could be found between DO-178B Level A and
Level B code
Hampshire BCS – February 2002
68
abc
Agenda
• Introduction: a historical magic
• The software process
• What are “Critical Systems”?
• The need to “reason”
• Some obstacles
• Advantages of logic — project experience
• Summary
Hampshire BCS – February 2002
69
abc
“Any sufficiently advanced technology is
indistinguishable from magic."
Arthur C. Clarke, 1962
But this doesn’t mean that all magical illusions are
underpinned by advanced technology!
Hampshire BCS – February 2002
70
abc
Logic vs Magic
• Magic is attractive because life would be so much
easier if it worked!
• Magic is transient, fashion driven and ultimately futile
• Software engineering is hard because we are trying
to solve hard problems
• The engineering response to solving hard problems
is to use clever people and good mathematics
• Being the best screwdriver user in the world might
make you a mechanic but not an engineer!
Hampshire BCS – February 2002
71
abc
and finally
“Real life problems are those that
remain after you have systematically
failed to apply all the known
solutions”
Edsger Dijkstra, 1973
Hampshire BCS – February 2002
73