Formal Methods Tool Qualification for
DO178B & DO178C
Nick Tudor
tel: +44 1684 894489
email: njtudor@qinetiq.com
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Agenda
• Tool Overview and Qualification Approach
• The Current Guidance - DO178B
• Qualification Considerations – DO178B
• Applicant’s Claims – DO178B
• Qualification Considerations – DO178C
• Applicant’s Claims – DO178C and Supplements
• Summary
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Tool Overview and
Qualification Approach
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Overview of the tool
• The tool is a Formal Methods based software verification tool
− Automates the verification of automatically generated code
− Constraints include a subset of Simulink and a subset of the Ada programming
language
• Exploits the Z formal specification language
− Automatic generation of formal specification of the Simulink in Z
− Gives required independence
• Uses Carroll Morgan’s Theory of Refinement, also automatic
• Built in automated use of an off-the-shelf Theorem Prover (ProofPower)
• Having used the tool manually for independent code verification, we
aimed to:
− Automatically prove automatically generated code for productivity reasons
− Understand the tool qualification issues
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Overview of the CLawZ Process
B4S
Simulink
Z Producer
Ada
Development
Verification
User Interface
Refinement Script Generator
Z
Supertac
Compliance Notation
Tool
Verification Conditions
ProofPower
Discharge proof
www.QinetiQ.com
© Copyright QinetiQ limited 2010
The GUI and some functions (PID example)
Creating
Defining
Interface
variables
for each unit
Creating Z specification,
Importing Simulink & Ada
doing refinement
files & defining units
and proof
Proof results
www.QinetiQ.com
© Copyright QinetiQ limited 2010
The ‘pre-qualification’ approach
• As part of the final development steps, we took advice from CAA
• This is new because:
− No-one has previously approached them to do any form of “pre-qualification” of
a tool
− NB This does not constitute certification authority pre-approval of this tool
− It’s a formal methods tool
− A formal methods tool which is intended to only automate code verification [and
will be qualified as such]
• We needed to examine the guidance from DO178B to ensure that we
knew what we were aiming at and to check likely impact of DO178C
− Particularly in light of both the Formal Methods and Tool Qualification
Supplements
www.QinetiQ.com
© Copyright QinetiQ limited 2010
How high is the bar?
RTCA DO178C/EUROCAE ED12C
Tool Qualification Supplement
RTCA DO178B/EUROCAE ED12B
“Plus”
RTCA DO178B/EUROCAE ED12B
Minimum
QinetiQ Internal Processes
www.QinetiQ.com
© Copyright QinetiQ limited 2010
A virtual, parallel approach
Tool Development
Theoretical Applicant
Theoretical System
DO178B
Theoretical DO178C
Includes (theoretically):
Core text,
Tool Qualification,
FM Supplement
MBD Supplement
www.QinetiQ.com
© Copyright QinetiQ limited 2010
The Current Guidance - DO178B
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Not Shooting for the Minimum
• RTCA DO178B states that for a Verification tool:
− “The qualification criteria for software verification tools should be achieved by
demonstration that the tool complies with its Tool Operational Requirements under normal
operational conditions”
− Production of Tool Operational Requirements, the Verification Plan, the Verification Results
and the Tool Accomplishment Summary
• NB FAA-8110-49 says you could also have a TQP and TAS, but specifically:
− “Tool Operational Requirements satisfies the same objectives as the Software
Requirements Data of the airborne software”. …ie (from 12.2.3.2):
− “Tool Operational Requirements describe the tool's operational functionality. This data
should include:
− a. A description of the tool's functions and technical features. [For software development
tools, it includes the software development process activities performed by the tool].
− b.
User information, such as installation guides and user manuals.
− c.
A description of the tool's operational environment.
− d. [For software development tools, the expected responses of the tool under abnormal
operating conditions.] ”
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Adapted from FAA-8110-49 Fig 9-2
Data
Applicability Available/
Submit/Note
RTCA/DO-178B Ref.
Plan for Software Aspects of
Certification (PSAC)
Verification & Development
Submit
12.2, 12.2.3.a, & 12.2.4
Tool Qualification Plan
Development
Submit
12.2.3.a(1), 12.2.3.1, & 12.2.4
Tool Operational
Requirements
Verification & Development
Available
12.2.3.c(2) & 12.2.3.2
Software Accomplishment
Summary (SAS)
Verification & Development
Submit
12.2.4
Tool Accomplishment
Summary
Development
Submit
12.2.3.c(3) & 12.2.4
Tool Verification Records (for
example, test cases,
procedures, and results)
Verification & Development
Available
12.2.3
Tool Qualification
Development data (for
example, requirements,
design, and code)
Development
Available
12.2.3
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Qualification considerations –
DO178B
www.QinetiQ.com
© Copyright QinetiQ limited 2010
An Applicant’s perspective
• We needed to consider how an Applicant would use the tool and
subsequently be able to claim credit
− Despite confused guidance, what artefacts would need to be made available
− Given that this is a tool based upon FM, what “language” to use (this is non-trivial!)
• We needed to be cognisant of context, ie the overall software development
life cycle
• We have defined CLawZ Simulink Subset (CSS) used for Independent
Verification and used a well known subset of code (Ada)
− This constrains the possible input space to something that is well defined, but also
very flexible
− Also constrained the output space (code)
• By doing this we set the Tool Operational Requirements and hence the
scope of the qualification
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Proposed approach
Histor
y
Available information includes:
Development Plan
Configuration Management Plan
QA Plan/records (not for 178B)
Design Description
Configuration Records
Development Environment Index
User Guide
Upgrade/
production
of Plans
Configuration
Control
Tool
Operational
Requirements
Verification
Plan
Configuration
Index
Tool
Qualification
Plan
Applicants
PSAC
Tool
Accomplishment
Summary
Applicants
SAS
Verification
Cases and
Procedures
Verification
Results
QA and Technical Audit
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Applicant’s Claims – DO178B
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Verification Objectives
– DO178B
System
Requirements
A-3.2 Accuracy & Consistency
A-3.3 HW Compatibility
A-3.4 Verifiability
A-3.5 Conformance
A-3.7 Algorithm Accuracy
A-3.1 Compliance
A-3.6 Traceability
(A-2: 1, 2)
High-Level
Requirements
A-6.1 Compliance
A-6.2 Robustness
NB Verification
of the Verification,
QA, DER liaison
and other
documentation also
required
A-4.1 Compliance
A-4.6 Traceability
A-4. 8 Architecture Compatibility
(A-2: 3, 4, 5)
A-4.9 Consistency
A-4.10 HW Compatibility
A-4.11 Verifiability
A-4.12 Conformance
A-4.13 Partition Integrity
Software
Architecture
Low-Level
Requirements
A-5.2 Compliance
(A-2: 6)
A-5.3 Verifiability
A-5.4 Conformance
A-5.6 Accuracy & Consistency
A-4.2 Accuracy & Consistency
A-4.3 HW Compatibility
A-4.4 Verifiability
A-4.5 Conformance
A-4.7 Algorithm Accuracy
A-5.1 Compliance
A-5.5 Traceability
Source Code
A-6.3 Compliance
A-6.4 Robustness
(A-2: 7)
A-5. 7 Complete & Correct
Executable
Object Code
A-6.5 Compatible With Target
Compliance: with requirement
Conformance: with standards
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Claims relating to source code verification Table A-5
Objective
Description
Applicability by
Software Level
Ref.
A
B
C
1
Source Code complies with low-level
requirements.
6.3.4a
l
l
m
2
Source Code complies with software
architecture.
6.3.4b
l
m
m
3
Source Code is verifiable.
6.3.4c
m
m
4
Source Code conforms to standards.
6.3.4d
m
m
m
5
Source Code is traceable to low-level
requirements.
6.3.4e
m
m
m
6
Source Code is accurate and
consistent.
6.3.4f
l
m
m
7
Output of software integration process
is complete and correct.
6.3.5
m
m
m
D
Partial
We can show that eg there are no uninitialised or unused variables or constants.
There are no claims re stack usage, resource contention, WCET, etc
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Verification Objectives
System
Requirements
Major Contribution
Simulink [Continuous]
(A-2: 1, 2)
High-Level
Requirements
Simulink [Discrete]
(A-2: 3, 4, 5)
Software
Architecture
Low-Level
Requirements
Simulink for CLawZ
A-5.2 Compliance
(A-2: 6)
A-5.3 Verifiability
A-5.4 Conformance
A-5.6 Accuracy & Consistency
A-5.1 Compliance
A-5.5 Traceability
Source Code
Autocode, proof
(A-2: 7)
A-5. 7 Complete & Correct
Executable
Object Code
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Qualification considerations –
DO178C
www.QinetiQ.com
© Copyright QinetiQ limited 2010
DO178C introduces 3 categories of tools (see section12.2.2)
DO178B
Development tool
?
Verification tool
DO178C
Criteria 1
Criteria 2
Criteria 3
DAL
Criteria 1
Criteria 2
Criteria 3
A
TQL1
TQL4
TQL5
B
TQL2
TQL4
TQL5
C
TQL3
TQL5
TQL5
D
TQL4
TQL5
TQL5
Criteria
2: A1:tool
that whose
automates
verification
process(es)
and thus
Criteria
A tool
output
is part of
the airborne
could
fail to detect
an error,
whose
output is used to justify the
software
and thus
couldand
insert
an error.
elimination or reduction of:
Criteria 3: A tool that, within the scope of its intended use,
1. Verification process(es) other than that automated by the tool, or
fail to detect
an error.
2. could
Development
process(es)
that could have an impact on
the airborne software.
www.QinetiQ.com
© Copyright QinetiQ limited 2010
DO178C Tool Qualification Supplement
• We wanted to know what we might be under ED12C/DO178C we
we’re either
− A verification tool (Criteria 3) and hence TQL5 would be required, or
− A super verification tool (Criteria 2) and hence TQL5 or even TQL4
• We believe CLawZ would be a TQL5 tool, but this depends largely on
what the Applicant will wish claim as a result of using the tool
• And whether this is acceptable for the certifier
• More on this at the end…
• We will also examine TQL4 …just in case!
www.QinetiQ.com
© Copyright QinetiQ limited 2010
DO178C TQL 4 – Tables T-1 & T-2
• We needed to keep and eye on what TQL4 required at the same time
• Referring to Table T-1 and T2, there is considerable extra material required
over DO178B
Document
178B
TQL4
Note
The Tool Qualification Plan (TQP)
Y
Y
Tool Development Plan (TDP)
N
Y
Not strictly needed, but certifier will look for it
Tool Verification Plan (TVP)
Y
Y
Not entirely clear if this is needed, but…
Tool Configuration Management Plan
(TCMP)
N
Y
Not entirely clear if this is needed, but the
Configuration Index is required
Tool Quality Assurance Plan (TQAP)
N
Y
Not needed, but needed for eg TickIT?
• No requirement for Tool Operational Requirements, but is implicit
• Interesting that QA Plan appears to be required, but not QA records
• Also, that Tool Plans do not have to comply with the Supplement (T-1-6)
www.QinetiQ.com
© Copyright QinetiQ limited 2010
TQL4 – DO178C – Tables T-3, T-4 and T-5
• There is large concentration of effort in Table T-3 (requirements) which
is not “needed” under DO178B
− It is implied that the Tool Operational Requirements are the “requirements”.
• Suspect that in practice, for all but trivial tools under qualified under
DO178B, there is a more clearly defined set of requirements contained
within the TOR
• It was not immediately clear why the only requirement for Table T4 is T4-10 (protection mechanisms). Sought guidance from FAQ #2 Tool
Qualification Supplement :
− Seeks to show evidence for separation of functions, essentially through
architecture – fine, but…
− There are no other architecture requirements for TQL 4 (ie compatible with
requirements, consistency, standards), so does this objective help safety?
• No objectives for Table T-5 verification of the coding process…
www.QinetiQ.com
© Copyright QinetiQ limited 2010
TQL4 - Tables T-4 and T-5 - A closer examination
• From DP #2: Rationale for Tool Qualification Criteria Definition, para 2.2.3.2 Rationale for introducing Criteria 2
“These additional uses of tools raise some concerns about the rigor required for qualification. For example:
− From a safety perspective, the impact of these tools on the final software can vary.
− The required confidence may not be able to be assessed by considering only the Tool Operational Requirements.
− The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required
confidence
• The major concern would appear to be the implementation, so no examination of low level requirements or the source
code?
• Reliance is therefore left to Table T-6…
• NB No objectives are required to be met for TQL5
www.QinetiQ.com
© Copyright QinetiQ limited 2010
TQL4 – ED12C/DO178C – Tables T-6, T-7, T-8 and T-9
• Focus here is on how the tool functions when actually undertaking
verification
• Examines object code with respect to the Tool Requirements
− Checks compliance and robustness
• Also looks at verification results
− Checks that results were correct/discrepancies explained and
− Coverage of Tool Requirements
• Ensures that the configuration was identified
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Applicant’s Claims – DO178C and
Supplements
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Possible claims relating to source code verification –
DO178C FM Supplement
• Specific claims for formal verification could be made in accordance with
the Formal Methods Supplement (see table FM A-5)
− Specific claims can be made FM A5-1 to 6 (FM7 refers to core text as above)
− Also need to examine possible to claims to Extra Objectives FM8-11
Objective
Description
Applicability by
SW Level
Ref.
A
B
C
FM 8
Formal analysis cases and
procedures are correct.
FM.6.3.6a
FM.6.3.6b
l
m
m
FM 9
Formal analysis results are correct
and discrepancies explained.
FM.6.3.6c
l
m
m
FM 10
Requirements formalization is
correct.
FM.6.3i
l
l
m
FM 11
Formal method is correctly defined
and sound.
FM.6.2
l
m
m
D
m
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Perspective of OTS tools for DO178C vs
in-house tool development for DO178B
• Number of views from Tool Qualification Supplement
− Re-use – see section 11.2
− OTS - see section 11.3
− Service History – see section 11.4 (not applicable)
− Alternative methods – see section 11.5 (not applicable)
• Section 11.3.2 shows activities required by the tool developer
− Which are to produce: TOR, TQP, TCI, TAS
− Also outlines which objectives are modified
• Section 11.3.3 shows activities for the tool user
− To assess the TOR, TQP, TCI and TAS and plan extra activities such as division
of responsibilities
www.QinetiQ.com
© Copyright QinetiQ limited 2010
Summary
• We believe we can support an Applicant in system certification to
DO178B
• We have gone further than was required because this tool was “predeveloped” and is a formal methods tool
• We believe that it would be a TQL5 tool under DO178C Tool Qualification
Supplement, but…
• Could be TQL4, depending on development processes of the Applicant
• Interesting that our approach has mirrored the COTS advice to tool
qualification
• So does TQL4 achieve anything extra in terms of system safety?
www.QinetiQ.com
© Copyright QinetiQ limited 2010