Introduction to Safety Cases Tim Kelly E-mail: tim.kelly@cs.york.ac.uk High Integrity Systems Engineering Group Department of Computer Science Copyright © 2011, Tim Kelly Overview Historical Context Definitions Role of Argument & Evidence Safety Case Reports Safety Arguments Copyright © 2011, Tim Kelly Safety Cases - 2 A Brief History of Safety Cases Number of serious accidents, e.g. Windscale Nuclear Accident (late 1950s) Piper Alpha Off-shore Oil and Gas Platform Disaster (1990s) Clapham Rail Disaster (1990s) Prompted reconsideration of how safety is managed in the safety-critical sector Industries were not ignorant of safety Safety standards existed – but often based on prescriptive codes What Was Missing: Systematic and thorough consideration of safety, and communication of this to a regulator Prescription Designers / operators claim safety through satisfaction of the regulator’s requirements ‘Goal-based’ standards Up to the designers / operators to demonstrate that they have an adequate argument of safety in support of high level objectives (e.g. ALARP) Copyright © 2011, Tim Kelly Safety Cases - 3 Motivation for Safety Cases Completeness – hard to judge … … when evidence is distributed and diverse … when arguments are implicit Rationale behind prescriptive requirements missing Knowledge Imbalance – developers know more about their products than the regulators Some existing forms of assurance are increasingly considered too indirect (e.g. software) Copyright © 2011, Tim Kelly Safety Cases - 4 Fault Tree Analysis Example 1 Unannunciated Loss of All Wheel Braking LSSALLWBP3 5.00E-07 Requirement is per 5 hour flight Loss of All Wheel Braking Loss of Annunciation Capability LOSSALLWB NOANCREDIT 5.00E-07 Requirement is per 5 hour flight PROB: 1.00E+00 (0) No Credit Taken Normal Brake System Does Not Operate Alternate Brake System Does Not Operate Emergency Brake System Does Not Operate LOSSNORMP3 LSSALTERP3 LSSEMERGP3 PROB: 5.00E-03 (0) Budgeted Prob. PROB: 1.00E+00 (0) No Credit Taken Copyright © 2011, Tim Kelly Loss of Green Hydraulic Supply Loss of Normal Brake Loss of BSCU Ability System HydraulicSafety Cases - 5 to Command Braking Components LSSGRHYDP3 LSSNRMBRP3 PROB: 3.30E-05 (0) Budgeted Prob. LSSOF2BSCU 3.30E-05 (0) Redundant BSCUs Required to Meet Budgeted Probability PROB: 3.30E-05 (0) Budgeted Prob. BSCU Fault Causes Loss of Braking Commands Loss of Aircraft Electrical Power to Both BSCU Systems BOTHBSCUSF LBSCUPWRP3 Fault Tree Analysis Example 1 PROB: 1.00E-07 (0) Historical Data 3.30E-05 BSCU 1 Failure Causes Loss of Unannunciated Loss of All Wheel Braking Commands Braking BSCU 2 Failure Causes Loss of Braking Commands BSCU1FAILS BSCU2FAILS LSSALLWBP3 PROB: 5.75E-03 (1) PROB: 5.75E-03 RATE: 1.15E-03 5.00E-07 /H RATE: 1.15E-03 EXPO: 5 H Requirement is per EXPO: 5 Budgeted F.R. 5 hour flight Budgeted F.R. Loss of All Wheel Braking Loss of Annunciation Capability LOSSALLWB NOANCREDIT 5.00E-07 Requirement is per 5 hour flight (1) /H H PROB: 1.00E+00 (0) No Credit Taken Normal Brake System Does Not Operate Alternate Brake System Does Not Operate Emergency Brake System Does Not Operate LOSSNORMP3 LSSALTERP3 LSSEMERGP3 PROB: 5.00E-03 (0) Budgeted Prob. PROB: 1.00E+00 (0) No Credit Taken Copyright © 2011, Tim Kelly Loss of Green Hydraulic Supply Loss of Normal Brake Loss of BSCU Ability System HydraulicSafety Cases - 6 to Command Braking Components LSSGRHYDP3 PROB: 3.30E-05 (0) Budgeted Prob. LSSNRMBRP3 LSSOF2BSCU PROB: 3.30E-05 (0) Budgeted Prob. 3.30E-05 (0) Redundant BSCUs Required to Meet Budgeted Probability BSCU Fault Causes Loss of Braking Commands Loss of Aircraft Electrical Power to Both BSCU Systems BOTHBSCUSF LBSCUPWRP3 Loss of All Wheel Braking Annunciation Capability LOSSALLWB NOANCREDIT 5.00E-07 Requirement is per 5 hour flight PROB: 1.00E+00 (0) No Credit Taken Normal Brake System Does Not Operate Alternate Brake System Does Not Operate Emergency Brake System Does Not Operate LOSSNORMP3 LSSALTERP3 LSSEMERGP3 Fault Tree Analysis Example 2 PROB: 5.00E-03 (0) Budgeted Prob. PROB: 1.00E+00 (0) No Credit Taken Loss of Green Hydraulic Supply Loss of Normal Brake System Hydraulic Components Loss of BSCU Ability to Command Braking LSSGRHYDP3 LSSNRMBRP3 LSSOF2BSCU PROB: 3.30E-05 (0) Budgeted Prob. 3.30E-05 (0) Redundant BSCUs Required to Meet Budgeted Probability PROB: 3.30E-05 (0) Budgeted Prob. BSCU Fault Causes Loss of Braking Commands Loss of Aircraft Electrical Power to Both BSCU Systems BOTHBSCUSF LBSCUPWRP3 PROB: 1.00E-07 (0) Historical Data 3.30E-05 Unannunciated BSCU 1 Failure Loss of All Wheel Causes Loss of Braking Braking Commands BSCU 2 Failure Causes Loss of Braking Commands BSCU1FAILS LSSALLWBP3 BSCU2FAILS PROB: 5.75E-035.00E-07 (1) is per RATE: 1.15E-03 Requirement /H 5 hour EXPO: 5 H flight Budgeted F.R. Loss of Loss of All Safety Cases - 7 Annunciation Wheel Braking Capability LOSSALLWB PROB: 5.75E-03 RATE: 1.15E-03 EXPO: 5 Budgeted F.R. (1) /H H Copyright © 2011, Tim Kelly NOANCREDIT 5.00E-07 Requirement is per 5 hour flight PROB: 1.00E+00 (0) No Credit Taken Normal Brake System Does Not Operate Alternate Brake System Does Not Operate Emergency Brake System Does Not Operate LOSSNORMP3 LSSALTERP3 LSSEMERGP3 Fault Tree Analysis Example 2 PROB: 5.00E-03 (0) Budgeted Prob. PROB: 1.00E+00 (0) No Credit Taken Loss of Green Hydraulic Supply Loss of Normal Brake System Hydraulic Components Loss of BSCU Ability to Command Braking LSSGRHYDP3 LSSNRMBRP3 LSSOF2BSCU PROB: 3.30E-05 (0) Budgeted Prob. 3.30E-05 (0) Redundant BSCUs Required to Meet Budgeted Probability PROB: 3.30E-05 (0) Budgeted Prob. BSCU Fault Causes Loss of Braking Commands Loss of Aircraft Electrical Power to Both BSCU Systems BOTHBSCUSF LBSCUPWRP3 PROB: 1.00E-07 (0) Historical Data 3.30E-05 BSCU 1 Failure Causes Loss of Braking Commands BSCU 2 Failure Causes Loss of Braking Commands BSCU1FAILS BSCU2FAILS PROB: 5.75E-03 RATE: 1.15E-03 EXPO: 5 Budgeted F.R. (1) /H H Safety Cases - 8 PROB: 5.75E-03 RATE: 1.15E-03 EXPO: 5 Budgeted F.R. (1) /H H Copyright © 2011, Tim Kelly (Further) Motivation for Safety Cases Completeness – hard to judge … … when evidence is distributed and diverse … when arguments are implicit Rationale behind prescriptive requirements missing Knowledge Imbalance – developers know more about their products than the regulators Some existing forms of assurance are too indirect The role of evidence can otherwise be unclear The assumptions and implicit judgements in evidence need to be presented explicitly and argued Copyright © 2011, Tim Kelly Safety Cases - 9 The Purpose of a Safety Case Principal Objective: safety case presents the argument that a system will be acceptably safe in a given context ‘system’ could be ... physical (e.g. aero-engines, reactor protection systems) procedural (e.g. railway operations, off-shore) Software (in a system context) In practice: often series of safety cases produced — stages of development and/or operation safety cases are large, complex, technical and political documents Copyright © 2011, Tim Kelly Safety Cases - 10 Some Safety Case Definitions "A safety case is a comprehensive and structured set of safety documentation which is aimed to ensure that the safety of a specific vessel or equipment can be demonstrated by reference to: safety arrangements and organisation safety analyses compliance with the standards and best practice acceptance tests audits inspections feedback provision made for safe use including emergency arrangements” (JSP 430 Issue 1) "A Safety Case is a structured argument, supported by a body of evidence, that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given operating environment." (DS 00-56 Issue 4) Copyright © 2011, Tim Kelly Safety Cases - 11 Argument & Evidence A safety case requires two elements: Supporting Evidence Results of observing, analysing, testing, simulating and estimating the properties of a system that provide the fundamental information from which safety can be inferred High Level Argument Explanation of how the available evidence can be reasonably interpreted as indicating acceptable safety – usually by demonstrating compliance with requirements, sufficient mitigation / avoidance of hazards etc Argument without Evidence is unfounded Evidence without Argument is unexplained Copyright © 2011, Tim Kelly Safety Cases - 12 Safety Cases vs. Safety Case Reports The Safety Case is the totality of the safety justification + all the supporting material: testing reports, validation reports, relevant design information etc The Safety Case Report is the document that summarises all the key components of the Safety Case and references all supporting documentation in a clear and concise format Copyright © 2011, Tim Kelly Safety Cases - 13 Safety Case Reports Exact contents depends on regulatory environment The following are key elements of most standards: scope system description system hazards safety requirements risk assessment hazard control / risk reduction measures safety analysis / test safety management system development process justification conclusions Copyright © 2011, Tim Kelly Safety Cases - 14 Safety Arguments Safety Case is NOT just a collection of disparate pieces of information Safety Argument should form the ‘spine’ of the Safety Case showing how these elements are related and combined to provide assurance of safety within the limits defined [Scope], the system [System Description] is SAFE because all identified hazards [System Hazards] and requirements [Safety Requirements] have been addressed. Hazards have been sufficiently controlled and mitigated [Hazard Control / Risk Reduction Measures] according to the safety risk posed [Risk Assessment]. Evidence [Safety Analysis / Test] is provided that demonstrates the effectiveness and sufficiency of these measures. Appropriate roles, responsibilities and methods were defined throughout the development of this system [Development Process Justification] [Safety Management System] and defined future operation Copyright © 2011, Tim Kelly Safety Cases - 15 Presenting Clear Arguments Basic argument structure claim – what we want to show argument – why we believe the claim is met, based on evidence – test results, analysis results, etc. In general, argument broken down hierarchically claim, argument, sub-claims, sub-arguments, evidence easy to show graphically, although can be done in document structure (sub-section numbering, etc.) In practice, other concepts useful e.g. context to claims, assumptions Copyright © 2011, Tim Kelly Safety Cases - 16 The Goal Structuring Notation Purpose of a Goal Structure To show how goals are broken down into sub-goals, and eventually supported by evidence (solutions) whilst making clear the strategies adopted, the rationale for the approach (assumptions, justifications) •A/J and the context in which goals are stated Copyright © 2011, Tim Kelly Safety Cases - 17 A Simple Goal Structure Control System is Safe Hazards Identified from FHA (Ref Y) Tolerability targets (Ref Z) 1x10 -6 p.a. limit for Catastrophic Hazards H1 has been eliminated All identified hazards eliminated / sufficiently mitigated Probability of H2 occurring < 1 x 10 -6 per annum Probability of H3 occurring < 1 x 10 -3 per annum Software developed to I.L. appropriate to hazards involved Primary Protection System developed to I.L. 4 I.L. Process Guidelines defined by Ref X. Secondary Protection System developed to I.L. 2 J Formal Verification Fault Tree Analysis Process Evidence of I.L. 4 Process Evidence of I.L. 2 Copyright © 2011, Tim Kelly Safety Cases - 18 A Simple Goal Structure Safety Requirements & Objectives Control System is Safe Hazards Identified from FHA (Ref Y) Tolerability targets (Ref Z) All identified hazards eliminated / sufficiently mitigated Software developed to I.L. appropriate to hazards involved I.L. Process Guidelines defined by Ref X. Safety Argument 1x10 -6 p.a. limit for Catastrophic Hazards H1 has been eliminated Probability of H2 occurring < 1 x 10 -6 per annum Probability of H3 occurring < 1 x 10 -3 per annum Primary Protection System developed to I.L. 4 Secondary Protection System developed to I.L. 2 J Formal Verification Process Evidence of I.L. 4 Safety Evidence Fault Tree Analysis Process Evidence of I.L. 2 Copyright © 2011, Tim Kelly Safety Cases - 19 A Simple Goal Structure Control System is Safe Hazards Identified from FHA (Ref Y) Tolerability targets (Ref Z) 1x10 -6 p.a. limit for Catastrophic Hazards H1 has been eliminated All identified hazards eliminated / sufficiently mitigated Probability of H2 occurring < 1 x 10 -6 per annum Probability of H3 occurring < 1 x 10 -3 per annum Software developed to I.L. appropriate to hazards involved Primary Protection System developed to I.L. 4 I.L. Process Guidelines defined by Ref X. Secondary Protection System developed to I.L. 2 J Formal Verification Fault Tree Analysis Process Evidence of I.L. 4 Process Evidence of I.L. 2 Copyright © 2011, Tim Kelly Safety Cases - 20 Summary Production of a Safety Case is a key objective of all the safety lifecycle activities The objective of the Safety Case is to ‘pull together’ many forms of information and present a coherent argument of safety However, safety cases are not just documents Clear arguments essential for safety case approach Copyright © 2011, Tim Kelly Safety Cases - 21