@ - Presentation Caveat The following presentation was made by Marv Nuss of Nuss Sustainment Solutions at the 2013 Aircraft Airworthiness and Sustainment Conference – Australia. The presentation title is: An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach. Mr. Nuss describes the USA Federal Aviation Administration’s (FAA) approach to risk assessment and risk management for aircraft continued operational safety. Mr. Nuss, a retired FAA certification engineer, explains the FAA’s Transport Airplane Risk Assessment Methodology (TARAM) published in FAA document PS-ANM-25-05. He uses the 2013 Boeing 787 lithium-ion battery failures as an example. The presentation contains facts from publicly available documents, including published USA’s National Transportation Safety Board (NTSB) reports and newspaper reports. Mr. Nuss received no information from the FAA or Boeing for the presentation. The assumptions made for the example risk assessment using the TARAM method are those of Mr. Nuss, and are based only on publically available information. Mr. Nuss believes his assumptions to be realistic and a good example of the TARAM method. However, it is completely an “outsider’s” perspective. Mr. Nuss believes quantitative risk assessment is an effective means to manage risk. Concepts similar to those explained in the presentation provide valuable tools for continued operational safety managers – whether a regulator, manufacturer, or operator. The presentation is intended to demonstrate how such a risk assessment works, not as a critique of either the FAA or Boeing. The FAA’s risk assessment and management methods continue to evolve. Mr. Nuss understands that the FAA’s TARAM document used for the example in this presentation is likely to be revised with updated policy at some point in the future. Mr. Nuss asks the reader to view the presentation with the understanding of his outsider’s perspective. He also hopes the reader finds it informative and provides a better understanding of quantitative risk assessment and risk management. An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment Conference (Australia) Brisbane, QLD July 23-25, 2013 Marv Nuss 1 + 913-962-4683 marv.nuss@marvnuss.com www.marvnuss.com An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach • • • • © 2013 NuSS Background FAA approach Definitions Case study An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 1 RA&RM Background • Managing risk is the bottom line task for any enterprise Risk, one definition: the existence of a potential for an undesirable loss Undesirable loss could be: Life Health Money Time Customers • In the aerospace industry, these risks can be competing Safety risk vs. mission readiness risk Safety risk vs. design risk Safety risk vs. financial risk Financial risk vs. mission readiness risk RA&RM: Risk Assessment and Risk Management © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 2 RA&RM Background • In order to manage risk it must first be quantified Risk assessment is a means of quantifying risk Useful data are necessary for sensible risk assessment Computer computational speed enables easier risk assessments Ability to collect and store data Ability to analyze large amounts of data • Increased pressure to make processes more efficient puts increased pressure on understanding associated risks © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 3 Big Picture Perspective on Life Risk 1.0E-04 Professional Rodeo (Bull, Bronc) Motorcycle Individual Death Probability 1.0E-05 Probability, per hour Female Probability of Death - All Causes Male Probability of Death - All Causes 1.0E-06 Passenger Vehicle Individual Death Probability TARAM Individual Risk Threshold 1.0E-07 Average Accidental Death Probability Drowning in Bathtub Individual Death Probability Passenger Bus Individual Death Probability Commercial Airplane (part 121) Individual Death Probability 1.0E-08 0 © 2013 NuSS 10 20 30 40 50 60 70 80 90 Age An Outsider’s Perspective on FAA’s Aircraft Safety Risk(years) Assessment & Risk Management Approach Ref: FAA TARAM Handbook 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 4 www.marvnuss.com FAA’s Disciplined Approach to Risk Management • FAA established its “Safety Risk Management Policy” in 1998: “The FAA shall use a formal, disciplined, and documented decision-making process to address safety risks in relation to high-consequence decisions affecting the complete life cycle.” (FAA Order 8040.4, revised to 8040.4A in 2012) • FAA Aviation Safety organization followed with its Safety Management System (SMS) FAA Aircraft Certification Service SMS includes specific policies regarding risk assessment and management “Monitor Safety/Analyze Data” (MSAD) is FAA’s Aircraft Certification Service’s process to manage risk: “designed to promote data-driven, risk-based continued operational safety decision-making.” The MSAD process is documented in FAA Order 8110.107A (2012) © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 5 FAA MSAD Risk Management Process MSAD process relies heavily in-service data. Service difficulty reporting (SDR) is the major source of in-service data. Ref: FAA Order 8110.107A © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 6 FAA MSAD Risk Management Process • MSAD is an attempt to standardize the safety risk associated with any failure Mechanical systems Electrical systems Engine and fuel systems Structure • Each directorate developed specific process for risk assessment and management Transport Directorate: TARAM -- Transport Airplane Risk Assessment Methodology, published Nov. 2011 (PS-ANM-25-05) Small Airplane Directorate: SARA – Small Airplane Risk Analysis Rotorcraft and Engine/Propeller Directorates have similar processes © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 7 FAA Risk Analysis Specification Process Statistical Exposure (Time, Number of Flights) Probability (Exposure x Frequency of Occurrence) Quantitative © 2013 NuSS Frequency of Occurrence (Failure Rate, Event Rate) (Number of Occurrences per Hour, per Flight) Conditional Probability (Event to Severity) Risk (Expected Probability of an Fatal Accident) Severity (Fatal Accident) Measurable An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 8 FAA Safety Risk Definitions Severity definitions • Catastrophic: Multiple fatalities (or fatality to all on board) usually with the loss of aircraft • Hazardous: Multiple serious injuries; fatal injury to a relatively small number of persons (one or two) or a hull loss without fatalities • Major: Physical distress or injuries to persons and/or substantial damage to aircraft • Minor: Physical discomfort to persons and/or slight damage to aircraft • Minimal: Negligible safety effect Likelihood definitions • Extremely improbable: So unlikely that it is not expected to occur, but it is not impossible • Extremely remote: Expected to occur rarely • Remote: Expected to occur infrequently • Probable: Expected to occur often Ref: FAA Order 8040.4A • Frequent: Expected to occur routinely © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 9 FAA Risk Assessment and Management Definitions Causes: Underlying circumstances, occurrences, and/or failures that contribute, or could contribute, directly or indirectly, to an event. Corrected risk: Residual risk that remains after corrective action is taken. When highly effective corrective action is taken, residual risk is considered to be zero. Corrective action: Any action to mitigate a safety issue. Fleet: Aircraft, engine or propeller products of a type currently in service affected by a certain safety issue. Hazard: Any existing or potential condition that can lead to injury, illness or death to people; damage to or loss of a system, equipment or property or damage to the environment. A hazard is a condition that is a prerequisite to an accident or incident. Probability: Ratio of the number of actual occurrences to the number of possible occurrences. Risk: Expression of the severity and probability of an undesired event. Severity: The consequence or impact of a hazard in terms of degree or loss or harm. Uncorrected risk: Risk that accumulates over time in the affected fleet if no corrective action is taken for a certain safety issue. Ref: FAA Order 8110.107A © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 10 FAA Risk Assessment Definitions Conditional Probability (CP): the probability that an unsafe outcome, for which an injury ratio is known, will result from a particular condition under study. The conditional probability is the product of the individual conditional probabilities for all of the conditions that must occur, after the condition under study, to result in the defined unsafe outcome. Defect Airplanes(s) (DA): the predicted number of airplanes that would have the subject failure if the condition under study is left undetected during the timeframe being analyzed. Exposed Occupants (EO): the average number of persons expected to be exposed to fatal injury during an unsafe outcome or condition. Frequency of Occurrence (F): the rate at which the condition under study is expected to manifest itself within the affected fleet or sub fleet. For non-constant failure rates, such as wear-out failures, Weibull and log-normal analysis techniques are helpful in determining the distribution of failures over time. Hazard Function (h(t)): the instantaneous failure rate of a unit. Hazard function is analogous to the frequency of occurrence. Ref: PS-ANM-25-05 FAA TARAM Handbook © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 11 FAA Risk Assessment Definitions (con’t) Injury Ratio (IR): the average single-event probability that those exposed to a particular condition or outcome will suffer fatal injury. Utilization (U): the airplane’s flight hours or flight cycles per defined-unit time period (e.g., per day). Not Detected (ND): the probability that an occurrence of a defect will not be detected before the defect leads to an unsafe condition or outcome. ND is a conditional probability, but it is defined separately because of its importance in certain wear-out issues, such as structural fatigue. Number of Aircraft (Σ): the number of airplanes in the affected fleet during the time period under study. Time Period (T): the time period over which risk is calculated. See Paragraph 4.6 for specific guidance on time period determination. Ref: PS-ANM-25-05 FAA TARAM Handbook © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 12 Components of Fleet Risk (Total Uncorrected Fleet Risk) FREQUENCY of OCCURRENCE (Failure Rate / Event Rate per Hour / Flight / Cycle) OUTCOME SEVERITY RATE FLEET EXPOSURE (Time, Flights, Cycles ) NUMBER OF OCCURRENCES (Fleet Exposure and Frequency of Occurrence) -OR- (90-Day and Control Program Fleet Risk) EXPOSED OCCUPANTS (Passengers and Crew) (Injury Ratio) OUTCOME SEVERITY RATE (Injury Ratio) CONDITIONAL PROBABILITY SEVERITY (Probability of Outcome Given the Occurrence) (Weighted Events or Number of Expected Fatalities per Outcome) Risk (Probability of Fatality over Time or Number of Fatalities over Time ) Components of Individual Risk FREQUENCY of OCCURRENCE (Failure Rate / Event Rate per Hour / Flight / Cycle) CONDITIONAL PROBABILITY (Probability of Outcome Given the Occurrence) OUTCOME SEVERITY RATE (Injury Ratio) Risk (Individual Probability of Fatal Injury per Flight Hour) © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach Ref: FAA TARAM Handbook 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 13 www.marvnuss.com TARAM Risk Assessment Equations • • • • • Fleet exposure: U x T x Σ Predicted number of Occurrences: U x T x Σ x F Severity: S = IR Fleet Risk: R = (U x T x Σ x F) x CP x S Individual risk: R = F x CP x S Individual risk: Probability of individual fatal injury per flight hour Variants of these equations are used to determine: • Uncorrected and corrected risk (constant rate or wear-out) • Short term risk U: Utilization T: Time Period Σ: Number of Aircraft F: Frequency of Occurrence S: Severity IR: Injury Ratio R: Risk CP: Conditional Probability TARAM: Transport Airplane risk Assessment Methodology TARAM Handbook An Outsider’s Perspective on FAA’s Aircraft SafetyRef: Risk PS-ANM-25-05 Assessment & RiskFAA Management Approach © 2013 NuSS 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 14 Risk Assessment Case Study: B-787 Battery Fires Was the FAA justified in grounding the fleet? Information sources: • NTSB Interim Factual Report for NTSB No. DCA13IA037, March 7, 2013 • FAA special conditions No. 25-359-SC, Federal Register October 11, 2007, page 57842 • Wall Street Journal (WSJ), various dates • Aerospace Defense News (ASDNews), various dates An Outsider’s&Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach © 2013 NuSS 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 15 B-787 Battery Fires Information Timeline Jan. 7, 2013: APU battery event in Boston (JAL) Jan. 16, 2013: main battery event in-flight in Japan (ANA) Jan. 16, 2013: FAA issued emergency AD 2013-02-51 that grounds the fleet Pertinent incident information JAL: heavy smoke and fire coming from front of battery case (3” flames) JAL: “intense” smoke in cabin ANA: burning smell and smoke in the cabin, battery malfunction similar to JAL Fleet history as of Jan. 16, 2013 50 airplanes in service Total accumulation: 51,662 hours, 18665 cycles Certification information Battery vent and/or smoke (without fire) classified Hazardous (Extremely remote, 1 x 10-7/hour) Battery vents smoke/fire classified Catastrophic (Extremely improbable, 1 x 10-9/hour) © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 16 JAL JA829J APU 787-8 Battery Fire Photos Forward face of the battery case APU installation location with battery removed Ref: NTSB #DCA13IA037 Interim Factual Report, 3/7/13 Side face of the battery case © 2013 NuSS Battery case with sides folded down An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 17 ANA 787-8 Main Battery Fire Photo View into the battery case Ref: WSJ © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 18 TARAM Risk Assessment Frequency of Occurrence (F) = Occurrences/Fleet Exposure: F = 2/52,000 hrs. 4x10-5/hr. (400 times greater than cert (1x10-7)) A comparison example: • Fleet of 500 x 50 hrs/week x 50 weeks 1.25M hrs./year (.125x10-7) • Certificated expectation of hazardous battery vent and/or smoke (without fire) no more often than once/ 8 years for this example TARAM: Transport Airplane risk Assessment Methodology © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 19 TARAM Risk Assessment Conditional Probability (CP): the probability that an unsafe outcome, for which an injury ratio is known, will result from a particular condition under study. Per TARAM guidance, assume next failure results in a more severe outcome • First two battery failures could be classed as “Major” or “Minor” • Assume next failure either “Catastrophic” or “Hazardous” • CP = .333 (2 failures + next failure (with assumed higher severity) = 3 occurrences, one of which is an unsafe outcome) Severity (S): The consequence or impact of a hazard in terms of degree or loss or harm. Severity definitions • Catastrophic: Multiple fatalities (or fatality to all on board) usually with the loss of aircraft • Hazardous: Multiple serious injuries; fatal injury to a relatively small number of persons (one or two) or a hull loss without fatalities • Major: Physical distress or injuries to persons and/or substantial damage to aircraft • Minor: Physical discomfort to persons and/or slight damage to aircraft • Minimal: Negligible safety effect Ref: FAA TARAM Handbook, FAA Order 8040.4A © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 20 TARAM Risk Assessment Injury Ratio (IR): the average single-event probability that those exposed to a particular condition or outcome will suffer fatal injury. (IR = S) Per TARAM guidance example, IR = 0.16 for in-flight fire • Individual risk: R = F x CP x S • R = (4 x 10-5/hr. ) x 0.333 x 0.16 • R 2 x 10-6/hr. • TARAM guideline for normally accepted individual risk is R 1 x 10-7/hr. For the scenario of a battery failure causing an individual fatality due to a fire, the risk is 20 times higher than the TARAM guideline. This would justify the FAA’s AD. Ref: PS-ANM-25-05 FAA TARAM Handbook © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 21 TARAM Risk Assessment Fleet risk: R = (U x T x Σ x F) x CP x S R = (5 x 365 x 20 x 1000 x 4x10-5) x 0.333 x 0.16 80 Hrs./day Days/yr. Yrs. No. A/C Freq. CP S = IR TARAM guideline for normally accepted fleet risk is R 0.02 Note: Hrs./day and years are probably low estimates Ref: PS-ANM-25-05 FAA TARAM Handbook © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 22 TARAM Risk Assessment Fleet risk: R = (U x T x Σ x F) x CP x S R = (5 x 365 x 20 x 1000 x 4x10-5) x 0.333 x 0.16 80 Hrs./day Days/yr. Yrs. No. A/C Freq. CP S = IR TARAM guideline for normally accepted fleet risk is R 0.02 90 day fleet risk: R = (U90x T x Σ90 x F) x CP x IR x EO R = (5 x 90 x 50 x 4x10-5) x 0.333 x 0.16 x 150 7 Hrs./day Days No. A/C Freq. CP IR Exposed Occ’s. TARAM guideline for normally accepted 90 day fleet risk is R 0.5 Note: Hrs./day probably a low estimate © 2013 NuSS Ref: PS-ANM-25-05 FAA TARAM Handbook An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 23 TARAM Risk Assessment • It appears that risk of battery malfunctions is an unsafe condition Individual per hour risk (20x FAA normally accepted risk) Fleet risk (4000x FAA normally accepted risk) 90 day fleet risk (10x FAA normally accepted risk) • Assumptions appear to be reasonable • However, “injury ratio (IR)” may be too high for the 787 scenario One order of magnitude less would put 90 day risk within acceptable range From an outsider’s perspective, it appears the FAA was correct issuing its AD. However, additional data could substantiate a reduced injury ratio that would justify continuing operations for a short time to allow design and installation of a mitigating risk control. © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 24 Comparison of Old and New 787 Battery Redesign Mitigating redesign features • • • • • Increased space between cells Fireproof containment box Vent fumes outside Detailed pre-flight data review Repeated in-flight status checks Ref: WSJ 2/21/13 WSJ photo © 2013 NuSS Original An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach NTSB photo 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 25 New 787 Battery Reduces Risk • FAA issued AD 2013-08-12 on April 26, 2013 Mandates installation of new batteries per Boeing Service Bulletin • Mitigating redesign features should sufficiently reduce risk Increased space between cells Fireproof containment box Vent fumes outside Detailed pre-flight data review Repeated in-flight status checks • If each feature reduces risk by factor or 10: Fleet risk goes from 80 to 0.0008 Reduces risk to 25 times less than TARAM normally accepted fleet risk (0.02) © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 26 An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach Summary • RA&RM is a part of life • Quantitative RA&RM is should be a skill of every aircraft engineer • FAA has a rational approach to RA&RM The FAA AD to ground the B-787 appears justified Boeing’s mitigations appear to lower risk significantly © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 27 An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach • Questions? • Perspectives? Thanks for your attention! Marv Nuss 1 + 913-962-4683 marv.nuss@marvnuss.com www.marvnuss.com © 2013 NuSS An Outsider’s Perspective on FAA’s Aircraft Safety Risk Assessment & Risk Management Approach 2013 Aircraft Airworthiness & Sustainment (Australia) Conference Brisbane, QLD July 23-25, 2013 www.marvnuss.com 28