Anatomy of an Incident Michael E. Cournoyer LA-UR-16-20647 March 13, 2016 UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 1 UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 2 Accident pyramid UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 3 Five-step process of Integrated Safety Management UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 4 Anatomy of an incident ! The incident occurs ! The incident is investigated ! Causal factors are determined ! Conclusions are drawn ! Corrective actions are developed ! Lessons Learned are communicated UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 5 The incident occurs UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 6 Before The incident occurs After UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 7 The incident is investigated UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 8 The incident is investigated UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 9 The incident is investigated NA--LASO-LANL-CHEMLASER-2015-0001: Light Fixture Fire 11:13 The Acrylic Lens Ignites 11:00 11:07 Light Bulb Burst The Acrylic Lens Melts 11:23 A Small Fire Starts 11:00 11:25 UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 10 The incident is investigated • The metal halide lamp is designed around a sealed tube with an electrode in each end • The Arc Tube temperature range is between 1000°C and 1300°C • This type of metal halide lamp (ANSI M59/S) is position dedicated UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 11 The incident is investigated • Lamp position is horizontal • The fixture had an acrylic lens • The acrylic material has a flash point > 250°C and an AutoIgnition temperature > 400°C. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 12 Causal factors are determined NA--LASO-LANL-CHEMLASER-2015-0001: Light Fixture Fire Engineering Management 2 LLs on This The Lens Material Was Acrylic Zero Accidents LL Policy LTA Policy Not Enforced Management Methods LTA Error in Material Selection MOV LTA Design Output LTA The Wrong Bulb Was Installed. Inspection LTA Required Inspection Was Not Established Inspection Would Identify Wrong Bulb & Lens Material Corrective Maintenance LTA Preventive Maintenance for Equipment LTA Preventive Maintenance Was not Established for the Bulb That Failed. Property Damage The Lens Material Was Acrylic Defective, Failed or Contaminated Check of Work Was LTA Failed Material Skill Based Errors End of Life Failure Equipment Bulb Exploded The Wrong Bulb Was Installed. Wrong Action Man Power The Wrong Bulb Was Installed. The Wrong Bulb Was Installed. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 13 Causal factors are determined Frequency 25 Incident Causal Factors 20 15 10 5 0 Management Man Power Equipment Engineering UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 14 Conclusions are drawn • A wrong metal-halide lamp bulb caused a small fire. • The acrylic lens material exacerbated the incident. • The engineer should have required lenses that are compatible with metalhalide lamp. • The worker should have known that he or she was installing the wrong bulb. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 15 Conclusions are drawn • The worker’s supervisors should have checked the workers task and discovered that the wrong bulb was installed. • The worker’s manager should have monitored the workers task. • Implementing Lesson’s Learned from two previous incidences could have prevented the small fire. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 16 Corrective actions are developed • Position metal-halide lamp all in one direction • Store only one type of metal-halide lamp bulbs Replace acrylic lens with lenses that are compatible with metal-halide lamp • Iterate the importance of management monitoring programs UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 17 Lessons Learned integration with work performance Managers Start Identify improvement opportunities Communicate relevant LessonsLearned toworkers LLsCoordinator Workers “LessonsLearned”(LLs)Process Communicate keyLessons Learnedto managers Incorporate improvements intoWork Processes PerformWork Evaluatework toidentify lessonslearned ShareLesson Learned Lessons Learned Archive CaptureLessons Learned ScreenLessons Learned MonitorLesson Learned End UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 18 Defense in Depth UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 19 Slip simulator training UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 20 UNCLASSIFIED ADPSM Total With S/T/F Year Employees Trained* Injuries 2011 481 0 8 2012 484 229 8 2013 491 385 5 *None of the Trained Employees Had an Injury UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED UNCLASSIFIED | 21 UNCLASSIFIED P-Value for a Chi Square Test Injury (Slip, Trip, Fall) Trained Yes No Yes 0 614 No 21 800 Х2=15.9385 ThePvalueis6.5E-05 Thisresultissignificantatp<0.05 TheassociaAonbetweenrows(groups)andcolumns(outcomes) isconsideredtobeextremelystaAsAcallysignificant UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED UNCLASSIFIED | 22 Lessons Learned integration with work performance UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 23 Lessons Learned integration with work performance • Metal halide light failures - Small fires start when burning diffuser falls to ground, 2012-SRSRNS-0004, 2012 • Metal halide bulb fails, shatters outer glass envelope, hot glass shard chars wood crate, 2015-FSO-FNAL-022, 2015 UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 24 Lessons Learned integration with work performance • Assign a Metal-Halide Lamp Subject Matter Expert (SME) • Have the Metal-Halide Lamp SME make an inventory of metal-halide lamps • Have managers perform an inspection of metalhalide lamps in their areas of responsibility UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 25 Lessons Learned integration with work performance • Have the Metal-Halide Lamp SME revise the Metal-Halide Lamp procedure such that corrective actions have been incorporated • Have the Metal-Halide Lamp SME supervisor review Metal-Halide Lamp procedure UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 26 Lessons Learned integration with work performance • Replace all metal-halide lamps with light-emitting diode (LED) lamps. • LED lamps have a lifespan and electrical efficiency that is comparable to metalhalide lamps. • LED lamps contain no mercury. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 27 Lessons Learned integration with work performance • No ballast bypass or rewiring is required. • LED lamps operate at < 93 °C, are thermally regulated and may be positioned at any angle. In addition the operating life is 2 and half times longer than metal-halide lamps: 50,000 hr life vs 20,000 hr. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 28 Predicting the future UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 29 How the people involved saw it before the accident UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 30 How the investigator sees it after the accident UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 31 Fire the person who reported It UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 32 Latent organizational conditions • Managers going around telling there workers to be safe is difficult to track. • Tracking preventive maintenance, safety meeting attendance, and chemical container inventory is easy to track. 25 20 15 10 5 0 Management Man Power Equipment Engineering UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 33 What do you work on first? Injury Factors Factors Jun 06 - May 14 20 100% 15 75% 10 50% 5 25% 0 Chemical Agent Ergonomic Slip/Trip/Fall Struck against/by Frequency 06 - 14 Other Lift/Push/Pull 0% Cum. Frequency 06 - 14 UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 34 Safety improvements are hard to prove • Redundant defenses improve safety margins, but also increase complexity. • Flawed defenses and safety hazards become more difficult to detect. • Redundant defenses make safety improvements more difficult to identify as well. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 35 Improvement can’t be measured Injury Factors Factors Jun 06 - May 15 20 100% 15 75% 10 50% 5 25% 0 Chemical Agent Ergonomic Slip/Trip/Fall Struck against/by Other Lift/Push/Pull Frequency 06 - 14 Frequency Last 12 Months Cum. Frequency 06 - 14 Cum. Frequency Last 12 Months 0% UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 36 Summary ! A wrong type of metal-halide lamp bulb exploded causing a small fire. ! Latent organizational conditions created error-likely situations and weaken defenses. ! Corrective action included standardizing the type of metalhalide lamp bulbs, replacing acrylic lens with lenses that are compatible with metal-halide lamp, and adding the inspection of metal-halide lamp to management monitoring programs. ! A significant improvement to the task consists of replacement of metal-halide lamps with LED lamps. ! This improvement was shared through a Lessons Learned Program. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 37 Conclusions ! Latent organizational conditions that create error-likely situations and weaken defenses have been identified and controlled. ! Incorporating corrective actions selected from the DOE CAT provide corrective based on decades of incidences. ! Effective improvements have been implemented that reduce or eliminate the risk of another metal-halide lamp fire incidents. ! This increases technical knowledge and augments operational safety. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 38 Acknowledgements ! The author would like to acknowledge the U.S. Department of Energy and LANL's Plutonium Science & Manufacturing directorate for support of this work. ! The author would like to acknowledge James D. Jurney (MET-2) for information on the Light Fixture Fire incident. UNCLASSIFIED Operated by Los Alamos National Security, LLC for the U.S. Department of Energy's NNSA UNCLASSIFIED | 39