INPO Event Report Level 4 15-3 Issue Date: March 3, 2015 LIMITED DISTRIBUTION This INPO Level 4 document provides notable information on an industrywide trend in mechanical maintenance precursor events that contribute to equipment damage and safetyrelated equipment unavailability. INPO evaluation and peer review teams will not review Level 4 INPO Event Reports (IERs). Documentation is not required for the disposition of IER Level 4 documents. Each station is expected to review its performance based on lessons learned from this document and consider developing corrective actions as necessary. Mechanical Maintenance Performance Summary Worker performance shortfalls during mechanical maintenance activities are contributing to equipment damage and safety-related equipment unavailability. Although high-level consequential events have not yet shown an adverse trend, the behaviors and work practices are precursors to consequential events and need to be recognized and mitigated. Insights from nonconsequential events reported in the INPO Consolidated Event System (ICES®), INPO areas for improvement (AFIs), and observations during INPO assist visits offer a collective view of the underlying causes of this problem. The trend of mechanical maintenance precursor errors from early 2012 through mid-2014 worsened, predominately during valve and pump maintenance and mechanical joint reassembly. Insufficient awareness of proficiency challenges1 such as lack of preparation, procedure quality, and overconfidence on the part of supervisors and workers contributes to many of the mistakes. These problems are further amplified by inappropriate use of human performance tools, weak coaching, and ineffective oversight to mitigate the proficiency challenges. Many errors could have been prevented had worker fundamentals been fully understood, implemented and reinforced. Analysis The quarterly trend of mechanical precursor events reported in ICES from April 2012 through June 2014 is presented in the chart on page 12. In addition to the ICES reports, INPO AFIs and assist visit information were 1 2 See Reference 6 ICES reports coded as noteworthy but nonconsequential Page 1 of 6 INPO Event Report Level 4 15-3 reviewed to obtain further insights regarding underlying causes. The collective review identified three primary underlying causes of mechanical-maintenance-related precursor events as follows: Workers did not fully understand and apply appropriate fundamentals and standards, which led to inconsistent work quality. The fundamental that is most often overlooked is whether workers are prepared to perform high-quality corrective and preventive maintenance. Supervisors and workers did not understand and account for the risk significance of the work activity during work planning, assignment and execution. In most cases, had the importance of the component or system to reactor safety and station reliability been understood, additional measures would have been taken to provide and ensure highquality work. Inadequate preparation because of late work assignments challenges the depth of procedure and documentation review and exacerbates time pressures, whether selfimposed or schedule driven. To obtain perspective regarding frequency of these causes, each ICES report was assigned the best-fitting underlying cause, resulting in the following bar graph. Nearly half of the events and many AFIs reveal a weakness in the application of maintenance fundamentals and standards. In several cases, mechanical maintenance supervisors did not provide coaching and reinforcement of maintenance fundamentals and standards, which are vital to reducing maintenance errors. In other cases, maintenance supervisors and workers did not recognize the risk significance or potential plant impact of performing maintenance incorrectly. Several events were the result of latent equipment failures not revealed during postmaintenance testing that were caused by improper reassembly of critical and safety-related equipment. These events resulted in a perception that associated risks were low during these work activities. Subsequently, prejob briefings and job site reviews inadequately addressed risks identified in the work plan, contributing to the continuation of worker errors. In some cases, workers did not have an opportunity or sufficient time to review the work task before execution, limiting their effectiveness in identifying weaknesses in work instruction technical details or errors in the work instructions. An analysis of industry scrams3 noted that 3 See Reference 3 Page 2 of 6 INPO Event Report Level 4 15-3 over half of maintenance-related scrams involved insufficient maintenance work instruction detail. Assigning work late prevents adequate review and understanding of the work task by workers and supervisors. Lastly, supervisors and workers are not recognizing circumstances that can challenge the performance of even the most proficient workers; consequently, they do not establish controls to mitigate those challenges. For example, a main steam isolation valve was reassembled incorrectly, resulting in a forced outage extension. The error was partly caused by the work being reassigned from a specialized vendor to in-house mechanics. Even though the in-house mechanics had not performed this task before and were unfamiliar with the reassembly of the valve, the supervisor did not recognize and assess the proficiency challenges nor take the necessary actions to mitigate the errors. Lessons Learned Mechanical maintenance staff must understand and apply appropriate fundamentals and standards for ensuring consistently high-quality work. Supervisors should communicate and reinforce maintenance fundamentals during maintenance training, prejob briefings, and oversight of field activities. INPO 15-001, Nuclear Maintenance Fundamentals, can be used to improve the understanding of maintenance fundamentals of workers, supervisors and maintenance managers. Supervisors and workers should understand all risks associated with a maintenance activity so that actions can be identified and taken to mitigate the risks. Review the guidelines in INPO 05-004, Guidelines for the Conduct of Maintenance at Nuclear Power Stations, for using a graded approach to work planning commensurate with the level of task risk. Ensure work packages are assigned to workers far enough in advance of the scheduled start date to allow adequate review, understanding and clarification. Work should not be assigned on the morning of execution — this being the first opportunity for workers to prepare for the activity. Supervisors and workers should recognize proficiency challenges and take appropriate mitigating actions when work is being prepared, assigned and executed. Ensure supervisors understand the concepts of proficiency challenges and appropriately evaluate them when assigning and overseeing work. Consider using Attachment 1 to evaluate shortfalls in mechanical maintenance training based on observed behaviors. Attachments Attachment 1: Mechanical Maintenance Behaviors Observed During INPO Evaluations and Assist Visits References 1. INPO 15-001, Nuclear Maintenance Fundamentals 2. INPO 05-004, Guidelines for the Conduct of Maintenance at Nuclear Power Stations Page 3 of 6 INPO Event Report 3. 4. 5. 6. Level 4 15-3 IER L2-11-2, 2009-2010 Scram Analysis IER L4-12-82, Emergency Diesel Generator Leaks IER L4-14-30, Analysis of Vibration-Induced Piping and Tubing Leaks (2010 through 2013) INPO Member Website (Proficiency), Advancing Proficiency (Brochure) Page 4 of 6 INPO Event Report Level 4 15-3 Attachment 1 Mechanical Maintenance Behaviors Observed During INPO Evaluations and Assist Visits – When performing a crisscross final torque of flow orifice flange bolts, a supplemental mechanic lost position on the flange. This resulted in re-torqueing two bolts and not applying the final torque on another two bolts. (maintenance fundamental) – Supplemental valve technicians did not stop when unsure and question unexpected conditions when installing the air-operated valve (AOV) actuator into the valve bonnet. When the workers attempted to install the actuator into the bonnet, the actuator stem bottomed out before the actuator was fully inserted into the bonnet. Three additional attempts were made to insert the actuator into the valve bonnet, resulting in multiple contacts between the stem and bonnet. (maintenance fundamentals) – A mechanic's glove smeared a valve disc when installing the valve disc assembly into the valve body during performance of the diesel jockey pump check valve seat blue check. Smearing can result in a false indication of a seat and disc inspection if the blue check die is rubbed off during the installation. (maintenance fundamentals) – Mechanics did not secure fasteners, match mark components or label parts during disassembly of a vault vapor extractor fan. (maintenance fundamentals) – A mechanic inappropriately used a flat screwdriver and a ballpeen hammer to remove a used spiral-wound gasket from a tongue-and-groove flange joint connecting a main steam relief valve to piping. (maintenance fundamentals) – Mechanics working on an AOV inappropriately used a brass bar as a hammer, a Phillips screwdriver to apply lubrication, and a screwdriver to align the upper and lower air dome holes with the diaphragm. (maintenance fundamentals) – A mechanic improperly set up the test equipment before performing the auxiliary feedwater pump turbine oil cooler pressure test. This resulted in erroneous readings that the test was acceptable when in fact the test criteria had not been met. The work order step required a leak test of the lube oil cooler with air set to 50 pounds per square inch (psi) for 10 minutes. The test equipment setup was an air cylinder and a regulator set at 50 psi connected directly to the oil cooler. This setup would maintain the air pressure at 50 psi until the air cylinder emptied. (work practices) – A mechanical supervisor did not verify the required position of a hydraulic control unit valve before incorrectly directing the valve to be positioned to the as-found position. (work practices) – A mechanic used an inadequate work order to perform sealant stick leak repair for the auxiliary feedwater inlet valve body. (work practices) – Supplemental valve technicians did not follow work instructions and improperly tightened the actuator for a moisture separator reheater drain tank level control valve. (work practices) – While replacing and charging suction stabilizer and discharge pulsation damper bladders on a positive displacement charging pump, a mechanic did not perform a prescribed procedure step without concurrence from supervisors. (work practices) Page 5 of 6 INPO Event Report Level 4 15-3 LIMITED DISTRIBUTION: Copyright © 2015 by the Institute of Nuclear Power Operations. Not for sale or for commercial use. Unauthorized reproduction is a violation of INPO policy and U.S. and international copyright laws. The confidential, proprietary information herein is for use only by members and participants of INPO. This material is not authorized to be transferred or delivered to any third party, and its contents must not be made public, without INPO’s prior written consent. All other rights reserved. NOTICE: This information was prepared in connection with work sponsored by the Institute of Nuclear Power Operations (INPO). Neither INPO, INPO members, INPO participants, nor any person acting on behalf of them (a) makes any warranty or representation, expressed or implied, with respect to the accuracy, completeness, or usefulness of the information contained in this document, or that the use of any information, apparatus, method, or process disclosed in this document may not infringe on privately owned rights, or (b) assumes any liabilities with respect to the use of, or for damages resulting from the use of any information, apparatus, method, or process disclosed in this document. Page 6 of 6