Commissioning: Early and Often Ryan Orr Senior Consultant, Uptime Institute Webinar: 16 December 2014 © 2014 Uptime Institute, LLC Observations from the Field • Commissioning resources are being marginalized (schedule, financial, human resources) • TCCFs are illuminating issues that should be caught by basic commissioning processes • Research and experience reveals the need for enhanced focus on rigorous commissioning activities — catch problems in commissioning before an outage occurs 1 © 2014 Uptime Institute, LLC The Issues • Commissioning has become a wasted opportunity in the industry due to — a lack of planning — assumptions that proof of concept is adequate — a lack of understanding of the benefits of commissioning • Operations personnel are minimally involved, or not at all 2 © 2014 Uptime Institute, LLC Why is Commissioning Important? • Unique period of time to verify system performance as designed without risk to mission critical IT loads • Can reduce critical infrastructure early equipment failure rates • Opportunity for maintenance and operations teams to get hands-on equipment experience • Opportunity to verify detailed written procedures that will govern live facility maintenance and operations • Only opportunity to test the facility limitations 3 © 2014 Uptime Institute, LLC Commissioning Defined © 2014 Uptime Institute, LLC Level 1 – Factory Witness Testing (FWT) • Allows for basic verification of operation and capacity for critical infrastructure • Performed in the original equipment manufacturer’s (OEM) factory or in a third-party testing facility • Verifications are performed with conditions and procedures in accordance with manufacturer, national, or international standards • Helps prevent delivery of components with unrepairable defects • Reports generated by OEM, provided to client, and reviewed by the Commissioning team • Performed on engine generators, UPS systems, chillers, air conditioners, and switchgear 5 © 2014 Uptime Institute, LLC Level 2 – Receipt, Installation, & Post-Installation Checks • Inspection, verification, and tests upon delivery — equipment matches that procured and tested during Level 1 no damage no alterations • Post-installation checks — each and every component and auxiliary appurtenance installed in accordance with drawings, plans, and specifications, accessibility, maintainability, health and safety requirements, local code compliance, and manufacturer’s installation requirements and directives 6 © 2014 Uptime Institute, LLC Level 3 – Functional Component Testing • Verification that the installed component is operable at a basic level • Maintaining checklists for mechanical and electrical startup • Initial performance verification by OEM • Mechanical systems should go through a Pre-Test and Balancing effort to ensure accuracy before Level 4 7 © 2014 Uptime Institute, LLC Level 4 – Functional System Testing • Verification of system readiness for integration with other systems supporting the data center • Test, Adjust, & Balance (TAB) of the mechanical systems to ensure design airflow and water-flow rates • Demonstrations to ensure that related components, equipment, and ancillaries of a defined system operate and function to acceptance criteria — normal, maintenance, and emergency modes of operation to verify settings, alarms, capacities, and performance of associated monitoring and control functions 8 © 2014 Uptime Institute, LLC Level 5 – Integrated System Testing (IST) • Verification that all data center systems work together under a variety of load conditions as designed • Verification that systems respond to various actions, maintenance activities, or faults as designed per Sequences of Operation • Verification that each component and system as a whole respond as intended to expected and unexpected events 9 © 2014 Uptime Institute, LLC 10 © 2014 Uptime Institute, LLC 11 © 2014 Uptime Institute, LLC Commissioning Execution By Phase © 2014 Uptime Institute, LLC Commissioning Stakeholders • Owner or Owner’s Representative • Contractor (inclusive of OEM equipment vendors) • Architects & Engineers • Operations Personnel • Commissioning Agent (CxA) 13 © 2014 Uptime Institute, LLC Pre-Design Benchmarks Owner or Owner’s Representative Contracto r Architects Operations & Personnel Engineers Develop Request for Proposal (RFP) for Commissioning Agent ✓ ✓ Select Commissioning Agent ✓ ✓ Establish a Commissioning Team with key stakeholders ✓ ✓ Include Commissioning in the Overall Project Schedule ✓ ✓ Identify the budget for Commissioning ✓ Develop Owner’s Project Requirements (OPR) Documentation ✓ ✓ ✓ 14 ✓ ✓ ✓ ✓ ✓ ✓ ✓ Develop Commissioning Plan Review Commissioning Plan Commissioning Agent ✓ ✓ ✓ ✓ © 2014 Uptime Institute, LLC Pre-Design Requirements • Account for completion of all commissioning activities • Schedule has significant flexibility • Sufficient time allotted for correcting installation and performance deficiencies • Assess the requirement and/or capability for postoccupancy commissioning activities — include provisions for seasonal commissioning to assess the performance of critical components in a variety of ambient conditions 15 © 2014 Uptime Institute, LLC Design & Pre-Construction Benchmarks Owner or Owner’s Representative Contractor Review Design for OPR Concurrence ✓ ✓ Review Design for Operations Concurrence ✓ Review Design for Ease of Commissioning ✓ Develop Commissioning Plans, Checklists, and Reports for Level 1, 2, and 3 Review Commissioning Plans, Checklists, and Reports for Level 1, 2, and 3 16 ✓ Architects & Engineers Operations Personnel Commissioning Agent ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ © 2014 Uptime Institute, LLC Design & Pre-Construction Requirements • Review project schedule and budget — ensure adequate resources and time remain • Verify adherence to OPR and BOD — amend as necessary to keep up to date • Scalable and Phased: include enhancements to allow for future commissioning with reduced risk 17 © 2014 Uptime Institute, LLC Design & Pre-Construction Requirements • Equipment specifications — specified capacity is net of any deductions or tolerances allowed by national or international manufacturing standards for Level 1 • Determine if the systems can be commissioned per OPR — add additional design elements as required to allow for the commissioning program to meet the minimum OPR 18 © 2014 Uptime Institute, LLC Design & Pre-Construction Requirements • Ensure that RFPs include commissioning requirements — include OEM on-site technician support for Level 4 and Level 5 — assess adherence throughout equipment delivery and installation 19 © 2014 Uptime Institute, LLC Construction Benchmarks Owner or Owner’s Representative Contractor Architects & Engineers Commissioning Agent ✓ ✓ Execution of Level 1 Commissioning: Factory Witness Testing ✓ Execution of Level 2 Commissioning: Receipt, Installation, and Post-Installation Checks ✓ ✓ ✓ ✓ Execution of Level 3 Commissioning: Component Functional Testing ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Develop Commissioning Plans & Scripts for Level 4 and 5 Review Commissioning Plans & Scripts for Level 4 and 5 20 Operations Personnel ✓ ✓ ✓ © 2014 Uptime Institute, LLC Construction Requirements • Review project schedule and budget — ensure schedule continues to have adequate time and budget • Throughout construction protect equipment stored pending installation from hazards • Engineers should provide a finalized Sequence of Operations document to the Commissioning Agent — CxA creates the Level 4 and Level 5 commissioning scripts • Verification of circuit breaker settings per short circuit and breaker coordination study 21 © 2014 Uptime Institute, LLC Construction Requirements • Log critical asset information (e.g., make, model, serial number) as equipment is delivered • Repeat factory testing activities in actual data center environment • Verification that building management system (BMS) is functional and ready to support critical Level 4 and 5 22 © 2014 Uptime Institute, LLC Construction Requirements • Submit formal reports to Owner detailing all items tested, steps taken to test, and the results 23 © 2014 Uptime Institute, LLC Commissioning Benchmarks 24 Owner or Owner’s Representative Contractor Architects & Engineers Operations Personnel Commissioning Agent Execution of Level 4 Commissioning: Functional System Testing ✓ ✓ ✓ ✓ ✓ Execution of Level 5: Integrated Systems Testing ✓ ✓ ✓ ✓ ✓ ✓ ✓ Final Site Clean-Up ✓ Ensure all equipment is back in normal position ✓ © 2014 Uptime Institute, LLC Commissioning Requirements • Do not tolerate representative testing — each and every critical component must be tested thoroughly • Electronic systems involving programming and configuration — if a testing step does not complete successfully and programming or control wiring is altered to correct, repeat the entire testing procedure as it is possible to have an unexpected impact • Unsuccessful first tests should be considered for multiple retests to ensure subsequent successful test was not an anomaly • Level 4 load bank testing — engine generators, UPS, and UPS battery systems at design and rated capacities recommended: minimum continuous runtime durations of ≥ 8 hours best practice: continuous runtimes of ≤ 24 hours 25 © 2014 Uptime Institute, LLC Commissioning Requirements • As possible without causing damage, test emergency conditions—such as N-1 and no cooling with design • Install aisle containment as part of Level 4 and Level 5 • Identify, document, and validate normal operating set points, alarms, and component settings during Level 4 and Level 5 • Changes made during Level 5 to fix deficiencies must include evaluation to determine which, if any, tests must be repeated 26 © 2014 Uptime Institute, LLC Commissioning Requirements • Complete testing on both utility power and on enginegenerator power • Simulate multiple fault types across separate tests on each piece of equipment — on highly automated data centers that rely heavily upon field sensors, include sensor failures in testing scope • Load banks should be as small as reasonably possible to best simulate the actual IT environment for Level 5 27 © 2014 Uptime Institute, LLC Commissioning Requirements • Test a variety of load conditions (e.g., 25%, 50%, 75%, 100% step loads) • Distribute load banks within the critical areas to best simulate the actual IT environment distribution — physically located within racks — forced cooling on horizontal path • Locate Commissioning team members strategically throughout the data center to monitor all systems during Level 5 28 © 2014 Uptime Institute, LLC Commissioning Requirements • Before Level 5, complete the building management and control system (BMCS) graphics • Monitor alarms generated in BMCS and electrical power monitoring system (EPMS) to ensure accuracy and usefulness • Take electrical load readings and critical area temperature readings constantly during Level 5 — Ideally use calibrated automatic data loggers 29 © 2014 Uptime Institute, LLC Commissioning Requirements • Isolate equipment to simulate maintenance activities at the upstream circuit breaker — not the unit’s local disconnect • Perform Level 5 with active fire detection and suppression systems to ensure there are no adverse impacts 30 © 2014 Uptime Institute, LLC Commissioning Requirements • After Level 5, replace air filters for the electrical systems and heating, ventilation, and air conditioning (HVAC) systems • Flush and clean piping and ductwork to ensure construction debris does not impact future mechanical plant performance 31 © 2014 Uptime Institute, LLC Turnover-to-Operations Benchmarks • Expeditiously complete all formal activities turning the facility over to the owner and the operations team — final commissioning documentation and reports — final operating procedures • Critical juncture for operations team to apply lessons learned • Make all possible support resources available to operations team The longer it takes, the longer the facility will be at risk! 32 © 2014 Uptime Institute, LLC Re-Commissioning & Future Installation • Increasingly designs are made scalable for incremental buildouts to more efficiently deploy capital — leading to rigorous initial commissioning, but no Level 4 or 5 on the subsequent phases increased risk to the facility • Owners are not examining the risk of only light commissioning for future phases • Rigorous commissioning in the beginning will make follow-up commissioning activities far easier • Consider altering the deployment schedule to accommodate less risky commissioning 33 © 2014 Uptime Institute, LLC Re-Commissioning & Future Installation Elements and Benchmarks Owner or Owner’s Representative 34 Develop detailed commissioning scripts for all levels ✓ Review Commissioning Scripts ✓ Ensure IT hardware is dual corded (for those facilities that require it) ✓ Execute Commissioning ✓ Update SOPs, MOPs, and EOPs ✓ Contractor ✓ Architects & Engineers ✓ Operations Personnel Commissioning Agent ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ © 2014 Uptime Institute, LLC Re-Commissioning & Future Installation Elements and Requirements • Adequate notice to service owners to gain concurrence from IT end users — schedule, duration, risk, and countermeasures in place • Verification that existing critical load is appropriately dual corded (as required) • Consider load bank placement carefully to not impact critical IT equipment 35 © 2014 Uptime Institute, LLC Re-Commissioning & Future Installation Elements and Requirements • Prepare and follow detailed commissioning scripts to ensure minimal risk to existing IT equipment — priority to the live production IT environment — back-out procedures in place to ensure an optimal mean time to recovery (MTTR) in case of a power down event • Perform seasonal testing to verify performance in a variety of climatic conditions including extreme ambient conditions — ensures that economizers, where used, will be tested properly 36 © 2014 Uptime Institute, LLC Conclusions • Not an opportunity to be wasted—the benefits should not be undervalued • We, as an industry need to be better about identifying and addressing potential roadblocks for a rigorous commissioning program earlier in the project • Start With The End In Mind—additional emphasis needs to be placed on involving the operations personnel early and often • Sets up your Operations team for success • Education regarding the benefits of commissioning is the key!! 37 © 2014 Uptime Institute, LLC Questions? Ryan Orr Senior Consultant rorr@uptimeinstitute.com © 2014 Uptime Institute, LLC