Commissioning:
Early and Often
Ryan Orr
Senior Consultant, Uptime Institute
Webinar: 16 December 2014
© 2014 Uptime Institute, LLC
Observations from the Field
• Commissioning resources are being marginalized
(schedule, financial, human resources)
• TCCFs are illuminating issues that should be caught by
basic commissioning processes
• Research and experience reveals the need for enhanced
focus on rigorous commissioning activities
— catch problems in commissioning before an outage occurs
1
© 2014 Uptime Institute, LLC
The Issues
• Commissioning has become a wasted opportunity in the
industry due to
— a lack of planning
— assumptions that proof of concept is adequate
— a lack of understanding of the benefits of commissioning
• Operations personnel are minimally involved, or not at all
2
© 2014 Uptime Institute, LLC
Why is Commissioning Important?
• Unique period of time to verify system performance as
designed without risk to mission critical IT loads
• Can reduce critical infrastructure early equipment failure
rates
• Opportunity for maintenance and operations teams to get
hands-on equipment experience
• Opportunity to verify detailed written procedures that will
govern live facility maintenance and operations
• Only opportunity to test the facility limitations
3
© 2014 Uptime Institute, LLC
Commissioning Defined
© 2014 Uptime Institute, LLC
Level 1 – Factory Witness Testing (FWT)
• Allows for basic verification of operation and capacity for
critical infrastructure
• Performed in the original equipment manufacturer’s (OEM)
factory or in a third-party testing facility
• Verifications are performed with conditions and procedures in
accordance with manufacturer, national, or international
standards
• Helps prevent delivery of components with unrepairable
defects
• Reports generated by OEM, provided to client, and reviewed
by the Commissioning team
• Performed on engine generators, UPS systems, chillers, air
conditioners, and switchgear
5
© 2014 Uptime Institute, LLC
Level 2 – Receipt, Installation, &
Post-Installation Checks
• Inspection, verification, and tests upon delivery
— equipment matches that procured and tested during Level 1


no damage
no alterations
• Post-installation checks
— each and every component and auxiliary appurtenance installed
in accordance with drawings, plans, and specifications,
accessibility, maintainability, health and safety requirements, local
code compliance, and manufacturer’s installation requirements
and directives
6
© 2014 Uptime Institute, LLC
Level 3 – Functional Component Testing
• Verification that the installed component is operable at a
basic level
• Maintaining checklists for mechanical and electrical startup
• Initial performance verification by OEM
• Mechanical systems should go through a Pre-Test and
Balancing effort to ensure accuracy before Level 4
7
© 2014 Uptime Institute, LLC
Level 4 – Functional System Testing
• Verification of system readiness for integration with other
systems supporting the data center
• Test, Adjust, & Balance (TAB) of the mechanical systems
to ensure design airflow and water-flow rates
• Demonstrations to ensure that related components,
equipment, and ancillaries of a defined system operate
and function to acceptance criteria
— normal, maintenance, and emergency modes of operation to
verify settings, alarms, capacities, and performance of associated
monitoring and control functions
8
© 2014 Uptime Institute, LLC
Level 5 – Integrated System Testing (IST)
• Verification that all data center systems work together
under a variety of load conditions as designed
• Verification that systems respond to various actions,
maintenance activities, or faults as designed per
Sequences of Operation
• Verification that each component and system as a whole
respond as intended to expected and unexpected events
9
© 2014 Uptime Institute, LLC
10
© 2014 Uptime Institute, LLC
11
© 2014 Uptime Institute, LLC
Commissioning Execution By Phase
© 2014 Uptime Institute, LLC
Commissioning Stakeholders
• Owner or Owner’s Representative
• Contractor (inclusive of OEM equipment vendors)
• Architects & Engineers
• Operations Personnel
• Commissioning Agent (CxA)
13
© 2014 Uptime Institute, LLC
Pre-Design Benchmarks
Owner or Owner’s
Representative
Contracto
r
Architects
Operations
&
Personnel
Engineers
Develop Request for Proposal
(RFP) for Commissioning Agent
✓
✓
Select Commissioning Agent
✓
✓
Establish a Commissioning Team
with key stakeholders
✓
✓
Include Commissioning in the
Overall Project Schedule
✓
✓
Identify the budget for
Commissioning
✓
Develop Owner’s Project
Requirements (OPR)
Documentation
✓
✓
✓
14
✓
✓
✓
✓
✓
✓
✓
Develop Commissioning Plan
Review Commissioning Plan
Commissioning
Agent
✓
✓
✓
✓
© 2014 Uptime Institute, LLC
Pre-Design Requirements
• Account for completion of all commissioning activities
• Schedule has significant flexibility
• Sufficient time allotted for correcting installation and
performance deficiencies
• Assess the requirement and/or capability for postoccupancy commissioning activities
— include provisions for seasonal commissioning to assess the
performance of critical components in a variety of ambient
conditions
15
© 2014 Uptime Institute, LLC
Design & Pre-Construction Benchmarks
Owner or Owner’s
Representative
Contractor
Review Design for OPR
Concurrence
✓
✓
Review Design for Operations
Concurrence
✓
Review Design for Ease of
Commissioning
✓
Develop Commissioning Plans,
Checklists, and Reports for
Level 1, 2, and 3
Review Commissioning Plans,
Checklists, and Reports for
Level 1, 2, and 3
16
✓
Architects
&
Engineers
Operations
Personnel
Commissioning
Agent
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
© 2014 Uptime Institute, LLC
Design & Pre-Construction Requirements
• Review project schedule and budget
— ensure adequate resources and time remain
• Verify adherence to OPR and BOD
— amend as necessary to keep up to date
• Scalable and Phased: include enhancements to allow for
future commissioning with reduced risk
17
© 2014 Uptime Institute, LLC
Design & Pre-Construction Requirements
• Equipment specifications
— specified capacity is net of any deductions or tolerances allowed
by national or international manufacturing standards for Level 1
• Determine if the systems can be commissioned per OPR
— add additional design elements as required to allow for the
commissioning program to meet the minimum OPR
18
© 2014 Uptime Institute, LLC
Design & Pre-Construction Requirements
• Ensure that RFPs include commissioning requirements
— include OEM on-site technician support for Level 4 and Level 5
— assess adherence throughout equipment delivery and installation
19
© 2014 Uptime Institute, LLC
Construction Benchmarks
Owner or Owner’s
Representative
Contractor
Architects
&
Engineers
Commissioning
Agent
✓
✓
Execution of Level 1
Commissioning: Factory
Witness Testing
✓
Execution of Level 2
Commissioning: Receipt,
Installation, and
Post-Installation Checks
✓
✓
✓
✓
Execution of Level 3
Commissioning: Component
Functional Testing
✓
✓
✓
✓
✓
✓
✓
✓
Develop Commissioning Plans
& Scripts for Level 4 and 5
Review Commissioning Plans
& Scripts for Level 4 and 5
20
Operations
Personnel
✓
✓
✓
© 2014 Uptime Institute, LLC
Construction Requirements
• Review project schedule and budget
— ensure schedule continues to have adequate time and budget
• Throughout construction protect equipment stored
pending installation from hazards
• Engineers should provide a finalized Sequence of
Operations document to the Commissioning Agent
— CxA creates the Level 4 and Level 5 commissioning scripts
• Verification of circuit breaker settings per short circuit
and breaker coordination study
21
© 2014 Uptime Institute, LLC
Construction Requirements
• Log critical asset information (e.g., make, model, serial
number) as equipment is delivered
• Repeat factory testing activities in actual data center
environment
• Verification that building management system (BMS) is
functional and ready to support critical Level 4 and 5
22
© 2014 Uptime Institute, LLC
Construction Requirements
• Submit formal reports to Owner detailing all items tested,
steps taken to test, and the results
23
© 2014 Uptime Institute, LLC
Commissioning Benchmarks
24
Owner or Owner’s
Representative
Contractor
Architects
&
Engineers
Operations
Personnel
Commissioning
Agent
Execution of Level 4
Commissioning: Functional
System Testing
✓
✓
✓
✓
✓
Execution of Level 5: Integrated
Systems Testing
✓
✓
✓
✓
✓
✓
✓
Final Site Clean-Up
✓
Ensure all equipment is back in
normal position
✓
© 2014 Uptime Institute, LLC
Commissioning Requirements
• Do not tolerate representative testing
— each and every critical component must be tested thoroughly
• Electronic systems involving programming and configuration
— if a testing step does not complete successfully and programming or
control wiring is altered to correct, repeat the entire testing procedure
as it is possible to have an unexpected impact
• Unsuccessful first tests should be considered for multiple
retests to ensure subsequent successful test was not an
anomaly
• Level 4 load bank testing
— engine generators, UPS, and UPS battery systems at design and
rated capacities
 recommended: minimum continuous runtime durations of ≥ 8 hours
 best practice: continuous runtimes of ≤ 24 hours
25
© 2014 Uptime Institute, LLC
Commissioning Requirements
• As possible without causing damage, test emergency
conditions—such as N-1 and no cooling with design
• Install aisle containment as part of Level 4 and Level 5
• Identify, document, and validate normal operating set
points, alarms, and component settings during Level 4
and Level 5
• Changes made during Level 5 to fix deficiencies must
include evaluation to determine which, if any, tests must
be repeated
26
© 2014 Uptime Institute, LLC
Commissioning Requirements
• Complete testing on both utility power and on enginegenerator power
• Simulate multiple fault types across separate tests on
each piece of equipment
— on highly automated data centers that rely heavily upon field
sensors, include sensor failures in testing scope
• Load banks should be as small as reasonably possible to
best simulate the actual IT environment for Level 5
27
© 2014 Uptime Institute, LLC
Commissioning Requirements
• Test a variety of load conditions (e.g., 25%, 50%, 75%,
100% step loads)
• Distribute load banks within the critical areas to best
simulate the actual IT environment distribution
— physically located within racks
— forced cooling on horizontal path
• Locate Commissioning team members strategically
throughout the data center to monitor all systems during
Level 5
28
© 2014 Uptime Institute, LLC
Commissioning Requirements
• Before Level 5, complete the building management and
control system (BMCS) graphics
• Monitor alarms generated in BMCS and electrical power
monitoring system (EPMS) to ensure accuracy and
usefulness
• Take electrical load readings and critical area
temperature readings constantly during Level 5
— Ideally use calibrated automatic data loggers
29
© 2014 Uptime Institute, LLC
Commissioning Requirements
• Isolate equipment to simulate maintenance activities at
the upstream circuit breaker
— not the unit’s local disconnect
• Perform Level 5 with active fire detection and
suppression systems to ensure there are no adverse
impacts
30
© 2014 Uptime Institute, LLC
Commissioning Requirements
• After Level 5, replace air filters for the electrical systems
and heating, ventilation, and air conditioning (HVAC)
systems
• Flush and clean piping and ductwork to ensure
construction debris does not impact future mechanical
plant performance
31
© 2014 Uptime Institute, LLC
Turnover-to-Operations Benchmarks
• Expeditiously complete all formal activities turning the facility
over to the owner and the operations team
— final commissioning documentation and reports
— final operating procedures
• Critical juncture for operations team to apply lessons
learned
• Make all possible support resources available to operations
team
The longer it takes, the longer the facility will be at risk!
32
© 2014 Uptime Institute, LLC
Re-Commissioning & Future Installation
• Increasingly designs are made scalable for incremental
buildouts to more efficiently deploy capital
— leading to rigorous initial commissioning, but no Level 4 or 5 on
the subsequent phases
 increased risk to the facility
• Owners are not examining the risk of only light
commissioning for future phases
• Rigorous commissioning in the beginning will make
follow-up commissioning activities far easier
• Consider altering the deployment schedule to
accommodate less risky commissioning
33
© 2014 Uptime Institute, LLC
Re-Commissioning & Future Installation
Elements and Benchmarks
Owner or Owner’s
Representative
34
Develop detailed
commissioning scripts for
all levels
✓
Review Commissioning Scripts
✓
Ensure IT hardware is dual
corded (for those facilities that
require it)
✓
Execute Commissioning
✓
Update SOPs, MOPs,
and EOPs
✓
Contractor
✓
Architects
&
Engineers
✓
Operations
Personnel
Commissioning
Agent
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
© 2014 Uptime Institute, LLC
Re-Commissioning & Future Installation
Elements and Requirements
• Adequate notice to service owners to gain concurrence
from IT end users
— schedule, duration, risk, and countermeasures in place
• Verification that existing critical load is appropriately dual
corded (as required)
• Consider load bank placement carefully to not impact
critical IT equipment
35
© 2014 Uptime Institute, LLC
Re-Commissioning & Future Installation
Elements and Requirements
• Prepare and follow detailed commissioning scripts to
ensure minimal risk to existing IT equipment
— priority to the live production IT environment
— back-out procedures in place to ensure an optimal mean time to
recovery (MTTR) in case of a power down event
• Perform seasonal testing to verify performance in a
variety of climatic conditions including extreme ambient
conditions
— ensures that economizers, where used, will be tested properly
36
© 2014 Uptime Institute, LLC
Conclusions
• Not an opportunity to be wasted—the benefits should not
be undervalued
• We, as an industry need to be better about identifying and
addressing potential roadblocks for a rigorous
commissioning program earlier in the project
• Start With The End In Mind—additional emphasis needs to
be placed on involving the operations personnel early and
often
• Sets up your Operations team for success
• Education regarding the benefits of commissioning is the
key!!
37
© 2014 Uptime Institute, LLC
Questions?
Ryan Orr
Senior Consultant
rorr@uptimeinstitute.com
© 2014 Uptime Institute, LLC