Data Center Design
An Overview
Ronald P. Szkodny, P.E.
Presented to ISPE
New Jersey Chapter
October 14, 2010
1
Topics
 Identify your Critical Application
 Redundancy, Reliability, Availability &
Maintainability
 Define Mechanical and Electrical Load
Requirements
 The Tier Classification Approach to Site
Infrastructure Functionality
 Putting Together a Professional Team
 Testing & Commissioning
 Site Selection Considerations
 Energy Efficiency Considerations
Presented to ISPE New
Jersey Chapter
October 14, 2010
2
Identify Your Critical Applications
 What are the functional consequences
should the mission application
become unavailable?
 What financial loss will be incurred
should the application become
unavailable?
 Will the business support the ongoing
expenses in maintaining and
operating the infrastructure?
Presented to ISPE New
Jersey Chapter
October 14, 2010
3
Identify Your Critical Applications
 Can the application be shut down on
a regular basis for equipment
maintenance?
 Is the building functionally capable of
supporting a critical application or will
the cost of upgrade be too much?
 Can the process be seamlessly
backed up at another location?
Presented to ISPE New
Jersey Chapter
October 14, 2010
4
Identify Your Critical Applications
 Is the process unique and proprietary
to your company/industry?
 Are there available, either in house or
outsourced, personnel to operate &
maintain the equipment?
 What can your budget support?
 Is failure not an option?
Presented to ISPE New
Jersey Chapter
October 14, 2010
5
Identify Your Critical Applications
 This leads us to the next topic,
Redundancy, Reliability, Availability &
Maintainability
Presented to ISPE New
Jersey Chapter
October 14, 2010
6
Redundancy, Reliability, Availability
& Maintainability
 Redundancy
Quite simply stated mathematically as
N+1, with “N” the number of
components required to maintain the
application, with “1” a back-up
component
Example, 1,200 tons of cooling required
so use two 600 ton chillers (N=2).
Install three chillers (N+1=3)
Presented to ISPE New
Jersey Chapter
October 14, 2010
7
Redundancy, Reliability, Availability
& Maintainability
 Reliability
Reliability is the ability of a system or
component to perform its required
functions under stated conditions for a
specified period of time (IEEE 90)
Reliability is affected by the complexity of a
system. Less components = Less chance of
component failure
Components may have a published Mean Time
Between Failure (MTBF). MTBF for a
component may not be representative of
system reliability.
Presented to ISPE New
Jersey Chapter
October 14, 2010
8
Redundancy, Reliability, Availability
& Maintainability
 Availability
Availability is the degree to which a system or
component is operational and accessible
when required for use.
Availability
based on Site
Caused
Downtime
99.67%
99.75%
99.98%
99.99%
Annual Site
Caused
Downtime
28.8 Hours
22.0 Hours
1.6 Hours
0.8 Hours
Presented to ISPE New
Jersey Chapter
October 14, 2010
9
Redundancy, Reliability, Availability
& Maintainability
 Maintainability
 Ultimate desire is to have a system that
is concurrently maintainable, that is, the
critical operation is maintained while
service is underway.
 Single bus solutions and single points of
failure prohibit concurrent maintenance
 Are shutdowns for maintenance
tolerable?
Presented to ISPE New
Jersey Chapter
October 14, 2010
10
Questions?
We’ve covered Identifying your Critical
Application and Redundancy,
Reliability, Availability &
Maintainability
 How does this tie together?
 Fault tolerant design within the
established performance parameters and
economic limitations
Presented to ISPE New
Jersey Chapter
October 14, 2010
11
Define Mechanical and Electrical
Load Requirements
 Review
Electrical power is measured in watts
Mechanical cooling is stated in Btu/Hr
12,000 Btu/Hr = 1 Ton of Cooling Capacity
3.412 Btu/Hr = 1 Watt
 How much Power?
 How much Cooling
Presented to ISPE New
Jersey Chapter
October 14, 2010
12
Define Mechanical and Electrical
Load Requirements
 How to determine power and cooling
requirements
 If an existing or similar installation exists, obtain
data of load profile
 Obtain equipment power and cooling data from
IT department/equipment vendors
 Watts per square foot? Watts per rack?
 Density of rack layout (Square feet per rack)
 Help from the team (We’ll talk about later)
Presented to ISPE New
Jersey Chapter
October 14, 2010
13
Define Mechanical and Electrical
Load Requirements
 Things to be aware of
 If using existing installation as a data
point, understand how the data was
obtained. Best is actual meter readings
at the right place in the system.
 Rack load density – Increasing to levels
beyond air cooling, about 14 kW per
rack. Water Cooled an option? Available
but not in widespread use.
 Dual corded equipment
Presented to ISPE New
Jersey Chapter
October 14, 2010
14
Define Mechanical and Electrical
Load Requirements
 Things to be aware of (continued)
 Space requirements of equipment, on
the data center floor and back of house
 Hot and cold aisles
 Height of raised floor
 Air and power management
Presented to ISPE New
Jersey Chapter
October 14, 2010
15
The Tier Classification Approach to
Site Infrastructure Functionality
 The “Tier Classification and Performance
Standard” is internationally accepted as an
objective basis for comparing the
functionality, capacities and relative cost of
a site infrastructure design topology
 See The Uptime Institute White Paper
entitled “Tier Classification Define Site
Infrastructure Performance” Copyright 2008
Uptime Institute, Inc.
Presented to ISPE New
Jersey Chapter
October 14, 2010
16
The Tier Classification Approach to
Site Infrastructure Functionality
Four Tier Classifications
Tier 1 – Single path without redundant
components, non-redundant distribution path
Tier 2 - Single path with redundant components,
non-redundant distribution path
Tier 3 – Multiple independent paths with redundant
components. One path is active, the other is
passive
Tier 4 – Multiple independent and physically
isolated paths with redundant components.
Both paths are active.
Presented to ISPE New
Jersey Chapter
October 14, 2010
17
The Tier Classification Approach to
Site Infrastructure Functionality
“…the Tier topology rating for an entire site
is constrained by the rating of the
weakest subsystem that will impact the
site operation.” Uptime Institute, Inc.
Presented to ISPE New
Jersey Chapter
October 14, 2010
18
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 1
 Active components are N
 There is one distribution path
 It is not concurrently maintainable
 No fault tolerance for a single event
 Class C interruptible cooling
 No compartmentalization
Presented to ISPE New
Jersey Chapter
October 14, 2010
19
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 1 Topology
Presented to ISPE New
Jersey Chapter
October 14, 2010
20
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 2
 Active components are N+1
 There is one distribution path
 It is not concurrently maintainable
 No fault tolerance for a single event
 Class C interruptible cooling
 No compartmentalization
Presented to ISPE New
Jersey Chapter
October 14, 2010
21
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 2 Topology
Presented to ISPE New
Jersey Chapter
October 14, 2010
22
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 3
 Active components are N+1
 There are two distribution paths, one
is active and one is passive alternate
 It is concurrently maintainable
 No fault tolerance for a single event
 Class B Continuous cooling
 No compartmentalization
Presented to ISPE New
Jersey Chapter
October 14, 2010
23
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 3 Topology
Presented to ISPE New
Jersey Chapter
October 14, 2010
24
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 4
 Active components are N after any failure
 There are two distribution paths, both
active
 It is concurrently maintainable
 It is fault tolerance for a single event
 Class A uninterruptible cooling
 Compartmentalization
Presented to ISPE New
Jersey Chapter
October 14, 2010
25
The Tier Classification Approach to
Site Infrastructure Functionality
Tier 4 Topology
Presented to ISPE New
Jersey Chapter
October 14, 2010
26
Questions?
We’ve covered Identifying your Critical
Application, Redundancy, Reliability,
Availability & Maintainability, Define
Mechanical and Electrical Load
Requirements and The Tier
Classification Approach to Site
Infrastructure Functionality
 Now we’ll move on to Putting
Together a Professional Team
Presented to ISPE New
Jersey Chapter
October 14, 2010
27
Putting Together a Professional
Team
 Project programming is essential to success. The
successful outcome of a project is most influenced by
early and proper planning and programming
 The End User is the decision maker
 The Architect/Engineer manages the performance and
reliability specification required to meet the project
program
 The commissioning agent insures the product meets
the performance and reliability intended by the
project program and manages all the tests that prove
the design intent.
Presented to ISPE New
Jersey Chapter
October 14, 2010
28
Putting Together a Professional
Team
 User – Information Technology (IT)
Manager
 IT Equipment Manufacturers
 Project/Program Manager – In-house
or third party
 Facility Management – operations,
safety and security
Presented to ISPE New
Jersey Chapter
October 14, 2010
29
Putting Together a Professional
Team
 Architect/Engineer – Mechanical,
Electrical, Fire Protection, Cabling,
Structural, Civil and Security with
Data Center experience
 Utilities – Electricity, water, sewer,
natural gas
Presented to ISPE New
Jersey Chapter
October 14, 2010
30
Putting Together a Professional
Team
 Electrical and HVAC equipment
manufacturers
 Construction Manager – Contractors
and Subcontractors
 Commissioning Agent
Presented to ISPE New
Jersey Chapter
October 14, 2010
31
Testing and Commissioning
 The Commissioning Agent is engaged from
project inception. Commissioning should
be incorporated into the construction
documents.
 The Commissioning Agent is a third party
that protects your interests and is usually
not affiliated with other team members.
 Importance of commissioning cannot be
overemphasized. Real world conditions are
tested.
Presented to ISPE New
Jersey Chapter
October 14, 2010
32
Testing and Commissioning
 The Commissioning Agent will write the
scripts for on-site testing of equipment and
systems.
 Commissioning Pitfalls





Program changes
System re-designs
Shop drawing changes
Equipment substitutions
Confirm the credentials and experience of your
commissioning agent
Presented to ISPE New
Jersey Chapter
October 14, 2010
33
Testing and Commissioning
 Six Common Levels of the Commissioning
Process
Level 1 – Review of plans, specifications and shop
drawings
Level 2 – Factory testing of critical components and
systems
Level 3 – Verification of components received
Level 4 – Verification of critical system installation
Level 5 – Verification of critical system stand-alone
operation
Level 6 – Integrated system operation, site
commissioning and capacity benchmarking (Pull
the Plug Test)
Presented to ISPE New
Jersey Chapter
October 14, 2010
34
Questions?
We’ve covered Identifying your Critical
Application, Redundancy, Reliability,
Availability & Maintainability, Define
Mechanical and Electrical Load
Requirements, The Tier Classification
Approach to Site Infrastructure
Functionality, Putting Together a
Professional Team and Testing and
Commissioning
Presented to ISPE New
Jersey Chapter
October 14, 2010
35
Site Selection Considerations
 Natural
Earthquakes, tornados, hurricanes,
lightning, flooding, volcanoes, landslides,
wildfires
 Man-made
Oil & chemical plants, nuclear plants, major
transportation arteries, airport flight
paths, uphill water storage and
treatment facilities, freight railroad lines,
farms (dusty), interstate highways
Presented to ISPE New
Jersey Chapter
October 14, 2010
36
Site Selection Considerations
 Other Site Selection Considerations
Tax incentive
Noise zoning near residential
Building expansion potential
Space for exterior equipment
Community visual restrictions
Access to utilities
Back-up water supply
Height restrictions
Emissions from generators
Climate – Maximize use of outside air
Presented to ISPE New
Jersey Chapter
October 14, 2010
37
Energy Considerations
P.U.E. – Power Usage Effectiveness
What is it?
P.U.E. equals Total Facility Power
divided by IT Equipment Power
Presented to ISPE New
Jersey Chapter
October 14, 2010
38
Energy Considerations
 Increase server inlet temperature from 77 to 80.6
degrees F into racks (ASHREA 2008 Guidelines TC
9.9)
 Minimize humidification
 Reduce cold air/hot air mixing by moving cold air
supply and returns close to the load (air flow
management)
 Use raised floor grommets at penetrations to reduce
bypass airflow around cables
Presented to ISPE New
Jersey Chapter
October 14, 2010
39
Energy Considerations
 Optimize floor layout with Computational Fluid
Dynamics (CFD) analysis
 Maximize use of outside air in climates that have
enough days that make it worthwhile.
 High efficiency equipment (UPS, transformers,
chillers, motors, etc.)
Presented to ISPE New
Jersey Chapter
October 14, 2010
40
Questions?
We’ve covered Identifying your Critical
Application, Redundancy, Reliability,
Availability & Maintainability, Define
Mechanical and Electrical Load
Requirements, The Tier Classification
Approach to Site Infrastructure
Functionality, Putting Together a
Professional Team, Testing and
Commissioning, Site Selection
Considerations and Energy Considerations
Presented to ISPE New
Jersey Chapter
October 14, 2010
41
Lessons Learned
 Secure the services of a competent commissioning
agent as early as possible.
 Do not waive any factory witness testing for the sake
of schedule.
 Clearly understand the client expectations as to
installation of equipment on the data center floor.
 Engage the electric utility as early as possible.
 Be ready for last minute circuiting changes to the
racks.
 During any on-site testing, make sure the right people
are there to immediately address any problems that
arise.
Presented to ISPE New
Jersey Chapter
October 14, 2010
42
Lessons Learned
 Have spare parts kits.
 Complete water flow and air flow balancing prior to
any component or system testing.
 The UPS batteries must be properly installed, initially
charged and commissioned to the manufacturer’s
specified recommendations.
 For any load testing, make sure enough load banks
are on site, connected, with proper instrumentation to
accurately record all required data.
 Watch out for that high density load that will pop up
on the data center floor.
Presented to ISPE New
Jersey Chapter
October 14, 2010
43
Sources of Information
 The Uptime Institute
 American Society of Heating, Refrigerating,
and Air-Conditioning Engineers (ASHREA)
 Mission Critical Magazine
 The Data Center Journal Magazine
 Consulting-Specifying Engineer Magazine
 Datacenter Dynamics
 Equipment Manufacturer’s websites
(Liebert, Eaton Corporation, Schneider
Electric, etc.)
Presented to ISPE New
Jersey Chapter
October 14, 2010
44
Data Center Design
An Overview
Thank You!
Ronald P. Szkodny, P.E.
Presented to ISPE
New Jersey Chapter
October 14, 2010
45