Data Center Facility Basics
Ofer Lior, Spring 2015
Challenges in Modern Data Centers Management, Spring 2015
1
Information provided in these slides is
for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
2
Outline
• Introduction
• Four main elements of a Data Center facility
• Total Cost of Ownership (TCO)
• Power Usage Effectiveness (PUE)
• Challenges related to improving PUE
• Optimizing cooling
• Optimizing power feeding
• Optimizing refresh rate
• Open Compete Project (OCP)
Challenges in Modern Data Centers Management, Spring 2015
3
What is a Data Center?
A Data Center is a facility used to house computer systems and associated components, such as
telecommunications and storage systems. It generally includes redundant or backup power supplies,
redundant data communications connections, environmental controls (e.g., air conditioning, fire
suppression) and security devices. Large data centers are industrial scale operations using as much
electricity as a small town.
Challenges in Modern Data Centers Management, Spring 2015
4
How did Data Centers become the production
floor of most companies?
• Data Center history and development:
• Started as a facility to house old complex computing systems
• Big size systems, required special environment in which to operate
• Many cables were needed
• A single mainframe consumed a lot of power and was hard to cool
• Security requirements for computing facilities started in the military computing
environment
Challenges in Modern Data Centers Management, Spring 2015
5
How did Data Centers become the production
floor of most companies?
• Data center history and development:
• Data Center golden age started during the DotCom bubble
• Internet connection needed for business connectivity and stock exchange
• 24/7 companies’ representation on the net
• Data started to be centrally stored on computing devices and available on the web
• Cloud computing and Information explosion caused the Data Center industry
to grow dramatically
• Data Centers are the engine behind all known application and web use-cases .
• 100% availability and connectivity – data is available from everywhere, all the time
• Data is no longer kept on the personal device
Challenges in Modern Data Centers Management, Spring 2015
6
How are Data Centers used by big enterprises?
• Build vs. rent vs. use hosting facilities
• Room/rack/servers/service hosting
• What are the Data Centers
in Intel/Disney/Amazon/
Google/Facebook used for?
•
•
•
•
Visualized: ring around the world of data center power usage *
Storage farms
Computing farms
Hosting farms
SMB server room
*
Different articles may show
different numbers due to data
confidentiality inside large
organizations
Courtesy of Business2community.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
7
Four main elements of Data Center infrastructure
1. Power – 1 MW = 10,000 x 100 Watt light bulbs / medium neighborhood
• Main element and cost of the Data Center operation
2. Cooling – up to 50% of total Data Center power:
• Heat is created by the electronic elements of all computing devices.
• Multiply the heat of your laptop by many thousands to understand the challenge.
3. Space – each rack space
can cost up to $20,000/Y
Typical Data center power consumption – for a non efficient facility
• Floor space should be
populated with as many
devices as possible.
• Construction costs are high .
4. Network – the veins and
arteries of the data center
Challenges in Modern Data Centers Management, Spring 2015
Courtesy of blog.opower.com – for
educational purposes only
8
Data Center Power
What consumes more power?
OR
Courtesy of wgt.co.il– for educational purposes only
Courtesy of brimag.co.il - for educational purposes only
Behind every smartphone there are several hundred watts of server power!!
The cost of a Google search is estimated at 0.0003 kWh of energy.
Challenges in Modern Data Centers Management, Spring 2015
9
Data Center Power cont.
• Electricity used in global data centers in 2010 likely accounted for
between 1.1% and 1.5% of total electricity use. For the US that
number was between 1.7 and 2.2%.
Growth in data center electricity use 2005 to 2010 by Jonathan Koomey
•
•
•
•
Why do we need redundancy?
Main elements in Data Center power designs
Why is the Data Center power always growing?
What causes inefficiency?
Challenges in Modern Data Centers Management, Spring 2015
10
Data Center Power cont.
Courtesy of buildings.com – for educational purposes only
AC/DC current conventions in Data Center operation
Courtesy of embedded.com– for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
11
Data Center Cooling
• What creates heat?
• ~70% of all computing heat is created by the CPU
• The heat density of a CPU core is similar to the heat
density of the sun’s surface (energy per SQF)
• Charging and discharging capacitors even with
minimal electrical resistance creates heat
• High 20Amp Current flowing in a 8mm footprint
• Every CPU core can consume 20 watts and reach 100
degrees
• Current leaks at the transistor level create heat
Challenges in Modern Data Centers Management, Spring 2015
12
Data Center Cooling cont.
• How can heat impact Data Center uptime?
• How is heat evacuated and neutralized?
• Cooling methods
•
•
Basic Data Center cooling diagram
Cold water systems vs. DX units
• High-density computing challenges
Courtesy of datacenterjournal.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
13
Data Center Cooling cont.
• Cooling inefficiencies:
• Hot & cold air mix
• Over-cooling
• High cold air volume
• Heat exchanging cycles
• Electronics to air
• Air to water/gas
• Water to air
Courtesy of electronics-cooling.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
14
Data Center space and physical layout
• Data Center facility is extremely expensive to build.
• A single rack location construction cost can sum up to $80K*
• A single rack location maintenance, rent and power can sum up to $20K/y*
• Layout is arranged in hot/cold aisle design for better cooling efficiency
* Estimation for best of breed data center facility
Challenges in Modern Data Centers Management, Spring 2015
15
Data Center Network
• To be discussed in a coming lecture
Challenges in Modern Data Centers Management, Spring 2015
16
Total Cost of Ownership (TCO)
• TCO – the sum of initial capital expenditures (CapEx) added to
ongoing and long-term operational expenditures (OpEx)
• Basic formula:
•
•
•
•
•
•
•
Building the facility
Buying the computing
Power cost
Cooling cost
Space cost
Network cost
Staff
Courtesy of http://datacenter10.blogspot.co.il/ – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
17
Total Cost of Ownership
• TCO can be improved by the following:
• Balance between the requirements,
flexibility and expected growth
• Set the right amount of redundancy
• DC efficiency management
• Smart CaPex investment
• Why does 1 percent in energy
efficiency matter?
Courtesy of www.datacenterknowledge.com– for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
18
Power Usage Effectiveness
• PUE — a measure of how efficiently a data center uses energy; specifically,
how much energy is used by the computing equipment (in contrast to
cooling and other overhead)
• Pros 
• A way to measure power efficiency, benchmark, and start improving
• Easy to measure and calculate at basic level
• Commonly used in IT industry by all main players
• Cons 
• Hard to benchmark between different facilities — creates deviations
• Does not take into account computing utilization, usage, and business value
• Complicated formula in mixed facilities
Challenges in Modern Data Centers Management, Spring 2015
19
PUE – not so simple
Challenges in Modern Data Centers Management, Spring 2015
20
PUE benchmarking
Challenges in Modern Data Centers Management, Spring 2015
21
Optimizing PUE – it’s all about power!!!
• PUE is a power efficiency performance indicator
• Three main ways to improve PUE
1. Optimizing cooling
2. Optimizing power feeding
3. Optimizing refresh rate
• PUE impacts the carbon footprint of the facility
• Total set of greenhouse gas emissions caused by an organization (tones per year)
Challenges in Modern Data Centers Management, Spring 2015
22
www.epa.gov/cleanenergy/energy-resources/calculator.html
Challenges in Modern Data Centers Management, Spring 2015
23
Challenge #1: Optimizing cooling
• Why is cooling an issue?
• Efficiency of cooling mechanicals – efficient pumps, chillers, piping, and airflow
• Available solutions:
• Hot aisle – can improve PUE from 3 up to 2.5
• Hot/cold air containment – can improve PUE from 2.5 up to 1.8
• Free cooling – can improve PUE from 1.8 up to 1.05
• Data center physical location
• Building the data center where free cooling is available (ocean, cold areas)
http://www.youtube.com/watch?v=4l0KPek7Ibo
http://www.youtube.com/watch?v=SRn_xW7VtWc
Challenges in Modern Data Centers Management, Spring 2015
24
Cold and hot air separation – enclosing the
hot/cold air
Data center layout arranged in Hot/cold aisle method
Hot air enclosure by chimney racks – the room
area is cold
Cold air enclosure – the room area
is hot
Challenges in Modern Data Centers Management, Spring 2015
25
Cold and hot air separation – enclosing the
hot/cold air
Courtesy of psmtyech.com.sg – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
26
Challenge #2: Optimizing power feeding
• Redundancy conflict – uptime vs. efficiency and cost
• A 600Kw UPS consumes 10Kw (fans, logic…) in addition to 2%-10% of load for losses
• Dividing the load to 2 units multiplies the numbers above
• UPS connection mode can impact the losses
• Offline vs online mode
• Eco mode allows offline mode with clean power
Challenges in Modern Data Centers Management, Spring 2015
27
Challenge #2: Optimizing power feeding
• AC vs. DC
• Data center using AC power typically
converts power five times while a DC system
usually requires only two conversions
• Redundancy & low density create
inefficiency
• An under-loaded UPS is less efficient
• Pumps and AC units’ electrical efficiency
• Efficient motors
• Installation of VSD (variable-speed drive)
Courtesy of digikey.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
28
Advantages of DC power to a data center
• Lower line resistances
• Fewer power losses due to conversions
• The frequency of a DC system is zero Hz, eliminating synchronization procedures
when connecting to the power grid.
• High-voltage DC can transmit large amounts of power over long distances with
potentially lower capital costs and fewer losses than AC.
• DC power allows for simpler integration of renewable energy, such as
photovoltaic.
• Simplified connection of energy storage devices such as UPS batteries
Research on DC delivery system showed a minimum of 5%-7% efficiency gains
comparing to an AC delivery system to a data center.
Challenges in Modern Data Centers Management, Spring 2015
29
Challenge #3: Optimizing refresh rate
• Part of the data-center’s CapEx
• Optimal lifespan of a server in a data center is about four years
• What creates positive ROI (Return-of-Investment) for 4-year refresh plan?
• Warranty costs
• Malfunctions increase in old servers
• Technology improvements enable refresh rates that are higher that 1:1 (e.g., 2:1)
• Purchasing less servers to produce the same or even higher compute power
• Higher rack density creates better power/cooling/space efficiency
Courtesy of triplepundit.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
30
Optimizing cooling
•
•
•
Better layout alignment could
reduce PUE from 3 to 2.5
cold/hot air containment could
reduce PUE from 2.5 to 1.8
Free cooling could reduce PUE from
1.8 to 1.05
Optimizing power feeding
•
•
•
•
Have the right amount of
redundancy
Manage the data center power
consumption
Efficient UPS units and transformers
A DC Data Center is expensive but
very efficient
Optimizing refresh rate
•
•
•
A four-year lifespan has a positive
ROI – server refresh is a good
investment
Part of the data-center CapEX
Improvements in technology allows
to purchase less servers and get
same or high compute power
*PUE numbers are industry estimations and are impacted by
many variables
The total efficiency of a data center should be calculated by comparing the business value it brings to the company divided by
the TCO including run rate costs.
Challenges in Modern Data Centers Management, Spring 2015
31
Data center energy productivity
Courtesy of triplepundit.com – for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
32
Data center energy productivity cont.
• Equipment utilization has the highest impact on energy productivity
• Includes servers, storage, networks, etc.
• In the coming lectures you will learn (a lot) about utilization
• Challenges related to network management (design) in the data center
• Challenges related to resource management in the data center (job scheduling)
• Challenges related to data access in the data center
Challenges in Modern Data Centers Management, Spring 2015
33
Open Compute Project (OCP)
http://www.opencompute.org
Challenges in Modern Data Centers Management, Spring 2015
34
What is OCP?
• Facebook-sponsored program aiming to change the way data-centers
are built in order to scale computing infrastructure in the most
efficient and economical way possible
• Include all DC aspects:
• Storage
• Racks
• Networking
• Server Design
• Physical infrastructures • Hardware management
• All specs and documents are available online and open to all:
• http://www.opencompute.org/projects/
Project is the open source of DC industry with proven results of 38%
less energy use with a 24% cost reduction in DC facilities
Challenges in Modern Data Centers Management, Spring 2015
35
OCP: Design
• Designed in tandem with Open Compute servers
• The data center maximizes mechanical performance and thermal and
electrical efficiency
• Open racks and modified 19’ racks are used
• http://www.opencompute.org/assets/download/Open-Compute-ProjectDeploying-OCP-Hardware-in-a-Colo.pdf
Challenges in Modern Data Centers Management, Spring 2015
36
OCP: Electrical
• OCP data center design is a highly-efficient electrical system
• The facility utilizes an electrical system with a 48VDC UPS system
integrated with a 277VAC server power supply
• 2 choices of power supplies to the servers:
• OCP 700W-SH AC/DC power converter, a single voltage 12.5Vdc, closed frame,
self-cooled power supply used in high efficiency IT applications
• OCP 450W power supply
Challenges in Modern Data Centers Management, Spring 2015
37
OCP: Mechanical
• OCP data centers use a highly-efficient cooling system
• Mechanical system uses 100% airside economization with an evaporative
cooling system
• All PUE data is open to the public and available online
• https://www.facebook.com/PrinevilleDataCenter/app_399244020173259
Challenges in Modern Data Centers Management, Spring 2015
38
OCP: Energy efficiency and green aspects
• OCP server's vanity-free design eliminates nearly 6 pounds of
material per server, reducing the amount of materials that need to
be produced, transported, assembled, and eventually –
disassembled
• OCP servers can operate in a higher-temperature environment,
reducing the overall cooling load required in a data center (all non
relevant parts are removed from server EX front panel
• PUE of 2 largest DCs is 1.1 and 1.08
• WUE (Water usage efficiency is measured in real-time
• Data centers consumes 52% less energy than a data center built to
code requirements, consumes 72% less water for occupant use, and
meets 100% of irrigation needs through rainwater capture
Challenges in Modern Data Centers Management, Spring 2015
39
Visit to Intel’s largest data center outside the US
Courtesy of itsupport24.com– for educational purposes only
Challenges in Modern Data Centers Management, Spring 2015
40