MANAGING DISTRIBUTED UPS ENERGY FOR
EFFECTIVE POWER CAPPING IN DATA CENTERS
Vasileios Kontorinis, L.Zhang, B.Aksanli, J.Sampson, H.Homayoun,
E. Pettis*, D. Tullsen, T. Rosing
*Google
ISCA 2012
UCSD
Datacenter market is growing
2

World is becoming more IT dependent.

Internet users increased from 16% to 30% of world population
in 5 years [Internet World Stats]

Smart phones are projected to jump from 500M in 2011
to 2B in 2015 [Inter.Telecom.Union]

Internet heavily depends on Datacenters

Data center power will double in 5 years

Expected worldwide Datacenter Investment in 2012: 35B$
(equivalent to GDP of Lithuania) [DataCenterDynamics]
Important to build cost-effective Datacenters
Power Oversubscription - Opportunity
3
Datacenter
More servers
Server
Cost
Total Cost of Ownership / Server
No Oversubscription
One time
capital
expenses
Servers
Supporting equipment
Recurring
Costs
With Oversubscription
Same infrastructure
Power Oversubscription More Cost-effective Data centers
Power Oversubscription – Opportunity
4


[Barroso et al. + APC TCO calc]
Assumptions:






Server cost: 1500$
28000 servers (10MW)
Energy: 4.7c/KWh
Power: 12$/kW
Amort. Time DC: 10y, servers: 4y
Distributed LA-based UPS
Available at:
http://cseweb.ucsd.edu/~tullsen/DCmodeling.html
Utility Peak
5.5%
Facility Space
4.5%
Utility Energy
11.7%
Power
Infrastructure
7.9%
Cooling
Infrastructure
3.3%
PUE overhead
2.6%
Server Opex
2.0%
Rest
11.9%
DC opex
9.9%
Server
Depreciation
40.6%
UPS LA
0.2%
Power Oversubscription using Stored Energy
5
Power Profile
Pulse
Model
Shaping
Diurnal Power Profile
Power
Power
Peak
Power
M
Tu
W
Time



…
Peak Power
Pulse
Peak Power
Reduction
Low Power
Pulse
…
UPS stored
Energy
+
_
Su
Time
Leverage diurnal patterns of web services
Discharge UPS batteries during high activity (once per day)
Recharge during high (once per day)
Centralized UPS
6
Used in most small / medium data
centers
 Scales poorly
 High losses in AC-DC-AC conversion
(5-10%)
 Centralized single point of failure,
requires redundancy

X
Increasingly cost-inefficient for large data centers
Distributed UPS
7
Used in large data centers
 Scales with data center size
 Avoids AC-DC-AC conversion
 Distributed points of failure

Facebook
Cheaper UPS solution
Google
Related work and our proposal
8

Utility

Diesel
Generator
UPS

+
_
Centralized UPSs for power
capping [Govindan, ISCA 2011]
Distributed UPSs for rare power
emergencies [Govindan, ASPLOS
2012]
Our proposal:

PDUs
…



Racks

Provision distributed UPS for peak
power capping
Different battery technology
Shave power on daily basis
Place more servers under same
power infrastructure
Better amortize capex costs
Outline
9
Introduction
 Choosing the right battery for power shaving
 Datacenter workload and power modeling
 Policies and results
 Conclusions

Outline
10
Introduction
 Choosing the right battery for power shaving
 Datacenter workload and power modeling
 Policies and results
 Conclusions

Competing Battery Technologies
11

Lead Acid (LA)

Lithium Cobalt
Oxide (LCO)

Lithium Iron
Phosphate (LFP)
Electric
Metrics
12
Backup
 UPS batteries rarely used (3-4 times per year)
 Proper metrics:


Cost
Size
Wh / $
Volumetric Density (Wh / liter)
Backup + peak shaving
 UPS batteries used on daily basis
 Proper metrics:




Charge cycles
Cost
Size
Recharge speed
Wh * cycles / $
Volumetric Density (Wh / litre)
( % charge / hour)
Battery Technology Comparison
13
Backup: Lead Acid (cheaper)
Backup+Peak Shaving: Lithium Iron Phosphate (cost effective)
Battery Capacity-Cost Estimation
Power
14
Peak
Duration
E shaved
Peak
Reduction
Time
LFP
Lead Acid
Assumptions
15
Number of servers
28K
Server Type
Custom Sun Fire X4270
- Intel Xeon (8-core), 8 GB Mem.
- Idle Power: 175W
- Max Power: 350W
PSU efficiency
80%
Workload
Pulse Model, utilization 50%
Batteries
LFP (5$/Ah), LA (2$/Ah)
TCO savings with peak duration
16
LFP
LFP size constraint
LA
LA size constraint
LA
The more we shave, the more we gain!
LFP more space,energy efficient than LA, can shave more!
TCO savings with battery DoD
17

When shaving same energy:
Low DoD
High DoD
+
_
(a) LA
+
_
(b) LFP
Sweet DoD spot for TCO savings (LA: 40%, LFP: 60%)
Key points for battery selection
18
When using batteries for peak power shaving:
 Shave as much power as possible (reasonably sized
battery)
 There is a DoD sweet spot, maximizing TCO savings
 LFP better technology because:




lots of recharges
more efficient discharge
higher energy density
cheaper in the future
What if: - Servers with unbalanced load?
- Day-to-day variation in demand?
Outline
19
Introduction
 Choosing the right battery for power shaving
 Datacenter workload and power modeling
 Policies and results
 Conclusions

Workload Modeling
20
Whole year traffic data from Google Transparency Report
 Apply weights according to web presence:
(Search 29.2%, Social Networking 55.8%, Map Reduce 15%)
 Present results for 3 worst consecutive days
(11/17/2010-11/19/2010)

Workload Modeling (cont.)
21


Model 1000 machine cluster, with 5 PDUs, 10 racks per PDU,
20 servers (2u) per rack.
We simulate load based on M/M/8 queues and scale inter-arrival
time according to workload traffic
Interarrival Time
Job
Job
Job
Job
Job
Job
Job
Job
Job
Job
Job
Job
Job
Job
8 Cores (consumers)/ Server
Job
Job
Job
Job
Job
Scheduler
(Round Robin or Load-aware)
……..
Job
Service
Time
Outline
22
Introduction
 Choosing the right battery for power shaving
 Datacenter workload and power modeling
 Policies and results
 Conclusions

Policy goals
23
Guarantee power budget at specific level of power
hierarchy
 Discharge during only high activity,
charge during only low activity
 Effective irrespective of job scheduling
 Make uniform battery usage

Uncoordinated Policy
24
Power over Threshold
Recharge Complete
Available
In Use
Power below Threshold
Recharge
Reached DoD Goal
Applied at the server level
 Easy to implement
 Runs independently per server
 DoD goal set to 60% of
battery capacity (LFP)

Not
Available
(Power + Bat. Recharge Power) below Threshold
Uncoordinated Policy Results
25
Round Robin Scheduling

Batteries discharge when
not required

Batteries recharge during peak

Fails to guarantee budget
Budget
violation
Uncoordinated Policy Results (cont.)
26
Load-aware Scheduling

Batteries discharge all together
(wasteful)

Recharge all together
(violates budget)

Fails to guarantee budget
Coordination is required!!
Budget
violation
Coordinated Control
27
Applied at higher levels
(PDU, Cluster)
 Requires remote battery
enable/disable, initiate recharge
 Number of batteries enabled
proportional to peak magnitude
 Batteries used spatially
distributed
Overall Power

300 server
100 server
equivalent
equivalent
200 server 200 server
equivalent
equivalent
0 server
equivalent
Day1
Day2
rack1
Day3
rack2
Coordinated Policies
28
Pdu-level
Cluster-level
Power cap close to Average power (ideal) of 250W
Peak power reduction of 19%  23% more servers
 6.2% TCO/server reduction
Discussion: Energy proportionality
29
Modern Servers
Sharper, thinner peaks
 We can shave more power,
with same stored energy

Overall Power
Energy Proporional Servers
Day1
Day2
Day3
Peak power reduction of up to 37.5% with the 40Ah LFP battery
Concluding remarks
30
Battery provisioning of distributed UPS topologies to cap power
and oversubscribe data center is beneficial
 Critical to reconsider battery properties
(technology, capacity, DoD)
 Coordination of charges and discharges is required
 We cap peak power by 19%, allow 23% more servers and
better amortize capex costs
 Achieve 6.2% reduction in TCO/server ($15M -- 28k server DC)

31
BACKUP SLIDES
TCO savings with battery cost
32


LA is stable technology
LFP advancements expected, due to electric vehicles
TCO savings increase over time with LFP!
When things go wrong?
33

Scenario 1: Unexpected daily traffic
We use the additional 35% capacity in our
batteries (DoD optimized for TCO savings at 60%)

Scenario 2: Batteries are not replaced immediately
With 50% of batteries dead we can still reduce
peak by 15%
Grouping battery maintenance/replacement for cost
savings possible
Exploration of Dead Batteries
34
Discussion: DVFS
35
To DVFS or not DVFS?
Datacenter SLAs violations
likely during peak load
 DVFS bad during high demand
 Great during low demand
 Creates higher margins for
aggressive battery capping

Overall Power

Potential SLA violation
WITH
No DVFS
SLA violation unlikely
Day1
Day2
Day3
Battery Capacity-Cost Estimation
36
E Datacenter,shaved =
Power

PeakReduction * PeakDuration
Peak
Duration
E shaved
Peak
Reduction
Time
E server,shaved
= E Datacenter,shaved* PSUEff
# servers

Cbattery
1
1
Eserver,shaved
PE-1
*I
*
*
=
DoD 0.8
V

Cbattery *CostperAh * # servers
UPSdepreciation =
Min(servicelife, DoD(cycles) / 30)

LFP
Lead Acid (~twice volume)
Battery Related Assumptions
37
Workload partitioning
38
Distributed Algorithm
39