Accelerating business innovation; a Technology Strategy Board programme
Energy efficiency and research computing
RUGIT Away Day, 24th Jan 2008
Dave Berry
Deputy Director, NeSC &
Technology Lead, Grid Computing Now!
daveb@nesc.ac.uk
www.gridcomputingnow.org

Contents
Accelerating business innovation; a Technology Strategy Board programme
Background and goals
Desktop Grids for High Throughput Computing
Data centres and HPC centres
Ways forward www.gridcomputingnow.org

Accelerate the benefits to the UK economy of adopting modern computing technologies, i.e.:
The creation of scalable, secure, efficient ICT infrastructures,
For delivering IT services, linked to business processes through a service oriented architecture,
While achieving greater utilisation with reduced energy consumption and reduced costs www.gridcomputingnow.org
GCN! Aims
Accelerating business innovation; a Technology Strategy Board programme

Knowledge Transfer Network
Accelerating business innovation; a Technology Strategy Board programme
Part of the Technology Strategy Board’s Innovation
Programme
Run by Intellect, NeSC and CNR
Activities
Web platform; user case studies; events; webinars; active sector and regional programme
Communities of Practice
Green IT (including MBE KTN, BCS, …)
Transport Modelling, Grids in Health, Public sector IT, … www.gridcomputingnow.org

To expand our community of practice
Stimulate discussion
Share knowledge
Form relationships
Suggest a “best practices” document
Input to policy (Unis, RCs, government)
Want to learn as much as inform
Aims for this session
Accelerating business innovation; a Technology Strategy Board programme www.gridcomputingnow.org

Desktop Grids
Accelerating business innovation; a Technology Strategy Board programme
High Throughput Computing
Many experimental scientists are more interested in number of jobs/month rather than instantaneous computing power
One approach: use “Spare” Cycles
No need for air conditioning, etc.
But don’t we want to switch off machines at night?
Claim: desktop grids can give computing power for less cost and less electricity www.gridcomputingnow.org

Cardiff slides from James Osborne,
High-Throughput Computing Week
NeSC, 27-30 November 2008 Central Manager
Execute Nodes
1600 Workstations master, collector, negotiator
Submit Nodes
30 Workstations master, schedd, shadow master, startd, starter

Based on a P4 3GHz PC with 512MB RAM
Watts Consum ed
80
60
40
20
0
160
140
120
100
0
Off
100
0
Hibernate
5
Standby Idle
Machine State
112
150
Office Condor

Based on a P4 3GHz PC with 512MB RAM
• Makes sound financial sense
• Hibernate saves £60 per year
• Condor = £30 per year (max)
• Dedicated = £150 per year
• Condor is 5 times cheaper
Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24
Cost of Condor = Cost of 150W Electricity (Condor State) – Cost of 100W Electricity (Idle State)
Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

Based on a P4 3GHz PC with 512MB RAM
• Makes sound environmental sense
• Hibernate saves 650Kg CO
2
• Condor = 325Kg CO
2 per year per year (max)
• Dedicated = 1,625Kg CO
2
• Condor is 5 times greener per year
Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24
Cost of Condor = Cost of 150W Electricity (Condor State) – Cost of 100W Electricity (Idle State)
Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

Based on 10,000 P4 3GHz PCs with 512MB RAM
• Makes sound financial sense
– Hibernate would save £600,000 per year
• Hibernate 16 out of 24 hours
• Makes sound environmental sense
– Hibernate would save 6,500T CO
2
– Rainforest required = 52Km 2 per year
– Rainforest required = 40% area of Cardiff
Saving of Hibernate = Cost of 100W Electricity (Idle State) for 16 Hours out of 24
Cost of Condor = Cost of 150W Electricity (Condor State) – Cost of 100W Electricity (Idle State)
Cost of Dedicated = Cost of 150W Electricity (Condor State) + Cost of 100W Electricity (Air Con)

The best of both worlds?
Accelerating business innovation; a Technology Strategy Board programme
What if we could power-off idle machines and wake them up when Condor has jobs to run?
JISC Low Carbon ICT project http://projects.oucs.ox.ac.uk/lowcarbonict/
Develop an institution-wide wake-on-LAN service
Monitor energy consumption across the University
Write and implement a communications strategy
Towards Low Carbon ICT conference
Oxford, 19 th March 2008 www.gridcomputingnow.org

Data Centres and HPC Centres
Accelerating business innovation; a Technology Strategy Board programme
Need: 2xProcessing
Capacity per annum;
Target: 60% Energy
Reduction from 1990 levels by 2050 www.gridcomputingnow.org

Power consumption by data centres:
Estimated 1.5% of UK national electricity generation
Peak consumption of 8GW
Estimated to rise from 46 TWH in 2006 to 93 TWH in 2020
Comparable with consumption by refrigeration…
… or greenhouse gases emitted by aviation
The scale of the problem
Accelerating business innovation; a Technology Strategy Board programme www.gridcomputingnow.org

Technology Power Loss Chain:
Fossil Fuel – CPU Used
Fossil Fuel
Data Centre
Equipment
Servers
35%
40%
65%
95%
25%
65%
35%
20%
2.5
%
2.5
%
15%
CPU
30%
CPU Utilisation
20%
0.5% of Fossil Fuel Energy
45%
80%
25%
Slide from BCS DCSG
Heat Exhausted
Electricity Generated
Transmission Losses
Transformer Losses
Data Centre
Cooling Losses
Power Infrastructure
IT Equipment
Network Equipment
Storage Equipment
Servers
Power Supply
Other Components
CPU
Idle Time Power
CPU Load Power

Policies and measurement
Accelerating business innovation; a Technology Strategy Board programme
BCS DCSG model
Open source model of energy-efficiency for data centres and servers
The Green Grid
Vendor consortium
EU code of conduct on data centres
Development of baseline measures
Voluntary contribution of data from subscribing organisations www.gridcomputingnow.org

Press Examples
Accelerating business innovation; a Technology Strategy Board programme
Ultraspeed DC-based system in East London
Claims 30% power saving from use of DC
Extra 10% saving from diskless servers http://pcworld.about.com/od/recyclin1/Data-center-claimspower-cuts.htm
Plan for green data farm in Lockerbie
Using renewable energy sources
Waste heat used to heat new “eco village” http://www.redwasp.co.uk/newsitem.asp?id=280 www.gridcomputingnow.org

GCN! Webinar
Accelerating business innovation; a Technology Strategy Board programme
The Business Case and Methods for the Green Data
Centre
Recording available on the web
Zahl Limbuwala
Chair, BCS Data Centre Specialist Group
Motivation and an introduction to the BCS model
Kate Craig Wood
Managing Director, Memset Ltd.
Practical steps to running a “carbon-neutral” data centre www.gridcomputingnow.org

Finance and administration
Accelerating business innovation; a Technology Strategy Board programme
Who sees the power bills?
Who calculates the total cost of ownership?
Does this affect purchasing decisions?
Trade-off: cost of reliability vs. cost of downtime
Don’t overspecify
Staff requirements
Specialised HPC kit may need specialised staff
Other kit may not www.gridcomputingnow.org

Power engineering
Accelerating business innovation; a Technology Strategy Board programme
Location
Near power stations
Or local generation (CHP?)
AC or DC?
Provisioning levels
Nameplate provisioning is inefficient
Cables
Oversize for lower resistance (can halve losses)
Route through cool underfloor area www.gridcomputingnow.org

Dell 1U server power usage
2006 SC1425 Dual
Xeon
2007 860 Single dual core Xeon
0 100 200
Watts
300 400 500
Idle
Max load
Label
Slide from Kate Craig-Wood


Latest switch-mode based generation vastly more efficient

 96-96% vs. ~90%
Most efficient when fully loaded (98%)

Don't run at half-capacity

40KVA steps, rather than typical 500KVA steps
Slide from Kate Craig-Wood

Cooling
Accelerating business innovation; a Technology Strategy Board programme
What target temperature?
Water or air?
Store heated water externally?
External air or recycled air?
Savings from fresh air cooling of 72% (Source: BCS)
Modular cooling
Cabinets or rooms?
Modelling and controlling heat flow
Waste heat
How to sell or reuse?
www.gridcomputingnow.org

www.gridcomputingnow.org
Fresh Air Cooling
Accelerating business innovation; a Technology Strategy Board programme

State of the art?
Accelerating business innovation; a Technology Strategy Board programme
“By careful design, matching the specification of the plant to the needs of the machine(s) we have brought the summer time cooling overhead down from 60 - 65% (very typical of most computer rooms) to 25 - 30%.
Further, by using freecool this drops to 7% in the winter.” www.gridcomputingnow.org

System utilisation
Accelerating business innovation; a Technology Strategy Board programme
Server virtualisation
Load balancing
Note – this is typically done already for compute clusters
May still apply to other university systems
Choice of equipment
Multi-core, power management, etc.
Diskless servers www.gridcomputingnow.org

Best practice document(s)
Perhaps proven + experimental?
Shared facilities
Shared data centres?
Shared compute clusters (a la NGS)?
Training for IT staff?
Case studies?
Input to policy (Unis, RCs, government)
Ways forward
Accelerating business innovation; a Technology Strategy Board programme www.gridcomputingnow.org

Processor:
Memory:
Disks:
600
500
400
300
200
100
Power Report for <VENDOR> <DEVICE> <MODEL>
Under <BENCHMARK>
2 x 2.8GHz Quad Core
4 x 2GB 667MHz
2 x 146GB SAS
IO Cards:
IO Cards:
PSU:
2 x 10W PCI-E 10GBE
2 x 18W PCI-E FC
2 x 600W
Load (%) Sleep Idle 0 10 20 30 40 50 60 70 80 90 100 Peak PSU
Load
Power
Sleep Idle 0 10 20 30 40 50 60 70 80 90 100 Peak
25 190 243 267 286 304 320 334 347 360 371 381 390 440
PSU
600
BCS Data Centre Specialist Group
Slide from BCS DCSG

More references
Accelerating business innovation; a Technology Strategy Board programme
GCN! Webinar http://tinyurl.com/2gtslj
BCS Data Centre Specialist Group http://tinyurl.com/2dyy5t
EU Code of Conduct http://tinyurl.com/2drxoh
HTC week http://www.nesc.ac.uk/action/esi/contribution.cfm?Title=831
Technology Strategy Board http://www.berr.gov.uk/innovation/technologystrategyboard www.gridcomputingnow.org