!
!
!
!
!
WHITEPAPER
!
Executive0Summary0...........................................................................................................03!
The0CIO0view:0a0combination0of0cost0pressures,0risk,0and0opportunity0........................03!
The!information!factory!.........................................................................................................!4!
Summary!..............................................................................................................................!6!
The0explosion0of0data:0generation,0storage,0and0the0need0for0realHtime0processing0...06!
The!number!of!both!megabytes!and!containers!is!growing!..................................................!7!
The!value!of!realEtime!processing!........................................................................................!7!
Supercomputing!will!aid!weather!forecasting!.......................................................................!8!
More!powerful!computing!will!improve!management!of!the!electric!power!grid!...................!9!
Summary!............................................................................................................................!11!
The0challenge0for0the0electric0power0grid0......................................................................011!
The!challenge!for!power!planners!......................................................................................!11!
The!power!consumption!profiles!of!emerging!supercomputers!..........................................!12!
Summary!............................................................................................................................!13!
Clean0Sources:0is0green0always0fully0green?0................................................................013!
Source!–!100!percent!renewables!are!not!exactly!the!same!as!carbon!free!.....................!13!
Hydropower!and!geothermal!E!ideal!power!resources!for!a!low!carbon!computing!
environment!........................................................................................................................!14!
Summary!............................................................................................................................!14!
Reliability:0where0to0find0robust0and0reliable0grids?0.....................................................014!
The!UK:!Struggling!to!maintain!reliability!...........................................................................!15!
Continental!Europe:!Their!own!challenges!.........................................................................!17!
Germany:!in!a!state!of!expensive!transition!.......................................................................!17!
The!United!States!grid:!undergoing!a!structural!shift!.........................................................!19!
The!Nordic!markets:!largely!blessed!with!abundant!renewables!&!relatively!stable!grids!.!21!
Quebec:!A!Hydro!superpower!in!North!America!................................................................!22!
Summary!............................................................................................................................!22!
Cost:0look0for0hydropower0and0cooler0climates0............................................................022!
Summary!............................................................................................................................!25!
Conclusion0........................................................................................................................025!
Appendix0A:0Future0Coordination0with0power0grid0planners0and0operators0..............027!
!
!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!2!
!
WHITEPAPER
!
EXECUTIVE0SUMMARY00
0
Today’s!CIO!faces!a!host!of!challenges!and!new!trends.!Budgets!are!tight!and!in!many!cases!
decreasing.!At!the!same!time,!IT!is!becoming!more!strategic!as!the!amount!of!data!created!is!
exploding!and!the!value!of!accessing!that!data!in!real!time!is!continuously!increasing.!Much!of!this!
data!must!be!mined!and!acted!upon!instantaneously!while!it!is!still!of!value,!requiring!the!application!
of!bigger!and!more!powerful!data!centers.!Many!companies!cannot!meet!this!challenge!costE
effectively!inEhouse,!and!are!therefore!increasingly!resorting!to!outsourcing!operations!to!larger!and!
centralized!data!centers!around!the!world.!However,!while!they!support!the!virtual!world!and!the!
amorphous!cloud,!data!centers!exist!in!the!real!world.!One!of!the!main!challenges!they!face!is!
sourcing!clean,!reliable,!and!affordable!electricity.!This!is!critical,!as!the!power!grid!has!become!the!
central!nervous!system!of!the!dataEdriven!economy.!
!
However,!many!power!grids!are!facing!unprecedented!challenges.!They!are!having!to!cope!with!
aging!infrastructure,!reliability!issues,!cost!pressures!and!a!mandate!to!decarbonize!electricity!
supply!resources.!In!many!of!the!world’s!more!advanced!power!grids,!the!advent!of!large!quantities!
of!intermittent!renewable!resources!E!such!as!wind!and!solar,!coupled!with!the!retirement!of!many!
coalEfired!facilities!–!add!to!the!challenge.!With!many!of!these!renewable!resources!distributed!
behind!the!meter!at!the!customer!premise,!planning!and!revenue!generation!become!ever!more!
difficult.!In!this!complex!planning!environment,!many!utilities!face!significant!uncertainties!and!
difficulties!in!planning!new!resource!additions!and!keeping!costs!down.!In!some!regions,!this!poses!
new!challenges!to!grid!reliability!as!well!as!cost.!!
!
For!the!CIO!considering!outsourcing!to!a!data!center,!adequacy!of!electricity!resources!must!
become!part!of!the!decisionEmaking!process.!The!questions!then!become!about!the!power!source,!
reliability,!and!cost!of!operations.!Where!can!CIOs!find!data!centers!that!provide!them!with!clean!
and!lowEcost!power,!the!right!climate,!and!stable!and!secure!grids!to!meet!their!needs?!In!fact,!as!
we!shall!see,!there!are!only!a!small!number!of!markets!that!meet!this!combination!of!requirements.!!
THE0CIO0VIEW:0A0COMBINATION0OF0COST0PRESSURES,0RISK,0AND0
OPPORTUNITY0
Today,!our!globally!connected!world!is!becoming!increasingly!dataEintensive,!and!the!value!of!
information!is!becoming!ever!more!critical.!Ironically,!at!the!same!time!that!more!is!being!asked!of!
1
them,!CIOs!have!been!facing!static!–!or!even!declining!E!IT!budgets. !IT!groups!are!under!
continuing!pressure!to!do!more!with!less.!They!must!squeeze!all!available!productivity!out!of!their!
information!technology!assets,!increasing!the!burden!on!data!centers!to!keep!pumping!out!valuable!
and!actionable!information.!Analytics!are!emerging!as!an!increasingly!vital!part!of!many!industries,!
whether!this!involves!the!mining!and!assessment!of!customer!data,!monitoring!and!optimization!of!
machineEtoEmachine!activities,!or!predicting!future!outcomes.!The!value!is!becoming!increasingly!
obvious,!so!that!close!to!50!percent!of!CIOs!interviewed!in!2014!indicated!that!they!are!adopting!
2
analytics. !However,!while!there!is!tremendous!upside!to!optimizing!data!and!information!flows,!
there!is!also!enormous!potential!for!loss!if!the!information!stops!flowing.!!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!3!
!
WHITEPAPER
!
The0information0factory0
At!the!heart!of!this!IT!enterprise!lies!the!data!center,!the!information!factory!that!stores!and!
processes!data!and!creates!value!in!countless!ways.!Businesses!cannot!function!without!quick!
access!to!critical!information.!If!that!information!factory!stops!producing!useful!and!usable!data,!the!
pulse!of!the!business!stops.!And!these!data!centers!rely!on!a!continuous!feed!of!power!from!their!
electric!utility.!If!the!grid!goes!down,!data!centers!can!be!quite!vulnerable.!This!is!particularly!the!
case!because!backEup!systems!do!not!always!perform!as!anticipated!and!some!level!of!failure!is!in!
fact!quite!probable.!!
!
A!recent!survey!(2013)!on!U.S.!data!center!outages!reveals!some!disturbing!facts.!Of!the!584!data!
center!professionals!surveyed,!85!percent!reported!a!loss!of!primary!utility!power!in!the!past!24!
3
months. !And!16!percent!of!those!individuals!indicated!more!than!six!outages!per!year.!
!
4
Figure01:0Frequency0of0Utility0Outages0Experienced 0
60%
50%
49%
41%
40%
35%36%
30%
20%
13%
10%
10%
10%
5%
0% 0%
1% 0%
0%
Between!1!
and!2
Between!2!
and!5
Between!6!
and!10
FY!2013
Between!11! Between!16! More!than!20
and!15
and!20
FY!2010
!
!
The!average!frequency!of!utility!power!outage!was!once!every!six!months,!with!an!extrapolated!
5
average!duration!of!138!minutes!(up!from!105!minutes!in!the!2010!study). !While!half!of!those!
outages!lasted!less!than!five!minutes,!13!percent!exceeded!two!hours.!When!that!occurs,!data!
centers!switch!to!backEup!power,!but!that!transition!does!not!always!occur!smoothly!or!without!
difficulty.!In0fact,0of0those0who0reported0losing0utility0power,0910percent0reported0an0
6
unplanned0data0center0outage. !!
!
!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!4!
!
WHITEPAPER
!
7
Figure02:0Average0Duration0of0Power0Outages 0
45%
42%
40%
34%
35%
30%
25%
25%
20%
20%
16%17%
15%
12%11%
10%
10%
8%
3% 2%
5%
0%
<!1!minute
1!to!5!minutes
5!to!20!
minutes
20!minutes!to! 2!hours!to!1!
2!hours
day
FY!2013
FY2010
>!1!day
!
!
The!costs!of!an!hourly!shutdown!were!estimated!on!average!to!be!just!over!$200,000,!with!six!
percent!of!respondents!indicating!estimated!hourly!losses!exceeding!one!million!dollars.!Some!
industries!pay!more!for!failure!than!others,!and!experience!has!shown!that!the!cost!of!a!single!event!
can!indeed!be!quite!high.!In!2013!for!example,!Sears!Holdings!experienced!two!power!failures!at!its!
main!data!center!in!Troy,!Michigan,!resulting!in!estimated!lost!profits!of!over!$2!million.!The!first!
!8
fiveEhour!failure!during!the!postEholiday!period!in!January!resulted!in!hourly!losses!of!$300,000. !
!
Depending!on!the!business,!the!impacts!can!occur!immediately.!In!a!recent!survey!of!just!over!300!
global!IT!professionals,!17!percent!of!respondents!indicated!that!their!operations!were!affected!
immediately!by!downtime,!while!another!11!percent!felt!the!impact!within!the!first!five!minutes.!!
!
!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!5!
!
WHITEPAPER
!
9
Figure03:0The0Impact0of0IT0Downtime0on0Businesses 0
!
For0your0company,0approximately0how0many0minutes0of0IT0downtime0can0
occur0before0the0business0is0negatively0impacted?!
!
35%
29%
30%
25%
20%
17%
15%
17%
16%
11%
10%
6%
5%
4%
0%
Zero,!our! 1!E 5!minutes 6!E 15!minutes
business!is!
instantly!
affected!by!
downtime
15!E 60!
minutes
1!E 3!hours
3!E 6!hours
More!than!6!
hours
!
!
!
If!the!grid!goes!down,!data!centers!often!face!significant!risk.!Their!own!UPS!backEups!won’t!always!
work!and!other!unexpected!issues!may!occur.!The!key,!then,!is!to!try!and!minimize!the!likelihood!of!
system!power!outages.!!
Summary0
Electricity!grid!performance!and!power!reliability!matter,!since!the!cost!of!outages!can!be!crippling.!
Some!grids!are!more!reliable!than!others.!When!choosing!data!center!locations,!power!grid!
reliability!should!be!one!of!the!factors!considered.!!
THE0EXPLOSION0OF0DATA:0GENERATION,0STORAGE,0AND0THE0NEED0FOR0
REALHTIME0PROCESSING0
Power!reliability!will!matter!even!more!in!future!years!as!society!becomes!increasingly!digitized.!
The!last!decade!has!seen!an!astonishing!growth!in!production,!storage,!and!consumption!of!data.!
And!yet,!the!global!economy!is!about!to!experience!an!acceleration!of!this!trend.!A!recent!report!on!
the!topic!indicates!that!the!number!of!digital!bits!doubles!every!two!years,!and!will!increase!from!4.4!
10
zetabytes!in!2013!to!44!zetabytes!in!2020. !
!
The!table!below!highlights!one!major!contributing!factor:!the!global!growth!in!devices!–!each!of!
which!will!generate!enormous!and!endless!streams!of!data.!However,!while!the!growth!in!devices!is!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!6!
!
WHITEPAPER
!
significant,!the!biggest!driver!of!future!growth!is!projected!to!be!‘the!Internet!of!Things!(IoT),!which!
includes!machineEtoEmachine!(M2M)!communications!and!a!growing!network!of!ubiquitous!sensors.!
Today,!there!are!some!200!billion!‘things’!in!the!world,!with!about!20!billion!of!them!currently!
11
connected!and!communicatingk!this!number!is!expected!to!grow!to!30!billion!by!2020. !
!
12
Figure04:0The0Growing0Number0of0Devices 0
!
!
The0number0of0both0megabytes0and0containers0is0growing0
The!number!of!megabytes!that!must!be!stored!is!not!the!only!challenge!that!will!arise!from!the!IoT!
and!the!embedded!systems!that!contain!the!sensors!and!systems!that!monitor!and!measure!the!
physical!environment.!In!fact,!the!number!of!containers!or!‘files’!the!data!comes!in!is!growing!at!a!
much!faster!pace.!These!files!may!contain!large!amounts!of!information!or!as!little!as!that!
associated!with!an!RFID!tag!from!a!retail!store!with!only!32!bytes.!The!growth!rate!of!these!
13
containers!is!likely!to!be!staggering:!from!28!quadrillion!in!2014!to!4,200!quadrillion!by!2020. !
!
And!we!are!hesitant!to!throw!most!of!this!data!away.!As!far!back!as!1980,!one!observer!noted!E!
“Those!associated!with!storage!devices!long!ago!realized!that!Parkinson’s!First!Law!may!be!
14
paraphrased!to!describe!our!industry!–!‘Data!expands!to!fill!the!space!available.’” !Thus,!
increasingly!more!storage!capability!will!be!required.!At!the!same!time,!we!will!want!to!access!
increasingly!more!data!in!realEtime.!
The0value0of0realHtime0processing0
There!is!considerable!benefit!to!be!derived!if!we!can!better!analyze!and!extract!more!value!from!the!
data!we!create.!IBM!estimates!that!90!percent!of!all!data!generated!by!connected!devices!such!as!
electric!vehicles,!smartphones,!and!tablets!is!never!analyzed,!with!60!percent!of!this!data!losing!all!
15
value!almost!immediately!after!it!is!created.! !!
!
A!recent!study!on!the!topic!observed!that!in!2013,!only!22!percent!of!the!information!in!the!digital!
universe!would!even!be!a!candidate!for!analysis,!i.e.,!useful!if!it!were!tagged.!And!less!than!5!
percent!of!that!was!actually!analyzed.!However,!by!2020,!the!useful!percentage!could!grow!to!more!
16
than!35!percent,!mostly!because!of!the!growth!of!data!from!embedded!systems. !It!is!this!data!E!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!7!
!
WHITEPAPER
!
appropriately!analyzed!and!acted!upon!E!that!has!enormous!potential!economic!value!to!society.!
Cisco!has!reportedly!identified!14.4!trillion!dollars!of!global!value!over!the!coming!decade!through!
17
connecting!people!to!people,!people!to!machines,!and!machines!to!machines. !
!
!As!a!consequence,!global!interest!in!big!data!continues!to!grow.!In!fact,!a!2014!Gartner!survey!of!
302!‘Research!Circle’!members!worldwide!found!that!73!percent!of!those!surveyed!either!have!
invested!in!or!plan!to!invest!in!big!data!within!the!following!24!months!(up!from!64!percent!in!the!
18
prior!year).! !!
!
Processing!large!data!sets!to!extract!useful!information!will!drive!continued!growth!in!data!centers!
which!can!both!store!and!process!data,!and!lead!to!an!ever!larger!number!of!supercomputer!data!
centers!as!well!as!high!performance!computers!(HPCs).!An!increasing!amount!of!processing!and!
analysis!will!happen!in!real!time!(or!close!to!realEtime),!particularly!in!industries!where!the!data!
become!useless!if!not!analyzed!and!acted!upon!immediately.!!
!
A!few!examples!help!to!highlight!this!trend.!Weather!forecasting!is!becoming!increasingly!granular!
and!extremely!data!intensive,!with!the!need!for!highly!accurate!and!granular!forecasts.!The!electric!
power!grid!is!on!the!verge!of!getting!‘smart’!with!the!ability!of!millions!E!if!not!billions!E!of!devices!to!
respond!to!nodal!market!signals!and!localized!grid!conditions!in!real!time.!Medical!research!is!also!
increasingly!relying!on!supercomputing!capability,!and!other!industries!are!also!experiencing!similar!
gains.!But!this!dynamic!is!only!just!getting!started.!A!brief!overview!of!two!examples!–!weather!
forecasting!and!power!grid!management!–!will!serve!to!highlight!the!value!of!realEtime!analytics,!the!
need!for!big!data,!and!the!imperative!for!more!powerful!(and!powerEhungry)!computers.!
Supercomputing0will0aid0weather0forecasting00
Weather!forecasting!is!one!field!that!benefits!greatly!from!the!ability!to!process!increasingly!larger!
data!sets!more!quickly.!All!of!the!world’s!major!weather!services!are!continuously!working!to!create!
higher!resolution!weather!predictions!(more!locally!accurate),!and!to!look!out!further!into!the!future,!
as!far!out!as!30!days.!!
!
As!a!consequence,!these!institutions!are!making!big!bets!on!supercomputing.!The!U.S.!National!
Oceanic!and!Atmospheric!Administration!(NOAA)!announced!in!January!of!this!year!an!investment!
of!$44.5!million!to!increase!its!computing!capacity!to!five!petaflops!(two!machines,!each!at!2.5)!–!a!
19
tenEfold!increase!over!current!capabilities. !!
!
Similarly,!the!British!government!announced!in!2014!plans!to!invest!97!million!British!Pounds!
Sterling!in!a!supercomputer!13!times!more!powerful!than!the!current!system,!projecting!that!the!
increased!accuracy!of!forecasts!will!deliver!£2bn!of!socioEeconomic!benefits.!It!will!be!able!to!
deliver!highly!localized!forecasts,!for!example,!to!determine!the!likelihood!and!timing!of!fog!at!
20
airports!or!localized!impacts!of!extreme!weather!events. !
!
For!its!part,!the!European!Center!for!Medium!Range!Forecasting!recently!upgraded!to!two!
supercomputer!clusters,!each!of!which!has!over!3.5!petaflops!of!capability,!as!well!as!six!petabytes!
of!high!performance!parallel!storage,!and!38!terabytes!of!general!purpose!storage,!more!than!twice!
21
the!power!of!the!previous!machines. !
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!8!
!
WHITEPAPER
!
This!trend!will!only!increase!in!the!future.!Weather!forecasting!computer!models!will!utilize!everE
22
smaller!grid!boxes,!providing!increasingly!higher!resolution. !There!is!value!in!knowing!even!further!
out,!and!with!more!localized!granularity,!what!the!future!weather!will!likely!be,!whether!it!be!for!
safety!during!storms,!or!knowing!when!to!sow!and!harvest!crops.!!
!
23
Figure05:0More0Data0Increases0Forecast0Accuracy
!
!
Climate!change!modeling!will!also!be!vastly!improved!and!accelarated!by!stronger!and!faster!
machines.!The!17!petaflop!machine!at!Oak!Ridge!National!Laboratories,!for!example,!helps!
researchers!by!simulating!from!one!to!five!years!of!computing!time!in!a!day,!compared!with!three!
months!of!simulation!per!day!in!the!computer!it!replaced.!This!is!critical!in!understanding!not!only!
24
whether!the!earth’s!climate!is!being!altered,!but!where!and!how. !
More0powerful0computing0will0improve0management0of0the0electric0power0grid00
The!world’s!electric!power!grids!will!also!greatly!benefit!from!enhanced!computing!capabilities.!
Many!national!grids!involve!thousands!of!miles!of!transmission!lines,!countless!transformers,!
millions!of!miles!of!distribution!lines,!and!highly!varied!localized!grid!conditions.!The!emerging!grid!
of!tomorrow!will!include!millions!of!smart!meters,!capable!of!storing!and!sending!back!high!granular!
data!on!energy!consumption.!!
!
At!the!same!time,!distributed!energy!resources!at!the!far!end!of!the!grid!–!on!the!customer!premise!
–!are!rapidly!proliferating.!In!Australia,!for!example,!there!are!already!over!one!million!rooftop!solar!
installations!–!which!not!only!supply!electricity!to!the!local!premise,!but!also!export!power!back!to!
25
26
the!grid!in!times!of!surplus. !In!Hawaii,!over!50,000!customers!have!rooftop!solar!installations. !
This!phenomenon!of!distributed!energy!resources!will!only!accelerate!as!technologies!get!cheaper,!
but!these!resources!must!be!better!integrated!into!the!power!grid!and!optimized!for!maximum!
benefit.!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!9!
!
WHITEPAPER
!
Energy!storage!(including!electric!vehicles)!is!also!growing!rapidly.!California!now!has!over!100,000!
27
electric!vehicles,!while!Tesla!has!emerged!as!the!single!bestEselling!car!in!Norway. !This!rapidly!
growing!number!of!assets!will!have!to!be!tracked!and!integrated!into!the!power!grid.!Failure!to!do!
so!could!lead!to!grid!instability,!including!brownEouts!and!blackEouts.!Meanwhile,!effective!grid!
integration!brings!great!economic!potential.!GTM!Research!has!estimated!the!total!global!
28
cumulative!value!of!utility!data!analytics!at!$20!billion!between!2013!and!2020. !
!
RealEtime!grid!awareness!(‘situational!awareness’),!with!millions!of!distributed!assets,!poses!a!very!
significant!challenge.!Ultimately,!the!goal!will!be!to!coordinate!the!behaviors!of!numerous!assets!on!
the!‘smart!grid’!so!that!they!are!optimized!form!a!framework!that!takes!economic!value,!reliability,!
safety,!and!other!critical!factors!into!account.!!
!
Predictive!analytics!–!the!ability!to!forecast!ahead!and!to!accurately!predict!outcomes!–!will!add!
even!more!value!to!the!grid.!However,!as!the!chart!below!illustrates,!all!of!this!will!require!the!ability!
to!absorb!and!process!far!more!data!than!we!do!today.!
!
29
Figure06:0The0Growth0of0Data0in0the0Power0Industry 0
!!
!
Recognizing!the!challenges!inherent!in!this!sea!change,!the!U.S.!National!Renewable!Energy!
30
31
Laboratory!has!developed!a!supercomputer !to!aid!in!this!task. !The!Peregrine!machine!supports!
the!Department!of!Energy’s!(DOE)!Energy!Systems!Integration!Facility,!which!is!specifically!
devoted!to!helping!understand!issues!related!to!integration!of!various!energy!resources,!and!
32
developing!a!map!for!the!future!of!the!electric!power!grid. !
!
In!Europe,!similar!efforts!are!underway.!ITESLA!(an!organization!of!six!European!transmission!
service!operators)!is!utilizing!HPCs!to!assess!the!security!of!the!panEEuropean!transmission!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!10!
!
WHITEPAPER
!
network,!running!dynamic!simulations!from!two!days!out!to!realEtime.!Among!other!outcomes,!this!
effort!helps!to!increase!the!share!of!renewables!that!can!be!introduced!into!the!grid.!This!work!is!
33
partly!being!done!to!prepare!for!the!proposed!unified!European!energy!market. !
Summary0
As!both!the!quantity!of!the!data!and!the!value!of!processing!that!data!quickly!increases!in!the!
coming!years,!downtime!will!be!even!more!costly.!This!further!reinforces!the!fact!that!the!reliability!
of!the!power!grid!will!become!an!increasingly!important!element!to!consider!in!data!center!siting!
considerations.!
THE0CHALLENGE0FOR0THE0ELECTRIC0POWER0GRID0
As!data!quantities!grow,!so!too!will!the!requirement!for!electricity!to!store!and!process!the!data.!
While!increasing!data!center!efficiencies!are!helpful,!they!are!unlikely!to!keep!pace!with!the!surging!
demand!for!data!storage,!analytics,!and!information.!As!a!result,!overall!energy!use!–!in!terms!of!
megawatthours!(MWh)!–!can!be!expected!to!grow.!In!fact,!data!centers!represent!one!of!the!fastest!
growing!users!of!electricity!in!the!world.!Globally,!it!is!estimated!that!nearly!3!percent!of!all!
electricity!use!is!dedicated!to!some!kind!of!data!center,!an!increase!of!100!percent!over!the!past!
34
five!years. !!
!
It!is!estimated!that!in!the!U.S.!alone,!approximately!91!billion!kilowattEhours!(kWh)!were!dedicated!
to!feeding!data!centers!in!2013.!This!would!be!equal!to!the!output!of!about!thirtyEfour!500!MW!
power!plants.!Put!another!way,!it!would!be!equivalent!to!twice!the!power!use!of!all!households!in!
New!York!City.!That!number!is!forecasted!to!increase!by!almost!60!percent,!to!140!billion!kWh!–!
about!the!output!of!50!power!plants!E!by!the!end!of!the!decade.35!And!that’s!just!the!United!States.!
Globally,!the!energy!used!to!support!the!Internet!and!our!dataErich!economy!is!equivalent!to!the!
36
power!consumption!of!a!country!somewhere!in!size!between!Italy!and!Spain. !With!the!burgeoning!
IoT,!that!demand!will!only!increase.!A!recent!study!projects!that!in!just!the!next!four!years,!the!IoT!
will!require!750!percent!more!data!center!capacity!in!serviceEprovider!facilities!than!is!currently!
37
being!used. !
!
The!Lakeside!Technology!Center!(LTC)!in!Chicago!illustrates!just!how!much!power!a!large!data!
center!can!consume.!Second!only!to!O’Hare!Airport!in!its!demand!on!the!local!utility,!the!LTC!1.1!
million!square!foot!facility!draws!more!than!100!MW!of!electricity.38!
!
Not!to!be!outdone,!the!huge!SUPERNAP!data!center!in!Las!Vegas!Nevada!draws!approximately!
200!MW!of!power.39!When!all!of!the!SUPERNAP!facilities!planned!or!under!construction!are!built!
out,!they!will!consume!over!500!MW40!–!equivalent!to!just!under!half!of!the!capacity!of!NVPower’s!
(the!local!utility)!largest!generating!station.41!
The0challenge0for0power0planners0
As!noted!above,!many!datasets!lose!their!value!if!not!immediately!tagged!and!analyzed.!With!
increasing!data!streams!from!the!connected!IoT,!realEtime!analytics!will!become!ever!more!critical.!
In!other!words,!two!things!are!happening!simultaneously:!the!overall!level!of!data!is!skyrocketing,!
while!at!the!same!time!more!of!it!will!be!subjected!to!realEtime!analytics.!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!11!
!
WHITEPAPER
!
!
As!a!result,!observers!forecasts!a!growing!level!of!stress!on!most!enterprise!and!service!provider!
networks.!One!analyst!indicates!that,!“The!growing!importance!of!analytics!in!IoT!services!will!
ensure!that!hyperscale!data!centers!are!a!major!component!of!most!IoT!service!offerings!by!
2019.”42!!
!
So!how!will!the!power!grids!respond!to!the!power!demands!of!this!burgeoning!information!
processing!industry?!Certainly,!electricity!grids!will!require!new!generating!resources!and!the!
energy!to!support!data!centers.!That!in!itself!may!not!be!impossible.!A!new!natural!gasEfired!
43
generating!plant!can!be!built!in!24!months !(assuming!permitting,!which!is!often!the!lengthiest!part!
of!the!process,!especially!in!denser!urban!areask!and!ignoring!any!issues!related!to!carbon.)!
!
However,!just!as!important!as!the!baseload!(24x7)!energy!will!be!the!issue!of!generating!capacity!
and!supporting!transmission!infrastructure.!Grids!will!not!only!need!to!be!able!to!generate!and!
transmit!enough!electrons,!but!also!to!deliver!power!exactly!when!it!is!needed!in!order!to!support!
realEtime!data!processing!requirements!that!support!our!IoT!environment.!!
!
For!electric!power!planners,!this!challenge!involves!two!dimensions:!1)!ensuring!the!power!
infrastructure!will!be!there!to!support!whatever!is!requiredk!2)!coordinating!with!data!centers!and!
HPC!operations!to!optimize!computer!processing!loads!with!respect!to!both!overall!energy!
consumption!as!well!as!realEtime!demands!on!the!system.!!
!
For!the!latter,!it!will!be!critical!to!understand!which!high!value!applications!must!be!processed!
immediately,!and!those!that!are!less!timeEsensitive!and!can!be!processed!at!other!times.!The!goal!
will!be!to!spread!out!those!processing!loads!wherever!possible!to!minimize!stress!on!the!grid!and!
reduce!overall!electrical!costs.!
!
Two!things!seem!fairly!certain:!!
1)! Today’s!machines!and!data!centers!are!only!a!precursor!of!what!is!likely!to!come!in!the!
very!near!future.!!
2)! The!number!of!large!machines!and!high!performance!data!centers!will!proliferate!as!
more!value!is!extracted!from!rapid!analysis!of!data.!This!will!add!more!challenges!to!the!
stability!of!electric!power!grids.!
The0power0consumption0profiles0of0emerging0supercomputers0
U.S.!DOE!noted!in!a!recent!report!that,!“The!computing!environment!has!begun!to!change!as!the!
complexity!of!computing!problems!grows,!and!with!the!explosion!of!data...The!need!to!extract!
useful!information!from!this!explosion!of!data!becomes!as!important!as!sheer!computational!
44
power.” !
!
According!to!the!report,!these!new!machines!will!need!to!perform!a!variety!of!tasks,!such!as!
quantifying!“the!uncertainty!of!associated!with!the!behavior!of!complex!systemsEofEsystems”!(such!
as!hurricanes!or!nuclear!disasters)!and!predict!outcomes.!These!computers!will!also!be!able!to!
learn,!iterate,!and!refine!existing!models,!while!accommodating!the!ability!to!entertain!“what!if”!
45
questions. !
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!12!
!
WHITEPAPER
!
The!DOE’s!explicit!goal!is!to!develop!an!exaEscale!machine!capable!of!1,000!petaflops!(three!times!
greater!than!the!total!combined!capability!of!the!world’s!top!500!machines!as!of!the!end!of!2014.)!In!
terms!of!energy!use,!the!goal!will!be!to!limit!consumption!to!20!megawatts!(MW!E!to!put!that!in!
perspective,!that!is!the!equivalent!to!the!power!consumption!of!four!thousand!homes!in!Texas!
46
during!their!peak!demand.) !To!reach!that!20!MW!goal,!current!efficiencies!will!have!to!increase!
dramatically.!In!today’s!world,!the!best!petascale!machines!–!such!as!NREL’s!Peregrine)!function!at!
a!ratio!of!one!petaflop!per!MW.!Thus,!the!challenge!is!quite!formidable:!to!increase!the!capability!
1000x,!while!increasing!energy!consumption!only!twentyEfold.!Indeed,!the!goal!of!20!MW!may!take!
some!time!to!achieve.!By!one!estimate,!the!theoretical!peak!power!requirements!will!exceed!45!
47
MW. !
Summary0
From!an!energy!reliability!perspective,!we!are!looking!at!a!moving!target,!with!data!centers!putting!
more!stress!on!an!already!brittle!power!system.!The!questions!to!be!answered!are!not!only!“Will!
there!be!enough!electricity?”!but!“Will!it!be!there!when!my!data!center!needs!it?"!Some!countries!
and!regions!are!already!struggling!with!this!problem.!They!are!also!grappling!with!the!growing!issue!
of!carbon,!as!discussed!in!the!following!section.!
CLEAN0SOURCES:0IS0GREEN0ALWAYS0FULLY0GREEN?00
As!noted!in!the!Executive!Summary,!CIOs!are!looking!for!clean,!reliable!and!costEeffective!power!
sources.!At!the!same!time,!many!of!the!world’s!electricity!grids!are!dealing!with!unprecedented!
challenges!and!one!overEarching!issue:!How!to!supply!energy!to!growing!populations!while!at!the!
same!time!reducing!carbon!emissions.!In!the!U.S.!and!Europe!–!the!principal!markets!addressed!in!
this!paper!–!the!reliability!and!cost!challenges!are!growing!ever!more!acute!as!carbonEintensive!
power!plants!are!phased!out!in!favor!of!a!cleaner!generation.!!
Source0–01000percent0renewables0are0not0exactly0the0same0as0carbon0free00
Given!the!increasing!concern!over!increasing!carbon!emissions!and!the!move!towards!lowEcarbon!
computing,!more!data!centers!will!rely!on!growing!amounts!of!renewables.!Companies!such!as!
Google,!Apple,!and!Amazon!have!made!strong!moves!in!this!direction.!Apple!recently!made!an!
48
$850!million,!130!MW!deal!with!First!Solar !to!supply!its!headquarters,!California!stores,!and!a!data!
center.!Even!more!recently,!Amazon!announced!financing!of!a!208!MW!wind!farm!in!North!Carolina!
49
to!power!its!cloud!computing!services. !
!
Such!direct!purchase!contracts!may!focus!on!carbon!reduction!and!serve!the!laudable!cause!of!
‘additionality’.!In!other!words,!they!drive!positive!change!at!the!margin!by!creating!demand!for!
renewable!resources!that!might!not!otherwise!be!financed.!As!a!result,!they!help!to!‘green!up’!the!
grid.!But!by!themselves,!without!energy!storage!resources,!they!do!not!eliminate!the!need!for!fossilE
based!generation!resources,!owing!to!the!intermittency!of!renewable!supplies.!Apple’s!dedicated!
solar!resource,!for!example,!cannot!serve!as!the!sole!electricity!supply!option!for!its!locations!
(otherwise!the!company!would!have!to!stop!work!at!night,!and!slow!down!during!cloudy!days).!Nor!
can!the!Amazon!wind!turbines!fully!support!their!facilities.!While!both!Apple’s!and!Amazon’s!
investments!in!renewable!supply!help!to!deEcarbonize!the!power!grid,!they!also!currently!involve!coE
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!13!
!
WHITEPAPER
!
dependency!with!dispatchable!fossilEbacked!generation!resources,!and!must!necessarily!be!part!of!
a!much!larger!integrated!system.!!
!
If!a!company!is!buying!100!percent!directly!from!wind!or!solar!resources,!then!during!various!
periods!those!resources!are!overproducing!relative!to!the!facility’s!consumption,!and!dumping!that!
surplus!into!the!grid.!At!other!times,!the!facility!must!be!taking!electrons!from!the!grid!that!are!
“system!power,”!i.e.,!sourced!from!the!grid!as!a!whole!rather!than!a!unitEcontingent!purchase.!In!
almost!every!power!system,!this!means!that!a!large!portion!of!the!electrons!consumed!at!a!given!
moment!are!hydrocarbon!based.!The!only!locations!where!that!is!not!the!case!are!where!the!grid!is!
100!percent!renewable.!For!large!power!grids,!that!involves!hydropower,!and!E!in!more!limited!
cases!–!geothermal.!As!we!shall!see!below,!there!are!only!a!few!such!electric!grids!in!the!world.!
Hydropower0and0geothermal0H0ideal0power0resources0for0a0low0carbon0computing0
environment0
Geothermal!energy!and!hydropower!are!the!two!resources!that!can!best!address!both!the!need!for!
lowEcarbon!energy,!and!handle!the!rapid!rampEup!and!rampEdown!required!by!the!operation!of!the!
emerging!computing!environment.!Unlike!wind!and!solar,!they!are!both!dispatchable!loadEfollowing!
resources!that!can!meet!constantly!fluctuating!demand.!!
!
Geothermal!E!similar!to!any!fossilEfuel!fired!plant!E!involves!the!production!of!steam.!While!
geothermal!normally!provides!baseload!24x7!power,!newer!technologies!allow!geothermal!plants!to!
offer!all!of!the!attributes!necessary!for!balancing!the!power!grid,!including!rapid!rampEup!and!rampE
50
down!capabilities. !
!
ReservoirEbased!hydropower!(compared!with!run!of!river,!which!does!not!involve!reservoirs!and!is!
highly!variable)!can!be!quite!dispatchable!as!well,!changing!energy!output!by!restricting,!or!
increasing,!the!amount!of!water!flowing!through!turbines.!They!can!offer!the!same!types!of!grid!
51
reliability!and!balancing!services. !As!such,!either!resource!can!match!the!requirements!of!today’s!
computers.!!
Summary0
Companies!are!responding!to!the!pressure!for!carbonEfree!bits!and!bytes.!Direct!purchases!of!wind!
and!solar!facilities!help,!as!they!clearly!contribute!to!new!renewable!resources!on!the!margin.!But!
today,!they!still!require!the!existence!of!other!resources!to!back!them!up.!In!grids!with!hydrocarbonE
based!resources,!the!paper!transaction!is!green!but!the!actual!electrons!may!sometimes!come!from!
other!sources.!The!only!way!to!ensure!100!percent!carbonEfree!resources!is!to!locate!within!a!
carbonEfree!grid,!such!as!those!dependent!on!hydropower.!
RELIABILITY:0WHERE0TO0FIND0ROBUST0AND0RELIABLE0GRIDS?0
In!the!search!for!lowEcost!and!reliable!power!supplies,!the!CIO!is!often!involved!in!a!search!at!the!
global!level.!Unfortunately,!many!of!the!power!grids!up!for!consideration!may!not!meet!the!stringent!
requirements!for!reliability,!as!electricity!demand!growth!in!many!developed!countries!has!
outstripped!the!infrastructure!necessary!to!keep!up.!The!U.K.,!continental!Europe,!and!the!U.S.!all!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!14!
!
WHITEPAPER
!
face!challenges!unique!to!each!area,!while!the!Nordic!economies,!Quebec,!and!Iceland!fare!
significantly!better.!
The0UK:0Struggling0to0maintain0reliability0
The!UK!is!a!particularly!striking!example!of!a!vulnerable!power!grid!facing!complex!issues,!with!
some!real!challenges!ahead.!The!British!Office!of!Gas!and!Electricity!Markets!(Ofgem)!in!its!2014!
report!covering!the!issue!of!available!generating!capacity!stated!that!,“Even!during!the!relatively!
short!time!horizon!of!this!analysis,!there!is!significant!uncertainty!over!the!security!of!supply!
52
outlook.” !!
!
The!Ofgem!report!finds!that!deErated!margins!(i.e.,!supply!adjusted!to!take!into!account!the!
availability!of!the!power!plant)!could!vary!between!around!2!percent!and!8!percent!in!2015/16!and!
be!on!the!low!side!if!demand!projections!are!off!or!fewer!imports!from!the!continent!are!available.!
To!put!this!figure!into!perspective,!deErated!margins!in!the!UK!stood!at!14.1!percent!in!the!winter!of!
2010/11!and!in!the!U.S!are!typically!maintained!in!the!15!percent!range.!A!recent!update!released!
in!July!of!2015!indicates!that!the!situation!has!changed!significantly.!Generation!assets!have!fallen!
by!yet!another!2,200!MW,!which!National!Grid!is!supplementing!with!special!reserves!including!
demand!response.!The!result!is!a!reserve!margin!of!just!over!5!percent!for!the!coming!winter.!
!
Ofgem!and!utility!National!Grid!indicate!that!gains!will!be!achieved!when!new!generation!is!added!
to!the!mix!as!a!result!of!new!markets!to!be!implemented!in!2018/19.!However,!Ofgem!issued!a!
security!of!supply!report!in!July!that!indicates!an!even!greater!uncertainty!with!respect!to!the!winter!
53
of!2016/17!and!higher!“loss!of!load”!probability. !!
!
A!reserve!margin!that!falls!to!between!0!to!4!percent!in!2016/17!and!barely!clears!6!percent!over!
the!next!five!years!in!the!current!scenario!planning!should!give!data!centers!looking!to!locate!in!the!
UK!some!considerable!cause!to!reconsider.!Aside!from!concerns!related!to!supply,!overall!reliability!
of!the!grid!does!not!appear!to!be!improving!either.!An!analysis!evaluating!that!issue!shows!537!
54
outages!in!2014,!and!505!in!2013,!compared!with!246!in!2012,!and!241!in!2011. !
!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!15!
!
WHITEPAPER
!
55
Figure07:0Thin0Reserve0Margins0in0the0United0Kingdom 0
!
!
56
Figure08:0Even0Thinner0Margins0in02015 0
!
!
“Given!Current!Power!Sector!Uncertainties,!Is!Relocation!of!
Data!Centers!Out!of!the!UK!a!Viable!Option?”!
!
Currently,!the!number!of!data!centers!in!the!UK!is!estimated!to!be!close!to!5,000,!although!some!
consolidation!is!taking!place.!Even!as!the!number!of!centers!will!decline!somewhat,!and!efficiencies!
per!server!increase,!electricity!consumption!will!continue!to!increase!for!the!reasons!discussed!in!
this!paper.!Today,!these!data!centers!are!estimated!to!consume!somewhere!between!3,000!and!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!16!
!
WHITEPAPER
!
4,000!MWk!equal!to!1.5!to!2!percent!of!total!UK!power!use.!Perhaps!more!critically,!this!number!is!
!57
forecast!to!grow!by!3!to!6!percent!annually. !With!the!UK’s!current!and!precarious!limited!reserve!
margins,!there!is!precious!little!room!for!data!center!growth.!!
!
At!the!same!time,!as!discussed!previously,!data!centers!comprise!a!vital!nervous!center!for!
economic!growth.!Limiting!their!growth!would!be!counterproductive!and!costly!to!a!competitive!
economy.!So!what!are!the!alternatives?!One!potential!option!is!to!build!more!energy!infrastructure,!
but!as!discussed!above,!that!path!is!fraught!with!difficulties!and!uncertainties.!Another!option!might!
be!to!relocate!data!centers!offshore!to!countries!with!more!robust!power!infrastructure!and!lower!
costs.!!
!
Relocation!would!result!in!lower!operating!costs!and!less!exposure!to!the!risk!of!power!supply!
disruption.!It!would!also!help!to!take!some!stress!off!the!UK!power!grid,!providing!planners!with!
more!time!to!develop!costEeffective!options.!While!some!data!centers!absolutely!need!to!be!located!
close!to!customers!and!support!applications,!this!holds!true!mainly!for!financial!transactions!that!
require!minimal!latency.!Research!undertaken!by!the!consulting!firm!The!Broad!Group!indicates!
that!even!in!the!financial!sector,!only!10E15!percent!of!applications!actually!require!proximity!to!the!
58
London!Stock!Exchange. !!
!
Today,!more!and!more!individuals!are!increasingly!comfortable!with!the!cloud!and!the!knowledge!
that!their!data!can!literally!be!stored!anywhere!and!shipped!in!miliseconds!(for!example,!Verizon!
59
indicates!it!can!move!data!roundtrip!from!the!U.S.!to!Europe!in!90!miliseconds.) !As!a!
consequence,!there!is!little!true!economic!justification!for!keeping!most!data!centers!close!to!home.!
This!is!truly!a!global!market!where!cost!and!reliability!matter!far!more!than!proximity!for!the!majority!
of!data!centers!in!the!UK!or!elsewhere.!
!
In!fact,!one!estimate!suggests!that!fully!50!to!60!percent!of!data!centers!could!relocate!outside!of!
the!UK!without!any!negative!business!or!regulatory!(e.g.,!government!mandated!locations!for!
60
sensitive!data)!impact. !Indeed,!such!data!centers!moving!to!the!Nordic!countries!or!Iceland,!for!
example,!would!enjoy!both!lower!costs!AND!more!robust!electricity!infrastructure.!!
Continental0Europe:0Their0own0challenges00
Much!of!Europe!is!also!focusing!considerable!efforts!on!reducing!the!carbon!intensity!of!the!power!
grid.!In!some!countries,!such!as!Germany!and!Spain,!the!result!has!been!to!integrate!significant!
levels!of!wind!power!and!solar!energy,!but!at!significant!economic!cost!and!with!some!challenges!to!
the!power!grid.!Outside!of!the!Nordic!regions,!the!cost!of!electricity!in!Europe!is!quite!significant,!
and!too!expensive!to!lure!most!data!centers.!
!
The!continent!continues!to!focus!on!the!goal!of!a!panEEuropean!power!grid,!no!easy!task!given!the!
variety!of!power!grids!and!quality!of!some!interconnections,!but!the!implication!is!that!no!country!will!
go!it!alone.!!
!
Germany:0in0a0state0of0expensive0transition0
In!Germany,!the!continent’s!largest!grid,!there!are!effectively!two!power!systems!that!have!
essentially!been!built:!the!conventional!fossilEbased!system!and!the!new!renewables!system!driven!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!17!
!
WHITEPAPER
!
by!state!policies!and!heavily!subsidized.!The!challenge!with!the!renewable!resources!–!principally!
wind!and!solar!–!are!not!dispatchable.!That!is,!they!cannot!be!called!upon!when!required,!so!they!
have!to!be!backstopped!with!fossilEbased!supply.!At!the!same!time,!the!marginal!costs!for!wind!and!
solar!generation!are!close!to!zero,!so!that!fossil!fuel!plants!cannot!compete!during!many!hours!
when!those!resources!are!providing!electricity.!!
!
The!result!of!this!marginal!cost!problem!is!that!many!of!Germany’s!conventional!plants!are!no!
longer!economical.!More!than!half!lost!money!in!2013,!and!in!the!period!between!December!2012!
and!May!2014,!companies!announced!plans!to!take!8!percent!of!the!country’s!total!2012!
61
conventional!generating!capacity!offline. !The!government!responded!by!requiring!regulatory!
permission!prior!to!shutdown,!and!for!those!essential!plants!denied!permission,!capacity!(standby)!
prices!have!been!paid.!!
!
In!Germany’s!power!sector,!much!remains!in!a!state!of!confusion!at!present,!but!one!thing!is!clear:!
there!is!no!financial!incentive!to!build!any!new!dispatchable!power!plants.!Even!Irsching!5!–!among!
the!most!efficient!combined!cycle!gas!power!plant!in!the!world!(and!only!five!years!old)!–!is!
62
currently!receiving!subsidized!payments!to!stay!in!operation. !!
!
To!date,!these!major!changes!have!not!had!a!significant!impact!on!the!reliability!of!the!German!
power!grid.!In!2014,!the!total!number!of!blackouts!in!the!country!totaled!601,!with!over!600,000!
63
households!affected !and!the!average!outage!time!at!133!minutes.!That!still!makes!Germany!one!
of!the!most!reliable!grids!in!the!world!and!far!more!reliable!than!the!power!grids!of!the!U.S.!and!
64
UK. !It!is!also!quite!expensive,!as!will!be!discussed!below.!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!18!
!
WHITEPAPER
!
65
Figure09:0Comparing0Reliability0of0Some0Power0Grids 0
Minutes0of0power0outages0per0year0(excl.0exceptional0events),0based0on0Saidi0
Source:!CEER!and!own!calculations!
!
!
A!planned!1400!MW!interconnection!between!Germany!and!Norway!should!help,!allowing!Norway!
66
to!export!stabilizing!hydro!to!Germany!while!Germany!can!export!surplus!wind!power!to!Norway. !
But!with!high!prices!and!a!power!market!in!an!uncertain!state,!Germany!will!have!difficulty!providing!
low!cost!power!to!the!data!center!industry.!
The0United0States0grid:0undergoing0a0structural0shift0
In!the!U.S.,!the!focus!is!increasingly!on!the!EPA’s!proposed!Clean!Power!Plan,!which!requires!
states!to!reduce!utility!carbon!output!by!30!percent!between!now!and!2030.!This!will!require!a!
significant!restructuring!of!the!U.S.!power!sector,!involving!more!efficient!power!plants,!renewables,!
and!energy!efficiency.!Many!of!the!country’s!coalEfired!generators!are!already!scheduled!for!
retirement!simply!due!to!their!age.!It!is!estimated!that!some!90,000!MW!of!aging!coal!plants!will!
67
retire!between!2012!and!2020!(out!of!a!total!generating!fleet!of!1,060,000!MW). !
!
0
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!19!
!
WHITEPAPER
!
68
Figure010:0America’s0CoalHFired0Power0Plants0Are0Aging 0
Age0and0capacity0of0electric0generators0by0fuel0type,0as0of0yearHend020120
!
!
Some!of!these!coal!plants!will!be!replaced!by!natural!gasEfired!plants,!but!intermittent!wind!and!
solar!resources!will!also!play!an!increasing!role.!This!will!require!the!buildEout!of!new!infrastructure,!
with!likely!additional!costs!(such!as!storage)!to!the!consumer.!Even!without!the!new!environmental!
regulations,!much!investment!in!new!infrastructure!would!have!been!necessary.!The!American!
Society!for!Civil!Engineers!estimates!that!the!investment!gap!(what!is!spent!versus!what!is!needed)!
69
in!the!U.S.!electric!infrastructure!is!equivalent!to!approximately!$11!billion!annually. !Catching!up!
will!involve!upward!pressure!on!electricity!rates.!!
!
At!the!regional!level,!the!amount!of!available!new!energy!resources!varies!considerably.!This!is!a!
consequence!of!numerous!factors!including!economic!strength,!available!infrastructure,!such!as!
gas!pipelines,!and!characteristics!of!the!generating!fleet!(especially!age).!Some!regions,!such!as!
California,!New!England,!New!York,!and!Texas!are!more!constrained!than!other!areas!of!the!
country.!At!this!point!in!time,!with!known!technologies,!it!appears!that!the!U.S.!power!grid!will!
continue!to!face!challenges!in!offering!reliable!and!uninterrupted!power.!This!has!real!economic!
costs.!According!to!Dunn!and!Bradstreet,!59!percent!of!Fortune!500!companies!suffer!at!least!1.6!
hours!of!annual!downtime.!Just!the!lost!productivity!associated!with!labor!is!estimated!at!$46!million!
70
annually!per!company. !In!fact,!the!DOE!estimates!that!outage!and!power!quality!issues!costs!
71
American!businesses!over!$100!billion!on!average!each!year. !The!country!also!has!the!
unenviable!reputation!of!having!more!blackouts!than!any!other!industrialized!nation.!Unfortunately,!
72
that!trend!has!accelerated!since!1984,!when!data!collection!began,!by!284!percent. !In!general,!the!
recent!trend!–!in!terms!of!number!of!events!E!continues!upwards,!as!demonstrated!in!the!chart!
below.!
!
!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!20!
!
WHITEPAPER
!
73
Figure011:0Total0Number0of0U.S.0Power0Outages0Is0On0the0Rise !
Year0
Total0number0of0outages0
People0affected0
2008*!
2,169!
25.8!million!
2009!
2,840!
13.5!million!
2010!
3,149!
17.5!million!
2011!
3,071!
41.8!million!
2012!
2,808!
25.0!million!
2013!
3,236!
14.0!million!
2014!
3,634!
14.2!million!
*PartialFyear!data.!Data!collection!began!on!February!16,!2008!
The0Nordic0markets:0largely0blessed0with0abundant0renewables0&0relatively0stable0grids0
By!contrast,!the!Nordic!economies!(Denmark,!Finland,!Iceland,!Norway,!and!Sweden)!are!fortunate!
to!have!an!abundant!and!diversified!fuel!mix,!with!Denmark!a!leader!in!wind,!while!Iceland,!Norway,!
and!Sweden,!enjoy!significant!hydroelectric!resources,!and!Finland!has!significant!nuclear!power!
resources.!!
0
74
Figure012:0The0Nordic0Pool0Resource0Mix 0
!
Well!before!the!restructuring!of!U.S.!electricity!markets,!Norway!introduced!competitive!power!
markets!in!1990.!Finland!and!Sweden!followed!suit!in!the!midE90s!with!Denmark!completing!the!
75
process!in!1998,!allowing!large!customers!to!access!power!markets. !!
!
Thereafter,!a!Nordic!Power!Pool!was!established!under!the!regulation!of!NordREG!–!the!
organization!of!Nordic!energy!regulators,!including!the!aboveEmentioned!countries!as!well!as!
Iceland.!The!role!of!this!entity!is!to!facilitate!transmission!and!trading!across!the!affiliated!counties.!
In!the!future,!it!is!likely!that!all!customers!(with!the!exception!of!those!in!Iceland)!will!be!able!to!buy!
electricity!across!national!boundaries.!LongerEterm,!the!goal!is!participation!in!an!even!broader!
76
European!market. !
!
Since!the!Nordic!power!system!possesses!significant!interEannual!hydro!reservoir!storage!
capability,!fewer!costly!peaker!plants!are!required!to!meet!maximum!demand.!The!system,!
therefore,!has!the!ability!to!dampen!power!prices.!Energy,!stored!as!accumulated!water!during!high!
precipitation!periods,!can!also!be!consumed!in!the!winter!when!less!precipitation!falls!and!prices!
would!otherwise!be!higher.!To!the!extent!it!exists,!price!variability!is!more!about!predicted!longEterm!
77
precipitation!cycles!than!fossil!fuels. !
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!21!
!
WHITEPAPER
!
A!combination!of!falling!costs!for!renewable!generation!such!as!wind!and!hydro,!along!with!
expanded!grid!interconnections!to!the!continent,!could!result!in!the!Nordic!countries!becoming!large!
exporters!of!power.!Estimates!suggest!that!with!proper!infrastructure!and!market!pricing,!the!Nordic!
78
Pool!could!eventually!export!50!terawatt!hours!(TWh)!to!100!TWh. !Should!that!prove!to!be!the!
case,!the!European!system!may!begin!to!rely!on!the!Nordic!pool!as!a!very!large!storage!system!to!
increase!its!share!of!intermittent!renewables.!Of!course,!the!availability!of!surplus!electricity!for!
export!implies!availability!of!power!for!localized!data!centers!as!well.!Norway!and!Sweden,!in!
particular,!have!developed!policies!to!attract!new!computing!facilities.!In!both!countries,!greenfield!
sites!for!hundreds!of!megawatts!of!new!data!centers!have!already!been!identified!(to!be!discussed!
in!more!detail!below).!!
Quebec:0A0Hydro0superpower0in0North0America0
The!Province!of!Quebec!enjoys!a!similar!position!in!North!America,!with!a!stable!power!grid,!
terawattEhours!of!interEannual!hydroelectric!storage,!and!the!ability!to!export!relatively!cheap!
hydropower!to!the!northeastern!United!States.!The!provincial!utility!is!currently!expanding!its!
hydroelectric!system,!and!it!is!also!eyeing!additional!transmission!lines!to!further!boost!export!
79
capabilities. !!
0
Summary0
All!electric!grids!are!not!created!equal.!The!UK!is!skating!on!very!thin!ice!for!the!coming!years!with!
limited!generating!resources!and!little!room!for!maneuver.!The!UK!is!one!of!the!developed!countries!
where!data!centers!face!the!highest!level!of!uncertainty!with!respect!to!future!power!supply!(see!
box).!Meanwhile,!Germany!is!in!transition!and!regulatory!uncertainty,!with!resulting!high!costs.!
Meanwhile!the!U.S.!increasingly!struggles!with!an!aging!grid!and!reliability!issues.!By!contrast,!the!
Nordic!countries!(including!Iceland),!and!enjoy!an!abundance!of!renewable!resources!and!fairly!
reliable!power!grids.!The!province!of!Quebec!is!similarly!blessed.!In!the!context!of!a!20Eyear!
investment!in!siting!a!data!center,!these!reliability!issues!cannot!be!ignored.!Cost!will!be!an!
important!factor!as!well,!as!addressed!in!the!following!section.0
COST:0LOOK0FOR0HYDROPOWER0AND0COOLER0CLIMATES00
Given!the!primacy!of!hydropower!for!this!type!of!application,!the!logical!step!is!to!identify!where!the!
world’s!largest!available!sources!of!hydropower!are!located.!Today,!the!only!sizable!electric!power!
grids!in!the!northern!hemisphere!that!can!lay!claim!to!being!all!or!nearly!100!percent!renewable!are!
80
81
HydroEQuebec!(99!percent!E!mostly!hydro), !Norway!(97.5!percent!E!mostly!hydro), !and!Iceland!
82
(71!percent!hydrok!29!percent!geothermal). !Sweden!also!has!over!50!percent!hydro,!
supplemented!largely!by!nuclear!power.!
!
Not!surprisingly,!these!electric!grids!are!able!to!offer!some!of!the!lowest!kilowattEhour!prices!in!the!
world,!as!seen!in!the!chart!below.!Unlike!systems!that!are!primarily!based!on!expensive!
hydrocarbons!that!are!also!highly!volatile!in!price,!hydroEbased!grids!are!not!only!cheaper,!but!more!
stable!from!year!to!year!(though!in!Norway,!rainfall!variability!has!had!some!impact!on!prices).!!
83
Figure013:0A0Comparison0of0Power0Prices 0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!22!
!
WHITEPAPER
!
!
!
However,!the!composition!of!the!existing!generating!fleet!is!only!part!of!the!issue!when!considering!
the!ability!to!host!large!numbers!of!new!highEperformance!data!centers.!Perhaps!more!importantly!
is!the!composition!and!availability!of!power!generating!resources!on!the!margin.!If!the!question!is!
where!to!locate!new!and!energyEhungry!facilities,!the!answer!must!revolve!around!both!cost!and!
availability!of!existing!surpluses!and!new!resources!to!meet!growing!demand.!
!
HydroEQuebec!has!ample!hydroelectric!resources!on!the!margin.!It!just!delivered!the!first!640!MW!
of!its!1,550!Romaine!complex!in!2014,!with!910!additional!MW!to!come!online!between!2016!and!
84
2020. !!
!
Norway!and!Sweden!are!not!as!aggressively!developing!new!resources,!although!Norway!in!
85
particular!still!has!many!existing!resources!that!can!be!technically!exploited. !Both!Norway!and!
Sweden!currently!enjoy!existing!surpluses,!and!as!noted!above,!the!entire!Nordic!region!has!
significant!potential!export!capability.!!
!
For!its!part,!Iceland!has!up!to!2,000!MW!of!additional!renewable!power!capacity!being!considered!
86
for!developments!between!now!and!2030. !
!
Each!of!these!markets!is!able!to!offer!relatively!low!cost!power!to!industrial!users.!HydroEQuebec!
87
offers!industrial!rates!for!its!very!large!users!in!the!mid!threeEtoEfour!cent!range. !The!Nordic!
markets!are!cost!competitive,!with!recent!fixed!prices!to!data!centers!in!Norway!being!offered!tenE
88
year!contracts!in!the!4.5!cents!(U.S.)!per!kWh!range,!less!than!half!the!price!in!the!U.K. !Iceland’s!
local!utility,!Landvirkjun,!can!offer!a!fixed!price!contract!for!four!cents!per!kWh!to!data!centers!for!up!
89
to!12!years. !
!
Norway!has!already!identified!multiple!data!center!sites!–!many!of!which!are!designed!to!be!
90
consuming!over!100!MW!when!completed. !It!also!boasts!what!may!eventually!become!the!world’s!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!23!
!
WHITEPAPER
!
largest!data!center!–!Lefdal!Mine!E!repurposing!120,000!square!meters!of!space!in!an!abandoned!
91
sixEstory!olivine!mine!deep!inside!a!mountain. !Green!Mountain!has!also!built!out!one!13,600!m2!
data!center!(up!to!26!MW)!in!a!former!NATO!ammunition!store,!and!has!a!second!under!
construction!with!an!additional!25!MW+!of!capacity.!
!
In!Sweden,!dozens!of!potential!greenfield!sites!have!been!identified!as!part!of!The!Swedish!Data!
92
Center!Initiativek!a!part!of!the!Swedish!Trade!and!Investment!Council). !Facebook!already!has!an!
enormous!data!center!close!to!the!Arctic!Circle,!and!has!plans!to!add!an!additional!data!center!
93
using!preEfabricated!modules!to!speed!up!construction. !!
!
Quebec!has!been!a!laggard!in!the!area,!with!relatively!few!centers!and,!until!recently,!minimal!
efforts!to!attract!them.!That!has!changed!and!the!government!is!now!offering!a!tenEyear!tax!
94
holiday. !
!
Almost!as!important!as!the!low!power!prices!are!the!chilly!temperatures!in!which!these!facilities!are!
located.!The!lower!ambient!temperatures!allow!the!data!centers!to!utilize!outside!air!to!cool!the!
servers!at!little!to!no!cost.!In!fact,!some!data!centers!have!been!located!close!to!the!Arctic!Circle!for!
just!this!reason.!Since!cooling!is!responsible!for!up!to!half!the!energy!consumed!by!this!industry,!
this!matters!considerably.!The!following!chart!shows!the!relationship!between!temperatures!in!
Iceland!in!the!Nordic!High!Performance!Computing!system,!with!administrators!having!identified!“a!
high!correlation!between...power!usage!effectiveness!(PUE)!and!the!average!temperature!by!
95
day…” !!
!
96
Figure014:0Temperatures0and0PUEs 0
!
!
!
Taken!together,!power!costs!and!ambient!temperatures!have!the!greatest!impact!on!total!costs!over!
time.!The!following!chart!serves!to!highlight!Iceland’s!comparative!advantage!with!other!locations!
(although!it!does!not!include!such!potential!competitors!as!Quebec!and!Norway,!both!of!which!
would!likely!be!close!in!terms!of!total!costs.)!
!!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!24!
!
WHITEPAPER
!
97
Figure015:0Operating0Costs 0
100Year0cost0of0data0center0operations0
!
Summary0
Operating!costs!are!a!function!of!a!number!of!variables,!with!energy!costs!–!especially!for!cooling!–!
at!the!top!of!the!pyramid.!Low!power!prices!and!free!cooling!for!most!of!the!year!confer!a!real!
advantage!to!northern!markets!rich!in!hydropower!and!cool!weather.!Countries!that!can!offer!lower!
costs!and!grid!stability!have!a!real!advantage!as!sites!for!future!data!centers.0
CONCLUSION0
The!world!is!becoming!ever!more!dataEcentric.!We!produce!and!store!more!data!every!day,!a!trend!
that!continues!to!accelerate.!We!create!increasingly!more!economic!value!as!we!more!rapidly!
process!and!analyze!information.!IT!will!become!ever!more!important.!At!the!same!time,!CIOs!are!
under!a!growing!burden!to!create!value,!avoid!risk,!and!focus!on!environmental!costs,!while!having!
access!to!limited,!or!even!shrinking,!resources.!!
!
One!pathway!forward!for!these!CIOs!involves!migration!to!the!cloud!and!high!performance!data!
centers.!However,!these!super!data!storage!and!processing!facilities!will!require!reliable!power!
grids!with!abundant!sources!of!clean!and!affordable!power.!The!U.K.,!continental!Europe,!and!the!
U.S.!will!likely!continue!to!have!difficulty!with!offering!reliable,!lowEcarbon,!and!affordable!power.!
The!U.K.,!in!particular,!may!face!serious!challenges!to!grid!reliability!in!the!coming!years.!!
!
By!contrast,!countries!with!abundant!hydropower!and!geothermal!energy!represent!optimal!
locations!for!the!task!of!providing!affordable!and!lowEcarbon!services!to!data!centers.!Colder!
climates!that!minimize!cooling!requirements!are!also!highly!beneficial.!!
!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!25!
!
WHITEPAPER
!
Figure016:0A0Summary0View0of0Important0Considerations0
Low0
Carbon0
0
!
US!!!!!!
Germany!!!!!!!
Comments0
!
!
Increasing!power!outages,!
reliability!issues,!
underinvestment!in!
infrastructure!
!
!
Reserve!margins!falling!to!
near!zero!cause!for!concern!
!
High!costs!and!regulatory!
uncertainty!are!the!main!
concern!
!
!
UK!!!!!!!!!
High0
Low0
Reliability0 Cost0
!
!
!! !
!!!
!!!
!!!
Low!cost,!abundant!hydro,!
and!free!cooling!
!!!
!!!
!!!
Low!cost,!abundant!hydro,!
and!free!cooling!
!!!
!!!
!!!
Low!cost,!abundant!
hydro/geothermal!and!free!
cooling!
!
!
Nordics!!!
Quebec!!!!!!!
Iceland!!!!
!
!
!
!
!
Only!a!few!locations!meet!all!those!requirements.!Iceland,!Norway,!Sweden,!and!the!Province!of!
Quebec!all!enjoy!significant!quantities!of!affordable!and!abundant!renewable!resources,!stable!
governments,!and!the!required!connectivity.!These!locations!may!serve!as!a!viable!host!for!data!
centers,!and!indeed!many!of!them!have!already!taken!significant!steps!in!this!direction.!
!
It!should!be!expected!that!these!northern!locations!may!soon!become!the!preEeminent!destinations!
for!the!next!generation!of!high!performance!data!centers,!which!will!increasingly!connect!our!world!
and!add!enormous!value!to!the!global!economy.!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!26!
!
WHITEPAPER
!
APPENDIX0A:0FUTURE0COORDINATION0WITH0POWER0GRID0PLANNERS0AND0
OPERATORS0
One!critical!issue!concerning!the!machines!of!the!future!is!how!they!will!run.!One!report!suggests!
that!the!20!MW!HPC!machine!of!the!future!may!ramp!up!or!down!its!energy!consumption!by!up!to!
98
40!percent!within!miliEseconds. !One!or!multiple!such!machines!in!the!same!location!pose!an!
obvious!and!serious!challenge!to!the!stability!of!electric!power!grids!if!they!are!not!somehow!
buffered!and!coordinated.!!
!
The0necessary0conversations0with0grid0coordinators0
These!emerging!technologies!are!so!new!that!very!little!conversation!has!yet!occurred!between!
electric!utilities,!power!grid!coordinators,!and!the!computer/server!industry.!The!good!news!is!that!
there!is!historical!precedent!for!coordination!between!industry!and!utilities!to!coordinate!
consumption!in!areas!such!as!the!U.S.,!Australia,!and!Europe.!!
!
Indeed,!there!are!specific!ways!in!which!some!of!today’s!supercomputers!interact!with!power!grid!
planners,!depending!in!part!on!the!manner!in!which!markets!and!electricity!“products”!are!
structured.!Supercomputers!can,!for!example,!develop!and!deploy!software!to!manage!their!
electricity!consumption!and!coordinate!with!the!requirements!of!the!power!grid,!and!there!are!
specific!precedents:!Lawrence!Livermore!National!Labs!has!been!scheduling!its!Blue!Gene/L!
system!(ten!years!ago,!the!fastest!supercomputer!in!the!world)!with!the!local!utility!for!years!after!
the!initial!swings!in!power!consumption!prompted!the!utility!to!request!daily!demand!forecasts.!The!
University!of!California’s!San!Diego!Supercomputing!Center!enjoys!a!similar!relationship!with!its!
99
utility. !
!
New0technologies0may0help0
Since!power!consumption!now!represents!a!relatively!high!portion!of!Total!Cost!of!Ownership!of!
100
highEperformance!data!centers, !more!focus!is!being!paid!to!the!issue.!Two!emerging!techniques,!
power!capping!and!energy!capping,!may!help!to!deal!with!these!issues!to!some!degree.!!
!
Power!capping!is!similar!to!demand!limiting!strategies!that!have!existed!for!years!in!buildings!and!
other!facilities.!In!these!scenarios,!limits!are!set!above!which!power!consumption!is!not!allowed!to!
go.!In!buildings,!this!has!to!do!with!set!points!affecting!internal!temperatures.!In!computers,!such!
strategies!involve!scheduling!of!queued!jobs!and!evaluating!the!impact!on!cooling!requirements!to!
calculate!overall!load.!The!majority!of!the!time,!power!capping!utilizes!dynamic!voltage!frequency!
scaling.!The!general!result!is!that!while!peak!demand!is!limited,!overall!energy!consumption!
increases.!
!
0
0
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!27!
!
Energy0Capping
101
WHITEPAPER
!
00
!
!
Energy!capping!focuses!on!solving!a!different!problem:!how!many!kWh!will!be!consumed!over!a!
specific!period!of!time?!Elastic!and!low!priority!applications!can!potentially!be!moved!to!offEpeak!
hours!when!electrons!are!cheaper.!
!
Today!and!tomorrow’s!HPC!super!computers!differ!from!a!standard!data!center.!Among!other!
things,!they!are!currently!more!performanceEoriented!and!are!not!geographically!portable!owing!to!
security!concerns.!However,!data!centers!themselves!are!not!stagnant.!They!are!also!rapidly!
evolving!to!process!more!valuable!data!in!real!time.!Thus,!they!will!increasingly!become!more!
performanceEoriented!as!dynamic!loads!increase,!with!many!shifting!to!subscription!and!capacity!
102
onEdemand!type!of!service. !The!demands!they!place!on!the!power!grid!will!increase!as!they!
become!larger!and!create!the!need!for!both!more!energy!and!instantaneous!capacity.!In!areas!
where!the!grid!is!constrained,!they!too!may!be!required!to!coordinate!with!the!electric!utility!and!to!
employ!power!and!energy!capping!strategies.!
!
1
!
https://www2.deloitte.com/content/dam/Deloitte/fi/Documents/technology/CIO%20Survey%202014.
pdf!E!in!fact,!23!percent!of!CIOs!surveyed!in!the!Deloitte!report!indicated!a!decrease!in!budget!over!
the!past!year.!
2
!
https://www2.deloitte.com/content/dam/Deloitte/fi/Documents/technology/CIO%20Survey%202014.
pdf!
3
!http://www.emersonnetworkpower.com/documentation/enE
us/brands/liebert/documents/white%20papers/2013_emerson_data_center_outages_slE24679.pdf!E!
(that!number!was!88!percent!in!the!2010!survey,!so!the!issue!of!utility!reliability!appears!to!have!
remained!unchanged)!
4
!http://www.emersonnetworkpower.com/documentation/enE
us/brands/liebert/documents/white%20papers/2013_emerson_data_center_outages_slE24679.pdf!
!
5
!http://www.emersonnetworkpower.com/documentation/enE
us/brands/liebert/documents/white%20papers/2013_emerson_data_center_outages_slE24679.pdf!
6
!The!leading!cause!of!unplanned!outages!was!reported!to!be!UPS!battery!failure,!followed!closely!
by!human!error!or!UPS!battery!capacity!having!been!exceeded.!
!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!28!
!
WHITEPAPER
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
7
!http://www.emersonnetworkpower.com/documentation/enE
us/brands/liebert/documents/white%20papers/2013_emerson_data_center_outages_slE24679.pdf!
!
8
!http://www.chicagobusiness.com/article/20130604/BLOGS11/130609948/theEpriceEofEfailureEdataE
centerEpowerEoutageEcostEsearsE2E2mEinEprofit!
9
!http://info.xmatters.com/rs/alarmpoint/images/xMattersE2015ESurveyEReport.pdf!
10
!http://www.emc.com/leadership/digitalEuniverse/2014iview/executiveEsummary.htm!
11
!http://www.emc.com/leadership/digitalEuniverse/2014iview/internetEofEthings.htm!
12
!http://www.cisco.com/c/en/us/solutions/collateral/serviceEprovider/visualEnetworkingEindexE
vni/VNI_Hyperconnectivity_WP.html!
13
!http://www.emc.com/leadership/digitalEuniverse/2014iview/internetEofEthings.htm!
14
!http://www.forbes.com/sites/gilpress/2013/05/09/aEveryEshortEhistoryEofEbigEdata/!
15
!http://www.environmentalleader.com/2015/04/01/ibmEcreatesE3bnEinternetEofEthingsEunit/!
16
!http://www.emc.com/leadership/digitalEuniverse/2014iview/executiveEsummary.htm!
17
!http://blog.schneiderEelectric.com/buildingEmanagement/2014/10/06/bigEopportunityEinternetE
things/!
18
!http://www.gartner.com/newsroom/id/2848718!
19
!http://www.noaanews.noaa.gov/stories2015/20150105_supercomputer.htmlk!a!petaflop!is!a!
thousand!trillion!floating!point!operations!per!second.!
20
!http://www.metoffice.gov.uk/news/releases/archive/2014/newEhpc!
21
!http://www.ecmwf.int/en/computing/ourEfacilities/supercomputer!
22
!http://mashable.com/2015/01/06/weatherEserviceEsupercomputersEpetaflops/!
23
!http://www.ecmwf.int/sites/default/files/HPCEWSEMichalakes.pdf!
24
!https://www.olcf.ornl.gov/wpEcontent/themes/olcf/titan/Titan_Debuts.pdf!
25
!http://ret.cleanenergyregulator.gov.au/LatestEUpdates/2013/April/oneEmillionEsolarEpanelEsystemsE
installed!
26
!http://www.hawaii247.com/2015/01/20/hawaiianEelectricEplansEtoEtripleErooftopEsolarEpower/!
27
!http://cleantechnica.com/2014/04/04/7EteslaEnorwayEfactsEwillEblowEaway/!
28
!http://www.greentechmedia.com/articles/read/BigEDatasE5EBigEStepsEtoESmartEGridEGrowthEinE
2014!
29
!http://www.greentechmedia.com/articles/read/BigEDatasE5EBigEStepsEtoESmartEGridEGrowthEinE
2014!
30
!https://hpc.nrel.gov/users/systems/peregrine!E!With!1.19!petaflops,!a!PUE!of!1.06!or!better,!
utilizing!a!liquid!cooling!system,!and!consuming!just!over!1!MW,!this!is!one!of!the!most!efficient!
machines!in!the!world!to!date.!Among!other!initiatives,!Peregrin!is!supporting!the!Eastern!
Renewable!Integration!Study!–!a!multiEyear!U.S.!DOEEfunded!project!to!simulate!operations!of!the!
largest!power!system!in!the!world!(from!Northern!Texas!east!to!Florida!and!reaching!all!the!way!up!
to!Nova!Scotia.!“The!study!will!inform!critical!questions!on!how!system!operations!could!be!
impacted!by!various!wind!and!solar!deployment!strategies!and!operational!paradigms.!It!is!the!first!
study!to!simulate!the!entire!Eastern!Interconnection!and!HydroEQuébec!with!hourly!dayEahead!unit!
commitment,!5Eminute!realEtime!dispatch,!and!a!nodal!DCEpower!flow.”!
!
31
!http://www.nrel.gov/news/features/feature_detail.cfm/feature_id=2133!
32
!http://www.renewableenergyworld.com/rea/news/article/2014/07/researchersEdevelopingE
supercomputerEtoEtackleEgridEchallenges!
33
!http://phys.org/news/2014E11EelectricalEpowerEpanEeuropeanEelectricityEtransmission.html!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!29!
!
WHITEPAPER
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
34
!http://www.eweek.com/storage/slideshows/intelsElowEpowerEatomEchipEmakingEwayEintoEdataE
centers.html?kc=EWKNLESB09082013STR1&dni=77668363&rni=24932747!
35
!http://www.nrdc.org/energy/dataEcenterEefficiencyEassessment.asp!
36
!http://anthesisgroup.com/wpEcontent/uploads/2014/08/DataECenterEIssueEPaperEfinal826.pdf!
37
!http://www.idc.com/getdoc.jsp?containerId=prUS25580215!
38
!http://www.datacenterknowledge.com/specialEreportEtheEworldsElargestEdataEcenters/worldsE
largestEdataEcenterE350EeEcermak/!
39
!http://www.supernap.com/faq.html!
40
!http://www.supernap.com/unitedEstates.html!
41
!https://www.nvenergy.com/company/energytopics/where.cfm!
42
!http://www.idc.com/getdoc.jsp?containerId=prUS25580215!
43
!http://www.ieaEetsap.org/web/eEtechds/pdf/e02Egas_fired_powerEgsEadEgct.pdf!
44
!
http://energy.gov/sites/prod/files/2014/10/f18/SEAB%20HPC%20Task%20Force%20Final%20Repo
rt.pdf!
45
!
http://energy.gov/sites/prod/files/2014/10/f18/SEAB%20HPC%20Task%20Force%20Final%20Repo
rt.pdf!
46
!http://www.ercot.com/news/press_releases/show/26597!
47
!Bates,!Ghatikar,!et!al.,!“Electrical!Grid!and!Supercomputing!Centers:!An!Investigative!Analysis!of!
Emerging!Opportunities!and!Challenges,”!December!2014.!
48
!http://www.bloomberg.com/news/articles/2015E02E11/whatEappleEjustEdidEinEsolarEisEaEreallyEbigE
deal!
49
!http://fortune.com/2015/07/13/amazonEtoEpowerEcloudEwithEbigEwindEfarmEinEnorthEcarolina/!
50
!http://www.renewableenergyworld.com/articles/2014/04/flexibilityEbenefitsEmakeEgeothermalEaE
superiorEsolutionEforEtodaysEchangingEgrids.html!E!these!services!include!“scheduling!and!dispatch,!
reactive!power!and!voltage!control,!loss!compensation,!load!following,!system!protection,!and!
energy!imbalance.”!
51
!http://www.hydroworld.com/articles/hr/print/volumeE32/issueE3/articles/valuingEhydropowerEgridE
servicesEinEtheEfutureEelectricityEgrid.html!
52
!https://www.ofgem.gov.uk/ofgemEpublications/88523/electricitycapacityassessment2014E
fullreportfinalforpublication.pdf!
53
!
https://www.ofgem.gov.uk/sites/default/files/docs/2015/07/electricitysecurityofsupplyreport_final_0.p
df!“Our!analysis!shows!that!the!LOLE!could!range!between!2!and!15!hours!per!year!for!our!central!
view!of!the!risks!(corresponding!to!margins!between!0%!and!4%).”!!
!
54
!http://powerquality.eaton.com/BlackoutTracker/default.asp!
55
!https://www.ofgem.gov.uk/ofgemEpublications/88523/electricitycapacityassessment2014E
fullreportfinalforpublication.pdf!
56
!http://www.bbc.com/news/businessE33527967!
57
!Commissioned!Research,!September!2015:!Steve!Wallage,!Managing!Director,!The!Broad!Group!
58
!Commissioned!Research,!September!2015:!Steve!Wallage,!Managing!Director,!The!Broad!Group!
59
!http://www.verizonenterprise.com/about/network/latency/!
60
!Commissioned!Research,!September!2015:!Steve!Wallage,!Managing!Director,!The!Broad!Group!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!30!
!
WHITEPAPER
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
61
https://www.bcgperspectives.com/content/articles/sustainability_energy_environment_germany_e
nergiewende_end_power_market_liberalization/?chapter=2#chapter2_section4!
62
https://www.bcgperspectives.com/content/articles/sustainability_energy_environment_germany_e
nergiewende_end_power_market_liberalization/?chapter=2#chapter2_section4!
63
!
http://powerquality.eaton.com/BlackoutTracker/default.asp?id=&key=&Quest_user_id=&leadg_Q_Q
Required=&site=&menu=&cx=86&x=8&y=10!
64
!http://spectrum.ieee.org/energywise/energy/theEsmarterEgrid/germanysEsuperstableEsolarsoakedE
grid?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+IeeeSpectrum+%28I
EEE+Spectrum%29&utm_conten!
65
!http://spectrum.ieee.org/energywise/energy/theEsmarterEgrid/germanysEsuperstableEsolarsoakedE
grid?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+IeeeSpectrum+%28I
EEE+Spectrum%29&utm_conten!
66
!http://www.tennet.eu/nl/gridEprojects/internationalEprojects/nordlink.html!
67
!http://www.eia.gov/analysis/requests/powerplants/cleanplan/!
68
!http://qz.com/61423/coalEfiredEpowerEplantsEnearEretirement/!
69
!
http://www.asce.org/uploadedFiles/Issues_and_Advocacy/Our_Initiatives/Infrastructure/Content_Pie
ces/failureEtoEactEelectricityEreport.pdf!
70
!http://images.electricalsector.eaton.com/Web/EatonElectrical/{aa0d93cfE362aE4bd9E9005E
bb723dd40d97}_USBlackoutTracker2014ReportFinal.pdf!
71
!
http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/DOE_SG_Book_Single_Pages%281
%29.pdf!
72
!http://www.ibtimes.com/agingEusEpowerEgridEblacksEoutEmoreEanyEotherEdevelopedEnationE
1631086!
73
!http://images.electricalsector.eaton.com/Web/EatonElectrical/{aa0d93cfE362aE4bd9E9005E
bb723dd40d97}_USBlackoutTracker2014ReportFinal.pdf!
74
!www.iea.org/media/etp/nordic/net.pdf!
75
!http://www.nordicenergyregulators.org/aboutEnordreg/anEoverviewEofEtheEnordicEelectricityE
market/!
76
!http://www.nordicenergyregulators.org/projects/commonEnordicEendEuserEmarket/!
77
!http://www.nordicenergyregulators.org/wpEcontent/uploads/2014/06/NordicEMarketEReportE
2014.pdf!
78
!http://www.iea.org/media/etp/nordic/netp.pdf!
79
!http://www.rtoinsider.com/hydroEquebecEexportsE13648/!
80
!http://www.hydroquebec.com/publications/en/docs/annualEreport/annualEreportE2014.pdf!
81
!http://www.ssb.no/en/energiEogEindustri/statistikker/elektrisitetaar/aar/2015E03E25!
82
!http://askjaenergy.org!
83
!http://www.landsvirkjun.com/Media/lvwhitepaperreportnordicEhighEperformanceEcomputingE
system.pdf!
!
84
!http://hydroforthefuture.com/projets/9/developingEquebecEsEhydropowerEpotential!
85
!http://www.nortrade.com/sectors/articles/exploitingEnorwaysEhydropowerEpotential/!
86
!http://www.landsvirkjun.com/productsservices/energyproducts/datacenters/highEperformanceE
computing/!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!31!
!
WHITEPAPER
!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
87
!http://www.hydroquebec.com/publications/en/docs/distributionEtariff/distribution_tariff.pdf!
88
!http://www.datacenterdynamics.com/appEcloud/norwayEoperatorEgoesEafterEpowerEconsciousE
dataEcenterEmarket/80424.fullarticle!
89
!http://www.datacenterknowledge.com/archives/2014/06/16/icelandEutilityEtoutsEcheapEcleanE
energyElureEdataEcenters/!
90
!http://www.invinor.no/no/Industries/GreenEIT/kartEogEliste/?id=858&epslanguage=no!
91
!http://www.lefdalmine.com!
92
!http://www.businessEsweden.se/en/Invest/Industries/ICT/TheEDataECenterEInitiative/AvailableE
DataECenterEsites/GreenfieldE10Eha/!
93
!http://www.datacenterdynamics.com/designEstrategy/facebookEtoEbuildEsecondEdataEcenterEinE
sweden/85409.fullarticle!
94
!http://business.financialpost.com/news/cheapEhydroelectricEpowerEdrawsEdataEwarehouseE
companiesEtoEquebecEbEc!
95
!http://www.landsvirkjun.com/Media/lvwhitepaperreportnordicEhighEperformanceEcomputingE
system.pdf!E!run!by!Denmark,!Iceland,!Norway,!and!Sweden.!
96
!http://www.landsvirkjun.com/Media/lvwhitepaperreportnordicEhighEperformanceEcomputingE
system.pdf!
97
!http://www.landsvirkjun.com/Media/20130529_Iceland_DC_Report.pdf!
98
!Bates,!Ghatikar,!et!al.,!“Electrical!Grid!and!Supercomputing!Centers:!An!Investigative!Analysis!of!
Emerging!Opportunities!and!Challenges,”!December!2014.!
99
!http://link.springer.com/article/10.1007%2Fs00287E014E0850E0#pageE1!
100
!http://superfri.org/superfri/article/view/9!
101
!http://superfri.org/superfri/article/view/9!
102
!http://siliconangle.com/blog/2014/03/05/theEevolutionEofEtheEdataEcenterEtimelineEfromEtheE
mainframeEtoEtheEcloudEtc0114/!
!
!
Mind!the!Gap:!Energy!Availability!and!the!Disconnect!with!Data!
Page!32!