A Vision for a New Era in Computational Science
advertisement
Green Datacenter Initiatives at SDSC Matt Campbell SDSC Data Center Services Manager mattc@sdsc.edu SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO SDSC Data Center Overview • 19,000 sq ft • 13 MW of on-site power • 100+ projects operating IT equipment in data center 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 101 Gray system Blue jump-out A Liebert air handler Liebert air handler Liebert air handler C Vented Tile Vented Tile 104 D Vented Tile 103 B C 102 Black system distribution cabinet Vented Tile 35 Vented Tile 34 Vented Tile 33 Vented Tile 32 Liebert air handler Vented Tile 31 Vented Tile 30 Vented Tile 29 Vented Tile 28 Vented Tile 27 Vented Tile 26 Liebert air handler B Vented Tile 25 X 24 V 23 W 22 T 21 S 20 Liebert air handler U 19 R 18 Q 17 P 16 O 15 N 14 L 13 M 12 I 11 Liebert air handler J 10 K 9 F 8 H 7 E 6 G 5 B 4 Liebert air handler D 3 A 2 C 1 A D G UC colo UC colo UC colo Vented Tile UC colo UC colo UC colo UC colo UC colo UC colo UC colo UC colo Vented Tile UC colo Vented Tile UC colo Vented Tile UC colo UC colo Vented Tile UC colo UC colo UC colo UC colo UC colo UC colo Vented Tile UC colo UC colo UC colo UC colo Iso-base area U2G (empty) UC colo Vented Tile I J UC colo Vented Tile Vented Tile UC colo Vented Tile Vented Tile SAN 41 HPSS UC colo 240v Iso-base area UC colo Vented Tile UC colo Vented Tile Vented Tile Vented Tile UC colo Vented Tile Vented Tile 96 sq. ft. 240v Vented Tile UC colo O P Q R S T Vented Tile Vented Tile Vented Tile 118 240v Vented Tile UC colo 117 Vented Tile 116 Vented Tile Vented Tile Vented Tile R 118 Vented Tile UC colo Vented Tile R 117 UC colo UC colo Vented Tile Vented Tile R 116 UC colo Vented Tile O 116 Vented Tile UC colo G 121 Vented Tile G 120 Vented Tile UC colo G 119 UC colo UC colo 240v G 118 UC colo N Vented Tile UC colo SAN 44 HPSS 240v U2E U2D U2C U2B U2A Vented Tile UC colo 261 sq. ft. Vented Tile Vented Tile SAN 45 HPSS Vented Tile 261 sq. ft. Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile Vented Tile 240v Vented Tile UC colo SAN 46 HPSS Vented Tile Vented Tile 119 Vented Tile PDU-U2 UC colo R 119 UC colo UC colo O 125 UC colo O 124 Vented Tile Vented Tile O 123 UC colo O 122 UC colo Vented Tile O 121 Vented Tile UC colo O 120 UC colo UC colo O 119 UC colo 240v 240v Vented Tile O 118 Vented Tile Vented Tile O 117 UC colo Vented Tile UC colo SATA 3 & 4 J 124 UC colo R Iso-base area J 123 Q Distribution panel Vented Tile UC colo Iso-base area J 122 Vented Tile U1A J 121 Power Pillar J 120 Vented Tile UC colo U1B J 119 Power Pillar U2H J 118 Vented Tile UC colo U1C 240v J 117 Power Pillar U2I M Vented Tile UC colo SAN 43 HPSS Vented Tile Vented Tile Vented Tile 115 UC colo 240v Vented Tile Vented Tile Air return UC colo Vented Tile UC colo Vented Tile R 115 UC colo 316 sq. ft. UC colo Iso-base area240v 240v Vented Tile UC colo 316 sq. ft. L Vented Tile UC colo G 117 U2F U4B UC colo Power SAN 42 HPSS 349 sq. ft. 360 sq. ft. Vented Tile UC colo 240v U2F 9/11 & 13/15 & U2I 27/29 & 31/33 Vented Tile UC colo Iso-base area Vented Tile G 116 Pod A 1/3 & Pod B 29/31 UC colo Vented Tile UC colo Vented Tile O 115 UC colo 360 sq. ft. 240v Vented Tile UC colo Iso-base area Vented Tile UC colo Vented Tile K Vented Tile Vented Tile O 114 Vented Tile UC colo Iso-base area Vented Tile 240v Vented Tile 240v 240v New APC Vented Tile 240v Vented Tile UC colo U2E 32/34 & U2I 5/7 Vented Tile Pod B Pod A 10/12 & Pod B 47/49 UC colo Vented Tile 240v Power pod B 240v Vented Tile Vented Tile H Vented Tile UC colo 114 UC colo Vented Tile 113 Vented Tile Vented Tile 112 UC colo 111 UC colo UC colo 110 UC colo 109 Vented Tile Vented Tile 108 81 107 80 106 79 PDU-U1 78 105 77 R 114 76 R 113 75 R 112 74 R 111 73 R 110 72 R 109 71 70 R 108 69 Vented Tile R 107 68 Vented Tile R 106 67 R 105 66 O 113 65 Vented Tile UC colo 96 sq. ft. U2B O 112 Vented Tile UC colo Iso-base area UC colo O 111 UC colo 96 sq. ft. O 110 Vented Tile UC colo U2E 36/38 & U2I 14/16 Vented Tile 240v Vented Tile Vented Tile Vented Tile 240v U2F 32/34 & U2I 13/15 Vented Tile 240v U1C 2/4 & U2H 2/4 Power pod A Nesis and GEON UC colo O 109 240v U1C 5 UC colo UC colo G 115 Vented Tile P UC colo Iso-base area Vented Tile UC colo Air return 240v O Vented Tile UC colo UC colo O 108 Pod B 5/7 & U2I 38/40/42 Vented Tile UC colo O 107 U4B 26/28/30 & 37/ 39/41 Vented Tile SDSC Mars Vented Tile UC colo U2F 6/8 & U1C 26/28/30 Vented Tile 240v U2F 40/42 & U1C 22/24 240v U1C 31 & 33 & U2E 17 & U2I 21 Vented Tile Vented Tile O 106 Vented Tile IT-J14 HPSS tape movers 240v Pod A 30/32 & U2I 1/3 Pod A 26/28, 55/57 & Pod B 63/65 240v Vented Tile UC colo G 114 Vented Tile N UC colo UC colo O 105 240v M UC colo Iso-base area Vented Tile UC colo Vented Tile J 113 Vented Tile Vented Tile UC colo 240v Pod A 37/39 & U4B 9/11 (with in-rack UPS) U4B 32/34/36 & 38/ 40/42 Vented Tile Vented Tile J 112 IT-L7 F Vented Tile UC colo J 111 U2F 17 UC colo J 110 Pod A 64/66, 68/ 70 & Pod B 64/ 66, 68/70 Vented Tile G 113 Vented Tile Vented Tile UC colo 240v U2E 7/9 & 240v L UC colo G 112 U2F 26 Vented Tile Vented Tile 96 sq. ft. 240v K Vented Tile 240v 240v Vented Tile Vented Tile UC colo J 109 Vented Tile UC colo G 111 U2F 36/38 (both PDU’s feed from in-rack) Vented Tile G 110 U2E 24 Vented Tile UC colo PDU-N1 U1C 32 U2F 10/12 & 14/ 16 UC colo 240v U2E 31/33 & Vented Tile 6140 disk 240v Pod A 14/16 & U2K 26/28 240v J Vented Tile G 109 I 240v U2I 9/11 & U4B 21/23 240v U2I 26/28 & U4B 29/31 Vented Tile New APC 240v U2E 35/37 & U4B Vented Tile Engenio rack 240v Pod A 25/27 & Pod B 25/27 Vented Tile UC colo 240v G 108 Vented Tile E U2I 18 U1B 17/19 & U2H 25/27 Vented Tile 240v U2F 29/31 & U4B 17/19 Vented Tile New APC 240v H Vented Tile 240v U2F 5/7 & U1C 6/8 G 107 Pod A 22/24 & Pod B 22/24 G Pod B & U1C 10/12 240v J 108 U1C 14/16 & U2H 10/12 Pod A 6/8 & Pod B 55/57 J 107 Vented Tile 240v J 106 Vented Tile New APC 240v Pod A 21/23 & Pod B 21/23 J 105 Vented Tile G 106 New APC 240v F G 105 Pod A 18/20 & Pod B 18/20 E 240v Networking Core BB Vented Tile Vented Tile Vented Tile Vented Tile UC colo Vented Tile UC colo UC colo UC colo UC colo UC colo Vented Tile UC colo UC colo Vented Tile Vented Tile UC colo Vented Tile UC colo UC colo Vented Tile Vented Tile UC colo UC colo UC colo Vented Tile Vented Tile UC colo UC colo Vented Tile UC colo Vented Tile SAN 56 HPSS Vented Tile UC colo CC Vented Tile 240v UC colo Vented Tile UC colo UC colo UC colo Vented Tile UC colo UC colo UC colo Vented Tile Vented Tile Vented Tile Vented Tile SAM-QFS Vented Tile SAM-QFS DD G 132 UC colo 360 sq. ft. Vented Tile Vented Tile 240v U2J U3E Power Pillar Vented Tile UC colo U3D U3E-A Vented Tile UC colo UC colo Vented Tile UC colo UC colo Vented Tile UC colo UC colo Power Pillar Vented Tile Vented Tile UC colo Vented Tile GPFS-WAN UC colo Vented Tile Vented Tile GPFS-WAN UC colo Vented Tile UC colo Vented Tile Vented Tile GPFS-WAN Vented Tile U3A 1/3 & 5/7 Vented Tile Vented Tile Vented Tile Vented Tile 240v 240v U2I 39/41 HH EE FF GG SATA 1 & 2 Vented Tile UC colo U3A Distribution panel Vented Tile U3B-A HH Vented Tile GPFS-WAN server 136 UC colo 240v U3B Power Pillar U3C-A 135 Vented Tile UC colo Vented Tile UC colo U3D-A R 135 GG U3C Power Pillar Grey 1 UC colo Grey 2 G 135 U4A G 134 U2K FF Vented Tile G 133 Vented Tile UC colo U2K 13/15 (9311) U2K 10/12 (3590) PDU-U3 Vented Tile Iso-base area UC colo EE Vented Tile Vented Tile UC colo G 131 Vented Tile From Millennium DD Vented Tile Vented Tile PDB CC Vented Tile Vented Tile UC colo Vented Tile Vented Tile Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile UC colo 240v U2K 17/19 (9311) U2K 22/24 (3590) Vented Tile UC colo G 130 UC colo AA Vented Tile G 129 Vented Tile SAN 55 HPSS Vented Tile G 128 240v PDB Millennium U4A 1/3/5 Vented Tile Z SAN 54 HPSS Vented Tile Vented Tile Y Vented Tile Vented Tile From Millennium BB 184 sq. ft. Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile UC colo Vented Tile Vented Tile Vented Tile Vented Tile Vented Tile UC colo Vented Tile Vented Tile UC colo UC colo Vented Tile UC colo UC colo Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile UC colo UC colo 240v Vented Tile Vented Tile UC colo UC colo 240v Vented Tile Vented Tile Vented Tile SAN 53 HPSS Vented Tile 134 Vented Tile Vented Tile 133 UC colo G 125 UC colo Vented Tile PDU-N2 UC colo Vented Tile PDB 132 UC colo SAN 52 HPSS Vented Tile PDB Vented Tile 131 UC colo X 130 UC colo UC colo Iso-base area Power 240v From Millennium AA W Vented Tile 240v SAN 51 HPSS 129 UC colo UC colo Vented Tile Vented Tile 128 UC colo 127 UC colo Vented Tile 126 Vented Tile Vented Tile 125 Vented Tile UC colo 124 UC colo UC colo 123 UC colo Vented Tile Air return Vented Tile UC colo 122 UC colo Vented Tile Vented Tile Vented Tile Vented Tile 121 UC colo Vented Tile G 124 Vented Tile Vented Tile Vented Tile R 134 UC colo UC colo V R 133 UC colo UC colo U Vented Tile 240v R 132 UC colo 240v Vented Tile Vented Tile R 131 UC colo R 130 UC colo Vented Tile R 129 Vented Tile Vented Tile R 128 Vented Tile UC colo R 127 UC colo UC colo R 126 UC colo Vented Tile SAN 50 HPSS Vented Tile UC colo PDB Millennium U4B 1/3/5 U2K 9/11 (9311) U2K 18/20 (3590) U2I 35/37 (drive rack) Z UC colo R 125 UC colo U1C 7/9, 35/37 U2J 13/15, 17/19, 21/23, 25/ 27 Y Vented Tile SAN 49 HPSS colo ft. 129UCsq. R 124 Vented Tile Vented Tile Vented Tile R 123 UC colo G 123 U2K 5/7 (9311) U2K 6/8 (3590) U2I 6/8,10/12 (drive racks) X Vented Tile Iso-base area 240v W Vented Tile UC colo Vented Tile G 122 Vented Tile U3E-A 2/4, 6/8, 10/12 U4A10/12, 34/36, 37/39 Air return U1C 38/40 & U2J 26/28 V Vented Tile O 135 UC colo Vented Tile SAN 48 HPSS U2K 1/3 (9311) U2K 14/16 (3590) U2K 2/4 & U2F 35/37 (drive racks) U Vented Tile O 134 Vented Tile UC colo Vented Tile O 133 UC colo UC colo SAN 47 HPSS O 132 UC colo Vented Tile Vented Tile O 131 UC colo O 130 UC colo Vented Tile O 129 Vented Tile Vented Tile O 128 Vented Tile UC colo O 127 UC colo UC colo UC colo O 126 UC colo UC colo UC colo Vented Tile J 135 UC colo UC colo Vented Tile UC colo J 134 UC colo UC colo UC colo J 133 UC colo UC colo UC colo J 132 UC colo UC colo UC colo Vented Tile Vented Tile J 131 UC colo UC colo J 130 UC colo UC colo UC colo J 129 UC colo UC colo UC colo J 128 UC colo Vented Tile Vented Tile J 125 UC colo G 127 UC colo UC colo G 126 UC colo 240v Vented Tile UC colo 120 Vented Tile UC colo Vented Tile U2E 14/16 R 122 Vented Tile T R 121 S R 120 240v 240v 30 sq. ft. Vented Tile Vented Tile Liebert air handler JJ 240v GPFS-WAN Liebert air handler 240v KK Vented Tile Vented Tile Vented Tile UC colo UC colo Vented Tile Vented Tile Vented Tile UC colo Vented Tile UC colo Vented Tile Vented Tile Vented Tile Vented Tile Liebert air handler Vented Tile UC colo Vented Tile UC colo Vented Tile On-Demand cluster Vented Tile On-Demand cluster Vented Tile UC colo On-Demand cluster Vented Tile Vented Tile UC colo Vented Tile Vented Tile II Vented Tile U2K 29/31 & U2K 36 U3E-A 6/8 & 10/12 LL 141 240v MM MM Liebert air handler CDSB SBCDS2 Liebert air handler CDSC CDSD CDSE NN Liebert air handler 143 Liebert air handler 142 Liebert air handler NN OO OO PP Empty 240v 5 6 7 8 Physics storage U2K 38 Empty XFMR CDSA XFMR CDSB XFMR CDSC XFMR CDSD QQ XFMR CDSE Empty RR 240v 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 PDU-N3 147 4 PP CDSA 240v Empty 240v 146 3 Vented Tile 240v CMS networking 240v 2 Vented Tile 240v Empty 240v CMS disk 1 Vented Tile CMS blades Vented Tile RR Vented Tile 145 CMS disk QQ Empty 240v Vented Tile 144 Empty Vented Tile 140 240v U2K 21/23 LL 139 Vented Tile 240v KK Vented Tile GPFS-WAN server 240v 138 JJ 240v Iso-base area Vented Tile Vented Tile Vented Tile UC colo 240v Vented Tile On-Demand cluster UC colo Vented Tile 137 II 81 SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Key Points • How do we illustrate energy loss and quantify savings? • Establishing a common unit of measurement for data centers • What areas can be improved? • Mechanical • Electrical • IT equipment SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Power Usage Effectiveness Total Facility Power PUE = IT Equipment Power • Industry accepted unit of measurement • 2+ is still the data center industry ‘norm’ • SDSC’s datacenter came was audited last year with a resulting PUE of 1.35 – 1.42. • Systems located at SDSC would save 28-65% on energy alone! SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Energy Savings Multiply Courtesy Jack Pouchet, Emerson SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Working on Many Fronts Efficient System Lifecycles Distributing Higher-Voltage (277/480V) Power to the floor to reduce transformer and distribution losses. Experimenting with DC-Only Power Distribution “Free Cooling” Opportunities High-efficiency UPS Retrofit Air handling retrofit with variable frequency drive fans and modulating chilled water valves to match cooling to real-time loads. Cold / Hot Aisle Containment SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Data Center Improvements to Lower PUE • Mechanical Systems • Aisle containment • CRAH VFD retrofits • Blanking panels, floor brushes • Electrical Systems • High efficiency UPS • System Lifecycles • Virtualization • Consolidation of ‘ghost’ servers SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Cold/Hot Aisle Containment • • • Expect 50-80% better cooling airflow efficiency than traditional hot/cold aisle Separates cold/hot air, eliminating the need for over-cooling, over-blowing Allows for high temperature differentials, maximizing efficiency of cooling equipment Cold Aisle Containment • Knuerr Coolflex • • • • Hot Aisle Containment • First deployment in the United States Requires a standardized datacenter Enclosed cold aisles (70-78F) Servers exhaust to room as a whole, which runs hot (90-100F) • • • Essentially a “heat chimney” on top of hot aisles with doors at each end. Flexible, can be phased incrementally Cold room as a whole (70-78F) Exhaust to enclosed hot aisle (100+F), ducting directly to cooling systems SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Filling In The Gaps •Cost effective blanking panels •Minimizes leakage • Brushes to cover tile cutouts • Allows cables to pass through while blocking air SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO CRAH VFD Retrofits • Added variable frequency drives to computer room air handlers • Allows CRAH units to modulate based on control signal SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Electrical Systems • High efficiency UPS units that can save ~500,000 kWh/year • Higher voltage to IT equipment • Providing 277/480V power on the floor to more efficiently power HPC systems without transformer and distribution losses • Exploration of DC power for IT equipment SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO Hardware Lifecycle Efficiency • Hardware retirement • Equipment refresh more often leverages technology improvements • Virtualization • Maximize equipment utilization • Consolidation of ‘ghost’ servers • Underutilized IT equipment • ‘Forgotten’ equipment SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO QUESTIONS? SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
