Page 1 of 9
ECP-150014 ADDITIONAL INFORMATION NOTE
This note contains additional information pertinent to the SKA1 Mid CSP to INFRA ICD ECP, ECP-150014.
Nicolas Loubser
August 25, 2015
CBF Cooling Assumptions
Due to increased power in the CBF, and the indication that liquid cooling at KAP-B is feasible, liquid cooling
for these is now the baseline. INFRA will provide an (external-building, TBC) heat exchanger c/w circulating
pump(s) and radiator and closed-cycle water/glycol flow of O(15 L/sec) TBC to the MID.CBF rack via supply
and return pipes of O(3”) diameter each TBC. It is believed that the external heat exchanger can operate at
external ambient temperature, with radiator fan control to maintain ~constant cold-line liquid temperature
(i.e. to avoid diurnal temperature fluctuations at each LRU.)
Liquid Cooling System Control and Monitor
The plan is to employ ambient liquid cooling, provided by INFRA connecting water flow to CBF rack I/O
pipes. INFRA provides an external pump and heat exchanger, with inlet cold water temperature <~50
degrees C and seeing a temperature rise of a few degrees C on exit. Likely this means an external heat
exchanger can cool to external ambient air temperature, but with heat exchanger blower (or fluid flow
rate) control to maintain constant cold water temperature to avoid diurnal temperature variations which
cycle the electronics temperatures accordingly. It is better to be consistently running warmer rather than
cycling temperature. Very slow seasonal temperature variations are likely not a problem.
All mezzanine cards will be fitted with humidity sensors to (hopefully) detect even very small leaks likely
seen as humidity anomalies. The INFRA system should have pressure monitors to shut down the system if
there is a sudden pressure drop or anomaly. Also, there needs to be a guaranteed communication path
from MID.CBF Master to signal TM that a humidity anomaly has been detected so that INFRA can turn off
water pressure and shut down power to MID.CBF (by turning the -48 VDC power supply off).
CBF Blade Liquid Cooling Design
The cooling design consists of F, X, and BF blades with PowerMX-motherboard-monolithic liquid-cooling
plates (see: http://www.aavid.com/sites/default/files/products/liquid/pdf/Aavflow.pdf) attached to power
dissipating devices and with front-panel input and output liquid cooling lines attached to rack-mounted
manifolds.
Rough estimate is that at each of the top (or bottom corners, tbd by INFRA) corners of the CBF rack there
will be a ~3”-3.5” diameter pipe with 2 of those pipes being inputs, and 2 being returns. These tie into
piping provided by INFRA routing to an external-to-the-building heat exchanger (air-cooled radiators
Page 2 of 9
cooling to external ambient) and circulating. A rack-level liquid-cooling concept diagram is shown in Figure
1, with a basic flow/heat exchanger diagram shown in Figure 2.
Cool Liquid In
Warm Liquid Out
To/From External Heat Exchanger
Water Outlet Manifold
Water Inlet Manifold
Rail
Rail
B
l
a
d
e
B B B B B B B
l l l l l l l
a a a a a a a
d d d d d d d
e e e e e e e
B
l
a
d
e
B B B B B B B
l l l l l l l
a a a a a a a
d d d d d d d
e e e e e e e
Rail
Rail
Rail
Rail
B
l
a
d
e
B B B B B B B
l l l l l l l
a a a a a a a
d d d d d d d
e e e e e e e
B
l
a
d
e
B B B B B B B
l l l l l l l
a a a a a a a
d d d d d d d
e e e e e e e
Rail
Normal room air
flow from floor
Figure 1 Notional Correlator/Beamformer rack liquid cooling concept. A constant-pressure
Water Inlet Manifold is used to deliver constant flow to individual blades at an estimated
~250 mL/sec per blade for ~10 C Trise.
Liquid Cooling Concepts
A brief description of the thinking regarding liquid cooling concepts is warranted and is presented here.
The commercial Asetek (www.asetek.com) solution uses 3 stages of heat exchangers for their server box
liquid cooling scheme, 1) per-chip liquid plate/heat exchanger/pump, pumping liquid for the server box
with several such pumps in series to/from rack-level CDU (Cooling Distribution Unit) in closed loop; 2) Racklevel CDU with exchangers between server box flow and system fluid flow; 3) external building system heat
exchanger from system fluid flow to ambient air. It is unclear exactly why there are 3 stages, however it is
likely to keep each server box fluid level isolated to minimize spill due to failure, and that if there is a failure
in this last stage, the entire rack and perhaps data center does not have to shut down. Asetek also has a
closed-air cycle in each server box to allow for air cooling of components not amenable to liquid-plate
attachment such as memory DIMMs, disk drives etc.; there is no particular need in MID.CBF for this. Asetek
Page 3 of 9
indicates their system is capable of 50 kW per 19” rack and operates up to 40 C external ambient
temperature.
The MID.CBF is likely higher power density than can be handled by the Asetek system directly, although
perhaps the Asetek approach could be used if needed. There is also the freedom in MID.CBF—as long as
availability requirements can be met—that in the event of a blade feed line, liquid plate, or return line
failing, the entire system can be shut down and the liquid system evacuated to minimize spillage (noting
that there are millions of households with high-pressure hot and cold water distribution lines running in
enclosed/inaccessible places which are, by all indications, very reliable).
It is possible that each blade dissipates ~2.4 kW for a total of ~77 kW per shelf, double that per rack (noting
that the MID.CBF rack is ~1 m W x ~2.4 m D x ~1 m H). A two-stage heat exchanger (one is the external
heat exchanger, the other is blade electronics to cooling plate) is one possibility, the notional flow/control
diagram shown in Figure 2Error! Reference source not found.:
External Building Pad
Ambient air cooling
Heat exchanger
Pump
Flow/
Pressure/
Temp
Monitor
KAP-B Screened Room
MID.CBF Rack
Constant Pressure & Temp
Flow/
Pressure
Monitor
Control
System
Blade
Liquid
Cooling
Plates
MID.CBF
Blade
Electronics
Constant
flow valve
Humidity
Sensors
CSP
LMC
Figure 2 Two-stage heat exchanger concept for MID.CBF. The “External Building Pad” and
lines to/from the MID.CBF rack are an INFRA responsibility.
The external Control System maintains a constant (low) pressure (say <10 psi) and temperature (<50 C to
avoid diurnal variations) on the Water Inlet Manifold (2 of them one for the front and one for the rear of
the rack). Constant-flow bleeder valves then are used off the Inlet Manifold to feed each blade with ~250
mL/sec of fluid flow, with the warm return going back to the external system via a large low-pressure
return. The manifolds are engineered for very high reliability and long lifetime, as are all lines feeding it.
Flow/pressure monitors on the outgoing and return flow are used to maintain constant pressure and detect
leaks (via differential flow rates). Humidity sensors in the MID.CBF electronics report to CSP LMC which
provides alarms to the Control System about possible leaks.
Page 4 of 9
Key issues are believed to be, 1) dripless interconnects—perhaps Asetek technology can be used as they
have patented dripless technology; 2) blade liquid cooling plates—Aavid Thermalloy has “Aavflow”
parameterized plates that can be ordered to fit our needs (i.e. monolithic 4 x PMXM size); 3) external heat
exchanger, pump, and control system—commercial systems exist, but it is unknown if they will do exactly
what is required; 4) design of the rack and blade cooling mechanics to mitigate the impact in the event of
leaks.
A few more informative diagrams of key concepts are shown in the following figures.
Figure 3 Top: Aavid-Thermalloy Aavflow liquid cooling plate with large semi-conductor
poised to be attached. Bottom: thermal performance curves for ~38 in2 heat source area.
The PowerMX 4xPMXM plate would have a surface area of ~64.5 in2 although the thermal
resistance is likely largely governed by the heat capacity of coolant rather than surface area
for this size of plate. From the graph, the thermal resistance for 1 GPM (~63 mL/sec) is
~0.0068 C/W which, for P=600W, is Trise~=4 C. This would seem to violate fundamental H2O
heat capacity calculations that ~60 mL/sec, and ~600 W would see a 10 C temperature rise.
Page 5 of 9
Figure 4 Some initial thermal analysis with the Aavflow liquid cooling plate on a PMXM
module, complements of Scott Johnson at Imagination Machine Works, Kelowna, BC,
Canada.
CBF Electrical , Space and weight Footprint on INFRA
Find below in the figures an overview of the Electrical and Space footprint that CBF has on INFRA.
Also see the CSP_FloorAndPowerEstimates_2015_06_26” spreadsheet and the CSP_all_SLD_asof_2015-0615 Visio diagram for further details. The most recent power estimate for the CBF processor rack is 150KW.
The most recent weight estimate for the CBF processor rack is 1126 kg. The CBF Processor rack
dimensions including the cooling system is 2100mm high, 1050mm wide and 2500mm deep
Page 6 of 9
TM
100 MHz encoded
clock on fiber
SADT Clk&Timing
Air
MID.CBF LMC CPU
Air
-48VDC @ ~1700 A
Exhaust fan(?)
1000BASE-T
48-port + 4xSFP
GigE Switch
MID.CBF LMC CPU
52 x 100GBASE-SR4
PST Signal Gen.
10GBASE-SR(?)
COTS 19" rack
Cold liquid in,
Warm liquid out
(~16 L/sec) < 50C
Correlator Shelf
(~70 kW)
52 x 100GBASE-SR4
Cap. Transient Data
PSS, PST channels
96 x MPO fiber
MT-RJ fiber
48-port + 4xSFP
GigE Switch
2 x 19" COTS racks
16 x 100GBASE-SR4
VLBI beams to SDP fac.
32
208 x
100GBASE-4
1000BASE-T
64 x 100GBASE-SR4
vis. to SDP
Active Clk&Timing Splitter
1000BASE-T
Air
2X 16x16 100GBASE-4 Sw.
-48 VDC
Power
Rack(s)
32
2X 16x16 100GBASE-4 Sw.
133 x SKA1 DISH @
100GBASE-4 ea
+
64 x MeerKAT @
40GBASE-4 ea
16
Beamformer Shelf
(~77 kW)
(x16)
PSS beams
~7 kW
PST beams
2X 16x16 100GBASE-4 Sw.
Active Clk&Timing Splitter
Floor load
TBD
230VAC
(~10 kW)
~2330 kg
over 0.9 m^2
(~4.1 psi)
-48VDC @ ~1900 A
Low-flow
room air
-48VDC @ ~200 A
Floor load:
2 x 450 kg
208/230V or
480V 3ϕ AC
(~164 kW)
Figure 0-5 Total MID.CBF sub-element block diagram with updates due to design refinements, rebaselining, and with anticipated ECPs and
requirements changes, particularly in the beamformer (double-beamformer processing option [power] shown).
“Lane”
From
PST sky
signal gen
X
X
X
4 ants, 5 GHz/poln
X
X
X
X
4 ants, 5 GHz/poln
X
X
X
X
X
X
X
X
X
XX
4 ants, 54GHz/poln
ants, 5 GHz/poln
4 ants, 5 GHz/poln
16 ant
4 ants, 5 GHz/poln
X
X
X
X
X
X
XX
X
XX
X
X
X
X
XX
X
XX
X
X
X
X
XX
X
X
X
XX
X
X
F-Blade
16
16 corr X-blade
XX
m
XX
4 ants, 54GHz/poln
ants, 5 GHz/poln
4 ants, 5 GHz/poln
256x256
100G/40G
External switch
X
16 ant F-Blade
X
4 ants, 54GHz/poln
ants, 5 GHz/poln
16 ant
X
X
X
4 ants, 5 GHz/poln
4 ants, 5 GHz/poln
X
4 ants, 5 GHz/poln
~23" = ~0.59 m
16 corr X-blade
3
X X-blade
X
16 corr
2
F-Blade
16 corr X-blade
16 ant F-Blade
Visibilities + VLBI beams to SDP,
PSS/PST channels to beamformer
X
~0
.9
1
4 ants, 5 GHz/poln
4 ants, 5 GHz/poln
"=
2
16 ants in
3
133
SKA1
DISHes
Plug-in
“X-part”
16
64
MeerKATs
1
“F-part”
Plug-in
~3
6
Inputs from SKA1 and
MeerKAT DISHes
(256 max total)
To external
transient
processor
1
~94" = ~2.4 m
Liquid cooling
in/out
Residual
room airflow
Residual
room airflow
Liquid cooling
in/out
Figure 0-6 Modified MID Correlator simplified view. The front-end switch is external to the shelf and may ultimately be a COTS switch.
Plug-in
Plug-in
Aggregator
beams
beams
All PST, PSS beams, 1/64th BW th
All PST, PSS beams, 1/64 BW th
All PST, PSS beams, 1/64 BW
beams, 1/64th BW
All PST, PSS beams, 1/64th BW th
All PST, PSS beams, 1/64 BW
beams
beams
All PST, PSS
beams
beams
beams
beams
All PST,
PSS
beams
beams
All PST, PSS beams, 1/64th BW
th
All PST, PSS beams,
1/64th
BWBF-blade
1/16
BW
beams
th
All PST, PSS beams,
BW
1/16 1/64
BW
BF-blade
beams, 1/64th BW
All PST, PSS beams, 1/64th BW th
All PST, PSS beams, 1/64 BW
beams
beams
beams
beams
All PST, PSS beams, 1/64th BW
th
All PST, PSS beams,
BW
1/16th1/64
BW
BF-blade
th
All PST, PSS beams, 1/64th BW
th
th
All PST, PSS beams,
BW
1/161/64BW
BF-blade
1/16th BW BF-blade
1/16th BW BF-blade
1/16th BW BF-blade
~23" = ~0.59 m
Final
PSS beams
Final
PST beams
16
Final
PSS beams
Final
PST beams
Inputs from PSS, PST
channelizers in Correlator;
Aggregated Beam outputs to
PSS/PST
Aggregator
th
beams
beams
1/16
BW
BF-blade
beams
beams
All
PST, PSS beams, 1/64th BW
All PST, PSS beams, 1/64th BW
All PST, PSS beams, 1/64th BW
Aggregator
1/16th BW PST, PSS channels from Correlator
1/16th BW PST, PSS channels from Correlator
Final
PST beams
All PST, PSS beams, 1/64th BW
1/16th BW PST, PSS channels from Correlator
Final
PSS beams
beams
beams
beams
beams
All PST, PSS beams, 1/64th BW
All PST, PSS beams, 1/64th BW th
Final
All PST, PSS beams, 1/64 BW
PST beams
All PST, PSS beams, 1/64th BW
beams
beams th
PST, PSS beams, 1/64th BW
All PST, PSS beams, 1/64th All
BWPST, PSS beams, 1/64 BW
All PST, PSS beams, 1/64thAll
BW
Aggregator
1
beams
beams
Aggregator
2
1/16th BW PST, PSS channels from Correlator
3
All PST, PSS beams, 1/64th BW
Aggregator
Inputs from PSS, PST
channelizers in Correlator;
Aggregated Beam outputs to
PSS/PST
Final
PST beams
All PST, PSS beams, 1/64th BW
All PST, PSS beams, 1/64th BW
Aggregator
Final
PSS beams
beams
beams
All PST, PSS beams, 1/64th BW
1/16th BW PST, PSS channels from Correlator
16
1/16th BW PST, PSS channels from Correlator
“BF-part”
3 ~36" = ~0.91 m
2
1
~95" = ~2.4 m
Liquid cooling
in/out
Residual
room airflow
Residual
room airflow
Liquid cooling
in/out
Figure 0-7 Modified beamformer with “2X/double” processing capacity in the same shelf by plugging in BF-blades into the front and rear of the
shelf and integrating the Beam Aggregator into each blade.
CBF PROCESSOR RACK FOOTPRINT, POSITIONING
Short heavy copper bus bars connect the -48 VDC power supply racks to the CBF Processor rack. An example is provided
below of how the CBF Processor rack (CBF M-rack below) could be arranged with respect to the INFRA facility rack rows.