• Infiniband architecture
– Specification (Infiniband
architecture specification release 1.2,
Oct. 5, 2004) available at Infiniband
Trade Association
(http://www.infinibandta.org)
• Potential improvements
• Infiniband architecture overview
• Infiniband architecture overview
– Components:
•
•
•
•
Links
Channel adaptors
Switches
Routers
– The specification allows Infiniband wide area network,
but mostly adopted as a system/storage area network.
– Topology:
• Irregular
• Regular: Fat tree
– Link speed:
• 2.5Gbps (X), 10Gbps (4X), and 30Gbps (12X).
• Layers: somewhat similar to TCP/IP
– Physical layer
– Link layer
•
•
•
•
Error detection (CRC checksum)
flow control (credit based)
switching, virtual lanes (VL),
forwarding table computed by subnet manager
– Not adaptive
– Network layer: across subnets.
• No use for the cluster environment
– Transport layer
• Reliable/unreliable, connection/datagram
– Verbs: interface between adaptors and OS/Users
• Packet format:
• Local Route Header (LRH): 8 bytes. Used for local routing by
switches within a IBA subnet
• Global Route Header (GRH): 40 Bytes. Used for routing
between subnets
• Base Transport header (BTH): 12 Bytes, for IBA transport
• Reliable datagram extended transport header (RDETH): 4
bytes, just for reliable datagram
• Datagram extended transport header (DETH): 8 bytes
• RDMA extended transport header (RETH): 16 bytes
• Atomic, ACK, Atomic ACK,
• Immediate DATA extended transport header: 4 bytes,
optimized for small packets.
• Invalidate
• Invariant CRC and variant CRC:
– CRC for fields not changed and changed.
• Local Route Header:
– Switching based on the destination port address (LID)
– Multipath switching by allocating multiple LIDs to one
port
• Local Route Header:
– Switching based on the destination port address
(LID)
– Multipath switching by allocating multiple
LIDs to one port
• GRH: same format as IPV6 address (16
bytes address)
• Base transport header:
• Verbs
– OS/Users access the adaptor through verbs
– Communication mechanism: Queue Pair (QP)
• Support the four types of services, including reliable
connection service
• Each connection takes one QP on each end.
• Each QP has a send queue and a receive queue.
• Users can post send requests to the send queue and
receive requests to the receive queue.
• Three types of send operations: SEND, RDMA(WRITE, READ, ATOMIC), MEMORY-BINDING
• One receive operation (matching SEND)
• Queue Pair:
– The status of the result of an operation
(send/receive) is stored in the complete queue.
– Send/receive queues can bind to different
complete queues.
• Related system level verbs:
– Open QP, create complete queue, Open HCA,
open protection domain, register memory,
allocate memory window, etc
• User level verbs:
– post send/receive request, poll for completion.
• To communicate:
– Make system calls to setup everything (open QP, bind
QP to port, bind complete queues, connect local QP to
remote QP, register memory, etc).
– Post send/receive requests.
– Check completion.
– What if a packet arrives before a receive request is
posted?
• Not specified in the standard
• The right response should be a ‘receiver not ready (RNR)’
error. The sender is back-pressed in this case.
• Infiniband has a perfect software interface
(Chien'94 paper):
– The network subsystem realizes all user level
functionality.
– User level accesses to the network interface. A
few machine instructions will accomplish the
transmission task without involving the OS.
– Network supports in-order delivery and and
fault tolerance.
– Buffer management is pushed out to the user.
• SilverStorm 9024:
–
–
–
–
–
–
24 ports 4X(10Gbps) or 8 ports 12X(30 Gbps)
switch type:
cut-through
switch latency:
< 140ns
switch bandwidth: 480 Gbps
forwarding table size: 48K
VL support:
8 + 1 management
• SilverStorm 9240:
– 24 expansion slots, each expansion model 12
port 4X or 4 port 12X (24x12 = 288, 288 by
288 switch)
– switch type
cut-through
– switch latency: < 140ns to < 420ns
– switch bandwidth: 5.76Tbps
– forwarding table size: 48K
– VL support: 8 + 1 management
• Potential improvements on Infiniband using
compiled communication
– Improving the internal Infiniband fabric:
• Offline routing for static pattern (static SM for a
reduced traffic pattern) can be beneficial for
irregular networks.
• Simplify the layer architecture by having a direct
link model (for known patterns), the header can be
simplified, may not matter much (Infiniband layers
are thin).
• Simplify the protection mechanism.
• Circuit switch type Infiniband.
• Reliable communication protocol is still needed.
• Potential benefits can be evaluated by simulation.
• Improving the messaging software (software to
hardware interface): no chance.
• Improving the MPI implementation over
Infiniband: similar to our current work on Ethernet
– Message scheduling for collective/point-to-point
communications based on the network topology.
– Exploring NIC features (buffers in NIC, multicast)
– Reducing the number of instructions in a library routine
makes sense. Compiled communication can be used to
optimize the MPI library.
– Compiled communication can help improving the
library implementation (e.g. reducing the number of
message copies, early requests posting , using RDMA,
etc).
• One particular project:
– Design algorithms for Infiniband subnet manager
– Improving routing performance for Infiniband subnet
manager (SM).
• Objective: minimize the maximum channel load for an given
traffic pattern
• Optimize according to a given pattern: the traffic pattern in an
application is usually not all-to-all
– Default routing used in IBA SM
• For a sparse traffic pattern, the maximum channel load can
usually be minimized using the minimim interference
principle.
– Need to extend minimum interference routing for load balance
deadlock free routing.
– The best way to realize IBA SM is still not clear
(unknown) at this time, we can probably do something
here.
• Irregular network or Fat tree network