• Storage area network and System area network
(SAN)
– What are they?
– Network requirements
– Hardware/software issues
– References:
• Ulf Troppens, Rainer Erkens, a nd Wolfgang Muller, “Storage
Networks Explained - basic and application of Fibre Channel
SAN, NAS, iSCSI and Infiniband”, John Wiley & Sons, 2004.
• W. J. Dally and B. Towles, “Principles and Practices of
Interconnection Networks”, Morgan Kaufmann, 2004.
• Ajay V. Bhatt, “Creating a Third Generation I/O Interconnect,”
available at http://www.express-lane.org
• Storage area network (SAN):
– Server-centric IT architecture: storage devices
exist only with servers
• Storage-centric IT architecture: SCSI cables
are replaced by a network (storage is now
independent of servers).
• Storage area network (SAN) requirement:
– Serial transmission for high speed and long
distance
– Low transmission errors
– Low delay of transmitted data
• Needs to make it feel like using a local disk
• Low delay is a relative term:
– The disk subsystem has around 1ms – 10ms latency itself.
– The communication protocol should not use
CPU.
• Current Storage area network (SAN)
technology (IBM):
– Fibre Channel
– TCP/IP + Gigabit Ethernet (iSCSI)
– InfiniBand
• System area network: a network with a high
bandwidth and a low lantency that serves as
a connection between computers in a
distributed computer system.
• Why system area network:
– Historically, the system area network comes with a
particular parallel machine (supercomputer, e.g. Cray
T3D, Cray T3E, SGI origin 2000, IBM SP, Thinking
machine CM5, Intel Polygon)
• The network is very expensive due to low volume
• CPU is two generations behind
– A more cost effective way to build these system is to
decouple the processor technology from the networking
technology.
– To form cheaper clusters of workstations with the offthe-shelf system area network technology (compared to
traditional supercomputers).
– current system area networks:
• Myrinet, Quadrics, Infiniband
• System area network requirement:
– Low latency and high bandwidth at the
application level.
• Not just at the hardware level
• Not just at the system level
– Implicitation:
• Hardware, network interface, software messaging
layer should work together to achieve the goal.
– Infiniband is designed as both storage area
network and system area network.
• Hardware issues:
– High speed links:
• Infiniband: 2.5Gbps = 250MBps, 10Gbps=1GBps,
30 Gbps = 1GBps
• Fibre channel: 100MBps, 200MBps, 400MBps,
1GBps.
• Myrinet: up to 9.6Gbps
• As a reference PCI bus: 100MBps
– NIC may need to attach to the memory bridge
• A typical PC:
• A workstation connected to a system area
network:
• When the number of end points is large,
multiple switches will be needed.
• Topology
• Switching
• Routing
• Topology
– Static arrangement of channels and nodes in an
interconnection network
– Trade-off between cost and performance
• Cost: the number and complexity of chips, density
and length of the interconnections, etc.
• Performance:
– Bandwidth and latency: also depend on other factors other
than topology
– Topology performance metrics: Bisection bandwidth,
diameter, nodal degree, channel load
• A cut of a network is the set of channels that
partitions the set of all nodes into two disjoint sets.
• A bisection of a network is a cut that partitions the
network nodes in roughly half.
• The bisection bandwidth of a network is the
minimum bandwidth over all bisections of the
network.
• The diameter of a network is the largest minimal
hop count over all pairs of nodes.
• Under a particular traffic pattern, the channel that
carries the largest fraction of traffic determines the
maximum channel load of the topology.
• Example topologies:
– Regular or irregular
– Regular topologies are mostly derived from two
main families: butterflies (k-ary n-flies) or tori
(k-ary n-cubes)
• Switching: how a packet pass a switch
– Message/packet/flit
• Traditional scheme: store-and-forward
– Time = H (S + P/B)
• Cut-through switch:
– Forward to the next link after the header flit is
received. Stop only when the next hop buffer is
not available.
– Time = H S + P/B, when S << P/B, the time
does not depend on the number of hops!!!
• Wormhole routing:
– Cut-through switches still allocate buffer to
packets. May require a large amount of buffers
– Wormhole routing only allocates buffer for one
flit for each packet.
– Latency is the same as cut-through switching.
– When the packet is block, the whole flit “train”
is block, occupying links.
• Solution: add more virtual channels.
• The deadlock problem in wormhole routing:
– Need deadlock free routing scheme to select the
right path
• Cut-through switch and wormhole switch
are widely used in system are networks
– Routing in such systems is an issue!!
– Shortest path routing may result in deadlock.
– Deadlock free routing:
• Cut-through switch and wormhole switch
are widely used in system are networks
– Routing in such systems is an issue!!
– Shortest path routing may result in deadlock.
– Deadlock free routing:
• Basic idea: fix the priority of channels and using the
channels with increasing priority.
• Example: up/down routing
• Up/down routing:
– Select a node as the root
– Build a spanning tree from the root
– Nodes are partitioned into layers based on the position
in the spanning tree
– The channel from a lower layer node to a higher layer
node is the up link, the channel from a higher layer
node to a lower layer node is a down link, channels
between nodes in the same layer are marked as up or
down link based on the node number
– In the valid route: an up channel cannot follow an down
channel.
– These exists at least one valid path between each pair of
nodes.
• Problems with deadlock free routing:
– Load balancing is a problem, traffic are not
evenly distributed
– Non-adaptive version of the deadlock free
routing scheme is also a problem
• How to map the routes in order to get good
performance (metrics: maximum channel load?)
• More on the problem to be discussed later.
• Hardware/software codesign and software
API issues:
– What functionality should be implemented in
the hardware.
• E.g. adaptive routing may imply out of order packets
– Chien’04 paper gives good answers to some of
these questions.