Interconnection Topologies
• Networks scaling with N
• Logical Properties:
• distance: number of links on route
• degree: number of links in a switch
• diameter: maximum routing distance
• Physical properties
• length, width
Network Bandwidth
• Local bandwidth: bandwidth available to each node individually
effective local bandwidth
 n 

 b * 

n

n
E


• b is raw bandwidth (bytes/sec)
• n packet length
• nE header information
• Aggregate bandwidth
• Entry bandwidth: Sum of the bandwidth of all links of the nodes into
the network.
• Bisection bandwidth: Sum of the bandwidth of the minimum set of
links that, if removed, partition the network into two equal
unconnected sets of nodes.
Fully Connected network
• diameter = 1
• degree = N
• cost?
• bus => O(N), but BW is O(1)
• crossbar => O(N2) for BW O(N)
Linear Arrays and Rings
Linear Array
Torus
Torus arranged to use short wires
• Linear Array
• Diameter N-1
• Average Distance N/3
• Bisection bandwidth 1
• Route A -> B given by relative address R = B-A
– A and B are node numbers (0  N-1)
• Examples:
• FDDI, SCI, FiberChannel Arbitrated Loop, KSR1
Multidimensional Meshes and Tori
• d-dimensional array
• N = kd-1 X ...X kO nodes
• described by d-vector of coordinates (id-1, ..., iO)
• Routing
• relative distance: R = (bd-1 - ad-1, ... , b0 - a0 )
• traverse ri = bi - ai hops in each dimension
• dimension-order routing
Properties
• Routing
• relative distance: R = (bd-1 - ad-1, ... , b0 - a0 )
• traverse ri = bi - ai hops in each dimension
• dimension-order routing
• Diameter:
• d*(k-1) if k0=k1=...kd-1=k for mesh
• d*k/2 for torus
• Average Distance
• d * k / 3 for mesh
• d * k / 4 for torus
• Degree:
• d to 2d for mesh, 2d for torus
• Bisection bandwidth:
• k**(d-1) bidirectional links for mesh, 2 times for torus
Diameter (d=2)
70
60
50
40
Mesh
30
Torus
20
10
0
2
4
8
16
32
Hypercubes
• Also called binary n-cubes, n=#dimension, #nodes = N = 2n
• Diameter n=log N
• Bisection bandwidth 2n-1
0-D
1-D
2-D
3-D
4-D
5-D !
Gray Code
0-D
(0)
(0,1,1)
(1,1,1)
3-D
1-D
(0)
(1)
(0,0,1)
(1,0,1)
(0,1,0)
(0,1)
(1,1,0)
(1,1)
2-D
(0,0)
(1,0)
(0,0,0)
(1,0,0)
Routing in Hypercubes
• Nodes are connected to n nodes that differ by exactly one bit in address.
(Gray Code)
• Routing:
• length: number of ones in A xor B
• dimension order routing: hops in the
non-zero dimensions
Properties of Some Topologies
Topology
Degree
Diameter
Ave Dist
Bisection
Ave Dist N=1024
1D Array
2
N-1
N/3
1
huge
1D Ring
2
N/2
N/4
2
huge
2D Mesh
4
2 (N1/2 - 1)
2/3 N1/2
N1/2
21
2D Torus
4
N1/2
1/2 N1/2
2N1/2
16
Hypercube
n=log N
n
n/2
N/2
5
• All have some “bad permutations”
• many popular permutations are very bad for meshes (transpose)
• randomness in wiring or routing makes it hard to find a bad one!
K-ary Trees
• height d = logk N
• Fixed degree
• Diameter and avg. distance are logarithmic
• Bisection BW?
Addressing in k-ary Trees
Level 1
0
1
Level 0
(0,0)
(0,1)
(1,0)
(1,1)
• address specified as d-vector
• describing path down from root (kd,…,k0)
• Route up to common ancestor and down
• going from A to B: R = B xor A
• let i be position of most significant 1 in R, route up i+1 levels
(common ancestor)
• down in direction given by low i+1 bits of B
Example: Routing in a Tree Network
000
001
010
011
100
101
110
111
Route from 000 to 001: 000 xor 001 = 001
Route from 010 to 111: 010 xor 111 = 101
Position i+1= 1  1 level up
Position i+1= 3  3 level up
Last bit of B is 1
Last 3 bits of B is 111
Fat tree
• Routing AB:
• Select random switch C in the least common ancestor of A and B
• Take unique tree route from A to C
• Take unique tree route back from C to B
• Let i be position of most significant 1 in B xor A; then there are 2i root nodes to
choose from. The longer the routing distance the more the traffic can be
distributed.
• Tree network in a partition of the Altix 4700, Roadrunner
How Many Dimensions in Network?
• d = 2 or d = 3
• Short wires, easy to build
• Many hops, low bisection bandwidth
• Requires traffic locality
• d >= 4
• Harder to build, more wires, longer average length
• Fewer hops, better bisection bandwidth
• Can handle non-local traffic
• k-ary d-cubes provide a consistent framework for comparison
• N = kd
• scale dimension (d) or nodes per dimension (k)
Traditional Scaling: Latency(P)
140
Ave Latency T(n=40)
120
100
d=2
d=3
80
d=4
k=2
60
n/w
40
20
0
0
5000
10000
Machine Size (N)
• Assumes equal channel width and 1 cycle routing delay
• bandwidth 1 byte/cycle
• dominated by average distance
In the 3-D world
• For N nodes in a 3-cube, bisection area is O(N2/3 )
• For larger dimensions the bisection bandwidth is limited to O(N2/3 ), since
number of wires in physical space are limited. (Dally, IEEE TPDS, 1990)
Equal cost in k-ary d-cubes
• Equal bisection bandwidth?
• Equal number of pins/wires?
• What do we know?
• switch degree: 2*d
• diameter = d*(k-1)
• total links = N*d
• pins per node = 2wd (w is width of link)
• bisection = k**(d-1) = N/k links in each direction
• 2Nw/k wires cross the middle
Latency with Equal Bisection Width
• Number of wires crossing
bisection is constant.
• N-node hypercube has N
bisection links
• 2d torus
• has 2N bisection links
• each link can thus be
N / 2 times wider
• 1 M nodes, d=2, each link
can be 512 times wider than
in a hypercube.
1000
900
Ave Latency T(n=40)
800
700
600
500
400
300
256 nodes
1024 nodes
200
16 k nodes
100
1M nodes
0
0
10
20
Dimension (d)
30
Latency with Equal Pin Count
300
300
256 nodes
250
1024 nodes
Ave Latency T(n= 140 B)
Ave Latency T(n=40B)
250
16 k nodes
200
1M nodes
150
100
50
200
150
100
256 nodes
1024 nodes
50
16 k nodes
1M nodes
0
0
0
5
10
15
20
25
0
Dimension (d)
• Baseline d=2, has w = 32 (128 wires per node)
• fix 2dw pins => w(d) = 64/d
• distance down with d, but channel time up
5
10
15
Dimension (d)
20
25
Embedding d-dim. Dimension into physical space
• Wire density tends to be very high near the bisection and low near the
perimeter.
• 2D mesh has uniform density throughout.
• Higher-order dimensions require longer links. Cycle time for network
increases logarithmic in the wire length.
Topology Summary
• Rich set of topological alternatives
• Design point depends heavily on cost model
• nodes, pins, bisection, area, ...
• Wire length or wire delay metrics favor small dimension
• Long (pipelined) links increase optimal dimension