Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Multiprocessor Networks
Multiprocessor networks can be modeled as networks of processing elements, namely PE’s.
Each processing element has a CPU and its own separate memory. Despite traditional network
models, multiprocessor networks have a well structured fashion of communication, which makes
it more suitable for performing distributed tasks.
In our model, PE’s communicate using a message passing strategy. Sender, the PE that
initiates the communication, enters "send mode" in which it puts the message on the line. The
message here can be considered as a physical entity that can be put and taken from the line.
The receiver PE in turn, becomes blocked until the requested message arrives. As PEs may get
blocked while performing this way of communication waiting to receive data from other PE’s, we
should watch out for deadlocks. Deadlocks are situations in which each PE in a group of PEs is
waiting to receive a message from another PE in a way that their dependencies form a cycle
(Figure 1). Handling deadlocks, including deadlock avoidance, deadlock detection and deadlock
recovery has its own issues and algorithms we are not going to cover in this course.
Figure 1. A deadlock situation; each PE is waiting for another PE to
continue its work and the wait relations form a cyclic structure.
Multiprocessor network models can be put in two broad categories: interconnection networks
and direct networks. The categorization here is for the sake of organizing the material. Of
course this is not a globally standard categorization, and there may be other possible ways to
categorize these models. We now further describe each of these categories.
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Interconnection Networks
In an interconnection network, a set of PE’s are connected to each other via some links and
switches in a way that each PE can logically communicate with any other PE directly. Each PE
has its own unique ID. From logical point of view, the interconnection network which consists of
links and switches can be considered as a black-box through which the messages can be sent
directly from sender to receiver.
Figure 2. Each PE can communicate any other PE via an the
interconnection network of links and switches.
This black-box may have several different configurations, which differ in the degree of efficiency
they offer. Here, we’ll study bus networks and crossbar networks.
Bus Network
A bus network is an arrangement of PE’s in which each node is connected to a main link called
the bus. Figure 3 illustrates a bus network with five nodes.
Figure 3. A bus network.
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Ethernet is logically a bus network. Message passing is performed as follows: The sender puts
the message on the bus, which has the write property: only one message can exist on the bus
at any instant, or a collision occurs. To avoid this situation and conform to the write property,
arbitration strategies should be adopted. Receiving the message would be easy according to
the read property: every one in the network can see the message put on the bus. In order to
restrict receiving to the intended receiver, all the others simply discard the message after seeing
their IDs don't match the receiver ID.
Performance Parameters
Distance: Distance between PE’s u and v is defined as the number of hops between u and v.
Each hop is either a PE or a switching element. In the bus network, the distance between each
pair of PE’s is one, since there are no switches and no intermediate is needed for message
passing.

d (u, v)  # of hops between u and v = 1
(for any u and any v ≠ u)
Diameter: Diameter of the network is the maximum distance between pairs of PEs in the
network. As all the distances are 1 in a bus network, its diameter would also be 1.

D  max d (u, v)  1
u ,v
Throughput: Throughput is a measure of efficiency of the networks. Roughly speaking,
throughput determines the number of messages that can be sent across the network at a time
unit. It’s obvious that the network traffic affects throughput, but it must be noted that in a bus
network, the amount of traffic and throughput are not in a direct relationship. Although
throughput increases by an increase in traffic for low traffics, it might worsen for higher traffics
because of higher rate of collisions. Saturating traffic is the infinite load conditions, in which all
nodes have traffic to send at all times. Saturating traffic does not necessarily give the best
throughput. The ideal traffic is the configuration of traffic which leads to the maximum
throughput. Ideal traffic is achieved through choosing the best choice for input/output
(sender/receiver) pairs and traffic pattern (Figure 4).
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Figure 4. An illustration of how throughput and traffic interrelate in a bus network.
Throughput is affected by topology of the previously mentioned black box. Here is an example
of computing throughput of a bus network.
Example 1. Suppose that there are 64 PE’s connected via a bus network. The bus width is 8
bytes, meaning that data should be divided into 8-byte units (called cycles) to be put on the bus.
Size of messages is 256 bytes, and time to send is tx, which means it takes tx time units for a PE
to put an 8-byte unit of data on the bus. After the data is sent, a delay of ta time units is needed
to ensure that all PE’s can see it. This propagation delay is a property of the physical media
forming the bus and depends on the speed of data in it, which is usually denoted by a fraction of
light’s speed.
To better understand the distinction between tx and ta, consider interconnection networks
analogous to road networks. Roughly speaking, the time needed for a group of cars to reach
from one end of a road to the other, depends not only on the width of the road (which
determines tx), but also on how fast a single car may pass through the road considering its
length and surface properties (determines ta).
Thus, we have:
N = 64;
M = 8 bytes;
w = 8;
(Number of PEs)
(Message Size)
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
tx = 1 cycle;
And therefore, as throughput is number of messages per unit time per PE,
T
M
1
256
 
N 32  t a
M 
 t x  t a
w
Assuming ta has negligible value,
T
256 1

 0.12 bytes/cycle.
32   64
Crossbar Network
Crossbar network is another network topology in which N PE’s are arranged along height and
width of a grid of switching elements as in Figure 5. Each switching element can have different
configurations which determine which input should be routed to which output (Figure 5).
Figure 5. (Left) A crossbar network; (Right) Each switch can have several configurations to route messages
depending on which input links are connected to which output links.
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
In a crossbar network, messages can share switches but not edges.
Performance Parameters
Distance: Distance between nodes in a crossbar network is the Manhattan distance between
them.
Diameter: Diameter of the network is therefore the Manhattan distance between the PEs
located at the two ends of the grid’s diagonal, which are trivially the farthest apart. Thus,
N

D  2    1
2

Considering N/2 of PE’s as senders and the others as receivers, this network topology provides
simultaneous access among all the senders and receivers of the network providing that all the
requested receivers are different. Therefore, as the following example indicates, crossbar
topology provides significantly higher throughput compared to the bus topology.
Example 2. Consider 64 PEs on a 32x32 grid of switching elements, and M, w, and tx as in
Example 1.
N
 M
1
2
T  

N
M 
 t x  t a
w
32  256
1


64
 256 

t x  t a
 8 
4

 4.
tx
Although Ethernet protocol gives significant utilization compared to ALOHA because of the way
it handles collisions, it still suffers from low throughout. This does not mean that Ethernet
protocol is not good. In fact, the inherent limitation of bus topology imposes this problem to
every protocol designed for such kind of networks. As we can conclude from this example, other
topologies may allow much higher throughput. As a result, comparing performance parameters
of protocols designed for different network topologies doesn’t make sense.
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Direct Networks
In a Direct Network, every intermediate node is a PE. No black-box interconnection structure is
considered, and the network is modeled as a graph with PE’s as nodes and links as edges. The
Internet is an example of direct networks. Like interconnection networks, direct networks may
have different structures.
Performance Parameters
Like interconnection networks, direct networks can be evaluated using some parameters. We
discuss three of these parameters here: distance, diameter and bisection width.
Distance
Distance between two PE’s in the network is defined by the distance between their
corresponding nodes in the network’s graph. This would be simply the number of edges on the
shortest path connecting the two nodes. To achieve a good measure to evaluate the overall
performance of the network, the average distance between nodes in a network is taken into
account.
Diameter
Having the notion of distance, definition of the network’s diameter is the same as the case for
interconnection networks. It is equal to the distance between farthest nodes in the network.
Bisection Width
To achieve the desired communication among PE’s, the network graph must be connected.
When a link in the network becomes faulty, it fails to transfer data and the corresponding edge
in the graph is removed. So, a good measure of the network’s fault tolerance is to count the
number of edges that must be removed from the network to make the graph disconnected.
Bisection width is defined based on this intuition, and it is equal to the number of edges we have
to remove from the graph to obtain two connected components of fairly equal size.
Linear Network
In a linear network, PE’s are arranged along a line, one after another (Figure 6).
Figure 6. Linear Network.
In a linear network with N PE’s, diameter is N-1, average distance is
is 1.
N 1
, and bisection width
2
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
Ring Network
Connecting the nodes at two ends of a linear network, gives a ring network (Figure 7). In a ring
N 
network, diameter is   and the bisection width is 2. Comparing ring network with linear
2
network, we realize that adding just one edge can significantly improve performance of the
network, thus highlighting the importance of good network design.
Figure 7. A ring network.
k-Dimensional Meshes (with or without wraparound)
Generalizing linear and ring arrangement to higher dimensions, gives k-D meshes without
wraparound and k-D meshes with wraparound respectively (Figure 8).
Figure 8. A 3D mesh.
Lecture Notes #1
Computer Networks – Session 2 (11 Sep. 2009) and Session 3 (14 Sep. 2009)
Instructor: Funda Ergun
In a 2D without wraparound, each node is represented with a 2-tuple, indicating its relative
position in the mesh. Node (i1, i2) is connected to all valid tuples among (i1-1, i2), (i1+1, i2), (i1, i21) and (i1, i2+1). Connecting the two ends of each vertical and horizontal line in a 2D mesh, we
obtain 2D mesh with wraparound which is equivalent to a torus (Figure 9 and Figure 10).
Figure 9. 2D mesh (a) without and (b) with wraparound.
Figure 10. A torus is equivalent to a 2D mesh with wraparound.