NK ISC 101: Practice of
Parallel Computing
LECTURE 2
Software
Introduction
Parallel Programming Language
Observation 1: A regular sequential programming language
(C or Fortran or C++ etc) plus four communication statements
(send, receive, myid, numnodes) are necessary and sufficient to
form a parallel computing language.
Send: One processor sends a message to the network. Note this
processor does not have to know to which processor it is
sending this message, but it does give “name” for the message.
Receive: One processor receives a message from the network.
Note this processor does not have to know which
processor sends this message, but it retrieves the
message by name.
myid: Integer between 0 and P-1 identifying a processor. myid
is always unique within one partition.
numnodes: Integer showing the total number of nodes in the
system.
Basic Message Passing Protocol (single-sided with “virtual buffers”.
Sender: The circle on the left represents the "Sender"
whose responsibility is to send a message to the
"Network Buffer” without knowing who the receiver is.
Receiver:
The “Receiver” on the right has to issue a
message to the “buffer” to retrieve a message that is
labeled for it.
Note:
1. This is the so-called single-sided message passing, which
is popular in most distributed-memory supercomputer.
2. The Network Buffer as labeled, in fact, does not exist as
an independent entity and is only a temporary storage. It
created either in the sender’s RAM or in the receiver’s
RAM and depended on the readiness of the message
routing information. For example, if a message’s
destination is known but the exact location is not known at
the destination, the message will be copied to the
receiver’s RAM for easier transmission.
Three Ways to Communicate
Synchronous: The sender will not proceed to the next task
until the receiver retrieves the message from the network (hand
deliver: slow!)
Synchronous Message Passing: The circle on the left sends
a message-1 to the imaginary Network “Buffer”, which then
requests the destination to stop its current activities and
ready to receive a message from the sender. In synchronous
mode, the Receiver will immediately halt its current
processing stream by issuing an acknowledgement to the
Sender saying “OK” to send the message. After receiving
this message, the Sender will immediately dump the
original intended message to the Receiver at the exact
location.
Asynchronous:The sender will proceed to the next task whether
the receiver retrieves the message from the network or not
(mailing a letter: will not tie up sender!) No protection for the
message in the buffer.
Asynchronous message passing: The Sender issues a
message with the appropriate addressing header (envelope
information) and regardless of the arrival of the message at
the Receiver end or not, the Sender continues its execution
without waiting for any confirmation from the Receiver.
The Receiver, on the other hand, will also continue its own
execution stream until the “receive” statement is met.
Note: The advantage with the asynchronous message passing is
its speed. There is no need for either party to wait. The risk lies
in the misuse of the correct message.
One example for Asynchronous message passing
Asynchronous Message passing example: The Sender issues
a message and then continues on its execution regardless of
the Receiver’s response in receiving the message. While the
Receiver can have several options with regard to the
message issued already by the Sender; this message now
stays somewhere called Buffer:
1. The first option for the Receiver is to wait until the
message has arrived and then make use of it.
2. The second is to check if the message has indeed
arrived. If YES, do something with it; otherwise,
stay with its own-thing.
3. The third option is to ignore the message; telling
the buffer that this message was not for me.
4. The fourth option is to merge this message to
another existing message in the Buffer; etc.
Interrupt: The receiver interrupts the sender's current activity
for pulling messages from the sender (ordering a package:
interrupt sender!)
Interrupt message passing: The “Sender”first issues a short
message to interrupt the“Receiver”current execution stream
so that the “Receiver”is ready to receive a long message
from the Sender. After appropriate delay (for the interrupt
to return the operation pointer to the messaging process),
the Sender pushes through the message to the right location
for the Receiver without any delay.
Nine Communication Patterns
i.
ii.
iii.
iv.
v.
vi.
vii.
viii.
ix.
1 to 1
1 to Partial
1 to All
Partial to 1
Partial to Partial
Partial to All
All to 1
All to Partial
All to All
Nine Communication Patterns:
(A) A single processor can send one message (same) to one
processor, to a sub-group of M processors, or to the
entire system.
(B) A subgroup of M processors or all processors can send
M different messages or all different messages to one
processor.
(C) A sub-group of K processors (how the messages are
partitioned is a separate issue) can send messages to the
entire system. Finally, the entire system of P processors
can send P different messages to one processor, a subgroup of N processors, or to the entire system.
Note:
1. In the obvious case of one message to 1, K, or P
processors (same case in reverse), messages are
partitioned naturally.
2. But in the case of M messages sent to K processors, the
matter is different and we will discuss that later.
Parallel Programming Tools
(1) Parallel computing languages (parallel FORTRAN etc)
(1.1) Message-passing assistant,
(1.2) Portability helper: PVM, MPI......
(2) Debuggers
(3) Performance analyzer
(4) Queuing system (same as in sequential)