BSS 797: Principles of Parallel Computing
Lecture 3
Software Issues
Parallel programming languages
Observation 1: A regular sequential programming
language (C or Fortran or C++ etc) plus four
communication statements (send/recv/myid/numnodes)
are necessary and sufficient to form a parallel
computing language.


send: One processor sends a message to the
network. Note this processor does not have to know
to which processor it is sending this message, but it
does give ``name'' for the message.
recv: one processor receives a message from the
network. Note this processor does not have to know
which processor sends this message, but it retrieves
the message by name.


myid: a integer between 0 and P-1 identifying a
processor. myid is always unique within one
partition.
numnodes: a integer showing the total number of
nodes in the system.
Three Ways to Communicate:
Synchronous
the sender will not proceed to the next task until
the receiver retrieves the message from the
network (hand deliver: slow!)
Asynchronous
the sender will proceed to the next task whether the
receiver retrieves the message from the network or
not (mailing a letter: will not tie up sender!)
No protection for the message in the buffer.
Examples for Asynchronous message passing
Sender
... ...
do_sth_useful()
Receiver
msg_rid = irecv() /* no need of MSG */
... ...
... ...
... ...
msg_sid = isend()
... ...
... ...
do_sth_witout_needing_msg
... ...
... ...
msgwait(msg_rid); /*No Return Until
... ...
Done*/
do_something_without_needing_msg doing_sth_with_it.
CHOICE II (msg_done())
if(msg_done(msg_rid))
do_sth_with_it
else do_sth_else
msg_ignore(msg_rid) /*oops, wrong
CHOICE III (msg_ignore())
number*/
CHOICE IV (msgmerge())
mid = msgmerge(mid1, mid2)
Interrupt
the receiver interrupt the sender's current activity
for pulling messages from the sender (ordering a
package: interrupt sender!)
Nine Communication Patterns
1. 1 to 1
2. 1 to partial
3. 1 to all
4. partial to 1
5. partial to partial
6. partial to all
7. all to 1
8. all to partial
9. all to all
Parallel Programming Tools
1. parallel computing languages (parallel
FORTRAN etc)
1. message-passing assistant
2. portability helper: PVM, MPI... ...
2. debuggers
3. performance analyzers
4. queuing system (same as in sequential)