Dr Markus Hagenbuchner
markus@uow.edu.au
CSCI319 SIM
Distributed Systems
Chapter 4 - Communication
CSCI319
Chapter 4
Page: 1
Communication
Lecture notes based on the textbook by Tannenbaum
Study objectives:
1. Understand the role of communication in DS
2. Explain message oriented communication, stream oriented
communication, and multi-casting.
3. Understand the role and limitations of RPC.
4. Explain role and mechanisms of message queuing and
message broker systems.
5. Explain QoS in stream oriented communication.
6. Understand communication in wireless mesh networks
CSCI319
Chapter 4
Page: 2
Content
•
•
•
•
•
•
•
•
Communication protocols
Types of communication
Remote Procedure Calls (RPC)
Streaming
Multicasting
Synchronization
Quality of service
Introduction to Wireless Mesh Networks
CSCI319
Chapter 4
Page: 3
Layered Protocols (1)
Layers, interfaces, and protocols in the OSI model:
CSCI319
Chapter 4
Page: 4
Layered Protocols (2)
Typically, each layer adds a header to the message containing
information relevant to the layer. Example, a typical message as it
appears on the network:
CSCI319
Chapter 4
Page: 5
Interactive slide
What does the acronym OSI stand for?
Open System Interconnect, provides communication standards.
Give examples to what do the following layers in the OSI model
typically refer to:
Application: FTP, HTTP (application side protocol determines what to
do with an incoming message)
Transport: Transport Control Protocol (TCP) adds some flow control
(sequencing)
Network: Network/Internet Protocol (IP) contains information about
unique sender and receiver
Data link: Frames for error detection.
Physical: Refers to wire, line speed, voltage, plugs, etc.
CSCI319
Chapter 4
Page: 6
Interactive slide
What does the acronym OSI stand for?
Open System Interconnect, provides communication standards.
Give examples to what do the following layers in the OSI model
typically refer to:
Application: FTP, HTTP (application side protocol determines what to
do with an incoming message)
Transport: Transport Control Protocol (TCP) adds some flow control
(sequencing)
Network: Network/Internet Protocol (IP) contains information about
unique sender and receiver
Data link: Frames for error detection.
Physical: Refers to wire, line speed, voltage, plugs, etc.
CSCI319
Chapter 4
Page: 7
Middleware Protocols
• The standard OSI model does not provision a layer for
the middleware protocol.
• The presentation layer and the session layer are hardly
ever used in practice (their corresponding headers are of
zero-size).
• Distributed system leave the presentation layer empty,
and use of the session layer for the injection of
middleware layer protocols.
• Hence, the OSI model for distributed systems is defined
more specifically as follows:
CSCI319
Chapter 4
Page: 8
Middleware Protocols
An adapted reference model for Distributed Systems.
CSCI319
Chapter 4
Page: 9
Interactive slide
What is the role of the middleware layer in the Open
System Interconnection (OSI) model?
• Offers high level communication services and is placed
as an independent protocol layer between transport and
application layer.
• Helps to achieve transparency (from the application point
of view).
• Allows the realization of RPC…
CSCI319
Chapter 4
Page: 10
Interactive slide
What is the role of the middleware layer in the Open
System Interconnection (OSI) model?
• Offers high level communication services and is placed
as an independent protocol layer between transport and
application layer.
• Helps to achieve transparency (from the application point
of view).
• Replaces the session and presentation layer which are
rarely ever used (if needed, the middleware layer can
simulate these layers in a transparent fashion).
• Allows the realization of RPC…
CSCI319
Chapter 4
Page: 11
RPC
•
Many older distributed systems have been
based on explicit message exchange between
processes (i.e. using send and receive).
– Transparency is not achieved
• An idea was introduced to allow calling of
procedures located on other machines.
• The method is commonly known as
Remote Procedure Calls (RPC)
CSCI319
Chapter 4
Page: 12 of 47
RPC
•
•
Local procedure call: Parameters are passed via a stack.
Remote procedure call: Parameters are passed via a message over a
network.
•
In both cases: the receiver needs to know the type, format, and
purpose of the parameters.
Main difference: With RPC, the receiver may be of a differing
hardware architecture. Type conversion may be necessary.
•
•
Example: Assume that a function add() is defined to accept two integer
parameters such as in add(int i, int j). Then the differences between
calling a local procedure and RPC can be shown as follows:
CSCI319
Chapter 4
Page: 13 of 47
Conventional Procedure Call
Parameter passing when calling a local procedure add(int, int).
Left, the stack before the call to add(), right the stack while the
called procedure is active.
Main program’s local
variables
Main program’s local
variables
int i
int j
Return address
Local variables of add
CSCI319
Chapter 4
Page: 14 of 47
Remote Procedure Calls (1)
In general, a remote procedure call occurs in the following
steps:
1. The client procedure calls the client stub in the normal
way.
2. The client stub builds a message and calls the local
operating system.
3. The client’s OS sends the message to the remote OS.
4. The remote OS gives the message to the server stub.
5. The server stub unpacks the parameters and calls the
server.
Continued …
CSCI319
Chapter 4
Page: 15 of 47
Remote Procedure Calls (2)
A remote procedure call occurs in the following
steps (continued):
6. The server does the work and returns the result to the
stub.
7. The server stub packs it in a message and calls its local
OS.
8. The server’s OS sends the message to the client’s OS.
9. The client’s OS gives the message to the client stub.
10. The stub unpacks the result and returns to the client.
CSCI319
Chapter 4
Page: 16 of 47
Passing Value Parameters (1)
A visualization of the steps involved in a doing a remote
computation through RPC:
But there is a problem in heterogeneous systems:
CSCI319
Chapter 4
Page: 17 of 47
Passing Value Parameters (2)
But there is a problem:
 The byte order can differ between architectures. This is called
endianess.
Example: A integer consists of 4 bytes. Say, we store the value 5 in
an integer, which of the bytes in the integer will carry the value?
 In Big endian (i.e. Intel): the (last) forth byte.
 Little endian (i.e. Sun sparc): the first byte.
 Thus, an RPC issued by an intel based computer would send
the byte sequence 0005. But a sun workstation would interpret
this value as 5*224.
Solution:
 Reverse byte order if endianess differ between sender/receiver.
This worked for the integer but not for the string. Hence, we need
to understand what needs to be reverted and what not (since
we cannot ask the RPC system to guess).
CSCI319
Chapter 4
Page: 18 of 47
Communication
RPC is one example of process communication between two
machines.
In general, Process Communication can be divided into two
types:
(1) Message oriented
(2) Stream oriented.
We will have a closer look at these two.
CSCI319
Chapter 4
Page: 19 of 47
Message Oriented Communication
We will look at two message oriented
communication and their protocols:
1. The message passing interface (MPI) and the
message passing protocol.
2. Message Queuing (MQ) and the message
queuing protocol.
CSCI319
Chapter 4
Page: 20 of 47
The Message-Passing Interface (1)
MPI is a well established protocol commonly used for communication in
homogeneous systems (i.e. clusters). Some of the primitives of MPI:
MPI avoids low level primitives which are irrelevant to the
task of message passing -> More intuitive, more transparent
CSCI319
Chapter 4
Page: 21 of 47
Message Oriented Communication
 However, MPI requires that both, sender and
receiver are available at the time of
communication.


This is OK in cluster environments.
But it is unsuitable in DS which are distributed over a larger area.
 Message Queuing allows communication in
loosely coupled systems.

Through a buffering mechanism within the middleware layer.
CSCI319
Chapter 4
Page: 22 of 47
Message-Queuing Model (1)
With message queuing, there are four combinations for
loosely-coupled communications using queues.
CSCI319
Chapter 4
Page: 23 of 47
General Architecture of a MessageQueuing System (1)
 There can be multiple queues in a MQ system.
 Messages may be passed via a set of routers.
 Thus, the path of a message sent through a queue
must be defined: Routing tables…
CSCI319
Chapter 4
Page: 24 of 47
Message Transfer in MQ systems
The general organization of a message queuing network
using routing tables and aliases.
CSCI319
Chapter 4
Page: 25
IBM’s WebSphere Message-Queuing System
General organization of IBM’s message-queuing system:
CSCI319
Chapter 4
Page: 26
Channels
IBMs message-queuing systems uses channel agents
(MCA). Channels avoid the need of re-establishing a
connection when sending several messages through a
queue.
The collective of all channels define the topology of a
system. It becomes necessary to define routes to ensure
that a message can find a path to any destination. This
too is achieved through routing tables.
CSCI319
Chapter 4
Page: 27
Message Brokers
Large scale DSs are often heterogenous. Messages
send be one node may not be “understood” by the
receiving node. This can occur, for example, if sender
and receiver use differing software versions. This issue
can be addressed i.e. through the use of message
brokers as in WebSphere.
Message brokers are commonly found in message
queuing systems of large enterprises.
CSCI319
Chapter 4
Page: 28
Message Brokers
The general organization of a message broker in a
message-queuing system.
CSCI319
Chapter 4
Page: 29
Message Transfer (2)
We have seen that MQ systems can be powerful. Yet, the
primitives required can be kept quite simple. In practice,
the primitives available in the message-queuing interface
are:
CSCI319
Chapter 4
Page: 30
Next: Stream Oriented Communication
in distributed systems
CSCI319
Chapter 4
Page: 31
Data Stream
A general architecture for streaming stored multimedia
data over a network when using Quality of Service
(QoS) module.
CSCI319
Chapter 4
Page: 32
Streams and Quality of Service
Properties that define Quality of Service (QoS):
• The required bit rate at which data should be
transported.
• The maximum delay until a session has been
set up
• The maximum end-to-end delay .
• The maximum delay variance, or jitter.
• The maximum round-trip delay.
CSCI319
Chapter 4
Page: 33
Enforcing QoS (1)
A strategy to reduce jitter is by using a buffer as is
illustrated in the following:
The loss of a packet can be a problem since we often can
not wait for a packet to be re-sent. So what can be done
to reduce the effect of packet loss?
CSCI319
Chapter 4
Page: 34
Enforcing QoS (2)
The effect of packet loss in (a) non interleaved transmission
and (b) interleaved transmission.
CSCI319
Chapter 4
Page: 35
Enforcing QoS (3)
Note that on the previous slide:
1.Send refers to data sent by a multimedia server
2.Delivered refers to data as is delivered to the client-side
application layer by the client-side middleware. Thus,
buffering, and the recreating of the frame sequence is
handled by the middleware layer.
CSCI319
Chapter 4
Page: 36
Multicast Communication
With Multicast communication:
• One sender, multiple receivers
• Network link between sender/receiver may run
at different speeds, produce different cost, etc.
• Strategies for efficient message propagation?
• (Network) overlay construction (a structure that defines how
a message travels across the network).
• Dissemination models
CSCI319
Chapter 4
Page: 37
Overlay Construction
Aim of overlay construction:
 Reduce transport cost
 Reduce time requirement.
The relation between links in an overlay and actual
network-level routes will be demonstrated in an example:
A system consisting of
four hosts, four routers, a
cost (i.e. time, transport cost)
associated with each
network link, and network
level topology.
CSCI319
Chapter 4
Page: 38
Example 1 overlay construction
Assume sender A multicasts a message by sending the message to
each received one at a time. What would the total cost be?
Cost: Cost A->B + Cost A->C + Cost A->D
9
+
53
+ 57
= 119
CSCI319
Chapter 4
Page: 39
Example 2 overlay construction
Sender A attaches ID of all receivers. Routers chose the connection
of lowest cost to deliver the message to the first receiver. The
receiver removes its own ID from message, then becomes the new
sender. Repeat until no more IDs in the message. Cost?
Cost: Cost A->B + Cost B->C + Cost C->D
9 +
60
+
6
= 75
CSCI319
Chapter 4
Page: 40
Overlay construction
 Thus, the overlay network defines the efficiency of a
multi-cast system. A well designed overlay can
reduce costs significantly.
 But what is the optimal cost?
 In the example, the optimal cost is: 9 + 41 + 6 = 56
 In dynamic systems it is not generally possible to
achieve optimal cost. In static systems it is possible
to define static routing tables which can achieve
optimal cost.
CSCI319
Chapter 4
Page: 41
Information Dissemination Models (1)
An alternative to spreading information (i.e. as found in P2P)
is to spread information like an epidemic:
1. Anti-entropy propagation model
– Node P picks another node Q at random
– Subsequently exchanges updates with Q
2. Approaches to exchanging updates
– P only pushes its own updates to Q
– P only pulls in new updates from Q
– P and Q send updates to each other
In both situations, P forgets Q with probability (i.e. 1/k), then
picks another node at random.
The approach in (1), and (2) does not guarantee that all
nodes receive the message.
CSCI319
Chapter 4
Page: 42
Wireless Mesh Networks (1)
•
•
A wireless mesh network (WMN) is a communications
network made up of radio nodes organized in a mesh
topology.
Wireless mesh networks often consist of Nodes have
two functions:
– Generate/terminate traffic
– Route traffic for other nodes
Therefore, MMNs are:
•
Multihop Wireless network.
•
Support for adhoc networking and capability of self
forming, self healing and self organization.
CSCI319
Chapter 4
Page: 43
Wireless Mesh Networks (2)
SS = Subscriber Station; BS = BaseStation
CSCI319
Chapter 4
Page: 44
Wireless Mesh Networks (3)
•
•
Security enforcement and topology management is
much easier to achieve in Point-to-multipoint
communication.
Whereas robustness and range is improved vastly in
WMN.
This makes mesh communication particularly attractive for
distributed pervasive systems.
WMN is standardized (IEEE802.16a) for WIFI networks.
There is no such standard yet for 3G type of networks.
CSCI319
Chapter 4
Page: 45
Wireless Mesh Networks (4)
Note: Wireless mesh networks differ from Ad-hoc
networks and from Wireless sensor networks.
You will learn more about these differences during
the tutorial.
CSCI319
Chapter 4
Page: 46
Summary
• Standard protocols are an essential requirement.
• Types of communication (i.e. RPC)
Synchronous vs asynchronous RPC
• Message distribution models. These are….?
• Streaming = information dissemination
• Quality of Service
• Introduced Wireless Mesh Networks
CSCI319
Chapter 4
Page: 47