2. Interprocess Communication
Sistemes Operatius Distribuïts i en Xarxa (SODX)
Facultat d’Informàtica de Barcelona (FIB)
Universitat Politècnica de Catalunya (UPC)
2011/2012 Q2
Contents
• Introduction
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (2) Interprocess communication
Interprocess communication
• The ‘core’ of every distributed system
• Communication is always based on low-level
message passing offered by the underlying
network
– Established network facilities are too primitive
– Systems are too difficult to develop
• Communicating processes must agree on a
set of rules  protocols
• Format messages conform to protocols
SODX (3) Interprocess communication
Contents
• Introduction
– Types of communication
– Communication paradigms
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (4) Interprocess communication
Types of communications
A. Direct communication
– Senders explicitly direct messages/invocations to
the associated receivers
– Senders must know receivers identity and both
must both exist at same time
– e.g. chat, sockets, RPC
B. Indirect communication
– Communication through an intermediary with no
direct coupling between senders and receivers
– Indirect communication allows time and/or space
uncoupling between senders and receivers
SODX (5) Interprocess communication
Types of communications
A. Space uncoupling
– Senders do not need to know who they are
sending to
– e.g. publish-subscribe
B. Time uncoupling
– Sender and receiver do not need to exist at the
same time
– e.g. e-mail
– Time uncoupling also allows distinguishing
persistent from transient communications
SODX (6) Interprocess communication
Types of communications
A. Persistent communications
– The receiver need not be operational at the
communication time
•
•
The message is stored at a communication server as
long as it takes to deliver it at the receiver
e.g. e-mail
B. Transient communications
– A message is discarded by a communication
server as soon as it cannot be delivered at the
next server, or at the receiver
•
e.g. chat, sockets
SODX (7) Interprocess communication
Types of communications
A. Asynchronous communications
– A sender continues with other work
immediately upon sending a message to the
receiver
B. Synchronous communications
– A sender blocks, waiting for a reply from the
receiver before doing any other work
•
This tends to be the default model for RPC/RMI
technologies
SODX (8) Interprocess communication
Types of communications
• Transient synchronous communications
1. Receipt-based
•
•
Client blocks until server ACKs message reception
i.e. synchronization at request submission
2. Delivery-based
•
•
Blocks until server ACKs beginning processing request
i.e. synchronization at request delivery
3. Response-based
•
•
Blocks until server replies with response to request
i.e. synchronization after request processing
SODX (9) Interprocess communication
Types of communications
a) Persistent asynchronous communication
b) Persistent synchronous communication
SODX (10) Interprocess communication
Types of communications
c) Transient asynchronous communication
d) Receipt-based transient synchronous communication
SODX (11) Interprocess communication
Types of communications
e) Delivery-based transient synchronous
communication at message delivery
f) Response-based transient synchronous
communication
SODX (12) Interprocess communication
Types of communications
• Examples
A. Client/server model is generally based on
transient synchronous communication
• Client & server must be up at the time of communication
• Client issues request and blocks until it receives reply
• Server essentially waits only for incoming requests, and
subsequently processes them
– Drawbacks synchronous communication
• Client cannot do any other work while waiting for reply
• Failures have to be dealt with immediately (client is
waiting)
• In many cases the model is not appropriate (mail, news)
SODX (13) Interprocess communication
Types of communications
• Examples
B. Message-oriented middleware aims at high-level
persistent asynchronous communication
• Processes send each other messages, which are queued
• Sender need not wait for immediate reply, but can do
other things
SODX (14) Interprocess communication
Contents
• Introduction
– Types of communication
– Communication paradigms
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (15) Interprocess communication
Remote invocation
• Network transparent extension to traditional programming
– Code in a node can invoke a function in another node as if it were also local
• Point to point interactions
• Middleware handles the marshaling/unmarshaling of parameters
• Protocol is sessionless and server is stateless about client
– Function can change server's state, but client must maintain its state
SODX (16) Interprocess communication
Message passing
• Direct networked communication between components
– e.g. sockets
• Point to point (multipoint also possible)
• It is not middleware mediated
• Lacks transparency: exposes the network characteristics/issues
SODX (17) Interprocess communication
Message queuing
• Send message to a queue, receiver reads from queue
– e.g. mail
• Communication is indirect, by means of message queues
• Point to point
• Middleware stores/forwards messages
SODX (18) Interprocess communication
Group communication
• Messages sent to a group via the group identifier: do not need
to know the recipients (communication is indirect)
• Delivered to all group members
• One-to-many style of communication
• Middleware maintains group membership
Group
Leave
send
Multicast
communication
Fail
Join
Process group
SODX (19) Interprocess communication
Publish-subscribe
• Indirect communication by means of the propagation of events
–
–
Consumers (a.k.a. subscribers) subscribe to types of events
Events are delivered once published by any producer (a.k.a. publisher)
• One-to-many style of communication
• Middleware efficiently routes events generated by producers to
consumers who desire them
SODX (20) Interprocess communication
Shared data space
•
•
•
•
Indirect communication by means of shared persistent storage
Post items to shared space
Consumers pick up at a later time
One-to-many style of communication
SODX (21) Interprocess communication
Shared memory
•
•
•
•
•
Indirect communication by means of shared data structures
Extension to traditional parallel programming model
Standardized in the OpenMP library
Interaction is multipoint (many components share memory)
Middleware maintains the consistency of memory
SODX (22) Interprocess communication
Mobile code
• Code or running processes travel from one node to another and
interact locally with other components
– e.g. web applets
• Middleware transfers code and saves/restores state
SODX (23) Interprocess communication
Stream oriented
• Centered in the processing of large sequences of data streams
(e.g. video, sensor data) by a series of functions, some done in
parallel
• Middleware coordinates flows (splits, joins) and handles
communication issues such as congestion, delays, failures
SODX (24) Interprocess communication
Contents
• Introduction
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (25) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (26) Interprocess communication
Remote Procedure Call (RPC)
• Birrell and Nelson (1984)
• ‘Allow programs to call procedures located on
other machines’
• Programmers do not have to worry about all
the details of network programming (i.e. no
more sockets)
• Conceptually simple, but …
SODX (27) Interprocess communication
Remote Procedure Call (RPC)
• RPC issues
– Two machine architectures may not (or need not)
be identical
– Each machine can have a different address space
– How are parameters (of different, possibly very
complex, types) passed to/from a remote
procedure?
– What happens if one, the other, or both of the
machines crash while the procedure is being
called?
SODX (28) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (29) Interprocess communication
Basic RPC operation
• RPCs are transparent
– From the programmer viewpoint, a ‘remote’
procedure call looks and works identically to a
‘local’ procedure call
• Transparency is achieved by using stubs:
– The CLIENT stub
• Implements the interface on the local machine through
which the remote functionality can be invoked
– The SERVER stub
• Transforms requests coming in over the network into
local procedure calls
SODX (30) Interprocess communication
Basic RPC operation
• The steps of a RPC
client process
server process
Request
Reply
client stub
client
program
communication
module
server stub
communication
dispatcher
module
SODX (31) Interprocess communication
service
procedure
Basic RPC operation
• The steps of a RPC
1. Client procedure calls client stub in normal way
2. Client stub builds message, calls local OS
3. Client's OS sends message to remote OS
4. Remote OS gives message to server stub
5. Server stub unpacks parameters, calls server
6. Server does work, returns result to the stub
7. Server stub packs it in message, calls local OS
8. Server's OS sends message to client's OS
9. Client's OS gives message to client stub
10. Stub unpacks result, returns to client
SODX (32) Interprocess communication
Basic RPC operation
• Client-to-server binding
– Locate server machine, and then locate the server
• Use a ‘directory service’  binder
• e.g. DCE RPC
SODX (33) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (34) Interprocess communication
RPC parameter passing
• Client stub  parameter marshalling
– Take the function parameters
– Pack them into a message
– Send the message to the server stub
 Transform parameters/results into a byte stream
• Issues with parameter passing
– Passing value parameters
– Passing reference parameters
SODX (35) Interprocess communication
RPC parameter passing
• Passing value parameters
SODX (36) Interprocess communication
RPC parameter passing
• Issues with parameter passing
– RPC works really well if all the machines are
homogeneous
– Complications arise when the two machines:
• use different character encodings
– IBM mainframes: EBCDIC, IBM PC: ASCII
• use different byte-ordering
– Intel: little endian, Sun SPARC: big endian
– Solution: Agree on the protocol used
• Representation of data, format and exchange of
messages, etc.
• Use a standard representation
– Example: SUN eXternal Data Representation (XDR)
SODX (37) Interprocess communication
RPC parameter passing
• Passing reference parameters
– Pointers are meaningful only within a specific
address space
– By default, RPC does not offer call by reference
– Some implementations allow passing by reference
arrays (of known length) & structures
• Copy/restore semantics
– Pass a copy and the server stub passes a pointer to
the local copy
• IN/OUT/INOUT markers
– May eliminate one copy operation
SODX (38) Interprocess communication
Interface Definition Language (IDL)
• Definition of interfaces simplifies RPC-based
applications
– An interface consists of a collection of procedures
that can be called by a client, and which are
implemented by a server
– Stub compiler generates stubs automatically from
specs in an Interface Definition Language (IDL)
• e.g. rpcgen tool in SUN RPC or rmic tool in Java RMI
– IDLs are language-neutral
• They do not presuppose the use of any programming
language
SODX (39) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (40) Interprocess communication
Extended RPC models: Asynchronous RPC
a) Traditional RPC
•
BLOCKING occurs – the client waits
b) Asynchronous RPC
•
•
non BLOCKING
useful when the client does not need or expect a result
SODX (41) Interprocess communication
Extended RPC models: One-way RPC
• Client does not even wait for an ACK from the
server
• Limitation: reliability not guaranteed
– Client does not know if procedure will be executed
by the server
SODX (42) Interprocess communication
Extended RPC models: Deferred Synchronous RPC
• Interaction through two asynchronous RPC
• Allows a client to perform other useful work
while waiting for results
SODX (43) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (44) Interprocess communication
RPC semantics and failures
• The RPC mechanism works well as long as
both the client and server function perfectly
• Five classes of RPC failures can be identified
1.
2.
3.
4.
5.
The
The
The
The
The
client is unable to locate the server
client’s request to the server is lost
server crashes after receiving a request
server’s reply to the client is lost
client crashes after sending a request
SODX (45) Interprocess communication
RPC semantics and failures
1. Client cannot locate the server
– Error handling: Raise exceptions for missing
server/stub version mismatch, but …
•
•
This tend to be very language-specific
This tend to be non-transparent !!
2. Lost request messages
– Use timeouts: If no ACK arrives in time, the
request is resent
– Server must be able to deal with retransmissions
•
See ‘Lost reply messages’ section
SODX (46) Interprocess communication
RPC semantics and failures
3. Server crashes
a) Normal sequence of events
b) Crash after execution
•
See ‘Lost reply messages’ section
c) Crash before execution
•
•
Requires different handling than case b)
But client cannot distinguish b) from c)
SODX (47) Interprocess communication
RPC semantics and failures
3. Server crashes
– Three possible philosophies:
•
•
•
At-least-once semantics (e.g. SUN RPC)
– Keep trying until a reply has been received
» Guarantees that RPC has been carried out at
least once, but possibly more
At-most-once semantics (e.g. Java RMI)
– Give up immediately and report failure
» Guarantees that RPC has been carried out at
most once, but possibly not at all
Hope-for-the-best (e.g. Erlang)
– Send request and do not even wait for reply
– We would like exactly once semantics but …
SODX (48) Interprocess communication
RPC semantics and failures
4. Lost reply messages
– Difficult to solve
•
Why was there no reply?
– Is the server dead, slow, or did the reply just go
missing?
– Repeat the request until reply is received
•
•
Only if the operation is idempotent
– It can be repeated any number of times without
any nasty side-effects
» e.g. a read of a static web page
At-least-once semantics achieved
SODX (49) Interprocess communication
RPC semantics and failures
4. Lost reply messages
– Solution for non-idempotent operations
•
•
•
•
e.g. the electronic transfer of funds
Assign a unique sequence number to each request
Check on server against sequence number
– Need stateful server
– How long should server keep track of sequence
numbers?
Additionally, indicate retransmissions with additional
bits
SODX (50) Interprocess communication
RPC semantics and failures
5. Client crashes
– ‘Orphan’ computations: RPC is active but
nobody is waiting for the result
•
•
Waste of resources (CPU cycles, lock files)
Client can confuse reply after recovering
– Four strategies:
a) Extermination
– Log requests to stable storage before sending them
– Upon recovery, explicitly kill orphans
↓ Overhead of logging (for every RPC)
↓ Problems with grand-orphans
 Orphans themselves can do RPCs
SODX (51) Interprocess communication
RPC semantics and failures
5. Client crashes
b) Reincarnation
– Divide time into epochs (sequentially numbered)
– Upon reboot, client broadcasts start-of-epoch
» Upon receipt, all orphans from previous epochs
are killed
» If network is partitioned, orphans may survive,
but their responses will contain an obsolete
epoch number  easily detected
c) Gentle reincarnation
– Upon receipt of start-of-epoch, each server checks
to see if it has any remote computations
» Kill the orphan if the owner cannot be found
SODX (52) Interprocess communication
RPC semantics and failures
5. Client crashes
d) Expiration
– Each RPC is given a time quantum T to complete
» Must explicitly ask for another quantum if it
cannot finish in time
» After reboot, client only needs to wait a time T
to guarantee that all orphans are gone
» How to select a reasonable value for T ?
– However, none of these methods is perfect
•
Problems of killing orphans
– Locks held forever, traces of orphans (e.g. jobs in
queues), etc.
SODX (53) Interprocess communication
Contents
• Introduction
• Remote invocation
– Remote Procedure Call (RPC)
•
•
•
•
Basic RPC operation
RPC parameter passing
Extended RPC models
RPC semantics and failures
– Remote Method Invocation (RMI)
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (54) Interprocess communication
Remote Method Invocation (RMI)
• Idea: expansion of the RPC mechanism to
support object oriented systems
– e.g. Java RMI
• Distributed objects
– Object state is stored on the server and can be
accessed only by the methods of the object
– Every remote object has an interface that specifies
which of its methods can be invoked remotely
– Remote objects can receive remote invocations if
we have its remote object reference
– Tighter integration than RPC into the language
SODX (55) Interprocess communication
Remote Method Invocation (RMI)
• Distributed object
– The proxy can be thought of as the ‘client stub’
– The skeleton can be thought of as the ‘server stub’
SODX (56) Interprocess communication
RMI: Parameter passing
• Less restrictive than RPC
– Supports system-wide object references
– Local objects (O1) by value; Remote ones (O2) by reference
SODX (57) Interprocess communication
Contents
• Introduction
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (60) Interprocess communication
Message-oriented communication
• As a communications mechanism, RPC/RMI is
often inappropriate
– For example: what happens if we cannot assume
that the receiving side is “awake” and waiting to
communicate?
– Also: the default “synchronous, blocking” nature
of RPC/RMI is often too restrictive
• Something else is needed: Messaging
– All communications primitives are defined in terms
of passing messages
SODX (61) Interprocess communication
Transient vs. persistent messaging
• Transient
– No support for persistence of messages sent
– e.g. Sockets
• Persistent
– Sent messages stored in queue, delivered upon
request
– e.g. Message-Oriented Middleware (MOM)
SODX (62) Interprocess communication
Berkeley sockets
• Sockets API
Primitive
Meaning
Socket
Create a new communication endpoint
Bind
Attach a local address to a socket
Listen
Announce willingness to accept connections
Accept
Block caller until a connection request arrives
Connect
Actively attempt to establish a connection
Send
Send some data over the connection
Receive
Receive some data over the connection
Close
Release the connection
SODX (63) Interprocess communication
Berkeley sockets
• Low level of abstraction
– Supports only simple ‘send’ and ‘receive’ primitives
• Too closely coupled to TCP/IP networks
SODX (64) Interprocess communication
Message-queuing systems
• Also known as: ‘Message-queuing systems’ or
‘Message-Oriented Middleware (MOM)’
• Support for:
– Persistent asynchronous communication
– Message transfers allowed to take minutes instead
of seconds/milliseconds
– Intermediate storage for message while
sender/receiver are inactive
– Example application: email
SODX (65) Interprocess communication
Message-queuing systems
• The basic idea:
– Applications communicate by putting/taking
messages into/out from ‘message queues’
• Assume a common messaging protocol: all applications
agree on message format (i.e. structure and data
representation)
– Sender is only guaranteed that message will be
eventually inserted in recipient’s queue
– No guarantees on when or if the message will be
read
– This permits ‘loosely-coupled’ communications
SODX (66) Interprocess communication
Message-queuing systems
• Message-queuing system API
– This is a very simple, yet extremely powerful
abstraction
Primitive
Meaning
Put
Append a message to a specified queue
Get
Block until the specified queue is nonempty, and remove the
first message
Poll
Check a specified queue for messages, and remove the first.
Never block
Notify
Install a handler to be called when a message is put into the
specified queue
SODX (67) Interprocess communication
Message-queuing systems
• Message-Queuing System Architecture
– Sender puts messages into a source queue
– Receiver takes them out from a destination queue
– Obviously, a mechanism to move a message from
a source queue to a destination queue is needed
– This is the role of the Queue Manager
• Can interact directly with sender/receiver applications or,
• Can operate as ‘relays’ forwarding incoming messages to
other queue managers
– These systems may grow into an ‘overlay network’
SODX (68) Interprocess communication
Contents
• Introduction
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (69) Interprocess communication
READING REPORT
• Eugster03. The Many Faces of Publish-
Subscribe
SODX (70) Interprocess communication
Publish-subscribe systems
• Publishers publish structured events to event service
• Subscribers express interest in particular events
• Event service matches published events to subscriptions
SODX (71) Interprocess communication
Publish-subscribe systems
• API
Primitive
Meaning
publish(e)
Disseminate an event
subscribe(f)
Express an interest in a set of events via a filter
(pattern over all possible events)
notify(e)
Deliver event
unsubscribe(f)
Revoke interest in events
advertise(f)
Advertise the nature of future events
unadvertise(f)
Revoke an advertisement
SODX (72) Interprocess communication
Publish-subscribe systems
• The expressiveness of these systems is
determined by the subscription (filter) model:
1. Channel-based
– Publishers publish to named channels
– Subscribers get ALL events from channel
– Very simplistic, no filtering
2. Topic-based (a.k.a. subject-based)
– Subscriptions defined in terms of the topic of
interest (identified by keywords)
– Topics can be hierarchically organized
SODX (73) Interprocess communication
Publish-subscribe systems
3. Content-based
– Subscribe via compositions of constraints over the
values of event attributes
– More expressive than channel- or topic-based
4. Type-based
– Use object-based approaches with object types
• Clean integration with OO programming languages
– Subscriptions defined in terms of types of events
– Matching in terms of types or subtypes of the filter
– From coarse grained (type names) to fine grained
(attributes and methods of object)
SODX (74) Interprocess communication
Contents
• Introduction
• Remote invocation
• Message-oriented communication
• Event-based communication
• Stream-oriented communication
SODX (75) Interprocess communication
Stream-oriented communication
• With RPC, RMI and MOM timing has no effect
on correctness
– Time-independent communications
• Time-dependent communications
– Timing is crucial. If wrong, the resulting ‘output’
from the system will be incorrect
– e.g. audio, video, animation, sensor data
– a.k.a. ‘continuous media’ communications
– Examples:
• audio: PCM: 1/44100 sec intervals on playback
• video: 30 frames per second (30-40 msec per image)
SODX (76) Interprocess communication
Stream-oriented communication
• Transmission modes
– Asynchronous transmission mode
• The data stream is transmitted in order, but there is no
timing constraints placed on the actual delivery
– Discrete data streams (e.g. file transfer)
– Synchronous transmission mode
• The maximum end-to-end delay for each unit is defined
(but data can travel faster)
– Isochronous transmission mode
• Data units transferred ‘on time’. There is maximum and
minimum end-to-end delay (known as ‘bounded jitter’)
– e.g. multimedia systems
SODX (77) Interprocess communication
Stream-oriented communication
• Stream
– An unidirectional continuous data stream that
supports isochronous data transmission
– Generally a single source, and one or more sinks
A. Simple streams
• One single sequence of data (e.g. audio or video)
B. Complex streams
• Several simple streams (substreams) that are ‘related’
by time
• e.g. stereo audio, combination audio/video
• This leads to data synchronization problems
SODX (78) Interprocess communication
Stream-oriented communication
• Enforcing timing (QoS) requirements
1. Use a buffer to reduce end-to-end delay variance
SODX (79) Interprocess communication
Stream-oriented communication
2. Interleaved transmission
•
Gap is distributed but higher start delay
SODX (80) Interprocess communication
Summary
• Powerful and flexible communication facilities
are essential
• Network primitives are too low-level
• Middleware communication mechanisms
support a higher-level of abstraction
• RPC and RMI: synchronous, transient
• MOM: asynchronous, persistent
• Publish-subscribe: decoupled in space
• Streams: for ‘temporally related data’
SODX (81) Interprocess communication