Group Communications
Group communication: one source process sending a message to
a group of processes: Destination is a group rather than a single
process.
— Broadcast – destination is everybody.
— Multicast – destination is a designated group.
— Unicast – destination is a single process.
Useful: for many applications, the recipient of a message is a
group of related process. Many services in DS are
implemented by a group of processes, possibly distributed in
multiple machines.
Examples of use:
(a) Fault tolerance based on replicated servers:
The service is implemented by a group of (replicated) server
processes. A client process multicast(Send-to-group) request
message to the server group. Each group member performs
identical operation on a request.
(b) Locating objects in distributed service:
For example, a distributed file service is implemented by a
group of servers. The files are not replicated. A client
looking for a particular file multicast the request message to
the server group. The member holding the wanted file
responds.
(c) Updating replicated data:
Data may be replicated to improve performance (response
time) and reliability. The data is replicated on multiple
servers which forms a server group. Update requests are
multicasts to the group.
(d) Multiple notification:
For example, a group of processes need be notified of certain
event. When the event occurs, a message can be multicasted
to the group. A specific example is in the multicasting of
flight information to a group of display processes in an
airport terminal system.
A Group is a collection of processes: G = {P1, P2, …, Pn}
Group management functions:
Forming a group: Create-group (out gid: group-id)
returns a group identifier
Joining a Group: Join-group (in gid: group-id)
makes the caller process a member of the
group gid.
Leaving a group: Leave-group (in gid: group-id)
remove the caller process from the group.
Delete a group: Delete-group (in gid: group-id)
Only authorized process (such as the
creator) can delete a group.
- Dynamic group vs static group
Closed Group: Processes outside the group cannot multicast to
the group, although an outsider process can send to individual
member of the group.
Implementations (of group):
Centralised: Use a group management server.
• All group membership function requests are
handled by the server.
• Server keeps database of all active groups
(membership) in the system.
• Pro: implementation ease.
Con: server may be reliability and performance
bottleneck.
Distributed: Each group member keeps track of
membership. membership function requests
are multicast to all members.
• Pro: reliable
• Cons: complexity and overheads
Group Communication Functions
Send to group:
SendToGroup (in gid: group-id, msg: message)
Receive from group:
ReceiveFromGroup (in gid; group-id, out msg: message)
Receive any message sent to group gid.
Group Send and Receive have similar issues as IPC with respect
to:
Blocking and non-blocking
Implementations (of communication functions):
Unicast: A message is sent to each of the members of the
group.
- Number of message = group size
- Inefficient
If underlying communication system supports broadcast to
machines (e.g. Ethernet). A broadcast message is sent to all
machines. The group communication facility within the
kernel on respective machine takes in the message if there
are members (of the group) on that machine.
If underlying communication system supports multicast to
machines. Then the message can be multicasted to only those
machines on which there are members.
Group Atomicity
Group atomicity is concerned with whether the group is always
a whole or can be partitioned (with respect to group operations).
Atomic multicast:
When a message is sent to the group, all members of the group
receive the message. Otherwise, no member receives the
message. i.e., there is never the case that some members receive
the message while others do not. It is all (member) or none.
Example in which atomicity is needed:
- Replicated data update. If not, the replicated data may get out
of steps with each other.
Example in which atomicity is not needed:
- Locating objects in distributed service. It is sufficient that the
server holding the object receives the message. If the message
to this server is lost, the client can try again (say on time-out).
- Multiple notification in a flight information display system.
Implementation (of group atomicity)
No machine failure:
- Message (to individual member) can be lost.
- Can use a positive acknowledgment protocol (i.e., timeout retransmission).
- If network failure (which may make some members
unreachable) is possible, a protocol similar to the
following case can be used.
Machine failure possible:
- For example, the sending machine (or the machine on
which the group communication server runs) crashes.
Some member will have received the message while
others have not.
- More complicated protocol. e.g., a protocol similar to 2phase commitment protocol.
- A simple (inefficient) protocol: Sender sends to each
member, using positive acknowledgment protocol. When
a receiver receives a message. It determines if the
message has been received before. If not it sends the
message to all other members of the group.
Message Ordering
The Order of message received by a group member can be
important
- FIFO order
If a source process sends two messages to a group in the
order of M1->M2 (i.e., M1 precedes M2), then the
messages are received in that order.
FIFO is easy to implement.
Not adequate for some application. e.g., two client processes
P1 and P2 are multicasting update requests to a server group
whose member are holding replicated data. Suppose P1
sends M1, then M2 ; and at about the same time P2 sends N1
then N2. Further suppose the group consists of 2 server
processes S1 and S2. The message can arrive at the
following order and still satisfy FIFO order:
at S1: M1->N1->M2->N2
at S2: N1->M1->M2->N2
The above order may lead to inconsistent result.
- Consistent order
The relative order of messages received at each member
process is the same. That is, the order is consistent across
the group.
- Causal order
Assume S1 first multicasts m, upon reception of m, S3
multicasts m’, then m->m’. m and m’ are said to be
causally ordered.
Causal order could be violated. e.g., due to network delay.
A group communication mechanism is said t observe
casual order if it preserves the order.
Total order
When a group communication mechanism satisfies both causal
order and consistent order, we say the mechanism satisfies total
order. That is,
The message order at each member process preserves causal
order and that the message ordering is the same across the
entire group.
Implementation:
Use a logical clock global to the distributed system to
timestamp messages (e.g., Lamport’s timestamps – to be
discussed in a later chapter). The message are ordered by the
entire group.
Use a centralised server. A multicast message is first sent to the
server. The server assigns sequence number to the messages
(the server is also called a sequencer) and send them to all the
members. The messages are delivered (to the application
program) in order of the sequence number.
Use a token passing (logical) ring. Only the token holder can
multicast message.
Note that atomicity and ordering are two different aspects of
group communications.
One may have total ordering but not atomicity and vise versa.
Atomic totally ordered group communication is usually
expensive in terms of implementation and overheads.
- System preserving causal order is also said to be a virtually
synchronous system.
An Implementation (of causal order)
— Each process keeps a vector clock (V1, V2, …, Vn).
— Each message carries a vector (logical) timestamp:
(V1, V2, …, Vn). Where Vk increments each time process
Pk multicast a message.
— When message with timestamp (V1, V2, …, Vn) from Pj is
received. The message is accepted if
Vj = Lj + 1, and Vi <= Li
Where (L1, L2, …, Ln) is the local vector clock. And the
local clock is set to (V1, V2, …, Vn).
Otherwise, the message is held back until the condition is
satisfied.
Design Issues to be Considered
1. Design your client-user interface
1. Group Information Inquiring
2. Joint Group
3.
4.
…….
2. Use TCP or UDP ( I choose TCP)
3. Functional Specification
3.1 Membership related service
a) create a group b) Joint group c) Leave group d)
remove group …….
3.2. Multicast related services
a) send message to a group b) message ordering
(consistant order)
3.3 Miscellanceous Services
a) group information inquiring
4. Architectural Relationship between Clients and Server(s)
server
client1
client 2
…
clientn
Design Issues to be Considered
4. Architectural Relationships among processes and threads
each client owns a process and the client process may
contain several threads
a thread sending group related command
a thread sending message related command.
the server owns a process that contains several threads
each primitive corresponds to a tread, e.g.,
join group
create a group
……
send a message to a group
Note that, you can use other ways to organize threads.
Whatever you do, give reasons in the report.
5. Client-to-Server Protocol
If many functions are on server side. How does the clients
call them?
1#1gname --create a group with the name gname
1#gid
-- join the group with group in “gid”
6#gid#meg -- send message “msg” to the group “gid”
6. Atomicity and fault tolerance, etc will be considered in
semester B. However, you have to leave some
“space” for that.