Distributed Systems
CS 15-440
Consistency and Replication – Part IV
Lecture 13, Oct 23, 2013
Mohammad Hammoud
Today…
 Last Session:
 Consistency and Replication- Past III
 Client-centric consistency models and replica management
 Today’s Session:
 Consistency and Replication – Part IV
 Consistency Protocols
 Programming Models – Part I
 Announcements:
 Project II is due today by 11:59PM
 Project III will be posted by tonight (and is due on Nov 6, 2013)
2
Overview
Consistency Models
Replica Management
Consistency Protocols
3
Consistency Protocols
A consistency protocol describes the implementation of a
specific consistency model
We are going to study three consistency protocols:
Primary-based protocols
One primary coordinator is elected to control replication across
multiple replicas
Replicated-write protocols
Multiple replicas coordinate together to provide consistency
guarantees
Cache-coherence protocols
A special case of client-controlled replication
4
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
Replicated-Write
Protocols
Cache
Coherence
Protocols
5
Primary-based protocols
In Primary-based protocols, a simple centralized design is
used to implement consistency models
Each data-item x has an associated “Primary Replica”
The primary replica is responsible for coordinating write operations
We will study one example of Primary-based protocols that
implements Sequential Consistency Model
Remote-Write Protocol
6
Remote-Write Protocol
Rules:
All write operations are forwarded to the primary replica
Read operations are carried out locally at each replica
Approach for write ops: (Budhiraja et al. 1993)
Client connects to some replica RC
If the client issues write operation to RC:
RC forwards the request to the primary replica RP
RP updates its local value
RP forwards the update to other replicas Ri
Other replicas Ri update, and send an ACK back to RP
After RP receives all ACKs, it informs RC that
the write operation is completed
RC acknowledges the client, which in return
completes the write operation
x+=5
Client 1
Primary server
R1
R2
R3
=5
x1=0
x22=0
=5
=5
x3=0
Data-store
7
Remote-Write Protocol – Discussion
The Remote-Write protocol provides
A simple way to implement sequential consistency
Guarantees that clients see the most recent write operations
However, latency is high in Remote-Write Protocols
Clients block until all the replicas are updated
Can a non-blocking strategy be applied?
Remote-Write Protocols are applied to distributed
databases and file systems that require fault-tolerance
Replicas are placed on the same LAN to reduce latency
8
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
ReplicatedWrite
Protocols
Cache
Coherence
Protocols
Remote-Write
Protocol
9
Replicated-Write Protocol
In a replicated-write protocol, updates can be carried out
at multiple replicas
We will study one example on replicated-write protocols
called Active Replication Protocol
Here, clients write at any replica
The modified replica will propagate updates to other replicas
10
Active Replication Protocol
When a client writes at a replica, the replica will send the
write operation updates to all other replicas
Challenges with Active Replication
Ordering of operations cannot be guaranteed across the replicas
x+=2
x*=3
Client 1
Client 2
R1
W(x)
x+=2
R(x)2
R(x)0
R2
R1
R2
R3
xx11=0
=2
=6
=0
=2
xx22=6
x3=0
=2
=6
Data-store
R3
R(x)6
W(x)
R(x)2
11
x*=3
R(x)2
R(x)6
Centralized Active Replication Protocol
Approach
There is a centralized coordinator called the sequencer (Seq)
When a client connects to a replica RC and issues a
write operation
RC forwards the update to the Seq
Seq assigns a sequence number to the update operation
RC propagates the sequence number and the operation to other replicas
Operations are carried out at all the replicas in the order defined by
the sequencer
x+=5
x-=2
Client 1
Client 2
10
R1
Seq
10
R2
x+=5
R3
11
11
x-=2
12
Data-store
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
Replicated-Write
Protocols
Remote-Write
Protocols
Active
Replication
Protocol
Cache
Coherence
Protocols
13
Cache Coherence Protocols
Caches are special types of replicas
Typically, caches are client-controlled replicas
Cache coherence refers to the consistency of data
stored in caches
How are the cache coherence protocols in sharedmemory multiprocessor (SMP) systems different from
those in Distributed Systems?
Coherence protocols in SMP assume cache states can be
broadcasted efficiently
In DS, this is difficult because caches may reside on different
machines
14
Cache Coherence Protocols (Cont’d)
Cache Coherence protocols determine how caches are
kept consistent
Caches may become inconsistent when a data item
is modified:
1.
2.
at the server replicas, or
at the cache
15
When Data is Modified at the Server
Two approaches for enforcing coherence:
1. Server-initiated invalidation
Here, server sends all caches an invalidation
message (when data item is modified)
2. Server updates the cache
Server will propagate the update to the cache
16
When Data is Modified at the Cache
The enforcement protocol may use one of three techniques:
i.
Read-only cache
The cache does not modify the data in the cache
The update is propagated to the server replica
ii.
Write-through cache
Directly modify the cache, and forward the update to the server
iii. Write-back cache
The client allows multiple writes to take place at the cache
The client batches a set of writes, and will send the batched write
updates to the server replica
17
Summary of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
ReplicatedWrite Protocols
Cache
Coherence
Protocols
Remote-Write
Protocols
Active
Replication
Protocol
Coherence
Enforcement
Strategies
18
Consistency and Replication –
Brief Summary
Replication improves performance and fault-tolerance
However, replicas have to be kept reasonably consistent
Consistency Models
• A contract between the data-store and processes
• Types: Data-centric and Client-centric
Replication Management
• Describes where, when and by whom replicas should be placed
• Types: Replica Server Placement, Content Replication and Placement
Consistency Protocols
• Implement Consistency Models
19
• Types: Primary-based, Replicated-Write, Cache Coherence
Back-up Slides
20
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
Remote-Write
Protocols
Replicated-Write
Protocols
Cache Coherent
Protocols
Local-Write
Protocols
21
Local-Write Protocols
Can we make Remote-Write better (e.g., for mobile
computers that are able to operate in disconnected
mode)
Approach:
Client connects to some replica RC
If the client issues write op to RC:
RC becomes the primary replica RP
x+=5
Client 1
Primary server
Rest of the protocol is similar to
Remote-Write
R1
R2
R3
x1=0
=5
=0
x2=5
x3=0
=5
Data-store
22
Local-Write Protocol
Advantages
Primary replica can propagate a batch of write updates instead of
an individual update
Multiple, successive write operations can be carried out locally,
while reading processes can still access their local copies
Scenarios where Local-Write is applicable:
Mobile computers when disconnected (all updates can be carried
locally, while other processes can still perform read operations- but
not updates)
Scenarios where Local-Write is inappropriate:
When (multiple) clients are writing at multiple replicas
Overhead of reassigning primary replica is high
23
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
Remote-Write
Protocols
Local-Write
Protocols
Cache
Coherence
Protocols
Replicated-Write
Protocols
Active
Replication
Protocol
24
Two aspects of Cache
Coherence Protocols
In order to maintain consistent caches, we need
to perform two operations:
Coherence detection strategies
Detect inconsistent caches
Coherence enforcement strategies
Update caches
25
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
ReplicatedWrite Protocols
Remote-Write
Protocols
Active
Replication
Protocol
Cache
Coherence
Protocols
Coherence
Detection
Strategies
Coherence
Enforcement
Strategies
26
Coherence Detection Strategies
Detection strategies deal with predicting when
caches are inconsistent
Since different replicas may be written by client
processes, the protocol has to dynamically detect
cache inconsistency
27
Coherence Detection Strategies
In a distributed system, cache inconsistencies can be typically detected
at three stages:
1. Verify coherence before every access
Before every read or write operation
2. Verify coherence before a write access
Cache coherence is checked before every write operation
3. Verify coherence after a write access
First, an update is performed, and later cache consistency is verified
If cache was inconsistent
The write operation is rolled-back, and re-performed
28
Overview of Consistency Protocols
Consistency
Protocols
Primary-based
Protocols
ReplicatedWrite Protocols
Remote-Write
Protocols
Active
Replication
Protocol
Cache
Coherence
Protocols
Coherence
Detection
Strategies
Coherence
Enforcement
Strategies
29
Coherence Enforcement Strategies
Enforcement strategies determine how
caches are kept consistent
Caches may become inconsistent when
data item is modified:
1. at the server replicas, or
2. at the cache
30
When Data is Modified at the Server
Two approaches for enforcing coherence:
1. Server-initiated invalidation
Here, server sends all caches an invalidation
message when data item is modified
2. Server updates the cache
Server will propagate the update to the cache
31