iDIBS: Reliable and Efficient
Distributed Backup
Tam Chantem, Philip Little and
Faruck Morcos
Graduate Operating Systems
‹#›
Improving Peer-to-Peer Backup Systems
• Objective: Improve peer-to-peer backup
techniques.
• Approach: Built on DIBS, an existing peer-topeer backup system.
• Results: Improved 3 aspects of DIBS:
– Reliability, Network Load, and Computation Time.
Graduate Operating Systems
‹#›
iDIBS
File
Erasure
Codes
Encoding
Peer
Peer
iDIBS
Peer
Database
Peer List
Graduate Operating Systems
‹#›
iDIBS
File
Erasure
Codes
Encoding
Peer
Peer
iDIBS
Peer
Database
Peer List
Graduate Operating Systems
‹#›
Backup and Recovery
Encoding and Distributing:
Peer
File
a b
c d
Peer
e f
g h
RS Encoder
a d g j
b e h k
c f i l
Graduate Operating Systems
Peer
i j
k l
‹#›
Backup and Recovery
Recovering and Decoding:
Peer
File
a b
c d
Dead Peer
e f
g h
RS Decoder
a d
j
b
k
c
i l
Graduate Operating Systems
Peer
i j
k l
‹#›
Peer-List Backups
Unmodified DIBS after a crash:
Client
Data
Do you have my data?
Peer
List
?
?
a
b c
?
?
Graduate Operating Systems
d
e f
‹#›
Peer-List Backups
iDIBS after a crash:
Client
Data
Do you have my peer list?
Peer
List
?
?
a
b c
Peer
List
d
e f
Peer
List
?
?
Graduate Operating Systems
‹#›
Peer-List Backups
iDIBS after a crash:
Client
Data
Peer
List
a
b c
d
e f
Graduate Operating Systems
‹#›
Thresholds
• K – Required number of pieces to recover
• N – Total number of pieces transmitted
• Redundancy level:
– DIBS: N = 2K
– iDIBS: Recovery when up to 1 peer is dead
Graduate Operating Systems
‹#›
Example
Assume:
Peer 1
Peers = 4
Peer 2
K=5
Peer 3
N=6
Peer 4
Can’t recover!!!
So, N = 8
Graduate Operating Systems
‹#›
Reduction in Network Utilization
Graduate Operating Systems
‹#›
Luby Transform Codes
• Reed-Solomon (RS) codes not scalable
• Luby Transform (LT) codes
– Digital Fountain concept
– Probabilistic nature
– 15% redundancy needed for successful
decoding
– Flexible symbol size T, to speed up decoding
Graduate Operating Systems
‹#›
Implementation: LT Codes
• Encoder and Decoder modules
• Encoder:
– Calculates redundancy needed
– Encodes and splits file depending on T,
number of users, and probabilistic parameters
• Decoder:
– Decodes pieces
– Reconstructs file
Graduate Operating Systems
‹#›
Luby Transform Codes Encoder/Decoder Performance
LT codes vs. RS codes
Decoder T=256
Processing Time (sec)
Encoder T=256
File Size (Kb)
Graduate Operating Systems
‹#›
iDIBS vs. DIBS
network utilization behavior
• DIBS network utilization is flat.
• iDIBS has decreased network
utilization. (OH when users<10)
• Tradeoff in the theoretical
minium is better in iDIBS.
• Results shown for T=32, for
larger numbers iDIBS plots are
better!
Graduate Operating Systems
‹#›
The iDIBS tradeoff
iDIBS
Advantages
LT codes are:
Disadvantages
LT codes are
probabilistic.
1. Faster than RS
2. Flexible
performance
given T.
Decreased overall
Network Utilization.
Peer Lists:
1. Allow faster
recovery.
2. Increase reliability.
As T > 256 decoding
is less stable
LT need a minimum of
15% of extra overhead
+ ESI to decode.
Peer lists induce a
small amount of
overhead
Graduate Operating Systems
‹#›
Contribution
We contributed to the improvement of DIBS in the following ways:
• Increase of performance through the introduction of a new encoding
technology. (LT codes)
• Introduction of this encoding scheme to the application of peer-topeer backup systems.
•
More Reliability through redesign of the system recovery scheme.
(Peer-Lists)
• Reduces Network utilization by changing the philosophy of number
of transmitted pieces N and LT codes.
Graduate Operating Systems
‹#›
Questions??
Graduate Operating Systems
‹#›
Implementation: Timeouts & Thresholds
If: Peers > K then
N = K + 1;
else:
PiecesPerPeer = 1;
while PiecesPerPeer * (Peers - 1) < K
do:
PiecesPerPeer = PiecesPerPeer + 1;
N = N * K;
end if
If a peer is offline, recovery is still possible
Graduate Operating Systems
‹#›