Pond
The OceanStore Prototype
Introduction
Problem: Rising cost of storage
management
Observations:
Universal connectivity via Internet
$100 terabyte storage within three years
Solution: OceanStore
OceanStore
Internet-scale
Cooperative file system
High durability
Universal availability
Two-tier storage system
Upper tier: powerful servers
Lower tier: less powerful hosts
OceanStore
More on OceanStore
Unit of storage: data object
Applications: email, UNIX file system
Requirements for the object interface
Information universally accessible
Balance between privacy and sharing
Simple and usable consistency model
Data integrity
OceanStore Assumptions
Infrastructure untrusted except in
aggregate
Most nodes are not faulty and malicious
Infrastructure constantly changing
Resources enter and exit the network
without prior warning
Self-organizing, self-repairing, self-tuning
OceanStore Challenges
Expressive storage interface
High durability on untrusted and
changing base
Data Model
The view of the system that is
presented to client applications
Storage Organization
OceanStore data object ~= file
Ordered sequence of read-only versions
Every version of every object kept forever
Can be used as backup
An object contains metadata, data, and
references to previous versions
Storage Organization
A stream of objects identified by AGUID
Active globally-unique identifier
Cryptographically-secure hash of an
application-specific name and the owner’s
public key
Prevents namespace collisions
Storage Organization
Each version of data object stored in a
B-tree like data structure
Each block has a BGUID
• Cryptographically-secure hash of the block
content
Each version has a VGUID
Two versions may share blocks
Storage Organization
Application-Specific Consistency
An update is the operation of adding a
new version to the head of a version
stream
Updates are applied atomically
Represented as an array of potential
actions
Each guarded by a predicate
Application-Specific Consistency
Example actions
Replacing some bytes
Appending new data to an object
Truncating an object
Example predicates
Check for the latest version number
Compare bytes
Application-Specific Consistency
To implement ACID semantic
Check for readers
If none, update
Append to a mailbox
No checking
No explicit locks or leases
Application-Specific Consistency
Predicate for reads
Examples
• Can’t read something older than 30 seconds
• Only can read data from a specific time frame
System Architecture
Unit of synchronization: data object
Changes to different objects are
independent
Virtualization through Tapestry
Resources are virtual and not tied to
particular hardware
A virtual resource has a GUID, globally
unique identifier
Use Tapestry, a decentralized object
location and routing system
Scalable overlay network, built on TCP/IP
Virtualization through Tapestry
Use GUIDs to address hosts and
resources
Hosts publish the GUIDs of their
resources in Tapestry
Hosts also can unpublish GUIDs and
leave the network
Replication and Consistency
A data object is a sequence of read-only
versions, consisting of read-only blocks,
named by BGUIDs
No issues for replication
The mapping from AGUID to the latest
VGUID may change
Use primary-copy replication
Replication and Consistency
The primary copy
Enforces access control
Serializes concurrent updates
Archival Storage
Replication: 2x storage to tolerate one
failure
Erasure code is much better
A block is divided into m fragments
m fragments encoded into n > m
fragments
Any m fragments can restore the original
object
Caching of Data Objects
Reconstructing a block from erasure
code is an expensive process
Need to locate m fragments from m
machines
Use whole-block caching for frequentlyread objects
Caching of Data Objects
To read a block, look for the block first
If not available
Find block fragments
Decode fragments
Publish that the host now caches the block
Amortize the cost of erasure
encoding/decoding
Caching of Data Objects
Updates are pushed to secondary
replicas via application-level multicast
tree
The Full Update Path
Serialized updates are disseminated via
the multicast tree for an object
At the same time, updates are encoded
and fragmented for long-term storage
The Full Update Path
The Primary Replica
Primary servers run Byzantine
agreement protocol
Need more than 2/3 nonfaulty participants
Messages required grow quadratic in the
number of participants
Public-Key Cryptography
Too expensive
Use symmetric-key message
authentication codes (MACs)
Two to three orders of magnitude faster
Downside: can’t prove the authenticity of
a message to the third party
Used only for the inner ring
Public-key cryptography for outer ring
Proactive Threshold Signatures
Byzantine agreement guarantees
correctness if not more than 1/3 servers
fail during the life of the system
Not practical for a long-lived system
Need to reboot servers at regular
intervals
Key holders are fixed
Proactive Threshold Signatures
Proactive threshold signatures
More flexibility in choosing the membership
of the inner ring
A public key is paired with a number of
private keys
Each server uses its key to generate a
signature share
Proactive Threshold Signatures
Any k shares may be combined to
produce a full signature
To change membership of an inner ring
Regenerate signature shares
No need to change the public key
Transparent to secondary hosts
The Responsible Party
Who chooses the inner ring?
Responsible party:
A server that publishes sets of failureindependent nodes
• Through offline measurement and analysis
Software Architecture
Java atop the Staged Event Driven
Architecture (SEDA)
Each subsystem is implemented as a stage
With each own state and thread pool
Stages communicate through events
50,000 semicolons by five graduate
students and many undergrad interns
Software Architecture
Language Choice
Java: speed of development
Strongly typed
Garbage collected
Reduced debugging time
Support for events
Easy to port multithreaded code in Java
• Ported to Windows 2000 in one week
Language Choice
Problems with Java:
Unpredictability introduced by garbage
collection
Every thread in the system is halted while
the garbage collector runs
Any on-going process stalls for ~100
milliseconds
May add several seconds to requests travel
cross machines
Experimental Setup
Two test beds
Local cluster of 42 machines at Berkeley
• Each with 2 1.0 GHz Pentium III
• 1.5GB PC133 SDRAM
• 2 36GB hard drives, RAID 0
• Gigabit Ethernet adaptor
• Linux 2.4.18 SMP
Experimental Setup
PlanetLab, ~100 nodes across ~40 sites
• 1.2 GHz Pentium III, 1GB RAM
• ~1000 virtual nodes
Storage Overhead
For 32 choose 16 erasure encoding
2.7x for data > 8KB
For 64 choose 16 erasure encoding
4.8x for data > 8KB
The Latency Benchmark
A single client submits updates of
various sizes to a four-node inner ring
Metric: Time from before the request is
signed to the signature over the result
is checked
Update 40 MB of data over 1000
updates, with 100ms between updates
The Latency Benchmark
Latency Breakdown
Update Latency (ms)
Key
Size
512b
1024b
Update 5%
Size
Time
Median
Time
95%
Time
Phase
Time (ms)
Check
0.3
Serialize
6.1
4kB
39
40
41
2MB
1037
1086
1348
Apply
1.5
4kB
98
99
100
Archive
4.5
2MB
1098
1150
1448
Sign
77.8
The Throughput Microbenchmark
A number of clients submit updates of
various sizes to disjoint objects, to a
four-node inner ring
The clients
Create their objects
Synchronize themselves
Update the object as many time as
possible for 100 seconds
The Throughput Microbenchmark
Archive Retrieval Performance
Populate the archive by submitting
updates of various sizes to a four-node
inner ring
Delete all copies of the data in its
reconstructed form
A single client submits reads
Archive Retrieval Performance
Throughput:
1.19 MB/s (Planetlab)
2.59 MB/s (local cluster)
Latency
~30-70 milliseconds
The Stream Benchmark
Ran 500 virtual nodes on PlanetLab
Inner Ring in SF Bay Area
Replicas clustered in 7 largest P-Lab sites
Streams updates to all replicas
One writer - content creator – repeatedly
appends to data object
Others read new versions as they arrive
Measure network resource consumption
The Stream Benchmark
The Tag Benchmark
Measures the latency of token passing
OceanStore 2.2 times slower than
TCP/IP
The Andrew Benchmark
File system benchmark
4.6x than NFS in read-intensive phases
7.3x slower in write-intensive phases