Bigtable: A Distributed Storage
System for Structured Data
F. Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D.A. Wallach
M. Burrows, T. Chandra, A. Fikes, R.E. Gruber
Google, Inc.
Tianyang HU
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
2
Motivation
• Scalability
– worldwide applications & users
– huge amount of communication & data
3
Bigtable
• Distributed storage system
– petabytes of data, thousands of machines
– simple data model with dynamic control
– applicability, scalability, performance, availability
• Used by more than 60 applications of Google
4
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
5
Data Model – Overview
• Sparse, distributed, persistent multidimensional sorted map.
6
Data Model – Example
• “Webtable” stores copy of web pages & their related
information.
– row key: URL (reverse hostname)
– column key: attribute name
– timestamp: time that the page is fetched
7
Data Model – Rows
• Row key: string (usually 10-100KB, max 64KB)
• Every R/W of data under a single row key is atomic
8
Data Model – Rows
• Sorted by row key in lexicographic order
• Tablet: a certain range of rows
– the unit of distribution & load balancing
– good locality for data access
9
Data Model – Columns
• Column families: group of column keys (same type)
– the unit of access control
• Column key: family:qualifier
10
Data Model – Timestamps
• Timestamp: index multiple versions of the same data
– not necessarily the “real time”
– data clean up, garbage collection
11
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
12
Building Blocks
• SSTable file format
– persistent, ordered, immutable key-value (string-string) pairs
– used internally to store Bigtable data
13
Building Blocks
• GFS
– store log & data files
– scalability, reliability, performance, fault tolerance
• Chubby
– a highly-available and persistent distributed lock service
14
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
15
Bigtable Components
• A library that is linked into every client
• Many tablet servers
– handle R/W to tablets with clients
• One tablet master
– assign tablets to tablet servers
– detect addition & expiration of tablet servers
– balance tablet-server load
16
Architecture
17
Tablet Location
• Three-level hierarchy
– root tablet (Only one, stores addresses of METADATA tablets)
– METADATA tablets (stores addresses of user tablets)
– user tablets
18
Tablet Location
• Client caches (multiple) tablet locations
– if the cache is stale, query again
19
Tablet Assignment
• The tablet master uses Chubby to keeps track of
– live tablet servers
• each live tablet server acquires an exclusive lock on a corresponding file
– tablet assignment status
• compare tablets registered in METADATA tablet with tablets in tablet servers
20
Tablet Assignment
• Case 1: some tablets are unassigned
– master assigns them to tablet servers with sufficient room
• Case 2: a tablet server stops its service
– master detects it and assigns outstanding tablets to other servers.
• Case 3: too many small tablets
– master initiates merge
• Case 4: a tablet grows too large
– the corresponding tablet server initiates split and notifies master
21
Tablet Serving
• A tablet is stored as a sequence of SSTables in GFS
• Tablet mutations are logged in commit log
– the “commit log” stores redo records
– recent tablet versions are stored in memory (memtable)
– older tablet versions are stored in GFS
22
Tablet Serving
• Recover a tablet
– 1. Tablet server fetches its metadata from METADATA tablet, which
contains a list of SSTables that comprises a tablet and redo points.
– 2. The server reads the indices of the SSTables into memory.
– 3. The server applies all the mutations after the redo point.
23
Tablet Serving
• Write operation on a tablet
–
–
–
–
1. The tablet server checks the validity of the operation.
2. The operation is logged in the commit log.
3. Commit the operation.
4. The content of tablet is inserted into memtable.
24
Tablet Serving
• Read operation on a tablet
– 1. The tablet server checks the validity of the operation.
– 2. Execute the operation on a merged view of memtable & SSTables.
25
Compactions
• Memtable grows as write operations execute
• Two types of compactions
– minor compaction
– merging (major) compaction
26
Compactions
• Minor compaction (when memtable size reaches a threshold)
– 1. Freeze the memtable
– 2. Create a new memtable
– 3. Convert the memtable to an SSTable and write to GFS
27
Compactions
• Merging compaction (periodically)
– 1. Freeze the memtable
– 2. Create a new memtable
– 3. Merge a few SSTables & memtable into a new SSTable
28
Compactions
• Major compaction
– special case of merging compaction
– merges all SSTables & memtable
29
Compactions
• Why freeze & create memtable?
– Incoming read and write operations can continue during compactions.
• Advantages of compaction:
– release the memory of the tablet server
– reduce the amount of data that has to be read from the commit log
during recovery if this tablet server dies
30
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
31
Refinements
• Locality groups
–
–
–
–
group multiple column families together
different locality groups are not typically accessed together
for each tablet, store each locality group in a separate SSTable
more efficient R/W
32
Refinements
• Compression
– similar data in same column, neighbouring rows, multiple versions
– customized compression on SSTable block level (smallest component)
– two-pass compression scheme
• 1. Bentley and McIlroy’s scheme, compress long strings across a large window
• 2. fast compression algorithm, look for repetitions in small window
• experimental compression ratio: 10% (Gzip: 25-33%)
33
Refinements
• Caching
– two-level cache on tablet server
– Scan cache (high-level): caches key-value pairs
• Case: read the same data repeatedly
– Block cache (low-level): caches SSTable blocks
• Case: sequential read
34
Refinements
• Commit-log implementation
–
–
–
–
one commit log per tablet incurs a large # of disk seeks
use single commit log for all tablets on a tablet server
benefits significantly during normal operation
complicates recovery
• solution: sort the commit log entries first
35
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
36
Performance Evaluation
R/W rate per tablet server
aggregate R/W rate
37
Outline
•
•
•
•
•
•
•
Introduction
Data Model
Building Blocks
Implementation
Refinements
Performance Evaluation
Future Work
38
Future Work
• Resource sharing for different applications?
• Hybrid with relational database?
– complex query
– security
39