The Google File System
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
SOSP 2003
Presented by Wenhao Xu
University of British Columbia
Outline
•
•
•
•
•
•
•
•
•
Background
Assumption and Google Workloads
Architecture
System Interaction
Master
Relaxed Consistency Model
Experiment
Conclusion
Discussion
Background
• In 2003,
– 50+ GFS clusters
– Each with thousands of storage nodes
– Managing petabytes of data
– GFS is under lots of applications and other
systems in Google, such as bigtable, etc.
• Built on top of commodity machine and disks
Tailored for Google workloads
Assumption & Google Workloads
• Failures occur everywhere
– Disks, Network, Machines
High availability: Replicas,
Heartbeat, Fast recovery,
checksums
• Store a modest number of large filesLarge data chunk & No Client
data cache
– A few million files; Each typically 100MB or larger
in size
Throughput reads
is more
• Read: Large streaming reads and small random
important
– Typically read hundreds of KBs of an individual
operation,
more commonly 1MB or more
Make appending fast
– Small reads typically read a few KBs at arbitrary offsets,
which are usually batched and sortedRelaxed
to avoid
seeking
consistency model &
back and forth
Atomicity Record Append
• Mutation: Mostly appending new data
Customizable API & Don’t
• Concurrent clients
need to conform to POSIX
semantic!
• Design flexibility: Co-designing the application
and the
file system API
Architecture
Single Master
A Chunk
• Size: 64MB
• 64 bit handler
• Stored as a Linux file, lazy
space allocation
• 3+ replicas
• Single point of Failures?  Shadow masters,
operation log
• Performance bottlenecks?  Less involvment
• Simplicity: concurrency control, metadata
management, etc.
• Global knowledge: make sophisticated
placement and replication decisions.
System interaction – Mutation Data flow
• Leases – Primary replica
• Master is only involved in
1&2
• Separated data flow and
control flow
– Employs network
topology to transfer data
– Easily to control the
operation order
System Interaction - Heartbeat
• What is a heartbeat message like?
– “hey, I am still healthy. And I have chunks a, b, c ….
Their versions are v1, v3, v9, …”
• What is heartbeat for?
– Chunkserver status
– Chunk placement
– Garbage collection
– Stale Replica deletion
– Lease extension
Master - Metadata Management
• Three major types
– File and chunk namespaces
– (File, chunk) mapping
– Location of each chunk’s replicas
• All metadata is kept in the memory
– Possibility: (<64B/chunk, 226 (64M) * 64 = 4G, a few millions of files)
– Fast access & scanning
• Operation log
– Store First two types of metadata
– Any changes to the first two types of metadata are first written to the
operation log
– “It is replicated on multiple remote machines and respond to a client
operation only after flushing the corresponding log record to disk both
locally and remotely”
– Checkpoints are made to avoid larger log and long starting up time for
masters.
Master – Namespace Management & locking
• Namespace is organized into a lookup table
– (file name, metadata) mapping
– Does not support aliases for the same file or
directory
• Mutation lock
– Read lock of all parents directory
– Write lock of mutation directory
– E.g: /d1/d2/…/dn/leaf creation
• Read lock: /d1, /d1/d2, …, /d1/d2/…/dn
• Write lock: /d1/d2/…/dn/leaf
Master - Replica Creation and Placement
• Consideration
– Balance disk space utilization
– Performance: Limit the number of “recent” creations
(hot pot) on each chunkserver
– Availability: different racks; as far as possible
– Maximize network utilization: as less switches as
possible; exploit the aggregate bandwidth of multiple
racks;
– Multiple racks:
• Tradeoff: read bandwidth & write traffic
• Re-replication & rebalancing
Relaxed Consistency model
• Two state definitions:
– Consistent: All the clients will always see the same data, regardless of
which replicas they read from
– Defined: Consistent & Clients will see what the mutation writes in its
entirety
• The state after mutations depends on
– whether the ops succeeds or fails
– Whether there are concurrent mutations
• Move work to applications/clients
–
–
–
–
Error code  check consistency
Application checksum  check if it is defined
Identifier for record  figure out duplicates
Retry
• File namespace mutation are atomic
– Handled exclusively by the master
Atomic Record Appends
• Used heavily by Google applications
• Clients specify data
– Not specify offset in files
• GFS appends it to the files
– At least once atomically
– Successfully written regions are defined
– Does not guarantee that all replicas are identical
• Intervening regions are inconsistent
Experiments
Conclusion
• Gfs is designed for Google applications
• It is simple, but good enough for Google
applications
• It is important for Google to store data for
other applications.
Discussion (1/5)
• It is designed specifically to serve search
needs. For what other applications this
filesystem is suitable for. And for what
applications, it is not optimal?
• Reply
– Depends on:
• Major file size (small or large)
• Major file reads (streaming or not)
• Major writing modes (appending or not)
Discussion (2/5)
• When the Master decides on which replicas should a
particular chunk goes in, shouldn't it consider the potential
applications that is going to access that chunk ?
– Reply: Not really. More replicas if it is hot spot.
• We don't want to see unbalanced disk usages. In the paper
it mentioned that it "rebalances" for load balancing but a
way of doing it is not mentioned.
– Reply : Master knows the disk usages of all chunkservers
• In case of a shadow master is operating, it does not provide
any mutation operations. But the processes that got lease
from the primary before primary got crashed will continue
on writing to the disks right ? So how can we keep track of
those writes that is taken place while the shadow master is
in operation ?
– Reply : It can run until finished.
Discussion (3/5)
• What happens in GFS if the primary master is down? Will
one of the “shadow” masters become a new primary
master? If so, how to pick one from several “shadow” ones?
– Reply : Maybe.
• Will the centralized master-server architecture degrade the
GFS's scalability?
– Reply : Seems not under Google’s workload.
• GFS uses the "closest" client for replication. However, the
term "closest" is never explained - is it based on network
latency, or does it factor in network load on the closest
chunkserver, does it factor in the system latency? How does
one chunkserver figure out the closest server on the fly?
– Reply : Deduced from IP addresss
Discussion (4/5)
• The concept of 'records' appears frequently in the paper.
GFS can guarantee that a record is appended atomically,
and can associate an application defined checksum with
each record. This seems to imply that GFS keeps metadata
for each record. How is this done?
– Reply : The checksum is in the record data, not in the metadata.
• Clients cache chunk locations, so if one chunk server
cannot contact the master it may unwittingly serve stale
data to clients. Comments?
– Reply : The master includes the chunk version number when it
informs clients.
Discussion (5/5)
• What do you think about the performance gain of
“Concurrent appends to the same file operation” and the
“relaxed” consistency model under normal workload? (I
believe this is only suits well to google). Since appending is
key to GFS write performance in a multi-writer
environment, it might be that GFS would give up much of
its performance advantage even in large serial writes. What
do you think about this?
• Reply:
– It is a tradeoff between complexity between GFS and its clients.
– I think it is good for performance.
– It is tailored for google’s workloads (appending).
Thanks for your attention!