More I/O services and recovery
Jeff Chase
Duke University
The DeFiler buffer cache
File abstraction implemented in upper DFS layer.
All knowledge of how files are laid out on disk is at this layer.
Access underlying disk volume through buffer cache API.
Obtain buffers (dbufs), write/read to/from buffers, orchestrate I/O.
DBuffer dbuf = getBlock(blockID)
releaseBlock(dbuf)
DBufferCache
Device I/O interface
Asynchronous I/O to/from buffers
block read and write
Blocks numbered by blockIDs
DBuffer
read(), write()
startFetch(), startPush()
waitValid(), waitClean()
“Filers”
• Network-attached (IP)
• RAID appliance
• Multiple protocols
– iSCSI, NFS, CIFS
• Admin interfaces
• Flexible configuration
• Lots of virtualization: dynamic volumes
• Volume cloning, mirroring, snapshots, etc.
• NetApp technology leader since 1994 (WAFL)
What is clustered Data ONTAP?
 NetApp’s “scale-out” clustering solution
 1-24 “nodes” (filers), 10 Gb ethernet
interconnect
 Designed for extreme performance and
flexibility
 Key concepts:
– Vserver
– Aggregate
– Volume (FlexVol)
NetApp Confidential - Limited Use
5
Architecture of clustered Data ONTAP
Client
Client
Client
Client
Node
Network
SpinNP
N-Blade
N-Blade
Network,
Protocols
Network,
Protocols
Network,
Protocols
Network,
Protocols
D-Blade
D-Blade
D-Blade
D-Blade
WAFL,
RAID,
Storage,
HA,
NVRAM
WAFL,
RAID,
Storage,
HA,
NVRAM
WAFL,
RAID,
Storage,
HA,
NVRAM
WAFL,
RAID,
Storage,
HA,
NVRAM
HA Pair
N-Blade
HA
N-Blade
HA Pair
Client
Disks
NetApp Confidential - Limited Use
6
Notes on “scale-out”
• NetApp filer solution is “scalable”.
– “Just buy more blades.”
– Load is distributed: among the blades, interfaces, disks, etc..
– If we add another blade, it adds to the total load we can serve.
• What about failure?
– Redistribute load to the surviving blades.
– But what if the failed blade has some critical state?
• E.g., D-blades have writeback I/O caches.
– Replicate nodes in “HA pairs” (Highly Available).
– If one fails, the other can take over. (like the “buddy system”)
File data and metadata
A file’s data blocks could be “anywhere” on disk. The file’s inode maps them.
A fixed-size inode
has a fixed-size
block map.
How to represent large
files that have more
logical blocks than can
fit in the inode’s map?
attributes
block
map
inode
An inode could be “anywhere” on disk.
How to find the inode for a given file?
Once upo
n a time
/nin a l
and far
far away
,/nlived t
he wise
and sage
wizard.
data
blocks
Representing Large Files
inode
Classic Unix file systems
inode == 128 bytes
inodes are packed into blocks
direct
block
map
indirect
block
Each inode has 68 bytes of attributes
and 15 block map entries.
Suppose block size = 8KB
12 direct block map entries: 96KB of data.
One indirect block pointer in inode: + 16MB of data.
One double indirect pointer in inode: +2K indirects.
Maximum file size is 96KB + 16MB + (2K*16MB) + ...
double
indirect
block
indirect
blocks
Skewed tree block maps
• Inodes are the root of a tree-structured block map.
– Like multi-level hierarchical page tables, but
• These maps are skewed.
– Low branching factor at the root: just enough for small files.
– Small files are cheap: just need the inode to map it.
– Inodes for small files are small…and most files are small.
• Use indirect blocks for large files
– Requires another fetch for another map block
– But the shift to a high branching factor covers most large files.
• Double indirects allow very large files, at higher cost.
• Other advantages to trees?
Writes in the FS buffer cache
• Delayed writes
– Partial-block writes may be left dirty in the cache. The push is
deferred in expectation of later writes to the same block.
• Write-behind
– Dirty blocks are pushed asynchronously; write call returns
before push is complete.
– May lose data! Be sure you know the failure semantics of the
file systems you use in your life. A classic UNIX file system may
discard any subset of recent writes on failure.
– Fsync system call to wait for push, for use by applications that
really want to know their data is “safe”.
– Good file systems sync-on-close.
Failures, Commits, Atomicity
• Hard state is state (data) that the service needs to
function correctly. Soft state is non-critical.
• What guarantees does the system offer about the hard
state if the system fails?
– Durability
• Did my writes commit, i.e., are they saved? Can I get it back?
– Failure atomicity
• Can a set of writes “partly commit”?
– Recoverability and Corruption
• Can we recover a previous state?
• Is the state consistent?
• Is the metadata well-formed?
Failure/Atomicity
• File writes are not guaranteed to commit until close.
–
–
–
–
A process can force commit with a sync.
The system forces commit every (say) 30 seconds.
Failure may lose an arbitrary subset of writes.
Want better reliability? Use a database.
• Problem: safety of metadata.
–
–
–
–
How to protect the integrity of the metadata structures?
E.g., directory tree, inodes, block map trees
Metadata is a complex linked data structure, e.g., a tree.
Must be possible to restore with a scrub (“fsck”) on restart.
Write Anywhere File Layout (WAFL)
Metadata updates and recovery
• Metadata updates incur extra seek overhead.
– E.g., extending a file requires writes to the inode, direct
and/or indirect blocks, cylinder group bit maps and
summaries, and the file block itself.
• Metadata items are often updated repeatedly, so
delayed writes help.
• But delayed writes incur a higher risk of file system
corruption in a crash.
– If you lose your metadata, you are dead in the water.
• Option 1: careful write ordering, with scrub on recovery
• Option 2: shadowing (e.g., WAFL)
• Option 3: logging/journaling (used in databases)
Disk write behavior (cartoon version)
• Disk may reorder pending writes.
– Limited ordering support (“do X before Y”).
– Can enforce ordering by writing X synchronously: wait for write
of X to complete before issuing Y.
• Writes at sector grain are atomic (512 bytes?).
• Writes of larger blocks may fail “in the middle”.
• Disk may itself have a writeback cache.
– Even “old” writes may be lost.
– (Can be disabled.)
Metadata write ordering
A common approach to safety of file system metadata:
• Order essential metadata writes carefully.
– Various techniques to enforce a partial order of writes to the disk,
i.e., ensure that write A completes before write B begins.
• Maintain invariants! E.g., avoid dangling references.
– Never recycle a structure (block or inode) before zeroing all
pointers to it (truncate, unlink, rmdir).
– Never point to a new structure before it has been initialized.
E.g., sync inode on creat before filling directory entry, and sync a
new block before writing the block map.
• Traverse the metadata tree to rebuild derived structures.
– Can rebuild e.g., free block bitmaps, free inode bitmaps.
A quick peek at sequential write
in BSD Unix “FFS” (Circa 2001)
note sequential
block allocation
physical
disk
sector
write
write stall
read
sync command
(typed to shell)
pushes indirect
blocks to disk
sync
read next
block of
free space
bitmap (??)
time in milliseconds
Sequential writes: a closer look
physical
disk
sector
16 MB in one second
(one indirect block worth)
140 ms
delay for
cylinder seek
etc. (???)
time in milliseconds
Push indirect blocks synchronously.
longer delay
for head movement
to push indirect
blocks
write
write stall
Small-File Create Storm
50 MB
inodes and
file contents
(localized allocation)
physical
disk
sector
note synchronous
writes for some
metadata
sync
sync
delayed-write
metadata
sync
write
write stall
time in milliseconds
WAFL and Writes
• In WAFL, any modified data or metadata can
go anywhere on the disk.
– Don’t overwrite the “old” copies!
– Group writes together and write them in bursts!
– Avoid small writes!  better RAID performance
• An arbitrary stream of updates can be
installed atomically.
– Retain the old copy, switch to new copy with a
single write to the root of the tree (shadowing).
– Old copy lives on as a point-in-time snapshot.
Shadowing
1. starting point
2. write new blocks to disk
modify purple/grey blocks prepare new block map
3. overwrite block map
(atomic commit)
and free old blocks
WAFL’s on-disk structure (high level)
Root Inode
Metadata
File Data Blocks
NetApp Confidential - Limited Use
23
WAFL Snapshots
The snapshot mechanism is used for user-accessible
snapshots and for transient consistency points.
Another Look
Write Anywhere File Layout
(WAFL)
WAFL and the disk system
• WAFL generates a continuous stream of large-chunk
contiguous writes to the disk system.
– WAFL does not overwrite the old copy of a modified structure: it
can write a new copy anywhere. So it gathers them together.
• Large writes minimize seek overhead and deliver the full
bandwidth of the disk.
• WAFL gets excellent performance by/when using many
disks in tandem (“RAID”)…
• …and writing the chunks in interleaved fashion across
the disks (“striping”).
• Old copies of the data and metadata survive on the disk
and are accessible through point-in-time “snapshots”.
Building a better
disk: RAID 5
• Redundant Array of
Independent Disks
• Striping for high throughput
for pipelined reads.
• Data redundancy: parity
• Enables recovery from one
disk failure
• RAID5 distributes parity:
no“hot spot” for random
writes
• Market standard
Fujitsu
RAID Parity with XOR
• Bitwise exclusive or: xor
• “One or the other but not both”
• Bit sum, with no carry
– Commutative, associative
• Or: sum up the 1s, and choose parity to
make the sum even.
– “Even parity”
• If any bit is lost, xor of the surviving bits
gets it back.
• If any bit flips, the parity flips.
• What if two bits flip or are lost?
XOR
WAFL & NVRAM
 NVRAM provides low latency pending request
stable storage
– Many clients don’t consider a write complete
until it has reached stable storage
 Allows WAFL to defer writes until many are
pending (up to ½ NVRAM)
– Big write operations: get lots of bandwidth out
of disk
– Crash is no problem, all pending requests
stored; just replay
– While ½ of NVRAM gets written out, other is
accepting requests (continuous service)
NetApp Confidential - Limited Use
30
Postnote
• The following slides on logging were not covered in
lecture and won’t be tested.
Alternative: Logging/Journaling
• Logging can be used to localize synchronous
metadata writes, and reduce the work that must be
done on recovery.
• Universally used in database systems.
• Used for metadata writes in journaling file systems
• Key idea: group each set of related updates into a
single log record that can be written to disk
atomically (“all-or-nothing”).
– Log records are written to the log file or log disk
sequentially.
• No seeks, and preserves temporal ordering.
– Each log record is trailed by a marker (e.g., checksum) that
says “this log record is complete”.
– To recover, scan the log and reapply updates.
Transactions
Transactions group a sequence of operations, often on
different objects.
BEGIN T1
read X
read Y
…
write X
END
BEGIN T2
read X
write Y
…
write X
END
Recoverable Data with a Log
volatile memory (e.g., buffer cache)
Your program (or file system
or database software)
executes transactions that
read or change the data
structures in memory.
Your data structures
in memory
Log transaction
events as they occur
Push a checkpoint or
snapshot to disk
periodically
snapshot
log
After a failure, replay the log into the last snapshot to recover the data structure.
Transactions: logging
Commit
WriteN
Write1
Begin transaction
Append info about modifications to a log
Append “commit” to log to end x-action
Write new data to normal database
Single-sector write commits x-action (3)
Begin
1.
2.
3.
4.

…
Transaction Complete
Invariant: append new data to log before applying to DB
Called “write-ahead logging”
Transactions: logging
Commit
WriteN
…
Write1
Begin transaction
Append info about modifications to a log
Append “commit” to log to end x-action
Write new data to normal database
Single-sector write commits x-action (3)
Begin
1.
2.
3.
4.

What if we crash here (between 3,4)?
On reboot, reapply committed updates in log order.
Transactions: logging
WriteN
…
Write1
Begin transaction
Append info about modifications to a log
Append “commit” to log to end x-action
Write new data to normal database
Single-sector write commits x-action (3)
Begin
1.
2.
3.
4.

What if we crash here?
On reboot, discard uncommitted updates.
head
(old)
Anatomy of a Log
• A log is a sequence of records
(entries) on recoverable storage.
• Each entry is associated with
some transaction.
• The entries for transaction T
record execution progress of T.
• Atomic operations:
– Append/write entry to log tail
– Truncate entries from log head
Log
Sequence
Number
(LSN)
...
LSN 11
XID 18
LSN 12
XID 18
Transaction
ID (XID)
LSN 13
XID 19
commit
record
LSN 14
XID 18
commit
– Read/scan entries
• Log writes are atomic and durable,
and complete detectably in order.
tail
(new)
head
(old)
Using a Log
• Log entries for T record the writes
by T or method calls in T.
– Redo logging
• To recover, read the checkpoint
and replay committed log entries.
Log
Sequence
Number
(LSN)
– Skip the records of uncommitted Ts
• No T can affect the checkpoint
until T’s commit is logged.
– Write-ahead logging
LSN 11
XID 18
LSN 12
XID 18
Transaction
ID (XID)
LSN 13
XID 19
– “Redo” by reissuing writes or
reinvoking the methods.
– Redo in order (head to tail)
...
commit
record
LSN 14
XID 18
commit
tail
(new)
head
(old)
Managing a Log
• On checkpoint, truncate the log
– No longer need the entries to recover
• Checkpoint how often? Tradeoff:
– Checkpoints are expensive, BUT
– Long logs take up space
Log
Sequence
Number
(LSN)
– Is it safe to redo/replay records whose
effect is already in the checkpoint?
– Checkpoint “between” transactions,
so checkpoint is a consistent state.
– Lots of approaches
LSN 11
XID 18
LSN 12
XID 18
Transaction
ID (XID)
LSN 13
XID 19
– Long logs increase recovery time
• Checkpoint+truncate is “atomic”
...
commit
record
LSN 14
XID 18
commit
tail
(new)