Snapshots in a Flash with ioSnapTM
Sriram Subramanian, Swami Sundararaman, Nisha Talagala,
Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau
Copyright
©
2014 Fusion-io, Inc. All rights reserved.
Snapshots Overview
• Point-in-time representation of the state of a storage device
• Used for backup/recovery, audit trails
• Implemented at different levels
• Device, block layer, LVM, file systems, applications, databases
• Use Copy-on-Write (CoW) or Redirect-on-Write (RoW)
10
10
Update Data
Update Data
10
Existing Data
2
10
Existing Data
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Existing Data
Snapshots in a Flash with ioSnapTM (patent pending technology)
Why Rethink Snapshots for Flash?
• Flash is revolutionizing storage systems
•
Accelerating data centers, enterprise apps, desktops
• Natural fit for supporting snapshots
• Redirect-on-Write: never overwrite existing data
• Log-structuring: data ordered on their creation time (almost)
• Rate of data-change is higher for Flash devices
• e.g., multi-threaded 8KB IOs, TB capacity device
•
•
3
Flash: 30K IOPS and device fills in ~1 hour
HDD : 500 IOPS and device fills in ~3 days
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Why Implement Snapshots in Block Layer?
Free space
Namespace
Free space
Namespace
journaling
Mapping
journaling
Mapping
File Systems
File Systems
Free space
Block Driver
Dynamic
provisioning
Block Driver
Atomic write
Mapping
Block Driver
Disk-based Systems
4
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Flash-based Systems
Snapshots in a Flash with ioSnapTM (patent pending technology)
Snapshots in Flash Challenges
• Users are sensitive to performance variability
• Need predictable performance all the time
• NAND flash has low endurance & inefficient in-place writes
• In-place updates of reference counts not possible
• Data could be moved due to a variety of reasons
• Cannot waste storage space for storing snapshot metadata
• $/GB is high and need to keep costs low
5
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
ioSnapTM Overview
• First flash-aware snapshotting system
•
•
•
•
Leverages RoW in FTL to support snapshot operations
Proposes usage of epochs in FTL to maintain log-time ordering
Supports large number of writable snapshots (216)
Embraces rate-limiting to minimize performance implications
• Performance Results (prototype built into Fusion-io VSL driver)
• Instantaneous snapshot creation/deletion (~50usec, 4k metadata)
• Matches vanilla read/write performance numbers
• Provides predictable performance for foreground IOs
6
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Outline
• Introduction
• ioSnapTM Design
• Evaluation
• Conclusions
7
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Design Goals and Assumptions
Goals
•
•
•
•
Negligible impact on foreground performance
Predictable foreground performance
Minimal space overheads for snapshot metadata
Integrate with existing FTLs
Assumptions
• Snapshots are primarily used for backup, disaster recovery
• Creates/deletes are common operations
• Accesses/activates are rare operations
8
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Relevant components of Fusion-io VSLTM
• Organizes NAND Flash into segments & maintains a log
Free Space Mgmt.
Validity of blocks
Segment cleaner
Mapping Layer (FTL)
LBA
NAND ADDR
Crash Recovery
Reconstruct
forward and validity map
Virtual Storage Layer (VSLTM)
Segment 1
9
Segment N
NAND Flash Modules
Copyright © 2014 Fusion-io, Inc. All rights reserved.
01001100
00000000
Segment 1
Segment N
Validity Bitmap
Snapshots in a Flash with ioSnapTM (patent pending technology)
Creating / Deleting Snapshots in Flash
Segment
Boundary
Key concept:
Segment 1
S1
S2
Segment 2
S3
Snapshot
Segment N
Infinite size log
SN
Creation is just writing a snapshot create note in the log
Deletion is similar to creation – simply write a deletion note
Snapshot creation/deletion is fast & negligible (fixed) metadata
Leverage time ordering of data in a log to create snapshots
10
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Well What About Segment Cleaner?
10 20 30
10
Segment
Boundary
40 60
Snapshot
Active
S1
40 60 10 20 70 30 10
Epoch: notion of period of time
Epoch 1
Active
S1
Active S1 Active
Epoch 2
Epoch
10 20 30
10
40 60
40 60 10 20 70 30 10
Epochs help preserve
ordering
S1
Active log-time
Active S1 Active
11
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
LBA
Managing Liveness of blocks
• Issue: snapshots indirectly increase the reference count
• Validity bitmap with a single bit doesn’t work
• Possible solutions:
• Maintain more bits/block (216 snapshot implies 16x increase in bitmaps)
• Selectively maintain per sub-segment bitmap for snapshots
•
Only create a bitmap if a snapshot has (or modified) data in it
Insight: determine if a given block has at least one
reference to it
12
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Preserving Liveness Via CoW Validity Bitmaps
Epoch 1
10 20 30 40
Epoch 1
1
1
1
1
Segment
Boundary
60
1
0
Epoch 1
10 20 30 40
Epoch 1
1
1
1
1
0
0
Epoch 2
60
10
1
0
0
0
Validity
Map CoW
Epoch 2 (step1)
1
1
1
1
1
0
0
0
Epoch 2 (step2)
0
1
1
1
1
1
0
0
13
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Bits needed
to be flipped
Snapshot
Snapshot-Aware Segment Cleaner
Epoch 1
10 20 30 10
Epoch 2
60
10
Segment
Boundary
Snapshot
60 70
Valid Block
Epoch 1
0
1
1
1
1
0
0
0
0
1
1
1
Epoch 2
0
1
1
0
Merge Bitmaps
0
1
1
1
Epoch 1
Invalid Block
Epoch 2
Epoch 1
60 70
60
10
Epoch 1
1
0
0
0
1
1
1
Epoch 2
0
1
1
1
1
1
0
Cleaned Segment
20 30 10
Segment cleaner preserves log-time ordering
14
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Design Summary
• Leverage implicit time ordering in the Log
• Epochs preserve log-time ordering even with a cleaner
• Sub-segment-level bitmaps to track validity of blocks
• Lazily update reference counts for snapshotted blocks
• Snapshot-aware cleaner preserves log-time and block
validity
• Snapshot management
• Background snapshot activation
• Rate-limiting to provide predictable foreground performance
15
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Outline
• Introduction
• ioSnapTM Design
• Evaluation
• Conclusions
16
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Evaluation
• What’s the impact on user IOs in the absence of snapshots?
• Snapshot creation/deletion time? Implications on user IO?
• How does it compare with existing snapshotting systems?
• Implications of a snapshot-aware segment cleaner?
• How long does it talk to activate a snapshot?
• Implications on the crash recovery mechanism?
Setup: quad core Intel i7 processor, 1.2TB NAND flash, 12GB RAM, Linux
2.6.35, older generation of Fusion-io VSL driver, 4K Block sizes
17
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Bandwidth (MB/s)
Performance of Regular operations
Experiment: 4K read/writes using two threads. 16GB of data read/written.
Asynchronous writes to device.
1800
1600
1400
Vanilla
1200
1000
ioSnap
800
600
400
200
0
Sequential Random Write Sequential Random Read
Write
Read
Snapshot support does not impact regular operations
18
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Snapshot creation / Deletion
• Always 50us latency and 4K metadata (independent of data)
Impact on foreground operations?
• Write 512 byte blocks to
arbitrary logical addresses (total
data of 3GB).
• Create a snapshot and then
continue writing data to random
locations (of 8 MB).
19
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Validity bitmap CoW
Snapshots in a Flash with ioSnapTM (patent pending technology)
Comparison with BTRFS (1)
Impact on foreground latency upon snapshot creation
Around 8 GB of sequential writes followed by a random workload interspersed by a
snapshot every 5 sec
Snapshots in ioSnap does not impact foreground latency
20
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Comparison with BTRFS (2)
Impact of snapshots on sustained bandwidth
Around 200 GB of sequential writes followed by a random workload interspersed by
a snapshot once every 15 sec
Snapshots in ioSnap does not impact sustained bandwidth
21
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Outline
• Introduction
• ioSnapTM Design
• Evaluation
• Conclusions
22
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Conclusions
“Make everything as simple as possible, but not simpler.”
– Albert Einstein
• Flash is revolutionizing the storage industry
• Rethink data services to leverage flash’s capability & performance
• ioSnap: first flash-aware snapshotting system
• Leverages RoW capability in FTLs to implement snapshots
• Proposes usage of epochs to preserve log-time ordering
• Low-overhead instantaneous snapshots (performance & storage)
• Embraces rate-limiting to minimize impact to foreground traffic
• Activations are slow & can be 10s of sec for a TB size snapshot
23
Copyright © 2014 Fusion-io, Inc. All rights reserved.
Snapshots in a Flash with ioSnapTM (patent pending technology)
Thank You
fusionio.com | DELIVERING THE WORLD’S DATA. FASTER.