Deconstructing Storage Arrays
Timothy E. Denehy, John Bent, Florentina I. Popovici,
Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau
University of Wisconsin, Madison
Gray-box Research
•
Computer systems becoming more complex
•
•
•
•
Transistors
Lines of code
Each component is becoming more complex
Interactions between subsystems can affect
•
•
•
•
Performance
Reliability
Power
Security
Gray-box Research
•
Interfaces remain the same
•
Changes can be difficult and impractical
•
Support multiple platforms or legacy systems
•
Commercial acceptance for wide-spread adoption
•
Hardware and software phenomenon
•
•
IA-32 instruction set, POSIX OS, SCSI storage
Problem: lack of information
Gray-box Solution
•
Treat target system as a gray-box
•
•
Extract information from an existing interface
•
•
General characteristics are known
e.g. determine cache contents
Exploit information to control system behavior
•
e.g. access cached data first
Gray-box Information Techniques
•
Make assumptions about target system
•
Observe system inputs and outputs
•
Statistical methods
•
•
Draw inferences about internal structure
Microbenchmarks and probes
•
Parameterize system components
•
Observe system under controlled input
Gray-box Applications
•
Gray-box techniques have been used to identify
•
Memory hierarchy parameters [Saavedra and Smith]
•
Processor cycle time [Staelin and McVoy]
•
Low-level disk characteristics [Worthington et al.]
•
Buffer cache replacement algorithms [Burnett et al.]
•
File system data structures [Sivathanu et al.]
•
storage array characteristics: Shear
Shear
•
Software tool that automatically determines the
important properties of a storage array
•
Enables file system performance tuning with
knowledge of storage array characteristics
•
Acts as a management tool to help configure,
monitor, and maintain storage arrays
Outline
•
•
Introduction
Shear
•
•
•
Case Studies
•
•
•
Background
Algorithm
Performance: Stripe-aligned Writes
Management: Detecting Misconfiguration, Failure
Conclusion
Shear Goals
•
Determine storage array characteristics
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
SCSI
Shear Goals
•
Determine storage array characteristics
•
Number of disks
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
SCSI
Shear Goals
•
Determine storage array characteristics
•
•
Number of disks
Chunk size
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
SCSI
Shear Goals
•
Determine storage array characteristics
•
•
•
Number of disks
Chunk size
Layout and redundancy scheme
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
0 1 2 3
16171819
RAID-0
4 5 6 7
20212223
8 9 1011
24252627
12131415
28293031
SCSI
Shear Goals
•
Determine storage array characteristics
•
•
•
Number of disks
Chunk size
Layout and redundancy scheme
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
RAID-1
0 1 2 3
16171819
24252627
4 5 6 7
20212223
28293031
0 1 2 3
16171819
24252627
4 5 6 7
20212223
28293031
SCSI
Shear Goals
•
Determine storage array characteristics
•
•
•
Number of disks
Chunk size
Layout and redundancy scheme
SCSI
0 1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031
RAID-5
0 1 2 3
16171819
32333435
P P P P
4 5 6 7
20212223
P P P P
36373839
8 9 1011
P P P P
24252627
40414243
P P P P
12131415
28293031
44454647
SCSI
Shear Motivation
•
Performance
•
•
Tune file systems to array characteristics
Management
•
Verify configuration
•
Detect failure
Shear Techniques
•
•
Microbenchmarks and probes
•
Controlled, random access read and write patterns
•
Measure response time of access patterns
•
Measure steady-state performance
Statistical clustering
•
Automatically classify fast and slow regimes
•
Identify patterns that utilize only a single disk
Shear Assumptions
•
•
Storage array
•
Layout follows a repeatable pattern
•
Composed of homogeneous disks
System
•
Able to bypass the file system and buffer cache
•
Little traffic from other processes
Outline
•
•
Introduction
Shear
•
•
•
Case Studies
•
•
•
Background
Algorithm
Performance: Stripe-aligned Writes
Management: Detecting Misconfiguration, Failure
Conclusion
Shear Algorithm
•
Pattern size
•
Chunk size
•
Layout of chunks to disks
•
Level of redundancy
Determining the Pattern Size
•
Find the size of the layout's repeating pattern
•
•
Not always the stripe size
Choose a hypothetical pattern size
•
Perform random reads at multiples of that distance
•
Repeat for a range of pattern sizes
•
Cluster results and identify actual pattern size
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
2 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
4 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
6 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
8 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
10 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
12 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
14 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
16 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
18 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
20 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
22 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
24 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
26 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
28 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
30 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
32 KB
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Pattern Size Example
RAID-0
4 Disks
8 KB
Chunks
Actual
32 KB
cluster
cluster
cluster
Shear Algorithm
•
Pattern size
•
Chunk size
•
Layout of chunks to disks
•
Level of redundancy
Determining the Chunk Size
•
Chunk size
•
amount of data contiguously allocated to one disk
•
Find the boundaries between disks
•
Choose a hypothetical boundary offset
•
Perform random reads on both sides of that offset
•
Repeat for all offsets in the pattern size
•
Cluster results and identify actual chunk size
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
0 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
2 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
4 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
6 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
8 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
10 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
12 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
14 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Testing
16 KB
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Chunk Size Example
RAID-0
4 Disks
8 KB
Chunks
Actual
8 KB
cluster
cluster
Shear Algorithm
•
Pattern size
•
Chunk size
•
Layout of chunks to disks
•
Level of redundancy
Determining the Read Layout
•
Find mapping of chunks to disks
•
Choose a pair of chunks in the pattern
•
Perform random reads to both chunks
•
Repeat for all pairs of chunks
•
Cluster results and identify chunks on same disk
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 0 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 1 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 2 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 3 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 4 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 5 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 6 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 0, 7 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 1 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 2 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 3 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 4 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 5 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 6 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
Testing { 1, 7 }
3
4
Read Layout Example
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
3
4
Read Layout Example
cluster
RAID-0 ZIG-ZAG 4 Disks
0
7
1
6
2
5
3
4
Actual
{ 0, 7 } { 1, 6 } { 2, 5 } { 3, 4}
cluster
Shear Algorithm
•
Pattern size
•
Chunk size
•
Layout of chunks to disks
•
Level of redundancy
Determining Level of Redundancy
•
Ratio of read to write bandwidth reveals the type
of redundancy in the array
•
Expected R/W ratios:
•
RAID-0:
1
(no redundancy)
•
RAID-1:
2
(mirroring)
•
RAID-4:
•
RAID-5:
varies (examine write layout)
4
(parity)
Shear Experience
•
•
Shear has been applied to
–
RAID-0
–
RAID-4
–
RAID-1
–
RAID-5
–
Chained Declustering
–
P+Q
Linux software RAID
•
•
Poor RAID-5 parity updates
Adaptec hardware RAID controller
•
Implements RAID-5 left-asymmetric layout
Outline
•
•
Introduction
Shear
•
•
•
Case Studies
•
•
•
Background
Algorithm
Performance: Stripe-aligned Writes
Management: Detecting Misconfiguration, Failure
Conclusion
RAID-5 Performance
•
Small writes on RAID-5 are problematic
•
•
Require two reads, parity calculation, two writes
Writing in full stripes is more efficient
RAID-5
0 1 2 3
16171819
32333435
P P P P
4 5 6 7
20212223
P P P P
36373839
8 9 1011
P P P P
24252627
40414243
P P P P
12131415
28293031
44454647
Stripe-aligned Writes
•
Overcome RAID-5 small write problem
•
Modified Linux disk scheduler
•
•
Groups writes into full stripes
•
Aligns writes along stripe boundaries
•
Approximately 20 lines of code
Experiment
•
Hardware RAID-5, 4 disks, 16 KB chunks
•
Create 100 files of varying sizes
Stripe-aligned Writes Experiment
•
Simple modification has a large impact
Detecting Misconfigurations
RAID 5-LA
RAID 5-RS
RAID 5-RA
Correct
RAID 5-LS
•
Software RAID, 4 Disks, 8 KB Chunks
•
What if one disk is accidentally used twice?
Detecting Misconfigurations
Misconfig
Correct
RAID 5-LS
RAID 5-LA
RAID 5-RS
RAID 5-RA
Detecting Failures
Software RAID
• RAID-5 LS
• 10 disks
• 8 KB chunks
•
Detecting Failures
Software RAID
• RAID-5 LS
• 10 disks
• 8 KB chunks
•
•
Disk 5 fails
Outline
•
•
Introduction
Shear
•
•
•
Case Studies
•
•
•
Background
Algorithm
Performance: Stripe-aligned Writes
Management: Detecting Misconfiguration, Failure
Conclusion
Conclusion
•
Gray-box research
•
•
Extract / exploit information from existing interfaces
Shear
•
Extracts information
•
•
Automatically determines storage array properties
Exploits information
•
File system performance tuning
•
Storage management
Questions?
http://www.cs.wisc.edu/adsl/