PARAID: A Gear-Shifting
Power-Aware RAID
Charles Weddle, Mathew Oldham, Jin Qian, An-I Andy Wang – Florida St. University
Peter Reiher – University of California, Los Angeles
Geoff Kuenning – Harvey Mudd College
1
Motivation

Energy costs are rising

An increasing concern for servers


Energy consumption of disk drives




No longer limited to laptops
24% of the power usage in web servers
27% of electricity cost for data centers
More energy  more heat  more cooling  lower
computational density  more space  more costs
Is it possible to reduce energy consumption
without degrading performance while
maintaining reliability?
PARAID: A Gear-Shifting Power-Aware RAID
2
Challenges

Energy


Performance


Not enough opportunities to spin down RAIDs
Essential for peak loads
Reliability

Server-class drives are not designed for
frequent power switching
PARAID: A Gear-Shifting Power-Aware RAID
3
Existing Work

Most trade performance for energy
savings directly


e.g. vary speed of disks
Most are simulated results
PARAID: A Gear-Shifting Power-Aware RAID
4
Observations

RAID is configured for peak performance


Unused storage capacity



RAID keeps all drives spinning for light loads
Over-provision of storage capacity
Unused storage can be traded for energy savings
Fluctuating load


Cyclic fluctuation of loads
Infrequent on-off power transitions can be effective
PARAID: A Gear-Shifting Power-Aware RAID
5
Performance vs. Energy
Optimizations

Performance benefits


Realized under heavy loads
Energy benefits

Realized instantaneously
6
Power-Aware RAID

Skewed striping for energy savings

Preserving peak performance

Maintaining reliability

Evaluation

Conclusion

Questions
PARAID: A Gear-Shifting Power-Aware RAID
7
Skewed Striping for Energy Saving

Use over-provisioned spare storage

Organized into hierarchical overlapping subsets
1
2
3
4
5
RAID
PARAID: A Gear-Shifting Power-Aware RAID
8
Skewed Striping for Energy Saving

Each set analogous to gears in automobiles
1
2
3
4
5
RAID
Gears
1
2
3
PARAID: A Gear-Shifting Power-Aware RAID
9
Skewed Striping for Energy Saving

Soft states can be reclaimed for space

Persist across reboots
1
2
3
4
5
Soft
States
RAID
Gears
1
2
3
PARAID: A Gear-Shifting Power-Aware RAID
10
Skewed Striping for Energy Saving


Operate in gear 1
Disks 4 and 5 are powered off
1
2
3
4
5
Soft
States
RAID
Gears
1
2
3
PARAID: A Gear-Shifting Power-Aware RAID
11
Skewed Striping for Energy Saving


Approximate the workload
Gear shift into most appropriate gear

Minimize the opportunity lost to save power
Conventional RAID PARAID
Energy
( Powered
On Disks )
workload
Workload
( Disk Parallelism )
PARAID: A Gear-Shifting Power-Aware RAID
12
Skewed Striping for Energy Saving


Adapt to cyclic fluctuating workload
Gear shift when gear utilization threshold is met
load
utilization threshold
gear shift
PARAID: A Gear-Shifting Power-Aware RAID
time
13
Preserving Peak Performance

Operate in the highest gear



Maximize parallelism within each gear



When the system demands peak performance
Uses the same disk layout
Load is balanced
Uniform striping pattern
Delay block replication until gear shifts

Capture block writes
PARAID: A Gear-Shifting Power-Aware RAID
14
Maintaining Reliability

Reuse existing RAID levels (RAID-5)


Also used in various gears
Drives have a limited number of power
cycles

Ration number of power cycles
PARAID: A Gear-Shifting Power-Aware RAID
15
Maintaining Reliability


Busy disk stay powered on, idle disks stay powered off
Outside disks are role exchanged with middle disks
role exchange
Disk 1
Disk 2
busy
disks
Gear 1
Disk 3
Disk 4
power
cycled
disks
Disk 5
Disk 6
idle
disks
Gear 2
Gear 3
PARAID: A Gear-Shifting Power-Aware RAID
16
Logical Component Design
User space
Admin tool
Linux kernel
File system
RAID
Reliability manager
PARAID block mapping
Load monitor
Soft RAID
Gear manager
Disk device driver
PARAID: A Gear-Shifting Power-Aware RAID
17
Data Layout


Resembles the data flow of RAID 1+0
Parity for 5 disks does not work for 4 disks

For example, replicated block 12 on disk 3
Gear 1
RAID-5
Gear 2
RAID-5
Disk 1
Disk 2
Disk 3
Disk 4
(1-4)
8
12
((1-4),8,12)
16
20
(16,20,_)
_
1
2
3
4
(1-4)
5
6
7
(5-8)
8
9
10
(9-12)
11
12
13
(13-16)
14
15
16
(17-20)
17
18
19
20
PARAID: A Gear-Shifting Power-Aware RAID
Disk 5
18
Data Layout

Cascading parity updates

For example, updating block 8 on disk 5
Gear 1
RAID-5
Gear 2
RAID-5
Disk 1
Disk 2
Disk 3
Disk 4
(1-4)
8
12
((1-4),8,12)
16
20
(16,20,_)
_
1
2
3
4
(1-4)
5
6
7
(5-8)
8
9
10
(9-12)
11
12
13
(13-16)
14
15
16
(17-20)
17
18
19
20
PARAID: A Gear-Shifting Power-Aware RAID
Disk 5
19
Update Propagation

Up-shift propagation (e.g. shifting from 3 to
5 disks)
Full synchronization
 On-demand synchronization



Need to respect block dependency
Downshift propagation

Full synchronization
PARAID: A Gear-Shifting Power-Aware RAID
20
Asymmetric Gear-Shifting Policies

Up-shift (aggressive)


Moving utilization average + moving standard
deviation > utilization threshold
Downshift (conservative)


Modified utilization moving average + moving
standard deviation < utilization threshold
Moving average modified to account for fewer drives
and extra parity updates
PARAID: A Gear-Shifting Power-Aware RAID
21
Implementation

Prototyped in Linux 2.6.5




Open source, software RAID
Implemented block I/O handler, monitor,
disk manager
Implemented user admin tool to configure
device
Updated Raid Tools to recognize PARAID
level
PARAID: A Gear-Shifting Power-Aware RAID
22
Evaluation

Challenges
Prototyping PARAID
 Commercial machines
 Conceptual barriers
 Benchmarks designed to measure peak
performance
 Trace replay
 Time consuming

PARAID: A Gear-Shifting Power-Aware RAID
23
Evaluation

Measurement framework
client
RAID
USB cable
P4 2.8 Ghz, 1 GB RAM
160 GB 7200 RPM SATA
RAID
crossover
cable
RAID
multimeter
RAID
server
Xeon 2.8 Ghz, 512 MB RAM
36.7 GB 15k RPM SCSI
power
measurement
probes
RAID
BOOT
SCSI
cable
PARAID: A Gear-Shifting Power-Aware RAID
power
supply
12v & 5v
power lines
24
Evaluation

Three different workloads using two different
RAID settings

Web trace - RAID level 0 (2-disk gear 1, 5-disk gear 2)


Cello99 - RAID level 5 (3-disk gear 1, 5-disk gear 2)


I/O-intensive workload with writes
PostMark - RAID level 5


Mostly read activity
Measure peak performance and gear shifting overhead
Speed up trace playback


To match hardware
Explore range of speed up factors and power savings
PARAID: A Gear-Shifting Power-Aware RAID
25
Web Trace



UCLA CS Dept Web Servers (8/11/2006 – 8/14/2006)
File system: ~32 GB (~500k files)
Trace replay: ~95k requests with ~4 GB data (~260 MB unique)
0.6
0.5
0.4
GB/hour 0.3
0.2
0.1
0
1
6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
hours
PARAID: A Gear-Shifting Power-Aware RAID
26
Web Trace Power Savings
64x – 60 requests/sec
60
Energy Savings
50
40
RAID-0
watts 30
64x - 34%
PARAID-0
20
128x - 28%
10
256x - 10%
0
0
5
10
15
20
25
30
hours
128x – 120 requests/sec
256x – 240 requests/sec
60
60
50
50
40
40
RAID-0
watts 30
PARAID-0
20
20
10
10
0
RAID-0
watts 30
PARAID-0
0
0
5
10
15
20
25
30
hours
PARAID: A Gear-Shifting Power-Aware RAID
0
5
10
15
20
25
30
hours
27
Web Trace Latency
256x
1
Overhead
0.8
0.6
RAID-0
0.4
PARAID-0
256x - within 2.7%
64x - 240%
80ms vs. 33ms
0.2
0
1
10
100
1000
10000
100000
msec
128x
64x
1
1
0.8
0.8
0.6
RAID-0
0.6
RAID-0
0.4
PARAID-0
0.4
PARAID-0
0.2
0.2
0
0
1
10
100
1000
10000
100000
msec
PARAID: A Gear-Shifting Power-Aware RAID
1
10
100
1000
10000
100000
msec
28
Web Trace Bandwidth
256x
180
160
140
Overhead
120
MB/sec
256x - within
1.3% in high
gear
RAID-0
100
80
PARAID-0
60
40
20
0
0
5
10
15
20
25
30
hours
128x
64x
180
180
130
130
RAID-0
MB/sec 80
PARAID-0
PARAID-0
30
30
-20 0
RAID-0
MB/sec 80
5
10
15
20
25
30
hours
PARAID: A Gear-Shifting Power-Aware RAID
-20 0
5
10
15
20
25
30
hours
29
Cello99 Trace

Cello99 Workload




HP Storage Research Labs
50 hours beginning on 9/12/1999
1.5 million requests (12 GB) to 440MB of unique blocks
I/O-intensive with 42% writes
PARAID: A Gear-Shifting Power-Aware RAID
30
Cello99 Power Savings
32x – 270 requests/sec
50
Energy Savings
40
30
RAID-5
20
PARAID-5
watts
32x - 13%
64x - 8.2%
10
128x - 3.5%
0
0
10
20
30
40
50
hours
64x – 550 requests/sec
128x – 1000 requests/sec
50
45
40
35
30
watts 25
20
15
10
5
0
50
40
RAID-5
PARAID-5
30
RAID-5
20
PARAID-5
watts
10
0
0
10
20
30
40
50
hours
PARAID: A Gear-Shifting Power-Aware RAID
0
10
20
30
40
50
hours
31
Cello99 Completion Time
128x
1
Overhead
0.98
0.96
RAID-5
0.94
PARAID-5
0.92
0.9
1
10
100
1000
10000
100000
32x - 1.8ms,
26% slower
due to time
spent in low
gear
msec
64x
32x
1
1
0.98
0.98
0.96
RAID-5
0.96
RAID-5
0.94
PARAID-5
0.94
PARAID-5
0.92
0.92
0.9
0.9
1
10
100
1000
10000
100000
msec
PARAID: A Gear-Shifting Power-Aware RAID
1
10
100
1000
10000
100000
msec
32
Cello99 Bandwidth
128x
1000
Overhead
100
RAID-5
MB/sec
PARAID-5
10
< 1% degradation during
peak hours
1
0
500000
1000000
1500000
request number
64x
32x
1000
1000
100
100
RAID-5
MB/sec
PARAID-5
10
RAID-5
MB/sec
PARAID
10
1
1
0
500000
1000000
1500000
requests
PARAID: A Gear-Shifting Power-Aware RAID
0
500000
1000000
1500000
request number
33
PostMark Benchmark



Popular synthetic benchmark
Generates ISP-style workloads
Stresses peak read/write performance of storage
device
PARAID: A Gear-Shifting Power-Aware RAID
34
Postmark Performance
200
150
seconds 100
50
0
1K files, 50K trans
RAID-5
20K files, 50K trans
PARAID-5 high gear
PARAID: A Gear-Shifting Power-Aware RAID
20K files, 100K
trans
PARAID-5 low-gear
35
Postmark Power Savings
80
70
60
watts
50
R A ID 5
40
PA R A ID
30
20
10
0
1
11
21
31
41
51
61
71
81
91
10
111
12
13
14
151
16
171
seconds
PARAID: A Gear-Shifting Power-Aware RAID
36
Related Work
Pergamum
 EERAID
 RIMAC
 Hibernator
 MAID
 PDC
 BlueFS

PARAID: A Gear-Shifting Power-Aware RAID
37
Ongoing Work





Try more workloads
Optimize PARAID gear configuration
Explore asynchronous update propagation
Speed up recovery
Live testing
PARAID: A Gear-Shifting Power-Aware RAID
38
Lessons Learned





Third version of design, early design too
complicated
Data alignment problems
Difficult to measure system under normal load
Hard to predict workload transformations due to
complex system optimizations
Challenging to match trace environments
PARAID: A Gear-Shifting Power-Aware RAID
39
Conclusion

PARAID reuses standard RAID-levels without
special hardware while decreasing their energy
use by 34%.


Optimized version can save even more energy
Empirical evaluation important
PARAID: A Gear-Shifting Power-Aware RAID
40
PARAID Recovery

2.7 times slower than conventional raid

For example, 2 gear PARAID device




First, the soft state must recover
Second, data must be propagated
Third, conventional raid must recover
Recovery not as bad for read intensive
workloads
41
PARAID Gear-Shifting
Web Trace Gear-Shifting Stats
256x
128x
64x
Number of gear switches
15.2
8.0
2.0
% time spent in low gear
52%
88%
98%
% extra I/Os for update
propagations
0.63%
0.37%
0.21%
Cello99 Gear-Shifting Stats
128x
64x
32x
Number of gear switches
6.0
5.6
5.4
% time spent in low gear
47%
74%
88%
% extra I/Os for update
propagations
8.0%
15%
21%
42