Storage: Isolation and Fairness
Lecture 24
Aditya Akella
Performance Isolation and Fairness in MultiTenant Cloud Storage, D. Shue, M. Freedman
and A. Shaikh, OSDI 2012.
Setting: Shared Storage in the Cloud
Y
Z
Y
F
Z
T
T
Shared
S3 Key-Value
EBS Storage
SQS
3
F
Predictable Performance is Hard
Z
Z
Z
Y
Y
Y
Y
T
T
F
F
F
F
F
DD
DD
DDStorage
DD
Shared Key-Value
Multiple co-located tenants ⇒ resource contention
4
Predictable Performance is Hard
Z
Z
Z
Y
Y
Y
Y
T
T
F
F
F
F
F
Fair queuing
DD
DD
DDStorage
DD
Shared Key-Value
@ big iron
Multiple co-located tenants ⇒ resource contention
5
Predictable Performance is Hard
Z
Z
Z
Y
Y
SS
Y
Y
SS
T
T
SS
F
F
F
F
F
SS
Multiple co-located tenants ⇒ resource contention
Distributed system ⇒ distributed resource allocation
6
popularity
Predictable Performance is Hard
data partition
Z keyspace Y keyspace T keyspace
F keyspace
Z
F
Z
Z
Y
Y
SS
Y
Y
SS
T
T
SS
F
F
F
F
SS
Multiple co-located tenants ⇒ resource contention
Distributed system ⇒ distributed resource allocation
7
Predictable Performance is Hard
GET 10B
(small reads)
GET 1kB
(large reads)
Z
Z
Z
Y
Y
SS
Y
SET 1kB
(large writes)
Y
SS
T
T
SS
SET 10B
(small writes)
F
F
F
F
F
SS
Multiple co-located tenants ⇒ resource contention
Distributed system ⇒ distributed resource allocation
Skewed object popularity ⇒ variable per-node demand
Disparate workloads ⇒ different bottleneck resources
8
Tenants Want System-wide Resource Guarantees
80 kreq/s
120 kreq/s
Zynga
Z
Z
Yelp
Z
Y
TP
Y
Y
160 kreq/s
40 kreq/s
Y
T
Foursquare
T
F
F
F
F
F
Shared Key-Value Storage
SS
SS
SS
SS
Multiple co-located tenants ⇒ resource contention
Distributed system ⇒ distributed resource allocation
Skewed object popularity ⇒ variable per-node demand
Disparate workloads ⇒ different bottleneck resources
9
Pisces Provides Weighted Fair-shares
wz = 20%
wy = 30%
Zynga
Z
Z
wt = 10%
Yelp
Z
Y
TP
Y
Y
Y
T
wf = 40%
Foursquare
T
F
F
F
F
F
Shared Key-Value Storage
SS
SS
SS
SS
Multiple co-located tenants ⇒ resource contention
Distributed system ⇒ distributed resource allocation
Skewed object popularity ⇒ variable per-node demand
Disparate workloads ⇒ different bottleneck resources
10
Pisces: Predictable Shared Cloud
Storage
• Pisces
– Per-tenant max-min fair shares of system-wide
resources ~ min guarantees, high utilization
– Arbitrary object popularity
– Different resource bottlenecks
• Amazon DynamoDB
– Per-tenant provisioned rates
~ rate limited, non-work conserving
– Uniform object popularity
– Single resource (1kB requests)
11
Predictable Multi-Tenant Key-Value
Storage
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
GET 1101100
Controller
RR
RS
FQ
12
Predictable Multi-Tenant Key-Value
Storage
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
GET 1101100
Controller
RR
WA WA2 WB2
RS
FQ
WA1 WB1
13
WA2 WB2
Strawman: Place Partitions Randomly
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Controller
RR
WA WA2 WB2
RS
FQ
14
Strawman: Place Partitions Randomly
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Controller
Overloaded
RR
WA WA2 WB2
RS
FQ
15
Pisces: Place Partitions By Fairness Constraints
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Collect per-partition
tenant demand
Controller
RR
WA WA2 WB2
RS
FQ
Bin-pack partitions
16
Pisces: Place Partitions By Fairness Constraints
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Controller
RR
WA WA2 WB2
RS
FQ
17
Results in feasible partition placement
Strawman: Allocate Local Weights Evenly
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Controller
Overloaded
RR
WA WA2 WB2
RS
FQ
WA1 = WB1
18
WA2 = WB2
Pisces: Allocate Local Weights By Tenant Demand
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
Compute per-tenant
max
mismatch +/- mismatch
Controller
RR
WA WA2 WB2
RS
FQ
WA1W>A1W= B1
WB1
19
A←B
WA2W<A2W= B2
WB2
Reciprocal weight swap
A→B
Strawman: Select Replicas Evenly
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
GET 1101100
50%
RR
Controller
50%
WA WA2 WB2
RS
FQ
WA1 > WB1
20
WA2 < WB2
Pisces: Select Replicas By Local Weight
WeightA
WeightB
Tenant A
Tenant B
VM VM VM
VM VM VM
PP
GET 1101100
detect weight
mismatch by
request latency
60%
50%
RR
Controller
50%
40%
WA WA2 WB2
RS
FQ
WA1 > WB1
21
WA2 < WB2
Strawman: Queue Tenants By Single
Resource
Tenant A
Tenant B
VM VM VM
VM VM VM
Bandwidth limited
Request Limited
Controller
bottleneck resource
(out bytes) fair share
RR
out req out req
22
WA WA2 WB2
RS
GET 0100111
GET 1101100
WA1 > WB1
PP
FQ
WA2 < WB2
Pisces: Queue Tenants By Dominant
Resource
Tenant A
Tenant B
VM VM VM
VM VM VM
Bandwidth limited
Request Limited
Controller
dominant resource
fair share
RR
PP
WA WA2 WB2
RS
FQ
Track per-tenant
resource vector
WA2 < WB2
out req out req
23
SS
SS
24
WA WA2 WB2
FAST-TCP
basedreplica
selection
RS
DRR tokenbasedDRFQ
scheduler
FQ
Weight Allocations
Controller
RR
PP
weight exchange
...
RR
local
...
System Visibility
global
Pisces Mechanisms Solve For Global
Fairness
Maximum bottleneck flow
fairness and capacity
constraints
Replica Selection
Policies
microseconds
seconds
Timescale
minutes
Issues
• RS < -- > consistency
– Turn off RS. Why does it not matter?
• What happens to previous picture?
• Translating SLAs to weights
– How are SLAs specified?
• Overly complex mechanisms: weight swap and FAST-based replica
selection?
• Why make it distributed?
– Paper argues configuration is not scalable, but requires server
configuration for FQ anyway
• What happens in a congested network?
• Assumes stable tenant workloads  reasonable?
• Is DynamoDB bad?
Pisces Summary
• Pisces Contributions
– Per-tenant weighted max-min fair shares of systemwide resources w/ high utilization
– Arbitrary object distributions
– Different resource bottlenecks
– Novel decomposition into 4 complementary
mechanisms
PP
26
Partition
Placement
WA
Weight
Allocation
RS
Replica
Selection
FQ
Fair
Queuing
Evaluation
Evaluation
• Does Pisces achieve (even) system-wide fairness?
- Is each Pisces mechanism necessary for fairness?
- What is the overhead of using Pisces?
• Does Pisces handle mixed workloads?
• Does Pisces provide weighted system-wide fairness?
• Does Pisces provide local dominant resource fairness?
• Does Pisces handle dynamic demand?
• Does Pisces adapt to changes in object popularity?
28
Evaluation
• Does Pisces achieve (even) system-wide fairness?
- Is each Pisces mechanism necessary for fairness?
- What is the overhead of using Pisces?
• Does Pisces handle mixed workloads?
• Does Pisces provide weighted system-wide fairness?
• Does Pisces provide local dominant resource fairness?
• Does Pisces handle dynamic demand?
• Does Pisces adapt to changes in object popularity?
29
Pisces Achieves System-wide Pertenant Fairness
Ideal fair share: 110 kreq/s (1kB requests)
Unmodified Membase
0.57 MMR
8 Tenants - 8 Client - 8 Storage Nodes
Zipfian object popularity distribution
Min-Max Ratio: min rate/max rate (0,1]
Pisces
0.98 MMR
Each Pisces Mechanism Contributes to
System-wide Fairness and Isolation
Unmodified Membase
0.57 MMR
FQ
FQ
PP
WA
FQ
PP
RS
WA
0.59 MMR
0.64 MMR 0.93 MMR
0.90 MMR 0.98 MMR
0.58 MMR
0.74 MMR 0.96 MMR
0.89 MMR 0.97 MMR
2x vs 1x demand
0.36 MMR
31
Pisces Imposes Low-overhead
> 19%
< 5%
32
Pisces Achieves System-wide Weighted
Fairness
0.98 MMR
0.89 MMR
4 heavy hitters
20 moderate demand
0.91 MMR
40 low demand
0.91 MMR
0.56 MMR
33
Pisces Achieves Dominant Resource
Fairness
Bandwidth (Mb/s)
76% of bandwidth
Time (s)
34
10B workload
request limited
GET Requests (kreq/s)
1kB workload
bandwidth limited
76% of request rate
24% of request rate
Time (s)
Pisces Adapts to Dynamic Demand
Tenant
Demand
Constant
Diurnal (2x wt)
Bursty
even
~2x
35