advertisement

Dynamic Scaling of Virtual Clusters with Bandwidth Guarantee in Cloud Datacenters Lei Yu Zhipeng Cai Department of Computer Science, Georgia State University, Atlanta, Georgia Email: [email protected], [email protected] Abstract—Network virtualization with bandwidth guarantee is essential for the performance predictability of cloud applications because of the shared multi-tenant nature of the cloud. Several virtual network abstractions have been proposed for the tenants to specify and reserve their virtual clusters with bandwidth guarantee. However, they require pre-determined ﬁxed cluster size and bandwidth, and do not support the scaling of the cluster in size and bandwidth requirements. On the other hand, the existing works on virtual cluster scaling focus on dynamically adjusting the cluster size without considering any bandwidth guarantee targeted by current network abstractions. To ﬁll the gap, this paper considers the problem of scaling up a virtual network abstraction with bandwidth guarantee. Eﬃcient algorithms are proposed to ﬁnd the valid allocation for the scaled cluster abstraction with optimization on the VM locality of the cluster. We also point out the case that a virtual cluster cannot be scaled without changing its original VM placement, and propose an optimal allocation algorithm that exploits the VM migration to address this issue while minimizing the total migration cost for the virtual cluster scaling. Extensive simulations demonstrate the eﬀectiveness and eﬃciency of our algorithms. I. Introduction With modern virtualization technologies, cloud computing oﬀers infrastructure as a service that provisions virtualized computing resources over the internet with pay-as-you-go pricing model, such as Amazon EC2 [1]. The resource virtualization in the cloud datacenter enables eﬃcient resource sharing and multiplexing across multiple tenants as well as on-demand resource provision. However, a major concern for moving applications to the cloud is the lack of performance guarantee in the cloud. The applications of diﬀerent tenants compete for the shared datacenter network resources, and no bandwidth guarantee in toady’s public cloud platforms causes them to suﬀer from unpredictable performance [2]. To eﬃciently virtualize the network resource with bandwidth guarantee in the cloud, several virtual network abstractions [3]–[7] have been proposed, which are used to express the network requirements of the tenants. In these abstractions, a tenant can specify the number of virtual machines (VMs), the topology of the virtualized network, and bandwidth requirements between VMs. Most of these abstractions are variants of the hose model, in which all VMs are connected to a central virtual switch by a bidirectional link with the speciﬁed bandwidth, like Oktopus [4], TIVC [5] and SVC [6]. Other network abstractions, either specify bandwidth requirement between every pairs of VMs as virtual pipes [3], or are derived based on the true application communication structure [7]. All these abstractions not only provide a simple and accurate way for the tenants to specify their network demands, but also facilitate the network resource allocation for the datacenter. Various algorithms are proposed along with them for allocating VM and bandwidth resources in the datacenter to realize corresponding network abstractions. The resource virtualization of the datacenters provides elasticity [8], a key feature of the cloud, which enables capacity scaling up and down on demand for the cloud applications to adapt to the changes in workloads. It is achieved through ondemand dynamic resource scaling either at the VM level [9]– [12], namely, on the CPU, memory and I/O resources allocated to a VM, or at the cluster level [12]–[14], namely, on the number of VMs or VM types with diﬀerent capacities in the virtual cluster. Such resource scaling not only is an eﬃcient approach to maintain the performance of cloud applications under varying workloads but also improves the resource utilization of the datacenter. The existing works on virtual cluster scaling [12]–[14], however, overlook the network requirement targeted by virtual network abstractions. They focus on dynamically adjusting the number of VMs in a virtual cluster, but without maintaining any bandwidth guarantee along with the cluster scaling. On the other hand, existing virtual network abstractions [3]–[7] use pre-determined ﬁxed bandwidth and number of VMs, and they cannot support the elastic scaling of the cluster in size and bandwidth requirements. Given a virtual cluster that has been deployed in the datacenter, the shrinking of its size and bandwidth requirement can be trivially performed by releasing the unneeded VMs and bandwidths. However, its expansion is not trivial. The increase of cluster size and VM bandwidth in a virtual network abstraction is limited by available VM slots and network bandwidth in the datacenter. The cloud might not be able to accommodate the scaled network abstraction without changing its original VM placement. It is critical to properly and eﬃciently allocate/re-allocate the VMs for scaling while maintaining the bandwidth guarantee. To the best of our knowledge, no previous works have been proposed to address the scaling of virtual network abstractions. Therefore, we aim to ﬁll this gap in this paper. In this paper, we address the problem of scaling up a virtual network abstraction with bandwidth guarantee. We consider the common hose model based virtual cluster abstraction [4]– [6], denoted by < N, B >, in which N VMs are connected to a virtual switch with links each having speciﬁed bandwidth B. We focus on its scaling in size from < N, B > to < N , B > (N > N) with unchanged bandwidth B. We identify the challenging issues arising from such scaling, and propose eﬃcient allocation algorithms to determine the VM placement for the scaled cluster. Speciﬁcally, we ﬁrst propose a dynamic programming algorithm to search for the placement of N − N 2 additional VMs in the abstraction that can still ensure the bandwidth B for each VM through appropriately reserving bandwidth on network links. The algorithm is further improved to maximize the VM locality such that the newly added VMs can be put as closer as possible to the pre-existing VMs to reduce the communication latency and network overhead. Further more, we point out that a virtual cluster might not be scaled without making changes to the original VM placement and in this case the above algorithm cannot ﬁnd a valid placement. To address this issue, we exploit the VM migration and develop an optimal algorithm to allocate the scaled cluster that minimizes the total VM migration cost. Extensive simulations demonstrate the eﬀectiveness and eﬃciency of our algorithms. The results indicate that the scalability of virtual clusters is seriously hindered by their pre-existing VM placement but can be signiﬁcantly improved at small VM migration cost. Last but not the least, in our discussion part, we show that our algorithms can be easily used to solve other kinds of virtual cluster scaling, including the scaling from < N, B > to < N, B >(B > B) and from < N, B > to < N , B >(N > N, B > B). The rest of this paper is organized as follows. Section II introduces the related work. Section III describes the problem and presents our ﬁrst scaling allocation algorithm. Section IV and V present the scaling allocation algorithms with locality optimization, and with allowing VM migration while optimizing its total cost, respectively. Section VI shows our simulation results. Section VII presents our discussion and section VIII concludes the paper. II. Related Work A. Virtual Network Abstraction with Bandwidth Guarantee Several virtual network cluster abstractions [3]–[7] have been proposed for eﬃcient network resource allocation with bandwidth guarantee. Oktopus [4] proposes a virtual cluster abstraction based on the hose model, denoted by < N, B >, in which N VMs are connected to a central virtual switch by a bidirectional link with the speciﬁed bandwidth B. Based on that, a temporally interleaved virtual cluster model TIVC [5] is further proposed, which allows diﬀerent bandwidth speciﬁcations during diﬀerent time intervals to address the time-varying bandwidth requirements of many cloud applications. SVC [6] incorporates the information of the statistical distributions of bandwidth demand into the virtual cluster abstraction to address the demand uncertainty. CloudMirror [7] proposes a network abstraction that represents the true application communication structure and distinguish bandwidth requirements for diﬀerent components and their inter-communication in an application. Secondnet [3] proposes a virtual datacenter abstraction that describes the bandwidth requirements between each pair of VMs and uses a central controller to determine the ﬂow rate and the path for each VM-to-VM pair. To achieve bandwidth guaranteed virtual to physical mapping, various VM allocation algorithms and bandwidth enforcement mechanisms have been proposed along with these network abstractions. B. Elastic Resource Scaling A number of works have been proposed to dynamically scale the resources either at VM level or at cluster level or both TABLE I: Notations Notation C, C N, N Tv Mv T v [k] S v [k] mv Description The original cluster and the corresponding scaled cluster The original cluster size (the number of VMs) and new size (N > N) The subtree rooted at the vertex v The allocable #VM set of the vertex v The subtree consisting of the root v and its ﬁrst k child subtrees The set containing the number of VMs that can be allocated into the subtree T v [k] regardless of v’s uplink bandwidth constraint The number of VMs of the original cluster C located in the subtree T v to meet the performance requirement of cloud applications under workload changes. Padala et al. [9] proposed a feedback control system to dynamically allocate CPU and disk I/O resources to VMs based on an online model that captures the relationship between the allocation of resource shares and the application performance. Shen et al. [10] proposed a prediction-driven automatic resource scaling system, which uses adaptive estimation error correction for resource demand to minimize the impact of under-estimation error, and support multi-VM concurrent scaling with resolving scaling conﬂicts on the same PM by VM migration. Gong et al. [11] proposed a lightweight online prediction scheme to predict the resource usage of VMs in an application and perform resource scaling based on the prediction results. AGILE [13] is a resource scaling system at the cluster level, which dynamically adjusts the number of VMs assigned to a cloud application with live VM replication to maintain to meet the application’s service level objectives. Herodotou et al. [14] proposes an automatic system to determine the cluster size and VM instance type required to meet desired performance and cost for a given workload. Han et al. [12] proposed an approach that integrates both ﬁne-grained resource scaling at the VM level in addition to cluster size scaling. These cluster level scaling approaches do not consider the bandwidth guarantee targeted by the virtual network abstractions. III. Virtual Cluster Scaling A. Problem description In this paper, we consider the hose model based virtual cluster abstraction [4]–[6], which consists of N homogenous VMs, with bandwidth B for each VM, denoted by < N, B >. There are three types of problem instances for scaling up a virtual cluster < N, B >, which are either to increase the cluster size from N to N (N > N), or to increase the bandwidth from B to B (B > B), or to increase both. We mainly focus on the scaling of the abstraction < N, B > to < N , B >, and in Section VII we show that the algorithms proposed for it can also eﬃciently solve other two types of scaling problems. As previous works [4]–[6], we assume typical tree topology for the datacenter, from physical machines at level 0 to the root at level H. Each physical machine is divided into multiple slots where tenant VMs can be placed. For the clarity, Table I lists the notations used in this paper. B. Scaling Allocation Algorithm Suppose that a virtual cluster C < N, B > has been deployed in a datacenter. Considering a link L in the network, it con- 3 ο ൌ ͵ ο ൌ ͳ RL= 100 RL= 100 L L 2 VMs 1 VM Bandwidth reservation on L : 100 3VMs 4 VMs Bandwidth reservation on L : 300 Fig. 1: This example shows the diﬀerence on the valid allocation between virtual cluster scaling and virtual cluster allocation. nects two separate network components in the tree topology. Suppose that the virtual cluster has m (N ≥ m ≥ 0) VMs in one component and hence N − m VMs in the other. Because each VM cannot send or receive at a rate more than B, the maximum bandwidth needed on link L is min{m, N − m} ∗ B. Therefore, the amount of bandwidth reservation for the virtual cluster on link L should be min{m, N −m}∗ B in order to guarantee access bandwidth B for each VM. When the size of the virtual cluster needs to be increased to N (N > N), Δ = N − N number of new VMs have to be allocated in the datacenter. Let the number of new VMs allocated to the two components connected by link L be ΔrL (ΔrL ≥ 0) and ΔlL (ΔlL ≥ 0), respectively. Then, the bandwidth reservation on link L should be increased to min{m + ΔrL , N − m + ΔlL } ∗ B, which obviously is not less than previous bandwidth reservation min{m, N − m} ∗ B. If min{m + ΔrL , N − m + ΔlL } > min{m, N − m}, residual bandwidth on link L has to be enough for the increase of the bandwidth reservation. The increment depends on the allocation of new VMs resulting in diﬀerent ΔrL and ΔlL . If L does not have enough bandwidth for the increment of bandwidth reservation, such allocation is not valid. Therefore, the problem of increasing the size of a virtual cluster can be solved through ﬁnding the valid allocation for Δ VMs to be added. We can regard these VMs as a complete new virtual cluster and search its valid allocation in the datacenter with a dynamic programming approach similar to the methods in TIVC [5] and SVC [6]. Let the residual bandwidth of link L be RL . The key diﬀerence compared to the allocation for a new virtual cluster is that, the validity of VM allocation across any link L should be determined by the following condition RL ≥ min{m + ΔrL , N − m + ΔlL } − min{m, N − m} ∗ B (1) rather than RL ≥ min{ΔrL , ΔlL }∗ B. These two conditions are not equivalent, exempliﬁed in Figure1. It shows a virtual cluster of size 3 having 2 VMs and 1 VM in two subtrees respectively before scaling. The link L has 100Mbps residual bandwidth. Suppose that the size is increased from 3 to 7, and ΔlL = 1 and ΔrL = 3 VMs are placed to two subtrees respectively. Considering new VMs as a separate virtual cluster, 100Mbps residual bandwidth on L is suﬃcient for such allocation. However, for scaling, the resulting allocation for the scaled virtual cluster requires 300Mbps bandwidth on link L and hence 200Mbps increment of bandwidth reservation for the virtual cluster, which cannot be satisﬁed by link L. Based on (1), we introduce the dynamic programming based search algorithm for allocating additional VMs for cluster scaling. The algorithm requires the information of the network topology, the number of empty slots in each physical machine (PM, for short), the residual bandwidth of each link, and the current placement of the virtual cluster to be scaled. The algorithm starts from the leaves (i.e., PMs) and traverses the tree topology level-by-level in bottom-up manner. The vertices at the same level are visited in left-to-right order. For each vertex v, the algorithm determines the number of VMs that can be allocated into the subtree rooted at v based on the recorded results for each of its children. The number of VMs that can be allocated into a subtree may be multiple. Thus, the algorithm computes an allocable #VM set for each vertex that contains all feasible numbers of VMs. Given a vertex v, its allocable #VM set is denoted by Mv . We ﬁrst introduce the computation process at the leaves and then describe the general procedure at a vertex that has children. Consider a virtual cluster C of size N and VM bandwidth B, and Δ number of new VMs to be added for scaling. (1) Suppose that v is a leaf at level 0, (i.e., a PM), which has cv number of empty slots and mv VMs from the virtual cluster C. The number of VMs that can be allocated in v, denoted by Δv , is constrained by both the available VM slots and the residual bandwidth on the link L connecting v to the upper level. Thus, the ﬁrst condition for the allocability is that Δv ≤ cv . If Δv VMs are placed at v, Δ − Δv VMs are placed at the other side of link L. As a result, the link L splits the scaled virtual cluster into two components each with Δv +mv VMs and N − mv + Δ − Δv VMs respectively. Accordingly, the bandwidth reservation required on link L is min{Δv +mv , N−mv +Δ−Δv }∗B. Based on (1), the second condition for the allocability is min{Δv +mv , N −mv +Δ−Δv }−min{mv , N −mv } ∗ B ≤ RL (2) where RL is the residual bandwidth of link L. To compute v’s allocable #VM set Mv , the algorithm checks the condition (2) for each number from 1 to min{Δ, cv }, and adds the number to Mv if it is true. (2) Suppose that v is a vertex that has n number of children, denoted by v1 , v2 , . . ., vn . The corresponding allocable #VM set of each child vk is Mvk . Similar to previous works [5], [6], the algorithm visits all of v’s children in left-to-right order. For each child vk , the algorithm iteratively computes a set, denoted by S v [k], that contains the number of VMs that can be allocated in the subtree consisting of v and the ﬁrst k child subtrees, without considering the uplink bandwidth constraint of v. Figure 2 shows the part counted by S v [k], computed as S v [k] = {m | m = a + b where a ∈ Mvk and b ∈ S v [k − 1]} (3) where S v [0] = 0. Finally, we can obtain S v [n]. Each number in S v [n] is a candidate for Δv , i.e, the number of VMs that can be allocated in the subtree T v rooted at v, regardless of the bandwidth constraint of v’s uplink that connects v to the upper level. Then, for each candidate, the algorithm checks its validity by verifying the condition (2), given RL being the residual bandwidth of v’s uplink and mv being the number of preexisting VMs of the virtual cluster C in T v . If the value passes the validation, it is added to Mv . During the search process, the algorithm memorizes the element a contributing to m in (3) for each m in S v [k]. Such information is then used by backtracking to ﬁnd the number 4 v v1 v2 vk vn ܶ௩భ ܶ௩మ ܶ௩ೖ ܶ௩ ܶ௩ ሾ݇ሿ Fig. 2: The computation of S v [k] is based on the subtree T v [k]. of VMs that should be allocated to vertex vk . The search continues until it ﬁnds a vertex v, of which Mv contains Δ, i.e., the number of VMs to be added. Such vertex is referred to as allocation point. After that, starting from Δ at the allocation point, a backtracking algorithm uses the memorized information to ﬁnd the number of VMs allocated to each subtree level-by-level in top-down manner. The backtracking procedure is the same as in previous work [5], [6] . Eventually, we can obtain the number of VMs to be added on each PM. The search algorithm has the time complexity of O(Δ2 |V|D) where |V| is the number of vertices in the network and D is the maximum number of children that any vertices have. We also note that this algorithm can be reduced to an algorithm for allocating a complete virtual cluster, similar to those in [5], [6], regarding Δ as the size of the virtual cluster and zero VMs for its existing allocation. IV. Scaling with Locality Optimization The previous algorithm adds new VMs to the datacenter without considering the existing placement of the virtual cluster. This may cause that the newly added VMs are distant from the locations of previously allocated VMs. However, good spatial locality is desired for the allocation of a virtual cluster because it not only reduces the communication latency among VMs but also can conserve the bandwidth of the links in the upper levels. In this section, we consider how to scale a virtual cluster with maximizing its VM spatial locality. We measure the spatial locality of a virtual cluster by the average distance (in hops) between any pair of VMs in the cluster. Let C be a virtual cluster of size N and C be the scaled C with larger size N , and VC , VC be their VM sets, respectively. The VM set added for the scaling can be represented by VC \ VC . The average VM-pair distance of C , denoted by avgdC , is calculated as avgdC 2 = h(i, j) N (N − 1) i, j∈V C ⎞ ⎛ ⎟⎟⎟ ⎜⎜⎜ 2 ⎜ ⎜⎜⎜ = h(i, j) + h(i, j)⎟⎟⎟⎠⎟ ⎝ N (N − 1) i, j∈V i∈V \V , j∈V C C C (4) C where h(i, j) is the number of hops between two physical machines where VM i and j are placed. Smaller avgdC indicates higher spatial locality. i, j∈VC h(i, j) is determined by the pre-existing placement of C. Thus, to minimize avgdC , we need to ﬁnd the optimal placement of the new VMs that minimize the total distance between each new VM and all the other VMs in C . A straightforward approach is to use the algorithm in the previous section to ﬁnd all the feasible allocations for the scaling, calculate the average VM pair distance of the scaled cluster for every allocation and choose the one with the minimum avgdC . However, the number of feasible allocations can be huge. By observing Formula (3), we can see that the diﬀerent combinations of a and b can result in the same value of m, like m = 3 for both (a = 1, b = 2) and (a = 2, b = 1). This indicates possibly many valid choices for determining how the VMs are allocated into each child subtree at every vertex. From an allocation point down to the physical machines, the combinations of diﬀerent choices at each vertex are multiplied level-by-level, each leading to diﬀerent spatial locality. Thus, this approach needs to examine all these combinations, which is ineﬃcient. Instead, we can show that the problem has the optimal substructure and can be eﬃciently solved by dynamic programming. Consider placing Δv new VMs to v’s subtree T v for scaling C. Let VT v be this new VM set and VTv be the VM set containing C’s pre-existing VMs in T v . Then, given a feasible allocation A, the sum of the distances between each new VM in VT v and all the other by dTv (Δv , A), VMs in VTv ∪VTv is denoted i.e., dTv (Δv , A) = i∈VT , j∈VT ∪VTv h(i, j). Let d∗ (T v , Δv ) be the v v minimum of dTv (Δv , A) among every feasible allocation A that places Δv VMs into T v . Now we consider a subtree that consists of the vertex v and its ﬁrst k child subtrees, as shown in dashed box in Figure 2, denoted by T v [k]. Similarly, given an allocation A that places e new VMs to T v [k], dTv [k] (e, A) represents the sum of the distances between each new VMs and all the other VMs of the scaled cluster in T v [k], and d∗ (T v [k], e) is the minimum one among all feasible allocations. Suppose that an allocation A assigns x new VMs to the k + 1-th child subtree T vk+1 and e new VMs to the subtree T v [k]. Then, it is easy to see that dTv [k+1] (e , A) is the sum of distances of the VM pairs from T v [k] and T vk+1 , and also the VM pairs between x VMs in T vk+1 and e VMs in T v [k]. If vertex v is at level l, the distance between any two VMs from v’s two diﬀerent child subtrees is 2l. Thus, the distance between any VM pair between T vk+1 and T v [k] is 2l. The total number of VM pairs between T vk+1 and T v [k] is x·(e+ ki=1 mvi ) where mvi is the number of C’s pre-existing VMs in its child subtrees T vi . Thus, we have dTv [k+1] (e , A) = dTv [k] (e, A) + dvk+1 (x, A) + 2 · level · x · (e + k mvi ) (5) i=1 For each element e ∈ S v [k + 1], according to (3), there can exist multiple combinations of the VM numbers that can be allocated to the subtrees from T v1 to T vk+1 to sum up to e . Based on the above equation (5), we can derive the following recursive equation: d∗ (T v [k + 1], e ) = ⎫ ⎧ k ⎪ ⎪ ⎪ ⎪ ⎬ ⎨ ∗ ∗ min ⎪ mvi )⎪ ⎪ ⎪d (T v [k], e) + d (T vk+1 , x) + 2 · level · x · (e + ⎭ (x,e)∈Ψ(e ) ⎩ d∗ (T v [1], e) = d∗ (T v1 , e) i=1 (6) where Ψ(e ) = {(x, e) | x + e == e , x ∈ Mvk+1 , e ∈ S v [k]}, the min is operated over all the pairs of x and e of which the sum is equal to e , x ∈ Mvk+1 and e ∈ S v [k]. For each x ∈ Mv , d∗ (T v , x) = d∗ (T v [n], x) where n is the number of v’s children. The above formula indicates the optimal substructure of our 5 problem, and based on that we can easily derive a dynamic programming algorithm to solve the problem. To scale a virtual cluster with bandwidth guarantee while achieving optimal locality, we combine the above algorithm with the algorithm for searching a valid allocation in the previous section. Actually, during the computation of the allocable #VM set Mv of a vertex v, we can simultaneously compute d∗ (T v , e) for each e ∈ Mv . Besides, note that there can be multiple allocation points in the tree and each may lead to diﬀerent average VM-pair distance for the scaled cluster. Thus, to ﬁnd the optimal allocation among all possible allocation points, the algorithm traverses the whole tree, discovers all the allocation points, and chooses the one with minimum d∗ (T v , Δ) where v is an allocation point and Δ = N − N. For clarity, the pseudo code of the complete algorithm is given in Algorithm 1. The dynamic programming for maximizing the locality is between Line 14∼23 and the backtracking information is recorded in Dv [x + e, i]. It is easy to see that Algorithm 1 has the same time complexity O(Δ2 |V|D) as the previous algorithm in Section III. Algorithm 1: Scaling Allocation Algorithm 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 V. Scaling with VM Migration The algorithm proposed in the previous section aims to properly allocate VMs to be added for scaling a virtual cluster. However, it may not be able to ﬁnd a feasible solution, even when the datacenter has resource to accommodate the virtual cluster with scaled size. Figure 3 shows an example where a network consists of four PMs, each with four VM slots and connected by 1Gbps links. Two virtual clusters VC1 and VC2 are deployed. VC1 has 5 VMs and the bandwidth for each is 500Mbps. VC2 has 2 VMs with bandwidth 200Mbps. On link L, the bandwidth reservation for VC1 is 500 and for VC2 is 400, and thus the residual bandwidth on L is 100. Suppose that a new VM needs to be added to scale VC2 up to 6 VMs. It can be easily veriﬁed that, no matter to which empty slot it is allocated, the scaled virtual cluster needs 600 bandwidth on link L. It requires the 200 increment of bandwidth reservation for VC2 on L, more than the 100 residual bandwidth of L. That is, no feasible allocations can be found for the new VM and thus VC2 cannot be scaled up. However, in Figure 3(a), it is obvious that the subtree T u2 can accommodate the whole virtual cluster VC2 with scaled size 6. This indicates that existing VM placement could hinder the scalability of a virtual cluster. To address this issue, we exploit VM migration, with which it is possible to change the layout of existing VMs to accommodate new VMs. As an example, in Figure 3(b), after a VM is migrated from T u1 to T u2 , VC2 can add another new VM in T u2 without increasing the bandwidth reservation on link L. Thus, in this section we consider VM allocation with allowing VM migration for scaling a virtual cluster. Because VM migration incurs VM service downtime [15] while performance isolation among diﬀerent virtual clusters is desired in cloud, we only consider migrating VMs from the virtual cluster that is to be scaled, rather than VMs in other virtual clusters, such that the scaling of a virtual cluster does not interrupt other virtual clusters. Since VM migration incurs signiﬁcant traﬃc and service downtime [15], [16], it is a must to minimize the VM migration cost for virtual cluster scaling. Thus, only if no proper 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 Input: Tree topology T , a virtual cluster C =< N, B > to be scaled to < N , B >, bandwidth allocation information on each link. Δ ← N − N ; AllocationpointS et ← ∅; for level l ← 0 to Height(T ) do for each vertex v at level l do n ← the number of v’s children; if l = 0 then // leaf v is a PM S v [0] ← {0, 1, . . . , min{cv , Δ}} ; // cv is the number of empty VM slots of v else S v [0] ← {0}; for i from 1 to n do S v [i] ← ∅; for each x ∈ Mvi do for each e ∈ S v [i − 1] do if x + e S v [i] then d∗ (T v [i], x + e) ← in f ; S v [i] ← S v [i] ∪ {x + e}; if i = 1 then d(T v [i], x + e) ← d∗ (T v1 , x) ; else d(T v [i], x + e) ← d∗ (T v [i − 1], e) + i−1 d∗ (T vi , x) + 2 · level · x · (e + k=1 mvk ) ; if d∗ (T v [i], x + e) > d(T v [i], x + e) then d∗ (T v [i], x + e) ← d(T v [i], x + e); Dv [x + e, i] ← x; Mv ← ∅ ; for each e ∈ S v [n] do if min{e+mv , N −mv +Δ−e}−min{mv , N −mv } ∗ B ≤ RLv then // RLv is the residual bandwidth of v’s uplink Mv = Mv ∪ {e} ; d∗ (T v , e) ← d∗ (T v [n], e); if Mv = ∅ then return false; if Δ ∈ Mv then AllocationpointS et ← AllocationpointS et ∪ (v, d∗ (T v , Δ), Dv ); if AllocationpointS et ∅ then Choose v that has the minimum d∗ (T v , Δ); Alloc (v, Δ, Dv ); return true; return false; Procedure Alloc(v, x, Dv ) if v is a machine then allocate x VMs into v; else for v’s child i from n to 1 do Alloc(vi , Dv [x, i]); Update bandwidth allocation information on v’s downlink to its i-th child; x = x − Dv [x, i]; allocation can be found for new VMs with keeping existing VM placement, we turn to the allocation approach with exploiting VM migration. The complete solution is described with answering “when to migrate” and “how to migrate”. A. When to migrate Algorithm 1 can ﬁnd a feasible allocation for new VMs if it exists. If the solution does not exist, the algorithm fails, which occurs in the following two cases: 6 u1 VC 1 VM 500Mbps VC 2 VM 200Mbps u2 Link capacity: 1Gbps L u1 u2 L (a) (b) Fig. 3: Figure (a) shows the case that no feasible allocations exist for scaling a virtual cluster. Figure (b) shows that the virtual cluster VC2 can be scaled with one VM migration. (a) The search reaches at the root, and the algorithm ﬁnds no vertices (including the root) in the tree of which the allocable #VM sets contain Δ, i.e., the number of VMs to be added. (b) During the search, if any vertex is found to have empty allocable #VM set, the solution does not exist and the algorithm terminates. Given a vertex v, its allocable #VM set Mv is generated by ﬁltering any elements that do not satisfy v’s uplink bandwidth constraint out from the candidate set ( i.e., S v [n] in Formula (3)). Mv = ∅ means that the uplink bandwidth constraint cannot be satisﬁed for any placement of new VMs in the tree. In this case, even placing zero VMs in v’s subtree T v is not a valid choice, which indicates an essential diﬀerence between placing a complete virtual cluster and scaling an existing cluster. Zero VMs in T v imply that all new VMs are placed outside of T v but additional bandwidth needs to be reserved for the traﬃc to and from existing VMs in T v . If any of the above cases happens, we terminate the previous algorithm with returning false at line 30 and 37 in Algorithm 1 and turn to VM allocation with migration. B. How to migrate In this section we propose our allocation approach that allows VM migration. We ﬁrst look into the following fact: Proposition 1: Assume that a virtual cluster C with size N is scaled up to C with larger size N in the datacenter DC. If a valid allocation can be found for the virtual cluster C in DC where the original cluster C and its corresponding bandwidth reservation are hypothetically removed, then C must be able to scale up to size N with allowing VM migration, and vice versa. This proposition is straightforward, since we can always migrate C and additional VMs to the slots where C is allocated hypothetically. If we cannot ﬁnd any solutions for C , C cannot be scaled up even with VM migration. Based on that, our allocation algorithm ﬁrst hypothetically removes the virtual cluster C and releases its bandwidth reservation, and ﬁnds valid allocations for the scaled cluster C , from which it derives the ﬁnal solution. This also guarantees that as long as the datacenter has enough resource to accommodate the scaled virtual cluster, our algorithm always ﬁnds a valid allocation and will not falsely reject a scaling request that can be satisﬁed. On the other hand, the allocation should minimize the migration of VMs considering its signiﬁcant overhead. To do that, our algorithm aims to ﬁnd the allocation that has maximum overlap with existing VM placement of the virtual cluster, which let the most VMs of C stay the same. We ﬁrst introduce the overlap measurement as follows: Deﬁnition 1 (Allocation overlap size): Given a set of PMs {P1 , P2 , . . . , Pl }, suppose that an allocation assigns ai VMs to each Pi , and another allocation assigns bi VMs to each Pi , then the overlap size between two allocations is l min{ai , bi } (7) i=1 For example, an allocation assigns 2, 3 and 1 VMs to P1 , P2 and P3 respectively, and another one assigns 1, 2 and 2 VMs to P1 , P2 and P3 . Their overlap size on each single machine is 1, 2, 1 respectively, and hence the overlap size between them with respect to all three machines is 4. If the latter allocation represents a virtual cluster C’s existing VM placement and the former one represents its VM placement after being scaled, then 4 VMs of C can stay the same and only one VM in P3 needs to be moved out. Next we describe how to ﬁnd the allocation having the largest overlap size with the original allocation of the cluster C and how to migrate. Because here we are considering the allocation of a whole cluster instead of only new VMs, for a vertex v, its allocable #VM set Mv is redeﬁned as the set containing the number of VMs out of C that can be allocated to v’s subtree T v . 1) Finding the allocation with the largest overlap size: For each element x ∈ Mv , there are one or multiple valid ways to allocate x VMs to the PMs in the subtree T v . Every allocation can have diﬀerent overlap size with the VM placement of original cluster C in T v . Let OTv [x] be the maximum overlap size among all these allocations which place x VMs in the subtree T v . Our algorithm calculates OTv [x] for each x ∈ Mv . The procedure is similar to that for computing the maximum locality in Section IV. Reminding that T v [k] is the subtree consisting of v and v’s ﬁrst k child subtrees rooted at v1 , . . . , vk , and v has total n child subtrees. Let OTv [k] [x] be the maximum overlap size among all the feasible allocations that place e VMs to T v [k] where e ∈ S v [k]. Then, for each element e ∈ S v [k + 1], OTv [k+1] [e ] = max OTv [k] [e] + OTvk+1 [x] (x,e)∈Ψ(e ) OTv[1] [e] = OTv1 [e] (8) where Ψ(e ) = {(x, e) | x + e == e , x ∈ Mvk+1 , e ∈ S v [k]}. Finally, for each element e ∈ S v [n], we have OTv [n] [e ]. If e satisﬁes the uplink bandwidth constraints, it is added into Mv and the corresponding OTv [n] [e ] is kept along with it, referred to as OTv [e ]. The algorithm starts the search from the leaf vertices (i.e., PMs). For a leaf u, T u is u itself. For each x ∈ Mu , OTu [x] is the smaller of x and the number of VMs that the cluster C originally has in the machine u. Then, level-by-level, for each vertex, the algorithm computes its allocable #VM set and the maximum overlap size associated with each element in the set. As opposed to the previous algorithm in Section III, the algorithm needs to ﬁnd an allocation point that not only has N in the allocable #VM set, but also should achieve the largest overlap size, because there can be multiple allocation points that have diﬀerent maximum overlap sizes associated with N . Take Figure 3 as an example. At the level of u1 and u2 , the algorithm can ﬁnd that u2 is an allocation point. However, the 7 Fig. 4: The allocation point with largest maximum overlap size can appear at the root. The boxes ﬁlled with backslash represent VMs from other virtual clusters. allocation point with largest overlap size 4, shown in Figure 3(b), actually exist in the subtree at the upper level. Let T rC be the lowest-level subtree that contains the original cluster C, rooted at rC . Let S C be the set that contains the vertices of T rC and the ancestors of rC . It is obvious that only the allocation points in S C may have allocations overlapping with C. Any allocation points not in S C must have zero maximum overlap size. To ﬁnd the optimal allocation that achieves the largest overlap size, the algorithm needs to determine all the allocation points in S C and their maximum overlap sizes. The reason to examine all the vertices in S C is that the optimal allocation point can be at any level. See Figure 4 for an example, in which 3 more VMs are added to extend a virtual cluster of 3 VMs (represented by blue box) shown in Figure 4(a). Two downlinks of u1 have residual bandwidth 200 and 100. We assume suﬃcient bandwidth on other links. The allocation point u1 with the allocation shown in Figure 4(b) has the maximum overlap size of one. But the allocation point u at the upper level achieves a higher maximum overlap size, two, as shown in Figure 4(c). This example simply indicates the algorithm has to traverse the tree until to the root to determine the optimal allocation point. Denote the set of the allocation points with non-zero maximum overlap size in S C by S Ca . S Ca = ∅ if either no allocation points exist in S C or all the allocation points in it have zero maximum overlap size. Then, if S Ca ∅, the algorithm chooses the one that has the largest maximum overlap size in S Ca as the ﬁnal allocation solution for C ; otherwise, the algorithm chooses the allocation point at the lowest level among all allocation points in the whole tree as the ﬁnal solution. Locality v.s. Overlap size The above algorithm aims to maximize the overlap size with the original cluster C, regardless of VM locality. Good locality implies that a virtual cluster needs to be placed in a subtree as lower as possible, but the largest overlap size may appear in a higher level. We can see that the allocation in Figure 4(c) achieves the largest overlap size but the VMs of C are distributed to diﬀerent subtrees T u1 and T u2 . In contrast, Figure 4(b) has smaller overlap size but higher locality with VMs being in the same subtree T u1 . To make a trade-oﬀ, we propose a simple solution that evaluates an allocation point based on the weighted sum of locality and overlap size. Given an allocation point v at level L in the tree of height H, let OTv be its maximum overlap size for C ’s allocation in its subtree, v’s goodness is measured by α OT v H−L + (1 − α) N H (9) where N is the size of original cluster C and 0 ≤ α ≤ 1. The weights can be determined by the cloud providers, and diﬀerent weights indicate diﬀerent preferences to either mitigating the migration cost or compacting the virtual cluster. If α = 0, the goal is to maximize the locality for the scaled cluster, regardless of the migration cost; if α = 1, the migration cost is the only focus. Based on that, we can adapt the above algorithm to choose the allocation point with the maximum goodness among all as the ﬁnal solution. The algorithm for ﬁnding optimal allocation of C is similar to Algorithm 1 in that the search is also based on dynamic programming but uses a diﬀerent goodness measurement and searches for the allocation of N VMs. It has the time complexity O(N 2 |V|D) . 2) Determining how to migrate VMs: Given the allocation for the cluster C , the next step is to decide how to migrate non-overlapped VMs from C to the slots allocated to C . Migrating a VM to diﬀerent PMs may go through routing paths of diﬀerent lengths and hen diﬀerent durations. To reduce the service downtime and network overhead, the mapping between VMs and slots should minimize the total number of hops of VM migrations. To this end, we transform it to the minimum weight perfect matching problem in bipartite graph [17] that is to ﬁnd a subset of edges with minimum total weight such that every vertex has exactly one edge incident on it. Suppose the datacenter has ND PMs. The allocation for C can be represented by {mi |1 ≤ i ≤ ND } which means the allocation assigns mi VMs to PM Pi . Similarly, the original allocation for C is {mi |1 ≤ i ≤ ND }. We construct the corresponding bipartite matching problem as follows. Let V and S be two sets of vertices representing VMs and slots respectively. Initially they are empty. For each PM Pi , we compute δi = mi − mi . If δi > 0, which means that δi VMs need to be migrated out from Pi , then δi vertices are added to V, each labeled with its PM identiﬁer Pi ; if δi < 0, which means that |δi | empty slots on Pi are allocated to C ’s VMs, then |δi | vertices are added to S , each also labeled with Pi . After that, Δ = N − N vertices are added to V to represent new VMs to be added for scaling, labeled with “New”. It is easy to prove that V and S have the same number of vertices, and no vertices in S have the same labels as the vertices in V. An edge is added for each pair of vertices u (u ∈ V) and v (v ∈ S ), and ﬁnally we obtain a complete bipartite graph. If u’s PM label is Pu and v’s is Pv , then the weight of edge (u, v) is the number of hops between Pu and Pv , i.e., the hops for migrating the VM at u to the slot at v. If u’s label is “New”, the edge between u and any vertex in S has zero weight because no migration is needed for newly added VMs. Based on the complete bipartite graph constructed above, we can use well-known Hungarian algorithm [17] to ﬁnd its minimum weight perfect matching where each edge indicates the assignment of a VM in V to a slot in S and the total number of hops for VM migrations is minimized with such assignment. Note that, since we assume that all VMs in a virtual cluster are homogenous, we do not distinguish C’s VMs that are placed in one PM from each other, nor distinguish the newly added VMs. That is, for the assignment (u, v), we can migrate any one of C’s VMs on Pu to v, or add any one of the newly added VMs to v if u’s label is “New”. In the worst case that the overlap size is zero, N VMs need to be migrated and the Hungarian algorithm takes O(N 3 ) time complexity. 8 VI. EVALUATION In this section we evaluate the eﬀectiveness and eﬃciency of our algorithms for scaling a virtual cluster by simulations. A. Simulation Setup Datacenter topology We simulate a datacenter of three-level tree topology with no path diversity. A rack consists of 10 machines each with 4 VM slots and a 1Gbps link to connect to a Top-of-Rack (ToR) switch. Every 10 ToR switches are connected to a level-2 aggregation switch and 5 aggregation switches are connected to the datacenter core switch. There are total 500 machines at level 0. The oversubscription of the physical network is 2, which means that the link bandwidth between a ToR switch and an aggregation switch is 5Gbps and the link bandwidth between an aggregation switch and the core switch is 25Gbps. Workload Our simulations are conducted under the scenario of dynamically arriving tenant jobs. A job speciﬁes a virtual cluster abstraction < N, B >. The jobs dynamically arrive over time, and if a job cannot be allocated upon its arrival, it is rejected. Each job runs for a random duration and its virtual cluster is removed from the datacenter when the job is completed. The number of VMs N in each virtual cluster request, i.e., the job size, is exponentially distributed around a mean of 20 at the default. The bandwidth B of each job is randomly chosen from [100, 200]. The job arrival follows a Poisson process with rate λ, then the load on a datacenter with M VMs is λN̄T c /M where N̄ is the mean job size and T c is the mean running time of all jobs. The running time of each job is randomly chosen from [200, 500]. During the job arrival process, we randomly choose a job running in the datacenter and generate a scaling request for it with a given probability Pr scal before the arrival of the next job. The size increment for a virtual cluster < N, B > is determined by a given percentage increase %inc, then the new size N = (%inc + 1)N. We simulate the arriving process of 500 tenant jobs with varying the load, the scaling probability Pr scal , and the percentage increase of the cluster size %inc. B. Simulation Results We evaluate and compare our scaling algorithms in terms of the rejection rate of the scaling requests, the locality of the scaled cluster, and the number of VM migrations for the scaling. The ﬁgures present the average across multiple runs. For brevity, we refer to the scaling algorithm in Section III as “Scaling”, the algorithm with locality optimization in Section IV as “Scaling-L” and the algorithm allowing VM migration in Section V as “Scaling-M”. 1) Rejection rate of the scaling requests: Figure 5 and 6 show the rejection rate with diﬀerent %inc under light load and heavy load settings where load = 0.2 and 0.6 respectively. In Figure 5, the rejection rates of Scaling-M are zero. Except that, the rejection rate increases with %inc for all three algorithms. It can be seen that Scaling-L can achieve lower rejection rate than Scaling under light load. The reason can be that the new VMs are put as closer as to the pre-existing placement with the locality optimization, which actually reduces the fragmentation and saves more continuous space to accommodate incoming job requests. But under the heavy load, limited resource is main bottleneck and locality optimization does not help, thus, Scaling-L and Scaling have close rejection rates in Figure 6. Compared with Scaling-M, both Scaling-L and Scaling have much higher rejection rates, up to 50% under the heady load. This indicates that the preexisting VM placement can seriously hinder the scalability of virtual clusters. However, with allowing VM migration for the scaling, Scaling-M can achieve much lower rejection rate. We also vary Pr scal to change the arrival rate of the scaling requests and show the change of the rejection rate in Figure 7. The rejection rate increases with Pr scal , and Scaling-M still has the lowest rejection rate. From the evaluation blow, we will see that Scaling-M achieves such low rejection rate only at much small VM migration cost. 2) Locality: Scaling-L searches for the allocation with the minimum average VM-pair distance for scaling a virtual cluster. To verify its eﬀectiveness, we consider how much improvement can be obtained from Scaling-L compared with Scaling. Using Scaling and Scaling-L to scale a cluster can lead to diﬀerent average VM-pair distance. Thus, given the same workload and scaling requests under load = 0.2, we calculate the diﬀerence of the average VM-pair distances between two scaled clusters resulted from Scaling and Scaling-L, respectively, for the same job j, that is, avgdCj − avgdCj . S caling S caling−L Figure 8 shows the empirical cumulative distribution of such diﬀerence under %inc = 20% and %inc = 30% respectively. As we can see, 80% of all the diﬀerences are larger than zero and 20% less. The 20 percent that are less than zero indicate that Scaling-L does not always generate an allocation for the scaled cluster with smaller average VM-pair distance than the allocation obtained by Scaling. This is to be expected, because two algorithms lead to diﬀerent VM layout and bandwidth usage in the datacenter during the dynamic job arrival process. Even for the same job, Scaling-L and Scaling may work on the diﬀerent pre-existing placement and network status. Nevertheless, in terms of the overall eﬀect, Scaling-L reduces the average VM-pair distance for the large majority of scaled jobs. Besides, from the distribution we can tell that the improvement by Scaling-L becomes more pronounced under higher %inc. For %inc = 30%, 20 percent of distance reduction fall in [0.2, 0.6] but for %inc = 20% only 10%. 3) Migration Cost: The Scaling-M algorithm minimizes the number of VM migrations by maximizing the overlap size with pre-existing cluster placement. To show its eﬀectiveness, we compare it with the cluster allocation algorithm in [5], [6], referred to as “C-A”, which is used to search the allocation for the scaled cluster in the datacenter where the pre-existing cluster is hypothetically removed, without considering the overlap with pre-existing VM placement. We replay the same workload and scaling request sequences and measure the total number of VM migrations incurred by these algorithms for all the scaling operations. Let load = 0.6 and Pr scal = 0.2. From Table II we can see that without considering the overlap, the number of VM migrations can be prohibitive, from 748 to 915. Scaling-M signiﬁcantly reduces the total number of migrated VMs for the scaling by at least 90%. Combining with Figure 6, it can be concluded that Scaling-M can greatly decrease the rejection rate at very small migration cost. We also evaluated the eﬀect of α in Formula (9) by varying it from 0 to 1 by step 0.2. Given a workload with load = 0.6, 9 0.5 Scaling-M 0.2 0.15 0.1 0.05 Scaling Scaling-L 0.5 Scaling-M 0.4 0.3 0.2 0.1 0 0 0.2 0.3 0.4 0.5 %inc Scaling-M C-A 0.2 0.6 20% 34 748 0.3 0.4 0.5 0.6 %inc Fig. 6: The rejection rate under load = 0.6,Pr scal = 0.2. 30% 38 820 40% 70 852 50% 34 915 TABLE II: Total number of VM migrations for scaling with diﬀerent %inc. α #VM Migrations Total Avg pair distance 0 1193 73 0.2 1057 97 Scaling-L 1 Scaling-M 0.4 0.3 0.2 0.1 0 %inc Fig. 5: The rejection rate under load = 0.2,Pr scal = 0.2. Scaling cumulative probability distribution Scaling-L Rejection rate Scaling Rejection rate Rejection rate 0.3 0.25 0.4 841 125 0.6 581 134 0.8 300 128 1 38 133 TABLE III: The number of VM migrations and average VMpair distance with diﬀerent α. %inc = 30% and Pr scal = 0.2, we examine the total number of VM migrations for the scaling and the sum of all the scaled clusters’ average VM pair distance under diﬀerent α. The result is shown in Table III. Both two measurements vary with α signiﬁcantly and monotonously overall1 The overlap size increases with α, leading to decreasing number of VM migrations when α increases; the average VM-pair distance increases with α. Such results verify the eﬀectiveness of our weight sum design in (9), in which α can act as a control knob to capture users’ preferences. VII. Discussion In this paper we mainly focus on the scaling of virtual cluster abstraction in size, i.e., from < N, B > to < N , B > (N > N). Nevertheless, we can use the algorithms in this paper to solve two other types of scaling that increase the bandwidth requirement. To scale a cluster C from < N, B > to < N, B > (B > B), we ﬁrst hypothetically increase the bandwidth of C’s VMs to B and check whether there are any links that do not have enough residual bandwidth for the increment of C’s bandwidth reservation. If no such links exist, C can be scaled in-place with accordingly increasing the bandwidth reservation on links; otherwise, C’s original placement has to be changed for the scaling and we use the allocation algorithm with VM migration in Section V to ﬁnd the allocation of new cluster abstraction < N, B > that has the largest overlap size with C’s original placement and conduct the corresponding VM migrations. For the scaling from < N, B > to < N , B >, we ﬁrst try to scale the cluster in-place with two steps: ﬁrst < N, B >→< N, B > and then < N, B >→< N , B >. We use the algorithm in Section III or IV for the second step. If both two steps succeed, the cluster can be scaled in-place; If any of two steps failed, we turn to the algorithm with VM migration to allocate < N , B > to minimize the number of VM migrations. 1 In Table III 134 is an exception. For a particular workload it may have such small variance. The observations from multiple random workloads indicate that the overall trend is monotonic. 0.2 0.3 Prscal 0.4 0.5 Fig. 7: The rejection rate under load = 0.6, %inc = 30% 0.9 0.8 %inc=30% %inc=20% 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 Difference of average VM-pair distance Fig. 8: CDF of avgdCj avgdCj S caling−L S caling − . VIII. Conclusion This paper addresses the scaling problem of virtual cluster abstraction with bandwidth guarantee. We propose eﬃcient algorithms to scale up the size of the virtual cluster with and without allowing VM migration. Besides demonstrating the eﬃciency of these algorithms, simulation results also indicate that the pre-existing VM placement can seriously hinder the scalability of virtual clusters and the scalability can be signiﬁcantly improved at small VM migration cost. Acknowledgement: This research is supported by NSF CNS-1252292. References [1] “Amazon EC2,” http://aws.amazon.com/ec2/. [2] J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz, “Runtime measurements in the cloud: observing, analyzing, and reducing variance,” Proc. of VLDB Endow., vol. 3, no. 1-2, pp. 460–471, Sep. 2010. [3] C. Guo, G. Lu, H. J. Wang, S. Yang, C. Kong, P. Sun, W. Wu, and Y. Zhang, “Secondnet: a data center network virtualization architecture with bandwidth guarantees,” in Proc. of Co-NEXT. New York, NY, USA: ACM, 2010, pp. 15:1–15:12. [4] H. Ballani, P. Costa, T. Karagiannis, and A. Rowstron, “Towards predictable datacenter networks,” in Proc. of ACM SIGCOMM. New York, NY, USA: ACM, 2011, pp. 242–253. [5] D. Xie, N. Ding, Y. C. Hu, and R. R. Kompella, “The only constant is change: incorporating time-varying network reservations in data centers,” in Proc. of ACM SIGCOMM, 2012, pp. 199–210. [6] L. Yu and H. Shen, “Bandwidth guarantee under demand uncertainty in multi-tenant clouds,” in ICDCS, 2014, pp. 258–267. [7] J. Lee, Y. Turner, M. Lee, L. Popa, S. Banerjee, J.-M. Kang, and P. Sharma, “Application-driven bandwidth guarantees in datacenters,” in Proc. of ACM SIGCOMM, 2014, pp. 467–478. [8] N. R. Herbst, S. Kounev, and R. Reussner, “Elasticity in cloud computing: What it is, and what it is not,” in ICAC. San Jose, CA: USENIX, 2013, pp. 23–27. [9] P. Padala, K.-Y. Hou, K. G. Shin, X. Zhu, M. Uysal, Z. Wang, S. Singhal, and A. Merchant, “Automated control of multiple virtualized resources,” in Proceedings of the 4th ACM European conference on Computer systems. ACM, 2009, pp. 13–26. [10] Z. Shen, S. Subbiah, X. Gu, and J. Wilkes, “Cloudscale: elastic resource scaling for multi-tenant cloud systems,” in Proceedings of the 2nd ACM Symposium on Cloud Computing. ACM, 2011, p. 5. [11] Z. Gong, X. Gu, and J. Wilkes, “Press: Predictive elastic resource scaling for cloud systems,” in Network and Service Management (CNSM), 2010 International Conference on. IEEE, 2010, pp. 9–16. [12] R. Han, L. Guo, M. M. Ghanem, and Y. Guo, “Lightweight resource scaling for cloud applications,” in ACM/IEEE CCGrid, 2012, pp. 644– 651. [13] H. Nguyen, Z. Shen, X. Gu, S. Subbiah, and J. Wilkes, “Agile: Elastic distributed resource scaling for infrastructure-as-a-service,” in ICAC. San Jose, CA: USENIX, 2013, pp. 69–82. [14] H. Herodotou, F. Dong, and S. Babu, “No one (cluster) size ﬁts all: Automatic cluster sizing for data-intensive analytics,” in Proc. of ACM SOCC. New York, NY, USA: ACM, 2011, pp. 18:1–18:14. [15] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warﬁeld, “Live migration of virtual machines,” in Proc.of NSDI. Berkeley, CA, USA: USENIX Association, 2005, pp. 273–286. [16] L. Yu, L. Chen, Z. Cai, H. Shen, Y. Liang, and Y. Pan, “Stochastic load balancing for virtual resource management in datacenters,” IEEE Transactions On Cloud Computing, 2016. [17] C. H. Papadimitriou and K. Steiglitz, Combinatorial Optimization: Algorithms and Complexity. Upper Saddle River, NJ, USA: PrenticeHall, Inc., 1982. 0.6