1. Introduction
Cloud computing has changed the way in which modern enterprises are evolving. It preaches the paradigm
of - focusing more on the business aspects and IT infrastructure will be taken care of. This has given birth
to a new breed of organizations that focus on providing software, infrastructure and platform as a service.
Some of them are Google, Salesforce, Rackspace. This entire concept comes under the umbrella term Cloud
Computing.
As this is an emerging technology it has various issues related to performance, scalability security, fault
tolerance, etc. This wiki focuses on evaluating scalability in cloud computing. Many researchers have
published papers [1] that talks about the evaluation of cloud scalability in terms of some metrics and models.
To get a good insight into scalability evaluation in cloud it is necessary to understand what scalability in
cloud means. “Clouds’ scalability is reflected by its capability to dynamically increase or decrease the
number of server instances assigned to an application to cater to the usage demands of the application“[2].
To put it simply, it is the ability of the cloud to dynamically increase the resource allocation to a user that
desires them and then decrease the allocation of those resources to the user when it no longer requires that
increased amount of resources. Scalability is one of the key features in cloud and one of the major factors
why people around the world have taken to it so willingly. There are various services provided on cloud
today such as SaaS, PaaS & IaaS and a massive number of people use these services concurrently and hence
it is imperative that a cloud system scales well.
This naturally leads to the requirement for methods by which one can evaluate the scalability of a cloud,
and hence the services hosted on it too. For instance, evaluation of cloud scalability would be important for
the organization hosting the cloud since it would want to know how the system is reacting to different
amounts of load and whether it can provide the resources promised to its customers. On the other hand
users of a cloud would also want to evaluate if the cloud scales well and provides it the desired resources.
While organizations hosting the cloud have their own tools to test and evaluate its scalability, such tools
are not readily available to the users of the cloud. For instance, someone hosting a service on a cloud system
will want to know how well the cloud scales to provide the required resources to that service, if the number
of concurrent users of that service suddenly spike.
2. Models And Metrics For Evaluating Scalability In Clouds
Now that the importance of cloud scalability evaluation has been established, this section details how
evaluation of the scalability of the cloud is carried out. As compared to regular scalability testing, the testing
of cloud scalability offers a varied set of challenges. So to evaluate the scalability of a cloud, metrics apart
from the regular scalability metrics are required to be defined. [3] proposes the following set of metrics to
do so:
2.1 Testing Scalability of Cloud Applications
a. Performance/Resource Ratio (PRR) Measurement: Just the time required to perform a
computation is not enough to gauge the performance of a system running on the cloud. It
is important to take note of the resources utilized to perform the computation too.
PRR = (1/Tw)*(1/CR),
CR = ∑ Ri * Ti, where Ri is the allocation of resource i and Ti is the time for which the
resource is used. Tw is the waiting time and Tw = Tq + Te where Tq is queuing time and Te
the
execution
time.
b. The scalability of the system can be measured using Performance Change (PC) for varying
workloads
PC = (PRR (t)* W (t))/ (PRR (t’)*W (t’))
Where
W(t)
represents
the
workload
at
time
t.
In an ideally scenario the value of PC should be 1. But the PC might vary between different
test runs due to the complexity of the cloud system. So not only PC but also another metric
called Performance Variance (PV) needs to be measured. The PV effectively measures the
scalability, given the PC.
PV = E [(PCi - (1/n) * ∑ni=1 PCi) 2]
In an ideal scenario PC should be unity and PV should be as close to 0 as possible.
2.1.1 Methodology: Intelligent Scalability Testing
Data Mining techniques can be applied to help the scalability testing process. The following
methods can be used.
a. Feature
Selection
to
mine
the
bottleneck
in
scalability
testing
this method suggests that only the important features that affect scalability be mined to
reduce the complexity of testing scalability.
b. Association Rule to decide correlation between parameters and scalability
This method suggests to gauge the relationship of different parameters with scalability
using the association rule.
c. Mining
input-output
relationship
to
reduce
test
cases
Understanding the relationship between the input and output helps reduce the number of
test cases.
2.1.2
Platform Support: Partitioning For Scalability Testing
A two-layer model for partitioning as suggested in [4] partitions horizontally by tenants and
vertically by columns. This model can help in both read and update operations.
SaaS (Software as a Service) is a software distribution model in which software is published on a cloud
by a service provider and it then can be used by users. The data required by the software too is hosted by
the cloud. From the above description it can be deduced that performance and scalability of a SaaS is
correlated to that of a cloud system. Hence evaluating the scalability of a SaaS will provide a fair idea
regarding the scalability of the cloud system since the SaaS is hosted on the cloud.
The following material is based on the SaaS implementation in clouds and thus the described metrics and
models are specific to it. Many researchers have provided metrics and models to evaluate scalability in
the past. But none of them were purely concentrated on cloud scalability. Rather they were centric to
parallel and distributed systems.
In order to define these metrics and models for cloud, application researchers [1] have used an analytical
graphical model known as the radar chart/spider chart/ web chart. This chart represents the quantitative
variables involved on the axes starting from the same point thereby representing the multivariate data in
the form of a two dimensional chart.
2.2 System Resources Allocation and Utilization
In order to evaluate the system allocated resources and their utilisation in the cloud, the
following terms were described in [1].
CRAM(S,t) - Computing Resource Allocation Meter
CRAM is used as an indicator to represent the total amount of resource allocation for a SaaS(S)
at an evaluating time t. Now radar charts are used as a base to represent this entity graphically.
Further, the axes in the radar chart represent an allocated computing resource that belongs to
S in cloud.
In order to calculate the total number of allocated resource to S we need to calculate the area
of the radar chart. In order to compute the area we just need to calculate the area of the
subsequent triangles. The formula can be listed as:
CRAM(S,t) = 0.5 * sin(27l' / n) * L ai * ai + 1
where a1, a2… an represents the different types of allocated resource to S at time t
Figure 1. Computing Resource Allocation Meter [1]
CRUM(S,t) - Uniform Resource Allocation Meter
Now, what would interest the clients and vendors is the knowledge of actual consumption of
resources by the system while providing services. Thus, CRUM is used as a virtualization tool
to represent and measure the cloud resource and its utilization. CRUM is also represented
using the radar chart but in this case the axes represent the utilization value of an allocated
resource of a deployed SaaS application at time t. A similar formula is given for CRUM.
CRUM(S,t) = 0.5 * sin(27Z' / n) * LU . U + 1
i
i
where U U ,..., U represent the utilization of a resource allocated to application S. They
are generally helpful for clients and engineers to evaluate and monitor the system
performance.
1,
2
i
Figure 2. Computing Resource Utilization Meter [1]
2.3 System Scalability Measurement
Wikipedia states that “The key characteristic of scalable system is that with increased load the
application can perform at the same throughput level by addition of more resources and
avoiding major system modifications”. Thus using this concept researcher [1] proposed the
concept of determining scalability via their proposed metrics and models for system load,
system utilization and system performance.
Their system is based on the system’s effective capacity at time t. Thus they define the
Effective System Scalability (ESS) as the ratio of system load increase and system capacity
increase.
ESS = [SL(t) -SL(min)]/[SEC(t) - SEC(min)] ... [1]
where SL(t) stands for system load at time t and SL(min) is the minimum system load and
similarly the SEC(t) stands for system effective capacity at time t and SEC(min) refers to the
minimum capacity of the system.
Further, the authors defined the ESR (Effective Scalable Range) as the difference of
SEC(max) and SEC(min).
ESR = SEC(max) - SEC(min) … [1]
3. Case Study of Amazon EC2 for the Scalability Evaluation
In the earlier section of the wiki we saw that researchers [1] have provided metrics and models to evaluate
the scalability of cloud. In order to verify the correctness and effectiveness of these metrics and models
they also performed a test on the Amazon EC2 cloud technology. They performed the test on all five
instances of the Amazon EC2 i.e. Micro, Small, Large, and Extra-Large
Main purpose of this case study was:
1. To learn what scalability parameters and performance indicators are actually collectable and traceable in
cloud applications.
2. To check whether the proposed models and metrics fit in the real world scenario.
Following technologies and configurations were used to perform the case study: In this case study, a java
based application program to calculate Fast Fourier Transform is deployed on each instance of EC2 and
each EC2 instance is configured and monitored to analyse the performance and evaluate scalability of the
cloud environment.
APIs provided by Amazon like CloudWatch, Autoscaling and Elastic Load Balancing are used to configure
the instance of EC2 and gather required metrics.
 Amazon CloudWatch API allows its users to obtain information about resource utilization,
application performance, and operational health. For Amazon EC2 instances, Amazon CloudWatch
collects metrics for CPU utilization, disk usage activity and data transfer from each Amazon EC2
instance. [5]

Auto Scaling APIs are used to scale capacity of Amazon EC2 instance up or down automatically
according to defined conditions. With the help of this, user can ensure that the number of Amazon
EC2 instances will increase when the demand increases to maintain performance of the system and
decreases automatically if demand drops to minimum cost. When average CPU utilization goes
above upper bound of threshold value, auto group scaling APIs can be used to set condition to add
new Amazon EC2 instances in increments of 3 instances to the group and similarly, condition to
remove Amazon EC2 instances in the same increments can be set when CPU Utilization falls below
upper bound of threshold value.[6]

Elastic load balancing service provided by amazon distributes incoming traffic across multiple
instances of Amazon EC2. Elastic load balancing APIs are used to apply different loads on EC2
instances collect the values of parameters required for evaluating. [7]
Following steps are followed to set up [1] :
 Amazon EC2 instance management API is used to set up and configure the EC2 application
instance
 CloudWatch API is used to configure and monitor resource utilization metrics and performance
parameters.
 Autoscaling API is used to configure and monitor EC2 instance for computing resource utilization
and performance data.
Following figure shows set up done for the case study of scalability evaluation and performance analysis
of cloud environment [1]:
Figure 3. The EC2 cloud infrastructure and procedure steps for EC2 instance performance and scalability
evaluation [1]
As we discussed earlier, resource utilization in cloud environment can be considered as one of the major
factors for scalability evaluation. We defined CRUM model for calculating resource utilization. CRUM
stands for Computing Resource Allocation Meter. In this case study, EC2 CPU, disk storage, network-out,
I/O operations and memory are considered as computing resources and to obtain the utilization of these
resources five different traffic loads are applied. As per the observation, resource utilization in terms of
CPU usage and system and user I/O operations increases as traffic load increases but it remains constant
for the last two traffic loads. Following diagram depicts these observations:
Figure 4. Two key performance indicator in CRUM model [1]
Then for performance evaluation, threshold values for CPU utilization for EC2 instance are set using
autoscaling APIs as follows: lower bound = 54% and upper bound = 73%. Utilization of instance scalability
is one of the most important indicators out of five major performance indicators described in System
Performance Meter [1]. Following values are observed under different system loads [1]
Table 1. Load configuration and AutoScaling [1]
In this case study, for particular instances of EC2, SEC is evaluated using the System Effective Capacity
Model and as per discussed earlier in section ( Methods and metrics) value of Effective Scalable Range is
calculated using minimum value of SEC and maximum value of SEC by using ESR formula which is
nothing but 7.8E6.
ESR = SEC(max) - SEC(min) [1]
4. Conclusion
Cloud Computing and its related technologies have successfully scored their place in the list of hot topics
related to Computer Science. This is mainly due to their emerging nature. In the current scenario, industry
and its customers are looking for more equipped tools that could strongly comment on the scalability and
its related entities. While for the future, we need to rely on the existing and upcoming generation of
researchers to perform more deep studies which would not only solidify but will also establish new
foundations related to the field.
5. References
[1] Jerry Gao, Pushkala Pattabhiraman,, Xiaoying Bai and W. T. Tsai, “SaaS Performance and Scalability Evaluation
in Clouds”, Proceedings of The 6th international IEEE Symposium on Service Oriented System Engineering, 2011.
[2] Jian Wu, Qianhui Liang and Elisa Bertino, “Improving Scalability of Software Cloud for Composite Web
Services”, 2009 IEEE International Conference on Cloud Computing
[3] Wei-Tek Tsai, Yu Huang and Qihong Shao, “Testing the Scalability of SaaS Applications”
[4] W. Tsai, Y. Huang, Q. Shao, and X. Bai, “Data partitioning and redundancy management for robust multi-tenancy
SaaS.”, Int J Software Informatics, 4(4), 2010.
[5] http://aws.amazon.com/cloudwatch/
[6] http://aws.amazon.com/autoscaling/
[7] http://aws.amazon.com/elasticloadbalancing/
6. Other References
[1] Jerry Gao, K. Manjula, P. Roopa, and E. Sumalatha & Xiaoying Bai & W. T. Tsai & Tadahiro Uehara “A Cloud-Based
TaaS Infrastructure with Tools for SaaS Validation, Performance and Scalability Evaluation”, 2012 IEEE 4th
International Conference on Cloud Computing Technology and Science
[2] Madhvi Kamra, Ratnamala Manna “Performance of Cloud-based Scalability and Load with an Automation
Testing Tool In Virtual World”, 2012 IEEE Eighth World Congress on Services
[3] Trieu C. Chieu, Ajay Mohindra, and Alexei A. Karve “Scalability and Performance of Web Applications in a
Compute Cloud”, 2011 Eighth IEEE International Conference on e-Business Engineering