Software Testing in the Cloud
Leah Riungu-Kalliosaari
Objective and Background




Objective: To understand how organizations can
successfully use the cloud for testing through empirical
observations
Cloud computing: A model for enabling convenient, on-demand
network access to a shared pool of configurable computing
resources (e.g. networks, servers, storage, application and services)
that can be rapidly provisioned and released with minimal
management effort or service provider interaction (NIST definition)
STaaS: A model for software testing whereby testing of an
application is provided as a service across the internet.
Cloud-based testing is provided on-demand and billed on payper-use basis, so that the user pays only for the resources they have
used
Work distribution during projects
MASTO/ESPA
- 3 interview rounds (Fall 2009, Spring 2010 & Fall 2010)
- 4 publications (2 conference + 1 conference + 1 journal)
- Researcher exchange (04-06/2011, Limerick, Ireland)
- About half the dissertation work completed
STX
- About half of the dissertation work to be completed
- 1 interview round (Fall 2011)
- 1 Journal article; 1 book chapter in progress
- Data analysis  formulating results of the thesis
- More publications and thesis finalization
What has been done so far?
Conditions that influence software
testing as an online service
 Research Issues for software testing in
the Cloud
 The effect of testing in the cloud (most
recent)

Effect of Testing in the Cloud (1)

On actual testing work
 More efficient performance testing - e.g.
Running 40 Amazon instances for generating
about 20 000 realistic user experiences
 Quicker testing – e.g. Being able to test run a
developed feature in say 10 different
environments and getting (almost) immediate
results
 More realistic test results
Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”,
Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012
Effect of Testing in the Cloud (2)

On delivery and support of testing services
 Better availability of testing tools and options –
e.g. being able to carry out technical feasibility
studies when choosing the tools to use.
 Improved developer-tester communication – due
to equal access to the system
 Enhanced service delivery for vendors – quicker
time to market and meeting of customer
demands.
Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”,
Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012
Roadmap towards testing in the Cloud

Develop an understanding of cloud computing


Carry out pilot projects


Consider what to test; select a cloud provider
Enhance team interaction and prepare for complexities


Safe way to explore potential benefits
Come up with elaborate strategies


Consider the benefits and risks
Developer/tester interaction while working with cloud-based environments
for development and testing
Enhance co-operation between research and industry

Aim at addressing cloud-related issues faced by the software industry
Riungu-Kalliosaari, L., Taipale, O., Smolander, K., ”Testing in the Cloud: Exploring the Practice”,
Special Issue on Software Engineering for Cloud Computing, IEEE Software, March/April 2012
Work in Progress (1)


18 interviews from 11 organizations (Autumn 2011)
Data analysis – an iterative process
 Data


from previous project also in use
Evaluation of quality attributes
Development of cloud-based testing strategy that
can be used while considering adoption of cloudbased testing
WiP (2)… Quality Attributes

Cloud based testing improves testing of certain quality
characteristics, for example, performance and
scalability
 Justification:
Cloud testing offers , for example,
computing resources in performance testing, human
resources in crowd testing, resources for testing
scalability. Further cloud testing reduces testing costs
e.g. pay per use pricing model.

Cloud-based testing improves certain areas of
testing, but it introduces also new issues e.g. security
issues, and testing associated with huge data sets.
WiP (3)…Quality Attributes
Cloud-based Testing
improves
controls
Peformance
Scalability
supports
is part of
Maintainability
Elasticity
WiP (4)…Cloud-based testing strategy
Consider/Evaluate
The line of business
Assess
The need
Critical vs. noncritical
applications
Size
What would you like
to achieve?
Security risks/threats
Trade-offs
Maintainability
The cloud
provider
Change
management e.g.
skills development
SLAs, TOS, etc
Choose
/
The delivery
approach
Select
Achieved vs. nonachieved needs,
security, trade-offs
Pleriminary observations from most recent
data (1)

There seems to be a correlation between quality and
requirements management
 Improved
requirements management leads to improvement
in quality

Global software development
 Managing

globally distributed teams efficiently
Change management is unique within each
organization
 Different
types of changes
 To introduce change, identify the weakest link
Pleriminary observations from most recent
data (2)


Cloud awareness is rising
There is need for the benefits of cloud computing to be
clearly understood and appreciated before cloud computing
can be taken into use
4 out of 11 organizations are using the cloud
 1 has a customer that is using the cloud
 1 has a customer that is considering to use salesForce.com in
future


Very small companies are more willing to use the cloud

No upfront costs
 Getting up small functionality up and running quickly
In general, open source tools can be used if deemed necessary
