Engineering Multi-Tenant
Software-as-a-Service Systems
Bikram Sengupta (IBM Research India)
Abhik Roychoudhury (National University of Singapore)
1
ICSE 2011 workshop PESOS 2011, Honolulu
What is SaaS?

Software as a service (SaaS) is a software delivery
model in which software and its associated data are
hosted centrally (typically in the Internet/cloud) and are
typically accessed by users using a thin client, normally
using a web browser over the Internet. — wikipedia

Gartner predicts that the SaaS market will grow at a
compound annual growth rate (CAGR) of 22.1% through 2011
 Gartner/Dataquest Insight: “SaaS Demand Set to Outpace
Enterprise Application Software Market Growth.”
2
ICSE 2011 workshop PESOS 2011, Honolulu
SaaS vs. On-Premise Deployment
SaaS
Simplicity
Flexibility
Accessibility
Cost
3
On-Premise
web-based,
simple deployment
less flexible,
requires adaption
internet reliability
installation,
maintenance
full ownership,
customizable
higher availability
Low initial cost,
ongoing subscription
Software license,
infrastructure
ICSE 2011 workshop PESOS 2011, Honolulu
A. Lenk et al,
ICSE 2009 WS
on SE-Cloud
4
ICSE 2011 workshop PESOS 2011, Honolulu
Challenges in building Multi-tenant SaaS
vendor-offered
customizations
tenant-specific
changes
Fixed-set SLA:
• High commonality
• Ease development/maintenance
• Little variability
• Tenants need to compromise
5
Diverse requirements:
• Improve Flexibility
• Meet customer needs
• Degrade level of sharing
• Increase vendor’s overhead
ICSE 2011 workshop PESOS 2011, Honolulu
Broader perspective
Evolving programs / program versions


Testing [ASE08, ASE10, FSE10, …]


Test suite augmentation to stress changes
Debugging [ESEC-FSE99, ESEC-FSE’09, FSE10, …]

DARWIN method [ESEC-FSE99] uses symbolic execution for
regression debugging.
Design and development of multi-tenant SaaS software
can benefit from research on evolving programs.


6
Customization of SaaS to accommodate new tenants – can be
systematized using testing / debugging research.
ICSE 2011 workshop PESOS 2011, Honolulu
A Scenario that did not happen
??
http://www.asxgroup.com.au/asx-sgx-merger.htm
The Federal Treasurer, The Honourable Wayne Swan, has made a statement that he has
rejected the proposed merger of ASX Limited (ASX) and Singapore Exchange (SGX).
In these circumstances the parties have agreed to terminate the Merger Implementation
Agreement entered into on 25 October 2010.
8
ICSE 2011 workshop PESOS 2011, Honolulu
A Business Case
Suppose SGX and ASX make an agreement to offer joint online trading
services for a range of stock transactions. This has to be made operational
very soon to exploit a favorable economic climate.
 Several new business processes have to be supported with a high
number of variants tailored to the needs of different grades of investors
 A number of third party services may need to be leveraged, some of
which support large-scale data analytics off remote cloud infrastructures
 Grades of investors have to be offered specialized SLAs based on their
usage patterns and needs (performance expectations, security, isolation,
recoverability requirements etc)
 A highly-scalable multi-tenant SaaS solution has to be developed very
rapidly, and tested thoroughly before being deployed on a cloud.

9
ICSE 2011 workshop PESOS 2011, Honolulu
Challenges

Propose a SaaS system management model (from the
vendor’s perspective )


Tenant management



Enable quantitative analysis of SaaS system
On-boarding decision (return-on-investment)
Variability/commonality management
Application evolving



10
Add new functionalities
Testing
Re-factoring
ICSE 2011 workshop PESOS 2011, Honolulu
SaaS Management Model: Overview
Service I
Feature 1
v1
Feature 2
v1
subscription
v2
T1
v3
T2
v2
…
T3
Service II
…
Service X
11
ICSE 2011 workshop PESOS 2011, Honolulu
SaaS System

Service: a group of offered functionalities


service invariants
Feature: a particular functionality
 feature invariants

12
Variant v = <pre(v), meth(v), post(v)>: a variation of the feature to
support certain customization/changes
ICSE 2011 workshop PESOS 2011, Honolulu
Case study

Apache Roller



Free open source Java blog server roller.apache.org/
Powers the USA government blog, employee blogs at Sun and IBM,
Java blogging community …
Convert to a multi-tenancy SaaS system



Tenants: individual blogging sites
End user: site admin, blogger, reader…
Services:


Admin services
Weblog services



13
Features: write blog, edit blog, delete blog, category management …
Comment services
…
ICSE 2011 workshop PESOS 2011, Honolulu
Write Weblog Variants
New request/variant (v1)
Existing variant (v0)
Pre-conditions:
1. logged-in blogger
2. Word limit 2000
3. Rich/plain text
Implementation
Meth();
Post-conditions:
1. Publish time is displayed
2. Email notifications are
sent to owner and
newsfeed subscribers
14
Pre-conditions:
1. Logged-in blogger
2. Word limit 1000
3. Plain text only
v1 vs. v0
1. Stronger pre-condition
2. Weaker post-condition
3. v0 is a refinement of v1
Less impl.
cost 
Implementation
Configuration + Meth();
Post-conditions:
1. Email notifications are sent
to newsfeed subscribers
ICSE 2011 workshop PESOS 2011, Honolulu
Write Weblog Variants
New request/variant (v2)
Existing variants (v0 & v1)
Pre-conditions:
1. logged-in owner
2. Word limit 2000
3. Rich/plain text
v0
Implementation
Meth();
Post-conditions:
1. Publish time is displayed
2. Email notifications are
sent to owner and
newsfeed subscribers
15
v1
v2 vs. existing variants:
1. No refinement  new
impl. Is needed
2. Derive the impl. from the
closest existing variant
3. Create a new variant or
feature in the model
(depends on the distance)
1.
2.
3.
4.
Pre-conditions:
Authorized blogger
Select a category
Word limit 2000
Rich/plain text
Implementation
Meth1()
Post-conditions:
cost ∝ distance
1. Weblog is stored in the
selected category
2. Publish time is displayed
ICSE 2011 workshop PESOS 2011, Honolulu
On-boarding Cost

Accommodate a new service request

Direct on-boarding: exist a same variant, or a refinement


Feature expansion: create a new variant of an existing feature




16
No new implmentation  constant cost
Associate cost with weakening/strengthening of pre/post conditions
Assumption:
weaken post < strengthen pre < weaken pre <> strengthen post
Feature set augmentation: create a new feature
Service set augmentation: create a new service
ICSE 2011 workshop PESOS 2011, Honolulu
Onboarding Profit
revenue
Profit of
onboarding a
new tenant
17
Subscription fee
Expected pay-by-use
ICSE 2011 workshop PESOS 2011, Honolulu
cost
Aggregated
accommodation cost
for each request
Commonality

Commonality characterizes the level of reuse of a SaaS
system


Higher commonality  higher degree of sharing of features by
tenants
We expect that for each feature, a significant fraction of
tenants should subscribe to it (including its variants)
COMM S 
18

f Fs
Nf
| TS |  | FS |
ICSE 2011 workshop PESOS 2011, Honolulu
Tenant Selection Problem


A set of tenants to be onboarded on an existing system
Multi-objective optimization problem


maximize profit and maximize commonality
Find optimal subset of the tenants
commonality
Pareto front
Commonality threshold
profit
19
ICSE 2011 workshop PESOS 2011, Honolulu
SaaS Testing

When a new tenant is on-boarded, how to test that
existing tenants are not impacted by the changes
introduced?


No update of existing operations not relevant in our approach
How do we efficiently test that the SaaS system meets the
needs of the different tenants that have been onboarded ?


large degree of commonality
test only the changes introduced by a tenant


exploits similarities amongst tenants during testing

20
Test suite augmentation problem [ASE10, for evolving programs]
Test tree
ICSE 2011 workshop PESOS 2011, Honolulu
Test Tree
300 tests
100 tests
150 tests
Tenants {t3, t4, t6}
30 tests
Tenant {t1}
21
50 tests
Tenants {t2, t5}
ICSE 2011 workshop PESOS 2011, Honolulu
Initial Experiments

Conference Online Management System




Randomly generated tenant sets (20) and requests
Create two sets (set1 and set2) of costs associated with
each type of the onboarding cost factors



http://www.conference-service.com/index.html
Chair/ Reviewer/ Author/ Participant/ …
direct onboarding < weaken post cond. < strengthen pre cond.
< weaken pre cond. < strengthen post cond. < add new feature
consider only the implementation cost (no hardware or
maintenance cost)
Compute the Pareto optimal for the tenant selection
problem
24
ICSE 2011 workshop PESOS 2011, Honolulu
Experimental Results
Commonality
Pareto optimal solutions (set1)
onboard all (set1)
Pareto optimal solutions (set2)
onboard all (set2)
0.25
0.2
0.15
0.1
0.05
0
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
Profit
25
ICSE 2011 workshop PESOS 2011, Honolulu
Concluding Remarks

Towards a formal model for SaaS system




Variant management
Cost model
Tenant onboarding management
Based on the model, we propose various vendor
operations:

System testing



Test suite augmentation, Exploit commonality of test suites
System maintenance/re-factoring
Larger scale experiments – future work.
26
ICSE 2011 workshop PESOS 2011, Honolulu