Scaleout vs. Scaleup
Robert Barnes
Microsoft
7/27/2016
HPTS'99 Scale Out vs Scale Up
1
Why 1BTPD?
We had two main obstacles in bringing TP
to Microsoft:
Lack of products and infrastructure
 Industry mindshare and/or credibility

After starting to address the first obstacle,
we decided to build something to
demonstrate Microsoft has a credible
platform for large-scale applications
CPU
50 GB Disc
5 GB RAM
7/27/2016
HPTS'99 Scale Out vs Scale Up
2
Why 1BTPD?
We liked Jim Gray’s “cyber-brick” machine
and wanted to build one - only the technology
wasn’t there yet
The closest thing to it was a collection of
laptops - so we wanted to tie together 1000
laptops. We decided DebitCredit would be a
good workload to run, and 10KTPS*
DebitCredit was born
*10 K Transactions per Second
CPU
50 GB Disc
5 GB RAM
7/27/2016
HPTS'99 Scale Out vs Scale Up
3
7/27/2016
HPTS'99 Scale Out vs Scale Up
4
Why DebitCredit?
We needed a DB-only benchmark

Do not want to configure one million terminals
A simple transaction profile to stress DTC

Many little transactions
Need distributed transactions to stress DTC
TPC-B is dead (can not report numbers)
TPC-C is not server-only (need 1 million terminals)
and requires distributed data transparency
So, we returned to the Datamation Benchmark (1985)
7/27/2016
HPTS'99 Scale Out vs Scale Up
5
10KTPS  1BTPD
Work began on the benchmark in mid 1995
A vendor released a 30K TPM-C number
Even though 10KTPS is 600Ktpm and
DebitCredit is very different from TPC-C
transactions, we needed a different metric
11574.07 tps = 694444.2 tpm
= 41666652 tph
= 1 Billion Transactions
Per Day*
*thanks, Jim
7/27/2016
HPTS'99 Scale Out vs Scale Up
6
Scaling
Transactions Per
Second
DebitCredit Scaling
16000
14000
12000
10000
8000
6000
4000
2000
0
2
TPS
4
6
8 10 12 14 16 18 20
Number of Nodes
Linear (TPS)
7/27/2016
HPTS'99 Scale Out vs Scale Up
7
Scaleable Systems
SMP and Loosely Coupled Systems
SMP
Scaleup with SMP
8P is new standard (SHV)
Super Server
Scaleout with Loosely Coupled Systems
(LCS) using inexpensive parts
Departmental
Server
LCS
of PCs
or SHVs
Personal
System
7/27/2016
HPTS'99 Scale Out vs Scale Up
8
SMP Advantages
Single system image


no change to operations
(relative to uniprocessor)
no change to applications
SMP
Simple system resources



shared memory
shared disk
shared net
Super Server
Load balancing in OS kernel Departmental
Server
8x SMP soon commodity
SHV
Personal
System
7/27/2016
HPTS'99 Scale Out vs Scale Up
9
SMP Problems
Problems:





More then 8 processors not a
commodity today
More than 16 processors may SMP
require partitioning, scale out
needed anyway
Super Server
scale-down problem
(starter systems expensive
or fork lift upgrade)
Potential single point of failure Departmental
Server
Eventually stops scaling
Personal
System
7/27/2016
HPTS'99 Scale Out vs Scale Up
10
SMP Performance
Diminishing returns
TPC-C Scaling
600000
Linear (theoretical)
Microsoft (adjusted)
Sun
500000
TPM-C
400000
Linear
extrapolation
for 16
300000
200000
100000
0
0
10
20
30
40
50
60
70
Number of processors
7/27/2016
HPTS'99 Scale Out vs Scale Up
11
LCS Advantages
Advantages:
No hardware limit to scale, given
a scaleable interconnect and an
application that partitions
 Uses high volume,commodity
components
 No single point of failure
 Upgrade by incremental growth

Common
Function
7/27/2016
HPTS'99 Scale Out vs Scale Up
12
LCS Problems
Operations complexity

When not designed for LCS, costs
can be very expensive
Change in system architecture;
e.g. Exchange, SQL Server
(relative to uniprocessor)

Need additional system services





Load Balancing
LCS Membership
Configuration replication
Failure Retry (within LCS)
Parallelism needed for data and
applications


7/27/2016
explicit parallelism exposes
partitions to the application
implicit parallelism requires
transparency
HPTS'99 Scale Out vs Scale Up
Common
Function
13
givememoney.com
The givememoney.com business is brokering
– their goal is to not directly handle any
products
Site is currently <10 stores, plan to be
hundreds in a few years


Books, music,
video,computers,software,games,surplus
It takes over 90 days to add a new store
Over 90 servers make up current site
Today, checkouts/day 10K

Business plan calls for 500K by 2002
7/27/2016
HPTS'99 Scale Out vs Scale Up
14
givememoney.com FE
5
5
2
2
Cache Server
ASP SSL
FARM B
ASP SSL
FARM A
Basket/Ad/Surplus
SQL Product ASP File
Server
Server
ASP File SQL Product
Server Server
Receipt/Fulfillment
Games/Music
Comp/Soft Books
Search
7/27/2016 Servers
Videos
Music
Games/Music
Monitor and cache
HPTS'99 Scale Out vs Scale Up
Videos
Comp/Soft Books
Search
Servers
Music
15
givememoney.com
“We don’t have a scaling problem – if
we need more capacity, we just add a
server. We have a management
problem…”
7/27/2016
HPTS'99 Scale Out vs Scale Up
16
Scale Out Business Need
givememoney.com is at 10K orders/day,
business plan is to grow to 500K by 2002
Moore’s law says scale improves by 4X in 3
years, givememoney.com wants 50X , over
10X Moore’s law.
Even if you believe Moore’s law solves the
growth problem, it requires many forklift
upgrades
We still have to scale up – as it minimizes the
number of servers needed…
7/27/2016
HPTS'99 Scale Out vs Scale Up
17
Consolidation
Not consolidation vs. scale out
Consolidation motivations




Business
Site (Space)
Operations
Storage
Reasons for never wanting to have 1 huge
system




Single point of failure
Dis-economies of scale up
Mixed workload has operations challenges
similar to scale out
Difficulty of dealing with growth
7/27/2016
HPTS'99 Scale Out vs Scale Up
18
Scaleout is a software
challenge
Scaleup is hardware architecture & OEM driven

Optimized by Software


e.g. minimize locking, parallel threads, reduce contention on
critical sections, etc.
Ultimately, scale up begins to look like scale out
(partitioning)
Scaleout is software architecture driven

Optimized by hardware



7/27/2016
e.g. SAN (system area nets), low latency/high bandwidth
interconnects
Primarily an application design and operational
management problem
Possible today, but mostly left to application
design/development and ad hoc management tools
HPTS'99 Scale Out vs Scale Up
19
Summary
Scaleup begins to look like scaleout as
you add processors
Scaleout with commodity hardware is
economically compelling
Every large successful web site uses
scaleout – it is practical, but hard
Operations complexity is the key barrier
to scaleout
7/27/2016
HPTS'99 Scale Out vs Scale Up
20
CyberWall
7/27/2016
HPTS'99 Scale Out vs Scale Up
21