Storage: Alternate Futures
Jim Gray
Microsoft Research
Research.Micrsoft.com/~Gray/talks
NetStore ’99
Seattle WA, 14 Oct 1999
Yotta
Zetta
Exa
Peta
Tera
Giga
Mega
1
Kilo
Acknowledgments: Thank You!!
• Dave Patterson:
– Convinced me that processors are moving to the
devices.
• Kim Keeton and Erik Riedell
– Showed that many useful subtasks can be done by
disk-processors, and quantified execution interval
• Remzi Dusseau
– Re-validated Amdhl’s laws
2
Outline
• The Surprise-Free Future (5 years)
–
–
–
–
–
500 mips cpus for 10$
1 Gb RAM chips
MAD at 50 Gbpsi
10 GBps SANs are ubiquitous
1 GBps WANs are ubiquitous
• Some consequences
–
–
–
–
–
Absurd (?) consequences.
Auto-manage storage
Raid10 replaces Raid5
Disc-packs
Disk is the archive media of choice
• A surprising future?
– Disks (and other useful things) become supercomputers.
– Apps run “in the disk”
3
The Surprise-free Storage Future
•
•
•
•
•
•
•
1 Gb RAM chips
MAD at 50 Gbpsi
Drives shrink one quantum
Standard IO
10 GBps SANs are ubiquitous
1 Gbps WANs are ubiquitous
5 tips cpus for 1K$
and 500 mips cpus for 10$
4
1 Gb RAM Chips
• Moving to 256 Mb chips now
• 1Gb will be “standard” in 5 years,
4 Gb will be premium product.
• Note:
– 256Mb = 32MB: the smallest memory
– 1 Gb = 128 MB: the smallest memory
5
MAD at 50 Gbpsi
• MAD: Magnetic Aerial Density:
3-10 Mbpsi in products
20 Mbpsi in lab
50 Mbpsi = paramagnetic limit
but…. People have ideas.
• Capacity: rise 10x in 5 years (conservative)
• Bandwidth: rise 4x in 5 years (density+rpm)
• Disk: 50GB to 500 GB,
• 60-80MBps
• 1k$/TB
• 15 minute to 3 hour scan time.
6
Disk vs Tape
• Disk
• Tape
–
–
–
–
–
–
–
–
–
–
47 GB
15 MBps
10 ms seek time
5 ms rotate time
9$/GB for drive
3$/GB for ctlrs/cabinet
– 4 TB/rack
40 GB
5 MBps
30 sec pick time
Many minute seek time
5$/GB for media
10$/GB for drive+library
– 10 TB/rack
The price advantage of tape is narrowing, and
the performance advantage of disk is growing
Guestimates
Cern: 200 TB
3480 tapes
2 col = 50GB
Rack = 1 TB
=20 drives
7
System On A Chip
• Integrate Processing with memory on one chip
–
–
–
–
chip is 75% memory now
1MB cache >> 1960 supercomputers
256 Mb memory chip is 32 MB!
IRAM, CRAM, PIM,… projects abound
• Integrate Networking with processing on one chip
– system bus is a kind of network
– ATM, FiberChannel, Ethernet,.. Logic on chip.
– Direct IO (no intermediate bus)
• Functionally specialized cards shrink to a chip.
8
500 mips System On A Chip
for 10$
• 486 now 7$
233 Mhz ARM for 10$ system on a chip
http://www.cirrus.com/news/products99/news-product14.html
AMD/Celeron 266 ~ 30$
• In 5 years, today’s leading edge will be
–
–
–
–
System on chip (cpu, cache, mem ctlr, multiple IO)
Low cost
Low-power
Have integrated IO
• High end is 5 BIPS cpus
9
Standard IO in 5 Years
• Probably
• Replace PCI with something better
will still need a mezzanine bus standard
• Multiple serial links directly from processor
• Fast (10 GBps/link) for a few meters
• System Area Networks (SANS) ubiquitous
(VIA morphs to SIO?)
10
Ubiquitous 10 GBps SANs
in 5 years
• 1Gbps Ethernet are reality now.
– Also FiberChannel ,MyriNet, GigaNet,
ServerNet,, ATM,…
• 10 Gbps x4 WDM deployed now (OC192)
1 GBps
– 3 Tbps WDM working in lab
• In 5 years, expect 10x,
progress is astonishing
• Gilder’s law:
Bandwidth grows 3x/year
120 MBps
(1Gbps)
http://www.forbes.com/asap/97/0407/090.htm
80 MBps
40 MBps
5 MBps
20 Mbsp
11
Thin Client’s mean HUGE servers
•
•
•
•
•
•
AOL hosting customer pictures
Hotmail allows 5 MB/user, 50 M users
Web sites offer electronic vaulting for SOHO.
IntelliMirror: replicate client state on server
Terminal server: timesharing returns
…. Many more.
12
Standard Storage Metrics
• Capacity:
– RAM:
– Disk:
– Tape:
MB and $/MB: today at 512MB and 3$/MB
GB and $/GB: today at 50GB and 10$/GB
TB and $/TB: today at 50GB and 12k$/TB (nearline)
• Access time (latency)
– RAM:
– Disk:
– Tape:
100 ns
10 ms
30 second pick, 30 second position
• Transfer rate
– RAM:
– Disk:
– Tape:
1 GB/s
15 MB/s - - - Arrays can go to 1GB/s
5 MB/s - - - striping is problematic, but “works”
13
New Storage Metrics:
Kaps, Maps, SCAN?
• Kaps: How many kilobyte objects served per second
– The file server, transaction processing metric
– This is the OLD metric.
• Maps: How many megabyte objects served per second
– The Multi-Media metric
• SCAN: How long to scan all the data
– the data mining and utility metric
• And
– Kaps/$, Maps/$, TBscan/$
14
For
the
Record
(good 1999 devices packaged in system
http://www.tpc.org/results/individual_results/Compaq/compaq.5500.99050701.es.pdf)
Unit capacity (GB)
Unit price $
$/GB
Latency (s)
Bandwidth (MBps)
Kaps
Maps
Scan time (s/TB)
$/Kaps
$/Maps
$/TBscan
DRAM
1
5000
3300
1.E-7
1000
9.E+5
1.E+3
1
6.E-11
5.E-8
$0.05
DISK
9
900
12
2.E-3
15
6.E+2
14.67
600
1.E-8
6.E-7
$1
TAPE robot
40 X 100
20000
12
3.E+1
20
3.E-2
3.E-2
24500
6.E-3
6.E-3
$129
15
Tape is 1Tb with 4 DLT readers at 5MBps each.
For the Record
(good 1999 devices packaged in system
)
http://www.tpc.org/results/individual_results/Compaq/compaq.5500.99050701.es.pdf
1.E+7
DRAM
1.E+5
DISK
1.E+3
TAPE
1.E-7
$/
TB
sc
an
ap
s
$/
K
(s
/T
B)
ti m
e
Sc
an
1.E-5
M
ap
s
1.E-3
Ka
ps
1.E-1
$/
M
ap
s
1.E+1
1.E-9
1.E-11
Tape is 1Tb with 4 DLT readers at 5MBps each.
16
The Access Time Myth
• The Myth: seek or pick time dominates
• The reality: (1) Queuing dominates
•
(2) Transfer dominates BLOBs
•
(3) Disk seeks often short
• Implication: many cheap servers
better than one fast expensive server
– shorter queues
– parallel transfer
– lower cost/access and cost/byte
Transfer
Wait
Transfer
Rotate
Seek
• This is obvious for disk arrays
• This even more obvious for tape arrays
Rotate
Seek
17
Storage Ratios Changed
• DRAM/disk media price
ratio changed
• 10x better access time
• 10x more bandwidth
• 4,000x lower media price
1
1980
1990
Year
0.1
2000
Storage Price vs Time
Megabytes per kilo-dollar
100
10,000.
1,000.
MB/k$
Accesses per Second
1.
Capacity (GB)
seeks per second
bandwidth: MB/s
10.
10
1970-1990
100:1
1990-1995
10:1
1995-1997
50:1
today ~ 0.1$pMB disk 30:1
3$pMB dram
Disk accesses/second
vs Time
Disk Performance vs Time
100
–
–
–
–
10
100.
10.
1.
1
1980
1990
Year
2000
0.1
1980
1990
Year
18
2000
Data on Disk
Can Move to RAM in 8 years
Storage Price vs Time
Megabytes per kilo-dollar
10,000.
30:1
MB/k$
1,000.
100.
10.
6 years 1.
0.1
1980
1990
Year
2000
19
Outline
• The Surprise-Free Future (5 years)
–
–
–
–
–
500 mips cpus for 10$
1 Gb RAM chips
MAD at 50 Gbpsi
10 GBps SANs are ubiquitous
1 GBps WANs are ubiquitous
• Some consequences
–
–
–
–
–
Absurd (?) consequences.
Auto-manage storage
Raid10 replaces Raid5
Disc-packs
Disk is the archive media of choice
• A surprising future?
– Disks (and other useful things) become supercomputers.
– Apps run “in the disk”.
20
The (absurd?) consequences
• 256 way nUMA?
• Huge main memories: now:
•
•
•
•
•
•
1 GB RAM chips
MAD at 50 Gbpsi
Drives shrink one quantum
10 GBps SANs are ubiquitous
500 mips cpus for 10$
5 bips cpus at high end
500MB - 64GB memories
then: 10GB - 1TB memories
• Huge disks
now: 5-50 GB 3.5” disks
then: 50-500 GB disks
• Petabyte storage farms
– (that you can’t back up or restore).
• Disks >> tapes
– “Small” disks:
One platter one inch 10GB
• SAN convergence
1 GBps point to point is easy
21
•
•
•
•
The Absurd? Consequences
Further segregate processing from storage
Poor locality
Much useless data movement
Amdahl’s laws: bus: 10 B/ips io: 1 b/ips
Processors
~ 1 Tips
RAM Memory
~ 1 TB
Disks
22
~ 100TB
Storage Latency:
How Far Away is the Data?
10 9
Andromeda
Tape /Optical
Robot
10 6 Disk
100
10
2
1
Memory
On Board Cache
On Chip Cache
Registers
2,000 Years
Pluto
Olympia
2 Years
1.5 hr
This Hotel
10 min
This Room
My Head
1 min
23
Consequences
•
•
•
•
•
AutoManage Storage
Sixpacks (for arm-limited apps)
Raid5-> Raid10
Disk-to-disk backup
Smart disks
24
Auto Manage Storage
• 1980 rule of thumb:
– A DataAdmin per 10GB, SysAdmin per mips
• 2000 rule of thumb
– A DataAdmin per 5TB
– SysAdmin per 100 clones (varies with app).
• Problem:
– 5TB is 60k$ today, 10k$ in a few years.
– Admin cost >> storage cost???
• Challenge:
– Automate ALL storage admin tasks
25
The “Absurd” Disk
• 2.5 hr scan time
(poor sequential access)
• 1 aps / 5 GB
(VERY cold data)
• It’s a tape!
100 MB/s
200 Kaps
1 TB
26
Extreme case: 1TB disk:
Alternatives
• Use all the heads in parallel
– Scan in 30 minutes
– Still one Kaps/5GB
500 MB/s
1 TB
200 Kaps
• Use one platter per arm
– Share power/sheetmetal
– Scan in 30 minutes
– One KAPS per GB
500 MB/s
200GB
each
1,000 Kaps
27
Drives shrink (1.8”, 1”)
•
•
•
•
150 kaps for 500 GB is VERY cold data
3 GB/platter today, 30 GB/platter in 5years.
Most disks are ½ full
TPC benchmarks use 9GB drives
(need arms or bandwidth).
• One solution: smaller form factor
– More arms per GB
– More arms per rack
– More arms per Watt
28
Prediction: 6-packs
• One way or another, when disks get huge
– Will be packaged as multiple arms
– Parallel heads gives bandwidth
– Independent arms gives bandwidth & aps
• Package shares power, package, interfaces…
29
Stripes, Mirrors, Parity (RAID 0,1, 5)
• RAID 0: Stripes
– bandwidth
0,3,6,..
1,4,7,..
2,5,8,..
• RAID 1: Mirrors, Shadows,…
– Fault tolerance
– Reads faster, writes 2x slower
0,1,2,..
0,1,2,..
• RAID 5: Parity
– Fault tolerance
– Reads faster
– Writes 4x or 6x slower.
0,2,P2,.. 1,P1,4,.. P0,3,5,..
30
RAID 10 (strips of mirrors) Wins
“wastes space, saves arms”
RAID 5:
• Performance
– 225 reads/sec
– 70 writes/sec
– Write
• 4 logical IO,
• 2 seek + 1.7 rotate
• SAVES SPACE
• Performance
degrades on failure
RAID1
• Performance
– 250 reads/sec
– 100 writes/sec
– Write
• 2 logical IO
• 2 seek 0.7 rotate
• SAVES ARMS
• Performance
improves on failure
31
The Storage Rack
Today
• 140 arms
• 4TB
• 24 racks
24 storage processors
6+1 in rack
• Disks = 2.5 GBps IO
• Controllers = 1.2 GBps IO
• Ports
500 MBps IO
32
Storage Rack in 5 years?
• 140 arms
• 50TB
• 24 racks
24 storage processors
6+1 in rack
• Disks = 2.5 GBps IO
• Controllers = 1.2 GBps IO
• Ports
500 MBps IO
• My suggestion: move
the processors into
the storage racks.
33
It’s hard to archive a PetaByte
It takes a LONG time to restore it.
•
•
•
•
Store it in two (or more) places online (on disk?).
Scrub it continuously (look for errors)
On failure, refresh lost copy from safe copy.
Can organize the two copies differently
(e.g.: one by time, one by space)
34
Crazy Disk Ideas
• Disk Farm on a card: surface mount disks
• Disk (magnetic store) on a chip:
(micro machines in Silicon)
• Full Apps (e.g. SAP, Exchange/Notes,..)
in the disk controller
(a processor with 128 MB dram)
The Innovator's Dilemma: When New
Technologies Cause Great Firms to Fail
Clayton M. Christensen
.ISBN: 0875845851
ASIC
35
The Disk Farm On a Card
•
•
•
•
•
•
•
•
The 500GB disc card
An array of discs
Can be used as
100 discs
1 striped disc
50 Fault Tolerant discs
....etc
LOTS of accesses/second
bandwidth
14"
36
Functionally Specialized Cards
• Storage
P mips processor
ASIC
Today:
P=50 mips
• Network
M MB DRAM
M= 2 MB
In a few years
ASIC
P= 200 mips
M= 64 MB
• Display
ASIC
37
It’s Already True of Printers
Peripheral = CyberBrick
• You buy a printer
• You get a
– several network interfaces
– A Postscript engine
•
•
•
•
cpu,
memory,
software,
a spooler (soon)
– and… a print engine.
38
All Device Controllers will be Cray 1’s
• TODAY
– Disk controller is 10 mips risc engine
with 2MB DRAM
– NIC is similar power
• SOON
– Will become 100 mips systems
with 100 MB DRAM.
Central
Processor &
Memory
• They are nodes in a federation
(can run Oracle on NT in disk controller).
• Advantages
–
–
–
–
–
Uniform programming model
Great tools
Security
Economics (cyberbricks)
Move computation to data (minimize traffic)
Tera Byte
Backplane
39
With Tera Byte Interconnect
and Super Computer Adapters
• Processing is incidental to
– Networking
– Storage
– UI
• Disk Controller/NIC is
– faster than device
– close to device
– Can borrow device
package & power
Tera Byte
Backplane
• So use idle capacity
for computation.
• Run app in device.
• Both Kim Keeton (UCB) and
Erik Riedel (CMU) thesis investigate this
show benefits of this approach.
40
Implications
Conventional
• Offload device handling to
NIC/HBA
• higher level protocols:
I2O, NASD, VIA, IP, TCP…
• SMP and Cluster
parallelism is important.
Central
Processor &
Memory
Radical
• Move app to
NIC/device controller
• higher-higher level
protocols: CORBA /
COM+.
• Cluster parallelism is
VERY important.
Tera Byte
Backplane
41
How Do They Talk to Each Other?
Applications
Each node has an OS
Each node has local resources: A federation.
Each node does not completely trust the others.
Nodes use RPC to talk to each other
– CORBA? COM+? RMI?
– One or all of the above.
Applications
?
RPC
streams
datagrams
• Huge leverage in high-level interfaces.
• Same old distributed system story.
SIO
?
RPC
streams
datagrams
•
•
•
•
SIO
SAN
42
Outline
• The Surprise-Free Future (5 years)
– Astonishing hardware progress.
• Some consequences
–
–
–
–
–
Absurd (?) consequences.
Auto-manage storage
Raid10 replaces Raid5
Disc-packs
Disk is the archive media of choice
• A surprising future?
– Disks (and other useful things) become supercomputers.
– Apps run “in the disk”
43