Put Everything
in Future (Disk) Controllers
(it’s not “if”, it’s “when?”)
Jim Gray
http://www.research.Microsoft.com/~Gray
Acknowledgements:
Dave Patterson explained this to me a year ago
Kim Keeton
Erik Riedel Helped me sharpen
these arguments
Catharine Van Ingen
1
Remember Your Roots
2
Kilo
Mega
Giga
Tera
Peta
Exa
Zetta
Technology Drivers: Disks
• Disks on track
• 100x in 10 years
•
•
2 TB 3.5” drive
Shrink to 1” is 200GB
Disk replaces tape?
Yotta
• Disk is super computer!
3
Data Gravity
Processing Moves to Transducers
• Move Processing to data sources
• Move to where the power (and sheet metal) is
• Processor in
» Modem
» Display
» Microphones (speech recognition)
& cameras (vision)
» Storage: Data storage and analysis
4
It’s Already True of Printers
Peripheral = CyberBrick
• You buy a printer
• You get a
» several network interfaces
» A Postscript engine
• cpu,
• memory,
• software,
• a spooler (soon)
» and… a print engine.
5
All Device Controllers will be Cray 1’s
•
TODAY
» Disk controller is 10 mips risc engine
•
»
with 2MB DRAM
NIC is similar power
SOON
Central
Processor &
Memory
» Will become 100 mips systems
•
•
with 100 MB DRAM.
They are nodes in a federation
(can run Oracle on NT in disk controller).
Advantages
» Uniform programming model
» Great tools
» Security
» economics (CyberBricks)
» Move computation to data (minimize traffic)
Tera Byte
Backplane
6
Basic Argument for x-Disks
• Future disk controller is a super-computer.
» 1 bips processor
» 128 MB dram
» 100 GB disk plus one arm
• Connects to SAN via high-level protocols
» RPC,
HTTP, DCOM, Kerberos, Directory Services,….
» Commands are RPCs
» management, security,….
» Services file/web/db/… requests
» Managed by general-purpose OS with good dev environment
• Move apps to disk to save data movement
» need programming environment in controller
7
The Slippery Slope
Nothing =
Sector Server
• If you add function to server
• Then you
add more function to server
• Function gravitates to data.
Everything =
App Server
8
Why Not a Sector Server?
(let’s get physical!)
• Good idea, that’s what we have today.
• But
» cache added for performance
» Sector remap added for fault tolerance
» error reporting and diagnostics added
» SCSI commends (reserve,.. are growing)
» Sharing problematic (space mgmt, security,…)
• Slipping down the slope to a 2-D block server
9
Why Not a 1-D Block Server?
Put A LITTLE on the Disk Server
• Tried and true design
» HSC - VAX cluster
» EMC
» IBM Sysplex (3980?)
• But look inside
» Has a cache
» Has space management
» Has error reporting & management
» Has RAID 0, 1, 2, 3, 4, 5, 10, 50,…
» Has locking
» Has remote replication
» Has an OS
» Security is problematic
» Low-level interface moves too many bytes
10
Why Not a 2-D Block Server?
Put A LITTLE on the Disk Server
• Tried and true design
» Cedar -> NFS
» file server, cache, space,..
» Open file is many fewer msgs
• Grows to have
» Directories + Naming
» Authentication + access control
» RAID 0, 1, 2, 3, 4, 5, 10, 50,…
» Locking
» Backup/restore/admin
» Cooperative caching with client
• File Servers are a BIG hit: NetWare™
»SNAP! is my favorite today
11
Why Not a File Server?
Put a Little on the Disk Server
• Tried and true design
» Auspex, NetApp,
» Netware
...
• Yes, but look at NetWare
» File interface gives you app invocation interface
» Became an app server
• Mail,
DB, Web,….
» Netware had a primitive OS
• Hard to program, so optimized wrong thing
12
Why Not Everything?
Allow Everything on Disk Server
(thin client’s)
• Tried and true design
» Mainframes, Minis, ...
» Web servers,…
» Encapsulates data
» Minimizes data moves
» Scaleable
• It is where everyone ends up.
• All the arguments against are short-term.
13
The Slippery Slope
Nothing =
Sector Server
• If you add function to server
• Then you
add more function to server
• Function gravitates to data.
Everything =
App Server
14
Disk = Node
• has magnetic storage (100 GB?)
• has processor & DRAM
• has SAN attachment
• has execution
Applications
environment
Services
DBMS
RPC, ...
File System
SAN driver
Disk driver
OS Kernel
15
Technology Drivers:
System on a Chip
• Integrate Processing with memory on one chip
» chip is 75% memory now
» 1MB cache >> 1960 supercomputers
» 256 Mb memory chip is 32 MB!
» IRAM, CRAM, PIM,… projects abound
• Integrate Networking with processing on one chip
» system bus is a kind of network
» ATM, FiberChannel, Ethernet,.. Logic on chip.
» Direct IO (no intermediate bus)
• Functionally specialized cards shrink to a chip.
16
Technology Drivers: What if
Networking Was as Cheap As Disk IO?
• TCP/IP
• Disk
»Unix/NT
100% cpu @ 40MBps
»Unix/NT
8% cpu @ 40MBps
Why the Difference?
Host does
TCP/IP packetizing,
checksum,…
flow control
small buffers
Host Bus Adapter does
SCSI packetizing,
checksum,…
flow control
17
DMA
Technology Drivers:
The Promise of SAN/VIA:10x in 2 years
• Today:
http://www.viarch.org/
» wires are 10 MBps (100 Mbps Ethernet)
» ~20 MBps tcp/ip saturates 2 cpus
» round-trip latency is ~300 us
• In the lab
» Wires are 10x faster Myrinet, Gbps Ethernet, ServerNet,…
» Fast user-level communication
• tcp/ip ~ 100 MBps 10% of each processor
• round-trip latency is 15 us
18
SAN:
Standard Interconnect
Gbps Ethernet: 110 MBps
PCI: 70 MBps
UW Scsi: 40 MBps
• LAN faster than
•
•
•
memory bus?
1 GBps links in lab.
100$ port cost soon
Port is computer
FW scsi: 20 MBps
scsi: 5 MBps
19
Technology Drivers:
100 GBps Ethernet replaces SCSI
• Why I love SCSI
» Its fast (40MBps)
» The protocol uses little processor power
• Why I hate SCSI
» Wires must be short
» Cables are pricy
» pins bend
20
Functionally Specialized Cards
• Storage
P mips processor
ASIC
Today:
P=50 mips
• Network
M MB DRAM
M= 2 MB
In a few years
ASIC
P= 200 mips
M= 64 MB
• Display
ASIC
21
Technology Drivers
Plug & Play Software
• RPC is standardizing: (DCOM, IIOP, HTTP)
» Gives huge TOOL LEVERAGE
» Solves the hard problems for you:
• naming,
• security,
• directory service,
• operations,...
• Commoditized
programming environments
» FreeBSD, Linix, Solaris,…+ tools
» NetWare + tools
» WinCE, WinNT,…+ tools
» JavaOS + tools
• Apps gravitate to data.
• General purpose OS on dedicated ctlr can run apps.
22
Basic Argument for x-Disks
• Future disk controller is a super-computer.
» 1 bips processor
» 128 MB dram
» 100 GB disk plus one arm
• Connects to SAN via high-level protocols
» RPC,
HTTP, DCOM, Kerberos, Directory Services,….
» Commands are RPCs
» management, security,….
» Services file/web/db/… requests
» Managed by general-purpose OS with good dev environment
• Move apps to disk to save data movement
» need programming environment in controller
23