Rules of Thumb in Data Engineering Jim Gray International Conference on Data Engineering San Diego, CA 4 March 2000 Gray@Microsoft.com, http://research.Microsoft.com/~Gray/Talks/ 1 Credits & Thank You!! Prashant Shenoy U. Mass, Amherst analysis of web caching rules. shenoy@cs.umass.edu Terrance Kelly, U. Michigan, lots of advice on fixing the paper, tpkelly@mynah.eecs.umich.edu interesting work on caching at: http://ai.eecs.umich.edu/~tpkelly/papers/wcp.pdf Dave Lomet, Paul Larson, Surajit Chaudhuri how big should database pages be? Remzi Arpaci-Dusseau, Kim Keeton, Erik Riedel discussions about balanced systems an IO Windsor Hsu, Alan Smith, & Honesty Young, also studied TPC-C and balanced systems (very nice work!) http://golem.cs.berkeley.edu/~windsorh/DBChar/ Anastassia Ailamaki, Kim Keeton cpi measurements Gordon Bell discussions on balanced systems. 2 and Apology….. Printed/Published paper has MANY bugs! Conclusions OK (sort of ), but typos, flaws, errors,… Revised version at http://research.microsoft.com/~Gray/ and in CoRR and MS Research tech report archive. Sorry! By 15 March 2000. Sorry! Woops! 3 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 4 Trends: Moore’s Law Performance/Price doubles every 18 months 100x per decade Progress in next 18 months = ALL previous progress New storage = sum of all old storage (ever) New processing = sum of all old processing. E. coli double ever 20 minutes! 15 years ago 5 Trends: ops/s/$ Had Three Growth Phases 1890-1945 Mechanical Relay 7-year doubling 1945-1985 Tube, transistor,.. 2.3 year doubling 1985-2000 Microprocessor 1.0 year doubling 1.E+09 ops per second/$ doubles every 1.0 years 1.E+06 1.E+03 1.E+00 1.E-03 doubles every 7.5 years doubles every 2.3 years 1.E-06 1880 1900 1920 1940 1960 1980 2000 6 Trends: Gilder’s Law: 3x bandwidth/year for 25 more years Today: 10 Gbps per channel 4 channels per fiber: 40 Gbps 32 fibers/bundle = 1.2 Tbps/bundle In lab 3 Tbps/fiber (400 x WDM) In theory 25 Tbps per fiber 1 Tbps = USA 1996 WAN bisection bandwidth Aggregate bandwidth doubles every 8 months! 1 fiber = 25 Tbps 7 Trends: Magnetic Storage Densities Amazing progress Ratios have changed Capacity grows 60%/y Access speed grows 10x more slowly Magnetic Disk Parameters vs 1000000 Time 100000 10000 1000 100 tpi kbpi MBps Gbpsi 10 1 0.1 0.01 year 84 88 92 96 00 04 8 Trends: Density Limits Bit Density The end is near! Products:11 Gbpsi Lab: 35 Gbpsi “limit”: 60 Gbpsi But limit keeps rising & there are alternatives b/µm2 Gb/in2 3,000 2,000 Density vs Time b/µm2 & Gb/in2 ?: NEMS, Florescent? Holograpic, DNA? 1,000 600 300 200 SuperParmagnetic Limit 100 60 30 20 10 6 3 2 Wavelength Limit DVD ODD CD 1 0.6 Figure adapted from Franco Vitaliano, “The NEW new media: the growing attraction 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 of nonmagnetic storage”, 9 Data Storage, Feb 2000, pp 21-32, www.datastorage.com Trends: promises NEMS (Nano Electro Mechanical Systems) (http://www.nanochip.com/) also Cornell, IBM, CMU,… • 250 Gbpsi by using tunneling electronic microscope • Disk replacement • Capacity: • • • • 180 GB now, 1.4 TB in 2 years Transfer rate: 100 MB/sec R&W Latency: 0.5msec Power: 23W active, .05W Standby 10k$/TB now, 2k$/TB in 2002 10 Consequence of Moore’s law: Need an address bit every 18 months. Moore’s law gives you 2x more in 18 months. RAM Today we have 10 MB to 100 GB machines (24-36 bits of addressing) then In 9 years we will need 6 more bits: 30-42 bit addressing (4TB ram). Disks Today we have 10 GB to 100 TB file systems/DBs (33-47 bit file addresses) In 9 years, we will need 6 more bits 11 40-53 bit file addresses (100 PB files) Architecture could change this 1-level store: System 48, AS400 has 1-level store. Never re-uses an address. Needs 96-bit addressing today. NUMAs and Clusters Willing to buy a 100 M$ computer? Then add 6 more address bits. Only 1-level store pushes us beyond 64-bits Still, these are “logical” addresses, 64-bit physical will last many years 12 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 13 Storage Latency: How Far Away is the Data? 10 9 Andromeda Tape /Optical Robot 10 6 Disk 100 10 2 1 Memory On Board Cache On Chip Cache Registers 2,000 Years Pluto Olympia 2 Years 1.5 hr This Hotel 10 min This Room My Head 1 min 14 Storage Hierarchy : Speed & Capacity vs Cost Tradeoffs 1012 Disc Secondary 109 Main 106 Price vs Speed Cache 102 Offline Main Tape 100 Secondary Online Online Tape Disc Tape 10-2 Offline Nearline Tape Tape 10-4 Cache 103 10-6 10-9 10-6 10-3 10 0 10 3 Access Time (seconds) 10-9 10-6 10-3 10 0 10 3 Access Time (seconds) 15 $/MB Typical System (bytes) 1015 Size vs Speed Nearline Tape Disks: Today Disk is 8GB to 80 GB 10-30 MBps 5k-15k rpm (6ms-2ms rotational latency) 12ms-7ms seek 7K$/IDE-TB, 20k$/SCSI-TB For shared disks most time spent waiting in queue for access to arm/controller Wait Transfer Transfer Rotate Rotate Seek Seek 16 Standard Storage Metrics Capacity: RAM: Disk: Tape: MB and $/MB: today at 512MB and 3$/MB GB and $/GB: today at 40GB and 20$/GB TB and $/TB: today at 40GB and 10k$/TB (nearline) Access time (latency) RAM: Disk: Tape: 100 ns 15 ms 30 second pick, 30 second position Transfer rate RAM: Disk: Tape: 1-10 GB/s 20-30 MB/s - - -Arrays can go to 10GB/s 5-15 MB/s - - - Arrays can go to 1GB/s 17 New Storage Metrics: Kaps, Maps, SCAN Kaps: How many kilobyte objects served per second The file server, transaction processing metric This is the OLD metric. Maps: How many megabyte objects served per sec The Multi-Media metric SCAN: How long to scan all the data the data mining and utility metric And Kaps/$, Maps/$, TBscan/$ 18 Storage Ratios Changed 10x better access time 10x more bandwidth 100x more capacity Data 25x cooler (1Kaps/20MB vs 1Kaps/500MB) 4,000x lower media price 20x to 100x lower disk price Scan takes 10x longer (3 min vs 45 min) 1 1980 1990 Year 1970-1990 100:1 1990-1995 10:1 1995-1997 50:1 today ~ 0.03$/MB disk 100:1 3$/MB dram 0.1 2000 Storage Price vs Time Megabytes per kilo-dollar 100 10,000. 1,000. MB/k$ Accesses per Second 1. Capacity (GB) seeks per second bandwidth: MB/s 10. 10 Disk accesses/second vs Time Disk Performance vs Time 100 DRAM/disk media price ratio changed 10 100. 10. 1. 1 1980 1990 Year 2000 0.1 1980 1990 Year 22 2000 Data on Disk Can Move to RAM in 10 years Storage Price vs Time Megabytes per kilo-dollar 10,000. 100:1 MB/k$ 1,000. 100. 10. 10 years1. 0.1 1980 1990 Year 2000 23 More Kaps and Kaps/$ but…. 1970 1980 1990 1000 100 10 2000 100 GB 30 MB/s 24 Kaps/disk Kaps/$ Disk accesses got much less expensive Better disks Kaps over time Cheaper disks! 1.E+6 Kaps/$ But: disk arms 1.E+5 1.E+4 are expensive the scarce resource 1.E+3 1.E+2 45 minute Scan Kaps 1.E+1 vs 5 minutes in 1990 1.E+0 Disk vs Tape Disk 40 GB 20 MBps 5 ms seek time 3 ms rotate latency 7$/GB for drive 3$/GB for ctlrs/cabinet 4 TB/rack 1 hour scan Tape 40 GB 10 MBps 10 sec pick time 30-120 second seek time 2$/GB for media 8$/GB for drive+library 10 TB/rack 1 week scan Guestimates Cern: 200 TB 3480 tapes 2 col = 50GB Rack = 1 TB =20 drives The price advantage of tape is narrowing, and the performance advantage of disk is growing At 10K$/TB, disk is competitive with nearline tape. 25 It’s Hard to Archive a Petabyte It takes a LONG time to restore it. At 1GBps it takes 12 days! Store it in two (or more) places online A geo-plex (on disk?). Scrub it continuously (look for errors) On failure, use other copy until failure repaired, refresh lost copy from safe copy. Can organize the two copies differently (e.g.: one by time, one by space) 27 The “Absurd” 10x (=5 year) Disk 2.5 hr scan time (poor sequential access) 1 aps / 5 GB (VERY cold data) It’s a tape! 100 MB/s 200 Kaps 1 TB 28 How to cool disk data: Cache data in main memory See 5 minute rule later in presentation Fewer-larger transfers Larger pages (512-> 8KB -> 256KB) Sequential rather than random access Random 8KB IO is 1.5 MBps Sequential IO is 30 MBps (20:1 ratio is growing) Raid1 (mirroring) rather than Raid5 (parity). 29 Stripes, Mirrors, Parity (RAID 0,1, 5) RAID 0: Stripes bandwidth RAID 1: Mirrors, Shadows,… Fault tolerance Reads faster, writes 2x slower RAID 5: Parity Fault tolerance Reads faster Writes 4x or 6x slower. 0,3,6,.. 1,4,7,.. 0,1,2,.. 2,5,8,.. 0,1,2,.. 0,2,P2,.. 1,P1,4,.. P0,3,5,.. 30 RAID 10 (strips of mirrors) Wins “wastes space, saves arms” RAID 5 (6 disks 1 vol): Performance 675 reads/sec 210 writes/sec Write RAID1 (6 disks, 3 pairs) Performance 750 reads/sec 300 writes/sec Write 4 logical IO, 2 logical IO 2 seek + 1.7 rotate 2 seek 0.7 rotate SAVES SPACE Performance degrades on failure SAVES ARMS Performance improves on failure 31 Auto Manage Storage 1980 rule of thumb: A DataAdmin per 10GB, SysAdmin per mips 2000 rule of thumb A DataAdmin per 5TB SysAdmin per 100 clones (varies with app). Problem: 5TB is 60k$ today, 10k$ in a few years. Admin cost >> storage cost !!!! Challenge: Automate ALL storage admin tasks 33 Summarizing storage rules of thumb (1) Moore’s law: 4x every 3 years 100x more per decade Implies 2 bit of addressing every 3 years. Storage capacities increase 100x/decade Storage costs drop 100x per decade Storage throughput increases 10x/decade Data cools 10x/decade Disk page sizes increase 5x per decade. 34 Summarizing storage rules of thumb (2) RAM:Disk and Disk:Tape cost ratios are 100:1 and 3:1 So, in 10 years, disk data can move to RAM since prices decline 100x per decade. A person can administer a million dollars of disk storage: that is 1TB - 100TB today Disks are replacing tapes as backup devices. You can’t backup/restore a Petabyte quickly so geoplex it. Mirroring rather than Parity to save disk arms 35 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 36 Standard Architecture (today) System Bus PCI Bus 1 PCI Bus 2 37 Amdahl’s Balance Laws parallelism law: If a computation has a serial part S and a parallel component P, then the maximum speedup is (S+P)/S. balanced system law: A system needs a bit of IO per second per instruction per second: about 8 MIPS per MBps. memory law: =1: the MB/MIPS ratio (called alpha ()), in a balanced system is 1. IO law: Programs do one IO per 50,000 instructions. 38 Amdahl’s Laws Valid 35 Years Later? Parallelism law is algebra: so SURE! Balanced system laws? Look at tpc results (tpcC, tpcH) at http://www.tpc.org/ Some imagination needed: What’s an instruction (CPI varies from 1-3)? RISC, CISC, VLIW, … clocks per instruction,… What’s an I/O? 39 TPC systems Normalize for CPI (clocks per instruction) TPC-C has about 7 ins/byte of IO TPC-H has 3 ins/byte of IO TPC-H needs ½ as many disks, sequential vs random Both use 9GB 10 krpm disks (need arms, not bytes) KB IO/s MHz/ Disk Disks MB/s CPI mips / / s / cpu / cpu cpu IO disk Amdahl 1 1 1 6 TPC-C= random 550 2.1 262 8 100 397 50 40 TPC-H= sequential 550 1.2 458 64 100 176 22 141 Ins/ IO Byte 40 8 7 3 TPC systems: What’s alpha (=MB/MIPS) ? Hard to say: Intel 32 bit addressing (= 4GB limit). Known CPI. IBM, HP, Sun have 64 GB limit. Unknown CPI. Look at both, guess CPI for IBM, HP, Sun Alpha is between 1 and 6 Mips Memory Alpha Amdahl 1 1 tpcC Intel 8x262 = 2Gips 4GB tpcH Intel 8x458 = 4Gips 4GB tpcC IBM 24 cpus ?= 12 Gips 64GB tpcH HP 32 cpus ?= 16 Gips 32 GB 1 2 1 6 241 Amdahl’s Balance Laws Revised Laws right, just need “interpretation” Balanced System Law: A system needs 8 MIPS/MBpsIO, (imagination?) but instruction rate must be measured on the workload. Sequential workloads have low CPI (clocks per instruction), random workloads tend to have higher CPI. Alpha (the MB/MIPS ratio) is rising from 1 to 6. This trend will likely continue. One Random IO’s per 50k instructions. Sequential IOs are larger One sequential IO per 200k instructions 43 PAP vs RAP Peak Advertised Performance vs Real Application Performance Application Data File System CPU System Bus 550 x4 Mips = 2 Bips 1600 MBps 500 MBps PCI 1-3 cpi = 170-550 mips System Bus 133 MBps 90 MBps SCSI PCI Bus 1 PCI Bus 2 160 MBps 90 MBps Disks 66 MBps 25 MBps 44 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 45 Ubiquitous 10 GBps SANs in 5 years 1Gbps Ethernet are reality now. Also FiberChannel ,MyriNet, GigaNet, ServerNet,, ATM,… 10 Gbps x4 WDM deployed now (OC192) 1 GBps 3 Tbps WDM working in lab In 5 years, expect 10x, wow!! 120 MBps (1Gbps) 80 MBps 40 MBps 5 MBps 20 MBps 47 Networking WANS are getting faster than LANS G8 = OC192 = 8Gbps is “standard” Link bandwidth improves 4x per 3 years Speed of light (60 ms round trip in US) Software stacks have always been the problem. Time = SenderCPU + ReceiverCPU + bytes/bandwidth This has been the problem 48 The Promise of SAN/VIA:10x in 2 years http://www.ViArch.org/ Yesterday: 10 MBps (100 Mbps Ethernet) 250 ~20 MBps tcp/ip saturates 200 2 cpus round-trip latency ~250 µs 150 Now Wires are 10x faster Myrinet, Gbps Ethernet, ServerNet,… Fast user-level communication tcp/ip ~ 100 MBps 10% cpu Time µs to Send 1KB Transmit receivercpu sender cpu 100 50 0 100Mbps Gbps SAN round-trip latency is 15 us 1.6 Gbps demoed on a WAN 49 How much does wire-time cost? $/Mbyte? Gbps Ethernet 100 Mbps Ethernet OC12 (650 Mbps) DSL POTs Wireless: Cost Time .2µ$ .3µ$ .003$ .0006$ .002$ .80$ 10 ms 100 ms 20 ms 25 sec 200 sec 500 sec Seat cost $/3y GBpsE 2000 100MbpsE 700 OC12 12960000 OC3 3132000 T1 28800 DSL 2300 POTS 1180 Wireless ? Bandwidt h B/s $/MB Time 1.00E+08 2.E-07 0.010 1.00E+07 7.E-07 0.100 5.00E+07 3.E-03 0.020 3.00E+06 1.E-02 0.333 1.00E+05 3.E-03 10.000 4.00E+04 6.E-04 25.000 5.00E+03 2.E-03 200.000 2.00E+03 8.E-01 500.000 seconds in 3 years 94608000 50 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 52 The Five Minute Rule Trade DRAM for Disk Accesses Cost of an access (Drive_Cost / Access_per_second) Cost of a DRAM page ( $/MB/ pages_per_MB) Break even has two terms: Technology term and an Economic term PagesPerMBofDRAM PricePerDi skDrive BreakEvenReferenceInterval AccessPerSecondPerDi sk PricePerMB ofDRAM Grew page size to compensate for changing ratios. Now at 5 minutes for random, 10 seconds sequential 53 The 5 Minute Rule Derived $ T =TimeBetweenReferences to Page Breakeven: RAM_$_Per_MB = PagesPerMB T= _____DiskPrice . T x AccessesPerSecond DiskPrice x PagesPerMB . RAM_$_Per_MB x AccessPerSecond 54 Plugging in the Numbers BreakEvenReferenceInterval PagesPerMBofDRAM PricePerDi skDrive AccessPerSecondPerDi sk PricePerMB ofDRAM PPM/aps Random 128/120 Sequential 1/30 ~ disk$/Ram$ Break Even ~1 1000/3 ~300 5 minutes .03 ~ 300 10seconds Trend is longer times because disk$ not changing much, RAM$ declining 100x/decade 5 Minutes & 10 second rule 55 When to Cache Web Pages. Caching Caching Caching Caching saves user time saves wire time costs storage only works sometimes: New pages are a miss Stale pages are a miss 56 The 10 Instruction Rule Spend 10 instructions /second to save 1 byte Cost of instruction: I =ProcessorCost/MIPS*LifeTime Cost of byte: B = RAM_$_Per_B/LifeTime Breakeven: NxI = B N = B/I = (RAM_$_B X MIPS)/ ProcessorCost ~ (3E-6x5E8)/500 = 3 ins/B for Intel ~ (3E-6x3E8)/10 = 10 ins/B for ARM 57 Web Page Caching Saves People Time Assume people cost 20$/hour (or .2 $/hr ???) Assume 20% hit in browser, 40% in proxy Assume 3 second server time Caching saves people time 28$/year to 150$/year of people time or .28 cents to 1.5$/year. connection cache R_remote seconds LAN LAN Modem Modem Mobile Mobile proxy browser proxy browser proxy browser 3 3 5 5 13 13 R_local seconds H hit rate 0.3 0.1 2 0.1 10 0.1 0.4 0.2 0.4 0.2 0.4 0.2 People S avings ¢/page 0.6 0.3 0.7 0.5 0.7 58 1.4 Web Page Caching Saves Resources Wire cost is penny (wireless) to 100µ$ LAN Storage is 8 µ$/mo Breakeven: wire cost = storage rent 4 to 7 months Add people cost: breakeven is ~ 4 years. “cheap people” (.2$/hr) 6 to 8 months. Time = A/B A B C $/10 KB $/10 KB download storage/mo cache Internet/LAN network 1.E-04 8.E-06 storage time 18 months $ 0.02 Modem Wireless 2.E-04 1.E-02 8.E-06 2.E-04 36 months 300 years 0.03 0.07 Break-even People Cost Time = (A+ C )/B of download Break Even 15 years 21 years >99959years Caching Disk caching 5 minute rule for random IO 11 second rule for sequential IO Web page caching: If page will be re-referenced in 18 months: with free users 15 years: with valuable users then cache the page in the client/proxy. Challenge: guessing which pages will be re-referenced detecting stale pages (page velocity) 60 Outline Moore’s Law and consequences Storage rules of thumb Balanced systems rules revisited Networking rules of thumb Caching rules of thumb 61