NPS: A Non-interfering Web Prefetching System Ravi Kokku, Praveen Yalagandula, Arun Venkataramani, Mike Dahlin Laboratory for Advanced Systems Research Department of Computer Sciences University of Texas at Austin Summary of the Talk Prefetching should be done aggressively, but safely Safe: Non-interference with demand requests Contributions: A self-tuning architecture for web prefetching • Aggressive when abundant spare resources • Safe when scarce resources NPS: A prototype prefetching system • Immediately deployable July 12, 2016 Department of Computer Sciences, UT Austin 2 Outline Prefetch aggressively as well as safely July 12, 2016 Motivation Challenges/principles NPS system design Conclusion Department of Computer Sciences, UT Austin 3 What is Web Prefetching? Speculatively fetch data that will be accessed in the future Typical prefetch mechanism [PM96, MC98, CZ01] Client Server Demand Requests Responses + Hint Lists Prefetch Requests Prefetch Responses July 12, 2016 Department of Computer Sciences, UT Austin 4 Why Web Prefetching? Benefits [GA93, GS95, PM96, KLM97, CB98, D99, FCL99, KD99, VYKSD01, …] Reduces response times seen by users Improves service availability Encouraging trends Numerous web applications getting deployed • News, banking, shopping, e-mail… Technology is improving rapidly • capacities and prices of disks and networks Prefetch Aggressively July 12, 2016 Department of Computer Sciences, UT Austin 5 Why doesn’t everyone prefetch? Extra resources on servers, network and clients Interference with demand requests Two types of interference • Self-Interference– Applications hurt themselves • Cross-Interference– Applications hurt others Interference at various components • Servers – Demand requests queued behind prefetch • Networks – Demand packets queued or dropped • Clients – Caches polluted by displacing more useful data July 12, 2016 Department of Computer Sciences, UT Austin 6 Example: Server Interference Common load vs. response curve Avg. Demand Response Time (s) Constant-rate prefetching reduces server capacity 0.7 Pfrate=5 0.6 0.5 0.4 Pfrate=1 Demand 0.3 0.2 0.1 0 100 200 300 400 500 600 700 800 Demand Connection Rate (conns/sec) Prefetch Aggressively, BUT SAFELY July 12, 2016 Department of Computer Sciences, UT Austin 7 Outline Prefetch aggressively as well as safely Motivation Challenges/principles Self-tuning Decoupling prediction from resource management End-to-end resource management NPS system design Conclusion July 12, 2016 Department of Computer Sciences, UT Austin 8 Goal 1: Self-tuning System Proposed solutions use “magic numbers” Prefetch thresholds [D99, PM96, VYKSD01, …] Rate limiting [MC98, CB98] Limitations of manual tuning Difficult to determine “good” thresholds • Good thresholds depend on spare resources “Good” threshold varies over time Sharp performance penalty when mistuned Principle 1: Self-tuning Prefetch according to spare resources Benefit: Simplifies application design July 12, 2016 Department of Computer Sciences, UT Austin 9 Goal 2: Separation of Concerns Prefetching has two components Prediction – What all objects are beneficial to prefetch? Resource management –How many can we actually prefetch? Traditional techniques do not differentiate Prefetch if prob(access) > 25% Prefetch only top 10 important URLs Wrong Way! We lose the flexibility to adapt Principle 2: Decouple prediction from resource management Prediction: Application identifies all useful objects • In the decreasing order of importance Resource management: Uses Principle 1 • Aggressive – when abundant resources • Safe – when no resources July 12, 2016 Department of Computer Sciences, UT Austin 10 Goal 3: Deployability Ideal resource management vs. deployability Servers • Ideal: OS scheduling of CPU, Memory, Disk… • Problem: Complexity – N-Tier systems, Databases, … Networks • Ideal: Use differentiated services/ router prioritization • Problems: Every router should support it Clients • Ideal: OS scheduling, transparent informed prefetching • Problem: Millions of deployed browsers Principle 3: End-to-end resource management Server – External monitoring and control Network – TCP-Nice Client – Javascript tricks July 12, 2016 Department of Computer Sciences, UT Austin 11 Outline Prefetch Aggressively as well as safely Motivation Principles for a prefetching system Self-tuning Decoupling prediction from resource management End-to-end resource management NPS prototype design Prefetching mechanism External monitoring TCP-Nice Evaluation Conclusion July 12, 2016 Department of Computer Sciences, UT Austin 12 Prefetch Mechanism Prefetch Requests Client Demand Requests Demand Server Hint Lists 1. 2. 3. 4. July 12, 2016 Prefetch Server Hint Server Fileset Munger Server m/c Munger adds Javascript to html pages Client fetches html page Javascript on html page fetches hint list Javascript on html page prefetches objects Department of Computer Sciences, UT Austin 13 End-to-end Monitoring and Control Client while(1) { getHint( ); prefetchHint( ); } Hint Server if (budgetLeft) send(hints); else send(“return later”); GET http://repObj.html Monitor Demand Server 200 OK… Principle: Low response times server not loaded Periodic probing for response times Estimation of spare resources (budget) at server – AIMD Distribution of budget • Control the number. of clients allowed to prefetch July 12, 2016 Department of Computer Sciences, UT Austin 14 Monitor Evaluation (1) Avg Demand Response Time(sec) 0.7 Manual tuning, Pfrate=5 0.6 0.5 Manual tuning, Pfrate=1 0.4 No-Prefetching 0.3 Monitor 0.2 0.1 0 0 100 200 300 400 500 600 700 800 Demand Connection Rate (conns/sec) End-to-end monitoring makes prefetching safe July 12, 2016 Department of Computer Sciences, UT Austin 15 Monitor Evaluation (2) No-Prefetching Bandwidth (Mbps) 80 60 40 Demand: pfrate=1 20 Prefetch: pfrate=1 0 0 100 200 300 400 500 600 700 800 Demand Connection Rate (conns/sec) Manual tuning is too damaging at high load July 12, 2016 Department of Computer Sciences, UT Austin 16 Monitor Evaluation (2) No-Prefetching Bandwidth (Mbps) 80 Prefetch:Monitor Demand:Monitor 60 40 Demand: pfrate=1 20 Prefetch: pfrate=1 0 0 100 200 300 400 500 600 700 800 Demand Connection Rate (conns/sec) Manual tuning too timid or too damaging End-to-end monitoring is both aggressive and safe July 12, 2016 Department of Computer Sciences, UT Austin 17 Network Resource Management Demand and prefetch on separate connections Why is this required? HTTP/1.1 persistent connections In-order delivery of TCP So prefetch affects demand How to ensure separation? Prefetching on a separate server port How to use the prefetched objects? Javascript tricks – In the paper July 12, 2016 Department of Computer Sciences, UT Austin 18 Network Resource Management Prefetch connections use TCP Nice TCP Nice A mechanism for background transfers End-to-end TCP congestion control Monitors RTTs and backs-off when congestion Previous study [OSDI 2002] • Provably bounds self- and cross-interference • Utilizes significant spare network capacity Server-side deployable July 12, 2016 Department of Computer Sciences, UT Austin 19 End-to-end Evaluation Measure avg. response times for demand reqs. Compare with No-Prefetching and Hand-tuned Experimental setup Network Cable modem, Abilene Client httperf Trace IBM server July 12, 2016 PrefSvr Apache:8085 Fileset DemandSvr Apache: 80 HintSvr PPM predict Department of Computer Sciences, UT Austin 20 Avg. Dem. Resp. Time (ms) Prefetching with Abundant Resources 50 37.3 40 30 20 24.3 19.1 11.8 14.5 18 10 0 Cable modem No prefetching Abilene Hand-tuned NPS Both Hand-tuned and NPS give benefits Note: Hand-tuned is tuned to the best July 12, 2016 Department of Computer Sciences, UT Austin 21 Avg. Dem. Resp. Time (ms) Tuning the No-Avoidance Case 30 24.3 23.3 20 16.4 15.7 18.3 11.8 19.1 14.5 10 0 No prefetching T=0.05 Cable modem T=0.5 T=0.02 T=0.2 T=0.01 T=0.1 NPS Hand-tuning takes effort NPS is self-tuning July 12, 2016 Department of Computer Sciences, UT Austin 22 Avg. Dem. Resp. Time (ms) Prefetching with Scarce Resources 250 221.1 200 150 94.6 100 50 41.6 31.8 41.1 31.6 0 Loaded Server No prefetching Loaded Network Hand-tuned NPS Hand-tuned damages by 2-8x NPS causes little damage to demand July 12, 2016 Department of Computer Sciences, UT Austin 23 Conclusions Prefetch aggressively, but safely Contributions A prefetching architecture • Self-tuning • Decouples prediction from resource management • Deployable – few modifications to existing infrastructure Benefits • Substantial improvements with abundant resources • No damage with scarce resources NPS prototype http://www.cs.utexas.edu/~rkoku/RESEARCH/NPS/ July 12, 2016 Department of Computer Sciences, UT Austin 24 Thanks July 12, 2016 Department of Computer Sciences, UT Austin 25 Avg. Dem. Resp. Time (ms) Prefetching with Abundant Resources 160 140 120 100 80 60 40 20 0 134.9 73.3 75.5 24.3 37.3 19.1 18 11.8 14.5 Cable modem Abilene No prefetching Hand-tuned Trans-atlantic NPS Both Hand-tuned and NPS give benefits Note: Hand-tuned is tuned to the best July 12, 2016 Department of Computer Sciences, UT Austin 26 Client Resource Management Resources – CPU, memory and disk caches Heuristics to control cache pollution Limit the space prefetch objects take Short expiration time for prefetched objects Mechanism to avoid CPU interference Start prefetching after all demand done • Handles self-interference – more common case What about cross-interference? • Client modifications might be necessary July 12, 2016 Department of Computer Sciences, UT Austin 27