Slow Throughput Best Practice Guide Version 1 November 2013 About this document This document is a best practice guide to identifying and eliminating general slow throughput issues. It looks at key areas such as local hardware (customer kit etc.), contention, threading, MTU, packet loss and latency, as well as providing a general guide to throughput expectations and finally a checklist. 1 Contents • Slow Throughput: Introduction (page 3) • Local Hardware Checks (End User environment) (pages 4-5) • MTU, Remote Servers and Upload Saturation: (page 6) • Contention: (page 7) • DSL Layer Errors and Threading: (page 8) • Packet Loss and Latency: (page 9) • BT Wholesale Throughput Ranges (pages 10-11) • Checklist: (page 12) 2 Slow Throughput: Introduction It is important to remember that a number of factors can influence throughput, including: • contention on the BT network • congestion on the internet • customer domain issues. Analysis indicates that the least likely cause of slow throughput is contention on the BT network. Internet LINE SYNCH RATE Internet THROUGHPUT (DOWNLOAD SPEED) 3 Local Hardware Checks (End User environment) The most common cause of speed issues are End User customer environment problems. A virus-affected PC or similar device, or an older operating system, may affect throughput. Check with the End User on their modem connections list to determine what devices are using the broadband bandwidth. Make sure that it is not showing a sync speed that would limit the available throughput. If checking a wired connection, confirm the router and network adapter on the computer have negotiated the correct speed. For example, there has been cases where a router and a computer that were both GigE-capable actually negotiated a link speed of 10Mbps. Wireless connections can be very problematic with the signal being significantly affected by barriers, including thick walls or ceilings, large metallic surfaces such as radiators, mirrors and refrigerators or inference from other devices. Changing wireless channels may help. There are generally three types of wireless router: 1. Wireless-B: routers that can provide speeds of around 5-6Mbps 2. Wireless-G: routers that can provide speeds of around 20-54Mbps 3. Wireless-N: routers that can provide speeds of around 54-100Mbps. Most routers don’t have the ability to serve different Wi-Fi protocols simultaneously, so connecting one wireless-G device to an N network can drop every connected device to wireless-G. Connecting one B device may drop the network down to B speeds. Some routers may not recover to higher speeds even if the slow device is removed from the network and a router reboot may be needed. Check the wireless card is configured correctly and to eliminate local issues disable the wireless signal altogether. 4 Local Hardware Checks (End User environment) – (Continued) Please note: Tablets and smartphones can still communicate wirelessly even when their screen is turned off. For Apple devices you will need to hold the power button for several seconds until you see the ‘slide to power off’ message. Turn off wireless printers, Wi-Fi Extenders, etc. Power line Adapters 6-200Mbps: Use of Powerline adapters needs to be checked. Faults have been observed where the customer has used these instead of Wi-Fi or a LAN cable connection. In extreme cases, throughput on ‘up to 200Mbps’ adapters has been seen to be as low as 3Mbps. Other devices: Try two different machines using an Ethernet cable directly into the router. Make sure that there are no other devices connected, i.e. disconnect any other wired devices or turn off completely. Cable connections: Make sure that the cables are plugged into the correct sockets on the modem/router/hub. Certain routers have a known feature where you can plug the cable into another port and get lower speeds (though this may appear as lower line synch rate and not just lower throughput). Speed Tests: Some speed tests are Flash-based, such as speedtest.net, and others such as thinkbroadband are Java-based. If the End User device has a corrupt installation of the software it can affect the results of the respective tests. Recommended Speed Test: To get a recognised result a BT Wholesale speed test needs to be run at www.speedtest.btwholesale.com. Clicking ‘Further Diagnostics’ will upload the results to BT Wholesale. Please note that currently only the most recent test result is stored by BT Wholesale. Additional Checks: • Power cycle the router, leaving it off for several minutes and then turn it back on. Restart the PC in Safe Mode and ensure that your End User does not connect on a VPN. • Try different browsers as they may perform differently. For example, on a PC try Internet Explorer and Chrome, on a Mac try Safari and Firefox. For all of these, try a number of different websites. 5 MTU, Remote Servers and Upload Saturation MTU (Maximum Transmission Unit): The route a packet takes through BT Wholesale’s network determines the optimum MTU value to be set in the router. As this route is transparent to BT Wholesale and is dynamic, BT Wholesale cannot confirm the optimum value or guarantee it will stay the same. Suggested test values are 1478, 1458, 1428 and 700. If you notice a significant throughput change on one of these values, it’s likely to be an MTU issue. Remote Server: It is important to remember that you can only download data as fast as the sending server can upload it. As such if a VPN (Virtual Private Network) is being used it could be this running slowly and causing the issue. It is important to note that any speed test can be inaccurate depending on the server load when the test was run. If the customer is uploading to one specific site then this could be the cause of the problem. Upload Saturation: This is a common issue on DSL. A line generally has a limited amount of upload throughput. As well as the obvious uses of upstream bandwidth, users may be uploading large emails, backing up data and using bandwidth-intensive applications such as file-sharing software. The upload bandwidth is also used to request data for downloading. If you are uploading a large amount of data and then request a file for download, the request for the data blocks will wait in a queue to be sent; this will cause a noticeable degradation in downstream throughput. Upstream over-utilisation can be checked on the RADIUS LOG check tool provided as part of the Knowledge Based Diagnostic tool or as a standalone tool. Both are available on the btwholesale.com website or via the XML/B2B option for customer use. 6 Contention Contention generally occurs at peak times. In residential areas this tends to be evenings and weekends, while in business areas it is during office hours. If you have a fault that is slow at all times of the day, then it may not be not contention. If the issue does appear to be contention, you can check the published reports for BT Wholesale’s network at the following links: 20CN: Go to the BT IPstream Support & Tools page at: https://www.btwholesale.com/pages/static/Products/Broadband/BT_IPstream/supportandtools.htm See ‘Virtual Path (VP) Congestion Management’ section of this page to download the report. 21CN: Go to the Wholesale Broadband Connect (WBC) Support & Tools page at https://www.btwholesale.com/pages/static/Products/Broadband/Wholesale_Broadband_Connect//supp ortandtools.htm See the ‘S-VLAN Re-grade Threshold Management’ section of this page to download the report. The reports indicate exchanges where VP/S-VLANs may be operating outside of BT Wholesale planning guidance, but are still performing within product specification. Guidance on the next steps is also indicated on the reports and will consist of information to support any recommended action to be taken: 1. 2. 3. 7 If 20CN consider migration to 21CN Wholesale Broadband Connect Date for planned capacity regrade of VP (20CN) or S-VLAN (21CN). No regrade date planned at present. DSL Layer Errors and Threading DSL layer errors can affect throughput. Normally they will create other fault conditions such as intermittent connection, but it’s possible to have a stable connection and for throughput to still be affected. Interleaving: If the line is erroring then data will be lost due to a lack of error correction, in which case interleaving can be applied. If interleaving is already applied then the errors may be too severe to be repaired, or may slow down the rate at which any data is transmitted between devices. The best way to check for such issues is to view graphical RRT (Reactive Repair Tool ) data via the BT Wholesale Knowledge Based Diagnostic (KBD) tool, which is available on the btwholesale.com website. Check for fluctuating SNR (Signal to Noise Ratio) and also what the MTBE (Mean Time Between Errors) is on the downstream. It is normal for MTBE to fluctuate, but ideally this should not be less than 50 seconds. The theory is that 20 errored seconds, or more, in a 15 minute period may indicate an issue. If you believe the issue is related to the DSL layer proceed by following running KBD and following the Recommendation contained in the test outcome. If the MTBE is <50 upstream this is more likely to be an issue with the customer’s environment. Threading: Another potential impact on throughput are threading issues. The symptoms of a threading issue are that the line can only download at a certain speed per thread. This will affect single thread downloads and streaming video; but segmented downloads and BitTorrent (and the BT Speedtester – which opens 4 threads) will not be affected. Normally threading issues are caused locally by a computer. However we have seen instances where it was the DSLAM or MSAN port not allowing more than 2MB per thread. 8 Packet Loss and Latency The first thing to check with packet loss or latency issues is that this is not related to a DSL layer issue (see previous section). If this is not the case, then the issue is likely to be either local (up to the router/modem, or on the End User customer’s computer/network) if they're pinging out from their device and issues are seen, or remote (within the BT core network). If you're confident it's not a customer issue after trying multiple machines in safe mode over Ethernet with wireless disabled, and that it's not a DSL layer issue, then the problem might be within the BT backhaul network. In this case it is most likely to be a VP (Virtual Path) bearer issue. Where you suspect that this is a common issue, it needs to be reported to the BT Wholesale proactive incident helpdesk either via • the online eChat facility at https://www.btwholesale.com/pages/static/echat.html • calling 0800 085 3608 and selecting Option 2, and then Option 2 again. Where there is only evidence of a single circuit issue, then please refer to or run a KBD test and follow the outcome recommendation. Please note: Currently, the KBD logic for 20CN checks for VP contention issues against the circuit being tested. KBD logic for 21CN is in the process of being updated to enable this type of check. 9 BT Wholesale Throughput Ranges The network dimensioning rules are designed to ensure that customers receive an IP throughput based on the product they have purchased, regardless of the activity of other users within this shared part of the network and such that users can achieve a throughput of: at least 8Mbit/s for 90% of the time over the busiest 3 hour period for the Standard WBC (12Mbit/s for FTTC); at least 12Mbit/s for 90% of the time over the busiest 3 hour period for Elevated WBC (16Mbit/s for FTTC). In actuality the average speed experienced in the busiest 3 hour period each day is generally much higher than the speeds quoted above. The IP throughput stated above is always lower than the headline ATM ADSL synchronisation speed due to the protocol overheads introduced by the ATM and PPP layers. In the tables below and on the following page, the absolute minimum throughput level below which slow speed faults are accepted is shown as the lower of the two throughput figures. Product Option: WBC Standard Product Option: WBC Elevated ADSL Line Synch Rate 0 to 288kbps Throughput Range ADSL Line Synch Rate Throughput Range 100kbps to 250kbps 0 to 288kbps 100kbps to 250kbps 288kbps to 576kbps 100kbps to 500kbps 288kbps to 576kbps 100kbps to 500kbps 576kbps to 1152kbps 400kbps to 1000kbps 576kbps to 1152kbps 400kbps to 1000kbps 1152kbps to 2272kbps 800kbps to 2000kbps 1152kbps to 2272kbps 800kbps to 2000kbps 2272kbps to 4544kbps 1200kbps to 4000kbps 2272kbps to 4544kbps 1200kbps to 4000kbps 4544kbps to 8128kbps 2000kbps to 7150kbps 4544kbps to 8128kbps 3000kbps to 7150kbps 8128kbps to 24384kbps 4000kbps to 21000kbps 8128kbps to 24384kbps 6000kbps to 21000kbps 10 BT Wholesale Throughput Ranges (continued) Product Option: IP Stream ADSL Line Synch Rate Throughput Range BT IP Stream Home 500 100kbps to 500kbps BT IP Stream Home 1000 200kbps to 1000kbps BT IP Stream Home 2000 400kbps to 2000kbps BT IP Stream Office 500 200kbps to 500kbps BT IP Stream Office 1000 400kbps to 1000kbps BT IP Stream Office 2000 800kbps to 2000kbps BT IP Stream BET 1M 160kbps to 1000kbps BT IP Stream BET 2M 160kbps to 2000kbps Product Option: IP Stream MAX Standard Product Option: IP Stream MAX Premium ADSL Line Synch Rate Throughput Range ADSL Line Synch Rate Throughput Range 288kbps 50kbps to 250kbps 288kbps 100kbps to 250kbps Above 288kbps and up to 576kbps 50kbps to 500kbps Above 288kbps and up to 576kbps 100kbps to 500kbps At or above 576kbps up to 1152kbps 200kbps to 1000kbps At or above 576kbps up to 1152kbps 400kbps to 1000kbps At or above 1152kbps up to 2272kbps 400kbps to 2000kbps At or above 1152kbps up to 2272kbps 800kbps to 2000kbps At or above 2272kbps up to 4544kbps 600kbps to 7150kbps At or above 2272kbps up to 4544kbps 1200kbps to 7150kbps 11 Checklist 1. Without making any changes run the speed tester on the customer's PC during the WORST time at http://speedtest.btwholesale.com then record the results and refer to the throughput SLA section. 2. Then test against a wired connection to the modem and connected to the test socket whilst checking that the cable is not damaged and ensuring the wireless is turned off on the router and PC. Please ensure where possible that all additional wiring and equipment is disconnected. 3. Start the PC in safe-boot mode. 4. Make sure a VPN is not connected. 5. Try a number of browsers. E.g. On a PC try Internet Explorer and Chrome, on a Mac try Safari and Firefox. 6. Try a number of different websites. 7. Run the speed tester again at the end of Points 5-6. If the result is good then this suggests something in the customer environment or the internet is interfering. 8. If the result is still bad then power cycle the router and retry the speed test. 9. If the result is still poor try a different router and repeat from steps 2- 8. 10. Assuming the performance result still fails go into ‘Further Diagnostics’ on the speed tester so that the poor result is logged on the BT Wholesale database. 12