Summary WAN Optimisation Test Report Blue Coat Systems vs. Riverbed Enterprises increasingly have distributed employee bases – remote offices, mobile workers – even as applications become increasingly consolidated and centralized. This creates networking challenges that are required to deliver acceptable user performance while contending with continually rising bandwidth costs. WAN Optimisation has become a requirement to enable this new world. Applications and networks, however, are also changing. Enterprise applications have shifted to web technologies – HTML front ends over HTTP/SSL. Video is dominating traffic, over 52% of worldwide Internet traffic and increasingly important for enterprise training and communications. Cloud delivery is changing the structure of how traditional enterprise applications are deployed. We at Broadband‐Testing felt it was important to test not only the traditional WAN Optimisation use cases of file access, email and backup/DR optimisation, but also to test these critical emerging areas. We tested two key vendors – Blue Coat and Riverbed, comparing the Blue Coat MACH5 SG600 and Riverbed Steelhead 1050. We used a WAN simulator from Shunra to induce 100 ms of latency, and leveraged applications and various third party test tools. The results are summarized here. Traditional WAN
Optimization
(CIFS files, FTP
Backup, email)
Traditional WAN Optimization: Blue Coat and Riverbed TIE.
This tested resulted in a TIE between Riverbed Steelhead and Blue Coat MACH5. In the CIFS cold
Open and Save As operations, Riverbed had an edge. In CIFS warm Opens, as well as FTP and
email, Blue Coat had an edge. We declare a tie in this “traditional” WAN Optimisation use cases.
Web Applications
Web Applications: Blue Coat Wins, 1.5x-10x Better than Riverbed in Warm Cases.
(HTML front ends,
HTTP/SSL
transport;
SharePoint used)
We tested web applications – HTML front ends with HTTP and SSL network transport – using
Cloud-based
Cloud-based Applications: Blue Coat Wins, Optimizing Cloud apps 3x-93x Better than
applications
Riverbed and the Baseline. Riverbed Can’t Optimize Asymmetrically.
(Asymmetric case,
branch office to
cloud app)
Microsoft SharePoint. In cold operations – the first time data was seen - the vendors were
relatively even. When the data had been seen before, the warm scenario, Blue Coat was a clear
winner. In these warm scenarios, Blue Coat was 1.5x to 10x faster than Riverbed.
This test was a bit different, configuring the traffic to travel from the branch office to the Internet
to the cloud SaaS. In this configuration, Riverbed could not optimize the traffic at all. Though
cold results showed little optimization, Blue Coat provided significant speed up and bandwidth
savings in warm test cases, accelerating performance 3x to 93x.
Streaming Video
Video: Blue Coat Clearly Beats Riverbed, 500 to 20.
(Windows Media
RTSP, on demand)
We tested on-demand video files for Microsoft Windows Media Server. Blue Coat delivered video
to 500 simultaneous users, while leaving 65-70% of the bandwidth of a 1.5 Mbps T1 available for
other applications – astounding optimisation. Riverbed max’d the link at 100% with just 20 users
- leaving no room for other applications. Riverbed started creating video user errors with 50
users.
Traditional WAN Optimisation Tests: For traditional WAN Optimization, we tested CIFS file access using Cisco WAFS benchmarking tool. We also tested FTP transfer times to simulate both file transfer and backup like operations. Both CIFS access and FTP measured the time to conduct the various operations. For email, we used Exchange 2007 and measured the bandwidth, as it was difficult to engineer a “time test”. The results were basically a TIE between Riverbed Steelhead and Blue Coat MACH5. In the CIFS cold open, Save As operations, and warm Save operations, Riverbed had an edge, as well as in cold email and FTP tests. Blue Coat had an edge in CIFS warm Opens, as well as Warm FTP and email operations. In each of these tests, each vendor was relatively close to the others performance, with no significant differences. Web Applications – Internal to the Enterprise Most applications used in the enterprise have moved to web‐based delivery technologies. HTML front ends, via a web server, with HTTP and SSL‐based transport. Certainly ERP, CRM and other enterprise applications have already made the jump. Document access and collaboration has also transitioned to web document management and Intranets, including the biggest of those Microsoft SharePoint. We tested SharePoint as a representative application for internally deployed applications. Results for this test showed that the cold results were relatively even between the two vendors. On warm runs, however, Blue Coat performed significantly better. Blue Coat outperformed Riverbed in warm tests for web applications, ranging from 1.5x to 10x faster. Note on the warm runs, the baseline is not presented. We did this to be able to contrast the performance between the two vendors more clearly. Cloud Delivered Applications This test was a bit different, configuring the traffic to travel from the branch office directly to the Internet and to the Cloud SaaS. A “hub and spoke” test, where traffic came from the cloud, then through the data center and to the branch looked similar to internal SharePoint application deployment; also, we did not account for optimization from the cloud to the datacenter as part of the results. So we will focus on the asymmetric branch office to internet to cloud case. In that asymmetric configuration, Riverbed could not optimize the traffic at all, whether it was a cold run or a warm run. Blue Coat, however, delivered significant speed up and bandwidth savings in warm test cases for asymmetric cloud applications, accelerating performance 3x to 93x times. Note they did not provide much optimization on the cold runs. Also note, Blue Coat graphs here look very tiny – hard to see the one second delivery times in a graph! Video Besides cloud delivered applications, video is another area where Blue Coat demonstrated clear superiority. We tested on‐demand video files for Microsoft Windows Media Server with Microsoft’s own scaling test tool on the same 1.5Mbps link with 100ms of latency. The Baseline here saw ten simultaneous client accesses consuming the 100% of the link, leaving no additional bandwidth for other applications and errors. Riverbed also consumed 100% of the link with just 20 users ‐ leaving no room for other applications, similar to the baseline. We saw a large number of client errors at 50 simultaneous users. Blue Coat, on the other hand consumed only 30‐35% of the bandwidth of a 1.5 Mbps for 20 users with no video errors. In fact, they delivered similar results at 30 simultaneous clients, 50 clients, 100 clients. In fact, Blue Coat optimized up to 500 concurrent video clients taking only 30‐35% of the bandwidth for a short period of time, then LESS THAN 5% for the majority of the test. Astounding video optimization by Blue Coat, the clear winner. This graph illustrates the bandwidth consumed by 500 video simultaneous accesses of on demand video over a T1 link. Only 30‐35% of the link is consumed for less than two minutes, with almost no bandwidth used for the majority of the tests. Successful delivery of 500 on demand videos over a T1 – with plenty of room for other applications. About the Test Our test bed simulated a classic office environment. Using virtual machines, we created a set of clients that
attached directly to each of the WAN Optimisation products we had under test. We used a WAN simulator from
Shunra to enforce bandwidth and latency typical for WAN environment, and injected 100ms of latency.
For the Cloud application test, the configuration was different – the branch offices were connected directly to
the Internet, without a second device (asymmetric test scenario).
Who is Broadband‐Testing Based in Andorra, Broadband-Testing is Europe’s foremost independent network testing facility and consultancy organisation
for broadband and network infrastructure products. Broadband-Testing provides a range of specialist IT, networking and
development services to vendors and end-user organisations throughout Europe, SEAP and the United States. The full test
report can be accessed at: www.broadband-testing.co.uk.
©2011 Broadband-Testing. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the
express written consent of the authors. All trademarks used in this Report are the trademarks of their respective owners.