ELECTRONIC COPY
SAMKNOWS ANALYSIS OF
ROGERS’ BROADBAND PERFORMANCE
IN FEBRUARY 2015
Delivered by Email to: Shane Jansen
Rogers
Dated:
February 25, 2015
ELECTRONIC COPY
ISSUED TO ROGERS
FEBRUARY 25, 2015
[THIS PAGE LEFT INTENTIONALLY BLANK]
SamKnows | 2
ISSUED TO ROGERS
FEBRUARY 25, 2015
Contents
A
EXECUTIVE SUMMARY
5
B
TESTING METHODOLOGY
6
B.1
Recruitment
6
B.2
Sample Composition
8
B.3
Test Configuration and Schedule
9
B.4
Data Analysis
11
C
KEY PERFORMANCE INDICATORS
12
C.1
Download Throughput
12
C.2
Upload Throughput
15
C.3
Latency
18
C.4
Packet Loss
19
C.5
US providers vs Rogers comparison
20
SamKnows | 3
ISSUED TO ROGERS
FEBRUARY 25, 2015
About This Document
This document is a summary report, which has been commissioned by Rogers
and completed by SamKnows. For the purposes of this study 440 households
across Rogers’s customer base were given a specially configured hardware device
(SamKnows Whitebox), which runs a series of purpose-built tests to measure
Internet performance.
The analysis in this report is carried out on data collected over the date period of
20th January to 20th February 2015.
Any comments on the analysis in this document should be directed to Roxanne
Robinson (Roxanne@samknows.com).
SamKnows | 4
ISSUED TO ROGERS
A
FEBRUARY 25, 2015
Executive Summary
This report presents the findings of a project commissioned by Rogers to study
broadband performance across their network. This study was conducted during
20th January to 20th February 2015 using a sample of 440 fixed measurement
devices (‘Whiteboxes’) deployed to Rogers customers.
SamKnows has been engaged with other regulators and broadband providers
around the world and the methodology used has been developed to meet the
needs of both regulators and operators to enable them to better understand and
characterize their networks.
Tests were performed in a similar fashion to the FCC’s Measuring Broadband
America project, gathering statistics on upload and download speeds.
This report focuses on both the upload and the download tests specifically.
These were run every other hour each day for a duration of 30 seconds.
SamKnows created the panel using standard recruitment methodology. Potential
volunteers were contacted via an email campaign run by Rogers. Recipients of
this email were then directed to a SamKnows operated website where they could
register for the project. Following specific stages SamKnows then selected the
final panel of participants who could receive a Whitebox.
The study found:
-
All Rogers current in market packages exceed 100% of advertised download
and upload speeds based on 24/7 results.
-
Speedboost™ pushes download speeds far beyond the advertised speed for
the majority of packages based on 24/7 results, particularly those with lower
advertised rates.
-
Rogers’ peak-hour download average of 108% of advertised speed exceeds the
US average of 101% and US cable operator average of 103%.
SamKnows | 5
ISSUED TO ROGERS
B
Testing Methodology
B.1
Recruitment
FEBRUARY 25, 2015
For the purposes of this study, SamKnows sought to recruit 440 panellists using
the following method:
1) Potential volunteers were contacted via an email campaign run by Rogers.
This was targeted at all their customers on current in market service packages
which were representative of the overall subscriber base. Following this
SamKnows assumed full control over volunteer selection.
2) Recipients of this email were directed to a SamKnows-operated website to
register for the project. This involved completing an online form, requesting
information about which Rogers package they were subscribed to, and where
they resided.
3) SamKnows reviewed the demographics of this initial panel to ensure a
representative sample and to identify any deficiencies or shortfalls that would
prevent us achieving the goals of the project.
4) A sub-set of the volunteers were then selected based on the statistical
requirements of the panel. Selected volunteers were then asked to complete
an acknowledgment of User Terms and Conditions that outlined the
permissions to be granted by the volunteer in key areas such as privacy.
5) Of those volunteers that completed the User Terms and Conditions,
SamKnows selected the final panel of participants, each of whom received a
Whitebox for self-installation. SamKnows arranged for dispatch of the
Whitebox, and SamKnows provided full support during the Whitebox
installation phase.
6) Panellists were asked to install the Whitebox per the provided instructions and
leave it connected for the duration of the study.
SamKnows | 6
ISSUED TO ROGERS
FEBRUARY 25, 2015
NETWORK SCHEMATIC The installation of the SamKnows Whitebox is as follows:
Option B: Option A: To home computers
To home computers
SamKnows Whitebox
SamKnows Whitebox
Personal router
SamKnows | 7
ISSUED TO ROGERS
B.2
FEBRUARY 25, 2015
Sample Composition
440 Rogers customers were provided with a Whitebox, which they were
instructed to leave powered on and connected to their Internet Modem/Gateway.
151 panellists were excluded for the following reasons:
1) The Whitebox hadn’t run tests for at least 5 days in the month (due to it not
being turned on, or the panellist was actively using the broadband connection
continuously)
2) The panellist’s service package could not be reliably determined
3) The results were highly erratic and/or intermittent
4) On a legacy Rogers speed package.
5) As this report focuses on new products, most of the excluded panellists
belong to older products not suitable for the purpose of this study.
These exclusions were made to ensure the integrity of the panel and ensure each
speed package was statistically representative. As a result the report is based on
289 panellists.
SamKnows | 8
ISSUED TO ROGERS
FEBRUARY 25, 2015
The table below depicts the breakdown of the sample:
ISP
Package (Download x
Upload)
Sample Size
Rogers
15Mbps x 1Mbps
12
Rogers
30Mbps x 5Mbps
48
Rogers
60Mbps x 10Mbps
93
Rogers
70Mbps x 10Mbps
41
Rogers
100Mbps x 10Mbps
46
Rogers
250Mbps x 20Mbps
49
TOTAL
289
Table 1: Sample size split by Download Throughput service package
SamKnows typically recommends a minimum of anything between 40 and 45
samples per service package being reported upon, as this is what is considered
statistically significant based on research carried out by regulators around the
world. This is also the minimum sample size used by the FCC in their Measuring
Broadband America program. The 15Mbps plan therefore does not have a
sufficient representative sample size to be statistically significant, although it
remains included in this study.
B.3
Test Configuration and Schedule
All measurements for download and upload were conducted on a series of ‘offnet test nodes’. The same metrics, test parameters and testing schedule as the
FCC’s Measuring Broadband America project were used.
A detailed description of the methodology underpinning each metric can be
found at
http://www.samknows.com/broadband/uploads/foundationmodule/SQ301002-EN-Test-Suite-Whitepaper-D02.pdf
Four servers were used in total and they were hosted at a data centre in Toronto.
SamKnows | 9
ISSUED TO ROGERS
FEBRUARY 25, 2015
A summary of the speed related parameters and measurement frequency for each
metric is detailed below:
Test Schedule (Fixed) for Rogers
Test
Target(s)
Frequency
Duration
Download speed
1 off-net test node
Every other hour, 24/7
Fixed 30 seconds***
Upload speed
1 off-net test node
Every other hour, 24x7
Fixed 30 seconds***
Latency/Loss
1 off-net test node
Every hour, 24x7
Video Streaming
1 off-net test node
Every hour, 24x7
Web Browsing speed
1 off-net test node
Every hour, 24x7
DNS response time
1 off-net test node
Every hour, 24x7
Jitter
1 off-net test node
Every hour, 24x7
*** The speed tests run for a fixed duration of 30 seconds, but output the cumulative average
transfer speed every 5 seconds (i.e. the first average is for seconds 0-5, the second is for 0-10,
etc.). This allows us to measure performance over multiple durations, whilst running only one
test.
Below is also an outline of the number of tests carried out for each of the major
metrics along with the volume of data used for these tests:
Test
Number of Tests
Volume of data
Consumed
Download speed
845,316
180.87 TB
Upload speed
830,592
16.77 TB
Latency/Loss
277,233
0.99 GB
Web Browsing speed
2,210,636
0.82 TB
SamKnows | 10
ISSUED TO ROGERS
B.4
FEBRUARY 25, 2015
Data Analysis
A large raw dataset was collected over the period 20th January to 20th February
2015. The following process describes how the data was analysed to produce the
charts and tables in this report:
1. Failed measurements and significant outliers were identified and removed
from the raw dataset. A significant outlier is one where there has been a clear
error and failed measurements occur when a test fails (this could be due to the
user being online when the test was trying to run or something similar).
The FCC define the removal of any problem units as: removed measurements
for any unit that exhibited greater than or equal to 10% failures in a particular
one hour period (the purpose is to remove periods where units were unable to
reach the Internet).
2. Panellists that changed service package intra-month were identified and data
was removed for the service package that they spent the shortest duration on.
This is in line with SamKnows standard global methodology.
3. Per-Whitebox averages were computed for the period of the day for each
metric.
4. Per-service-package averages were computed from the per-Whitebox averages
calculated in step 3.
Please note that the analysis is not weighted by Rogers’s customer distribution.
However a sample plan was built to mitigate this based on the customer base.
SamKnows | 11
ISSUED TO ROGERS
FEBRUARY 25, 2015
C
Key Performance Indicators
C.1
Download Throughput
Sustained download throughput is a measure of the speed a broadband
connection is able to deliver for sustained periods of time. This metric is the one
most commonly used by regulators when reporting on speed.
All of Rogers' packages exceed their advertised rates significantly. The 100Mbps
product outperforms all other products, achieving 118% of the advertised speed.
All other packages achieve between 106% and 110% of advertised speed. The
250Mbps plan also displays a considerable improvement since March 2014,
increasing from 105% to 110% of headline speed. The 30Mbps and 60Mbps plan
instead show a slightly lower level of throughput.
Sustained Download Throughput, Percentage of Advertised Speed
140%
Throguhput (%)
120%
117.81%
108.21%
107.74%
107.81%
106.85%
15x1
30x5
60x10
70x10
110.08%
100%
80%
60%
40%
20%
0%
100x10
250x20
Package
Figure a.1: Sustained Download Speed as a percentage of Advertised Speed, by package
SamKnows | 12
ISSUED TO ROGERS
FEBRUARY 25, 2015
Burst download throughput is a measure of the short-term speed that can be
achieved on broadband connections. This is commonly much faster than the
sustained rate for cable service providers thanks to technologies such as
SpeedBoost™.
Rogers employs SpeedBoost™ for their packages, whose effects can clearly be
seen in the figure below. Most broadband services show a considerable increase in
throughput thanks to burst technology with the exception of the 250Mbps
product, although it also displays a slight increase from 110% to 118% of
advertised speed, again showing an improvement since March 2014. In contrast,
the effects of burst technology are less significant in this study’s testing period for
the 30Mbps and 60Mbps plans, although they still display a considerable increase
from sustained throughput. The 15Mbps product shows the largest increase from
sustained throughput, rising to 183% from 108% of advertised speed.
Generally, packages with lower advertised rates benefit the most from burst
technology as it is designed to use spare bandwidth. Therefore, it is to be expected
that plans with higher advertised rates display less significant improvements on
throughput performance, as seen with the 250Mbps plan.
Throughput (%)
Burst Download Throughput, Percentage of Advertised Speed
200%
180%
160%
140%
120%
100%
80%
60%
40%
20%
0%
182.77%
142.53%
154.68%
133.52%
135.21%
117.80%
15x1
30x5
60x10
70x10
100x10
250x20
Package
Figure a.2: Burst Download Speed as a percentage of Advertised Speed, by package
SamKnows | 13
ISSUED TO ROGERS
FEBRUARY 25, 2015
All of Rogers’s packages experience a very small decline during the peak period,
with the 250Mbps package showing the most significant decrease in throughput
compared to all other plans. This decline in performance still remains very small,
with throughput dropping by 3.32% points. All packages continue to perform
noticeably above the advertised speeds.
Throughput (%)
106.76%
110.08%
117.81%
116.57%
106.40%
106.85%
107.81%
106.86%
106.74%
107.74%
120%
108.21%
140%
107.54%
Off-peak vs Peak Sustained Download Throughput, Percentage of Advertised
Speed
100%
80%
60%
24hr
40%
Peak
20%
0%
15x1
30x5
60x10
70x10
100x10
250x20
Package
Figure a.3: Peak vs Off-peak Sustained Download Speed as a percentage of Advertised Speed, by package
SamKnows | 14
ISSUED TO ROGERS
Upload Throughput
Sustained upload throughput is a measure of the speed a broadband connection is
able to send for sustained periods of time.
As with download speed, all plans exceed their advertised rates, achieving a similar
performance ranging between 104% and 108% of headline speeds. The 30Mbps
and 60Mbps packages exhibit the highest level of throughput as a percentage of
advertised speed compared to all other plans, both achieving 108% of headline
speeds. This is a considerable improvement for the 60Mbps plan, rising from 98%
in March 2014, contrasting with the 30Mbps plan which dropped from 124%.
Sustained Upload Throughput, Percentage of Advertised Speed
120%
105.66%
108.41%
108.44%
15x1
30x5
60x10
104.37%
104.17%
105.17%
70x10
100x10
250x20
100%
Throughput (%)
C.2
FEBRUARY 25, 2015
80%
60%
40%
20%
0%
Package
Figure b.1: Sustained Upload Speed as a percentage of Advertised Speed, by package
SamKnows | 15
ISSUED TO ROGERS
FEBRUARY 25, 2015
Burst upload throughput is a measure of the burst speeds attained during an
upload measurement. Some service providers enable Boost technologies only in
the downstream direction whilst others enable it in both directions (this is less
common).
Most of Rogers’ packages deliver burst upload throughput closely resembling
sustained throughput, with all plans exhibiting a small increase, showing only
small improvements unlike what is seen in the downstream direction. The
15Mbps package shows a more significant increase in throughput, although it is
also a much smaller improvement compared to burst download speed.
Burst Upload Throughput, Percentage of Advertised Speed
140%
124.47%
Throguhput (%)
120%
109.75%
111.42%
30x5
60x10
106.42%
105.78%
107.50%
70x10
100x10
250x20
100%
80%
60%
40%
20%
0%
15x1
Package
Figure b.2: Burst Upload Speed as a percentage of Advertised Speed, by package
SamKnows | 16
ISSUED TO ROGERS
FEBRUARY 25, 2015
As is the case with download speed, upload speed of all of Rogers’s packages
display a very small decline in average performance during peak hours. All plans
continue to exceed headline speeds during the peak period as well.
105.17%
104.32%
104.17%
103.81%
104.37%
104.21%
108.44%
108.17%
108.41%
107.66%
105.73%
120%
105.66%
Off-peak vs Peak Sustained Upload Throughput, Percentage of Advertised
Speed
Throughput (%)
100%
80%
60%
24hr
40%
Peak
20%
0%
15x1
30x5
60x10
70x10
100x10
250x20
Package
Figure b.3: Peak vs Off-peak Sustained Upload Speed as a percentage of Advertised Speed, by package
SamKnows | 17
ISSUED TO ROGERS
Latency
Latency is a measure of the time it takes for a single packet of data to reach its
destination from its point of origin and back. It is also known as round-trip
latency.
The latency of Rogers' packages ranges from 24.52ms to 39.76ms, achieved by the
15Mbps and 70Mbps plans respectively. The average level of latency across all
plans is slightly higher than in March 2014, although this is mostly due to the high
latency of the 70Mbps product. The 60Mbps plan exhibits a higher level of
latency than in the previous testing period, contrasting with the 250Mbps package
which shows a noticeably lower latency.
Latency
45
39.76
40
35
Latency (ms)
C.3
FEBRUARY 25, 2015
30
25
33.68
31.10
24.52
27.06
26.82
20
15
10
5
0
15x1
30x5
60x10
70x10
100x10
250x20
Package
Figure c: Round-trip latency, in milliseconds, by package
SamKnows | 18
ISSUED TO ROGERS
Packet Loss
Packet loss is a measure of the percentage of data packages sent from the home to
the destination that are lost during the transmission of data.
On average, packet loss is considerably lower than in March 2014, ranging from
0.04% to 0.098% with the 30Mbps, 60Mbps and 250Mbps products all displaying
significant improvements. Contrasting with latency results, the 70Mbps package
displays the lowest level of packet loss compared to all other products.
Packet Loss
0.12%
0.098%
0.10%
Packet Loss (%)
C.4
FEBRUARY 25, 2015
0.071%
0.08%
0.06%
0.077%
0.078%
100x10
250x20
0.050%
0.040%
0.04%
0.02%
0.00%
15x1
30x5
60x10
70x10
Package
Figure d: Packet loss, by package
SamKnows | 19
ISSUED TO ROGERS
US providers vs Rogers comparison
The chart below shows a comparison between the sustained download
performance of Rogers to US ISPs. The US data is sourced from the FCC’s April
2014 Measuring Broadband America report. The measurement methodology used
by the FCC and Rogers is directly comparable, although some small differences
exist (such as the measurement servers in use).
All US broadband providers achieve an average of 105% of advertised speed
during the 24-hour period, with US cable operators reaching 108% of advertised
speed during this period. These results are exceeded by Rogers’ 110% for this
study’s measuring period. During the peak period, Rogers achieves an average of
108%, continuing to exceed US performance with the latter’s overall average of
101%. Rogers’s peak period performance also exceeds that of cable operators,
which achieve an average of 103% of advertised speed.
Rogers’ overall average drops by 1.33% between the 24-hour and peak periods.
This is a better show of consistency for throughput compared to US cable
services, which exhibit a decline of 4.62%.
Sustained Download Throughput, Percentage of Advertised Speed
160%
140%
120%
Throughput (%)
C.5
FEBRUARY 25, 2015
100%
24hr
80%
Peak
60%
40%
20%
0%
ISP
Figure e: Sustaned Download Speed as a percentage of Advertised Speed comparison between Rogers and US
providers, by ISP
SamKnows | 20
ISSUED TO ROGERS
FEBRUARY 25, 2015
US cable providers show a very slightly higher average latency of 31.49ms
(compared to Rogers's 31.19ms). The following is a breakdown by tier for Rogers
and US cable providers.
Speed Tier 15Mbps 30Mbps 50-­‐60Mbps 100-­‐105Mbps Rogers 25.14453 27.81497 31.87972 27.74366 US cable 28.0625 31.2083 36.2796 37.1205 Table 2: Latency comparison between Rogers and US cable operators, in milliseconds, by speed tier
As seen above, individual tiers perform much better on Rogers's side, particularly
the 100-105Mbps tier. Packet loss sees similar results, with the average across
individual tier being slightly higher for US cable operators with 0.12% compared
to the 0.07% average of Rogers's speed tiers where applicable. Packet loss of all of
Rogers’s packages achieves an average of 0.08%. US cable operators only
outperform Rogers in the 50-60Mbps tier.
Speed Tier Rogers US cable 15Mbps 0.04% 0.12% 30Mbps 0.08% 0.09% 50-­‐60Mbps 0.10% 0.08% Table 3: Packet Loss comparison between Rogers and US cable operators, by speed tier
SamKnows | 21