Comparing IPv6 and IPv4 Performance
John Berg – Lead Engineer September 16, 2014
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
1
Background
• CableLabs has been conducting IPv6 interops since 2009
– Observed subtle but persistent IPv6 performance benefit
• Collected real performance data from one of our member operators
• Conducted lab testing to measure IPv4/IPv6 performance
– Native IPv6
– IPv4 with one layer of NAT
– IPv4 with two layers of NAT
2
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
2
How Service Provider Data Was Collected
• Service Provider produced >1.5 million IPv4/IPv6 data records
– Analysis performed jointly by CableLabs and MSO
• Metrics collected on data records:
– Average Round Trip Time (AVGRTT) for a variety of popular IPv4 and IPv6 web sites
– Statistics Include: Minimum, Maximum, Mean, Median, Standard Deviation, etc.
• Measurement agents configured on backbone POPs in ten hubs
– Randomized latency checks executed each hour
– RTT interval = completion of TCP handshake between client and server
TCP SYN
SYN + ACK
TCP ACK
3
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
3
Where the Data Was Collected
Seattle
New York
Chicago
Washington, DC
San Jose
Charlotte
Los Angeles
Atlanta
Dallas/Ft.
Worth
Houston
4
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
4
Is IPv6 Really Faster?
These visits to apple.com make it appear that is so…..
5
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
5
Is IPv6 Really Faster?
RTT for wikipedia.org is still measurably faster for IPv6 on average…..
6
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
6
Is IPv6 Really Faster?
But on Facebook RTT for IPv6 and IPv4 are nearly identical…..
7
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
7
Is IPv6 Really Faster?
Again, nearly identical results for Netflix…..
8
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
8
Is IPv6 Really Faster?
And this time IPv4 is significantly faster, by nearly 25 ms…..
9
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
9
Is IPv6 Really Faster?
IPv4 wins again. What are we to make of these results?
10
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
10
IPv4 vs IPv6 Aggregate View
11
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
11
Lab Testing
12
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
12
Lab Testing Methodology
• Conducted matched IPv4-IPv6 ping tests
over the course of an IPv6 interop
– First sets: 2 NATS enabled
– Second sets: 1 NAT enabled
• Identify effect of NAT without IPv4-IPv6
path differences
• Did NOT measure effects of traffic
engineering or differentiated peering
13
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
13
Results – One Layer of NAT
Common Home Network Scenario
• Could NAT be responsible for slower IPv4
performance?
• IPv6 performed measurably better than
IPv4
– Lower mean, median, standard deviation, and
minimum
– Statistically significant differences
14
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
14
Results – Two Layers of NAT
Possible CGN Effect
• What happened?
• One NAT showed differences, two NATs
show nearly identical results
– 81% chance that the means are really the same*
– IPv4 in both cases still shows higher variability
– IPv6 performed better half the time across 6 test
runs
• Could NAT performance vary per device?
*Based on Students t-test analysis
15
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
15
Interpreting The Results Data
• How do we explain the disparity in test results?
– Analysis shows a subtle IPv6 performance advantage
– However, there are more questions than answers….
• Some possible explanations:
–
–
–
–
Does NAT performance vary across some IPv4 platforms?
Is hop count a factor for IPv4 vs. IPv6?
How do network effects, such as tunnels, traffic engineering and congestion factor in?
What about Content Provider IPv6 transition strategies?
• Better tools and methodologies for gathering end-to-end performance
metrics are needed
• Bottom line – further research is needed
16
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
16
CableLabs Initiative
• CableLabs is developing performance measurement tools and
methodologies as part of our IP Performance Evaluation & Reporting
(CLIPPER) project
• Project will standardize testing methodologies for MSOs
– Aligns with IETF IPPM & LMAP and BBF WT-143 working groups
– Introduce common test platform
– Gather data on jitter, delay, packet loss, throughput and DNS response time for both
IPv4 and IPv6
– Better correlation of speed tests between home router and wireless devices
• Recommendation: Systematic performance metric collection should
consider differences between IPv6 and IPv4
17
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.
17
© Cable Television Laboratories, Inc. 2014. Do not share this material with anyone other than CableLabs Members, and vendors under CableLabs NDA if applicable.