Managed Bandwidth
Next Generation
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
1
Motivation
SuperJANET-3 offered a pilot “Managed Bandwidth” service based upon the old ATM
network.
~3 years ago UKERNA+PPNCG put together a joint project
- to pilot an equivalent service on the IP based SuperJANET-4
- to use the SuperJANET development network
Then came eScience initiative – giving opportunity to make advances in networking aimed
at applications (HEP, Radio Astronomy, Computational Science (HPC))
….so the project grew legs, and MB-NG was formulated ….
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
2
Industrial and research partners
Academic partners:
UCL, Manchester, RAL.
Industrial partners:
UKERNA, Cisco, Spirent and Acterna
Funded by EPSRC & PPARC
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
3
Achievements of the project
(..and talk outline…)
 Built a leading edge, multi-domain, QoS enabled network running at 2.5 Gbit/s
 Proved Carrier” and “Campus” class router equipment
 Demonstrated use of QoS in scientific environment
 A managed bandwidth solution for SuperJANET based on MPLS
 High performance data transport achieved and understood (leveraged DataTAG)
 24 hours > Gbit/s
 800 Mbits/s disk to disk for real data
 Demonstrated edge control of QoS (with sister project “GRS”)  see demo
 Successful External activities
 DataTAG
 QoS work with Geant/DANTE
 UKLIGHT
 RealityGrid and SC2003
 Dissemination
 Technical reports
 Conferences and papers
 A dissemination workshop in July
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
4
The MB–NG Network
Manchester
Domain
UCL Domain
SuperJANET
Development
Network
Cisco 7609
Routers
RAL
Domain
High
performance
servers
Cisco 12000
Carrier class
Routers
1 Gigabit Ethernet
2.5 Gigabit SDH
RAL Domain
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
5
Quality of Service in a nutshell
What is “Quality of Service” = QoS
= Packet Prioritisation
= Diffserv

IP packets can be “marked”
Layer 3
IPV4
Version ToS
Len ID Offset TTL Proto FCS IP-SA IP-DA Data
Length 1 Byte
7
6
5
4
3
IP Precedence
2
1
0
Unused
Bits;
DSCP

At each node (router)
in chain, marked packets
can given “business class”
treatment
21st April 04
Identify & Classify
Sort
EPSRC Open Call / NeSC / MB-NG e-science presentation
Dequeue
6
QoS in action
Results
Congestion
point 2.5Gbit/s
Background flow
• 1st long distance network running
QoS at 2.5 Gbit/s in UK
• Characterises equipment used in
SJ4 and Campuses
Priority flow
• Inform community:
•SJ5 QoS rollout
•Networkshop ..etc
•Technical Reports
• Used QoS for managed bandwidth
2.5 Gbit/s
background
Rate
Priority flow
• Many joint tests with Geant
• Piloted a “less than best efforts”
service”
Time
21st
April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
7
Managed Bandwidth for SJ5
• Problem statement: How to provide a managed (protected)
bandwidth service over the production IP network
• Background: UKERNA procured SJ5 routers with “MPLS”
(“Multi Protocol Label Switching”) to allow this.
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
8
“MPLS” in a nutshell
Label inserted between IP and frame
Headers
3
IP Headers
IP payload
Ethernet
Frame
Allows engineering of
route packets take by
switching on label #
Normal Traffic
Special Traffic
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
9
Managed Bandwidth
•Problem statement: How to provide a managed (protected)
bandwidth service over the production IP network
• Background: UKERNA procured SJ5 routers with “MPLS”
(“Multi Protocol Label Switching”) to allow this.
• Used MPLS+QoS to “carve out” protected pipes
MPLS+QoS “tunnel”
• Solution proven – and will be fully documented
• …however … events overtaken by UKLIGHT and the new
development network …
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
10
High Performance Data Transport
Client
Campus
The
“network”
Typically
2.5 – 10
Gbit/s
Regional
SuperJANET
(10 Gbit/s
User gets
10 -100
Mbit/s
To 1st order the wide
area network is not the
problem
Issues are:

TCP – the internet workhorse

Servers


Regional
Campus



Disk I/O
Network Cards
Architecture
OS
Local network and firewall
Server
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
11
TCP subject to 1 packet
drop per million
This is a very active research area,

Normal




Rate
High Speed
HS-TCP: HighSpeed TCP
S.
Floyd
http://www.icir.org/floyd/hstcp.html
Scalable TCP: T. Kelly http://wwwlce.eng.cam.ac.uk/~ctk21/scalable/
FAST-TCP: S. Low
http://netlab.caltech.edu/FAST/
H-TCP: Hamilton institute
http://hamilton.ie/net/
BIC-TCP: Binary Increase
Control I. Rhee
http://www.csc.ncsu.edu/faculty/rhee/
export/bitcp/
Time
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
12
MB-NG 24 hour High Speed TCP test
…using all the tricks …
940 Mbit/s
24 hour
memory-to-memory
transfer from
Manchester  UCL
(limited by 1 Gbit/s NIC)
Rate
Messages:
 Getting 1 Gbit/s out of
network alone is a known
solution
Time
 Actually, getting
5 Gbit/s out of network
has been done
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
13
…using real data ….
BaBar Data Transfers from Manchester to RAL
BaBar @ Manchester
Current transfers of 900 GBytes of BaBar
data from RAL to Manchester ~40 Hours
100 Mbit/s
The two rates are BBFTP “verbose and “non-verbose” mode
MB-NG
MB-NG does the same transfer in ~10 Hours
400 Mbit/s
CCLRC ATLAS Centre
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
14
Edge Control of Network Services : The GRS project
Application/GUI
Remote resource
1: requests
3: data
packets
NRSE
(Network Resource
Scheduling Entity)
NRSE
2: configure routers
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
15
Trailer for the demos
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
16
External activities & leverage
DataTAG
Research & Technological Development
for a
Data TransAtlantic Grid
EU funded project which followed from MB-NG
UK (CCLRC-DL) managed the QoS and High Performance work package
Recently concluded with great success at final review.
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
17
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
18
UKLIGHT
•
•
•
Global Optical R&D network
UKLIGHT brings UK into this
NorthernLi
ght
SunLight
UKLight
Supported by EPSRC eScience &
CERN
JISC
•
•
Czech
Light
Led to first UK R&D network
MB-NG ready to jump on first
connections
Pacific NW
GigaPOP
MANLAN
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
19
Extended JANET
Development Network
CA*net
FERMILAB
StarLight
Chicago
UKLight
London
10Gb/s
2.5Gb/s
10Gb/s
Abilene
CERN
10Gb/s
10Gb/s
10Gb/s
NetherLight
Amsterdam
CzechLight
GEANT
UKLight – showing connections to
selected International peer facilities
EPSRC Open Call / NeSC / MB-NG e-science presentation
21st April 04
20
Reality Grid & SC2003 in Phoenix
Mancheste
r
“Site”
MB - NG
21st April 04
SuperJANE
T
Dev net
UCL
“Site”
BT provision
RAL
“Site”
EPSRC Open Call / NeSC / MB-NG e-science presentation
21
Dissemination
Technical Reports (www.mb-ng.net)







Less than best efforts tests between UCL and INFN
Benchmarking of QoS and CISCO 12000 (GSR) Core Routers
A Map of the Networking Code in Linux Kernel 2.4.20
How to achieve Gigabit speeds with Linux
How To Set up QoS in Gigabit Networks
Benchmarking of CISCO 7600 (OSR) Enterprise Routers
MPLS configuration for managed bandwidth
Conferences





Networkshop 2003, 2004,
AHM 2003, 2004
PFLDnet Feb 2004,
GNEW, CERN Feb 2004
Terena Network Conference , June 2004
Papers
 Benchmarking QoS Network Interface Cards and Beyond, Eunice
 Quality of Service Networking of High Performance GRID Applications
(accepted for publication in Journal of Grid Computing)
 On the joint use of new TCP proposals and IPQoS on high BW-RTT product
paths (Accepted for TERENA Networking Conference 2004)
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
22
Dissemination
Dissemination workshop
“Networks for non-networkers:
Bringing high-performance to your application”
UCL , July 13-14
NeSC October ????
to attend email:nfnn@dl.ac.uk
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
23
Thank You…
http://www.mb-ng.net
21st April 04
EPSRC Open Call / NeSC / MB-NG e-science presentation
24