Specification of the Hardware Infrastructure for the MB-NG Project Draft 0.3 27 Nov 01 R. Hughes-Jones HEP Manchester P. Myers Network Group MCC This is a draft – Comments are very welcome. Introduction The purpose of this note is to set down the functionality and the specification of the IP network hardware infrastructure and the peripheral test components for the MB-NG Project. A “working model” of the connectivity is proposed to position the components. It attempts to draw on the discussions presented at the September meeting in London and the experience of managing regional and campus networks. The equipment must allow the MB-NG Project to meet its aims, that is to investigate and study three major areas: MPLS switching and engineering High Performance High Throughput data transfers Managed Bandwidth and QoS These areas may well need slightly different arrangements of the equipment or different configurations of the software involved. As well as conventional network statistic gathering, many of the tests will require status and control information from the equipment, e.g. test PCs & routers, to be monitored and recorded as the test proceeds. Thus we have the requirements: The equipment should be able to be used in a flexible manner. This indicates uniformity of kit were possible. The equipment should be IP accessible preferably via a path independent of the current links under test. Access to current configurations and statistics must be available to all the collaborators. The equipment at the C-PoPs should be able to be power-cycled and managed remotely. UKERNA already operate in this manner for the SuperJANET production network. Much of the kit will be situated on campus and laboratory sites and in general all the test traffic will flow on links independent of the IP production traffic. However access to the test equipment may well involve traversal of the campus network, and some of the demonstrations being considered might need traffic from production computing systems, e.g. the operation of AccessGrid video rooms over MB-NG QoS. It is clearly understood that were the work impinges on the site domains, no arrangements can be made without discussion and the full involvement of the network teams concerned. In fact we believe this input is essential to the smooth operation of the project. Similarly, issues that involve the C-PoPs will require discussion with UKERNA. Thus it is emphasised that at this point in time, this is very much a discussion document. Physical Network Infrastructure Figure 1 shows the SuperJANET4 Development Network Core together with the access links to the sites involved. Core Routers The Core routers are Cisco GSR 12416s with two 10 Gbit POS cards and one 4 port Gigabit Ethernet Card. These routers will be connected to the UKERNA ISDN based out of band power management and terminal access system. Access to this power management system will be restricted to the UKERNA NOCs. Access Links The access and cross-campus links have the following specifications: Manchester a 2.5G POS link from the Warrington C-PoP to MCC and a fibre to the HEP Group across Manchester Campus. RAL There is dark fibre to Reading, which could be used for Gigabit Ethernet. UCL a 2.5G POS link from the London St Pancreas C-PoP to ULCC and a 1 Gbit Ethernet link to UCL over the London University fibre Equipment for the External and Edge Domains Figure 2. shows the equipment for the external Backbone and Edge domains. It consists of a backbone router connected to the C-PoP development access link, an Edge router connected “over Campus” to the backbone router, and several hosts capable of generating test and load traffic. In addition IP access and possible 100 Mbit Ethernet connections to other computing systems must be provided. Edge Routers The edge routers must be able to: Connect to the test systems in the IP domain. Accept marked packets from the test systems and be able to re-mark them Mark the packets on input according to at least: Source – dest IP address IP port Layer3 and Layer4 headers TCP/UDP Support Diffserve code points … Perform policing and admission control Provide a range of Queue type options – Several queues with different priorities Real-time queue Provide flexible Queue Scheduling Weighted round robin Weighted fair queuing WorldCom Manc MCC SJ4 Dev C-PoP Warrington 12416 SuperJANET4 Production Network Gigabit Ethernet 2.5 Gbit POS 10 Gbit POS SJ4 Dev C-PoP Reading 12416 Dark Fibre (SSE) RAL / UKERNA Leeds RAL / UKERNA Figure 1. SuperJANET4 Development Network and the Site Access links for MB-NG SJ4 Dev C-PoP London 12416 WorldCom ULCC UCL Gigabit Ethernet 2.5 Gbit POS 10 Gbit POS IP or MPLS Load PC Test PC 10 Gbit POS Test PC 3 * 1 Gigabit Ethernet blade Test PC Load PC 2.5 Gbit POS 10 Gbit POS Test PC 4 * 1 Gigabit Ethernet blade QoS Marking 4 * 1 Gigabit Ethernet blade QoS Marking MPLS 2.5 Gbit POS MPLS MPLS Load PC Supervisor Supervisor Load PC SJDN Core MPLS Backbone Router 12416 Figure 2. Equipment for the External and Edge Domain MPLS Backbone Router OSR Difserve router OR MPLS Edge Router Test PC Load PC Provide Congestion control such as RED ECN Be able to assign MPLS labels to the packets in a flexible manner. As shown in Figure 2, the edge routers have one 4-port Gigabit Ethernet interface. Three ports are connected to test PCs and one to the Backbone router. Question to Cisco: Can one card perform the IP and MPLS functions we need or do we require two cards – one for IP input functions, the other for MPLS output? Question to Cisco: what is the performance of these 4-port Gigabit Ethernet interfaces? The supervisor cards have 2 Gigabit Ethernet ports but it is believed these do not support the required options, they can be used for Load PCs and/or IP access to the router. For the High Performance High Throughput tests, and for future MAN operation, it is important to study the operation and performance of a 2.5 Gbit POS link. To do this it is proposed that Manchester connect the Edge and Backbone routers with a POS link across campus. Study of transport protocols at Gigabit rates will require knowledge of the packet headers, it is proposed to use 3508 Gigabit Ethernet switches to monitor the flows between two other ports in the switch, directing the traffic to a dedicated monitor PC to record the information. External Backbone Routers The Backbone routers must be able to support essentially the same features as the Edge routers but in the MPLS domain. [We should clarify the details here. Rich] There is also a requirement for these routers to be connected to test or load PCs so that cross traffic (similar priority to the test traffic ) or background traffic (of lower importance) may be injected or sinked at the router ports. Some of this traffic may be directed towards the Edge router and some towards the Development Core. The aim is to load the links causing queuing. Question to Cisco: Can the 6500 OSR switch IP cross traffic (ports a->b) as well as MPLS (ports c->d) ? What about mixing IP and MPLS on a port? What are the load implications for the router blades / CPU card? Test Hosts It is proposed that 1 GHz PCs with 64bit 66 MHz PCI bus and >=133 MHz front side bus running Linux be used, these motherboards include 100Mbit Ethernet interfaces. The SysKonnect Gigabit NICs will be tested for suitability. [More details to follow - Rich] The groups of PC test systems should have a Keyboard Mouse Video switch and screen / keyboard to provide suitable local access. IP Access to the External Systems To provide IP access it is proposed to connect the PCs and their local router to the campus network using a Cisco 3512 switch as shown in Figure 3. Some care is required with the configuration; the following arrangement is suggested: The default route from the PC is set to the Gigabit link to the development network. Static routes for the researchers only are set for the 100 Mbit interface, e.g. to Manchester HEP, MCC, RAL, UCL, ULCC, UKERNA. The link between the Cisco 3512 and the Campus is set to 10 Mbit – this would minimise any unintentional traffic. 4 * 1 Gigabit Ethernet blade QoS Marking Test PC 100 Mbit switch Test PC Cisco 3512 Load PC No routing traffic Supervisor Load PC Backbone or Edge Router Gigabit Ethernet 100 Mbit Ethernet 10 Mbit Ethernet Figure3. A possible way to provide IP access to the MB-NG test systems. To Campus Network IP Access to the Core Routers Loging into a test PC or external router would provide in-band access to the SJDN Core routers. With care the access could be via a non-stressed route e.g if testing was between London and Manchester access could be from RAL. An independent route might well be advantageous. Initial Kit List C-PoP GSR kit Backbone OSR kit Edge OSR kit MCC Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch 1 GSR POS 1 1 1 2 1 M/c HEP Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch 3508 Eht. switch 1 1 1 1 1 RAL1 RAL2 Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch 3508 Eht. switch 1 GSR 3 GigEth 1 1 1 1 1 1 1 1 ULCC/UCL1 Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch 1 GSR POS 1 1 1 1 1 UCL2 Chassis Supv etc OSR Ethernet OSR 2.5G POS 3512 Eht. switch 3508 Eht. switch 1 1 1 1