Study on existing Last Mile solutions for circuit
emulation applied to research environments
Version 3.1
November 2010
Jordi Jofre1, Joan A. García-Espín1,
Ana M. Medina2,
Ronald van der Pol3, Pieter de Boer3, Igor Idziejczak3, Sander Boele3
1
Network Technologies Cluster, i2CAT Foundation (Barcelona, Spain)
2
Network Area, RedIRIS (Madrid, Spain)
3
SARA Computing and Networking Services (Amsterdam, Netherlands)
Table of Contents
1.
Introduction ..................................................................... 4
2.
Lambda Station ................................................................ 5
2.1. Description..................................................................................... 5
2.2. Working principles ......................................................................... 5
2.3. Promoters ...................................................................................... 8
2.4. Last Activities ................................................................................. 8
2.5. Users .............................................................................................. 8
2.6. Conclusions .................................................................................... 8
3.
TeraPaths ......................................................................... 9
3.1. Description..................................................................................... 9
3.2. Working principles ......................................................................... 9
3.3. Promoters .................................................................................... 12
3.4. Last Activities ............................................................................... 12
3.5. Users ............................................................................................ 12
3.6. Conclusions .................................................................................. 13
4.
Phoebus ......................................................................... 13
4.1. Description................................................................................... 13
4.2. Working principles ....................................................................... 14
4.3. Promoters .................................................................................... 15
4.4. Last Activities ............................................................................... 16
4.5. Users ............................................................................................ 16
4.6. Conclusions .................................................................................. 16
5.
Virtual Routing and Forwarding (VRF) ............................. 16
5.1. Description................................................................................... 16
5.2. Working principles ....................................................................... 18
5.3. Promoters .................................................................................... 20
5.4. Last Activities ............................................................................... 20
5.5. Users ............................................................................................ 20
5.6. Conclusions .................................................................................. 20
6.
Generalizaed Token Based Networking (gTBN) ............... 21
6.1. Token Based Networking concepts .............................................. 21
6.2. Previous work on Token Based Networking ................................. 21
6.3. Last Mile Token Based Networking .............................................. 23
6.3.1.
Architecture .............................................................................................. 23
6.3.1.1.
Token Builder (TB) ....................................................................................... 24
6.3.1.2.
Token Based Switch (TBS-IP) ....................................................................... 25
6.3.1.3.
AAA server ................................................................................................... 26
6.3.1.4.
Use Case: User submits a request ............................................................... 26
6.3.2.
Integration with AutoBAHN ...................................................................... 27
6.4. Promoters .................................................................................... 28
6.5. Last Activities ............................................................................... 28
6.6. Users ............................................................................................ 29
6.7. Conclusions .................................................................................. 29
7.
Comparison Table .......................................................... 30
1.
Introduction
Connectionless networks, such as the ones based on TCP/IP protocol stack, are present in most
of the premises where scientists run their experiments nowadays (distributed research
institutes laboratories or university campuses, among others). Almost all applications requiring
network connectivity rely on packet networks for inter-process communication. Cutting edge
research activities need to ensure their applications get the precise data in the precise
moment, leading to experiment failure otherwise, which in many cases is direct synonym of
time and money losses.
Since permanent connections are expensive and not as scalable as desired, dynamic ondemand circuit emulation in the access network is foreseen to provide a solution. Therefore,
there is the necessity to reduce the gap between end-user applications and bandwidth ondemand provisioning systems that operate in the core networks to be able to bring dynamic
circuits until end-user sites.
Hybrid networking concepts within networks as SURFnet, Internet2, Dynamic Circuit Network
(DCN), CA*Net, UCLP, G-Lambda and GEANT AutoBAHN allow applications to reserve and use
circuits on demand. Within these networks it is however unclear how to bring those dynamic
circuits until end-user hosts.
This study is born from a necessity to reduce the gap between end-user applications and
bandwidth on-demand provisioning systems (last mile network section) that operate in the
core networks. A token-based networking architecture is described for overcoming the last
mile issue. The document revisits existing work done in PHOSPORUS project and proposes a
new definition in the context of a Last Mile solution.
This study is born from a necessity to reduce the gap between end-user applications and
bandwidth on-demand provisioning systems that operate in the core networks. Industry
solutions are considered as objects for comparison, but do not fit the needs of GÉANT3 project
and thus are discarded. Scientific-oriented solutions for overcoming the last mile issue (or
“first mile”, seen from user’s perspective) have focused on:



TCP/IP protocol stack enhancement
Low layer circuit provisioning, mainly in layer 2
High performance network processors at the edges
Considering these areas, the set of solutions under consideration in this study are:





Lambda Station
Terapaths
Phoebus
Virtual Routing
generalizedToken-Based Networking
2.
Lambda Station
2.1.
Description
The upcoming era of LHC high data volume distributed GRID computing implies extra
requirements on the resource management and network aware applications. There is a large
demand for applications that are capable to dynamically allocate a high performance network
path with predictable performance characteristics. Over the past several years, there has been
a great deal of research effort and funding put into the deployment of optical-based, advanced
research networks, such as National Lambda Rail, DataTag, CAnet4, Netherlight, UKLight, and
most recently, the DOE Science UltraNet.
The Lambda Station project is aimed to enable dynamic allocation of alternate network paths
for traffic of production SciDAC applications and to forward designated flows across LAN,
negotiates with reservation and provisioning systems of WAN control planes. It creates EndTo-End circuit between single hosts, computer farms or networks with predictable
performance characteristics. Lambda Station project also explores Network Awareness
capabilities.
2.2.
Working principles
Lambda Station is a service that addresses the need for dynamic reconfiguration of the local
network infrastructure to enable use of high bandwidth, alternate network paths by specific
applications for designated traffic flows. The next Figure 1 illustrates the main idea of the
project:
Figure 1. Lambda Station architecture
How it works: local applications requests high bandwidth WAN paths that aren’t available over
the normal production WAN network path. Having received such a request, Lambda Station
would negotiate the availability of the alternate WAN path, and if available, coordinate its
setup and teardown. Since these advanced research networks are typically restricted-use
resources, Lambda Station could schedule their use, if necessary. Finally, Lambda Station
would coordinate dynamic reconfiguration of the local network infrastructure to selectively
forward the application’s traffic over the alternate WAN path.
The Lambda Station (LS) is a host running special software capable of interact to the storage
system, local network infrastructure and advanced WAN. This SW consists in five modules:

Local Demand module: The interface to host systems or resource manager. Accepts
alternate WAN paths requests, coordinates flow identification keys (i.e.,
source/destination addresses and ports) and provides state information on alternate
path to requesting end system (setup status, bandwidth, etc.).

Local Steering Service: Customizable interface to local network infrastructure. Modifies
local forwarding policies for each specific flow.

WAN module: Customizable interface to alternate path networks. It checks alternate
path availability and negotiates alternate WAN path setup/teardown.

Peer Lambda Station module: Coordinates with peer Lambda Station at remote site to
establish path symmetry. Exchanges flow identification selectors. Acknowledges WAN
setup/teardown, and LAN forwarding reconfiguration. Verifies alternate WAN path
continuity. Provides notification of WAN path or traffic termination.

Authentication/Authorization module: Provide AA for alternate path requests.
Figure 2. Components of Lambda Station
Lambda Station must be able to forward traffic on a per-flow basis. That’s why LS identifies
PBR clients, which means entities that generate flows of traffic that could be subjected to
policy based routing (PBR). So, each PBR client has a group of attributes: source or destination
addresses, ports, ranges, etc. Hence, LS is capable of dynamically reconfiguring local network
devices based on these attributes.
Initially, flow identification is based on DSCP (differentiated services code point) and Lambda
Station software does support two different modes to work with DSCP:

In the first mode, a site may choose to use fixed DSCP values to identify all traffic that
will be switched by LS. Then, Lambda Station advises applications when to apply that
DSCP value, and router configurations remain constant. This mode will typically be
used by sites that do not want their network devices dynamically reconfigured under
Lambda Station's control.

In the second mode, a DSCP value is assigned on a per ticket basis by the local LS. The
same DSCP code can be used by multiple tickets as long as the source and/or
destination IP addresses are used as additional flow selectors.
Once, LS has identified the flow (PBR client), needs to reconfigure network devices with a set
of rules (PBR rules). Configuring these rules involves creating route map statements, applying
them to appropriate interfaces and creating access control list to match traffic. The changes
must be applied for one or both directions of traffic. At the moment, LS uses statically preconfigured route map statements. However, PBR rules should be created dynamically on
demand of applications. At this point, Lambda Station deals with the last-mile problem.
Typically, a campus network can be presented by a hierarchical design with several logical
layers. It consists of work group, core and border layers. Depending on site's specific network
structure, access to R&D networks (Research and Dynamic Networks) for different user groups
may need to be configured at several layers. For the architecture in Figure 3 below, outbound
traffic of WG-B can be switched to R&D networks at the work group layer because it has a
direct connection to the R&D admission devices. In order to get incoming traffic from R&D
networks forwarded via a symmetric path, the inbound route for WG-B needs to be configured
at the R&D layer. WG-A has no direct connection to R&D from its work group layer, so PBR
rules must be applied at the network core and R&D layer for both inbound and outbound
traffic.
Figure 3. A hierarchical network model
2.3.



2.4.


2.5.


2.6.
Promoters
U.S. Departament of Energy (DOE)
Fermi National Accelerator Laboratory (Fermilab)
California Institute of Technology (CALTECH)
Last Activities
JointTech/ESCC Winter 208, oral presentation, Honolulu, Hl, January 20-25, 2008
CHEP07, oral presentation, Victoria BC, Canada, September 2-4, 2007
Users
Fermilab Storage Resource Manager, Tier1 SRM
Center and Computing for LHC Physics in the US, CMS Tier1
Conclusions
The initial release of Lambda Station Software (1.0) is a fully functional prototype, including
interactions between: applications and Lambda Stations, Lambda Station and the site LAN and
pairs of Lambda Stations, but no interoperability across different platform.
This software uses multiple API-calls or primitives, but we focus this section describing an
operational function: openServiceTicket, used by applications and remote Lambda Stations to
request an alternative network path.
In the simplest case, an application or a remote Lambda Station places an openServiceTicket
request, and specifies source and destination PBR clients, desired bandwidth, boarding (a time
when Lambda Station can begin configuring the network), start and end times for data
movement. A unique ID will be returned immediately in response to an authenticated and
authorized openServiceTicket request. This ID can be used by applications to track the status of
path provisioning, getting additional information needed for flow marking, e.g. DSCP assigned
by remote Lambda Station to the corresponding ticket at its end, as well as to synchronize
actions with the remote site if, for example, the remote application cancels the ticket.
The way to deal with last-mile problem will be to apply the same set of policy rules to a logical
grouping of devices. As long as the same PBR rules are applied on several layers of hierarchical
work group architecture Lambda Station network model can be represented by only one group
of devices.
At this time, a site’s infrastructure must be configured for PBR and described by the network
administrator in the configuration of a site’s Lambda Station. There is a work in progress to
implement a feature that allows LS to discover a site’s PBR configuration automatically.
3.
TeraPaths
3.1.
Description
TeraPaths is a U.S. Department of Energy (DOE)-funded project conceived to address the
needs of the high energy and nuclear physics scientific community for effectively protecting
data flows of various levels of priority through modern high-speed networks. The primary
motivation of the project comes from the world of modern high energy and nuclear physics
(RHIC, LHC, ATLAS, U.S. ATLAS, CMS), where extremely large quantities of experimental and
analysis data need to be transferred through high-speed networks across the globe, to be
shared among scientists participating in various experiments
Despite the fact that not all network flows are of equal priority and/or importance, the default
behaviour of the network is to treat them as equal priority. Thus, the competition among flows
for network bandwidth can cause severe slowdowns for all flows, independent of their
importance. TeraPaths offers the capability to distinguish between various data flows and
enable the network to treat each flow differently in a pre-determined way through the use of
the differentiated networking services architecture (DiffServ).
The TeraPaths project at Brookhaven National Laboratory (BNL) investigates the combination
of DiffServ-based LAN QoS with WAN MPLS tunnels in creating end-to-end (host-to-host)
virtual paths with bandwidth guarantees. These virtual paths prioritize, protect, and throttle
network flows in accordance with site agreements and user requests, and prevent the
disruptive effects that conventional network flows can cause in one another. TeraPaths
automates the establishment of network paths with QoS guarantees between end sites by
configuring their corresponding LANs and requesting MPLS paths through WANs on behalf of
end users.
3.2.
Working principles
TeraPaths offers the capability to distinguish between various data flows and enable the
network treat each flow differently in a pre-determined way through the use of the
differentiated networking services architecture.
The network devices of the LAN of each end site are under control of a TeraPaths system
instance. They follow the approach of conceptually dividing the end-to-end route between a
source and destination end-sites in to three significant segments:
1) The segment within the LAN of the source end-site: from the host where the data
reside at the beginning of a transfer to the site’s border router.
2) The segment within the LAN of the destination end-site: from this site’s border
router to the host where the data will arrive.
3) The segment connecting the source and destination site border routers (WAN): This
may consist of multiple network segments belonging to different administrative
domains.
Figure 4. End-to-end route segment division
Terapaths follows a hybrid star/daisy chain setup model, where the initiating end-site
coordinates with the target site. If a mutual agreement is achieved, the initiating system
contacts the appropriate WAN provider and relying on that provider to coordinate, if
necessary, with other WAN domains along the desired route. The configuration of the path is
successful if all three parties are successful in configuring their segment of the path.
Figure 5. Star/daisy chain hybrid model
The Terapaths software configures and manages LAN QoS paths from end host to border
routers. Terapaths partitions an end-site’s bandwidth into multiple classes of service with
different priorities with statically or dynamically assigned bandwidth. DiffServ architecture is a
packet-oriented approach and the distinction between data packets is done by means of their
Differentiated Service Code Point (DSCP) markings. Packets receive different treatment
according to their DSCP (utilizes six bits of a header’s ToS IP byte). Traffic needs to be
conditioned (policed/shaped/marked) only at network boundaries. Therefore, the host routers
play the role of control valves, throttling and marking data flows outgoing from host and
incoming to the site’s network. The rest of the network simply honours the packet markings all
the way out the site and into the WAN.
The TeraPaths system does not have direct control over the network devices of the WAN route
segment. For TeraPaths, the WAN segment is a “cloud” through which packets travel from
source to destination. The cloud representation is relevant in the sense that the WAN domain
is an independent entity that can be contacted in a number of ways to make arrangements for
a specific data flow.
End-sites are clients to WAN providers and have the capability to request special treatment for
specific data flows to the degree that their provider allows. Furthermore, WAN providers are
likely to adopt service level agreements between them so as to make certain QoS options
available to their clients.
The automated configuration of the WAN providers themselves is more complex, requiring the
existence of suitable software mechanism exposed by the WAN providers and collaboration
between the responsible parties. TeraPaths is designed to request guaranteed bandwidth
WAN paths by invoking the web services of a WAN provider. To deal with any number of
different WAN providers web service interface implementations, as well as possible
compatibility issues, TeraPaths uses façade objects in the form of proxy server modules. Each
such module, one per provider, wraps the services the corresponding providers makes
available and exposes a common interface compatible to the core TeraPaths module
requirements. For instance, is here where AutoBAHN should provide an interface if wants to
use TeraPaths.
The level of QoS that is possible within WAN segments affects the level of QoS guarantees that
TeraPaths can provide. Typical cases are the following:
-No WAN QoS available: Prioritization/protection is possible only within end site LANs,
while the flow will be treated as standard “best effort” traffic in the WAN. If the WAN
becomes increasingly loaded, the requested bandwidth may not be honoured due to
the lack of QoS
-WAN QoS statically available: End-to-end data flow prioritization/protection is
possible; however, the WAN providers along a route have to agree to configure in
advance their network devices using DiffServ techniques similar to those at the end
sites or MPLS.
-WAN MPLS tunnel dynamic configuration: End-to-end prioritization/protection is
possible on demand. This case assumes that WAN providers have publicly available
services allowing the dynamic configuration of MPLS tunnels between specific endsites. A user is able to set up a virtual path with flow-level granularity
Figure 6. End-to-end path setup
The TeraPaths project demonstrate that the combination of LAN QoS techniques, based on the
DiffServ architecture combined with WAN MPLS tunnels, is a feasible and reliable approach to
providing end-to-end, dedicated bandwidth path to data flows in a demanding, distributed,
computational environment. Furthermore, TeraPaths technology offers a flexible way to
partition a site’s available bandwidth into predetermined bandwidth slots, and to protect
various data flows from competing against each other.
3.3.


U.S. Department of Energy (DOE)
Brookhaven National Laboratory (BNL)
3.4.


Last Activities
TeraPaths demo in SC07: International Conference for High Performance Computing,
Networking, Storage and Analysis.
GridNets 2006.
3.5.







Promoters
Users
The Relativistic Heavy Ion Collider (RHIC) at BNL
The Large Hadron Collider (LHC)
The ATLAS experiment
The USATLAS project
The CMS experiment
The ESnet OSCARS project
Ultralight project
3.6.
Conclusions
TeraPaths solution is compatible with AutoBAHN’s needs. TeraPaths follow the approach of
conceptually dividing the end-to-end route between a source and a destination end sites in to
three significant segments:
- The segment within the LAN of the source end-site.
- The segment connecting the source and destination site border routers (WAN).
- The segment within the LAN of the destination end-site.
For TeraPaths, the WAN segment is a “cloud” through which packets travel from source to
destination. The cloud representation is relevant in the sense that the WAN domain is an
independent entity. This WAN segment is where AutoBAHN set-up the edge-to-edge
connections.
This solution can be deployed implementing a TeraPaths instance in each local network
connected to an NREN and providing an interface between TeraPaths and AutoBAHN.
4.
Phoebus
4.1.
Description
Phoebus is an environment for high-performance optical networks that seamlessly creates
various adaptation points in the network to maximize performance. By splitting the network
path into distinct segments, Phoebus minimizes the impact of packet loss and latency by
leveraging the best performance attributes of each network segment. Using an end-to-end
session protocol, transport and signalling adaptation points can be controlled and better
performance is possible. The Phoebus adaptation library also allows existing applications to
take advantage of advanced networks with little or no modification. The Phoebus project aims
to help bridge the performance gap by bringing revolutionary networks like Internet2's
Dynamic Circuit Network (DCN) Infrastructure to users. Many applications can begin to use
DCS (Dynamic Circuit Services) with no modification by being transparently authenticated and
redirected to the circuit network via a Phoebus service node.
Martin Swany, Assistant Professor, Department of Computer and Information Sciences,
University of Delaware and Internet2 faculty fellow explained, "Due to the huge success and
growth of the Internet, it has been difficult to deploy disruptive technologies broadly. Phoebus
(developed by researchers at the University of Delaware) builds on standard Internet
infrastructure at the edge of the network, while creating a bridge to advanced optical and
packet networks like the new Internet2 network, enabling application communities like
bioinformatics and satellite/telescope data to benefit from hybrid networking in the short
term. In the longer term, Phoebus represents an architectural evolution of the Internet in
which commercial network providers can offer a richer set of services to their users because
they will no longer be constrained by the performance limitations of TCP."
4.2.
Working principles
Phoebus works by changing the current Internet model, which binds all end-to-end
communication to Transport layer protocols such as TCP and UDP. Using an end-to-end layer
called the Session layer, Phoebus can segment a connection into a series of Transport layer
connections. The edge connections can still use TCP and in many cases, existing software
works with no modification. These edge connections are serviced by Phoebus Gateways (PG).
The flow of traffic over the backbone network is handled by a series of these gateways. The
gateways can choose the best protocol and settings for sending the data across the network,
and can allocate additional network capacity on behalf of the user.
Figure 7. TCP model vs. Phoebus model
So Phoebus Gateways are placed at network access points and can be thought of as network
“on-ramps”. These gateways can then intelligently move the data as efficiently as possible to
the other edge of the network. At the other edge, data are received as regular TCP/IP traffic,
thus destination client does not need to be modified. It’s important to note that this method is
not altering the path, just terminating TCP connections and providing short-term buffering,
also known as “cascading TCP connections”.
TCP on its own, as end to end protocol cannot mitigate the problem of the edge networks. TCP
is heavily dependent on the RTT (Round-Trip Time). RTT is one of the most important
performance parameters in a TCP exchange. It’s defined as the time required for a host
transmits a TCP packet to its peer, plus the time for an acknowledgment to be received. If the
reply does not come within the expected period, the packet is assumed to have been lost and
the data is retransmitted. RTT time is also known as the ping time. It’s a big factor in helping
how to deal with timeouts and retransmissions and may be used to find the best possible
route, due to a decrease in the RTT can produce an increase in throughput.
In Phoebus context, the RTT of an individual connection is less than the RTT of the end-to-end
connection. So, establishing two TCP connections, each having a shorter RTT, the speed of
each of the shorter connections should be faster than the speed of the overall connection. If
this is the case, the throughput of the overall connection in the Phoebus case should run close
to the speed of the slowest individual connection.
On the other hand, when a loss occurs on an end-to-end connection, it causes a reduction in
the throughput for the entire connection. However, with the shorter, independent
connections, the throughput is only dropped for one portion of the connection. The others
continue running at the same speed. This works to prevent intermittent drops from
substantially affecting the entire connection.
In the Phoebus model, the end-to-end connection is broken into at least three segments: a
short RTT connection between the end user and the edge network PG, the negotiated inter-PG
connections and the connection from the far-side edge PG to the destination host. Also, in the
Phoebus model, once a PG has received the data, it becomes responsible for ensuring that that
data is received at the next hop. These two properties combine to provide for improved endto-end performance.
Thus, we can think of it as controlling an end-to-end session comprised of segment-specific
transport protocols and signalling technology. That’s the function of Phoebus Session Protocol
(PSP), used to manage a multi-layer connection.
Figure 8. Phoebus Session Protocol on protocols stack
After these previous concepts, the main point is the software that runs on PG and clients. The
current version of the software is 0.9 and is integrated with the DC Network Software Suite.
Once, this software is installed in PG, these gateways are able to signal the cross-domain
circuit infrastructure. In clients, Phoebus can be enabled on Linux systems (Windows support
under investigation) and it not involves modifications to the users’ workstations.
Figure 9. Phoebus Gateway communication
4.3.


Promoters
Internet2
University of Delaware, Newark

Recently: GEANT2 and ESNET
4.4.


Peformance Update at Winter 2008 Joint Techs Workshop
Supercomputing SC2007
4.5.


Last Activities
Users
GridFTP from the Globus project
The Large Hadron Collider (LHC)
4.6.
Conclusions
Phoebus model separates the whole end-to-end connection in segments: source host – PG,
inter-PG connections and PG – destination host. The point is to make short TCP connections to
get good throughput. That’s how PSP (Phoebus Session Protocol) works. This new protocol
requires that applications can use these new features.
Then, user is able to send data via these PGs withous extensive host tuning. Phoebus solves
last mile issue locating the PGs at the edge of backbone networks and using two methods to
establish connections to them:
5.

Wrapper Library. Overriding the standard operating system socket calls to that any
application linked against it, can transparently use Phoebus infrastructure. The only
requeriment from the users perspective is that he must install the software package
and then set two environmental variables: one to load the library and another to
specify the first PG along the path.

Transparent redirection. Enabling applications to use Phoebus infrastructure available
in Linux. In this scenario, the administrator would setup a machine on his network
running the Phoebus code. The end users would then route their traffic through this
box. The administrator could then setup some firewall rules that would transparently
redirect TCP connections destined for specific hosts or services through the forwarding
software. The end user would no longer need to perform any local changes except to
route their traffic through the forwarding box.
Virtual Routing and Forwarding (VRF)
5.1.
Description
If the campus network supports VRF, this solution requires minimal changes. VRF needs to be
enabled only on a router inside the campus, which is in the IP routing path between end host
and the light path. No changes are needed on the end hosts. Packets are routed between the
VRF router and the end host via the generic campus infrastructure using normal IP routing.
This solution does not provide explicit QoS. Usually, this is not a problem inside a campus
network because over provisioning in a LAN is easy and cheap. However, there is nothing that
prevents QoS technology to be used in combination with virtual routing.
Light paths are often used to build Optical Private Networks (OPNs). Many of these OPNs have
a strict use policy. E.g. only traffic related to the project is allowed to use the OPN; all other
traffic must be routed via other paths. Figure shows an example of policy requirements:
Figure 10. Example traffic requirement/flows (this picture does not show all possible
combinations)
In the above example both the red and blue lines represent valid traffic, where red lines use an
OPN, while blue lines do not. As you can see, all hosts are reachable via the generic routed
internet. The OPN links may only be used if both the source and destination host are part of
that OPN. The table below shows an overview of the allowed traffic flows between hosts at
site 1 and site 2 is allowed over the OPN A. This is also shown in Figure
Host related to OPN A Other host at site 2
at site 2
Host related to OPN A Allowed over OPN A Not allowed over OPN
and general internet (if A, should use general
on site 1
available)1
internet
Other host on Site 1
1
Not allowed over OPN Not allowed over OPN
This might be policy related, if the desire is there to have a backup via the general routed internet this
can be use, but it could also be a policy to explicitly not use the general routed internet.
A, should use general A, should use general
internet
internet
So, only traffic between a host related to OPN A on both site 1 and 2 is allowed to use OPN A.
The host may optionally use the general internet as backup. Any other host at site 1 is not
allowed to use OPN A to reach a host related to OPN A at site 2. These hosts should use the
general internet instead. We could extend this scheme with multiple other OPNs where a host
can even be part of multiple OPNs. But the hosts can only use an OPN if both the sending and
the receiving host are part of it; otherwise they should use the general internet.
Segregating traffic from different OPNs is a challenge in itself. Where normal routers do not
care about policies and just focus on the shortest route from A to B, OPNs with use policies
have a requirement to evaluate the source address in the decision for the best route. This
could be solved by simply by buying dedicated routers and configuring static routes, but this is
expensive and does not scale. Another solution is to use VRF by placing interfaces and links of
an OPN into a VRF. VRF is a technology that allows a router to maintain multiple routing tables
or instances. It is even possible to create views where a host or a group of hosts can have
access to multiple OPNs and on the other hand maintain their visibility to/from the general
internet, and thus not becoming exclusively part of an OPN.
5.2.
Working principles
To illustrate a VRF setup, Juniper’s VRF technology will be described. With VRF, multiple
routing table instances are used. Those routing tables are used to send traffic to the
corresponding interface. Figure 1 shows an example of the policies of various light path
services.
For example, only OPN related traffic is allowed on the OPN A between Site 1 and Site 2.
Traffic of Site 1 users accessing the Site 2 website should go via the routed internet. Or users at
Site 3 are not allowed to use the routed internet via Site 1. These policies can be enforced
quite elegantly with VRF.
A router with VRF enabled has several routing table instances. Each routing table instance has
one or more interfaces assigned to it. The routing table instance is used for traffic on one of
those interfaces. One can think of it as a virtual router with a routing table and a couple of
interfaces. The challenge is to make resources available in multiple VRF’s but maintain the
separation of routing between different VRFs.
Consider, for example, the following setup where a privileged host on some network is allowed
access to a site through an OPN or dynamic light path whereas an unprivileged host is not. The
networks these two hosts belong to are routed by a VRF enabled router.
Figure 11. Example traffic flow and route distribution diagram
Traffic originating from the privileged host enters the router and immediately hits a routing
policy that states that all route lookups are to be done in the routing table of VRF shared. This
VRF shared is the glue between the global connectivity the privileged host requires and the
OPN the host needs exclusive access to. This is achieved by importing all OPN routes from VRF
A to VRF shared (3) and by installing a default route in VRF shared (4) for global connectivity of
the privileged host.
VRF A contains only static routes for traffic destined for the privileged host (2) and dynamic
routes learned via BGP to site B (which can also be static routes). This ensures the OPN is kept
private without a default or full routing in its routing table. This means that traffic from site B
can never go anywhere but to the privileged host, simply because there are no routes to other
networks in the routing table of VRF A. The OPN or dynamic lightpath is thus exclusively used
between site B and the privileged host.
Consider traffic originating from the privileged host that is destined for site B. This traffic hits
the routing policy, so the router does a lookup in the routing table in VRF shared. It finds a
route via the OPN and takes the shortcut through the OPN to site B (1). Return traffic that is
routed by site B via the OPN is looked up in the routing table of VRF A and is statically routed
to the privileged host (2).
Traffic from the privileged host destined for other networks than the ones at site B hit a
default route in VRF shared and is routed via the internet (6). Because the privileged host’s
interface is a connected route in the global routing table, traffic from the internet can reach
the privileged host without a hitch.
Traffic from the unprivileged host that is connected to the same VRF enabled router is routed
through global routing (5).
The previous example is based on a static lightpath setup. But the same VRF mechanism can be
used in the case of dynamic lightpaths. A nice example is a cloud by-passing setup where traffic
is normally sent over the routed internet. But when a lot of data needs to be transferred, a
dynamic lightpath is set up as a temporary high bandwidth channel. This can be handled by
putting the interface connected to the dynamic service in a dedicated VRF (VRF A in our
example). A static default route pointing to the global routing table is added to this VRF. BGP is
used to receive routes from the dynamic service. As long as the dynamic lightpath is not
established, the default route entry is the only active route entry. Because this entry is
pointing to the global routing table, the traffic is send over the routed internet. When a
dynamic lightpath is set up, the BGP peering becomes active and routes from the peer are
received. These routes will be inserted in the VRF Shared and these more specific routes take
precedence over the default route. The result is that the traffic will flow via the dynamic
lightpath.
5.3.

Dutch Tier1 (SARA and Nikhef)
5.4.

Last Activities
VRF is present in most current high-end routers.
5.5.

Promoters
Users
Used in production network of SARA Computing and Network Services for LHCOPN,
DEISA,
LOFAR,
etc.:
http://www.terena.org/activities/e2e/ws2/slides1/1_SARA_Peter.pdf
and
http://indico.cern.ch/getFile.py/access?contribId=11&resId=0&materialId=slides&conf
Id=50345
5.6.
Conclusions
Using VRF’s to separate traffic and with that complying to use policies of OPN’s, a very flexible
and scalable setup can be build. Using this technology it is possible to provide access for a host
to multiple OPN’s, while the next host only has access to one OPN and another doesn’t have
access to any. Still complete reach ability of the nodes from the general routed internet is
ensured. Scalability is only limited by configuration, while for a setup with hardware enforced
separation a new router needs to be bought for every OPN.
6.
Generalizaed Token Based Networking (gTBN)
6.1.
Token Based Networking concepts
Token based networking (TBN) is a new concept for a network that handles packets according
to a specific tag, built-in the packet, called token. TBN uses tokens built from small encrypted
pieces of data. By its mean, a token allows for applying different access / enforcement/
filtering rules/criteria when processing the packets that carry the tokens. For instance, a token
may represent an exclusive right to access a service at a particular for a measured amount of
time.
The word token is an overloaded term. The term is likely to create confusion if we do not
define it in the context of our research. A token in this context is defined as: “A shared abstract
permission that is presented as part of an access request”. The permission is a small piece of
encrypted data that unambiguously references the specific resource request context.
A policy enforcement point (Token based Switch) authorize the resource usage (e.g., a packet
takes a certain path) only if the built-in token (Token Builder) matches a cryptographic result
applied to the packet using a set of keys provided beforehand by an authority (AAA server).
Tokens can bind to different semantics (in particular, it can be associated with one user, a
group of users, a institute or a grid and share it use when accessing designed resources), and
they decouple the time of authorisation decision from the time of use (e.g., enable the use of
resource for 10 hours from now or from tomorrow at 5 o'clock). Hence, a token provides a
generic way to match applications to their associated network services.
Moreover, tokens are protocol independent, which has an important advantage; the token can
be regarded as an aggregation identifier to a network service. Generally, we see four types of
aggregation identifiers that can be combined, as follows:

Identifier to point a service to the NE (multicast or transcoding)

Identifier that defines the service consumer (grid application)

Identifier that defines the serviced object (network stream)

Identifier that defines the QoS (security, authorization, robustness, deterministic
property)
6.2.
Previous work on Token Based Networking
A Token Based Networking architecture is implemented in UvA under the PHOSPORUS FP6
project. The Token Based Network (TBN) testbed uses Token Based Switch over IP (TBS-IP) as a
low-level system for traffic switching at high speeds (multi gigabits/sec) based on packet
authentication. TBS-IP is fast and safe and uses the latest network processor generation (Intel
IXP2850).
Token Based Switch over IP traffic (TBS-IP) is a ForCES router designed and implemented in a
prototype at UvA. Using specific encrypted tokens built-in the IP packets, TBS-IP allows
connections between a regular connectionless IP (Campus) network and a connection-oriented
GMPLS network of an Optical Network Service provider. The TBS-IP router provides a webservice configuration/set-up interface to a high level authorisation and accounting service
(e.g., AAA servers from UvA, Harmony from UniBonn, OSCARS from Internet2). The TBS-IP
router can be programmed for a number of reserved paths/service levels using an XML
Authorization ticket proposed and implemented in the PHOSPHORUS AAA/AuthZ
infrastructure. Briefly the process is depicted in Figure 1 works as follows:
1)
User application requests to the UvA-TBN gateway router (ForCEG) a path
between the IP addresses (IPsrc, IPdst, startTime, duration, and bandwidth), where
IPsrc must be within its IP campus network, and IPdst should be available within the
GMPLS networks.
2) The ForCEG service will authenticate the user application and request a path to
the GMPLS authority (Harmony IDB) on behalf of its user applications. As a result, IDB
will generate and return the credentials of using authorised paths in format of a Global
Reservation Identifier (GRI) and a TokenKey used of in-band encryption.
3) IDB will prepare all intermediate domains involved in the path provisioning with
the requested credentials (GRI, TokenKey, etc).
4) Once the start-time of the requested path arrives, the received credentials from
IDB are enforced into the TBS-IP gateway router at the lowest level (IP packets and
circuits).
5)
When a user application (e.g., vlc) is authorised to connect to an IPdst over
GMPLS network, the Token Builder inserts encrypted tokens within the outgoing
packets (media streams) and redirect them towards TBS-IP.
6) TBS-IP authenticates every received packet identified by the plain-text GRI as a
lookup computation (finds GRI-entry into the local AuthZ-table), then it does:
a.
If the Packet comes from IPcampus network (PKT.IPdst==GRIentry.Port1.IPaddr), then it will re-write its IPdst to the correct endpoint
(PKT.IPdst=GRI-entry.IPdst) and will push the packet into the proper GMPLS
path, as indicated by GRI-entry.Port2;
b.
If the Packet comes from GMPLS network (PKT.IPdst!=GRIentry.Port2.IPaddr), then it will push the packet into the proper outgoing port
into IPcampus as indicated by GRI-entry.Port1;
Figure 13 TBN PHOSPHORUS testbed
The PHOSPORUS Token Based Switch (TBS-IP) prototype implementation shows that a packetbased admission control system can be used to dynamically select a fast end-to-end
connection over hybrid network at gigabit speeds.
6.3.
Last Mile Token Based Networking
Last mile comprises the network section between the end-user site (campus, lab, organization,
etc) and the dynamic circuit access point. Generally, these campus networks work at IP packet
level and do not offer QoS. Hence, this can lead in a poor performance of high-demanding
applications.
In this chapter, we investigate the implementation of a Token Based Networking as a Last mile
mechanism. For this investigation, we take in consideration the concepts of a Token Based
Networking and the related work done in the field in PHOSPORUS FP6 project.
6.3.1. Architecture
In a nutshell, the presented approach performs authorized path selection between end-user
sites and the campus network edge. A cryptographic hardware acts as a real time path
selection mechanism, using a token-key to recognize tokens inside IP packets. These tokenkeys must be requested, authorized and created first. A client can make this request using web
services mechanism to an AAA server. In Figure 2 is depicted the Last Mile TBN architecture
Figure 14. Last mile Token Based Networking architecture
Last mile TBN architecture consists of three parts:
6.3.1.1. Token Builder (TB)
The Token Builder (TB) is software installed at the end-user host to tokenise application traffic.
In our approach, the token is inserted into the IP-option field. This field can be of variable
length and resides in the IP header. An IP packet containing options should be forwarded
unmodified by a router. A router does not need to understand IP options. This makes the use
of IP options transparent to non-TBS network devices. In PHOSPORUS testbed, hijackers are
used for token insertion.
A token created for an IP packet is essentially the result of applying an HMAC algorithm over a
number of packet fields as illustrated in Figure 3. An HMAC algorithm is a key-dependency to
create a one-way hash (RFC 2401). For instance, in PHOSPORUS testbed, a HMAC-SHA1
algorithm generates the unique token (20 bytes) that will be injected into the packet’s IP
option field. As an IP option field has a header of two bytes, and network hardware systems
commonly work most efficiently on chunks of four or eight bytes, they reserve 24 bytes for the
IP option field. In order to ensure the token uniqueness for packets, they need to include fields
that are different in every packet. Therefore, some part of the packet data will be included
together with the packet header in the HMAC calculation. We mention that some of the first
64 bytes of an Ethernet frame are masked in order to make the token independent of the IP
header fields which change when a token is inserted (e.g., header length, total length, IP
checksum) or when the packet crosses intermediate routers (e.g., TTL). The mask also provides
flexibility for packet authentication, so that one could use the (sub)network instead of end
node addresses, or limit the token coverage to the destination IP address only (e.g., by
selectively masking the IP source address).
Figure 15. Token creation process for a TBN IPv4.
Note that although in the document we refer to token as an entire tag built-in the packet, in
our proposal, the tag consist of two parts: a plain-text aggregation identifier (GRI) and an
encrypted token.
We may associate a token-key with an IP source / destination pair, so all traffic between two
machines is handled with the same key. However, other forms of aggregation are possible. For
instance, we may handle all traffic between two networks with the same key, or all machines
participating in a Grid experiment, etc.
6.3.1.2.
Token Based Switch (TBS-IP)
The Token Based Switch (TBS-IP) is a cryptographic hardware able to enforce selective path
forwarding on per-packet basis. For each received packet, the TBS-IP checks whether the
current packet has the appropriate IP option field. On success, a Global Resource Identifier
(GRI) field, identifying a specific application instance, is extracted. Next, TBS-IP uses the GRI
entry to retrieve the token-key from the authorization table, already provisioned in the TBS-IP
by an AAA authority. Finally, the token-key is matched with the packet token to authenticate
it.
Figure 16 Authorization table
Currently it is possible to implement such programmable network element using specialized
hardware (network processors, FPGAs or using powerful PCs). For instance, in PHOSPORUS
TBS-IP uses the latest network processor generation (Intel IXP2850) able to switch at multigigabit speeds.
6.3.1.3.
AAA server
The AAA server is an authority responsible for fetching a user policy from the policy repository
that validates the request and creates the corresponding token-key. After the request is
validated and the token key is created, the AAA server distributes the token-key and the global
resource identifier (GRI) to the end-user and to every TBS-IP in the selected path.
6.3.1.4.
Use Case: User submits a request
Suppose a campus network with several faculty buildings like the one depicted in Figure 5. An
end-user located at the Engineering faculty would like to forward his application traffic
through a privileged path within the campus network, which is expected to have a better
performance than the default IP-routing. The procedure to submit such a request is as follows:
1. End-user sends a request to the AAA server authority. AAA server authority
validates the request from a policy repository, if affirmative, generates a tokenkey.
2. AAA server distributes the token-key and the GRI to the end-user Token Builder
and to each TBS-IP in the selected path. This token-key will be used for both
generating tokens at end-user site (Token Builder) and for verifying tokens at each
TBS-IP.
3. End user application starts generating traffic, which is tokenised by the Token
Builder using the previously obtained token-key. For each received packet at the
TBS-IP equipment, it checks the packet token to check the packet authenticity /
validity/authorisation. An authenticated packet is then forwarded to its
corresponding privileged path.
4. All non-authenticated packets are forwarded to the default path.
Figure 17. User submits a request scenario
Note that, to forward selectively tokenised traffic between end-user site and campus network
edge, we must implement a TBS-IP in every hop. Otherwise, a non-TBS-IP router won’t process
the packet token and will take his forwarding decision based on its routing table. Furthermore,
every TBS-IP hop must have at least two routes to select, one to forward the tokenised traffic
and the second one to forward the default traffic.
6.3.2. Integration with AutoBAHN
Two end-users located at different domains require a connection among them. We suppose
both campus network implement TBS-IP capable devices. The dynamic circuit among domains
is requested and provisioned by AutoBAHN. Here we would like to merge both systems, TBS-IP
campus network and the dynamic circuit provisioned by AutoBAHN. Such a way, we can
provide to the end-users a host-to-host circuit.
In this scenario, instead of having the AAA server at the campus network, the AAA server
authority has to be implemented as an AutoBAHN service. Hence, there is a single AAA server
for all NREN campus networks, in other words, an AAA server per domain.
The procedure to establish a host-to-host circuit works as follows:
1.
End-user sends to AutoBAHN a host-to-host request. Note that, this request has
to indicate that the campus networks implements a TBN last mile system. In such
a way that the circuit can be extended until the end host and not until the
campus network edge as usual.
2.
AutoBAHN AAA authority validates the credit of the client, if is positive,
AutoBAHN accepts the request. Then AutoBAHN set-up the dynamic circuit
among domains/NRENs.
3.
When the dynamic circuit is successfully reserved, AutoBAHN AAA server
distributes the token-key and the GRI to the end-user Token Builder and to each
TBS-IP in the campus network path. This token-key will be used for both
generating tokens at end-user site (Token Builder) and for verifying tokens at
each TBS-IP.
4.
End user application starts generating traffic, which is tokenised by the Token
Builder using the previously obtained token-key. For each received packet at the
TBS-IP equipment, it checks the packet token to check the packet authenticity /
validity/authorisation. An authenticated packet is then forwarded to its
corresponding campus privileged path. At the campus edge, the tokenised traffic
is forwarding to the dynamic circuit established by AutoBAHN.
5.
All non-authenticated packets are forwarded to the default path.
Figure 18. Last Mile TBN and AutoBAHN integration
6.4.

University of Amsterdam (UvA)
6.5.

Promoters
Last Activities
TERENA Networking Conference 2009: gTBN over IPv4 is proposed as well as the
presentation of experiments in a live-demo
6.6.


Users
University of Amsterdam (UvA)
Phosporus: ForCES Token Based Switch Architecture (pdf).
6.7.
Conclusions
The main advantage of a Last mile TBN solution is allows binding application traffic from endsites to established authorized paths within the campus network. Hence, we can guarantee
that no rogue users will use the reserved resources. In other words, TBS architecture offers to
user applications a flexible access control approach/solution to the reserved paths, where the
token represents the right to use a pre-established network connection at a specific time.
For its own, TBN does not improve the network performance, it just allows to selective
forward packets according to a built-in token. However, network administrators can apply
traffic engineering. In the sense that, they can intentionally for instance, keep the tokenised
path working in low load to achieve better performance. Or assign as tokenised path the route
with less latency or jitter.
Probably, further detail on how AAA server can be integrated with AutoBAHN to implement
TBN feature in campus networks. Also some more detail about the token-key distribution
among domains.
The main drawback is the hardware requirements; we do need to install cryptographic
hardware able to enforce selective path forwarding on per packet basis in campus network
infrastructure.
7.
Comparison Table
Lambda Station
TeraPaths
Phoebus
VRF
Reconfiguration of
campus network
infrastructure.
Selectively forwarding
of designated flows.
Policy based routing
(PBR)
Traffic prioritization.
Uses Differentiated
Service architecture
(DiffServ) for
configuring QoS paths
within the LAN
Selective forwarding of
tokenized traffic. A
cryptographic token is
inserted in every packet
of user application
traffic. Binds
applications to
networks using a token
as a glue
Split end-to-end
connection into several
segments to improve
performance of each
segment. Phoebus
Session Protocol (PSP)
Virtual/VPN Routing
and forwarding
enabled on router
inside the campus
(multiple routing tables
per router enabled). No
changes needed on the
end hosts. Using
normal IP routing to
selectively forward
traffic from a host on a
static route out of the
campus. QoS not
explicitly provided but
possible
Lambda Station
software at every
campus network
TeraPaths software at
every campus network
Token Builder software
to tokenise traffic at
the end-sites. AAA
server to provide/
generate token keys.
Phoebus Gateway at
the campus network
edge
None
Working principle
Software
Requirements
gTBN
Routers/switches with
PBR , ACL capability
Routers/switches with
DiffServ capability
Token Based Switch.
Hardware able to
selectively forward
traffic according a
cryptographic built-in
token. Needed at each
router/switch device at
the campus network
IP router able to deploy
Phoebus Gateway
technology
Changes needed on
end hosts
YES
NO
YES
Limited
NO
Middlebox needed
NO
NO
NO
YES
NO
Used in production
environment
YES
YES
NO
NO
YES
Fermilab and CALTECH
RACF Computing
Facility, a division of
Brookhaven National
Laboratory
University of
Amsterdam (UvA)
Internet2 and
University of Delaware
Dutch Tier1 (SARA and
Nikhef), used in
production network of
SARA Computing and
Network Services for
LHCOPN, DEISA, LOFAR
Hardware
Requirements
Who is behind?
Required capabilities
present in most highend routers (VRF in
Juniper, VPN Routing
and forwarding in
Cisco)