Lab for Advanced Networking
University of Kentucky
Lexington, KY
AT&T Labs - Research
Florham Park, NJ
Other Project Members
Zongming Fei (Kentucky)
Eric Boyd (Internet 2)

GEC7

ProtoGENI ShadowNet

Leveraging AT&T ShadowNet
March 17, 2010

GEC7
March 17, 2010

GEC7

Problem: ProtoGENI backbone router resources are limited and can be challenging to use.

Idea: Leverage the logical router features of Juniper routers to dynamically create virtual routers (slivers) in the backbone that provide carrier-grade performance and services .

Challenge 1: Creating the control software needed to virtualize the Juniper M7i and integrate with the ProtoGENI network

Challenge 2: Make it easy for users to “see” what is happening on their backbone router slivers.
March 17, 2010

GEC7
Deploy “virtualizable” commercial routers (Juniper m7i) in the ProtoGENI backbone that support commercial OS/software.
Add software support to these virtual routers that will enable per-slice monitoring and measurement.
Develop tools and interfaces that will allow slice users to use the measurement infrastructure in simple and easy ways.
March 17, 2010

Source: http://groups.geni.net/geni/attachment/wiki/presentations/protogeni_Ricci_gec3.pdf
GEC7 March 17, 2010

Source: http://groups.geni.net/geni/attachment/wiki/presentations/protogeni_Ricci_gec3.pdf
GEC7 March 17, 2010

ProtoGENI Backbone Node Architecture
Internet 2
Gigabit Ethernet Switch
Non-sliced PC
GEC7
General Purpose
Slivers
Sliced PC
March 17, 2010

ProtoGENI Backbone Node Architecture
Internet 2
Gigabit Ethernet Switch
Non-sliced PC
GEC7
General Purpose
Slivers
Sliced PC
Virtual
Server
Juniper
Component
Manager
ShadowBox
Controller
Juniper M7i Router
Logical
Router 1
Logical
Router 2
Logical
Router n
ShadowBox Router
March 17, 2010

ProtoGENI Backbone Node Architecture
Internet 2
Gigabit Ethernet Switch
Non-sliced PC
GEC7
General Purpose
Slivers
Sliced PC
Measurement
Slivers
Virtual
Server
Juniper
Component
Manager
ShadowBox
Controller
Juniper M7i Router
Logical
Router 1
Logical
Router 2
Logical
Router n
ShadowBox Router
March 17, 2010

GEC7
March 17, 2010


ShadowNet is roughly addressing same problem as GENI, however
 Less clean slate…
 Focus on services and network management…

Need the ability to more rapidly evolve the way we run our network and the services we offer in our network (pull):

Inherently difficult:
–
Potential impact to existing services
 Networks are shared, new service/feature might negatively interact with existing services
 Gets worse with time: networks are “cumulative” (hardly ever gets switched off)
 Very long test cycles
–
Need for support systems
 Configuration management, network management, service monitoring, provisioning, customer interfaces, billing, fault management

Legacy lock in: Existing (complicated) systems need to be modified to support new services
 Extremely long development time

New vendor technologies (push):
 Programmability and virtualization available from major vendors
–
Allow non-vendor code to execute on routers
–
Loosen the tight coupling between physical boxes and logical functions
 Rethink the way we deploy services and operate our network

 “National footprint” network/platform/testbed for research and service trials
–
Connected to, but separate from production network
 Limit impact on operational network
 Look like a customer to AT&T network
–
In between lab and production
 Stable enough for service trials
 Open/flexible enough for research experiments
– “General purpose”, shareable testbed facility
 Would like to make this a widely available/useful facility, akin to general purpose computing facilities
 The role of ShadowNet:
 Operational (but non-production) environment to enable:
–
Evaluation of new technologies/vendor capabilities
 No impact on existing network/services
–
Service testing/trials in a realistic environment (including customer trials)
 Utilize virtualization and partitioning capabilities to limit interaction and reduce risk
–
Evolution of network support systems
 Free from legacy lock
–
Research in operational setting
 Both networking and “Internet services”
 Safe playground for network evolution
–
This model might become the way we want to build our network

ShadowNet rack
Sun Fire
X4150 Server
GigE
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Router
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Juniper
M7i
Sun Fire
X4150 Server
Cisco Catalyst
3560G-48TS
Sun Fire
X4150 Server
Juniper
M7i
Router
Juniper
M7i
 Set of building blocks that can be flexibly combined into an operational network
(or networks)

Operational nodes:

Richardson, TX

Pleasanton, CA

Chicago, IL
Waiting for network connectivity:

Middletown, NJ
Page 14
•
Each node:
– “Gateway” router, Juniper M7i
–
2 X GigE connectivity to AT&T network
–
7 X SunFire x4150 servers
– 2 X “multiservice” routers, Juniper M7i
–
Cisco GigE switch (Catalyst 3560)
–
OOB access
• AS 5105:
–
Full BGP table
–
4 /24 prefixes
–
Advertise up to /32

• Sharable and composable infrastructure
• Strong separation between physical and logical devices:
• Physical machines -> virtual machines
• Physical routers -> logical routers
• Physical links -> logical gigE links: pseudowires, tunnels, VLANs etc
• ShadowNet slices consist of logical devices that have been plumbed together
• However, allow allocation of physical devices to a slice
Page 15

GEC7 March 17, 2010

"The interesting thing about cloud computing is that we've redefined cloud computing to include everything that we already do. I can't think of anything that isn't cloud computing with all of these announcements.”
Larry Ellison, CEO Oracle
Wall Street Journal, September 26, 2008
• CloudNet experimentation
•
•
Combining cloud computing with VPN
•
•
•
•
Fairly elaborate setup involving many components
Create VPLS VPN between three sites
Prototype dynamic VPN connectivity
Experiment with (live) virtual machine and storage migration
Mechanisms for optimizing WAN migration
•
•
•
In the works:
Cloud control architecture
Slice with bunch of VMs for “architectural support for network debugging”
Declarative approach to network management
Extend to provide mobility functionality
March 17, 2010


Existing cloud platforms do not meet the needs of enterprise customers

Insufficient security controls

Need isolation at server and network level
 Deployment is difficult - transparency
 Cloud resources are completely separate from local ones
 Can’t make VMs look like part of existing enterprise network
 Limited control over network resources

Cannot specify network topology or IP addresses

Cannot reserve bandwidth or request QoS guarantees for network links
Page 18

Use VPNs to separate customer resources
Customer’s cloud resources are only reachable from other VPN end points
More flexible control of how IP addresses are assigned
Physical network is transparent to customer
Assume a virtual machine abstraction
VPNs provide both network resource isolation and strong security
Page 19

VPC A
Cloud Site X
Server
Server
VPN A
PE
PE
AT&T Backbone
VPC B
Cloud Site Y
Server
Server
Server
PE
PE
VPN B
VPN A
VPN B
Virtual Private Cloud:
•
Collection of cloud resources presented to customer as a private set of cloud resources, transparently and securely connected to customer VPN
•
Manage network resources in the same dynamic manner as cloud resources
Page 20

CloudNet Portal
VPN A
VPN B
Cloud Platform
Cloud
Manager
Network
Manager
PE
Server
Server
Server
Server
Server
CE
Router
PE
AT&T Backbone PE
VPN A
VPN B
Cloud Domain
High level abstraction:
• Create compute resources
• Map into VPN
• Cross domain interaction
Page 21
Network Domain
Cloud Manager:
• Create compute resources
• Map into VPN (cloud side)
Network Manager (IRSCP):
• VPN management (network side)

Physical nodes involved CloudNet slice
ShadowNet rack
PLTN
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Juniper
M7i
Juniper
M7i
ShadowNet rack
Cisco Catalyst
3560G-48TS
AT&T backbone
(7132)
Juniper
M7i
ShadowNet rack
Juniper
M7i
Cisco Catalyst
3560G-48TS
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Juniper
M7i
Juniper
M7i
RCSN
Juniper
M7i
Cisco Catalyst
3560G-48TS
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Juniper
M7i
Juniper
M7i
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
Sun Fire
X4150 Server
CHCG
GRE tunnels
Page 22

VPLS MPLS VPN in a slice
PLTN
RR/IRSCP
PLTN5 PE1 P1
VLAN circuit cross connect
Logical tunnel
Physical ethernet
Logical link: VLAN cross connect example
P1
P1
VLAN
Cisco
Switch
VLAN
Juniper
Router
Page 23
VLAN-CCC
P3
CHCG
PE3 CHCG6
P2
RCSN
PE2 RCSN6
P3
Juniper
Router
VLAN
Cisco
Switch
VLAN
P3

Laptop
Game
Client
VpnRemap
PLTN ipsec
RR/IRSCP
CHCG6 r0
VM0
P3
CHCG
PE3
PLTN5
PE1 P1 drbd
PE2
RCSN
P2
RCSN6 r0
VM0
•
Ipsec client on laptop provides remote access to VPN
•
Run game server on VM
•
Run game client on laptop
•
Game server move with VM
•
Application very sensitive to network impairments
•
Client monitor typically shows game detects minor changes
• VM migration across WAN “just works” using VPLS VPNs
•
Optimize for WAN conditions:
•
Storage: moving between asynchronous and synchronous replication
•
VM: optimizing migration logic + redundancy elimination
Game
Server

This material is based upon work supported in part by the National Science Foundation.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of GPO Technologies,
Corp, the GENI Project Office, or the National Science Foundation .