Optical Grids in USA
Gigi Karmous-Edwards
gigi@mcnc.org
March 5th, 2006
International Workshop on
Future of Optical Networking
Outline
•
Motivation
•
GLIF
•
Research Challenges
•
OBS for Grids
•
EnLIGHTened Computing
•
Conclusions
Motivation
E-science and Grid Computing
• E-science: global, large scale scientific collaborations enabled
through distributed computational and communication
infrastructure
• Combines scientific instruments and sensors, distributed data archives, computing
resources and visualization to solve complex scientific problems
• In physics, molecular biology, environmental, Health, Entertainment, etc.
• Grid computing: main enabler of E-science
• Grid is concerned with "coordinated resource sharing and problem solving in
dynamic, multi-institutional virtual organizations." (Foster)
• E-science migrated to Grid for the reasons of affordability of high-bandwidth
communication infrastructure, affordability of resources and inter-disciplinary nature
of the research
E-Science and Grid Opportunity
• Governments world-wide promoting global E-science research
programs
• New Zealand, $43 million to establish the Advanced Network
• United States: from Optiputer over Cyber-infrastructure to DDDAS ,
and DoE ($60M)
• Canada: CA*net4 and i-Infrastructure
• Australia: e-Research Initiative (Started Oct 2004)
• Europe: 7th framework (€1B/year), GEANT2 (€93M), Netherlight,
Geodise, UK E-science, etc.
• Private sector setting a strong pace
• Google building their own L1/L2 network
• Multi $B investments from large businesses: IBM, HP, Intel,…
• Smaller businesses: Grid-based enterprise applications, data centers
solution, commercialization of Globus toolkit,…
Advances in Optical Technologies
• 1000 channels per fiber….. Experimentation with 160G per
channel
• Dark Fiber every where ….
• Fiber is much cheaper…US Headlines: Google buys Fiber
• All-optical switches are getting faster and smaller (ns switch
reconfiguration) - useful for OBS
• Control Plane protocols, SOA, continue to mature - but
should revisit
• Layer one Optical switches relatively cheaper than other
technologies
• Electronic Dispersion Compensation - advances in state of
the art
• Fiber, optical impairments control, and transceiver
technology continue to advance while reducing prices
New Demands on Networks
Emerging High-End applications
• High bandwidth connectivity of supercomputers (teraflops+)
• Large file transfers, over long distances
• Applications/end-users/sensors/instruments requesting optical
networking resources host-to-host connections
• Determinism (QoS), jitter and latency requirements
• Coordination of network with computational and noncomputational resources (CPU, databases, sensors, instruments )
• Mechanisms for retrieving near-real-time information about
network resources and network states
• Mechanism for both advance and fast on-the-fly reservation and
set-up
• Low latency on-demand connection requests - real-time weather
prediction
• Policy and security enforcement in open scientific environments
Why should this community work with Driving
High-end Applications?
When thinking of future Optical Architectures
• Using advance applications to better understand the
requirements on the network in order to architect
future optical network architectures
–Requires Interdisciplinary teams
–Speaking the same language…
• Help experiment with novel Optical technologies
on experimental testbeds - NLR --> GENI (GLIF?)
GLIF
www.glif.is
Global Lambda Integrated
Facility
World Map
www.glif.is
Visualization courtesy of
Bob Patterson, NCSA.
GLIF Automation?
Client A
Grid
middleware
Network
Management
NREN
Control Plane
WEB
Services ?
Network
Management
NREN
Control Plane
Client B
Grid
middleware
Network
Management
NREN
Control Plane
GLIF Control Plane and Grid
Middleware Integration wg
Mission: To agree on the interfaces and protocols to
automate and use the control planes of the
contributed Lambda resources to help users on a
global scale access optical resources on-demand or
pre scheduled.
Research Challenges
Research Challenges
•
Coordination of resources per request for both on-the-fly and advanced
reservations
•
Dynamic and Adaptive on-demand use of end-to-end networking resources Identification of functions and interactions between the control plane,
management plane, and Grid middleware
•
Monitoring information of resources - identification of information, abstraction
of information, and frequency of updates
•
Advanced software to support multiple classes of software including highlydynamic, and event-driven applications
•
–
Dynamic lightpaths routing and resource allocation under constraints and
•
The more dynamic these reconfigurable lightpaths the closer they get to OBS
Behavioral Control
•
Control/management planes interacting with middleware
•
Centralized vs. distributed functionality
Applications
Edge
Routers
Workflow
Engines
Resource Manager
Co-Scheduler
Resource
Allocation
Resource
Monitoring
One Definition of Control Plane
“Infrastructure and distributed intelligence that controls
the establishment and maintenance of connections in
the network, including protocols and mechanisms to
disseminate this information; and algorithms for
engineering an optimal path between end points.”
Draft-ggf-ghpn-opticalnets-1
Centralized vs. Distributed…
Key areas for Today’s Control Plane are:
1) Provisioning
Network
2) Recovery
Behavioral
Control
Network
Management
(Hierarchical )
Network
Management
Migration
NE
NE
NE
Centralized (vertical)
NE
Protocols
NE
Protocols
NE
Distributed (Horizontal)
Control Plane Functions OCS/OBS/OPS
• Routing - Intra-domain and Inter-domain
1) automatic topology and resource discovery
2) path computation (How do we use the infrastructure)
• Signaling - standard communications protocols between
network elements for the establishment and
maintenance of connections
• Neighbor discovery - NE sharing of details of
connectivity to all its neighbors (very powerful tool)
• Local resource management - accounting of local
available resources
Centralized vs. Distributed Behavioral Control of Networks
•
Re-thinking control functionality in terms of (centralized or Distributed):
–
–
–
–
–
–
–
•
•
Information exchanged
Algorithms for path computation and recovery
Discovery and advertising of resources
Scalability
Timing
Interdomain interactions
Policy enabled
Centralized Control: Management Plane
–
Management plane mechanisms rely on client/ server model, usually
involving one or more management applications (structured hierarchically)
communicating to each network element in its domain via a management
protocol, (i.e., SNMP, Tl1, XML, etc).
Distributed Control: Control Plane
–
“Infrastructure and distributed intelligence that controls the establishment
and maintenance of connections in the network, including protocols and
mechanisms to disseminate related information, and algorithms for
engineering an optimal path between end points.”
Control Plane Focus today
1. Network Configuration
2. Network Recovery
GMPLS IP Control Protocols:
Signaling: RSVP-TE, OBS
Routing: OSPF-TE based on Link State protocols (LSA)
Discovery: LMP
OBS for Grids
OBS vs. GMPLS signaling comparison,
Advantage of OBS is fast reaction time
• GMPLS RSVP-TE connection setup
– PATH messages (from sender to receiver)
– RESV messages (from receiver to sender)
– Processing at each node for both message types to
set up an end-to-end connection.
• GMPLS RSVP-TE connection tear down
– PATHTear message (from sender to receiver)
– RESVTear message (from receiver to sender)
– Processing at each node for both message types to
set up an end-to-end connection.
–Trade-offs
• Two RTT to create and release end-to-end connection,
However, it is a Reliable connection
• OBS signaling can react much quicker - needed for nearrealtime reaction to events for High-end applications
OBS more efficient use of the network than OCS
OBS Control Plane Signaling Requirements
Provides flexibility
• Capability to send data w/out waiting for RTT - reduction of latency
• Implicit resource release
– Setup message contains duration of connection
– Resources are released automatically w/out second control message
• Explicit resource release
– Separate release message is sent from application if duration of data
transfer is unknown
• Capability to behave in Connection Oriented manner
– Wait for end destination ack prior to sending data
• Variable payload sizes - faster switches and faster signaling processing
– Very small payloads - packet
– Long lived lightpaths
• Hop-by-hop Path
Fast Switch Reconfiguration
• OBS signaling
– Setup message processing - microseconds in
hardware
• Message parsing
• Forwarding table look-up
– Cross-connection - dependent on switching
technology
– MEMs - 12 - 15 ms
– Polymer-based switches and SOA based - ns
– Etc.
– Advantages of Fast Switch Re-configuration times
…
– The smaller the payload … could be packet
– The smaller the offset time
– Quicker reaction to changing environments
US Grid Testbeds
•
•
•
•
•
•
•
Optiputer
TeraGrid
Dragon
Cheetah
OmniNet
UltraLight
Enlightened
EnLIGHTened Computing
EnLIGHTened Computing
Highly-dynamic Applications Driving Adaptive
Grid Resources
NSF seed funded project
EnLIGHTened team significance
key Institutions collaborating on the research efforts
•
•
•
•
MCNC (Network research), -PI -Gigi Karmous-Edwards, Yufeng Xin, Steve Thorpe,
Bonnie Hurst, Lina Battestilli, Mark Johnson, Joel Dunn
LSU (Application and Grid research), PI -Ed Seidel, PI - Gabriele Allen, PI - Seung
Jong (Jay) Park , Jon Maclaren, Andrei Hutanu, Lonnie Leger
(Sub-Contract to MCNC) Renaissance Computing Institute, RENCI (Grid
Middleware research): (a joint institute between UNC, Duke and NC State ), Dan
Reed, Lavanya Ramakrishnan,
NCSU (Network research), Savera Tanwir, Harry Perros, John Moore
Key Partner Institutions (cost share)
•
•
•
•
•
•
•
Cisco, Javad Boroumand, Russ Gyurek, Wane Clark, Kevin McGratten
AT&T Rick Schlichting, John Strand, Matti Hiltunen
SURA Gary Crane
Naval Research Lab (NRL) , Hank Dardy
Calient Networks, Olivier Jerphagnon, Ron Mackey
UCSB/Calient, John Bowers
NLR
EnLIGHTened Computing connectivity diagram with partners
To Asia
To Canada
To Europe
SEA
POR
BOI
Chicago
CAVE wave
OGD
EnLIGHTened wave
(Cisco/NLR)
DEN
KAN
SVL
PIT
CHI
CLE
WDC
Cisco/UltraLight wave
L.A.
LONI wave
San Diego
Raleigh
TUL
DAL
Baton Rouge
HOU
Members:
- MCNC GCNS
- Renaissance Comp. Inst.
- LSU CCT
Official Partners:
- AT&T Research
- SURA
- NRL
- Cisco Systems
- Calient Networks
- NLR
NSF Project Partners
- OptIPuter
- UltraLight
- WAN-in-LAB
- DRAGON
International
Partners
•LUCIFER
•G-Lambda
-GLIF
Enlightened/LUCIFER
Sister Projects - Similar Goals
Testbeds
♣ 3 EU NRENs are partners + 3
national test-beds + 3 research
networks in US and Canada + 5
expressed interest through LoIs
♣ These community
representatives are willing to
monitor project progress,
collaborate and exploit its results
GLIF
www.glif.is
Conclusions
Further Reference
IEEE Communications Magazine
Feature Topic
Optical Control Plane for Grid Networks: Opportunities, Challenges and
the
Vision
http://www.comsoc.org/pubs/commag/cfpcommag306.htm
Guest Editors: Gigi Karmous-Edwards and Admela Jukan
March, 2006
Conclusions
1.
2.
3.
4.
5.
6.
7.
8.
Control Plane research is vital to meeting future generation Grid computing - with
a strong focus on “vertical integration”
GLIF resources should be used for both network research and E-applications
(morphnet concept)
Reconfigurability is essential to bring down cost and meet application
requirements. OBS or OCS
Currently, we have a view of the behavior of potential future enterprise
applications by focusing on the needs of high-end driving applications, but it is also
important to understand the requirements of Industry.
Next generation networks could be vastly different than today’s mode of operation
- should not constrain research to today’s model
The Research networks are the ones that will take these bold steps not the
carriers… apply lessons learned to production quickly.
International Collaboration is a very Key ingredient for the future of Scientific
discovery - The Optical network plays the most critical role in achieving this!
The OBS community can benefit by experimenting on national scale networks like
NLR and working with E-science applications