“Internet2 Presentation”
Brian Stengel, Director of Operations, Kinber
Peter J. Heverin, Project Manager, Kinber
Mike Carey, Network Engineer, Kinber
Jon Paul Herron, Director of Engineering, GlobalNOC
Luke Fowler, Senior Manager, Systems Engineering, GlobalNOC
Marianne Chitwood, Director of Operations, GlobalNOC
September 20th, 2012
What is PennREN
a high-speed, state-wide, research & education serving healthcare,
K-20 and the public good
• Capital Budget - $128,958,031
– Federal Stimulus Funds - $99,660,678
– Matching Funds - $29,297,353
• Outside Plant Infrastructure Constructed for PennREN
– 48 Stands of NZD Fiber optic Cable
– 1700± Route Miles
– Outsourced Fiber Maintenance
• 13 Optical Regeneration Service Nodes
• 56 Service Distribution Access Nodes
Project Route
Total PennREN Fiber route is estimated at 1,613 miles.
The route consists of the following;
•
•
•
•
1,086 miles of new aerial construction.
486 miles of aerial overlash.
14 miles of new underground construction.
27 miles of leased underground conduit.
Network Backbone Engineering is 95% complete overall.
3
Project Route
4
PennREN Locations
Service Nodes
Amp Nodes
Lehigh University
East Stroudsburg University of Pa
401 N. Broad
Kutztown University of Pa
Penn State Hershey Medical Center
West Chester University of Pa
Indiana University of Pennsylvania
Millersville University of Pa
Allegheny Center Mall
Shippensburg University of Pa
Slippery Rock University of
Pennsylvania
Pitt Johnstown
Penn State Erie (Behrend)
Community College of Beaver Co
Clarion University of Pennsylvania
Allegheny College
Penn State Dubois
Pitt Titusville
Penn State – State College
Penn State Hazelton
Windstream
Bucknell University
University of Scranton
Access Nodes
46 Locations on PennREN fiber
Project Teams
•
•
•
•
•
•
Fiber OSP
Building entrances
Cabinets, Installation
Splicing
Testing
Last mile solutions
•
•
•
•
•
•
•
•
•
•
Hardware procurement
Design
Integration
Project Management
Professional Services
Service Desk, Tier 1, Tier 2
Systems and Network Tools
Network Engineering
Net, Perf, Change Management
Operations support
•
•
•
•
•
•
•
•
•
•
•
•
Staging
Shipping
Configuration
Testing
Turn-up
Field Services
Earthlink sourcing for
OOB MPLS Network
Last mile solutions
Plan/Design
Optical engineering
Field services
Professional services
•
•
•
•
•
Plan/Design
Consulting
Engineering
Support
Training
Node Installation Status
Segment in
Production
Equipment
Installed
Under
Construction
Co-Location at Member Sites
Construction Diagrams
Optical Core/DWDM System
Packet Core – MPLS
Plan – Build – Integrate
Peering Points (Initial)
Internet
Internet
R&E
R&E
Network Management
PS1
PS2
PS3
Performance Measurement
Servers at every Service
Node
In-band management
Out-of-band access
External
Networks
DB
External
Networks
Operations - Support
GRNOC provides to KINBER:
• Service Desk – 24x7x365 call center support, ticket management, technical
support coordination, and workflow support
• Network Engineering – Expert network engineers work with the Service
Desk to ensure fast problem resolution, provisioning, and strategic
engineering and planning
• Software and Systems – Provides support through a fully integrated system
of network management, measurement, and visualization tools
The GlobalNOC at Indiana University provides carrier-grade operations, tools,
and network expertise while placing a singular focus on the unique
requirements of the research and education (R&E) Community
GRNOC supports 20+ R&E networks across the country
Operations - Maintenance
•
•
•
•
•
Fiber Maintenance
Emergency Restoral
Routine
Maintenance
OSP Records
•
•
Warranty
Hardware replacement
(pre-ship)
Software maintenance
•
•
•
•
KINBER Network
Engineers
Host IT/Site
teams provide
remote eyes and
hands support
upon request
•
Warranty
Hardware
replacement
(pre-ship)
Software
maintenance
Foundation for Services
Multi-Degree
ROADMs
MPLS PE
Switches
West
DWDM
Ring
3rd Party
Providers
East
DWDM
Ring
ASBR
Router
External
Networks
External
Networks
On-Net: Delivery
Access
Node
Access
Node
Access
Node
Access
Node
Access
Nodes
Internet
R&E
Internet
R&E
Off-Net: Delivery
Carrier
CX
Commercial
Co-Lo
NNI
First/Last
Mile
Private
Member
Off-Net
Internet
Internet
R&E
R&E
EPC – Ethernet Port Connection Service
R&E networks
VPLS (pt – pt, multipoint)
Customer Router
KMEX
Service Node: PE
Switch
Commodity Internet
Customer Subscribes
to 1/10GE Ethernet
Port(s)
PennREN
Services
• Services available to a customer with an EPC
Member B
Member A
Member C
Member A
Services
•
•
•
•
•
•
•
•
•
Community-wide, distributed service for members to
exchange traffic across a common network. Similar to
an Internet Exchange
Member-to-Member peering, ad-hoc R&E activities
Best-effort traffic exchange within the community
Transit service to major R&E networks such as
Internet 2, ESNet, NLR…
Provided by KINBER affiliates
Virtual private networks with committed bandwidth
can be established using VPLS instances in the
PennREN network
VPWS – Virtual Private Wire Service – Point to Point
VPLS – Virtual Private LAN Service - Multipoint
Access to commodity Internet service is available
over the PennREN network
Optical – Wave/Lambda Services
• Optical Waves 10G can be provisioned across the network
Member A
Member A
Member B
Member A
Member B
Performance Measurement
• PerfSONAR Measurement Archives allow
exchange of data with other network
operators
• Regularly scheduled testing across the
backbone
• User-initiated testing for applications such as
problem diagnosis
• Multiple routing tables allow us to support
both 1G and 10G testpoints on a single host
Performance Measurement
Deploying performance measurement servers at each
service node
3 servers per site:
– Active throughput measurement (1G and 10G)
– Active latency measurement
– local data collection / ad hoc performance measurement
Specific performance measurement tools include:
BWCTL
OWAMP
MaDDash
24
Network Measurement
• GlobalNOC tool-set updated to support all
PennREN devices, including:
– SNAPP – High-resolution SNMP-based network
utilization data
– LLAMA – DWDM layer performance measurement
data
– Central storage of other passive data like syslog,
configuration, flow, etc. for regular and ad-hoc
processing & analysis.
PennREN Service Desk
• GlobalNOC Specialized Support Technician
• Footprints PennREN project created – integrated to
TickMon, Operations Calendars, Trouble Tickets
• Telephone number for PennREN customers integrated
into shared GlobalNOC phone queue
• Email established noc@pennren.net to receive customer
inquiries and or communications from vendors….this is
monitored 24x7x365
PennREN Service Desk
• Network and Member impact guidelines
defined
• Web form for customers/vendors to submit
trouble tickets into Footprints
• Change management process/form created
• Internal documentation developed/published
for staff training and reference
PennREN Service Desk
• Pro-active network monitoring
• Support for scheduled maintenances and
changes
• Vendor coordination
• Customer install process
• Reporting
• Security
• Tools and Communications
Weighing network needs
A new network is more than a
construction project
• It’s a system:
–
–
–
–
–
–
Goals
Infrastructure
People
Services
Operations
Business
• Make decisions and plans early (also, there will
be more decisions than you think, so watch for
bottlenecks)
• The work on all of these areas starts right away
Service Definition
• Unrealistic to imagine services will be
completely defined from the beginning of the
design
• But, enough understanding is needed to guide
the design and build plan
• Early “anchor” users help a lot!
Communications
• Communication needs:
– to be high-bandwidth
– Early is good
– Changing staff can be disruptive, do it carefully.
– Informal is good
– Multi-channel is good
– reliable technologies are good
– Face to face is good
Documentation
• Documentation:
– Have a place for documents
– Keep the place for documents clean
– Keep the purpose of each document clear/distinct
Operations
• Operations Preparedness:
– Start early, there are a LOT of things to think
about, especially:
• Turn-up/acceptance process
• Expectations for facilities
• How and where to keep network data
Build-outs make for lots of information
• Need to be ready to put it somewhere where
it’ll be usable later
• Data entry is cheap
• Entering/documenting is best when the
information and its context are fresh
The lowly management
network
• Sometimes doesn’t get enough attention
• It can get crazy complicated or crazy expensive
• NOTE: in an SDN world, this becomes even
MORE important!