Janet Update
GridPP Collaboration meeting
Sept 2012
David Salmon
Topics
• Janet6
• Janet & e-Infrastructure
• Issues & futures....
Janet6
Neil Shewry
Rob Evans
SuperJANET4/5
• Getting us out of the cycle of networks that were bursting at the
seams at the end of their life
• Dedicated fibre backbone, allowed us to grow.
– Subject to contract
• SJ5 was originally due to finish October 2011
– Extended to October 2013
– Still no congestion
– Originally 10Gbit/s, but have upgraded through 40Gbit/s to 100Gbit/s
SuperJANET5
• Greater visibility of the optical layer than previous networks
–
–
–
–
–
Managed by Verizon Business
Janet NOC had read-only access to (most) transmission equipment
Regular meetings with Verizon on implications of capacity requests
Good communications between Janet’s senior engineers and Verizon’s.
40G and 100G trials before deployment
• However, still some distance away from the ‘laser-face’
– Deployment issues that were contractual rather than technical or
operational
Janet6: Building up ideas
• Thinking & reflection as SJ5 progressed
– Greater visibility of optical layer
– New features of transmission systems
• ROADMs: Colourless; Directionless; Contentionless (more later)
• Direct experience of transmission equipment
– TVN
• Developments in Janet and other R&E networks
– Use of point-to-point circuits
– Dynamic provisioning
Requirements Gathering
Requirements Gathering
Bandwidth, bandwidth, bandwidth...
Off-net
•
Flexibility
•
Anytime, anywhere access
•
Agility
•
•
Cost control
Internationalisation of
education
Service delivery
Management of costs
•
Delivery of third party services
•
Funding environment
•
Cloud services
•
Costs of change
•
Reliability and resilience
Partnerships
•
Public/public and public/private
•
Information assurance
Requirements Gathering
Reliability
High reliability by minimising single points of failure;
flexibly coping with major breaks; assisting in
resilience to customer organisations
Scalability
Ability to increase bandwidth at controllable cost
Separability
Protection of interests of teaching and learning and
research sectors
Flexibility
Responsiveness to additional network service
requirements
Functionality
Support for the provision of wide range of
services: e.g. High information assurance; Unified
comms
With agility
End-to end
Across JANET
Procurement Strategy
Procurement Strategy
Options appraisal
Bespoke arrangement
•
Procure dark fibre infrastructure
•
Procure optical transmission equipment
•
Management in-house by the Janet NOC
competitive
dialogue
procedure
Procurement Strategy
Fibre Procurement
•
Lot 1: Fibre Infrastructure in the UK
•
Options: Scotland, Wales, Ireland, Aurora, London, Research
•
Lot 2: Fibre Infrastructure in Ireland
•
Framework Agreement(s) for 4 years
•
Initial order from 5+2 years to 10+5 years
Procurement Strategy
Transmission equipment procurement
•
Supply, design, delivery, installation, commissioning, maintenance, training,
specialist technical support
•
Framework Agreement for 7 years
•
Initial order 3 + 2 + 2 years
Project Timetable
Project Timetable
Procurement Update:
Fibre Procurement
Prequalification phase
•
Launched October 2011
•
13 responses
•
Shortlist 6 bidders
Fibre Procurement
UK Government approval to proceed – Cabinet Office
July 2012
Contract Signed
July 2012
Transmission Equipment Procurement
Prequalification phase
•
Launched November 2011
•
17 responses
•
Shortlist 6 bidders
Transmission Equipment Procurement
Contract signed last week
September 2012
Current Status
Current Status
From requirements to services
• Capacity
– Exponential growth continues
• ‘Commodity’ IP networking
• R&E requirements
– Must be able to scale the network
– Not just more channels, but faster channels over the network’s life
• 100Gbit/s to start
• 400Gbit/s?
• 1Tbit/s?
From requirements to services
• Ubiquitous connectivity
– Access to Janet from anywhere
– Not all in the scope of Janet6
• Other activities in the company
– Feeds the capacity requirements
– Feeds the external connection requirements
– Feeds the reliability requirements
From requirements to services
• Distributed campuses
– Inter-site, intra-organisation connections
– VPNs
• Separacy
• Centralised firewall
From requirements to services
• Outsourced services
–
–
–
–
Compute, data storage, applications
Low latency for interactive services
High bandwidth for bulk data services
Reliability, reliability, reliability.
From requirements to services
• Research
– National and global collaboration
• Interoperation with GEANT and other R&E networks worldwide
• LHC, SKA, ITER, Bioinformatics, Climate…
– ‘Data deluge’
– Services
• Dynamic provisioning of layer 2 circuits
Janet6: Core Fibre and Backbone Resilience
• Similar topology to SJ5.
• Two north-south paths
• New east-west path from
Birmingham to Nottingham
– Erdington to Lowdham
• No backbone PoP at Reading
– Still have a regional PoP
– STFC RAL as a backbone PoP
• New PoP in Acton, West
London
Janet6: Regional fibre
• Illustrative only
– The details are subject to
change
• Similar to SJ5
– ‘Collector arcs’
– Spurs
• Regional networks connect to
two different routers
• Over 70 optical nodes
– Amplifier sites, add/drop sites,
and router sites
Lighting the fibre
• DWDM transmission equipment
• Still some way to go in the procurement
• Theoretically 80 channels of 100Gbit/s per channel
• Roadmap for 400Gbit/s and 1Tbit/s with better spectral density
• ROADMs
• ‘Native’ capacity on the core to be 100Gbit/s
– 10Gbit/s multiplexed over that where needed
• Muxponders or OTN switching
– Coherent technology required for 100Gbit/s doesn’t sit well with
dispersion compensation required for 10Gbit/s
– RNEP fibre may still carry 10G native
10G Circuits: muxponders or switched?
Muxponders
Switching
ROADMs
• Will be more reconfigurable than SJ5
– One of the collector arcs in Scotland required 13 engineers to be in
dotted along it at the same time to commission
• Get away from the ‘air gap’ between RNEP fibre and backbone
fibre that exists on SJ5.
– Different transmission systems on collector arcs and backbone
– Requires two client cards and a patch cable to bridge a circuit from one
to the other
Building the network and the NOC
• What will this give us?
– Janet NOC will have a view from the fibre up the stack to the routers
– We will see when fibres/lasers/receivers degrade
• Lower signal levels
• Increase in error rate
– Forward Error Correction (FEC) working harder
– We will see where faults are
– Fewer administrative boundaries to cross
– Directly translate the community’s requirements into engineering
IP service
• As you were
• Finish moving away from SDH access towards ethernet
– 802.1ad bonding for 20Gbit/s, 30Gbit/s etc.
– 100GE if you can afford the optics (have seen a dramatic fall)
• Still IPv4, IPv6, unicast, multicast
• In London, retaining Telehouse and Telecity Harbour Exchange as
major points of external connection
• Continuing to expand Telecity Manchester
– Content providers and content delivery networks
– ‘IX Manchester’ operated by LINX
VPNs
• VPNs over primary access
–
–
–
–
–
–
Inter-site connectivity
Low bandwidth layer 2 circuits
Other services?
Statically configured VLANs (site/RN boundary, perhaps RN/core)
BGP signalled VPNs (RN/core, perhaps some sites)
Bandwidth management between IP and VPNs
• These are layer 2/3 VPNs
– Not IPsec
– If you want to be sure it is encrypted, best to do that yourself
• Would you trust this man with your encryption keys?
Lightpath service
• Good question
• Ethernet over MPLS network will remain
– Upgrade the MX960s to 100GE interconnects
– Move 10Gbit/s circuits from dedicated wavelengths to EoMPLS
• Investigating OTN switching as part of transmission procurement
– May be too early to do much
– …especially as we’ll also have MX960 platform
• Don’t need guaranteed bandwidth?
– See VPN service
Migration
• 66.666666666666667% (ish) chance we’ll be moving core PoPs
• Moving routers from PoP to PoP overnight would be ‘challenging’
– Requires all regional networks connected to a PoP to move
simultaneously (no link between old PoP & new PoP)
– Can’t move anything until all the regional network fibres to a PoP are
ready
• One delay holds up everything
• Remember the newts!
• Need to upgrade the switching fabric anyway
– T640 to T1600
– T1600 to T4000
• Investigating options & costs
Migration
• Aiming to have the network installed and ready by the end of
March 2013
• Migration between March and August
• Post-Clearing until end of October is contingency only
• End of October is a hard cut-off.
Janet6 summary
• Extensive fibre footprint
• Direct management of optical transmission
• On-schedule
• Very capable strategic foundation – 10 years+
e-Infrastructure
Networking
David Salmon
Topics
• e-Infrastructure funding
– What has Janet been doing ?
• Emerging Issues
Janet – e-Infrastructure networking
• £26M
With HEFCE
Breakdown
• £10M
Contribution to Janet6 backbone
• £12M Network provision within “classic” Janet community,
including key locations which may need additional fibre access
• £4M
“Industry connection”
e-Infrastructure programme support
• CTO Bob Day on ELC
• Paul Lewis secondment to BIS programme Office
– 8 months
• STFC Michael Wilson secondment
– just started
– funded from Janet’s e-Infrastructure allocation
e-Infrastructure Strategic Sites
• Potential fibre / capacity requirements
• Work with organisations to understand requirements
• Build cases for provision
• If approved, integrate provision with deployment of Janet6
Patterns of use
• Data movement – working patterns
– How much – data volume ?
• Beware Bytes vs. bits !
–
–
–
–
How quickly ?
How often ?
Where to ?
National & International
• Network engineers need bit-rates !
• Working with research communities to understand
emerging/evolving requirements
Norwich Bioinformatics Cluster
• The Genome Analysis Centre
- TGAC
• The Sainsbury Laboratory
• John Innes Centre
• Institute for Food Research
• University of East Anglia
• Norwich Research Park
Hinxton Genome Campus
• European Molecular Biology Laboratory - European Bioinformatics
Institute
• The Sanger Centre
• Babraham Institute
Exeter & locale
• Met Office
• University of Exeter
• Weather & Climate research
• Monsoon HPC – NERC & M.O.
Broaden scope of Janet’s fibre infrastructure
• Complementary &
integrated with
Janet6



e-Infrastructure projects funded
• HPC
– 5 regional consortia
– National facility – Hector / Archer
– Dirac
• Particle physics
• NERC
• STFC
– RAL
– Daresbury
• will engage to discuss requirements & evolution
Outcome
• Outline proposals made
• Approvals discussions with Research Councils
• Must converge soon to optimise Janet6 fibre layout
• Some budget remains
• Likely to be a further proposals
• Academic & Industry locations ?
Some implications
Throughput challenges
• Infrastructures can deliver very high capacities
• Realising high-throughput end-to-end will take some
effort
• Can’t just take systems “out of the box” and expect to
get full line-rate data-transfers
• This needs to be communicated & understood more
widely
• Janet & NREN community needs to work more closely
with research communities to understand and solve
issues.
Achievable Throughput
• What is realistic ?
– Challenge:
1Gbit/s, 24hr ~= 10TBytes
• Holistic approach
– End systems - Hardware & Software - Both ends !
– Local networks
– Wide area networks
• Highlight successful examples
– Technical case studies
• Janet role ?
– Co-ordination/collaboration
– Advice & troubleshooting support ?
Network structure implications ?
• Large research communities
• International/global overlays or private networks
• Particle physics - LHC OPN (LHCONE)
• Radio Astronomy – eVLBI, LOFAR, (SKA...)
• Future – Bioinformatics ?? ELIXIR
• Campus implications ?
point to point services (lightpaths)
more complex multi-homing – routing/switching issues
Industry ?
• Existing Janet policy
– Supporting collaborative R&D
– Business and Community Engagement (BCE)
• Evolution - more needed ?
• e-Infrastructure
– “Open and accessible Janet”
Connecting Industry – meaning ?
• Network connections ?
• Access to community systems & resources connected to Janet ?
• Access to Janet community knowledge & expertise ?
– Mix of all three...
• BIS & HPC Consortia views ?
Issues & futures
• Good planning & dialogue between Janet & GridPP
– continue !
– understand evolving requirements
– address these as they emerge
• Involved in LHCOPN & LHCONE meetings
– Rob Evans (Janet)
– Technical input
– Balancing UK requirements, GEANT & other NRENs in Europe, and US &
global NRENs
GidPP networking archaeology....(2000)
Conclusion
• Janet6 on-track
– Very capable strategic platform supporting UK
Research and Education for 10 years+
• e-Infrastructure
– Approvals soon hopefully
– Challenge of delivery & deployment
– Industry ?
• Using the network
– Achieving higher data throughput
?
Janet Aurora
Janet Aurora
• Last presented at CEF in 2010
• Very good results produced
– High profile papers at conferences
• But, funding issues have held back full development
Context - Funding History
• e-Science and JISC Funding
– £4.35M
– 2007-2012
• Fibre infrastructure only
–
–
–
–
Fibre access (single pair)
ILA rental – space for researchers equipment – 1 rack
basic telecomms for management access
Tax – lit fibres are subject to business rates in UK
• NOT
– Research equipment
– Staff effort
JANET Aurora
University of
Cambridge
550km of fibre-pairs
72km
50km
VM Bury St Edmonds PoP
54km
VM Ipswich PoP
VM Chelmsford PoP
JANET Lightpath access to
other UK Locations
VM Enfield PoP
CAL483331-1 (28km)
UCL
VM Brentford PoP
31km
52km
VM Camberley PoP
58km
VM Crawley Court PoP
43km
University of
Southampton
Intermediate equipment
co-location point
University / JANET access
point
Fibre spans
57km
University of
Essex
55km
30km
Telehouse
London
International Lightpath access
to other NRENs via JANET &
GEANT
Current position
• Janet will maintain Aurora in current form until Sept/Oct 2013
• Aurora / Dark Fibre facility WILL need to evolve over next 12-18
months
– Can not extend contracts for current infrastructure any further
(procurement law)
– The Essex research group is moving to Bristol (Prof. Dimitra Simeonidou)
• Research Council (EPSRC) funding for a “Dark-Fibre Facility” is
still being considered
– Equipment and staff effort to exploit dark-fibre as a test platform
e-Infrastructure
• New funding form UK government
• £100M+ into HPC
– National
– Regional
• £26M to Janet to continue and extend capability
–
–
–
–
Extend Janet6 to key locations
Biosciences
Climate and Weather Research
HPC
• Dark-Fibre facility / Aurora MAY also be possible
– Exploring with Research Councils
Recent Results
• ECOC – next week
– Dimitra Simeonidou & colleagues will present interesting new results
• Collaborations with NPL continue
– Ultra-stable lasers – NPL and UCL
– Optical comb transfer – NPL and Southampton
• Janet in discussion with NPL about participation in Aurora
• NPL has independent funding, and plans to deploy fibre for combtransfers
• Following slides from Radan Slavik
– UFE, Prague – ORC, Southampton
JANET Aurora Dark Fiber Link
Southampton to London part of the link: 200 km, dispersion-compensated,
Aston
University
University of
6 double in-line flat-gain CLA EDFAs with maximum input/output powers of -5/15 dBm.
Cambridge
850km of fibre-pairs
Experiments:
(A) Southampton-London roundtrip for phase regeneration;
(B) Southampton-Essex roundtrip for optical wavelength conversion.
University of
Essex
JANET Lightpath access to
other UK Locations
UCL
60 km
40 km
Telehouse
Telehouse
London
London
International Lightpath access to
other NRENs via JANET & GEANT
60 km
40 km
University
Universityofof
Southampton
Southampton
Intermediate equipment
co-location point
University / JANET access
point
Fibre spans
Phase and amplitude all-optical regenerator
Taken from University of Southampton, U.K. (R. Slavík et al, Nature Photonics 2010 and JLT 2011)
All-optical wavelength conversion
CLA amplifiers used
Taken from University of Southampton, U.K. (V. Rancano; results under progress)