Gig – BYU
Kelly McDonald, BYU (ret.)
RMCMOA Workshop, Tempe Az.
January 13, 2015
Background – since 2011
• Glenn Ricart gave a presentation on Gig-U at the June, 2011 Westnet
CIO meeting
• Pat Burns spoke again about the concept, in the January 2012
Westnet CIO meeting
• Regular reports of Gig – U and its value to community outreach
This caused me to have some
Self Reflection…..
• What value would Gig-E provide to me?
• Awesome HD, (perhaps even 4K)
• Much more compelling gaming for family and friends
• New source of home heating. (Cisco 6509)
• Cool new neighborhood status symbol
• New home signage: “1 Gb onboard “
More Serious Thinking…
• For the things I am interested in, does Gb make a difference?
(probably not, but I’m not a researcher)
• I asked a neighbor, who does heavy Landsat data analysis. (So, Dr.
Long, what would you do, if you had Gb network capability from your
home to the BYU campus?)
• After picking him up off the ground, we began to discuss it more
seriously.
This More Serious Analysis Spawned my
Epiphany…
• The Project was largely in the RFI/RFP stage to secure
partners who will assist in the eventual build-out
• Eventually, universities would enjoy high speed (fiberbased) connections through their service providers in the
community, but much utility work needed to be done
• BYU and it’s municipality (Provo, Utah) already had this
capability
Veracity and BYU
• Veracity provided BYU’s local dial-tone, as well as long-distance
services.
• They also serviced most of the apartments that BYU students live in.
• Veracity operated the Provo City’s optical fiber network, but
struggled to keep it afloat
• In November of 2011, we approached Veracity about becoming a
peering and bandwidth partner.
Provo, Utah was Google Fiber’s Third City
• Provo, Utah and Google-Fiber have worked together to transition the
residential fiber network to Google Fiber
• Google offers Internet and television services. They do not offer
residential phone services
• 5 mb-down/ 1 mb-up – Free, 1 gb-down/1 gb-up - $70/month
• Television - 300 channels, $50/month
• I have used this service for the past year, and have been very pleased
with the reliability and performance
What do I think is needed?
• Bandwidth is what was needed when aggregation was the goal
• Enabling the individual researcher is a different problem. Here,
reducing RTT is the key.
• Much more peering should be considered. Perhaps we should
establish peering zones at our institutions and invite all of the ISPs to
connect.
• Perhaps we push the borders of our network deeper into our campus
to further reduce RTT
• For 90% of our constituents, they already have more than enough
bandwidth available to them.
Fort Collins Community
Network
Scott Baily, CSU
FCCN - General Description
• CSU provides low-cost bandwidth to public sector entities in the Fort
Collins area
• Sharing infrastructure, aggregating BW
• Made possible by:
• Fiber from Platte River Power Authority
• Commitment from CSU’s Leadership
• Fostering the right relationships early on
• Nothing fancy, but effective
Initial Participants (circa 1999)
• CSU – Local bandwidth aggregator & “hub”
• Platte River Power Authority – local fiber provider and participant
• City of Fort Collins
• Larimer County
• Poudre School District (Fort Collins)
• Front Range Community College
Later Participants
• City of Loveland
• Poudre Valley Hospital System (now UC Health)
• Liberty Common Charter School
Original participants are all still there
FCCN Members:
• Lease fiber circuits from PRPA
• Share equipment purchase and maintenance costs
• Participate in meetings
• Vote on significant decisions
CSU Provides
• Best effort support (really pretty good!)
• Connectivity to the FRGP
• Quarterly invoices, based on proportional usage
• Performance and monitoring tools
• Bandwidth at CSU’s actual cost
• Meeting coordination, agendas, minutes, etc.
CSU Does Not Provide
• Other services typical of an ISP, e.g.:
• DNS, Websites, mail spooling, etc.
Governance
• Very lightweight governance
• MOU between CSU and each participant
• Simple majority vote required on purchasing and other significant
decisions
• Governance seems sufficient here, so no compelling reason to
change it
Results
• Quite favorable – public sector entities receive affordable and
dependable connectivity
• Reliability has been excellent, overall
• Membership has been very static over time
• Outreach/public service benefits to CSU, without requiring
unreasonable resources
FRGP
CSU Border1
90
CSU Border2
20
0M
FCCN GW1
0M
FCCN GW2
Larimer
County
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Spd
PoE
Usr
Reset
Console
Clear
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Auxiliary Port
Spd
PoE
Reset
Clear
Usr
Auxiliary Port
Console
ProCurve Networking
City of FC City Hall
HP Innovation
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
Act
FDx
LED
Mode
Spd Mode
RPS Status of the Back
Act
FDx
LED
Mode
ProCurve Networking
HP Innovation
ProCurve
Switch 3500yl
J8692A
Mdl
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
Link
1
on = 1000Mbps
Mode
3
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
PoE
Power
Act
Status
PoE
Tmp
Fault
Fan
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Auxiliary Port
Console
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
ProCurve Networking
ProCurve
Switch 3500yl
J8692A
HP Innovation
Tmp
Fan
Mdl
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Fan
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Act
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Console
Auxiliary Port
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Act
Spd
PoE
Reset
Clear
Usr
Auxiliary Port
Console
ProCurve Networking
HP Innovation
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
PoE
Tmp
Use only one (T or M) for each Port
ProCurve
Switch 3500yl
J8692A
Status
PoE
RPS Status of the Back
FDx
LED
Mode
Test
Fault
EPS
PoE
Status
PoE
Power
24 M
Fault
Power
Mdl
Use only one (T or M) for each Port
Fan
Test
Fan
Test
Use only one (T or M) for each Port
Mdl
Tmp
Fault
PoE
Status
PoE
Tmp
Fault
EPS
PoE
Status
PoE
Power
Use only one (T or M) for each Port
ProCurve
Switch 3500yl
J8692A
Power
Mdl
Use only one (T or M) for each Port
ProCurve
Switch 3500yl
J8692A
PSD
ProCurve Networking
HP Innovation
PVH Harmony
FRCC
Fan
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Act
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Console
Auxiliary Port
ProCurve Networking
HP Innovation
ProCurve
Switch 3500yl
J8692A
Mdl
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
PoE
Act
Status
PoE
Power
Tmp
Fault
Fan
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Auxiliary Port
Console
Use only one (T or M) for each Port
Mdl
PoE
Status
PoE
Tmp
Fault
Use only one (T or M) for each Port
ProCurve
Switch 3500yl
J8692A
Power
Liberty
Common
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
Spd
PoE
Reset
Clear
Usr
Console
Auxiliary Port
ProCurve Networking
ProCurve
Switch 3500yl
J8692A
HP Innovation
Mdl
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
PoE
Power
Act
Status
PoE
Tmp
Fault
Fan
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Console
Auxiliary Port
ProCurve
Switch 3500yl
J8692A
ProCurve Networking
Power
FCCN 10G
Oct. 2014
MCR
Mdl
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
PoE
Act
Status
PoE
Tmp
Fault
Fan
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
Console
Auxiliary Port
EPS
RPS Status of the Back
Spd Mode
off = 10Mbps
flash = 100Mbps
on = 1000Mbps
Dual-Personality Port 10/100/1000-T (T) or Mini-GBIC (M)
PoE-Integrated 10/100/1000Base-T Ports (1-24T) - Ports are IEEE Auto MDI/MDI-X
Link
1
Mode
3
5
7
9
11
Link
13
Mode
15
17
19
21T
23T
Link
21 M
Mode
23 M
Link
2
Mode
4
6
8
10
12
Link
14
Mode
16
18
20
22T
24T
Link
22 M
Mode
24 M
PRPA
Act
Status
PoE
Tmp
Fault
ProCurve
Switch 3500yl
J8692A
Power
City of Loveland
Mdl
PoE
HP Innovation
Fan
FDx
Spd
LED
Mode
PoE
Reset
Clear
Usr
Test
ProCurve Networking
Console
Auxiliary Port
Use only one (T or M) for each Port
EPS
Act
FDx
LED
Mode
Use only one (T or M) for each Port
Fan
Test
Use only one (T or M) for each Port
Mdl
PoE
Status
PoE
Tmp
Fault
Use only one (T or M) for each Port
ProCurve
Switch 3500yl
J8692A
Power
ProCurve Networking
HP Innovation
City of FC –
FRII
HP Innovation
ProCurve Networking
HP Innovation
Futures
• Talking with City of Fort Collins about a community Gigabit initiative
• Perhaps to become the next generation FCCN
• Things are progressing slowly …
• Boulder seems to be less risk-adverse, and may set a good example
for Ft. Collins
Lessons Learned
• Develop and maintain key relationships
• Document everything
• Take the initiative, if you believe this is the right thing to do in your
community
• Codify agreements
• Lots of personnel changes since we started
• Provide self-help tools and basic training to participants
Boulder Research and Administration
Network (BRAN), Front Range GigaPoP
(FRGP), Bi-State Optical Network (BiSON)
Marla Meehl
UCAR/FRGP/BiSON Manager
UCAR/NCAR
2014-04-29
1
24
Colorado Network History
• NSFnet national R&E network
•Funded by the National Science Foundation (NSF)
•Operated mid-1980s to 1995
•Encouraged creation of regional networks to aggregate
• Westnet (Colorado, Wyoming, Arizona, New Mexico, and Utah)
•Still operates as an affinity group
•Meets bi-annually managed by Marla
• Operated and managed by Colorado State University and the University of Colorado
•56Kbps, T1, to T3
25
Colorado Network History – Cont’d
• Very high-speed Backbone Network Service (vBNS) funded by the
NSF to connect the supercomputer centers including NCAR
• OC-3 (155Mbps) to OC-12 (622Mbps)
• Left out universities and other entities
• Internet2 formed and created the Abilene Network – 1999
• Formed by 34 member universities
• Still in operation with a 100Gbps backbone
• Own fiber for expansion
26
Colorado Network History – Cont’d
• National LambdaRail (NLR), Inc.
• Founded in 2003 and operational in 2004
• 14 members formed including the FRGP
• Owned 12,000 mile national Level3 fiber optic footprint
• In November 2011 the control of NLR was purchased from its university
membership by billionaire Patrick Soon-Shiong for $100M
• His intention to upgrade NLR infrastructure and repurpose portions of it to support an
ambitious healthcare project
• The upgrade never took place. NLR ceased operations in March 2014
27
Boulder Research and Administration Network
(BRAN) – January 2000
• http://www.branfiber.net/
• The City of Boulder (the City), the University of Colorado at Boulder
(CU), the National Center for Atmospheric Research (NCAR), and the
Department of Commerce Laboratories (DOC) share an important fiber
optic network that interconnects the participating organizations
• BRAN provides a significant, local, next-generation, fiber optic
infrastructure that gives the partners a competitive advantage in
research, higher education, and administrative functions. The network
links greatly improve the ability of each of the partner organizations to
conduct their business, as well as enhance their ability to serve the
public.
• The fiber network is over 11 miles in length. The cable path extends
generally along a north-south corridor within the City, and the route of
the network links the partners' many facilities to capitalize on the
network infrastructure already in place, to utilize rights-of-way, and to
transect properties that the partners control.
28
What is the FRGP?
• For over 14 years, the FRGP has advanced the
research and educational goals of government,
nonprofit, and educational participants in the
region by establishing and maintaining a unique
multi-state network infrastructure that is owned
and controlled by the FRGP participant research
and education community.
• Starting new 5-year agreement cycle 7/1/14
29
What is the FRGP – Cont’d
• The FRGP is a consortium of Universities, non-profit
corporations, government agencies, colleges, K-12s,
non-profit corporations, and non-profit health care
entities who cooperate as part of a Regional Optical
Network (RON) in order to share wide area
networking services including the commodity
Internet, Internet2, intra-FRGP, caching, and peering
connectivity.
• UCAR provides the management, engineering, and
network operations center support for the FRGP.
30
FRGP Benefits
• FRGP participants typically enjoy reduced costs,
shared expertise, shared services, increased buying
power, and economies of scale.
• Aggregated and direct intra-participant network
access
• This includes access to all participants, secondary
connections behind primary FRGP participants
• Direct peering connections including Comcast,
Google, CoreSite Any2, and the Energy Sciences
network (ESnet).
31
FRGP Benefits
• Access to multiple commodity Internet service providers at
10Gbps offering resiliency and diversity
• TeliaSonera and Level3
• Traffic aggregation and FRGP membership in The Quilt, Inc.
provide cost savings for commodity Internet services
• The FRGP also participates in the commercial network
peering program known as TransitRail (TR)/Commercial
Peering Service (CPS) managed by Internet2
• Designed to improve network performance through
peering relationships with commercial networks to help
participants reduce the overall cost of, and reliance on,
commodity Internet transit services.
• Carries 30-50% of FRGP traffic
32
FRGP Benefits
• Provides caching services as part of the Akamai and
Netflix (carrying ~7Gbps)
• Provides connectivity to the Internet2 network
• Either directly for Internet2 members or qualifying
community anchor institution via the US UCAN program
(formerly known as SEGP)
• 50Gbps to Chicago
• 10Gbps to Salt Lake City
• 10Gbps to Los Angeles (soon to be 100G)
• Seattle soon to be 100G
33
FRGP Traffic
• FRGP Traffic (excludes caching and intra-FRGP)
• ~14% commercial IP transit (Level3 and TSIC)
• ~60% peering (ESnet, Google, TR/CPS, Any2)
• ~26% Internet2
34
FRGP Benefits
• Has supported a doubling in traffic growth every 2-3
years since inception
• Currently serves approximately one million users
whose organizations participate in our research and
education community in Colorado and Wyoming
• Multiple locations to terminate circuits for
redundancy
35
Western Regional Network (WRN)
• The Western Regional Network is a multi-state
partnership to provide advanced, robust high-speed
networking for research, education, and related
uses. WRN is a collaboration of the Pacific
Northwest GigaPoP (PNWGP) in Washington, the
Front Range GigaPoP (FRGP) in Colorado, the
University of New Mexico on behalf of the State of
New Mexico, the Corporation for Education
Network Initiatives in California (CENIC), and the
University of Hawaii.
• Share Internet2, intra-WRN, and commodity
Internet
36
Bi-State Optical Network (BiSON)
• The Bi-State Optical Network, or BiSON, is a collaboration of research
and educational institutions in Colorado and Wyoming that utilize fiber
optic links for high-speed optical networking
• BiSON participants receive robust, redundant access to regional and
Wide Area Network (WAN) services.
4