CANARIE
“Critical Role Universities Will Play in the
Future of the Internet”
The Customer Empowered Networking Revolution
http://www.canarie.ca
http://www.canet3.net
Bill.St.Arnaud@canarie.ca
Tel: +1.613.785.0426
Outline






The Message
CANARIE
CA*net 3
Customer Empowered Networking
CA*net 4 and GigaPOP
eScience
The Message
 In mid 1990s the prevailing wisdom was that commercial sector would drive
design of Internet infrastructure
 R&E networks would focus on applications or specialized services
 As a result in North America R&E networks were commercialized or discontinued
 e.g NSFnet & CA*net
 However new network architectures and most importantly dark fiber is allowing
R&E networks to once again redefine telecommunications and the future of the
Internet
 LAN architectures, technologies and most importantly LAN economics are invading
the WAN
 Carrier neutral IXs (in essence GigaPOPs) are the essence of this network
 Control and management of the optics and wavelengths will increasingly be under
the domain of the LAN customer at the GigaPOP, as opposed to the traditional
carrier in the center
 These new concepts in customer empowered networking are starting in the same
place as the Internet started – the university and research community.
 Internet 2, SURFnet5 and CA*net 3 & 4
CANARIE Inc
 Mission: To facilitate the development of Canada’s communications
infrastructure and stimulate next generation products, applications and
services
 Canadian equivalent to Internet 2 and NGI
 private-sector led, not-for-profit consortium
 consortium formed 1993
 federal funding of $300m (1993-99)
 total project costs estimated over $600 M
 currently over 140 members; 21 Board members
CA*net 3 National Optical Internet
Consortium Partners:
Bell Nexxia
Nortel
Cisco
JDS Uniphase
Newbridge
CA*net 3 Primary Route
CA*net 3 Diverse Route
GigaPOP
ORAN
Condo Fiber Network
linking all
universities and
Netera
hospital
BCnet
Calgary
Deploying a 4
channel CWDM
Gigabit Ethernet
network – 400 km
SRnet
MRnet
Regina
Condo Dark Fiber
Networks
connecting
universities and
schools
Winnipeg
ONet
Vancouver
Seattle
16 channel DWDM
-8 wavelengths @OC-192
reserved for CANARIE
-8 wavelengths for carrier
and other customers
Chicago
Multiple Customer
Owned Dark Fiber
Networks
connecting
universities and
schools
Deploying a 4
channel Gigabit
Ethernet transparent
optical DWDM–
1500 km
ACORN
St. John’s
Charlottetown
Fredericton
RISQ
Montreal
Halifax
Ottawa
STAR TAP
Toronto
New York
Customer Empowered Networks
 Universities in Quebec are building their own 3500km fiber condo network in
partnership with number of carriers
 Universities in Alberta are deploying their own 700 km 4xGbe dark fiber network
 Soon will be extended to all communities in Alberta
 School boards and municipalities throughout North America are deploying their
condo open access, dark fiber networks
 Illinois Iwire, Indiana Gwire, California DCP, Georgia SURAnet, Stockholm
condo fiber builds
 Chicago CivicNET, Alberta SuperNET, Canadian NBTF
 Carrier are selling “dim wavelengths” managed by customer to interconnect dark
fiber networks
 Williams, Level 3, Hermes
 Typical cost is one time $20K US per school for a 20 year IRU for condominium
fiber connection
What is condominium fiber?
 A number of organizations such as schools, hospitals, businesses and universities
get together to fund and build a fiber network
 Carrier partners are also invited to be part of condominium project
 Several next generation carriers and fiber brokers are now arranging condominium
fiber builds
 IMS, QuebecTel, Videotron, Cogeco, Dixon Cable, GT Telecom, etc etc
 Fiber is installed, owned and maintained by 3rd party professional fiber contractors
– usually the same contractors used by the carriers for their fiber builds
 Each institution gets its own set of fibers, at cost, on a 20 year IRU (Indefeasible
Right of Use)
 One time up front cost, plus annual maintenance and right of way cost approx
5% of the capital cost
 Institution lights up their own strands with whatever technology they want –
Gigabit Ethernet, ATM, PBX, etc
 New long range laser will reach 120 km
 Ideal solution for point to point links for large fixed institutions
 Payback is usually less than 18 months
Why Condo Fiber?
 First - low cost
 Up to 1000% reduction over current telecom prices. 6-12 month payback
 Second - LAN invades the WAN – no complex SONET or ATM required in
network
 Network Restoral & Protection can be done by customer using a variety of
techniques such as wireless backup, or relocating servers to a multi-homed site,
etc
 Third - Enables new applications and services not possible with traditional
telecom service providers




Relocation of servers and extending LAN to central site
Out sourcing LAN and web servers to a 3rd party because no performance impact
IP telephony in the wide area (Spokane)
HDTV video
 Fourth – Allows access to new competitive low cost telecom and IT
companies at carrier neutral meet me points
 Much easier to out source servers, e-commerce etc to a 3rd party at a carrier
neutral collocation facility
Quebec University Condo Network
Construit
Projet démarré
À venir
Bande passante louée
Val d’Or/Rouyn
MAN de Montréal
MAN d’Ottawa/Hull
MAN de Québec
MAN de Sherbrooke
Observatoire Mont-Mégantic
Lionel-Groulx
Lanaudière
Sorel-Tracy
Montreal Public Sector
Condominium Networks
Marie-Victorin
Rosemont
Montmorency
Maisonneuve
Ahuntsic
Édouard-Montpetit
Bois-de-Boulogne
Vers Québec
St-Laurent/Vanier
Champlain
Vieux-Montréal
Gérald-Godin
Construit
Dawson
Projet démarré
À venir
John-Abbott
André-Laurendeau
Bande passante louée
Capitale
Région-de-Sherbrooke
Rivière-du-Nord
Seigneurie-des-Mille-Iles
Amiante
Laval
Saint-Hyacinthe
Affluents
Bois-Francs
Draveurs
Grandes-Seigneuries
Hautes-Rivières
Laurentides
Patriotes
Premières-Seigneurie
Samares
Trois-Lacs
Chemin-du-Roy
Marie-Victorin
Sir-Wilfrid-Laurier
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Engineering
Study
School Board
Construction
List of Schoolboard Fiber Builds
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
PROJECT
140 km of fibre optics 80 schools
180 km of fibre optics 66 schools
175 km of fibre optics 52 schools
200 km of fibre optics 80 schools 4 partners
12 km of fibre optics 9 schools
170 km of fibre optics 111 schools 3 partners
250 km of fibre optics 51 schools
170 km of fibre optics 70 schools 4 partners
60 km of fibre optics 12 schools 4 partners
90 km of fibre optics 40 schools
210 km of fibre optics 58 schools
250 km of fibre optics 54 schools
200 km of fibre optics 35 schools
2 km of fibre optics 3 schools
190 km of fibre optics 73 schools
460 km of fibre optics 72 schools
45 km of fibre optics 15 schools
29 km of fibre optics 11 sites
6 km of fibre optics 5 schools
92 km of fibre optics 20 schools
Alberta SUPERnet
• Province wide network of condominium fiber to 420 communities in Alberta
• Guaranteed cost of bandwidth to all public sector institutions
• $500/mo for 10 Mbps, $700/mo for 100 Mbps
• Network a mix of fibre builds and existing supplier infrastructure
•
•
•
•
(swap/buy/lease)
Condominium approach: All suppliers can
• Buy (or swap) a share of the fibre (during build or after)
• Lease bandwidth at competitive rates
GOA has perpetual right to use (IRU)
• Ownership will be held at arms length
• GOA/stakeholder rates are costs to run divided over users
• Because of fibre capacity, bandwidth can be made available to businesses
at urban competitive rate
Total cost $193m
Bell Intrigna prime contractor
National Broadband Task Force
 Mandate:To map out a strategy and advise the Government on best
approaches to make high-speed broadband Internet services available to
businesses and residents in all Canadian communities by the year 2004.
 To ensure Canada’s competitiveness in a global economy
 To address the Digital Divide
 To create opportunities for all Canadians
 35 members including carriers, educators, librarians, communities,
equipment manufacturers, etc
 Chair – David Johnston
Fredericton Fiber Build






Started as Economic Development tool
MUSH, Govt., Research - ISP, carriers invited to participate
Build partners emerged quickly, $50,000 “donated” by three firms
Contracting now for 8 km phase 1, $110,000, complete Sept 2001
48 fiber min.
Unique experiment to extend off campus Internet access through
802.11 wireless
Ottawa Fiber Condominium
 Consortium consists of 16 members from various sectors including
businesses, hospitals, schools, universities, research institutes
 26 sites
 Point-to-point topology
 144 fibre pairs
 Route diversity requirement for one member
 85 km run
 $11k - $50K per site
 Total project cost $CDN 1.25 million
 Cost per strand less than $.50 per strand per meter
 80% aerial
 Due to overwhelming response to first build – planning for second
build under way
Condo Fiber Costs - Examples
 Des affluents: Total cost $1,500,00 ($750,00 for schools)





70 schools
12 municipal buildings
204 km fiber
$1,500,000 total cost
average cost per building - $18,000 per building
 Mille-Isles: Total cost $2,100,000 ($1,500,000 for schools)




80 schools
18 municipal buildings
223km
$21,428 per building
 Laval: Total cost $1,800,000 ($1,000,000 for schools)





111 schools
45 municipal buildings
165 km
$11,500 per building
Peel county: Total cost $5m – 100 buildings
 Cost per building $50,000
Typical Payback for school
(Real example – des affluents – north of Montreal)
 Over 3 years total expenditure of $1,440,000 for DSL service
 Total cost of dark fiber network for 75 schools $1,350,000
 Additional condominium participants were brought in to
lower cost to school board to $750,000
 School board can now centralize routers and network servers
at each school
 Estimated savings in travel and software upgrades
$800,000
 Payback typically 8 –16 months
 Independent Study by Group Secor available upon request
Reduction in the number of
servers
Before
fiber
Antennas
Novell Servers
SQL Servers
Lotus Notes Servers
Tape Backup Servers
Ethernet switches/hubs
Routers
Cache/proxy (Linux)
Fire walls (Linux)
78
82
13
2
12
10
108
12
1
After
fiber
0
1
3
1
4
98
3
0
1
Community Fiber Architecture
 A community consortia would put together a plan to fiber up all public sector
buildings in their community
 A community can be a province, a municipality, village, etc
 A fiber splice box that terminates the fiber at the street side nearby each public
sector building such as school, hospital, library is called a “Node”
 Community should must insure that potential facilities exist near the for private
sector equipment to connect up future home owners – colo facility
 Colo facility allows private sector to extend wireless, VDSL or HFC services to the
neighbourhood around the school
 Public sector buildings will have dedicated fiber strands that connect to a
“Supernode” which is a fiber splice box on the street beside outside of major
public sector central facility such as school board office, city hall, university,
etc
 Community should insure that facilities exist nearby the Supernode for the
private sector to install equipment to service home owners and businesses –
colo facility
 Additional fibers are made available from the Supernode to all Nodes such that
competitive service providers can purchase fiber to the node at some future
date
Possible architecture for large town
Central Office
For Wireless
Company
Carrier Owned
Fiber
School board office
Cable head end
Telco Central
Office
Condominium Fiber
with separate strands
owned by school and by
service providers
School
VDSL, HFC or Fiber
Provisioned by
service provider
Colo
Facility
School
802.11b
Average Fiber
Penetration to 250-500
homes
Benefits to Industry
 For cablecos and telcos it help them accelerate the deployment of high speed
internet services into the community
 Currently deployment of DSL and cable modem deployment is
hampered by high cost of deploying fiber into the neighbourhoods
 Cable companies need fiber to every 250 homes for cable modem
service, but currently only have fiber on average to every 5000 homes
 Telephone companies need to get fiber to every 250 homes to support
VDSL or FSAN technologies
 Wireless companies need to get fiber to every 250 homes for new high
bandwidth wireless services and mobile Internet
 It will provide opportunities for small innovative service providers to offer
service to public institutions as well as homes
 For e-commerce and web hosting companies it will generate new business in
out sourcing and web hosting
 For Canadian optical manufacturing companies it will provide new
opportunities for sales of optical technology and components
CA*net 4 Overall Objective
 To deploy a novel new optical network that gives GigaPOPs at the edge
of the network (and ultimately their participating institutions) to setup
and manage their own wavelengths across the network and thus allow
direct peering between GigaPOPs on dedicated wavelengths and optical
cross connects that they control and manage
 To allow the establishment of wavelengths by the GigaPOPs and their
participating institutions in support of QoS and eScience applications
 To allow connected regional and community networks to setup transit
wavelength peering relationships with similar like minded networks to
reduce the cost of Internet transit
 To offer an “optional” layer 3 aggregation service for those networks
that require or want such a facility
CA*net 4 Possible Architecture
Layer 3 aggregation service
Optional Service Available to any GigaPOP
St. John’s
Calgary Regina
Winnipeg
Large channel
WDM system
Charlottetown
Vancouver
Europe
Montreal
Customer controlled
Seattle optical switches
Fredericton
Halifax
Ottawa
Chicago
Toronto
New York
eScience
 The ultimate goal of e-science is to allow students and eventually members
of the general public to be full participants in basic research.
 We have seen in other fields like bird census, comet watching, SETI@home,
public are interested in participating in basic research
 Using advanced high speed networks like CA*net 4 and novel new concepts
in distributed peer to peer computing, called “Grids” many research
experiments that used to require high end super computers can now use the
computer capabilities of thousands of PCs located at our schools and in our
homes.
 High performance computers that are part of C3.ca can be seamlessly integrated
with eScience distributed computers using CANARIE Wavelength Disk Drive
over CA*net 4
 Allows researcher access to the significant computational capabilities of all
these distributed computers at our schools and homes
 Will also allow students and individuals to be a full participant in the analysis
and basic research.
 With e-science it might be possible that the next big scientific discovery
could be by a student at your local school.
SETI@home
Demonstrated that PC Internet Computing
Could Grow to Megacomputers



Running on 500,000 PCs, ~1000 CPU Years per Day
 Over Half a Million CPU Years so far!
 22 Teraflops sustained 24x7
Sophisticated Data & Signal Processing Analysis
Distributes Datasets from Arecibo Radio Telescope
Arecibo
Radio Telescope
Next StepAllen Telescope Array
Forest Grid– on CA*Net 4
Univ.
ORAN
Labs/Ministries.
Min. of For.
CFS
Northern
York
CFS
Laurentian
Min. of For.
U of A
Cornerbrook
Min. of For.
Min. of For.
Waterloo
Alberta
UVic
Sask.
CCRS
BC
EDC
RSI
Quebec.
CFS
GLFC
U. Ottawa
CFS-HQ
UBC
Seattle
UNB
Ont.
Man.
CFS
PFC
Quebec.
NSF
VBNS
Min. of For.
JPL
CSA
Min. of For.
CFS
Atlantic
Chicago
STARTAP.
Maritimes
Min. of For.
European
New York
UCAID
Abilene
NISN
ESnet
DREN
NREN
Neptune eScience Grid
 Joint US-Canadian project to build large undersea dark fiber network off
west coast of USA and Canada
 Undersea network will connect instrumentation devices, robotic submarines,
sensors, under sea cameras, etc
 All devices available to students and researchers connected to CA*net 4
and Internet 2 networks
 Neptune will be used to gather research data in a variety of fields –
seismology, sea vents, fish migrations and population, deep sea aquatic life,
etc
 Distributed computing and data storage devices on CA*net 4 and Internet 2
will be used to analyze and store data
Neptune – Undersea Grid
Neptune eScience
Wavelength Disk Drives
St. John’s
Regina
Calgary
CA*net 3/4
Winnipeg
Charlottetown
Montreal
Fredericton
Vancouver
WDD Node
Halifax
Toronto
Ottawa
Wavelength Disk Drives
 CA*net 3 and CA*net will be “nation wide” virtual
disk drive for grid applications
 Big challenges with grids or distributed computers is
performance of sending data over the Internet
 TCP performance problems
 Congestion
 Rather than networks being used for
“communications” they will be a temporary storage
device
Conclusion
 Many governments have recognized the
importance of access to low cost dark fiber
as fundamental economic enabler
 It will be the 21st century equivalent to the
roads and railways that were built in the 20th
century