Halloween 2001
CNRC
1
CU Network
Overview
Halloween 2001
Alan Crosswell
alan@columbia.edu
CNRC
2
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
3
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
4
Columbia People
• 40K people:
•
•
•
•
•
•
5,700 CU undergrads
2,500 non-degree
12,000 grad students
3,000 Barnard, TC, UTS students
7,700 CU faculty
7,600 staff
Halloween 2001
CNRC
5
Columbia Places
Halloween 2001
CNRC
6
Columbia Places
• Morningside Heights
campus - 116/Bway
Halloween 2001
CNRC
7
Columbia Places
• Morningside Heights
campus - 116/Bway
• Health Sciences
campus - 168/Bway
Halloween 2001
CNRC
8
Columbia Places
• Morningside Heights
campus - 116/Bway
• Health Sciences
campus - 168/Bway
• Lamont-Doherty Earth
Observatory - Palisades
Halloween 2001
CNRC
9
Columbia Places
• Morningside Heights
campus - 116/Bway
• Health Sciences
campus - 168/Bway
• Lamont-Doherty Earth
Observatory - Palisades
• several smaller facilities
Halloween 2001
CNRC
10
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
11
Internet & Internet2 too
• OC-3 Commodity Internet (Applied Theory)
• currently 150 Mbps outbound/60 Mbps inbound
Halloween 2001
CNRC
12
Internet & Internet2 too
• OC-3 NYSERNet ATM
• Abilene OC-12 POS at NYC, Buffalo POPs
• vBNS, CANARIE OC-3 ATM at NYC POP
Halloween 2001
CNRC
13
Internet & Internet2 too
• RCN fast Ethernet Private Peering
Halloween 2001
CNRC
14
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
15
Metro Network:
Lamont-Doherty
• DS-3 leased line ATM
• T-1 leased line backup
Halloween 2001
CNRC
16
Metro Network:
Health Sciences
• 2 x 100 Mbps Ethernet
microwave (23 GHz)
• DS-3 ATM microwave
• See also: Cornell Univ
• looking at dark fiber.
Halloween 2001
CNRC
17
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
18
Morningside
Heights area
• 10 Mbps Ethernet
microwave to Carleton.
Halloween 2001
CNRC
19
Morningside
Heights area
• 10 Mbps Ethernet
microwave to Carleton.
• 11 Mbps IEEE 802.11b to
three residences.
Halloween 2001
CNRC
20
Morningside
Heights area
• 10 Mbps Ethernet
microwave to Carleton.
• 11 Mbps IEEE 802.11b to
three residences via…
• 100 Mbps 5.8GHz hop.
Halloween 2001
CNRC
21
Morningside
Heights area
• 10 Mbps Ethernet
microwave to Carleton.
• 11 Mbps IEEE 802.11b to
three residences.
• 100 Mbps 5.8GHz hop.
• 420 Mbps 5.8GHz RSN™
Halloween 2001
CNRC
22
Morningside
Heights area
• 10 Mbps Ethernet
microwave to Carleton.
• 11 Mbps IEEE 802.11b to
three residences.
• 100 Mbps 5.8GHz hop.
• 420 Mbps 5.8GHz RSN™
• 9 T1s & 1 T3 to nearcampus buildings.
Halloween 2001
CNRC
23
Morningside
Heights “ISP”
• 736 V.90 dialups
• 5,700 student and
faculty apartments in
143 nearby buildings
Halloween 2001
CNRC
24
OCS
• Campus and
Neighborhood POTS,
CATV, and:
• ISDN (phasing out)
• DSL (~50-100 users)
• copper to switch room
• own DSL hub
• RCN Cable modems in
six(?) buildings.
Halloween 2001
CNRC
25
10baseT
Ethernet
• 20,000 Ethernet ports
• 4,400 10baseT pillows
in 15 undergrad halls
• 500 10baseT in 20
frats
• Mostly cat 3 wiring.
Halloween 2001
CNRC
26
100baseT
Ethernet
• 251ET: 72 Ultra-10s
• Server farm
• Rewired Mudd, CSB,
CEPSR w/1500 cat5e
jacks.
• All renovations &
new construction are
cat 5+ (~6 bldgs)
Halloween 2001
CNRC
27
Separately administered
School & Dept Nets
• Some gigE uplinked
• Business/Law (3com)
• Barnard College
• 100baseFX or FDDI (pending gigE):
• Law, Teachers College, Architecture
• Bringing old dept-run nets back into fold.
Halloween 2001
CNRC
28
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
29
GigE Core Network
• Centrally funded to upgrade all 10 Mbps
shared ports to 10 Mbps switched with a
gigabit Ethernet core.
• 4 Year $7.7M project (done)
• Replaced Cabletron and Xyplex shared
hubs and FDDI backbone.
• Started as an ATM core. And stopped.
Halloween 2001
CNRC
30
GigE Core Network
• Installation started
summer 99.
• Core network.
• All Residence Halls.
• VIP academic &
admin bldgs.
Halloween 2001
CNRC
31
GigE Core Network
• Reverted early ATM OC-3 LANE bldgs to
gigE (Butler, Journalism, Furnald, Carman,
Watson, Computer Center).
• Converted school/dept LAN uplinks from
10 Mbps Ethernet to gigE or 100 Mbps.
Halloween 2001
CNRC
32
A Building Hub
• Catalyst 55xx, 4006, 29xx, 35xx
• Singly gigE uplinked to core switch/router
• New construction and cat5 renovations
puts one of these typically every 3rd floor.
• No config changes needed by users.
• One or more VLANs with 802.1q trunk
Halloween 2001
CNRC
33
Floor
Edge
Router
Router
Edge
Switch
Router
Floor
Edge
Edge
Router
Router
Core
Core
Switch
Switch
Edge
Router
Router
Edge
Router
Router
Halloween 2001
Floor
Switch
Edge
Edge
Switch
CNRC
Floor
Switch
34
C55xx
C6509
Edge
Router
MSFC2
C6509
MSFC2
C4006
C6509
C6509
MSFC2
MSFC2
C6509
C6509
C6509
C6509
MSFC2
MSFC2
C6509
C6509
MSFC2
MSFC2
Halloween 2001
C35xx
CNRC
C29xx
35
Campus bldgs:
Old copper
• Cat 3: Unshielded
twisted pair copper
• Was installed for
phone system.
• Small FDFs.
• Vertical risers.
Halloween 2001
CNRC
36
Campus bldgs:
Old copper
• Cat 3: Unshielded
twisted pair copper
• Was installed for
phone system.
• Small FDFs.
• Vertical risers.
• BDF.
Halloween 2001
CNRC
37
Campus bldgs:
shared Ethernet
• Cat 3 phone wiring
was also good for:
• Apple LocalTalk
• IBM Token Ring (4M)
• Ethernet 10baseT
• Basement hubs were
squirreled away in
closets or hallways.
• Shared 10 Mbps
bandwidth for bldg.
Halloween 2001
CNRC
38
Campus bldgs:
shared Ethernet
• Cat 3 phone wiring
was also good for:
• Apple LocalTalk
• IBM Token Ring (4M)
• Ethernet 10baseT
• Basement hubs were
squirreled away in
closets or hallways.
• Shared 10 Mbps
bandwidth for bldg.
Halloween 2001
CNRC
39
Campus bldgs:
New copper
• Cat 5e: Unshielded
twisted pair copper
• New construction and
renovation standard.
• Max. 100 meter run.
• Max. 1 floor vertical.
• 100 Mbps.
• RJ45 patch bays in
floor closets.
Halloween 2001
CNRC
40
Campus bldgs:
switched Ether
• Ethernet switches in
floor closets.
• 10/100 Mbps
dedicated per jack.
Halloween 2001
CNRC
41
Campus bldgs:
switched Ether
• Ethernet switches in
floor closets.
• 10/100 Mbps
dedicated per jack.
• Gigabit Ethernet
(1000 Mbps) fiber
uplink per switch.
Halloween 2001
CNRC
42
Campus bldgs:
switched Ether
• Ethernet switches in
floor closets.
• 10/100 Mbps
dedicated per jack.
• Gigabit Ethernet
(1000 Mbps) fiber
uplink per switch.
• Standard closet size
accomodates 2 racks.
Halloween 2001
CNRC
43
Horizontal Wiring
• TIA-568 Commercial Building Standard for
Telecommunications Cabling.
• 300 ft. maximum distance to closet.
• No more than one floor run vertically.
• Lucent Systimax category 5e+ plenum cable.
Halloween 2001
CNRC
44
Outlet Placement
• TIA-569 Commercial Building Standard for
Telecommunications Pathways & Spaces.
•
•
•
•
Min. 1 outlex box/faceplate per workstation.
Min. 1 outlet box/faceplate per 100 sq. ft.
T568B jack pinout.
4 jacks per outlet (2 data, 2 phone).
Halloween 2001
CNRC
45
Closets
• TIA-569 Commercial Building Standard for
Telecommunications Pathways & Spaces.
• Standard says minimum 1 closet PER FLOOR.
• We cheap out w/1 per 3 floors.
Serving Area
10,000 sq. ft.
8,000 sq. ft.
5,000 sq. ft.
Halloween 2001
Closet Size
10 x 11
10 x 9
10 x 7
CNRC
46
Closets
• Bare minimum
4’Dx6’Wx7’H interior
clearance.
• Double louvered
doors opening out.
• Interior plywood
walls.
• No other utilities!
Halloween 2001
CNRC
47
Closets
• Electrical, Enviro.
• Two 20A dedicated
single NEMA 5-20R.
• Utility outlet(s),
lighting.
• Ventilation 75° F
12,000 BTUH
Halloween 2001
CNRC
48
Closets
• Cable terminations
• EIA 19” relay racks.
• RJ45 patch bay
horizontal jack
wire.
• Telephone riser
cross-connections.
• Network optical
fiber risers.
Halloween 2001
CNRC
49
Closets
• Electronics
• Network switch.
• UPS.
Halloween 2001
CNRC
50
A Core Router
•
•
•
•
Cisco Catalyst 6509
Redundant MSFC/2 router w/Supervisor II
MSM->MSFC->MSFC/2
Concentrates several building hubs with gigE
uplinks.
• Dual gigE uplinks to dual core L2 6509s
• Dual Power supplies w/dual UPS.
• Some generator backup (working on more).
Halloween 2001
CNRC
51
A Core Switch
•
•
•
•
Catalyst 6509
Two of them at opposite sides of campus
One’s currently on generator power
Eight gigE uplinks from the eight
switch/routers
• We might put routers in these too.
Halloween 2001
CNRC
52
Older Network & Routers
• FDDI backbone
• ATM OC-3 cloud
• Only thing left is two WAN DS-3 links.
• 7507s for WAN, FDDI, ATM
• Dual 100baseFX into core (for now)
• Temporarily gigE (due to bug in port channel
support between 7507/6509).
Halloween 2001
CNRC
53
Routing Protocols
• Interior routing: EIGRP
• Would like to stop being Cisco zombies and
run OSPF one day.
• Exterior routing: BGP
•
•
•
•
vBNS, Abilene
NYSERNet schools & routers
Applied Theory, Uunet, RCN
All 6509 routers speak iBGP w/egress routers
Halloween 2001
CNRC
54
Conversion “Issues”
• Had to convert right-angle 25pr telco
connectors to straight-in!
• Had to convert 15A outlets to 20A.
• Long vs. short-haul GBICs.
• Expensive MM to SM conditioning cables.
• Pulled a lot of new SM fiber.
• Fiber testing: Plug and Pray.
Halloween 2001
CNRC
55
Conversion “Issues”
• MSMs vs. MSFCs vs. MSFC/2s
• MSM: limited ACLs; No Appletalk routing.
• MSFC: too little RAM to run iBGP.
• DoS caused by unrestrained fast hosts on
a fast network (p2p apps, etc.).
• Shaping outbound dorm traffic to 50 Mbps
• cost of Smartnet service contract.
Halloween 2001
CNRC
56
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
57
Advanced Networks &
Applications
•
•
•
•
•
•
•
Video Conferencing
Voice over IP
Multicast
Wireless 802.11b
IPv6
CDNs
Network Management
Halloween 2001
CNRC
58
H.323 video conferencing
•
•
•
•
•
•
Internet2 Commons just starting up.
Member of Videnet (www.cavner.org)
Zydacron Z340, Z360 & COMstation.
Cisco (Radvision) ISDN (H.320) gateway.
Tandberg Codec 5000/RADvision VIU-323
Polycoms just work.
Halloween 2001
CNRC
59
MPEG2
conferencing
• Litton CAMVision 2
• Full D1 720x480 video
• 48 kHz 384 kbps
stereo audio
• 15.4 Mbps
• Amnis, Minerva,
VBRICK
Halloween 2001
CNRC
60
Voice over IP
• Have a few Cisco phones.
• Telecom dept is deploying Siemens
phones on a very limited basis.
• SIP (Session Initiation Protocol) is taking
over the world.
• Switching our Cisco phones over to SIP
cuz it’s cool and runs on a real OS.
Halloween 2001
CNRC
61
Multicast
• CGMP/IGMP snooping on Catalysts
• Cisco/Borg Collective at work.
• PIM sparse mode.
• MBGP, MSDP w/Abilene, vBNS, NYSERNet,
Cornell.
• IGMPv3 & SSM soon.
• Main apps: Norton Ghost, Video
Halloween 2001
CNRC
62
Multicast
• A few conferences so far
• IPTV 2.0 and Osprey-100 card (H.261)
• In parallel with RealProducer G2.
• IPTV 3.0 and an Optibase MPEG2 D1.
• Used for several music events (MPEG1) and
Virtual Internet2 Member Meeting 10/2-5.
• Does MPEG2 but viewers are licen$ed.
Halloween 2001
CNRC
63
Multicast
• Would rather be using a real operating
system.
• Many equivalent apps that use multicast
but that don’t work with each other (Real,
WMT, vic/vat/IPTV, videocharger, Litton,
etc.)
• Need to learn a lot about video production.
Halloween 2001
CNRC
64
Wireless 802.11b
• Central outdoor campus spaces covered.
• Putting them in lounges, libraries,
classrooms, Lerner, and so on.
• Roof-to-roof applications for backhoe
protection and near-campus buildings.
• Linked NYAM across Central Park.
Halloween 2001
CNRC
65
Campus bldgs:
Wireless LANs
• IEEE 802.11b
deployed on small
scale today.
• Not as fast as wired
Ethernet.
• Applications:
• Outdoors.
• Conference rooms.
• Lecture rooms.
Halloween 2001
CNRC
66
IPv6
• IPv6 via NYSERnet/Abilene IPv4 tunnels.
• Global /48 assigned by NYSERNet.
• EUI-64 addressing (MAC address)
• Wireless subnet
• CS, AcIS subnets
• More as requested
• Bind9 IPv6 on curta.cc.ip6.columbia.edu
Halloween 2001
CNRC
67
Content Delivery Network
• Akamai 15-machine Edge Node
• Content Provider
• CVN
• Columbia Interactive
• Special events
Halloween 2001
CNRC
68
Performance Management
•
•
•
•
•
•
Surveyor
NLANR Active Measurement Probe (AMP)
NLANR PMA (OC3mon) – optical splitter
Cricket
Sampson
Aperserv
Halloween 2001
CNRC
69
Performance Management
• Policy routing
• Forcing non-I2 connector traffic to I1.
• CS I1 vs. I2 comparison project.
• QoS
• Shaping of outbound commodity traffic.
• Fair queuing.
• Security
• NBAR attempting to label Code Red, etc.
Halloween 2001
CNRC
70
Outline
•
•
•
•
•
•
•
A little about Columbia
Wide area Internet[2] connections
Metro area inter-campus network
Main campus network
Gigabit Ethernet experience
Advanced networks and applications
Future plans
Halloween 2001
CNRC
71
Future Plans
• New Construction
• K-8 School @110th/Bway
• New School of Social Work @121st/Morn
• Expansion to apartments?
•
•
•
•
Many are already passed by our fiber.
Several have 1st floor networks for offices.
Fiber routes pass UTS, MSM, I-house.
Fiber routes to academic buildings are via apartments
Halloween 2001
CNRC
72
NYC Dark Fiber Project
•
•
•
•
NYSERNet built & owned or leased.
Fiber ring around Manhattan, parts of Bx.
Cost target: <= current OC-3 local loops.
36 initial potential connectors
• Universities, Hospitals, Museums, NYPL
• Vendor-neutral ISP colo hotel(s) possible
• Even more important post-9/11.
Halloween 2001
CNRC
73
www.columbia.edu/acis/networks/advanced
Halloween 2001
CNRC
74