History, Operational Practices and Principles
at the
Collocation/Exchange/Peering Facility
32 Avenue of the Americas, Manhattan, NY
The NYSERNet space at 32 Avenue of the Americas serves multiple functions. It is a
collocation facility, an exchange point where interconnects between any two parties there can be
done quickly, home to the northeast nodes of many networks including Internet2, ESnet, NLR
and, more recently NOAA, and a landing point in the Americas of GEANT, many other research
networks and home to the peering fabric MANLAN. The space is adjacent to the mandatory
meet me facility for the building where NYSERNet and every carrier in the building has a cross
connect panel. NYSERNet can complete requested connections between two entities within our
cage, and can complete connections to any other carrier with equal ease because of the adjacent
meet me space. As the importance of that facility for the R&E community grows, Rudin
Management has worked with us to enable opportunities not originally imagined by any of us.
We could not run a collocation/exchange/peering facility with the speed and responsiveness we
enjoy without their understanding of the importance of this global R&E access point.
History
Not long after the attack on 9/11, and in the midst of the dot-com collapse, when it was not
apparent whether New York City would recover, NYSERNet embarked on its fiber deployment
in NYC. The risk was tremendous, but the general uncertainty in New York also enabled
conversations with organizations such as Rudin Management, owners of the building we chose
as the ultimate location of our primary PoP in NYC. Were we to start such a project today, such
opportunities would be few. When we signed with Rudin, the 24th floor where we are located
was empty except for the Meet Me facility, the building only about 40% occupied. Had it stayed
that way, our project, NYSERNet, and perhaps even the facility would have folded.
We committed to as much space as we could afford, far more than we would ever use, and Rudin
gave us a right of first refusal on additional space up to 150% of our committed amount. Our
original space is almost full, and we have exercised our expansion rights twice, the major
addition to house Internet2 and ESnet nodes. The building is now essentially full, and Rudin has
been managing space in such a way as to protect space contiguous to our cage so that we can
expand, as we have before, continuously.
In the course of this effort, personnel from Internet2, and particularly Steve Corbato, visited and
offered valuable advice. Seeing the advantages of 32 A of A and our developing relationship
with Rudin Management, Internet2 decided to put a peering switch in the NYSERNet space. But
we had no template about how to build and run such a facility, and had to deal with ongoing
uncertainty about the health of New York City even as we continued to invest in both the fiber
plant in Manhattan and this space. That level of overarching doubt was a blessing. When any
operational detail was being considered, the uncertainty made considering the issue abstractly the
only path forward, always in the framework of “What is best for the community”. Working
with Internet2, the Global NOC, and organizations who would be collocated in our space or have
circuits into MANLAN, a set of practices developed. The notes below are a first attempt to
codify the principles by which we operate in that space.
Operational Practices and Principles
Peering and Collocation
The collocation facility was purposefully designed to promote collocation and network peering
relationships, relationships that may take any of these forms:
1. Peering between and among networks operated by NYSERNet's member institutions, like
NYU and Columbia University
2. Peering between and among regional networks, like NYSERNet and MAGPI
3. Peering between and among national and international research networks, directly and via
MAN LAN
The facility also supports egress connections to networks collocated in and outside the space.
Explicitly, and as permitted by the operational rules of various networks:
1.
2.
3.
4.
NYSERNet members can connect to NYSERNet.
Regional networks can connect to Internet2, ESnet (typically via MANLAN), NLR
Collocated networks can cross connect with each other
Collocated networks and NYSERNet members can connect to commercial carriers via
the building-operated Meet Me Room, directly adjacent to the facility.
5. Any R&E network or educational or research organization may collocate equipment.
Although it has not happened yet, commercial entities could collocate as well, provided
the purpose was research and education.
The network operators alone determine with whom they will peer/cross connect and to whom
they will provide egress. Rudin Management makes no restrictions whatsoever except for a
prohibition of commercial to commercial peering, which is done solely in the Meet Me Room.
On several occasions Rudin Management has demonstrated their willingness to think outside the
box when opportunities and/or challenges arose not anticipated in our contractual terms.
Allowed Participants:
1. Any researcher and/or research and education organization, typically with work which
relies on data flowing through the collocation facility. This includes the research
components of corporations.
2. Government and other research labs and organizations (like ESnet) that support their
work.
3. International, national, and region research networks
4. Not permitted: activities of organizations supporting a core mission of commercial
networking. Research activities by such organizations would be allowed.
Responsibilities of the space leaseholder:
1. Equal treatment for eligible participants (e.g. same cost per unit of power, for comparable
services, for rack space, and for remote hands service). Some users of the facility may
have specialized requirements, and for any such needs pricing should likewise be
equitable.
2. Commitment that obligations to hold the space (including long term obligations) can be
met so that facility users are not vulnerable.1 If an organization using 32 A of A should
leave, NYSERNet has an obligation to maintain the space for all those remaining.
3. Pricing should be based on recovery of actual costs with reasonable assumptions about
occupancy, but without constant changes in pricing because of churn.
4. Charging should also seek to enable research. For example, cross connects at 32 A of A
are $100, almost covering labor required and nothing else.
5. Remote hands service available for emergencies 24-7, and scheduled remote hands
service available essentially for free.
6. Maintenance of a strategic partnership with the building owner.2
7. While the ultimate contractual obligations fall on the leaseholder, the space should be
treated as a public good. This fundamental principle drives all the above.
Responsibilities of all those involved in managing components of a
collocation/exchange/peering facility:
1. Operationally, all parties must meet on a regular basis to discuss and resolve
implementations, technical issues and user needs. Internet2, the Global NOC, and
NYSERNet meet weekly for MANLAN operations
2. Be flexible and agile about accommodating new technology, guided by the principles
above.
3. Similarly, remain flexible and able to deal with multiple needs – e.g. power, specialized
space and/or weight, programmatic help when possible, still developing requirements for
education programs.
4. Maintain open communication with other collocation/exchange/peering facilities is
essential as these facilities must work as a framework through which the special traffic of
the R&E community can flow with predictable characteristics.
Ultimately these facilities, like the research networks linking them, serve the research and
education community, so the ultimate guidance on their operations, and input for their meeting
future needs, must be grounded in the research and education community, including the
corporate research community, and in the organizations that provide the networks for those
communities. An advisory body with wide range opportunities for input by others is essential.
1
NYSERNet holds funds in reserve to ensure our ability to meet long term obligations.
When committing to this facility, we were well aware of the risks in a weakened New York City with uncertain
recovery prospects. We probably did not appreciate fully the extraordinary access that our willingness to invest
millions of dollars afforded us. Were we to lose this space, recreating anything like it would be very difficult.
2