Telecommunications Infrastructure Standard for Data
advertisement
Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942 Chris DiMinico cdiminico@ieee.org MC Communications IEEE 802.3 HSSG Data Center Standards ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers Co-chairs: Chris DiMinico & Jonathan Jew Published 2005 – available through TIA at www.tiaonline.org ANSI/NECA/BICSI-002 Data Center Design and Implementation Best Practices co-chairs: Jonathan Jew & John Kacperski best practices – complements TIA-942 – 2007 target IEEE 802.3 HSSG Who Developed TIA-942? Developed by the TIA TR-42.1.1 Network Distribution Nodes subcommittee as Project No. 3-0092 Participants included: Architecture & Engineering Firms Consultants End Users Manufacturers IEEE 802.3 HSSG Purpose of TIA-942 Encourage early participation of telecom designers in data center design process Fill a void by providing standards for planning of data centers, computer rooms, server rooms, and similar spaces. The standard encompasses much more than just telecommunications infrastructure. Close to half of the technical content deals with facility specifications. IEEE 802.3 HSSG Purpose of TIA-942 Define a standard telecommunications infrastructure for data centers Structured cabling system for data centers using standardized architecture and media Accommodates a wide range of applications (LAN, WAN, SAN, channels, consoles) Accommodates current and future protocols (e.g., 10+ GbE ) Replaces unstructured point-to-point cabling that uses different cabling for different applications IEEE 802.3 HSSG Purpose of TIA-942 Specifications for data center telecommunications pathways and spaces Recommendations on media and distance for applications over structured cabling Establish a standard for data center tiers to replace several proprietary standards. The TIA data center tier standard is: • A tool to evaluate existing data centers • A tool to communicate design requirements IEEE 802.3 HSSG Unstructured Cabling C D E F G H I J K L M N O P Q R S T U Heater - A/C 1 TABLE 247-7589 AB AC AD AE AF TABLE FI L E F IL E CABINET CABI NET AI AJ AK AL AM AN AO AP AQ F IL E CABINET FIL E CABINET POMONA A/C UNIT #6 AR AS HP PRINTER C D ROM SCANNER UNIT N ETWORK CABINET E7 NETWOR K CABI NET I1 NE TWORK CABI NET I2 NE TWORK CABI NET I3 I5 NE TWORK C ABINET I4 NETWORK CAB INE T NETWORK CABINET I7 NETWOR K C ABINET I6 CAB INE T I8 DESK NETWOR K H8 N ETW OR K CABINET H7 N ETW OR K CABINET H6 NETWORK CABINET G7 IBM CAB CABINET F6 N ETWOR K CABINET G6 N ETW ORK C ABINET F5 NETWORK CABINET G5 N ETW ORK C ABINET NETWORK C ABINET E6 IBM 3745-A11 IBM 3746-900 NETWORK C ABINET IBM TAPE RACK POMONA A/C UNIT #8 TAPE RACK NETWORK CABIN ET H4 NETWORK CABIN ET G4 N ETW ORK C ABINET F3 NETWORK NETWORK CABIN ET N ETW ORK C ABINET F2 CABIN ET H3 NETWORK CABIN ET H2 IBM DASD IBM S-390 R34 IBM DASD F4 NETWORK CABIN ET H1 G3 NETWORK CABIN ET G2 LAN CABINET F1 NETWORK C ABINET E4 NETWORK C ABINET E3 NETWORK C ABINET E2 NETWORK C ABINET E1 3745-41D IBM 3745-17A IBM 3745-L13 SAA CABIN ET D2 N ETW OR K CABINET A6 NETW OR K CABINET NETWORK CABINET C6 B7 N ETW OR K N ETWOR K CABINET A5 N ETW ORK E5 11 AH Heater - A/C POMONA A/C UNIT #7 N ETW OR K CABINET NETWORK CABIN ET H5 8 AG AT &T 7 10 AA DEFINIT Y 6 9 Z TABLE He ater - A/C 5 Y NETW OR K CABINET C5 CABINET B6 HALON PANEL NETWORK CABINET A4 NETW OR K CABINET C4 N ETWOR K CABINET B5 NETW OR K CABINET C3 N ETWOR K CABINET B4 N ETW OR K CABINET NETW OR K N ETWOR K A2 CABINET C2 CABINET B3 NETW OR K CABINET C1 N ETWOR K CABINET B2 N ETW OR K CABINET A3 NETWORK SAA CABIN ET CI SCO ROU TE R 4 X CONS OLE 3 W Heater - A/C STORAGE CABINET DESK 2 V F/X 247-7543 HALON ABORT HALON RELEASE C ABI NE T A1 B PA C BELL A D1 N ETWOR K CABINET B1 NETWORK CABIN ET G1 12 247-7538 13 14 STORAGE C ABINE T TAPE RACK 15 IBM AS -400 IBM AS-400 INSTALL A CABLE WHEN YOU NEED IT (SINGLE-USE, UNORGANIZED CABLING) TABLE 16 IEEE 802.3 HSSG LOCKER STORAGE CABINET LIEBERT PDU -C 3174 CABINET STORAGE CABINET 3174 CABINET 3174 CABINET 3174 CABINET 3174 CABINET 3174 CABINET Structured Cabling B C D E F G H I J K L M N O P Q R S T U V Heater - A/C 1 TABLE 247-7589 NE TWORK C ABINET I1 NETWORK CABINET I2 CAB IN ET I4 NETWORK CABINET I3 NETWORK NE TWORK CABIN ET I6 NETWORK C ABI NET I5 NETWOR K CAB INE T I7 NE TWORK CABI NET I8 IDF NETWORK C ABINET H6 IDF N ETWOR K C ABINET H5 T APE RACK TABLE AE AF AG FIL E FILE CABINET FIBER MDF COPPER & CORE MDF POMONA A/C UNIT #8 NETWORK C ABINET E7 NETWORK CABINET G7 NETWORK CABIN ET G6 N ETWOR K C ABINET G5 NETWORK CABIN ET NETWOR K C ABINET F6 NETWOR K C ABINET F5 NETWORK CABIN ET F4 G4 NETWORK CABIN ET F3 N ETWOR K C ABINET H3 NETWORK CABIN ET G3 NETWORK CABIN ET F2 N ETWOR K C ABINET H2 NETWORK CABIN ET G2 LAN CABINET F1 N ETWOR K C ABINET H1 NETWORK CABIN ET G1 C ABINET H4 TAPE RACK N ETWORK CABINET E6 AH AI AJ AK AL AM AN AO CABINET FIL E CABINET FIL E CABINET POMONA A/C UNIT #6 CD ROM SCANN ER AP AQ AR AS HP PRINTER IBM S-390 R34 IBM DASD IDF IDF IBM 3745s IBM CAB NETWOR K NETWORK IBM 3745-A11 IBM 3746-900 C ABINET C6 NETWORK CABINET B7 NETWORK C ABINET C5 NETWORK CABINET B6 NETWORK C ABINET NETWORK CABINET C ABINET A6 CABINET E5 N ETWORK CABINET E4 N ETWORK CABINET E3 N ETWORK CABINET N ETWORK CABINET E1 IBM NETWOR K C ABINET A4 C4 B5 NETWOR K C ABINET NETWORK A3 C ABINET C3 NETWORK CABINET B4 NETWORK C ABINET C2 NETWORK CABINET B3 3745-41D IBM 3745-17A IBM 3745-L13 SAA CABINET D2 NETWORK IDF C ABINET C1 SAA C ABINET D1 IDF IDF F/X 247-7543 H ALON ABORT NETWOR K C ABINET A2 NETWORK CABINET B2 NETWORK CABINET B1 H ALON PANEL NETWOR K C ABINET A5 N ETWORK E2 Mainframe IBM DASD 11 AD Heater - A/C POMONA A/C UNIT #7 NETWORK CABINET H8 NETWORK C ABINET H7 N ETWOR K 10 AC UNIT 7 9 AB AT&T 6 8 AA DEFINITY DESK 5 Z TABLE Heater - A/C 4 Y CISCO ROUTE R CON SOLE 3 X Heater - A/C STORAGE CABINET DESK 2 W H ALON RELEASE PAC B ELL C A BIN ET A1 A IDF IDF IDF 12 247-7538 13 14 15 S TOR AGE C AB IN ET TAPE RACK IBM AS-400 IBM AS-400 STRUCTURED CABLING SYSTEM (ORGANIZED, REUSABLE, FLEXIBLE CABLING) T ABLE 16 IEEE 802.3 HSSG STORAGE CABINET LIEBERT PDU- C LOC KER 3174 CABINET STORAGE CABINET 3174 CABINET 3174 CABINET 3174 CABINET 3174 CABINET 3174 CABINET Design Elements Cabling Design Facility Design Network Design Informative annex’s: Provide best practices Annex A - Cabling Design Considerations Annex B- Telecommunications infrastructure administration Annex C-Access provider information Annex D- Coordination of equipment plans with other engineers Annex E- Data center space considerations Annex F- Site selection IEEE 802.3 HSSG Design Elements Cabling Design: Copper and fiber cabling performance Connectors, cables, distribution hardware Cabling distances Space management Facility Design: Data center sizing Power distribution methodologies Pathways and spaces HVAC, security, operations, and administration. Flexibility, scalability, reliability and space management IEEE 802.3 HSSG Design Elements Network Design: Support of legacy systems Enable rapid deployment of new and emerging technologies such as 10 GbE and 10+ GbE copper and fiber applications. IEEE 802.3 HSSG Relationship of Spaces BUILDING SITE BUILDING SHELL GENERAL OFFICE SPACE TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center OFFICE BUILDING SUPPORT SPACE DATA CENTER SUPPORT STAFF OFFICES OPERATIONS CENTER ENTRANCE ROOM(S) TELECOM ROOM(S) for data center support spaces COMPUTER ROOM IEEE 802.3 HSSG DATA CENTER ELECTRICAL & MECHANICAL ROOMS STORAGE ROOMS & LOADING DOCKS Data Center Topology Entrance Room Entrance Facility Carriers Carriers Active Equip interconnect Offices, Operations Center, Support Rooms Secondary Entrance Room Active Equip interconnect Active Equip interconnect MC MC Carriers Carriers Back bone Cabling HC IC HC IC HC IC Horizontal Cabling interconnect Active Equip Storage Area Devices interconnect Active Equip ICHC Zone Distribution interconnect interconnect Active Equip Active Equip Data Center IEEE 802.3 HSSG TIA-942 Spaces Entrance Room (ER) - location of interface with campus and carrier entrance facilities Main Distribution Area (MDA) – location of main cross-connect (MC) Horizontal Distribution Area (HDA) – location of horizontal cross-connect (HC) Zone Distribution Area (ZDA) – location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) – location of equipment cabinets and racks IEEE 802.3 HSSG Data Center Cabling Main Distribution Area MC MC Backbone Cabling Horizontal Cabling HC IC interconnect Active Equip interconnect Active Equip ICHC Horizontal Distribution Area interconnect Zone Distribution Area Active Equip Equipment Distribution Area Horizontal cabling is the cabling from the horizontal cross-connect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area. IEEE 802.3 HSSG Horizontal and Backbone Cabling Recognized Cables: a) 100-ohm twisted-pair cable (ANSI/TIA/EIA-568B.2), category 6 recommended (ANSI/TIA/EIA568-B.2-1) b) multimode optical fiber cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-568-B.3), 50/125 micron 850 nm laser optimized multimode fiber is recommended (ANSI/TIA-568-3-1) c) singlemode optical fiber cable (ANSI/TIA/EIA568-B.3) d) 75-ohm (734 and 735 type) coaxial cable (Telcordia Technologies GR-139-CORE) IEEE 802.3 HSSG Horizontal cabling distances The maximum horizontal distance is 90 m independent of media type. The maximum channel distance including equipment cords is 100 m. The maximum cabling distance in a data center not containing a horizontal distribution area is: 300 m for an optical fiber channel including equipment cords. 100 m for copper cabling including equipment cords. IEEE 802.3 HSSG Backbone Cabling Includes cabling from MDA to ER, HDA Optional cabling between HDAs allowed Maximum backbone cable lengths depend on applications to be supported Centralized optical fiber cabling supported with interconnect, splice, or pull-through at the HDA Star topology with no intermediate crossconnects Various topologies permit redundancy and flexibility to support various data center sizes IEEE 802.3 HSSG Computer Room & Entrance Room Requirements Min clear height of 2.6m/8.5 ft Min door size 1m/3ft wide 2.13/7ft high Min dist floor loading 7.2 kPA/150lbf/ft2, recommended min 12 kPA/250 lbf/ft2 20 degrees C to 25 degrees C 40% to 55% relative humidity (reduces ESD) Any sprinkler systems must be pre-action system Common bonding network (CBN) – equipotential ground reference IEEE 802.3 HSSG Main Distribution Area Location of Main Cross-Connect (MC), the central point of distribution for data center structured cabling system Centrally located to avoid exceeding maximum distance restrictions (typically for E1s, E-3s, T-1s and T-3s) Install separate racks for Fiber, UTP, and coaxial cable distribution IEEE 802.3 HSSG Horizontal Distribution Area Location of Horizontal Cross-Connect (HC), the distribution point for cabling to equipment distribution area Distribution LAN, SAN, KVM switches and console servers located in HDA MDA may also include an HC for nearby equipment distribution area Number of HDAs depends on the density of cabling and the size of the data center IEEE 802.3 HSSG Horizontal Distribution Area The capacity of the cable tray system and the size of the cross-connect creates practical limits on the size of the HC Guideline is maximum of 2,000 4-pair UTP or coax cable terminations per HDA Arrange patch bays to minimize patch cable lengths and to simplify cable management Separate racks for fiber, UTP, and coax Locate switches and patch panels to minimize patch cord lengths IEEE 802.3 HSSG Zone Distribution Area Rack, cabinet, or under floor enclosure that houses a zone outlet (ZO) or consolidation point (CP) ZO - structured cabling termination for floor-standing equipment that cannot accept patch panels (e.g. mainframes and large servers). CP - intermediate termination point (e.g. cabling to areas where floor plan is uncertain or dynamic) No cross-connects within the ZDA No active equipment shall be located in the ZDA Maximum of 144 connections in a ZDA Maximum of one ZDA within a horizontal cable run IEEE 802.3 HSSG Equipment Cabinets IEEE 802.3 HSSG Front rails of cabinets must be recessed to provide adequate room for patch cables and wire managers Recommend 1-to-1 ratio of patching to cable management Arrange switches and patch panels to minimize patching between cabinets & racks Perforated tiles at front of cabinets One edge of cabinets placed at edge of tile Raised Floor Better appearance than overhead cabling. Allows higher power densities, better control of cooling, and more flexibility in location of cooling equipment Most stand-alone computer systems are designed for cabling from below Coordinate under floor cabling with mechanical & electrical engineers Recommend wire basket cable trays in hot aisles for telecom cabling IEEE 802.3 HSSG Example of Wire Basket Cable Trays For Cabling Under Raised Floor 24" 1.375" RAISED FLOOR TILE DIVIDER 18" 9" 5" 4" FIBER COPPER CABLING SUPPORT STRUT 36" 12" 18" 5" 4" FIBER COPPER CABLING SUPPORT STRUT Open Area 13.5" 24" On Center IEEE 802.3 HSSG 1" 1" Overhead Cable Trays Less expensive than raised floor systems Cable trays can be attached to the top of racks and cabinets (if they are uniform in height) Cable trays suspended from the ceiling provides more flexibility for supporting cabinets/racks of various heights and for adding and removing cabinets/racks. Cable trays can be installed with several layers Coordinate location with lighting, ducts, overhead conduits, overhead power IEEE 802.3 HSSG distribution Overhead Cable Tray Example 3 Layer cable tray system: Bottom layer – signal Middle layer – power Top layer – fiber Signal Reference Grid in brackets attached to lower layer of trays Fiber patch cables may be in fiber duct attached to threaded rods IEEE 802.3 HSSG Infrastructure Administration Informative annex with TIA-606-A standards compliant labeling scheme for all components. Labeling scheme extended for use in data centers Cabinets and racks labeled by location using tile grid or row/position identifiers All cabinets, racks, patch panels, cables, and patch cords should be labeled IEEE 802.3 HSSG Labeling Example AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS AT AU AV AW AX AY "X" COORDINATE AZ BA BB BC BD 01 02 Cat 6 x 24 03 04 05 RIGHT FRONT CORNER 06 07 CABINET AQ03 CABINET AJ05 08 09 10 11 12 13 "Y" COORDINATE Patch Panel for 24 Cat 6 from 1st Panel in Cab AJ05 to 2nd Panel in Cab AQ03 AJ05-A to AQ03-B Ports 01-24 01 02 03 04 05 06 07 08 09 10 11 12 13 14 A IEEE 802.3 HSSG 15 16 17 18 19 20 21 22 23 24 A Site Selection Informative annex with guidelines for selection of a site for a data center Architectural Electrical Mechanical Telecommunications Security Other IEEE 802.3 HSSG Facilities Specifications & Tiers Informative annex with general architectural, structural, electrical, mechanical, and telecommunications recommendations requirements Annex includes detailed architectural, security, electrical, mechanical, and telecommunications recommendations for each Tier Recommended specifications by tier are a uniform way to rate aspects of a data center design and are a starting point for initiating design requirements with qualified architects and engineers IEEE 802.3 HSSG Data Center Tiers Tier 1 – basic data center no redundancy Tier 2 – redundant components Single distribution path with redundant components Tier 3 – concurrently maintainable Multiple distribution paths with only one active Tier 4 – fault tolerant Multiple active distribution paths IEEE 802.3 HSSG Redundant Topologies Carriers Carriers Carriers Secondary Customer Maintenance Hole (Tier 2 and higher) Primary Customer Maintenance Hole (Tier 1 and higher) r3 Tier 2 Primary Entrance Room (Tier 1 and higher) Tie Secondary Entrance Room (Tier 3 and higher) r4 Tie Main Dist Area (Tier 1 and higher) Telecom Room Tier 3 Tier 4 Tier 1 Offices, Operations Center, Support Rooms Tier 3 Tier 1 Data Center Tie Carriers r3 Tier 4 Secondary Dist Area (Optional for Tier 4) Computer Room Horiz Dist Area Optional Cabling Horiz Dist Area Horiz Dist Area Equip Dist Area Equip Dist Area Zone Dist Area Equip Dist Area IEEE 802.3 HSSG Data Center Tiers Higher tiers correspond to higher availability, but also have higher construction costs Data Center can have different tier ratings for different portions of its infrastructure (architectural, security, mechanical, electrical, telecommunications) The overall rating for the data center is equal to the lowest tier rating Capacity of systems may need to be upgraded to maintain tier rating as data center load increases Human error and operating procedures have a major impact on availability IEEE 802.3 HSSG Conclusion TIA-942 is the first standard to specifically addresses data center infrastructure. Primarily a telecom infrastructure standard, but about half of the content deals with facility requirements. Provides a flexible and manageable structured cabling system using standard media. Builds on existing standards, when applicable Guidelines on a wide range of subjects useful to someone designing or managing a data center. An official tiering standard for determining the quality of a center. A way to objectively compare one center with another. IEEE 802.3 HSSG
