ICT Standards and Guidelines Segment 106 Buildings, Rooms and Environment Main Document (Version 2.0) Disclaimer The Office of the Minister of State for Administrative Reform (OMSAR) provides the contents of the ICT Standards and Guidelines documents, including any component or part thereof, submission, segment, form or specification, on an 'as-is' basis without additional representation or warranty, either expressed or implied. OMSAR does not accept responsibility and will not be liable for any use or misuse, decision, modification, conduct, procurement or any other activity of any nature whatsoever undertaken by any individual, party or entity in relation to or in reliance upon the ICT Standards and Guidelines or any part thereof. Use of or reliance upon the ICT Standards and Guidelines is, will be and will remain the responsibility of the using or relying individual, party or entity. The ICT Standards and Guidelines are works in progress and are constantly being updated. The documentation should be revisited regularly to have access to the most recent versions. The last date of update for this document was June 2003. Table of Contents - Buildings, Rooms and Environment 1.0 2.0 3.0 4.0 5.0 6.0 Executive Summary for Buildings, Rooms and Environment ..................... 1 The Background of Buildings, Rooms and Environment ............................ 2 2.1 The Scope of Buildings, Rooms and Environment .................................. 2 2.2 The Benefits of Standardization ........................................................... 2 2.3 Policies to Follow for Buildings, Rooms and Environment ........................ 2 2.4 Risks Resulting from the Standardization Activities ................................ 3 2.5 Related Documents ........................................................................... 3 2.6 How to Use This Document? ............................................................... 3 2.7 Related Terms and Acronyms ............................................................. 3 2.8 Related Segments and Cross References .............................................. 4 2.9 Related International Standards .......................................................... 4 2.10 All Segments in the ICT Standards and Guidelines ................................. 5 Data Center Infrastructure and Physical Considerations .......................... 6 3.1 Security/Access ................................................................................ 6 3.2 Access to Restricted and Sensitive Data ............................................... 6 3.3 Secure Housing Structures ................................................................. 7 3.4 Visitor Restrictions ............................................................................ 7 3.5 Open Doors ...................................................................................... 7 3.6 Piggy-Back Access ............................................................................. 7 3.7 Intermediate Holding Area ................................................................. 7 3.8 Terminated and Transferred Employees ............................................... 8 3.9 Validated Employees List .................................................................... 8 Building Layout ........................................................................................ 9 4.1 Structural Considerations ................................................................... 9 4.2 Equipment Racks and Cabinets ......................................................... 10 4.2.1 Rack Depth .......................................................................... 10 4.2.2 Rack Width ........................................................................... 10 4.2.3 Rack Stability ....................................................................... 10 4.2.4 Seismic Reinforcement ........................................................... 11 4.2.5 Types of Racks ...................................................................... 11 4.3 Flooring ......................................................................................... 11 Environmental Control ........................................................................... 13 5.1 HVAC ............................................................................................. 13 5.2 Data Center Supplemental Air Conditioning Requirements Calculations .. 13 5.3 Capacity......................................................................................... 14 5.4 Redundancy ................................................................................... 14 5.5 Rack Mounted Equipment Cooling ...................................................... 14 5.6 Remote Monitoring and Control Capability .......................................... 14 5.7 Acoustic Noise ................................................................................ 15 5.8 Lighting ......................................................................................... 15 5.8.1 Ambient Light levels .............................................................. 15 Power Source ......................................................................................... 17 6.1 System Capacity ............................................................................. 17 6.2 DC (Direct Current) Input Power (-48 VDC) ........................................ 17 6.3 AC (Alternating Current) Input Power ................................................ 17 6.4 Data Center Input Power .................................................................. 17 6.5 Distributed DC Power Configuration ................................................... 18 6.6 Rotary BPS ..................................................................................... 18 6.7 Types of Redundancy ....................................................................... 19 6.8 Backup Power Supplies - BPS ........................................................... 19 6.9 Available BPS Solutions .................................................................... 20 7.0 8.0 9.0 10.0 6.9.1 On-line BPS .......................................................................... 20 6.9.2 Line-interactive BPS .............................................................. 20 6.9.3 Off-line BPS or Standby Power Supply (SPS) ............................ 21 6.10 BPS Solutions - Additional Criteria ..................................................... 21 6.11 Guidelines for Selecting a BPS .......................................................... 22 6.11.1 Transients ............................................................................ 22 6.11.2 Line Noise Rejection .............................................................. 23 6.11.3 Harmonics ............................................................................ 23 6.11.4 Isolation .............................................................................. 24 6.11.5 Waveform and voltage conditioning ......................................... 24 6.11.6 BPS Synchronization .............................................................. 24 6.11.7 Switching Circuitry ................................................................ 24 6.12 Simple Network Management Protocol (SNMP) and BPS ....................... 25 6.13 Backup Generators .......................................................................... 25 6.13.1 Capacity............................................................................... 25 6.13.2 Fuel Provisioning ................................................................... 25 6.14 Multiple Circuits .............................................................................. 25 6.15 Electric Company Supply Line ........................................................... 26 6.15.1 Capacity in kVA ..................................................................... 26 6.15.2 Redundancy ......................................................................... 26 6.16 Power Planning and Considerations ................................................... 26 6.16.1 Total Power Requirement ....................................................... 26 6.16.2 Number, Location and Type of Power Connections ..................... 26 6.16.3 The Potential for Expandability/Re-Configurability ..................... 27 6.16.4 The Backup Time Required In Case of Power Outage ................. 27 6.16.5 Load Priority ......................................................................... 28 6.16.6 Recovery from Failure of Power Protection Equipment ................ 28 6.16.7 Overload Protection ............................................................... 29 Fire Retardation ..................................................................................... 30 7.1 Primary System .............................................................................. 30 7.2 Inergen Gas ................................................................................... 30 7.3 Carbon Dioxide ............................................................................... 30 7.4 FM-200 .......................................................................................... 31 7.5 Redundant Fire Suppression System .................................................. 32 7.6 Smoke Detection ............................................................................. 32 7.7 Redundant Fire and Smoke Detection ................................................ 32 Grounding and Lightning Protection ....................................................... 33 Documentation ....................................................................................... 34 9.1 General .......................................................................................... 34 9.2 Definitions ...................................................................................... 34 9.3 System Description and Documentation ............................................. 35 9.3.1 System Overall Documentation ............................................... 35 9.3.2 Subsystem Documentation ..................................................... 35 9.4 Hardware Documentation ................................................................. 35 9.4.1 General ................................................................................ 36 9.4.2 Module Documentation .......................................................... 36 9.4.3 Sub-module (Hardware Device) Documentation ........................ 36 9.5 Installation Documentation ............................................................... 36 9.6 Site Documentation ......................................................................... 37 9.7 Operation and Maintenance Document ............................................... 37 9.8 Disaster Recovery Planning Documents .............................................. 38 Appendix A – Calculating HVAC Capacity ................................................ 39 Figures - Buildings, Rooms and Environment Figure 1 : Rack Stabilization ............................................................................. 11 1.0 Executive Summary for Buildings, Rooms and Environment One of the strategic guidelines for the ICT infrastructure for a Ministry or an Agency is that implementation of new concepts and products do not take place until functional requirements have been established, system specifications are stable and mature products are available that meet those specifications. This is particularly true for the requirements pertaining to the Buildings, Rooms and Environment that houses and contains the various ICT resources of the Lebanese Government. System specifications are to be updated and expanded as required to cater to the rapid evolution of new concepts and technologies within the ICT Data Center. Data Center within the context of this segment pertains not only to newly constructed data facilities, but also to existing facilities and rooms which are modified to accommodate and support computer systems and operations. This segment covers such issues as: Data Center Infrastructure and Physical Considerations Building Layout Environmental Control Power Source Fire Retardation Grounding and Lightning Protection Documentation Acquisition criteria and care for such equipment and facilities are discussed in the Evaluation and Selection Framework segment which can be downloaded from OMSAR's website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/EV. One of the crucial issues to consider in this segment is the relationship between ICT resources and their requirement in relation to Building laws and architectural and civil requirements. These disciplines have to be consulted when developing current or new Data Centers. Buildings, Rooms and Environment Page 1 2.0 The Background of Buildings, Rooms and Environment This section describes a variety of issues related to the background and scope of the segment. 2.1 The Scope of Buildings, Rooms and Environment The segment covers the following ICT environmental and physical aspects: 2.2 Data Center Infrastructure and Physical Considerations Building Layout Environmental Control Power Source Fire Retardation Grounding and Lightning Protection Documentation The Benefits of Standardization The benefits of standardizing the acquisition of equipment and practices in the area of buildings, rooms and environment are: 2.3 Reduce costs when bulk purchasing is followed Reduced training Increased experience when similar equipment is acquired Maintaining up to date specifications The private sector providing such equipment would gear its supplies accordingly and would hence improve its experience, availability, support and pricing. Experience can be shared regarding the acquisition and use of such equipment Policies to Follow for Buildings, Rooms and Environment System specifications are to be updated and expanded as required Data Center within the context of this segment should refer to newly constructed data facilities as well as to existing facilities and rooms Architects and civil engineers should be consulted on a variety of issues related to building laws and practices before final specifications are set. Coordination with the Standards and Guidelines set in related segments such as Information Integrity and Security and Hardware should be established to reduce conflicting requirements. (Review these segments at OMSAR's website for ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG). Attempts should be made to develop building law frameworks to ensure that both private and public sectors can benefit from such Standards and Guidelines. Buildings, Rooms and Environment Page 2 2.4 Risks Resulting from the Standardization Activities When standardization is implemented, the following risks may arise: 2.5 The mandatory criteria are not observed while acquiring various equipment and units The additional criteria to be used when evaluating such equipment are not used Technology changes are not incorporated while updating the specifications Using the specifications without a firm knowledge of the Ministry’s or Agency’s requirements Architectural and civil engineering considerations are not followed when setting the requirements of a Ministry or Agency. Related Documents There are no related documents to this segment. 2.6 How to Use This Document? The following summarizes the organization of this document: Data Center Infrastructure and Physical Considerations: Identifies some of the physical considerations and security measures needed to design a Data Center. See Section 3.0. Building Layout: Discusses the structural considerations of a Data Center and helps in choosing the appropriate ones. See Section 4.0. Environmental Control: Examines the setting of the Data Center and the means necessary to optimize it. See Section 5.0. Power Source: Identifies the different power configurations and power management tools and helps lay the ground for power planning and design. See Section 6.0. Fire Retardation: Helps in choosing the best fire protection means that would not ultimately harm neither the personnel nor the machines. See Section 7.0. Grounding: States the guidelines for efficient and correct grounding. See Section 8.0. Documentation: Gives all the necessary categories of documentation and what is required in each one. See Section 9.0. 2.7 Related Terms and Acronyms AC BTU/HR CO2 CD-ROMS DB ˚C Alternating Current British Thermal Unit / Per Hour Carbon Dioxide Compact Disc Read Only Memory Decibels Degree Celsius Buildings, Rooms and Environment Page 3 ˚F DC DP GWP HVAC HSSD ICT I/O KHz KVA LOAEL MTBF MHz MG NEC NOAEL O&M PIN PDU SNMP Telco TC THD UL UPS U VAC VDC 2.8 Degree Fahrenheit Direct Current Dispersion Penalty Global Warming Potential Heating, Ventilation and Air Conditioning High Sensitivity Smoke detector Information Communication Technology Input/Output Kilo Hertz Kilo Volt Ampere Lowest Observed Adverse Effects Limit Mean Time Between Failures Mega Hertz Motor Generator National Electrical Code No Observed Adverse Effects Limit Operation and Maintenance Personal Identification Number Power Distribution Unit Simple Network Management Protocol Telecommunication Telecommunication Closet Grounding Bus Bar Total Harmonic Distortion Underwriter Laboratory Uninterruptible Power Supply Units and in this case, EIA unit Voltage Alternative Current Voltage Direct Current Related Segments and Cross References The following segments have Standards and Guidelines that relate to this segment: 101 102 203 204 www.omsar.gov.lb/ICTSG/HW www.omsar.gov.lb/ICTSG/NW www.omsar.gov.lb/ICTSG/EV www.omsar.gov.lb/ICTSG/SC Hardware Systems Networks Evaluation + Selection Framework Information Integrity and Security Each page contains the main document and supplementary forms, templates and articles for the specific subject. 2.9 Related International Standards EIA-310-D: Electric Industry Alliance http://www.EIA.org ISO 3741 ISO 3743: International Standard Organization www.iso.ch/cate/d22818.html IEEE Standard 519: www.ieee.org/grouper.ieee.org/groups/519/ FM 200: http://www.e1.greatlakes.com/fm200/jsp/index.jsp NFPA 2001: http://www.nfpa.org/Codes/NFPA_Codes_and_Standards/ TIA/EIA 607: http://web.anixter.com/anixter/anixter.nsf/StandardsGuides/ANSITIAEIA607GroundingandBonding Buildings, Rooms and Environment Page 4 2.10 All Segments in the ICT Standards and Guidelines OMSAR's website for ICT Standards and Guidelines is found at www.omsar.gov.lb/ICTSG and it points to one page for each segment. The following pages will take you to the home page for the three main project document and the 13 segments: 101 101 102 103 104 105 106 201 202 203 204 205 206 207 www.omsar.gov.lb/ICTSG/Global www.omsar.gov.lb/ICTSG/Cover www.omsar.gov.lb/ICTSG/Legal www.omsar.gov.lb/ICTSG/HW www.omsar.gov.lb/ICTSG/HW www.omsar.gov.lb/ICTSG/NW www.omsar.gov.lb/ICTSG/TC www.omsar.gov.lb/ICTSG/DB www.omsar.gov.lb/ICTSG/OS www.omsar.gov.lb/ICTSG/EN www.omsar.gov.lb/ICTSG/QM www.omsar.gov.lb/ICTSG/SW www.omsar.gov.lb/ICTSG/EV www.omsar.gov.lb/ICTSG/SC www.omsar.gov.lb/ICTSG/DE www.omsar.gov.lb/ICTSG/RM www.omsar.gov.lb/ICTSG/CM Global Policy Document Cover Document for 13 segment Legal Recommendations Framework Hardware Hardware Systems Networks Telecommunications Database Systems Operating Systems Buildings, Rooms and Environment Quality Management Software Applications Evaluation + Selection Framework Information Integrity and Security Data Definition and Exchange Risk Management Configuration Management Each page contains the main document and supplementary forms, templates and articles for the specific subject. Buildings, Rooms and Environment Page 5 3.0 Data Center Infrastructure and Physical Considerations This segment outlines physical considerations critical to safe and secure operations which need to be taken into account when designing a reliable, high availability Data Center, facility or computer room. 3.1 Security/Access No information is secure if the premises and media are not properly protected from unauthorized access. The Agency Data Centers will be used to provide ICT, managed services and, in time, data backup capabilities. To protect Agency’s data, applications and systems, the Agency is advised to implement stringent physical and Data Center security measures on a 24x7x365 basis. For additional Standards and Guidelines pertaining to assuring the availability, integrity and confidentiality of government data and information systems, review the segment on Information Integrity and Security. This segment can be downloaded from OMSAR’s website on ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/SC. Whether acquiring a new computer facility or upgrading an existing facility to support computer operations, a security and safety risk assessment is mandatory to identify those management, operational and technical controls required to mitigate vulnerabilities which are unique to the locale and mission of the site. For Data Centers, related facilities and computer rooms, the following security measures are required: 3.2 Access to the Data Center shall have one point of entry and multiple (At least 2) emergency exits. The emergency exits shall have anti-pass back capability. Entry into the facilities shall require the use of magnetic cards and/or biometric data capture devices such as palm or hand scans. These latter devices can also be used in conjunction with PIN codes to enhance access controls to sensitive or classified information or systems. Loading docks shall be secure. The loading dock shall terminate in a secure receiving room. Access to the Data Center from the receiving room shall be restricted to authorized Data Center personnel. Security Cameras that record all activities from different angles shall be deployed to constantly monitor all Data Centers. Alarms and Notification: Security system shall implement automatic Police and Management notification. Alarm notification shall use both the wired and wireless cellular phone systems. 24 x 7 x 365 Security can be outsourced to an independent contractor. Guard station and Data Center glass walls shall be hardened for protection from firearm attacks. Access to Restricted and Sensitive Data Access to the Agency’s restricted or sensitive data will be limited to only those individuals who have been properly cleared for access. Restricted and sensitive data and its associated computer equipment will be housed in a secure environment, protected from unauthorized access and natural disruptions. Specific physical regulations required for sites housing ICT equipment Buildings, Rooms and Environment Page 6 which process restricted, sensitive or uniquely important data are indicated below. 3.3 Secure Housing Structures ICT equipment, which processes restricted, sensitive or uniquely important data, needs to be: 3.4 Visitor Restrictions 3.5 Located in a room, building or office, which can be closed and locked. Space will be locked during off-duty hours. Housed away from areas where serious man-made catastrophes could occur Limited access and restricted from unescorted or unprotected public access. Housed in areas with no identifying signs (e.g., computer room, wiring closet, etc.). Accessed through doors, which are self-closing, lockable (Accessible through either magnetic cards, cypher locks, combination locks, keyed locks) and have alarm feature for the door ajar for an extended period of time. Any type of nonkeyed lock should have a keyed backup. Visitors will not have in their possession inappropriate material including weapons, cameras, recording devices, etc. and must receive authorization/clearance for entry from appropriate Reclamation staff. All visitors must be escorted by an employee. Visitors will be limited to access during business hours or as appropriate scheduling requires. Visitors or the escorted employee must sign in and out. Visitors will wear temporary badges at all times during visits. Visitors without badges will be challenged by employees Open Doors When a door to a computer or data active/storage area must be propped open, an employee will be present to ensure no unauthorized access takes place. 3.6 Piggy-Back Access Each employee must be verified through his/her card key or visitor pass before being allowed access into controlled computer rooms. No piggy-back entrance behind an authorized employee/contractor is permitted. This access control depends upon good entry way design and regular training of security personnel and government employees, as well as dedicated enforcement at all levels. 3.7 Intermediate Holding Area For active computer areas, an intermediate holding area for visitors is strongly suggested. Buildings, Rooms and Environment Page 7 3.8 Terminated and Transferred Employees Upon the termination or transfer of an employee or contractor, access to the computer room will be terminated by close of business on the day the employee leaves. See the segment on Information Integrity and Security for additional guidance concerning administrative controls related to terminated and transferred employees. This segment can be downloaded from OMSAR’s website on ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/SC. 3.9 Validated Employees List A current list will be maintained of all validated employees who have been granted access to those spaces where ICT equipment and data is used and/or stored. Buildings, Rooms and Environment Page 8 4.0 Building Layout These are some considerations for the layout of the building housing the Data Center. The Agency will be hosting ICT resources. By their nature, these resources will require special consideration to avoid the risks that can damage the integrity of the Information Systems hosted therein. This section will discuss all of the required measurements to be taken into account when designing and building the Data Center. 4.1 Structural Considerations This subsection highlights the structural considerations to be examined during construction of Data Center, computer facilities and rooms. 1. Equipment and operations areas 2. Other areas 3. Presentation area Employee areas o Locker room/ Shower room o Recreation area / kitchen o Smoking Area Flooring 4. Main data room (Data Center) Telco room (Fiber cables and Telco equipment—separate from computer room) Server room Power conditioning room (Adjacent to switch room) HVAC (Heating, ventilation and air-conditioning) room Main switch panel room (Executive) Briefing Center Raised floors (Permit flexible layout and maintenance of cooling ducts and cabling) Cooling ducts for forced cooling Cabling o Cable trays (Overhead and beneath floor) o Shield Telco cabling from electrical systems General considerations Accessibility o Walkways o Equipment accessibility o Exit accessibility o Disadvantaged accessibility o Toilets for the disabled o Parking o Computers Buildings, Rooms and Environment Page 9 o 5. Elevators Floor space per employee Parking Secure storage rooms for equipment spares 4.2 Hardware Software Backup Media Computer furniture Consumables (Printer cartridges, diskettes, etc.) Equipment Racks and Cabinets The equipment racks and cabinets to be installed shall be freestanding high quality cabinets that save floor space organize equipment, eliminate cabling 'rat's nests', and physically protect the investment. The height of the enclosures will depend on the available ceiling height at the Data Center location. Enclosure manufacturers typically provide 42U (187 cm), 36 U (160 cm) or 22U (98 cm) of vertical equipment space for industry-standard 48 cm rack-mount equipment. Enclosures must meet EIA-310-D requirements. Enclosures must meet the standards required by the equipment the rack houses. 4.2.1 Rack Depth The EIA 310-D specification does not specify how deep a rack can or should be. The appropriate depth for a rack that will house rack-mountable components is determined by the depth of those components and by the space required for cable management. However, unless the rack is at least 74 cm deep, it will not be possible to use the cable management system that is recommended. Further, to service components in the racks, they must slide completely out of the rack. If the Agency intends to use the cable management arm and the rack depth is greater than 76 cm, the components will not slide all the way out of the rack. This is due to the restriction of the cable management arm which will therefore make it unserviceable. 4.2.2 Rack Width 48 cm racks shall adhere to the EIA 310-D specification. According to the specification, the inside distance between the vertical rails shall be a minimum of 45 cm). It should be noted that not all vendors hold rigorously to this dimension. 4.2.3 Rack Stability There are two main safety issues to consider when mounting components into racks: The stability of the rack Power distribution. Buildings, Rooms and Environment Page 10 For tip stability, stabilizing feet are required for the racks. The following table outlines the stabilization guidelines: Rack Height 22U 23U to 42U Requirements Two stabilizing feet extending 25 cm from the front of the rack. Six stabilizing feet - two fastened to the front and two to each side – each extending 25 cm Figure 1 : Rack Stabilization The height of the rack and of rack-mountable components is measured in units called ‘U’ (1U = 4.4 cm) Each rack shall be equipped with an integrated redundant power distribution scheme. 4.2.4 Seismic Reinforcement Additional stability may require the rack to extend through the raised floor and be secured to the room’s load bearing floor. In some cases, it may be necessary to retrofit the racks to achieve this goal. Furthermore, any such retrofit and or stabilization must neither interfere with the appropriate power and network connectivity nor prevent appropriate forced cooling if required. 4.2.5 Types of Racks Open Racks: Each rack shall have two power supplies that come from two different energy sources. Each rack should have two network ports each coming from a different cell for 100% server and power redundancy. This is particularly applicable to racks housing servers, switches/ routers; layer 4 redirectors, cache engines, firewalls, etc. This type of rack is appropriate when the equipment is housed in a secure limited access room. Closed Racks: Closed racks offer the Agency the same state-of-the art benefits and features as those in the rest of the Data Center, but with an elevated level of privacy and security. Caged Racks: For an even higher level of security for server equipment, the Agency should have individual enclosed and caged areas each with its own security. 4.3 Flooring The Data Center shall have a minimum raised floor height of .3 meter. In addition to cabling, this gives headroom for the HVAC ducts. When designing the raised floor, the following planning factors must be taken into consideration: 1. Floor Loading Average loading shall not exceed 750 Kg/m2. Buildings, Rooms and Environment Page 11 2. Peak loading shall not exceed 1500 Kg/m2. Dynamic loading and vibration Weight of personnel – Up to 10 people can be in the room. Weight of furniture Weight of equipment Seismic reinforcement. The size of tiles must accommodate the following: Cable management Forced cooling duct retrofits Buildings, Rooms and Environment Page 12 5.0 Environmental Control Equipment failures can be minimized if the appropriate environment is maintained. Furthermore, personnel comfort, safety and productivity is predicated upon the working conditions provided by the facility. This section will discuss the environmental elements needed to provide an optimal work setting for ICT employees. 5.1 HVAC The Agency shall equip its data and network operating centers with temperature and humidity control systems to assure optimal equipment performance and protection. To maintain a constant, cool temperature, the refrigeration system shall inject cold air into the racks from below through the raised floors, thus removing the excess hot air generated by the equipment. Each area should have its own precision temperature and humidity sensors that help maintain constant levels within established parameters. Like all equipment and systems in the data centers, the climate control system shall be totally redundant: 1. Ambient temperature range 2. Telco: 13–18C Computer areas:15–21C Work Areas:19–22C Humidity range Equipment Rooms: 20–50% Work Areas: 45–65% The design of supplemental HVAC systems for data centers should be contracted to experienced and reputable local mechanical engineering firms. Local contractors are fully familiar with the building materials and techniques utilized and with the range of conditions that can complicate relevant calculations (Orientation of facility, building warm-up, air infiltration, etc.) and impact cost. For a new facility, the responsibility for mechanical contractor selection should lie with the building’s general contractor. For upgrade of existing spaces, “blanket,” use-asrequired, contracts with proven contractors are recommended. It should be emphasized that data center HVAC systems must be sized conservatively, but reasonably accurately. Oversized systems may cycle on and off frequently, not ever running long enough to effectively de-humidify the environment. Undersized systems may allow room temperatures to rise well beyond an acceptable range for both occupants and equipment. 5.2 Data Center Supplemental Air Conditioning Requirements Calculations For budgetary purposes, a rough estimate of data center supplemental air conditioning requirements may be calculated as follows: Buildings, Rooms and Environment Page 13 1. If an equipment vendor will provide an estimate of the number of BTUs per hour produced by a piece of equipment, use that number and skip steps b, c and d below. 2. For all planned or installed equipment not covered by the previous paragraph, record the line input voltage (Volts or VAC) and continuous current requirement (Amps) from the manufacturer’s documentation. Multiply the line voltage of the piece of equipment by the continuous current that each draws at that input voltage to obtain volt-amps (VA). Add these numbers for all pieces of equipment. 3. Estimate the connected load in watts by multiplying the total from paragraph 2 above by .8 (Typical power factor for a varied load of this type). 4. Multiply the number derived at in paragraph 3 above by 3.412 to obtain an estimate of the number of BTUs per hour produced by the equipment. Add the result to the numbers obtained in the first paragraph. 5. Divide the result of paragraph 4 above by 12,000 to calculate the capacity, in tons, of air conditioning required. Use this result to obtain rough order of magnitude pricing for HVAC equipment, installation and operating costs from local suppliers. 5.3 Capacity To calculate the capacity of the HVAC, refer to Appendix A in Section 10.0. The calculations should account for the initial (Expected) load and should also account for projected expansion and future resource increases. 5.4 Redundancy 1+1 redundancy shall be deployed (Discussed in Types of Redundancy, Section 6.7). 5.5 Rack Mounted Equipment Cooling Many rack-mountable components draw in cool, ambient air through the front of the rack and exhaust hot air through the rear of the rack. The Agency shall install racks that have ventilated front and rear doors that support this method of cooling. This allows racks to be used without modification in raised floor computer rooms as well as remote locations. However, a vast majority of rack enclosures are designed to be used only in raised floor computer rooms, with cool air being forced in from the bottom of the rack and exhausted through the roof. Appropriate ducting must be designed to support this type of forced cooling rack. 5.6 Remote Monitoring and Control Capability It is desirable though not essential at the outset to remotely monitor and control the HVAC system. There does not appear to be any standard in the industry governing such control. Buildings, Rooms and Environment Page 14 5.7 Acoustic Noise The equipment shall not generate acoustic noise exceeding the following levels of sound pressure: 5.8 Not greater than 60 dB (A) re1 pW in premises where persons normally are present. Not greater than 80 dB (A) re 1 pW otherwise. For impulse noise or permanent tone the above levels shall be reduced by 10 dB (A). No installation shall introduce acoustic noise into its environment, which exceeds the existing ambient noise by 2 dB, during the quietest time normal to the area. The measurement shall be made at a distance of 30 meters from the installation. Testing procedures according to ISO 3741 or ISO 3743 shall be used to verify the requirements to acoustic noise. Lighting 5.8.1 Ambient Light levels Each area must be provided with ambient light levels appropriate for safely conducting the target operations for the area. For Data Centers, equipment rooms and spot (Or work area) light levels refer to the following standards for proper design, installation and maintenance: IEC 60050-845 (1987-12); International Electrotechnical Vocabulary. Lighting IEC 60064 (1993-12); Tungsten filament lamps for general lighting purposes Performance requirements. Maintenance Result Date: 2005-11-25. IEC 60081 (2002-05) Ed. 5.1 Consolidated Edition; Double-capped fluorescent lamps - Performance specifications. Maintenance Result Date: 2003-05-29. IEC 60357 (2002-11); Tungsten halogen lamps (non vehicle) - Performance specifications. Maintenance Result Date: 2004-11-18. IEC 60364-1 (2001-08); Electrical installations of buildings - Part 1: Fundamental principles, assessment of general characteristics, definitions. Maintenance Result Date: 2003-08-31. IEC 60364-5-55 (2002-05) Ed. 1.1 Consolidated Edition; Electrical installations of buildings - Part 5-55: Selection and erection of electrical equipment - Other equipment. Maintenance Result Date: 2006-12-31. IEC 60364-7-714 (1996-04); Electrical installations of buildings - Part 7: Requirements for special installations or locations - Section 714: External lighting installations. Maintenance Result Date: 2008-09-30. IEC 60364-7-715 (1999-05); Electrical installations of buildings - Part 7-715: Requirements for special installations or locations - Extra-low-voltage lighting installations. Maintenance Result Date: 2008-09-30. IEC 60598-2-3 (2002-12); Luminaires - Part 2-3: Particular requirements Luminaires for road and street lighting. Maintenance Result Date: 2006-07-31. IEC 60598-2-22 (2002-08) Ed. 3.1 Consolidated Edition; Luminaires - Part 2-22: Particular requirements - Luminaires for emergency lighting. Maintenance Result Date: 2005-08-22. IEC 61547 (1995-09); Equipment for general lighting purposes - EMC immunity requirements. Buildings, Rooms and Environment Page 15 Buildings, Rooms and Environment Page 16 6.0 Power Source The convergence of information technology equipment and telecommunications networks has brought about the need to provision equipment having a variety of input power requirements. The designer of ICT facilities today is faced with the challenge of how to configure the power system to support this variety of input power requirements as reliably and economically as possible. This section provides the guidelines for power management requirements. 6.1 System Capacity The power system capacity must be estimated based on expected initial loads and projected future expansion and resource increases. Capacity calculations must account for all equipment (Telco, ICT, HVAC, lighting, etc…) and must take into account redundancy of such equipment. 6.2 DC (Direct Current) Input Power (-48 VDC) Traditional telecommunications equipment requires -48VDC input power. Typical telecommunication power systems consist of multiple parallel-redundant rectifiers that convert commercial AC power to -48VDC power, charge lead-acid storage batteries and supply power to the critical load equipment. When other voltages are required, inverter/converter combinations are used to convert the -48 VDC. Long battery support times are required to support the critical load equipment in case of commercial AC power failure or rectifier failures. Battery support times range from a minimum of 1 hour to over 24 hours, with typical battery support times being 3 to 8 hours. Engine-generator pairs are used to provide AC power during sustained commercial power system failures. 6.3 AC (Alternating Current) Input Power Traditional information technology equipment, on the other hand, require AC input power, generally matching the commercially available AC power source configurations, typically 120 or 208 volts single phase AC. Typical information technology power systems include the use of AC BPS systems with battery systems sized to provide either the necessary time for an orderly shutdown or time to reliably get standby engine generator power systems on line. Virtually all-critical information technology facilities include permanently sited engine-generator systems and their associated automatic transfer switches to protect against sustained commercial AC power system failures. 6.4 Data Center Input Power The Data Center is a convergence of telecommunications and information technology equipment and will therefore require both -48VDC power and one or more of the commercial AC power voltages. This co-dependence demands equally high reliability and availability for the DC and AC power systems. Buildings, Rooms and Environment Page 17 DC power systems use -48VDC battery plants with long duration backup times (Low rate of discharge). DC power systems typically use 24 cells in series and are applied at a relatively low discharge rate. Further, the end-of-discharge and float voltages are carefully controlled. On the other hand, AC BPS systems typically use 120 to 240 cells in series and are applied at very high discharge rates (10 to 20 minute backup times) and deep discharge voltages down to 1.65 volts per cell or even lower. Note that there are other considerations (Apart from reliability) in the selection of the power system configuration. One of the basic considerations is the input power requirement of the critical load equipment. Other important considerations are size, installed cost, operating efficiency and maintenance cost. 6.5 Distributed DC Power Configuration Typical information technology Data Centers use commercially available packaged power distribution units (PDUs) to distribute AC power to the various load equipment. The PDU performs the conditioning, distribution and monitoring of power for the load equipment. The 480 VAC outputs of typical large BPS systems are distributed to a number of PDUs located throughout the center. The PDU typically contains an isolation transformer (To provide voltage step-down to 208/120 VAC, common mode noise isolation, local voltage adjustment and ground referencing) and output distribution panel boards with output circuit breakers, cables and receptacles to match load equipment requirements. The PDUs are intentionally located close to the load equipment to minimize the output distribution circuit length and voltage drop. A high availability power system approach for convergent telecom/ information technology systems has been named “hybrid distributed redundant power system.” On the surface, this approach may seem to be expensive but it often is the most costeffective means to achieve ultra high power reliability and availability [for critical electronic load equipment] due to reduced installation costs and smaller equipment footprints. This system has redundant rectifier systems—in an N+1 configuration—with integral output power distribution located very close to the load equipment and powered from the same AC BPS systems as other associated electronic load equipment. The footprint of these battery-less rectifier systems (DC PDUs) is often approximately the same as the normally required secondary DC distribution bays. To mitigate potential failures and allow maintenance within the AC or DC power systems without load shutdown, dual-BPS systems with redundant power paths, dual rectifier systems and dual-input load equipment can be utilized. 6.6 Rotary BPS More modern systems employ what is known as a ‘Rotary BPS’, which consists of a Motor-Generator (MG) pair. The basic advantages would depend on the overall configuration but can include lower cost and isolation from utility company supply fluctuations. More advanced systems would employ a hybrid configuration which would marry solid-state battery backed devices with MG pairs to provide redundant, isolated, high quality power systems. In this case MG pairs are located downstream of the DC/AC converters (Inverters) to provide a higher quality AC supply characterized by perfect sinusoidal waveforms. Buildings, Rooms and Environment Page 18 6.7 Types of Redundancy The ultimate goal of redundancy is to eliminate any single points of failure that would cripple the Data Center. There are various standard configurations for redundant BPS system deployment that attempt to do just that. A few examples are outlined below. 1. A+B (1+1) The 1+1 configuration is an example of parallel redundancy. In this configuration two BPS systems are connected in parallel. Each BPS is connected to the load via a static switch. A bypass circuit is provided for maintenance purposes. Central control is used to allow the systems to share the load and to control switching to the bypass circuit. All modules in this system energize a single bus to the load. 2. N + 1 The N+1 configuration is another example of parallel redundancy. In this configuration N BPS systems are needed to supply the system load and an extra BPS module is added for redundancy. Clearly this configuration can only provide protection for a single BPS failure. A bypass circuit is also provided and a central switch controls the system’s behavior. All modules in this system energize a single bus to the load. 3. Cascade Cascade redundancy connects two BPS systems in series to the system load. Both BPS systems are active at all times and each will support the full system load. 4. Ring-Bus and/or Tied Redundancy The ring-bus configuration is a refinement of the parallel redundancy approach. In a ring-bus configuration a set of bus switches are used to isolate parallel modules into one or more parallel systems each serving a subset of the load. This is more prevalent with rotary systems. Tied redundancy is an alternative to the ring-bus configuration. Two parallel BPS systems operate on separate buses, which can be tied by means of a circuit breaker. Each BPS services its own load but is capable of operating the entire load. 5. Parallel Distributed The distributed redundant or distributed parallel system is a configuration that attempts to create redundancy all the way to the input terminals of the load. This means redundant buses, power distribution systems and redundant input power. This system is particularly effective with dual input loads. This configuration will typically consist of two synchronized BPS systems supplying two power distribution units. Each power distribution unit would supply one of the load’s dual inputs. For single input loads a third power distribution unit is used which takes its inputs from the two PDUs. 6.8 Backup Power Supplies - BPS As power moves from the generating plant through the distribution grid to the computer installation, the power company gradually loses control of its quality. A good BPS protects against all the various power problems like: Blackout, a sudden, complete loss of voltage or drops when it is of very short duration Brownout, a significant reduction of voltage lasting from seconds to days Buildings, Rooms and Environment Page 19 Surge, where delivered voltage is substantially (20% to 100%) higher than nominal Spike, also called a transient, an extreme over-voltage of very short duration The damage from electrical problems, particularly spikes, is incremental and cumulative. There really is a difference between an Uninterruptible Power Supply (UPS) and a Standby Power Supply (SPS), but common usage now designates a unit properly termed SPS as BPS. We will call a unit of either sort a Backup Power Supply (BPS) All supplies have three common elements: 6.9 A battery, which stores electrical energy against power failures An inverter, which converts DC voltage supplied by the battery to the AC voltage required by the load A charging circuitry, which converts AC mains power to the DC voltage required to charge the battery Available BPS Solutions IEEE recognizes three categories of BPS, which they term BPS 6.9.1 On-line BPS On-line BPS: This is often called a true BPS to differentiate it from a SPS which connects the load directly to the inverter and the equipment always runs from battery power supplied by the inverter. Advantages: first, no switch-over time and no switch to fail ; second, the computer is effectively isolated from AC line problems; Disadvantages: first, cost is higher than an equivalent SPS; second, BPS batteries require more frequent replacement since they run constantly; third, the BPS running its inverter all the time results in a lower efficiency 6.9.2 Line-interactive BPS Line-interactive BPS, also called a single-conversion on-line BPS, differs from an online BPS in that the load normally runs primarily from utility power as long as the power is available. The inverter runs at all times and the load is always dynamically shared between inverter and utility power. Although line-interactive units do not isolate the load from the AC line to the extent that an on-line BPS does, they are quite good at maintaining clean steady AC to the load. Line-interactive BPS are common in data centers but uncommon in the PC environment. Line-interactive BPS is the recommended solution for Servers. Buildings, Rooms and Environment Page 20 6.9.3 Off-line BPS or Standby Power Supply (SPS) Any BPS used with a PC (Or even a small server) nowadays is almost certainly an off-line power supply, sometimes called a Standby Power Supply (SPS). They are often described as “uninterruptible” power supply which they are not. The defining characteristics of an SPS are that it has a switch and that the inverter is not always running. During normal operation the switch routes utility power directly to the load. When utility power fails that switch quickly disconnects the load from the utility power and reconnects it to the inverter. SPS are less expensive than on-line and line-interactive units because they can use a relatively inexpensive inverter, one rated for low duty cycle and short run time. Several SPS variants exist: Standard SPS has only two modes -full utility power or full battery power. Most entrylevel SPS models are standard SPS. Line-boost SPS adds line-boost mode to the two modes of the standard SPS. They have an extra transformer tap, which they use to increase output voltage by a fixed percentage (Typically 12% to 15%) when input voltage falls below threshold. Most midrange and high-end desktop SPS are line-boost SPS. Line-boost SPS are the recommended BPS for standalone desktops Ferro-resonant SPS uses a ferro-resonant transformer rather than the tap-change transformer used by a line-boost unit. These SPS units have several serious drawbacks. This is not recommended. 6.10 BPS Solutions - Additional Criteria The important characteristics of a BPS are the following: Volt-Ampere (VA) rating: the VA rating of a BPS specifies the maximum power the BPS can supply and is determined by the capacity of the inverter Run time: The runtime of a BPS is determined by many factors, including battery type and condition, Amp-hour capacity and state of charge; ambient temperature; inverter efficiency; and percentage load. The number of Amp-hours a battery an supply depends on how many amps you draw from it: the relationship between load and run time is not linear. Doubling load cuts run time by much more than half; halving load extends run time by much more than twice Output waveform: The output waveform is determined by the inverter, the most expensive component of a BPS. Better inverters-those that generate a sine wave or a close approximation- are more expensive. The cheapest units generate square wave output, which is essentially bipolar DC voltage with near zero rise-time and fall-time. Battery replacement method: Batteries must be replaced periodically. Better units have user-replaceable batteries. It’s less expensive and much more convenient to be able to replace batteries on site. Warranty: The length of warranty is a reasonably good indicator of the quality of the unit. Buildings, Rooms and Environment Page 21 Configuration options: Better BPS offer flexible options for setting such things as transfer voltage threshold, warning type (Audible, visual, email and or pager notification, etc.), delay before warning, warning duration and so on. Status indicator: Better units provide detailed LED or LCD status displays to indicate such things as load percentage, battery charge status, overload conditions and battery replacement required. Overload protection: Better units use a circuit breaker that can be reset by pressing a button instead of a fuse that is found on less expensive units. Receptacle configuration: Most units include two types of receptacle. The first sort are backed up by the battery; the second sort are surge-protected only and are useful for connecting items that need not to be run from the BPS. Manageability: All but entry-level BPS units include a network interface port that can be used with appropriate software for automatic shutdown. Midrange and high-end SPS may include Simple Network Management Protocol allowing SNMP manageability. (Simple Network Management Protocol) 6.11 Guidelines for Selecting a BPS Select BPS type according to application: Online and line interactive units are too large and expensive for most desktop applications. Consider them only for departmental servers and critical systems. For standard desktops and workgroup servers, buy an off-line unit. If your location is subject to frequent power problems, choose a line boost unit, which greatly extends runtime under brownout conditions. Here are some good practices: Consider buying one BPS for multiple desktops Get the best waveform you can afford Make sure the BPS has user-replaceable batteries 6.11.1 Transients Startup (Inrush) Currents The BPS must support short duration startup loads. Repeated power cycling should not affect the life expectancy or the MTBF of the system. Brownouts Brownouts are periods of low voltage in utility lines that can cause lights to dim and equipment to fail. Also known as voltage sag, this is the most common power problem, accounting for up to 87% of all power disturbances. The BPS must provide reliable protection from brownouts. Spikes Buildings, Rooms and Environment Page 22 Spikes are instantaneous high voltage transients that result from lightning and other (Potentially catastrophic) large-scale discharges. The power distribution system must provide reliable protection from spikes. Surge Protection Power surges are an increase in the voltage that powers your electrical equipment. Surges often go unnoticed, often lasting only 1/120th of a second, but they are quite common and are potentially destructive. The BPS must provide reliable surge protection. Blackouts Power failures, also known as blackouts, are the easiest power problem to diagnose. Redundancy is described above. The BPS must provide power during blackouts for the duration it takes the backup generator(s) to go live. 6.11.2 Line Noise Rejection The term "line noise" refers to random fluctuations - electrical impulses that are carried along with standard AC current. Standard BPSs include special noise filters that remove line noise. The amount of filtration is indicated in the technical specifications for each unit. Noise suppression is stated as Decibel level (db) at a specific frequency (kHz or MHz). 6.11.3 Harmonics In recent years, there has been an increased concern about the effects of nonlinear loads on the electric power system. Nonlinear loads are any loads, which draw current that is not sinusoidal and include such equipment as, solid-state motor drives, battery chargers, BPS systems and the DC power supply. While nonlinear loads are not new, their increased use means a larger percentage of any power system tends to be nonlinear. Nonlinear loads generate harmonic currents that flow from the load toward the power source, following the paths of least impedance. Harmonic currents are currents that have frequencies that are whole number multiples of the fundamental (Power supply) frequency. The harmonic currents superimposed on the fundamental current result in the non-sinusoidal current waveforms associated with nonlinear loads. There is also concern about the effects of harmonics on the rest of the power system. Harmonic current flows cause additional heating in the power system components, cause voltage distortion and may excite resonance or cause undesirable interactions in the power system. Very severe voltage distortion can result when the power system's inductive and capacitive reactance happen to be equal (Parallel resonance) at one of the nonlinear load's significant harmonic current frequencies (Typically the 5th, 7th, 11th or 13th harmonic). Harmonic current flows also reduce the system power factor. Simply adding power factor correction capacitors cannot compensate for the harmonic distortion power factor. To deal with the problems associated with harmonics, some utilities are even considering a harmonic current surcharge. For these reasons, IEEE Standard 519 is being revised to specify limits to the amount of current distortion a user may inject into the utility power system. These limits vary from 5% to 20% total harmonic distortion (THD), depending on how large the user is relative to the capacity of the utility system. Buildings, Rooms and Environment Page 23 Power system designs for Data Centers tend to be much more conservative than designs for general office areas or typical building wiring systems where harmonic currents from electronic loads (Like personal computers and terminals) have caused problems. Data Center designs usually anticipate nonlinear loads and specify capacities accordingly. Allowances for growth and expansion may increase capacities further. Because of the increasing incidence of building distribution transformers being overheated by nonlinear loads, UL introduced a transformer nonlinear load rating system in UL1561, called K-Factor, which is based on C57.110. K-Factor is a weighting of the harmonic load currents based on their heating effects on dry-type transformers. A KFactor of 1.0 indicates a linear load. The higher the K-Factor, the greater the harmonic heating effects. Transformers designed to accommodate the additional heating effects of particular levels of harmonic currents can be certified under UL1561 as having a particular K-Factor rating. Standard ratings are 1, 4, 9, 13, 20, 30, 40 and 50. Individual loads with K-Factors greater than 20 have not been widely observed. Computer rooms have been observed to have K-Factors of 4 to 9. Areas with high concentrations of single-phase computers and terminals have observed K-Factors of 13 to 17. 6.11.4 Isolation See the discussion in Harmonics in Section 6.11.3. 6.11.5 Waveform and voltage conditioning One important characteristic of BPS systems is the shape of the waveform downstream of the inverters. While it is a fact that many system power supply modules are the switching type, it is still important to provide these systems with a waveform that is as close to sinusoidal as possible. 6.11.6 BPS Synchronization A subtle aspect of certain redundant configurations is phase synchronization of the output AC signal of multiple BPS units. This is typically an issue when input power is absent and the BPS systems are operating on battery power. Out of phase transfers can cause undue system overloads and thus trip over-current protection devices (Circuit breakers). The BPS system must therefore be equipped with fail-safe synchronization circuitry. 6.11.7 Switching Circuitry Switching circuitry is perhaps the trickiest component to design in a redundant BPS. The MTBF of the system is inversely proportional to the complexity of the switching circuitry, which must accomplish the following: Isolate faulty components from the system bus Provide a maintenance bypass circuit Protect systems from current overloads (Except for startup load) Buildings, Rooms and Environment Page 24 6.12 Simple Network Management Protocol (SNMP) and BPS With Simple Network Management Protocol (SNMP) communications on a BPS, the unit can suddenly do much more than power conditioning and protection. It can: Log and print both events and data Provide multiple screen views, simple to complex Continuously monitor power quality Report on battery status, load and temperature Remotely reset locked equipment Perform self-diagnostics An SNMP-compatible BPS is able to report its status to multiple management consoles and it can be directed to change operating parameters. If the BPS has provisions for individually turning on and off the devices connected to it, the network manager can isolate sections of the network for security purposes, shut down devices to achieve electrical savings, even manage redundant portions of the network. In short, through SNMP, a BPS can become an intelligent part of a communications network. 6.13 Backup Generators 6.13.1 Capacity The Agency’s Data Center should be equipped with powerful, industrial generators that are capable of supporting the Data Center in case of prolonged power outage. Given the number and ratings on equipment and equipment-laden racks provided and which can be accommodated in the data room, a back-up generator should be placed in the most suitable configuration for redundancy purposes. 6.13.2 Fuel Provisioning The Data Center shall have high capacity fuel tanks that prolong the operation of the generators for several days. These are contingency measures and such tanks may not be used for normal operations. Environmental and city ordinances have to be met for such installations of fuel tanks. 6.14 Multiple Circuits Locating different equipment on different circuits allows a higher degree of flexibility in managing power source requirements. The following shall be on different circuits. Computer equipment Security/Access and monitoring equipment Environmental equipment Safety equipment Buildings, Rooms and Environment Page 25 6.15 Electric Company Supply Line 6.15.1 Capacity in kVA Each center shall have its usage calculated accounting for future expansion in the data hardware, HVAC and lighting needs. 6.15.2 Redundancy Redundancy in the utility company supply is mitigated by the use of backup generation and BPS equipment. A higher level of reliability however can be achieved by provisioning redundant multi-utility supply lines. 6.16 Power Planning and Considerations To maximize uptime and minimize problems and cost, a logical approach to power protection design is outlined below. A power protection solution must address the following elements: Total power requirement of the loads Number, location and type of power connections required by the loads Potential for expandability/re-configurability Method for management of the power protection equipment Load priority Recovery from failure of power protection equipment Overload protection 6.16.1 Total Power Requirement This needs to be calculated from the published equipment rating to be deployed. As with any installation, the load will grow with time and a careful log shall be maintained. 6.16.2 Number, Location and Type of Power Connections A typical BPS in the range of 1 to 5kVA provides 4 to 8 receptacles. In some cases, this may be enough receptacles. In most cases, more receptacles are required. It is not advisable to use extension cords or outlet strips to accomplish power distribution in a network room (Outlet strips are permitted only for temporary installations). This causes a problem if the loads are too many for the BPS or located too far from the BPS. To avoid that, it is necessary to permanently wire a BPS into the building wiring if more outlets or distant outlets are required. The network room must then be fitted with sufficient wall receptacles to supply all of the loads and these receptacles must be hard-wired back to the BPS. This is a significant cost that is often overlooked. In the case of a small network room with less than 8 loads, a single BPS may provide sufficient receptacles. If the BPS required is rated at less than about 1.4kVA, then the BPS can be plugged into a standard wall receptacle. If a larger capacity BPS is specified, Buildings, Rooms and Environment Page 26 then the BPS must be supplied by either a hard-wire arrangement or a special high power connector. Therefore, costly electrical wiring may be required at the BPS input even when the BPS output appears to be able to supply the load. In many installations, the number of power connections required by the loads is larger than the number of outlets available on the BPS. An additional problem is that the power cords of the protected equipment may not be long enough to bring them all to a central place. Therefore, when there is a significant number of a load connected to a single BPS some method of power distribution is required. The use of extension cords and outlet strips is not favorable, as explained above. The loads would then depend on the BPS to provide appropriate outlets. If the BPS needs to be shut down or removed for maintenance, this can lead to the situation where there may be no place for the loads to be plugged in. An alternative solution is to use multiple BPS systems in which each BPS has a common AC connector which plugs into a standard wall receptacle. Here, the BPS systems are distributed around the room such that the power cords of each protected component can reach a BPS. The loads can be plugged into the wall receptacle if the BPS needs to be removed for any reason. 6.16.3 The Potential for Expandability/Re-Configurability Most network rooms are subject to expansion or reconfiguration as user needs and networking technology change. The power protection scheme for the wiring center must be flexible, reconfigurable and expandable to allow for these inevitable changes. Purchase of a greatly oversized BPS along with permanent installation of a large quantity of wall receptacles hard-wired from the BPS is a solution. A more flexible scheme is to distribute BPS systems. BPS systems are chosen that can be plugged into standard wall power receptacles. This limits the power capacity of the BPS. Enough BPS systems are installed to meet the total power requirement of the loads. BPS systems can be added or moved to meet shifting requirements. Adaptability for future growth is assured and initial cost is reduced. 6.16.4 The Backup Time Required In Case of Power Outage The backup time required during an outage is directly related to the size of the BPS battery and has a large effect on the cost of the BPS. Therefore, it is important to consider this issue carefully. To determine the run time available from a BPS, it is necessary to know the size of the load accurately. Load sizes are usually overestimated during the BPS sizing process in order to ensure that the BPS is not overloaded. However, such overestimates can cause large errors in determining the run time and result in unnecessary expense. In most situations, the system administrator expects the BPS to provide power fault correction that is transparent to the users for brief outages, while providing safe and orderly system shutdown during extended outages. Although most operating systems can be shut down in less than 2 minutes, some database servers, particularly on UNIX based servers, can take up to 10 minutes to shut down. Therefore, a backup time of at least 10 minutes is recommended. Buildings, Rooms and Environment Page 27 For systems where the network must stay up, a BPS system with extended run capability must be used (This expense can be mitigated somewhat by using a "load priority" system as described later). One important factor in choosing an extended run BPS is the recharge time. The recharge time of all BPS systems increases as the number of batteries is increased. If there is a chance that the backup time requirement will increase in the future, then it is advisable to install a BPS that has expansion and heavy duty recharge capability. Additional battery packs ship by parcel post and can be plugged in by the administrator without the need to power down the load. 6.16.5 Load Priority In some Data Centers, it may be desirable that some loads continue operation for a very long time, while others simply need to be shut down gracefully. This objective can be accomplished using one of the following three techniques: Use a single large BPS with very long run time: This can be a good solution if the devices that require long run time consume most of the power. However, if the non-critical loads make up a large fraction of the total power consumption, then providing the unneeded extended run for these loads gives rise to a cost penalty, both in up-front and service costs. Use a large BPS with an intelligent load-shedding system: In this case the large BPS is equipped with a load-shedding accessory which allows the network manager to switch off less critical loads either manually or automatically. The accessory is expensive and requires programming but the cost can be more than offset by other savings. Use multiple smaller BPS with different run times: In this case, the critical loads are simply connected to BPS systems that have extended run capability while loads that require shorter run time are assigned to BPS systems with shorter run time. In this way the problem is solved in a very cost effective manner. 6.16.6 Recovery from Failure of Power Protection Equipment Every BPS will eventually require some form of attention or maintenance. The network administrator must have a plan for how to deal with this eventuality and try to ensure that the users of the network are affected as little as possible. There are basically only two kinds of failures of the BPS; either the BPS electronics can fail or the BPS battery can wear out. These failures can give rise to different or unexpected effects on the network users depending on how the power protection system was planned. A comprehensive recovery plan shall consider the following elements: Advance notification systems: Some BPS systems have the capability to test the battery system and determine that battery failure is imminent. Proper use of these systems can allow for scheduled (Instead of unexpected) maintenance and avoid system downtime. Hot-swap capability: Some BPS systems are capable of being removed for service or repair without the need to power down the load. These systems are useful Buildings, Rooms and Environment Page 28 when "after hours" shutdown for maintenance is not desired due to a 24hr uptime requirement. Single point failure minimization: A key advantage of using multiple BPS systems is that with minimal planning it can often be arranged that the failure of a single BPS will affect only a portion of system operations. It would be wise, for example, not to put all hubs on a single BPS. This benefit cannot be provided when using a single, large BPS. 6.16.7 Overload Protection All power systems are equipped with overload protection that is required for safety reasons. In the design of a network room, it is important to ensure that power circuits are not overloaded to avoid unexpected trips and downtime. There are three points of concern for overload protection; namely, the circuit breaker panels that feed the wiring center, the circuit breakers at the input of the BPS and those at the output of the BPS. The BPS will maintain the loads for a finite time if the side breaker trips. Unfortunately, in some cases this time is not adequate since only certain unavailable individuals may have access to the breaker panel. If the input breaker to the BPS trips, the BPS will again respond by maintaining the load. In this case, the administrator must identify the BPS affected, identify the breaker as the problem and reset it. There is no protection if the BPS output breakers trip. For this reason, this is the worst kind of fault condition. However, this problem can occur only with BPS systems of greater than approximately 1.5kVA capacity since these are the only BPS that have output breakers. An output breaker is required for any BPS that has a power capacity greater than the AC power receptacle rating. Unfortunately, the BPS warns only when the total load on all circuits is excessive. Buildings, Rooms and Environment Page 29 7.0 Fire Retardation This section will discuss the fire protection mechanisms needed to ensure the safety of the equipment as well as the employees. 7.1 Primary System It is recommended to utilize an FM-200 gas-based system that extinguishes fire by molecularly cooling the protected area. Inergen gas and carbon dioxide fire retardant systems can pose an unacceptable risk to humans in some operational environments and their use must be thoroughly reviewed prior to the decision for implementation. Water, which can damage sensitive electronic equipment, is not initially discharged. We shall compare briefly the three types of gas based extinguishing systems. 7.2 Inergen Gas This is a mixture of three gases: approximately 52% nitrogen, 40% argon and 8% carbon dioxide. The basic concept with Inergen is to flood an enclosure with a mixture of these three gases to a 43% to 52% concentration. This reduces the available oxygen concentration below that necessary for combustion. It should be noted that any gas that displaces oxygen could also create an atmosphere detrimental to human life. The claim of Inergen is that the small amount of carbon dioxide in the mixture causes humans exposed to extinguishing concentrations to breathe faster and deeper and promote the uptake of oxygen by the blood. Therefore, it depends on the modification of human physiology to claim safety for personnel exposed to extinguishing concentrations of the agent. The "envelope" of life safety with this concept is rather narrow. A little too much Inergen and there may not be enough available oxygen, despite the CO2. A little less Inergen and the extinguishing concentration may not be reached. Any modifications to the protected space – even the addition of equipment could change the volume of the space and thereby significantly alter the desired design criteria. To ensure safety, it is required that the design concentration result in at least 10% oxygen. If the oxygen concentration falls below 10%, personnel must be evacuated within 30 seconds. This agent is approved for use in occupied areas under NFPA 2001 if it meets the stated criteria. Because Inergen is stored as a gas, it cannot be discharged to achieve rapid buildup of total flooding concentration. Discharge times in excess of one minute and as long as three minutes have been noted. Additionally, because Inergen is stored as a very high-pressure gas, as opposed to FM-200 that is stored as a liquid, its storage efficiency is low. Substantial numbers of containers are necessary to store enough Inergen for even the smallest hazards. The additional floor space required to accommodate the numerous Inergen cylinders along with added service costs to maintain, weigh and check all these cylinders must also be considered. 7.3 Carbon Dioxide Carbon dioxide suppression systems are another clean agent alternative to Halon. When released the stored pressure acts as a propellant. Normally, there is about 21% oxygen Buildings, Rooms and Environment Page 30 in air. The addition of CO2 reduces the oxygen content to a point where combustion cannot exist and the fire is literally suffocated. As with Inergen, however, life safety consideration must be heavily weighed. It is not recommended that total flooding CO2 systems in normally occupied spaces unless arrangements can be made to ensure evacuation before discharge. The same restriction applies to spaces that are not normally occupied but in which personnel may be present for maintenance or other purposes. Evacuation can be difficult once the discharge starts because noise, greatly reduced visibility and the physiological effects of the carbon dioxide concentration may confuse the occupants. Other technical difficulties relative to temperature and humidity can arise with the release of carbon dioxide. When released, this agent is very cold and will cause a sudden drop in the temperature of the room. Some sensitive electronic equipment is limited to a maximum change of 15 F (9 C) per hour. Concurrent with a rapid drop in temperature will be a rapid increase in the relative humidity. Experimental evidence shows that the dew point can be reached quickly. This causes condensation to form on many equipment components. 7.4 FM-200 Chemically known as heptafluoropropane is a clean agent alternative to Halon 1301 which NFPA 2001 accepts for use in total flooding situations where human exposure is expected. FM-200 contains no ozone depleting chlorine or bromine. Additionally, it has a very low GWP (Global warming potential) of 0.7 and an atmospheric lifetime of 17–31 years. This is the shortest atmospheric lifetime zero-DP new clean agent available. In assessing the toxicity of halocarbon alternatives, the main concern is the consumer and worker exposure during a fire. The principal guideline is cardiac sensitization that is defined as increased susceptibility of the heart to adrenaline that may result in potentially fatal heart arrhythmias. Two terms, NOAEL and LOAEL, are used in evaluating cardiac sensitization. NOAEL is the No Observed Adverse Effects Limit when dogs are subjected to a predetermined concentration. LOAEL is the Lowest Observed Adverse Effects Limit and is the lowest concentration where at least one dog has begun to experience cardiac sensitization. The criteria for human exposure in alternative total flooding agents are judged on these two terms as to suitability for a normally occupied space. FM-200 has a low toxicity. It is an acceptable substitute for Halon 1301 and is currently listed by U.L. in systems. It has a NOAEL of 9% and a LOAEL of at least 10.5%, while it’s design concentration is 7%. Therefore, it can be used in normally occupied spaces. FM-200 is a low-pressure gas and can be housed in cylinders similar to the traditional Halon 1301 equipment. The concentration requirement of 7% by volume avoids any concerns of enclosure over–pressurization that must be carefully considered with the installation of Inergen and carbon dioxide. With a discharge of FM-200 suppression agent, total extinguishing concentration is reached in approximately 10 seconds. FM-200 is life supporting within design concentrations and has been found to be electrically non–conductive and safe for use on electrically charged equipment. The relatively high boiling point of FM-200 reduces the danger of thermal shock to delicate electronics that might possibly occur from the direct discharge of other agents such as CO2. The Environmental Protection Agency has stated that "FM-200 does not deplete Buildings, Rooms and Environment Page 31 stratospheric ozone and that it is the most effective of the proposed substitutes for Halon 1301". 7.5 Redundant Fire Suppression System A secondary dry-pipe water system is utilized only as a backup should the gas-based systems not fully extinguish a raging fire. The dry-pipe system is preloaded with water while the gas-system attempts to extinguish the fire and only discharges water in the event of a prolonged fire, which cannot be extinguished by gas alone. 7.6 Smoke Detection Early warning devices monitor the air for signs that indicate an impending fire condition. As a recommendation, for conventional smoke detection in sensitive electronic areas there should be a combination of photoelectric and ionization principle smoke detectors. photoelectric smoke detector is normally most responsive to fuels whose products of combustion are best defined as cool smoke. This type of smoke characteristically accompanies plastic type materials. Therefore when a raised floor is utilized it is recommended that photoelectric detectors be exclusively used in the sub floor. These detectors are most responsive to the cool type wiring/plastic fire that would be anticipated and they are very stable in high airflow areas. The ionization detector is most responsive to fires that typically are associated with hotter and/or flaming combustion such as paper. Because it is not known whether a plastic or a paper fire would be had depending upon the fire loading/contents, it is recommended that the ceiling be protected with both ionization and photoelectric detectors. Special emphasis must be made to strict adherence to the manufacturers and the NFPA No. 72 spacing requirements especially when high air flows are involved. 7.7 Redundant Fire and Smoke Detection An alternate and preferred method for many areas may be the use of an air sampling type system. This detection generally is up to 1000 times faster than conventional smoke detection however despite its high sensitivity it is a very stable and reliable detector. Unlike conventional detectors that are static devices that must wait until the smoke comes to the sensor. The HSSD air-sampling device actually draws air throughout the system. This sensor is relatively unaffected due to high airflow and stratification affects that typically reduce the sensitivity of conventional detectors. Buildings, Rooms and Environment Page 32 8.0 Grounding and Lightning Protection This section discusses standards and guidelines for proper grounding and lightning protection. All Data Centers, facilities and rooms shall adhere to the grounding guidelines set forth in TIA/EIA-607 (COMMERCIAL BUILDING GROUNDING AND BONDING REQUIREMENTS FOR TELECOMMUNICATIONS) plus any additional codes in Article (250 – GROUNDING) and (800 - COMMUNICATIONS SYSTEMS) of the NEC 1999. (http://www.tiaonline.org/standards). For lightning protection conform to standards established by the International Electrotechnical Commission (IEC) (http://www.iec.ch/), as discussed below. For an explanation of what constitutes a proper ground point for the Telecommunications bus bar to be attached to in a Telecommunications Closet see NEC-1999 Article 800-40. Briefly: Any surface to be grounded must be free of paint or any other coating that may affect a proper ground to be achieved. The surface must be prepared to provide a proper path to ground. Paint should be scraped or filed away until a metallic surface has been exposed. Then the proper grounding component can be attached to complete the system. Below are three general possibilities of acceptable ground points as long as they meet all the detailed requirements of the above-mentioned TIA/EIA-607 (COMMERCIAL BUILDING GROUNDING AND BONDING REQUIREMENTS FOR TELECOMMUNICATIONS) plus any additional codes in Article (250 – GROUNDING) and (800 - COMMUNICATIONS SYSTEMS) of the NEC 1999. Attach to Building or Structure grounding system. Metallic power service raceway or equipment enclosure Properly installed 2.7 meter ground rod to earth. All system components (i.e. ladder-rack, equipment racks, etc.) will be connected together and eventually will connect to the TC’s Grounding Bus Bar with a minimum of a # 6 solid or stranded copper wire with a green insulator The Bus bar shall be connected to the above mentioned building ground systems in such a manner as that it meets the above mentioned requirements set forth in TIA/EIA-607 (COMMERCIAL BUILDING GROUNDING AND BONDING REQUIREMENTS FOR TELECOMMUNICATIONS) plus any additional codes in Article (250 – GROUNDING) and (800 - COMMUNICATIONS SYSTEMS) of the NEC 1999. The Telecommunications Closet Grounding Bus bar shall attach to the above mentioned grounding system by a wire that is a minimum of # 6 solid or stranded green insulator copper wire. One of the greatest threats to computer equipment in Lebanon is lightning. Proper lightning protection is built upon the foundation for proper grounding, discussed above. Lightning protection level requirements are a function of the computing infrastructure supported by the Data Center, facility or room in question. Follow the IEC 61024-1-1 standard for tailoring the lightning protection systems to the computing protection levels dictated by the scope of the facility. For design, installation, maintenance and inspection of these lightning protection systems comply with IEC 61024-1-2. To establish or improve lightning protection for equipment in existing rooms and structures refer to IEC/TS 61312-4. Buildings, Rooms and Environment Page 33 9.0 Documentation This section specifies the overall requirements for documentation covering all systems and elements included as part of the project. The documents stated below are not necessarily complete and shall be considered as general guidelines. 9.1 General Documentation includes all information such as text, tables, drawings, computer printouts, listings, printed circuit board layouts and schematics, films, video recordings, microfilm aperture cards, microfiche, magnetic disks/tapes, CD-ROMs etc. 9.2 Documentation for all hardware, software, firmware, installation, testing, maintenance, diagnostics, training, procedures and operations shall be provided. The documentation shall be expressed in clear, consistent, readily understandable and defined terms and be well structured to: o Support installation, testing and commissioning of new equipment. o Support operation and maintenance of installed equipment. o Plan and dimension new systems or extensions to existing systems. The documentation shall be top-down structured from overall system descriptions down to the details of individual hardware elements and software source code. The documentation shall be organized in a form adapted and applicable for its specific use by specialized groups of staff. A catalogue showing the contents of each binder and a description and detailed list of all documentation for a system shall be provided. A Table of Contents shall be provided as an integral part of each manual/handbook. An index representing the level of detail in the documentation shall be provided for each manual/handbook. The Vendor shall describe the updating procedures applicable to the documentation System. The procedures used to maintain compatibility between the installed and/or modified equipment (Hardware and software) shall also be stated. In order to plan and dimension future extensions, the Vendor shall provide detailed records for each system he has delivered to the Agency. These records shall be kept for the lifetime of each system. Electronic documentation should be cross-referenced and linked as appropriate Definitions Terms that are relevant for documentation are defined as follows: System: This term will comprise a set of specified functions (e.g. an exchange) that is implemented as a group of individual (sub) systems. Subsystem: A subsystem is a part of the system that takes care of closely related functions of the system. Each subsystem can comprise combinations of both hardware and software. Module: A module is a part of the subsystem that takes care of closely related functions of the subsystem. Each module can comprise combinations of both hardware and software. Sub module: A sub module is a part of a module that takes care of either: Buildings, Rooms and Environment Page 34 o o 9.3 A special software task (Program) within a module A special hardware task (Hardware device) within a module. System Description and Documentation Any system deployed within the Data Center or network of the Agency shall have documents that describe the system architecture, its features and facilities and how the system interoperates with the telecommunication network and the Data Center. The system description documentation shall be organized to various degrees of detail and as a minimum comprise the following: 9.3.1 System Overall Documentation This document is the top-level part of the documentation and shall describe the main functions (Subsystems) and components of the System. The system overall document shall as a minimum comprise documentation of the following: Overall system architecture and its partitioning in subsystems (e.g. network peripherals, switching blocks, transmission links, control, maintenance aids). Operation and maintenance (O&M) features Equipment and technology Reliability Dimensioning and capacity Grounding of the overall system Subsystem documentation Module documentation 9.3.2 Subsystem Documentation The subsystem documentation shall provide details of each functional subsystem identified in the overall system description and how the subsystems are divided into modules. 9.4 The subsystem documentation shall describe both hardware and software subsystems. The division of functions performed by hardware and software and their interaction shall be explained for each subsystem. Module documentation: Detailed description of the hardware and software aspects of the module documentation shall be according to the hardware and software documentation sections respectively. The module documentation shall provide details of each functional module identified in the subsystem description and how the modules are divided into submodules. The module documentation shall describe both hardware and software modules. The division of functions performed by hardware and software and their interaction shall be explained for each module. The lower level documentation, i.e. the sub-module documentation, shall be according to the requirements specified in the hardware documentation and software documentation sections respectively. Hardware Documentation Buildings, Rooms and Environment Page 35 9.4.1 General Detailed hardware documentation as defined in this section shall be provided for each module identified and described in the previous sub-section. The detailed hardware documentation shall be provided in sufficient and appropriate levels and provide a comprehensive coverage of system principles of at least the following two levels: Module documentation. Sub-module (Hardware device) documentation. 9.4.2 Module Documentation The hardware functions of the module documentation shall provide hardware details of each functional module identified in the subsystem description. The hardware module documentation shall describe the physical realization of the equipment and contain diagrams, drawings and descriptions of all hardware elements and the following as a minimum: System description comprising, e.g. for switching, routers and server systems: The processor system The switching network Peripheral units The transmission network Survey schematics and module descriptions Functional block diagrams Functional descriptions Circuit descriptions comprising: o Functional description o Component list, including part numbers o Component data o Test points with indication of normal values o Adjustment data including test sets, cords and set up details. Rack layout Shelf layout 9.4.3 Sub-module (Hardware Device) Documentation The sub-module documentation shall provide details of each hardware sub-module identified in the module description. 9.5 Installation Documentation The installation documentation shall cover all activities to ensure that the system is installed and tested as specified and according to schedule. The installation documentation shall contain the following as a minimum: Detailed procedures for the installation and operational checks to verify installation. Detailed configuration documentation comprising: o System drawings o System capacities (e.g. exchange size) Buildings, Rooms and Environment Page 36 9.6 o Number of lines Installation drawings comprising: o Site drawings o Floor plans o Assembly drawings. Material list. The material items shall be listed under the various subsystem headings and the existing and added quantities for each item shall be reflected. Cabling plans comprising: o Cable route lists o Cable connection schematics. Wiring diagrams Grounding grid diagrams Installation schedule documentation Earthquake bracing diagrams Test lists and procedures List of test equipment required Total index (Catalogue) of all documentation Site Documentation The site document shall provide all as-built plant dependent documents such as mounting, allocation, cabling and termination documents as well as the site configuration. The site document shall contain the following as a minimum: 9.7 Site configuration Floor plan Cable running list Trunking diagram Power allocation Straps Grounding diagram Alarm connection I/O device manual Operation and Maintenance Document This documentation, covering all functions for operation and maintenance, shall adequately guide the operation and maintenance (O&M) staff in their assigned activities. Documentation for operation and maintenance shall contain: Description of O&M principles User manual Operation manual comprising: o Normal operation routines o Emergency routines o Back-up routines Maintenance manual comprising: o Maintenance routines o Description of routines to change data Buildings, Rooms and Environment Page 37 9.8 Fault diagnostic manual comprising: o Fault diagnostic routines o Fault localization programs Test and repair manual Command descriptions Alarm descriptions System analysis and statistics Documentation of performance monitoring Principles and description of special test equipment Functional test and procedure description to check special test equipment Operational limits on the following parameters: o Temperature o Humidity o Voltage supply o Earth resistance o Radio frequency fields o Electrostatic fields Disaster Recovery Planning Documents See the segment on Data Integrity and Security for documentation pertaining to Disaster Recovery Planning and associated requirements pertaining to Data Center and computer facility planning. This segment can be downloaded from OMSAR’s website on ICT Standards and Guidelines at www.omsar.gov.lb/ICTSG/SC. Buildings, Rooms and Environment Page 38 10.0 Appendix A – Calculating HVAC Capacity All Temperatures are in deg F. Windows exposed to the Sun Area (Sq feet) Max outside Temperature BTU/HR South E/W/SE NW NE N Maximum value 0 0 0 0 0 0 Inside Windows and Sky Lights Walls exposed to Sun 0 Linear Feet Max outside Temperature BTU/HR Light Construction Heavy Construction 0 0 Shade Walls not included in line 13 Partition 0 0 Ceiling or roof Area Max outside Temperature Ceiling with unconditioned or unoccupied space above Ceiling with attic space above (NO Insulation) Ceiling with attic space above (2 inch Insulation) Flat roof with no ceiling and no insulation Flat roof with no ceiling and 2 inch or more insulation Floor People (includes allowance for ventilation) Lights (if total wattage is known use line 34 else 35) Enter watts in next column Enter area in the next column Computer Load Enter total BTU/Hr for all machines if available Enter max wattage for computers if BTU/Hr value is not known Total Load Factor Sum all the load factors BTU/HR 0 0 0 0 0 0 0 0 0 0 0 0 Refer to Section 5.3 for a discussion on the requirements calculations. Buildings, Rooms and Environment Page 39