All rights reserved. Unless otherwise specified, this standard, or part thereof, may not be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, scanning, extracting, OCR and microfilming, without permission in writing from EPI Singapore at the address below. Published by EPI® (Enterprise Products Integration Pte Ltd) 37th Floor, Singapore Land Tower 50 Raffles Place Singapore 048623 Website: www.epi-ap.com Email: support@epi-ap.com Publication date: 10-June-2022 juan.martinez1113@gmail.com Juan_Martinez juan.martinez1113@gmail.com CRUR® Computer Room Utilization Ratio CRUR® - COMPUTER ROOM UTILIZATION RATIO Juan_Martinez juan.martinez1113@gmail.com This page intentionally left blank juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 2 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 1. Foreword Data centers are now, more than ever, under pressure to ensure that they are optimized and utilized to its maximum effectiveness and efficiency. Various metrics have been developed over time looking at optimizing the supporting facilities such as the power and cooling infrastructure. Metrics such as PUE, WUE etc. have made their inroads in the data center industry years ago and, although often abused as well, they have helped the industry to create awareness and improvements. However, it is EPI’s view that, in addition to existing facilities-oriented metrics, the best savings can be made at the computer room level. The reason is simple in that for every kW at the computer room floor there will be a multiple of it at the facility entry side. Furthermore, MTDC’s (Multi Tenanted Data Centers) could end up being penalized due to oversubscribed, and underutilized, computer room utilization factors as they would have an impact on achieving the MTDC’s overall PUE, WUE and other energy efficiency factors. In certain countries these requirements are mandated by law and it is anticipated that more countries will follow and adopt similar regulations. To address this issue, EPI has developed a new metric, CRUR (Computer Room Utilization Ratio), which data center owners, operators and end-users can immediately use to optimize the computer room. In addition, it can be applied for trend analysis which could positively impact the design criteria for future, to be build data centers, resulting in better efficiency, utilization and return on investment of those data center projects. CRUR is the first ever metric designed to measure and optimize the core of many data center efficiency issues which is the computer room itself. It is clear that CRUR, together with metrics such as PUE, will create a positive impact on the environmental, increasing its sustainability. This document provides clarity and guidance on the CRUR metric and how to apply it. This document is a living document and as such, revisions will occur over time to further enhance this document and the CRUR metric based on data center owners, operators and service providers’ feedback as well as new trends, technologies, methodologies and other influencing factors to this metric. We always welcome industry feedback. Please provide us your feedback and ideas for improvement via email at support@epi-ap.com juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 3 of 31 Juan_Martinez juan.martinez1113@gmail.com The other issue in the industry is that there tends to be a habit of overdesigning and underutilization, causing great wastage of valuable resources as well as creating negative impact on expected investment returns. CRUR® - COMPUTER ROOM UTILIZATION RATIO 2. Disclaimer and limitation of liability The document to which this Notice is an integral part of (the “Document”) has been prepared by EPI and other subject matter experts. The use or practice of contents of this Document may involve the use of intellectual property rights (“IPR”), including pending or issued patents, or copyrights, owned by one or more parties. EPI makes no search or investigation for IPR. When IPR consisting of patents and published pending patent applications are claimed and called to EPI’s attention, a statement from the holder thereof is requested, and EPI will take appropriate action to investigate the claim and subsequent action to ensure that “IPR” of the respected holder is protected. As such EPI reserves the right to modify this Document at any time should the need arise to protect the relevant “IPR”. This Standard does not attempt to address all regulatory requirements. It is the responsibility of the data center operator to investigate and comply with applicable rules and laws. ALL WARRANTIES, EXPRESS OR IMPLIED, ARE DISCLAIMED, INCLUDING WITHOUT LIMITATION, ANY AND ALL WARRANTIES CONCERNING THE ACCURACY OF THE CONTENTS, ITS FITNESS OR APPROPRIATENESS FOR A PARTICULAR PURPOSE OR USE, ITS MERCHANTABILITY AND ITS NONINFRINGEMENT OF ANY THIRD PARTY’S INTELLECTUAL PROPERTY RIGHTS. EPI EXPRESSLY DISCLAIMS ANY AND ALL RESPONSIBILITIES FOR THE ACCURACY OF THE CONTENTS AND MAKES NO REPRESENTATIONS OR WARRANTIES REGARDING THE CONTENT’S COMPLIANCE WITH ANY APPLICABLE STATUTE, RULE OR REGULATION, OR THE SAFETY OR HEALTH EFFECTS OF THE CONTENTS OR ANY PRODUCT OR SERVICE REFERRED TO IN THE DOCUMENT OR PRODUCED OR RENDERED TO COMPLY WITH THE CONTENTS. EPI SHALL NOT BE LIABLE FOR ANY AND ALL DAMAGES, DIRECT OR INDIRECT, ARISING FROM OR RELATING TO ANY USE OF THE CONTENTS CONTAINED HEREIN, INCLUDING WITHOUT LIMITATION ANY AND ALL INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING DAMAGES FOR LOSS OF BUSINESS, LOSS OF PROFITS, LITIGATION, OR THE LIKE), WHETHER BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR OTHERWISE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE FOREGOING NEGATION OF DAMAGES IS A FUNDAMENTAL ELEMENT OF THE USE OF THE CONTENTS HEREOF, AND THESE CONTENTS WOULD NOT BE PUBLISHED BY EPI WITHOUT SUCH LIMITATIONS. ** Note: The names of the standards listed are protected by copyright laws and are used in this standard for reference only. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 4 of 31 Juan_Martinez juan.martinez1113@gmail.com EPI is the certificate issuing body for the CRUR®. Only accredited CRUR® auditors are allowed to make claims for conformity to the standard based on the audit principles laid down in the EPI – Auditor Agreement. A list of valid EPI Certified Auditors can be found at the EPI website (www.epi-ap.com). While efforts are made to keep this list of certified auditors current some delay in posting is possible. The authenticity of any auditor can be verified by writing support@epi-ap.com. CRUR® - COMPUTER ROOM UTILIZATION RATIO 1. Foreword .................................................................................................................3 2. Disclaimer and limitation of liability .....................................................................4 3. Table of Contents ....................................................................................................5 4. Scope and application of the CRUR metric ..........................................................6 5. Normative References ............................................................................................7 6. Definitions of Terms, Acronyms, Abbreviations and Units of Measure ............8 7. Introduction to CRUR ..........................................................................................10 8. 8.1 8.2 Facilities: Power Capacity ....................................................................................12 SPCR: Stranded Power Capacity Ratio ....................................................... 12 PCUR: Power Capacity Utilization Ratio ................................................... 13 9.1 9.2 Facilities: Mechanical (Cooling) Utilization .......................................................14 MPUR: Mechanical vs Power Utilization Ratio ......................................... 14 MCUR: Mechanical Capacity Utilization Ratio ......................................... 14 10.1 Facilities: Physical Space .....................................................................................16 RUUR: Rack Unit Utilization Ratio ........................................................ 16 9. 10. 11. ICT: Data Storage .................................................................................................17 11.1 DSUR: Data Storage Utilization Ratio .................................................... 17 12. 12.1 ICT: Compute Power............................................................................................18 CPUR: Compute Power Utilization Ratio ............................................... 18 13. ICT: Memory ........................................................................................................19 13.1 MUR: Memory Utilization Ratio ............................................................. 19 14. ICT: Network ........................................................................................................20 14.1 NPUR: Network Port Usage Ratio .......................................................... 20 14.2 NBUR: Network Bandwidth Utilization Ratio ........................................ 20 15. CRUR: Calculation Models .................................................................................22 Weight factors .......................................................................................... 22 CRUR calculation models........................................................................ 22 CRURF calculation................................................................................... 23 CRURIT calculation.................................................................................. 23 CRURFIT calculation ................................................................................ 23 15.1 15.2 15.3 15.4 15.5 16. Monitoring & Reporting Tools ............................................................................24 17. Self-declaration, Audit and Certification.............................................................25 18. Appendix-A: Tips .................................................................................................26 18.1 CPU utilization......................................................................................... 26 18.2 Network device power consumption ....................................................... 26 18.3 UPS Distributed Redundant / Catcher system configuration ................... 27 19. Copyright ...............................................................................................................29 20. EPI: Company Profile...........................................................................................30 juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 5 of 31 Juan_Martinez juan.martinez1113@gmail.com 3. Table of Contents CRUR® - COMPUTER ROOM UTILIZATION RATIO 4. Scope and application of the CRUR metric The scope of the CRUR metric is the computer room and its direct supporting electrical (power) and mechanical (cooling) infrastructure as well as the ICT infrastructure. The CRUR metric can be applied to either the facilities only, ICT only or the combination. The notation of the CRUR will indicate what is under the scope; CRURF CRURIT CRURFIT : Computer Room Utilization Ratio for Facilities components only : Computer Room Utilization Ratio for IT components only : Computer Room Utilization Ratio for IT & Facilities components As mentioned, it is imperative for the CRUR metric to always include the scope notation to avoid incorrect and/or misleading information. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 6 of 31 Juan_Martinez juan.martinez1113@gmail.com The organization can choose either one, or all, of the metrics depending on the business requirements and scope of control. For example, an Enterprise Data Center might prefer to implement CRURFIT as it has full control over both the data center facilities as well as the ICT infrastructure whereas a commercial data center operator might prefer to only implement CRURF as they (might) have very little or no control over the ICT equipment deployed by their customers. Cloud providers may consider using the CRURFIT as they often have full control over both the facilities and ICT equipment. CRUR® - COMPUTER ROOM UTILIZATION RATIO 5. Normative References ASHRAE ANSI/TIA-942 BREEAM EN 50600 EU CoC ISA-71.04.2013 ISO 14001 ISO 22237 ISO 30134 ISO 50001 IPMVP LEED NABERS SS564 ITU-T L.1300 ITU-T L.1310 ETSI ES 203 136 V1.1.1 (2013-05) ETSI EN 303 470 V1.1.1 ETSI EN 303 215 V1.3.1 ETSI TS 105 1742 V1.3.1 American Society of Heating, Refrigerating and Air-Conditioning Engineers Telecommunication Infrastructure Standard for Data Centers Building Research Establishment Environmental Assessment Method Information technology - Data centre facilities and infrastructures European Union Code of Conduct Environmental Conditions for Process Measurement and Control Systems Environmental management systems Information technology - Data centre facilities and infrastructures Data Centers – Key Performance Indicators Energy Management System (EnMS) International Performance Measurement and Verification Protocol Leadership in Energy and Environmental Design National Australian Built Environment Rating System Sustainable data centers – Energy and environmental management systems Best practices for green data centres Energy efficiency metrics and measurement methods for telecommunication equipment Environmental Engineering (EE); Measurement methods for energy efficiency of router and switch equipment Energy Efficiency measurement methodology and metrics for servers Measurement methods and limits for power consumption in broadband telecommunication networks equipment Part 2: ICT Sites: Implementation of energy and lifecycle management practices juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 7 of 31 Juan_Martinez juan.martinez1113@gmail.com Use the following documents as additional reference sources for this Standard. Please note that all standards are ‘living’ documents. Care should be taken to always apply the ‘current’ version of the standard. CRUR® - COMPUTER ROOM UTILIZATION RATIO 6. Definitions of Terms, Acronyms, Abbreviations and Units of Measure Term Definition AC Benchmark Air Conditioner A standard or point of reference against which things may be compared. Building Management/Monitoring System Officially recognize (someone or something) as possessing certain qualifications or meeting certain standards. Cubic Feet per Minute / Cubic Meter per Hour Configuration Management System Computerized Maintenance Management System Carbon Usage Effectiveness Central Processing Unit Computer Room Air-conditioner To adhere to an act or its regulations To meet a requirement, standard, or procedure The room containing continuous operation for ICT equipment which processes and stores business processes and data Any individual or organization which uses the services provided by the organization The computer room plus all its supporting facilities Energy Efficiency Ratio of equipment Environmental Monitoring System Energy Management System End Of Life End Of Service A break or other defect in a system and/or equipment. A basic structure underlying a system and/or concept. Gigabyte Gigabits per second Heating, Ventilation and Air-conditioning High Voltage/High Tension Information and Communications Technology Key Performance Indicator Megabyte Megabits per second Massive Array of Idle Disks Maintenance Operation Procedure A contract that defines how various departments within the data center organization plan to deliver a service or set of services to support an SLA. The data center service provider. Power Distribution Unit Set of rules and guiding principles that shall be adhered to, while executing related process(es) and procedure(s). A set of specific steps that describe how an activity should be carried out, and by whom. A set of interrelated activities that delivers a desired outcome. Power Usage Effectiveness At least once a year or after every major change An area or room which requires additional security measures to ensure risk of disturbance to this room or area is minimized. For the definition of this standard, it includes (but is not limited to); BMS Certified CFM/CMH CMS CMMS CUE CPU CRAC Compliance Conformity Computer room Customer Data center EER EMS EnMS EOL EOS Fault Framework GB Gbps HVAC HV/HT ICT KPI MB Mbps MAID MOP Operational Level Agreement Organization (the) PDU Policies Procedure Process PUE Regular Restricted area juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 8 of 31 Juan_Martinez juan.martinez1113@gmail.com The following table describes some of the terminology used in this standard. CRUR® - COMPUTER ROOM UTILIZATION RATIO REF ROI Service Improvement Process (SIP) SHR SLA – Service Level Agreement Standard Supporting facilities Juan_Martinez juan.martinez1113@gmail.com TB UPS WAN WUE Transformer room/area, Generator room/area, UPS room, Computer room etc. Renewable Energy Factor Return on Investment A service expansion/improvement process that builds on the existing services and identifies opportunities to improve and/or add new services. Sensible Heat Ratio A contract between a service provider (either internal or external) and the end user that defines the level of service expected from the service provider. A document that provides requirements, specifications, guidelines or characteristics that can be used consistently to ensure that materials, products, processes and services are fit for their purpose. The technical and operational areas required to support the computer room environment such as UPS room, Generator area/room, Network Operations Center (NOC) etc. Terabyte Uninterruptable Power Supply Wide Area Network Water Usage Effectiveness juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 9 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 7. Introduction to CRUR The computer room is at the heart of every data center. The aim of any business is to ensure that the design is based on business requirements which includes today’s, as well as future requirements. This, and other factors, has led to many business and data center designers to “play safe” by doing a fair amount of oversizing with the “thinking” that it is better to have 100kW too much than 1kW too little resulting in a level of “waste”. Not only due to unused space, power etc. but also financial waste due to overspending which immediately negatively affects the ROI. This has also an impact on the environment as often underutilized data centers result in more in-efficient data center environments. Ideally, we could predict the future knowing exactly what we need to have 5, 10, 15 or even 20 years from now. However, this is simply impossible. The other issue is that in MTDC’s (Multi Tenanted Data Centers), the underutilization/ oversubscription on services could create issues for the data center operator as the underutilization/oversubscription of its tenants could have a (major) impact on the overall efficiency the MTDC can achieve. For example, if the data center operator assumed at least 80% load in the building which would then achieve the required PUE, the operator could possibly end up in great financial and legal trouble if the tenants do not create the loading factors for which they signed up. There is clearly a need to measure computer room utilization. However, the sad news is that many organizations don’t measure computer room waste as a way to optimize it and therefore not only create a better ROI but also to optimize it such that the data center has less impact on the environment. It is clear that a metric is required so that data center operators and owners gain insight in how much over dimensioning has taken place and how to make sure that rightsizing and optimization is achieved in the best possible way. To optimize the computer room, we need to review the design capacity and used capacity of the following components; Facilities components - Power capacity - Cooling capacity - Physical space juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 10 of 31 Juan_Martinez juan.martinez1113@gmail.com The ‘future requirements’ part is often at the heart of the problem when it comes the sizing the computer room as it is very difficult, if not impossible, to predict the future. This is not only difficult to predict due to unknown business climate and outlook of the future, but also due to technology changes that might happen including those unknown to anyone at this moment in time. One should not forget that data centres are often expected to have a lifespan of 10-20 years and more and with such a long lifespan it is virtually impossible to make any accurate long-term predictions. CRUR® - COMPUTER ROOM UTILIZATION RATIO ICT components - Data storage capacity - Compute power - Memory capacity - Network ports and bandwidth There are still several data centers who lack in the ability to centrally monitor resources usage such as detailed power consumption monitoring or CPU/memory utilization etc. Obviously, it is very important to install monitoring systems as CRUR analysis will be difficult, if not impossible, in a manual fashion. Section 16 of this document will discuss monitoring systems. Suffice to say that any usage and optimization should ensure that data integrity, availability and security should not be compromised to the point that Service Level Agreements might be at risk of being violated. Juan_Martinez juan.martinez1113@gmail.com The next few sections will provide an explanation on each of the metrics within the overall CRUR metric and how to calculate them. Section 15 will provide guidance on how to calculate CRURF , CRURIT and CRURFIT. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 11 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 8. Facilities: Power Capacity Power capacity utilization, which is directly related to the computer room, has a number of measure points. These include, but are not limited to; • UPS • PDU • Busbar • Rack • Rack density Whilst 100% utilization on all these points would be the theoretical desirable point, in practice capacity utilization is often capped at 80 or 90% depending on the load and diversity factors and the risk appetite of the organization. When understanding the power utilization, consideration should be given to the power infrastructure layout. For example, where the computer room is using an A & B power distribution configuration to allow for failover, we should consider that a 40/45% load condition on A and B feed could be considered the optimum utilization under normal operational mode. This to ensure that we can transfer the load from one feed to the other feed resulting in the surviving feed to then have 80/90% load. This will allow for a safe and uninterrupted operation of the computer room. The optimized load condition percentage on a single feed will be more difficult to establish in a power distribution where a catcher system principle has been deployed. A catcher system could be as simple as having an A and B feed with a single catcher system catering for either the A or B feed or it could be much more complex where multiple active distribution paths are supported by one or multiple catcher systems. When using a catcher system setup, it will be important to calculate the optimized power capacity ratio and apply that to benchmark the current ratio. 8.1 SPCR: Stranded Power Capacity Ratio Redundant power infrastructure often leads to stranded capacity. It is therefore important to factor in stranded capacity to drive for a more efficient data center setup. The SPCR metric can be calculated by the following formula; πππΆπ = π − πΆππππππ‘π¦ (∗) πππ‘ππ πππ πππππππ‘π¦ πππ π‘πππππ(∗∗) juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 12 of 31 Juan_Martinez juan.martinez1113@gmail.com Furthermore, measuring utilization at this level of granularity would introduce a higher level of complexity which defeats the purpose of the CRUR metric which is aimed at being easy to use and providing a high-level insight into the computer room utilization overall. However, if an organization wishes to go into greater level of detailed analysis of the design optimization, now or for future data centers, then this could of course be applied. CRUR® - COMPUTER ROOM UTILIZATION RATIO Note-*: N Capacity is the maximum base capacity designed for the ICT load in kW. Example-1 Example-2 N-Capacity = 100kW Total UPS Capacity = 200kW SPCR = 100/200 = 0.5 N-Capacity = 200kW Total UPS Capacity = 300kW SPCR = 200/300 = 0.66 8.2 PCUR: Power Capacity Utilization Ratio The power capacity installed vs. current power capacity (currently) in use indicates the effective sizing of electrical equipment power capacity vs. the actual current power load for the ICT equipment inside the computer room. Too much over capacity installed vs. ICT power load in use leads to in-efficiencies and consideration should be given to reduce the active (UPS) power capacity or to potentially consider rotating schedules etc. It should be noted that growth strategies play an important role for this metric. However, much can be done by having an appropriate capacity growth strategy i.e. full build out vs. a phased build out. The PCUR metric can be calculated by the following formula; ππΆππ = πππ‘ππ πΌπΆπ ππππ(∗) πππ‘ππ πππ πππππππ‘π¦ πππ π‘πππππ(∗∗) Note-*: Total ICT load is based on all current load in the computer room measured in kW. Where loads vary by more than 15% the average kW load over a 7 consecutive day period should be used. Note-**: Total UPS capacity installed is the total UPS capacity installed (in kW) minus redundancy in parallel or catcher designs. However, total UPS capacity does include all UPS capacity including rackmount UPS systems which might be installed in addition to the centralized UPS system(s) in a serial power train configuration. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 13 of 31 Juan_Martinez juan.martinez1113@gmail.com Note-**: Total UPS capacity installed is the total UPS capacity installed (in kW) including all UPS capacity including rackmount UPS systems which might be installed in addition to the centralized UPS system(s) in a serial power train configuration. CRUR® - COMPUTER ROOM UTILIZATION RATIO 9. Facilities: Mechanical (Cooling) Utilization Cooling capacity utilization has two important aspects; • Total mechanical (cooling/AC) capacity installed vs. total electrical power capacity installed • Total cooling mechanical (cooling/AC) capacity installed vs. current cooling capacity required 9.1 MPUR: Mechanical vs Power Utilization Ratio The cooling capacity installed vs. the power capacity installed will indicate how much oversizing has been done on the mechanical infrastructure compared to the maximum power, and therefore heat, dissipation in the room. This kind of oversizing is often done to cater for cooling in-efficiencies expected in the room as theoretically we only need 1kW of (sensible) cooling capacity for every 1kW of ICT heat load. Oversizing has an impact on both ROI as well as daily efficiency of the data center. ππππ = πππ‘ππ πππ πππππππ‘π¦ πππ π‘πππππ(∗) πππ‘ππ π΄πΆ πππππππ‘π¦ πππ π‘πππππ(∗∗) Note-*: Total UPS capacity installed is the total UPS capacity installed minus all redundancy. For example, if there are two feeds with each a 100kW UPS systems and they are designed to cater for CM – Concurrent Maintainability or FT – Fault Tolerance considerations, then only the single capacity must be used in this formula. In this example one would use 100kW instead of the total purchased 200kW. Note-**: Total capacity is determined by adding the total cooling capacity (in kW/SHR) of all installed mechanical equipment installed minus additional units for redundancy purpose if these units are not running. If additional units are running as part of the total cooling capacity pool, then these additional units need to be added to the total cooling capacity installed. 9.2 MCUR: Mechanical Capacity Utilization Ratio The cooling capacity installed vs. current cooling capacity (currently) required indicates the effective sizing of mechanical equipment vs. the actual current heat load. Too much over capacity in use vs. current heat load leads to in-efficiencies and consideration should be given to reduce the active cooling capacity by considering rotating schedules etc. It should be noted that growth strategies play an important role for this metric. However, much can be done by having an appropriate capacity growth strategy i.e. full build out vs. phased build out. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 14 of 31 Juan_Martinez juan.martinez1113@gmail.com The MPUR metric can be calculated by the following formula; CRUR® - COMPUTER ROOM UTILIZATION RATIO The MCUR metric can be calculated by the following formula; ππΆππ = πππ‘ππ πΌπΆπ ππππ(∗) πππ‘ππ π΄πΆ πππππππ‘π¦ πππ π‘πππππ(∗∗) Juan_Martinez juan.martinez1113@gmail.com Note-*: Total ICT load is based on all current load in the computer room measured in kW. Where loads vary by more than 15% the average kW load over a 7 consecutive day period should be used. Note-**: Total AC capacity installed is the total AC capacity installed (in kW/SHR) minus all redundancy unless the redundant units are actively running. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 15 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 10. Facilities: Physical Space The physical space for a computer room should be utilized effectively as real estate cost is often high. Assuming a rack-based setup, there could be a variety of factors influencing the number of racks which can be installed within a computer room. Such factors include; • The width and depth of the racks • Aisle width taking into consideration power density, work/maintenance space requirements, regulatory safety requirements • Cooling system setup e.g. perimeter/in-row etc. As the basis of usable space within the computer room the space within the rack itself is applied. This is expressed in RU – Rack Unit which is commonly abbreviated to U. One U is 1.75inch / 44.45mm. 10.1 RUUR: Rack Unit Utilization Ratio The rack unit utilization is calculated by using the following formula π πππ = π πππ ππππ‘ πΏππππ‘ππππ ππ ππ π πππ ππππ‘ πΏππππ‘ππππ πΌππ π‘πππππ(∗) Note*: - When having varying rack heights, multiply the number of each rack type with the available U for that rack type e.g. (16 Racks x 42U) + (18 Racks x 48U) = 1,536 - When deploying ToR type of network infrastructure, do count all U space in the rack including the U which is occupied by the network switch(es) juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 16 of 31 Juan_Martinez juan.martinez1113@gmail.com On the basis of keeping the metric simple, the following assumptions apply; • Computer rooms are most likely being populated with standard racks (width, depth and height might vary) • Computer rooms comply to local (fire) safety and safe workplace regulations • Computer rooms are most likely to be fully populated with racks CRUR® - COMPUTER ROOM UTILIZATION RATIO 11. ICT: Data Storage Data storage is a component where it is impossible to determine the design capacity for a computer room. Data storage capacity requirements are dynamic to the business (applications) and as such, capacity could easily, and most likely, change frequently over time. Technology advances in the future will also have a major impact on the total amount of data storage capacity available within a given space whether that is a computer room, a rack or even a server/storage unit. However, one objective is clear, it is important to right size it as best as possible at any given time to avoid underutilization and therefore over investment especially knowing that the price for storage often (dramatically) reduce e.g. the GB/USD of today is most likely much higher than that of the GB/USD one year later. It is for this reason that for this component of the computer room we only measure currently installed capacity vs currently used capacity. It is therefore required to create logical data storage pools. This will allow the organization to determine the overall utilization ration as well as “departmental/application specific” ratios. 11.1 DSUR: Data Storage Utilization Ratio The data storage utilization ratio is calculated by comparing the used vs. installed data storage for every logical storage pool and then average out. The formula for calculating each data storage pool is: π·πππππ1 = π·πππππ1 ππ ππ π·πππππ1 πΌππ π‘πππππ After calculating each individual Data Storage Pool utilization, we can calculate the total by using the formula below; π·πππ = π·πππππ1 + π·πππππ2 + π·πππππ πΈππΆ π X = Total number of data storage pools. Note; Pls ensure that one unit of measure is used e.g. MB, GB, TB for all Data Storage Pools. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 17 of 31 Juan_Martinez juan.martinez1113@gmail.com Consideration should be given to the fact that data storage cannot always be considered as one big data storage pool which dynamically can be shared amongst a variety of applications and business units within an organization e.g. the storage pool available for the human resource department is most likely (physically) separated from the online transaction processing application for various reasons such as performance, security etc. CRUR® - COMPUTER ROOM UTILIZATION RATIO 12. ICT: Compute Power As with data storage, compute power is also a component where it is impossible to predict the required design capacity for a computer room over its full lifespan. Many compute power benchmarks have been established over time. Some are based on total system performance such as FLOPS (Floating-Point Operations Per Second) or MIPS (Million Instructions Per Second) and other variations thereof whereby the applied choice and usage typically varies depending on what favours the architecture of the vendor. Other measure of compute power use metrics such as the clock-speed as a single parameter or in combination with bus-width etc. CRUR takes a very simple approach in that it does not try to establish a new or single benchmark number based on parameters as indicated above. CRUR takes the simple approach that a CPU runs at a maximum of 100% process load. Anything less than 100% theoretically means the CPU is not utilized to its full capacity. We also need to consider that CPUs typically run at varying load conditions at any given time and as such we should measure the CPU utilization over a period. Time recorded for determining the average load condition should not be less than 7 consecutive days. A final consideration is that not every server will run at its most optimum energy utilization rate when running under high load. As such one could consider setting the optimum threshold at a lower level where desired. 12.1 CPUR: Compute Power Utilization Ratio The compute power utilization ratio is calculated by calculating the average CPU load measured over a 7-day consecutive period. Where feasible, it would be desirable to consider CPU load per logic CPU pool allocated for a defined scope such as an application/application group or business group. The formula for calculating the CPU load pool is: πΆπππ ππππ1 = πΆππππππ1 πΏπππ % 100% After calculating each individual CPU Power Pool utilization, we can calculate the total by using the below formula; πΆπππ = πΆπππ ππππ1 + πΆπππ ππππ2 + πΆπππ ππππ πΈππΆ π X = Total number of compute power pools. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 18 of 31 Juan_Martinez juan.martinez1113@gmail.com However, we need to consider other factors such as that no data center wants to have all its CPUs running at 100% load all the time as then there would be no room for (temporary) increased workload requirements. CRUR® - COMPUTER ROOM UTILIZATION RATIO 13. ICT: Memory Memory utilization has similar properties as compute power. The memory pool is tied to the CPU pool meaning that memory has a direct relation to the CPU (or CPUs) driving them. As such, the memory pool definitions will be in a one-to-one relation to compute power pool definitions. Memory capacity will often change depending on application requirements by simply adding or removing memory modules. It is therefore that the memory utilization will also be measured as a relation between installed and used. We also need to consider that memory utilization typically varies at any given time and as such we should measure the memory utilization over a period. Time recorded for determining the average load condition should not be less than 7 consecutive days. 13.1 MUR: Memory Utilization Ratio πππππ1 = πππππ1 ππ ππ πππππ1 πΌππ π‘πππππ After calculating each individual memory pool, we can calculate the total by using the formula below; πππ = πππππ1 + πππππ2 + πππππ πΈππΆ π X = Total number of memory pools. Note; Pls ensure that one unit of measure is used e.g. MB, GB, TB for all Memory Pools. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 19 of 31 Juan_Martinez juan.martinez1113@gmail.com The memory utilization ratio is calculated by calculating the used vs. installed memory for every memory pool. The formula is: CRUR® - COMPUTER ROOM UTILIZATION RATIO 14. ICT: Network Network utilization has two primary components; • Number of ports installed vs. used • Bandwidth available vs. used The number of ports and their utilization is relatively simple to establish by counting the number of ports vs. active ports. Measuring bandwidth utilization is potentially more complicated as there can be various points to measure utilization such as network utilization within the computer room itself as well as connectivity of the total data center building to the outside world. For example, whilst data analytics applications might rely more on network bandwidth within the computer room itself, video streaming applications require appropriate bandwidth to the outside world. The network port used ratio is calculated by calculating the used vs. installed network ports within the computer room. The formula is: ππππ = πππ‘π€πππ ππππ‘π ππ ππ(∗) πππ‘π€πππ ππππ‘π πΌππ π‘πππππ Note(*); This term refers to the total number of ports on all the network switches (across all the line cards) that are active (with cables plugged in). The remaining ports on the switch should explicitly be disabled using the switch’s command line interface. Consideration should be given to setting limits to the line rate forwarding capacity of individual ports (typically, the capacity of a full-duplex 1 Gbps port can also be set to 10 Mbps and 100 Mbps) to limit the power usage of each port in relation to the capacity required. 14.2 NBUR: Network Bandwidth Utilization Ratio The network bandwidth utilization ratio, for the purpose of CRUR, is measured at the individual computer room level under the scope of the CRUR metric application. The capacity installed vs. used is related to the network going into the computer room under scope which is typically the capacity going into the MDA – Main Distribution Area. ππ΅ππ = ππ ππ π΅ππππ€πππ‘β (πΊπππ ) π΅ππππ€πππ‘β πΆππππππ‘π¦ (πΊπππ ) juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 20 of 31 Juan_Martinez juan.martinez1113@gmail.com 14.1 NPUR: Network Port Usage Ratio CRUR® - COMPUTER ROOM UTILIZATION RATIO Juan_Martinez juan.martinez1113@gmail.com We also need to consider that network bandwidth utilization typically varies at any given time and as such we should measure the bandwidth utilization over a period. Time recorded for determining the average load condition should not be less than 7 consecutive days. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 21 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 15. CRUR: Calculation Models Each of the ratios detailed in this document serve a certain purpose to assess the current utilization ratios in various components of a computer room and its direct supporting facilities infrastructure. It should be noted that certain utilization ratios achieved might be far from what could be considered to be the optimum. However, utilization ratios should take into account business factors such as the current state of the business and expected growth over a defined period of time. Hence, a low ratio does not necessarily indicate a poorly designed and/or used data center, it might have been over designed for the purpose of catering for future growth. The CRUR can then be used as a metric to keep track on planned growth and planned optimization targets for the years to come versus targets achieved. 15.1 Weight factors The CRUR metric does not have weight factors for the various sub-metrics such as CPUR, CMUR etc. First of all, weight factors are subjective. Secondly, they could vary from data center type and industry. Thirdly, weight factors could vary depending on the business objective. Therefore, having a fixed set of weight factors would not make sense. If we would allow the data center operator/owner to set their own weight factors based on business objectives, then the risk is that those weight factors might also be abused by a select few who want to boost the overall CRUR ratio for business/marketing purposes. This would not be in the best interest for the industry. Therefore, weight factors should not be applied as any improvement in any of the categories would automatically result in an overall improvement in the CRUR ration. Of course, as a data center operator owner, one can always select which of the sub-metric(s) is/are more important and deserves more attention for improvement programs. 15.2 CRUR calculation models As explained in the introduction, there are three types of CRUR being; CRURF CRURIT CRURFIT : Computer Room Utilization Ratio for Facilities components only : Computer Room Utilization Ratio for IT components only : Computer Room Utilization Ratio for IT & Facilities components It is compulsory for those using the CRUR metric to display which of the three is being used by having the subscript indicator. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 22 of 31 Juan_Martinez juan.martinez1113@gmail.com By keeping track of the CRUR, the organization can take lessons learned on how CRUR changes over time to understand whether a too aggressive oversizing model was used often leading to an impact to the overall ROI. CRUR® - COMPUTER ROOM UTILIZATION RATIO 15.3 CRURF calculation The CRURF is calculated by applying the following formula; πΆπ ππ πΉ = CRURF SPCR PCUR MPUR MCUR RUUR πππΆπ + ππΆππ + ππππ + ππΆππ + π πππ 5 : Computer Room Utilization Ratio - Facilities : Stranded Power Capacity Ratio : Power Capacity Utilization Ratio : Mechanical vs Power Utilization Ratio : Mechanical Capacity Utilization Ratio : Rack Unit Utilization Ratio 15.4 CRURIT calculation πΆπ ππ πΌπ = CRURIT DUSR CPUR MUR NPUR NBUR Juan_Martinez juan.martinez1113@gmail.com The CRURIT is calculated by applying the following formula; π·πππ + πΆπππ + πππ + ππππ + ππ΅ππ 5 : Computer Room Utilization Ratio - IT : Data Storage Utilization Ratio : Compute Power Utilization Ratio : Memory Utilization Ratio : Network Port Usage Ratio : Network Bandwidth Utilization Ratio 15.5 CRURFIT calculation The CRURFIT is calculated by applying the following formula; πΆπ ππ πΉπΌπ = CRURFIT CRURIT CRURF πΆπ ππ πΉ + πΆπ ππ πΌπ 2 : Computer Room Utilization Ratio – IT & Facilities : Computer Room Utilization Ratio - IT : Computer Room Utilization Ratio - Facilities juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 23 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 16. Monitoring & Reporting Tools It is highly recommended to automate the measurement and reporting on the CRUR metric and sub-metrics. There are a wide variety of data center monitoring tools which can support part or all the measurements and provide an automated view on the CRUR metric. Whilst such automation could be of great value, careful evaluation must take place that the measurement locations, calculation formulas as well as measurement evaluation intervals are correctly defined to provide a reliable metric reporting. Juan_Martinez juan.martinez1113@gmail.com An audit and certification of the data center for CRUR will include the evaluation of the implementation of the monitoring and reporting tools used by the data center to ensure an accurate and reliable reporting on CRUR. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 24 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 17. Self-declaration, Audit and Certification Organizations are free to use the CRUR metric and indicate the CRUR values as long as they clearly indicate the area under scope and that the value is based on self-declaration. EPI, and its authorized partners, can provide official audit and certification services for CRUR. This will provide the additional level of trust to the customers of the data center that the measurements and calculation have been audited by trained professionals. This 3rd party declaration will ensure that customers know that an independent and impartial review has taken place providing an accurate representation of the values of CRUR. This can of course also used by the data center operator/owner as values for which approvement plans can be established to optimize the data center. Juan_Martinez juan.martinez1113@gmail.com Please contact EPI at sales@epi-ap.com for further information. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 25 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 18. Appendix-A: Tips The aim of the CRUR metric is to have a relatively simple way to analyze how well the computer room is optimized from ‘installed’ vs. ‘used’ perspective. However, a data center operator/owner could further enhance the analysis at various levels depending on much time and resources are available. The data center operator/owner could/should also consider other factors which might impact the CRUR or optimization plans. The list below gives some guidance on some of the potential considerations. It is by no means an exhaustive list. EPI is very interested in realworld experiences in computer room optimization and tips so that over time the list below can be expanded for the benefit of all organizations who like to implement the CRUR metric. Theoretically one would consider running a CPU at 100% utilization is the optimum. However, running CPUs at that level of utilization has a number of potential downsides such as; a. Thermal throttling potentially causing the system to shut down b. Decreased lifespan of ICT hardware c. Increased demand for cooling which in turn increases the demand for power and therefore cooling capacity CPU manufacturers often indicate optimum performance levels balancing CPU performance vs. power capacity. 18.2 Network device power consumption There is a vast difference between various network devices and the power they consume. For example; With no port connectivity, the Netgear JGS524 consumes approximately 20 Watt whereas the Linksys LGS123 consumes only approximately 12 Watt despite both devices being a 24 ports network device. Please note that these numbers are not indicative of power consumption efficiency for the whole range of products for these two manufacturers so care should be taken on product-by-product basis. It is clear that when selecting network devices it is important to review the actual power consumption as the network device will be powered on 24x7 and, with many network devices installed in a computer room environment, the power consumption cost might far outweigh difference in purchase cost. Another consideration is to whether deploy unmanaged or managed network devices. In most organization, especially larger network devices, are fully managed which provides the juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 26 of 31 Juan_Martinez juan.martinez1113@gmail.com 18.1 CPU utilization CRUR® - COMPUTER ROOM UTILIZATION RATIO organization with the ability to setup the device based on business requirements. This includes the activation/de-activation of network ports which are not being used. Unfortunately, in many computer room environments this feature is not being used for the reason that its just easier to leave them all on so one can plug in a cable in any port knowing that it is active. However, consider that a typical active network port with no traffic still consumes approx. 20-25 Watt vs. a port running at approx. 50% bandwidth to consume typically 25-40 Watt. With potentially many ports unused, it is clear that good savings can be made considering that every 1kW of power in the computer room also requires energy to cool the heat generated. Besides that, unused ports which are being blocked will also not being ‘abused’ by plugging in a network cable without proper change management consideration leading to undocumented network changes to which so many organization are prone. Most data centers which are designed to Rated-3 or Rated-4 or similar standards or guidelines, have traditionally been designed around the A and B power feed configuration providing additional power resiliency. However, that requires that the UPS configuration for the computer room which has an A and B feed, should not cross more than 45% on each feed so that during a failover scenario the surviving UPS (cluster) does not carry more than 90% load ensuring the ICT systems can continue to operate without any interruption. Although UPS technology advancement has resulted in better efficiency even under lower load conditions, using the traditional A & B setup will also require a large financial commitment as basically two power infrastructures are being build. An alternative option is to configure the UPS systems in a distributed redundant or catcher system configuration. The illustrations below shows the advantage from a sizing point of view. Provides only A / B failover. During maintenance/fault only one feed will be active. Provides only A / B failover. During maintenance/fault of one UPS in a cluster the other UPS will still be active. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 27 of 31 Juan_Martinez juan.martinez1113@gmail.com 18.3 UPS Distributed Redundant / Catcher system configuration Provides only A / B failover. During maintenance/fault the catcher takes over the full load. Additional ATS/STS systems required Provides only A / B failover. During maintenance/fault catcher takes over the full load. Catcher is fed through maintenance bypass circuit of UPS or via wrap-around There are several other factors to consider when deciding on traditional A&B, distributed redundant or catcher systems, please consult your electrical consultant. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 28 of 31 Juan_Martinez juan.martinez1113@gmail.com CRUR® - COMPUTER ROOM UTILIZATION RATIO CRUR® - COMPUTER ROOM UTILIZATION RATIO 19. Copyright EPI® (Enterprise Products Integration Pte Ltd) is the Intellectual Property and Trademark owner of the CRUR metric. Organizations can use the CRUR metric to analyze and optimize their computer room. Where the organization declares CRUR value(s) it shall; - Indicate the appropriate notation to declare what is under the scope of the indicated CRUR (i.e. CRURF, CRURIT, or CRURFIT) - Indicate that the CRUR is based on self-declaration Juan_Martinez juan.martinez1113@gmail.com Only EPI, and its authorized organizations, can audit and certify CRUR. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 29 of 31 CRUR® - COMPUTER ROOM UTILIZATION RATIO 20. EPI: Company Profile EPI® (Enterprise Products Integration Pte. Ltd.) offers an extensive range of data center services includes services such as, data center consulting, design evaluation, design validation, surveys, assessments, audits and certification based on standards such as ISO, ANSI/TIA-942, DCOS© and other relevant international data center standards. Juan_Martinez juan.martinez1113@gmail.com EPI® also offers a wide range of globally accredited certified data center training programs such as, CDCP®, CDCS®, CDCE®, CTDC®, CTIA®, CTEA®, CTLA® CDMS®, CDFOM®, CDFOS® CITO®, CITM®, CITD®, CDESS®, CDRP® and CNCDP® which enable organizations to educate and align their staff on data center design principles, data center efficiency, green data center principles, data center management, data center operations management, security management, disaster recovery management. All courses are accredited by EXIN®. juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 30 of 31 Juan_Martinez juan.martinez1113@gmail.com CRUR® - COMPUTER ROOM UTILIZATION RATIO juan.martinez1113@gmail.com COPYRIGHT EPI ® NO DUPLICATION IN PART OR WHOLE WITHOUT PERMISSION OF EPI®. Page 31 of 31