Strategic Data Center Procurement Kirk A. Killian Corporate America Needs New Data Centers 88% of Mid-market Companies (< $500 million revenues) plan to acquire/build or expand data centers in the next 4 years. 91% of Mid-market Companies expect equal or greater annual expenditures next year than last year for data center facilities. • Most corporate IT departments will procure (build or buy/lease) additional data center capacity in the next few years. Strategic Data Center Procurement 2 Demand Drivers for Additional Data Center Capacity 46% - Consolidate infrastructure (shrink # of facilities) 41% - Replace aging facilities 38% - Support organic growth 31% - Expand geographic coverage 29% - Improve business continuity 25% - Improve energy efficiency 21% - Change sourcing models (colocation, cloud, etc.) 16% - Reduce operating costs • Needs vary by organization; many companies have multiple reasons for needing new data centers. • Cost reduction is always important, but is not usually the most significant factor. Strategic Data Center Procurement 3 Importance to IT Procurement Professionals Cross-functional team (procurement, IT, facilities, legal, CFO) • Cross-Organizational Visibility Expensive projects (often $20 - $250 million) • Upper Management Scrutiny Infrequent purchases (once every few years vs. many purchases every year for hardware & software) • Process Unfamiliarity This session will inform the IT Procurement Pro about this niche, and help you strategically provide value to your organization, avoiding expensive, high-visibility missteps. Strategic Data Center Procurement 4 Overview – Data Center Delivery Models Design, Build & Self-Operate (traditional corporate model) • Company builds its own non-shared data center and operates it for many years. All hardware and software is owned. Colocation Suite • Company leases a non-shared private suite within a large colo data center. All hardware and software is owned. Cloud • Company gets use of specific hardware/software owned by the provider. Location and operations practices determined by provider. Hardware (maybe software) owned by cloud provider. Strategic Data Center Procurement 5 Brief Discussion On Cloud Cloud delivery model not discussed in depth in this session • Rapidly growing sector – many companies are moving part of • • • • • their computing to the cloud, especially test/dev, storage, backup, and special projects with extreme scaling or short durations Can be economical, flexible, rapidly deployed Control measures less defined than in corporate data centers Can present control, reliability, audit and security issues (reliability varies widely by provider, location and service level) Embraced warmly by internet/emerging industries; cautiously by heavily regulated industries (finance, healthcare, insurance) Incorporating cloud into the overall mix is a powerful tool for adding flexibility and controlling costs Strategic Data Center Procurement 6 Overview – Traditional Delivery Model: Design, Build & Self-Operate Company designs a data center facility, builds it, and selfoperates it Design can start with raw land or “shell building” suitable for data center operations Design Decisions: Tier Rating, Electrical Density, Raised Flooring, Modular Tier Rating (reliability) and Electrical Density are biggest variables affecting construction cost • Owner/operator company maintains maximum control over its investment level, operating costs, and facility management Strategic Data Center Procurement 7 Construction Costs Data center facilities are very expensive Large companies commonly spend $10 million on land and >=$100 million on buildings and critical systems (plus hardware/software) with future phases costing even more • $5-30 million is more common for mid-sized Fortune 1000 Data center developers can construct “turnkey” build-to-suit data centers pursuant to a long-term lease, converting a CapEx spend to a long-term lease of building • To maximize negotiations leverage, keep multiple locations, developers and delivery paths open for consideration Strategic Data Center Procurement 8 Example Budget: Building a Ground-Up Data Center Example: (1) 10,000 SF data hall w/ 1,200 kW, expandable into (2) 10,000 SF halls with 1,200 – 2,400 kW each Land Building/Site Work Critical Systems Design/Fees Contingencies (~10%) Total (approx.) $ 2,100,000 (8 acres at ~ $6.00/SF) $ 4,500,000 (60,000 SF at $75/SF) $10,800,000 (1,200 kW @ $9,000/kW) $ 800,000 $ 1,800,000 $20,000,000 = $16,667/kW (first 1.2 mW) • Above example design allows physical expansion of raised floor area within building, and separately, increase in power/cooling density within both data center phases = Future Flexibility Strategic Data Center Procurement 9 Design/Build Timeline – New Construction Takes a Long Time! Example schedule below doesn’t include internal scoping & approvals (inherently political), which can add months/years! 6 months 3 Months 6 months 9 Months 3 Months 27 Months Site Selection/Approvals Site/Building Acquisition Design & Permitting Construction Commissioning & Move-In Total – Start to Finish • Converting an existing building to data center use can usually save 3-9 months • Really agile companies can compress and overlap tasks to complete new project in 18 months, but that is rare • Most common source of delays is getting internal approvals! Strategic Data Center Procurement 10 Facility Reliability & Tier Ratings The Uptime Institute has established the most widely used data center reliability rating system, with (4) tier levels: • • • • • Most corporate and colo data centers built since 2005 are Tier 3 Most 10-20 year old corporate data centers are Tier 2 Most small data centers within office buildings are Tier 1 Higher Tier levels cost more to construct and operate Few data centers get formal TUI certification, but are instead designed in “substantial compliance” w/ TUI guidelines Strategic Data Center Procurement 11 Electrical Density/ Physical Expandability Most corporate IT departments are deploying computing equipment at 100 – 150 watts/SF (some much higher) Most 1990’s era data centers provide +/- 75 watts/SF Most new data centers designed to deliver 150 – 300 watts/SF, including colocation facilities • “Value engineer” new design/build to initially deliver 150-175 watts/SF (or current needs + 20%), then add more power later • Design new data center to allow physical addition of future additional data halls within the original building structure Strategic Data Center Procurement 12 New Trend – Modular Data Center Construction Data Center “Modular Kits” have factory-built major components, shipped to site and inter-connected • • • • Less customization (standard sizes and configurations) 10-15% lower upfront costs Higher initial reliability (in theory: factory built vs. site assembled) Faster timeline – can reduce schedule by 6-12 months if no customization desired • Long-term reliability and expandability TBD • Vendor-specific reliance risks TBD • Modular components can be a great solution to construct a new data center quickly within a building you already own! Strategic Data Center Procurement 13 Geography in New Data Center Site Selection Preference by many in IT to locations near existing data centers or operations staff (“server huggers”) For “active/active” data center pairs, proximity to existing data centers important for telecom latency (<= 40 miles apart) Lower construction costs in rural and southern/midwestern metros; research resource is Means Construction Cost Index Cold, dry climates maximize potential for “free cooling”, reducing electricity load by 10-20%. Lower electricity costs in south/midwest, higher in northeast and California; “hydro” utilities are lowest Green “fuel mix” important to some companies Strategic Data Center Procurement 14 Avoidance of Hazards/Risks Avoid sites that are: • <= 12 miles to paired “active/active” data center • <= 100’ to nearby roads • In seismically active areas (UBC zones 3 & 4) • Within 100-year and 500-year flood zones • < .5 mile to freight railroad ROW or HazMat or explosive storage/production facilities • < .25 mile to major interstate highway • <= 2 miles to commercial airport & flight paths Note: These are guidelines only; many data centers are near airports, railroads and highways, and in seismic zone 4 (California) Strategic Data Center Procurement 15 Economic Incentives Available in many forms for new data center projects: • Property Tax Abatements (real and personal property taxes) • Sales Tax Waivers (for building systems and computers) • Utility Enhancements (e.g. redundant feeders) • Utility Tariff Subsidies (typically 1-5 year rate discounts) • Cash Grants (difficult to get) • Job Grants & Training Programs (small, but easy to get) • Expedited Permitting (accelerates construction) • Free/Discounted Land Sites Strategically negotiate incentives; amounts can be millions! Keep multiple locations in consideration to maximize value Strategic Data Center Procurement 16 Site Infrastructure Issues Desirable site attributes include: • Electricity from redundant sources (2 feeders, preferably from 2 • • • • • substations) Power availability at medium voltage (12,470v – 50,000v), which is better than 480v Multiple redundant fiber optic networks nearby Dark fiber optic network nearby Light Industrial zoning to allow diesel backup generators >= .25 mile to residential neighborhoods, where homeowners can protest and delay data center development Strategic Data Center Procurement 17 Data Center Build-Out in Existing Buildings Seek buildings with: • >= 16’ clear height • >= 100 pounds/SF floor loading capacity • Concrete panel or concrete block exterior construction • Single-story building preferable • No hazardous co-tenants (single-tenant buildings preferable) • No co-tenants above data center (water penetration risk) Suggest phased build-out (multiple “data halls” within building) to conserve capital and reduce operating costs Strategic Data Center Procurement 18 Summary: Design, Build & Self-Operated Data Centers Advantages • Lower operating costs if long-term needs predictable & stable • Maximum control over operations, audits & reporting (“It’s Yours – Operate It How You Want”) • Incentives offset some upfront costs Disadvantages • Slow to construct & deploy • Expensive to build (large CapEx investment) • Inflexible specifications/quantities (“You Built It – Live With It”) Strategic Data Center Procurement 19 “Top 10 List”: Procuring SelfOperated Data Centers Adding Value as an IT Procurement Pro 1. Allow ample project time; start planning 3-5 years before new data center must be operational. If less time is available, you may need to construct in an existing building or procure colo. Carefully define facility needs: space, power, reliability, and design preferences. Force users to forecast growth to avoid overbuilding or under-expandability. Resist “Taj Mahal” criteria. Create an effective multi-disciplined project team (IT, engineering, real estate, legal). If needed to supplement in-house knowledge, hire expert consultants (architects, engineers, site selectors, incentives advisors, legal counsel). Select location wisely to reduce utility costs, improve telecom connectivity and reduce business continuity risks. 2. 3. 4. Strategic Data Center Procurement 20 “Top 10 List”: Procuring SelfOperated Data Centers 5. 6. 7. 8. 9. 10. Challenge “server hugger” mentality that dictates new data center location must be near existing facilities. Suggest phased build-out allowing future expansion phases as space needs dictate. Suggest phased power increases; similar to space phasing methodology in #6 above. Investigate and procure economic incentives, often overlooked by IT departments - can be worth millions! Evaluate modular “plug-in” data center systems components, and suggest multi-vendor component consideration. Explore developer sale/leasebacks to shift CapEx expenditures to OpEx rent if desired by corporate finance. Strategic Data Center Procurement 21 2nd Delivery Model: Colocation In data center Colocation, a third party provider constructs and operates a data center (similar in design to corporate data centers), leasing private suites to corporate IT departments similar to an office building with different tenants • In the colo model, the corporate IT department owns its hardware and software, but places it within a private suite or cage, which is inside the larger colo data center building Strategic Data Center Procurement 22 Modern Colo: Dedicated Infrastructure Trend: “Private Suite” colocation = less infrastructure sharing • Key feature for corporate customers - suites unlikely to share PDUs, CRACs, UPS (“data center within a data center building”) Example: Colo suite provides 1,200 kW to (1) customer • PDUs and CRACs non-shared within suite • N+1 UPS modules (3 x 750 kVa) non-shared serving this suite • Primary switchgear, generators, chillers not shared Strategic Data Center Procurement 23 Key Colo Terminology Provider – the company operating the colocation facility Suite – the premises occupied solely by one customer, often with hard walls and non-shared infrastructure components Cage – same space concept as “Suite”, but smaller and less likely to have non-shared CRACs and UPS modules Critical Load – the IT equipment load, typically backed up by UPS and generators Carrier Neutral – customer selects whichever 3rd party telecom carriers it chooses to serve its suite; offers cost advantage. Managed Services – available from colo provider in addition to the basics of space, power and telecom circuits; may include server re-boots, rack and stack, server builds, tape changes, tape shipping, SAN/NAS usage, firewall, load balancing, intrusion detection, and “cloud”. Managed services leverages the colo provider’s staff for users too small for local FTEs. Strategic Data Center Procurement 24 Telecom Options in Colo Facilities Key competitive advantage of many colocation facilities is carrier selection and carrier neutrality • Colo may provide access to 6-30 telecom carriers’ networks, offering wide choice of carriers, services and pricing • Circuit installations are relatively fast (+/- 60 days) and cheap – they’re already in the colo’s “meet-me” room! • Most colos (except those operated by telecom companies) are “carrier neutral”; each customer can choose its carriers • Select a colocation facility with at least one dark fiber option Carrier variety creates negotiating leverage; savings and contract flexibility can be huge for IT Procurement Pros! Strategic Data Center Procurement 25 Colo Industry Development: 1998 to Present Industry founded in 1998; early pioneers were startups, and many crashed with “dot-com” bust. Huge growth since 2005. Newer facilities have high reliability - mostly TUI Tier 3 High electrical densities, mostly 125 – 250 watts/SF Experienced, sophisticated providers, SSAE16 audits, PCI compliance, and professional operations/reporting Financially stable providers, many now publicly-traded Occupancy costs have decreased steadily since 2007 More dedicated (non-shared) infrastructure now • Greater acceptance by Fortune 1000 IT departments! Strategic Data Center Procurement 26 Colocation Geographic Availability While colocation is available in all major cities, the largest clusters are in (10) U.S. metro areas: • San Jose, Washington, NY/NJ, Dallas, Chicago (the “Big 5”), plus Los Angeles, Boston, Atlanta, Phoenix, and Seattle • Emerging market leaders are Austin, Las Vegas, Houston, Portland, Minneapolis, Denver and Miami • Providers include telecom companies, colo specialists, and publicly-traded Real Estate Investment Trusts (REITs) Colo provider variety creates negotiating leverage; savings and contract flexibility in most competitive markets is significant! Keep multiple markets under consideration, and include at least one “discount” provider in selection process Strategic Data Center Procurement 27 International Colocation Procurement Many US companies procure colo space internationally to support local or regional IT initiatives • The largest international “money center” cities offer a variety of • • • • modern colocation data centers Telecom latency and taxation issues can significantly affect the location decision (e.g. Dublin popular for tax reasons) International data ownership and data privacy laws (e.g. China) also affect the data center location decision. Data center space/power tends to be 10-40% more expensive overseas than in the US; London is particularly pricey A “last resort” colo option in most countries is the local phone company; usually very expensive and inflexible Strategic Data Center Procurement 28 Colocation Pricing Models The most common colo pricing models are: • Wholesale: Rent in $/kW of Capacity + Metered Power; typically for critical loads >= 150 kW. Metered power includes “uplift” of ~ 60% for cooling loads. • Retail: Cost/SF + Electrical Circuits; typically for critical loads below 150 kW. Retail colo placements can be effective for small corporate deployments (2-10 cabinets) or those in remote locations Wholesale colo pricing models tend to be ~ 30% cheaper than Retail, and customized for corporate IT customers Seek “Wholesale” pricing even if quantity is “Retail” Strategic Data Center Procurement 29 Example Colocation Pricing Example: 4,000 SF data center suite with 500 kW of critical power (125 watts/SF) for (125) 42u IT equipment cabinets • Typical wholesale pricing is $150/kW/month for the installed critical power capacity (including raised floor space), plus a separate charge for the electricity consumed (kW hours x $.06/kWh), including a 60% cooling uplift factor. $75,000 $34,560 $109,560 Rent/month (500 kW x $150/kW) Metered Electricity (500 kW x 720 hours/month x $.06/kWh x 1.60 “cooling uplift factor”) Occupancy costs (monthly for 5 years) • In a “lease vs. buy” analysis, the $900,000 example annual rent ($75,000/month x 5 years) compares to a ~ $12 million construction cost to build a data center of similar capacity Strategic Data Center Procurement 30 Service Level Agreements (SLA) Colo operators provide an SLA stipulating downtime allowed before financial compensation is provided to the customer Colo SLAs are skimpy; one day’s fees credit for each downtime event. No customer wants downtime triggering SLA credits (downtime risks/costs exceed the compensation) One of the most powerful incentives toward proper facility maintenance is the provider’s reputation, and uptime performance is generally excellent at new colo facilities • Negotiate early termination options (plus financial credits) for repeated/extended events of downtime • Negotiate that any downtime event of any duration triggers a credit, not only events that exceed a minimum duration Strategic Data Center Procurement 31 “Chemistry” With Colo Provider Is Important Procuring a colo data center involves a continuous cycle of requests from both parties; getting along well with the provider is key to a successful long-term relationship Invest time during the selection process to insure that your IT team communicates well with the vendor’s team, in addition to verifying competency from the vendor’s staff Some companies want a formal, all business approach; others appreciate a looser “silicon valley” vendor style It’s a painfully long contract term if either party doesn’t like doing business with the other side • Check customer references from finalist providers to learn contract terms existing customers wish they had obtained Strategic Data Center Procurement 32 Colocation Compared To Self-Operated Fast Deployment. Users can build-out and occupy colo suites/cages in months instead of years. Low CapEx Costs. Colo users only pay upfront for physical construction of the suite/cage plus electrical and network distribution. Higher Occupancy Costs. Compared to owned data centers, colo occupancy cost (rent) tends to be a little higher. Flexible Contracts. Colo contracts offer expansion options for “Pay As You Grow”, and short lease terms. Managed Services Options. At most colos, managed services are optional and ala carte, and less expensive than hiring FTEs. Less Operational Control. Colo customers have less control over critical systems operations, audits, security, and reporting. Lower Staffing Requirement. The colo provides all facility and critical systems maintenance. Strategic Data Center Procurement 33 Colo Procurement Process/RFPs Colo procurement is similar to other IT procurement: Project Scoping, Vendor Candidate Selection, RFPs, Facility Tours, Proposal Analysis, Negotiations, and Contract Review. RFPs should be sufficiently detailed to state the need, solution criteria, contract term, and preferred pricing models. Use a “short form” RFP (6-8 pages) for small cages and a standard RFP (20-25 pages) for larger colo suites. Most IT Procurement Pros distribute 4-8 colo RFPs. • RFPs should request flexibility in contract term, early termination, expansion/contraction of scope, and ala carte managed services. • To create negotiating leverage, send an RFP to at least one vendor in the most competitive geographic market under consideration, and at least one “competitively priced” vendor. Strategic Data Center Procurement 34 Baker’s Dozen: (13) Recommendations for Procuring Colo Adding Value as an IT Procurement Pro 1. Allow ample project time; start planning 1-2 years before new data center must be operational, even though colo can be procured in as little as 60 days. Carefully define facility needs: space, power density, reliability, systemic redundancies & design preferences). Force IT users to forecast growth to avoid over- or under-provisioning. Create an effective multi-disciplined project team (IT, engineering, real estate, legal). If needed to supplement in-house knowledge, engage expert consultants (architects, engineers, site selectors, incentives advisors, legal counsel). Select locations strategically to meet project objectives, reduce occupancy/utility costs, improve telecom connectivity, and include colo providers best suited to meet the company’s long-term needs. 2. 3. 4. Strategic Data Center Procurement 35 Baker’s Dozen: (13) Recommendations for Procuring Colo 5. 6. 7. 8. 9. Challenge colo vendors (and your IT team) to creatively devise contract flexibility – future changes in suite and power size, multiple locations within their portfolio, and switching services across their organization; e.g. substitute cloud spend for space/power spend. Incorporate a combination of “must take” space/power increases with optional increases to best address expansion flexibility. In addition to negotiating aggressively on initial pricing, get expansion provisions with multiple pricing options, such as current rate, ROFR on adjacent space, and “most favored nation”. Get fixed “rate card” pricing for optional managed services, and early termination options on all managed services and cloud services. Obtain sublease and contract assignment rights. Strategic Data Center Procurement 36 Baker’s Dozen: (13) Recommendations for Procuring Colo 10. 11. 12. 13. Get the right to bring new telecom carriers into the colo’s “meet-me” room, especially if you already have carrier discounts. Pick a colo with dark fiber, so you can bring in new cheaper carriers easily. At the closing table, when you’re about ready to sign the new colo contract, don’t be bashful in asking the colo provider to cover all or part of your installation costs. (Many providers are willing to make one-time cost concessions to win contracts.) Similar to other IT vendor relationships, end-of-quarter and end-ofyear can be great times to sign new colo contracts, especially with publicly-traded providers eager to “make their numbers”. Carefully scrutinize any under-construction facility not yet fully operational – demand detailed construction and commissioning schedules. Some colo providers “over promise” delivery dates. Strategic Data Center Procurement 37 Follow-Up Questions? Strategic Data Center Procurement 38