Data Center Infrastructure Management
(DCIM)
(CFRT Peer Review Workshop)
11/20/08
BROADCOM PROPRIETARY & CONFIDENTIAL
1
Schedule
– 9:30am to 10:00am
– 10:00am to 10:30am
– 10:30am to 11:00am
– 11:00am to 11:30am
– 11:30am to 12:15pm
– 12:15pm to 1:30pm
– 1:35pm to 2:10pm
– 2:15pm to 2:45pm
– 2:50 pm to 3:15pm
Registration and CFRT Networking
CFRT Membership Business - Membership Status and
Planning
Broadcom’s Data Center Journey
(speaker: Dave Dalesandro)
Broadcom’s Review of Data Center Management Software
(speaker: Dave Holub)
Interactive Demo - Broadcom’s DCIM application
(presenter: Dave Eastman)
Lunch and Round Table Interaction Working
Lunch (served in meeting room)
Panel Q&A (Sorena, Holub, Eastman, Dalesandro, Vippin)
CFRT Wrap-up Networking and Informal Q&A
Broadcom Data Center tour (for those interested)
BROADCOM PROPRIETARY & CONFIDENTIAL
2
Presenters
• Dave Dalesandro
– Director, IT X-Functional Architecture
– Data Center Design and Management
– (ddales@broadcom.com)
• Dave Holub
– Principal Project Manager, - IT
– Data Center infrastructure design architect
– (holub@broadcom.com)
• Dave Eastman
– Sr. Principal Project Manager, - IT
– DCIM Prototype Developer
– (deastman@broadcom.com)
BROADCOM PROPRIETARY & CONFIDENTIAL
3
Goal
Provide data and build processes to
operationalize the management of our data
centers
BROADCOM PROPRIETARY & CONFIDENTIAL
4
Why?
Data Center Infrastructure Optimization
• Broadcom data centers have been self service from inception
– This only works ‘if’ you have excess capacity
– With excess capacity ‘Optimization’ is not as critical of an issue
• Data centers cannot be self service when you near capacity
– Finding available infrastructure is ‘not’ a trivial task
• We need to get the most out of our “large $” investments in data
centers
• We need to protect the data center from overload or unbalanced
situations
BROADCOM PROPRIETARY & CONFIDENTIAL
5
Compute Growth Over Time
compute power
300000
250000
200000
150000
compute power
100000
50000
0
2000
2001
2002
2003
2004
2005
2006
BROADCOM PROPRIETARY & CONFIDENTIAL
2007
2008
6
Compute Power
• Processor Per-core
• Xeon 2.8GHz 32-bit
• Xeon 3.0GHz 32-bit
• Opteron 2.2GHz
• Opteron 2.4GHz
• Opteron 2.6GHz
• Opteron 2.8GHz
• Intel 2.66GHz
• Intel 3.16GHz
EDA Points
6
6.4
11
12
13
13.9
17.4
19.1
BROADCOM PROPRIETARY & CONFIDENTIAL
7
Global Disk Trends
Site (All)
Doubling storage capacity every 2 years!!!
StorageType (All)
2500000
2000000
1500000
1000000
500000
20080723
20080625
20080326
20071226
20070926
20070627
20070328
20061227
20060927
20060628
0
20060329
GB
Data
Date
BROADCOM PROPRIETARY & CONFIDENTIAL
8
Data Center Infrastructure
• Data Centers are valuable resources as they get close to capacity
those resources must be carefully managed
• Infrastructure includes
– Racks
– Switches and switch ports
– VLANs
– Patch panels and cables (of all types)
– Power utilization and monitoring
– Generators
– High voltage power components
– HVAC components
BMS
• By accurately tracking the usage of systems and their placement
in the data center we can ensure that overload conditions do not
occur
BROADCOM PROPRIETARY & CONFIDENTIAL
9
How did we get here?
• Company growth and technology changes means growth in
•
•
•
•
•
•
•
storage and compute
Evaluated several commercial applications
We did not know what we needed until we took the journey
We had rules (3840W for example) but did not know the cascading
impact
We were always doing fire drills to meet the demands of the
business
Lead times for data center expansion are long
Our largest data centers were designed 10+ years ago
Finding all the components to install just a single system in the
data center was becoming increasingly time consuming
BROADCOM PROPRIETARY & CONFIDENTIAL
10
BROADCOM PROPRIETARY & CONFIDENTIAL
11
Data Center Management
A Software Schlep
David Holub
Principal Project
Manager
IS Infrastructure
11/20/08
BROADCOM PROPRIETARY & CONFIDENTIAL
12
Software Smorgasbord
Various Buckets - Where have we looked, What have we Implemented?
Building Management System
Controls Integration – Outside the Scope of this Meeting/Discussion
Mechanical Electrical Plant Monitoring and Alerts
Propriety – Not Implemented Everywhere / Consistently Across Facilities
Multi-Protocol – Modbus, SNMP
Costly Integration and Customization
Various Scales – UPS’s from 50-500kVA modules
Critical Information for Management – Must Have Data
Commercial Application Vendors
Large – Expensive/Comprehensive Packages
“I Need A Bus Load of Consultants to get by”
Other Extreme - Open Source – Best at Monitoring & Alerts
OS Airlines.. Unix Airline - Each passenger brings a piece of the airplane and a box of tools to
the airport. They gather on the tarmac…
Heat and Power modeling packages
Instrumentation – Tie dye
CFD Analysis
Visualization Packages
Drawing Pictures
Flat Databases – hard to populate and keep current..
Cable Plant Management
Telecom-Centric
Proprietary Patch – Panel Systems
Designers
BROADCOM PROPRIETARY & CONFIDENTIAL
13
Customized Views
• Data Center Floor – Not in all Places
BROADCOM PROPRIETARY & CONFIDENTIAL
14
PDU Panel Example
• BCM
– Yes/No
Probably not
in all places
BROADCOM PROPRIETARY & CONFIDENTIAL
15
Rack Power Over Time
• 5 Months .5 Amps Variation @ 110 Volts (Lost in the Sauce)
BROADCOM PROPRIETARY & CONFIDENTIAL
16
BRCM over Time
• 5 Months - 1.8 Amps @ 110 Volts (Wrong Granularity…)
BROADCOM PROPRIETARY & CONFIDENTIAL
17
Larger Electrical Scale
• Static Transfer Switch
BROADCOM PROPRIETARY & CONFIDENTIAL
18
6 Months in the life of a ASTS
• Power Growth – Refresh Cycles – Great Data
BROADCOM PROPRIETARY & CONFIDENTIAL
19
Open Source Monitoring & Alerts
• Nice Graphs – Alerts, Thresholds, Notification lots of work…
BROADCOM PROPRIETARY & CONFIDENTIAL
20
Poor Man’s CFD
Instrument the Data Center
Understand Change
w/o Blanking
w/Blanking
BROADCOM PROPRIETARY & CONFIDENTIAL
21
Uber Data Center Application
• Expensive in $ and Resources
– To Buy
– To Implement
• Simple Tasks Become:
• Cumbersome
• Require Great Expertise
• Canonical Issues
– Training
– Support
– Waiting for Godot
BROADCOM PROPRIETARY & CONFIDENTIAL
22
Cable Management Icky Problems…
• Who Cares?
• Some People Do..
BROADCOM PROPRIETARY & CONFIDENTIAL
23
Cable Management Applications
• Highly Generalized
– Not Data Center Specific
– Carriers
– Contractors
– Vendors Specific
BROADCOM PROPRIETARY & CONFIDENTIAL
24
Visualization Packages
BROADCOM PROPRIETARY & CONFIDENTIAL
25
Theoretical Models
BROADCOM PROPRIETARY & CONFIDENTIAL
26
Cool But I Need…
• Need to Roll Up the Data - Pull Together Various Pieces on a Dashboard
– Live Within Your Constraints
• Thresholds
• Deployment Rules
• Remediation – “Houston We Have a Problem” -Find All These
– Planning
• Build Consensus – Team Understanding of Data Center Ecology
• Capacity - Relationship to Redundancy and Resiliency
– Prediction
• ‘What if..?’ Various scenarios for: Refresh / Deployment / Growth
– Hierarchical Relationships
• Achieving Balance Across Infrastructure UPS’s, PDU’s, Floor Layouts/Cooling
• Efficiently using resources and capital – Getting the most out of Data Centers
– Change management
• Run a Project – We need to Deploy “X” (Space, Power, Cables Ports)
• Day –to- Day tasks – Translation into Work Orders
BROADCOM PROPRIETARY & CONFIDENTIAL
27
Data Center Infrastructure
Management
The Application Story
Due to the proprietary nature of the application and the fact that a
decision to commercialize the product is pending, the application slides
have been removed from this copy of the presentation. However, as
discussed in the workshop if there is interest to learn more and possibly
partner with Broadcom in the further development of the application
contact should be made directly with Dave Eastman at Broadcom.
Dave Eastman; Sr. Principal Project Manager’ IS Infrastructure; deastman@broadcom.com
BROADCOM PROPRIETARY & CONFIDENTIAL
28
Direct Benefits
• Thresholds are understood and tracked
• Efficient use of data center Infrastructure
• Predictable consumption enables non-reactive procurement
• Change Management process is key to success
• Issues that we had no idea of before the application are now
•
identified issues can be rectified during upgrades or planned
outages
What if scenario provides alternatives (stretches the $DC)
– Adjust thresholds to understand the impact
– Refresh opportunities
– Some redundancy can be reduced if the impact is understood
BROADCOM PROPRIETARY & CONFIDENTIAL
29
Indirect Benefits
• Unbalanced or over load conditions rectified before they become
•
•
issues
The journey has allowed us to get commitment from all involved
Next generation design improvements
– Cabling
– Power distribution
– Density
– Efficiency
– Instrumentation is critical and there are options
– Modular/POD designs allow latest technology to be implemented by only
building what you need when you need it
• Rebates for refresh from Power companies
BROADCOM PROPRIETARY & CONFIDENTIAL
30
What Next?
• This is a Peer Review Workshop so . . .
• Tell us what you’re thinking . . .
– Fill out the 1-page participant response survey
BROADCOM PROPRIETARY & CONFIDENTIAL
31
Lunch
Round Table Discussion
BROADCOM PROPRIETARY & CONFIDENTIAL
32
Panel Q & A
• Dave Dalesandro
– Director, IT X-Functional
Architecture
– Data Center Design and
Management
– (ddales@broadcom.com)
• Dave Holub
– Principal Project Manager, - IT
– Data Center infrastructure design
• Al Sorena
– Manager, Critical Facilities
– (asorena@broadcom.com)
• Vippin Chadha
• Regional IT Manager, North
America
• (vchadha@broadcom.com)
architect
– (holub@broadcom.com)
• Dave Eastman
– Sr. Principal Project Manager, - IT
– DCIM Prototype Developer
– (deastman@broadcom.com)
BROADCOM PROPRIETARY & CONFIDENTIAL
33
CFRT Wrap-up Networking and
Informal Q&A
BROADCOM PROPRIETARY & CONFIDENTIAL
34
Broadcom Data Center Tour
(for those interested)
THE END
BROADCOM PROPRIETARY & CONFIDENTIAL
35