Columbia University’s
Advanced Concept Data Center Pilot Project
May 15, 2009
Agenda
1:00pm
Introductions
1:15pm
Welcome and Opening Remarks
Candace Fleming, Vice President, CUIT
David Hirsh, Executive Vice President for Research
Nilda Mesa AVP, Environmental Stewardship
1:35pm
Green DC Program presentation
Alan Crosswell, Chief Technologist, CUIT
2:20pm
CU Data Center
Ian Katz, Manager, CUIT Data Center
Current Status
2:35pm
NYSERDA Project Overview
Richard Hall, Project Manager CUIT
3:00pm
Closing Comments & Questions from guests
Victoria Hamilton, Research Initiatives Coordinator
The “Big Potential”
Comments from NYSERDA (tentative)
2
Introductions
May 15, 2009
External Visiting Committee
•
Laurie Kerr, Senior Policy Advisor for Energy and Green Buildings, NYC Mayor’s Office of LongTerm Planning and Sustainability
•
Vace Kundakci, Assistant Vice President for Information Technology and Chief Information Officer,
City College of New York/City University of New York
•
Timothy Lance, President and Board Chair, New York State Education and Research Network
(NYSERNet)
•
Marilyn McMillan, Associate Provost and Chief Information Technology Officer, New York
University
Role:
The External Visiting Committee is charged with maintaining skepticism about our plans and our
accomplishments from the perspective of external institutions. To the extent we demonstrate
feasibility and potential impact, they will serve as powerful ambassadors to other institutions.
4
Internal Advisory Group
•
Wilmouth A. Elmes, Associate Vice President of Engineering/Technical Services, Manhattanville
Development Project
•
Doug McKean, Director, Capital Project Management
•
Arthur M. Langer, School of Engineering and Applied Science, Senior Director of the Center for
Technology, Innovation, and Community Engagement and Faculty & Assoc Director, Executive
Masters of Science in Technology Management at the School of Continuing Education
•
Nilda Mesa, Assistant Vice President, Environmental Stewardship
•
Scott W. Norum, Chief Administrative Officer for Arts and Sciences and Vice President, Office of
the Vice President for Arts and Sciences
•
Leonard Peters, School of Business, Associate Dean and Chief Information Officer – Information
Technology
Role:
The Internal Advisory Group serves two primary functions. Not only do we anticipate they will prove
a valuable sounding board for the execution of the pilot (implementation guidance/expert judgment),
but we expect them to pose the hard questions about issues surrounding scale-up (feedback).
5
Research Faculty User Group
•
Liam Paninski, Associate Professor, Dept. of Statistics
•
Lei Cong, Staff Associate, Dept. of Statistics
•
Kathryn V. Johnston, Associate Professor, Dept. of Astronomy
•
Mary Putman, Clare Boothe Luce Associate Professor of Astronomy
•
Greg L. Bryan, Assistant Professor , Dept. of Astronomy
Role:
Responsible for vetting the results of the combined HPC cluster for the Department of Astronomy
and the Department of Statistics
6
Working Group
•
•
•
•
•
•
•
•
•
•
•
•
Principal Investigator
Alan Crosswell, Associate Vice President and Chief Technologist, CUIT
Co-Principal Investigator
Victoria Hamilton, Director of the Office for Research Initiatives (ORI)
Richard Hall, Project Manager
Anthony Cirillo, AVP Systems and Operations
Peter Miner, Senior Director, Systems Engr
Ian Katz. DC Facilities Manager
Melissa Metz, Director of Systems Engineering
Fran Ovios, Director of Systems Engineering
Stew Feuerstein, Director and Chief Architect
Joseph Rini, Director of Network Engineering
Megan Andersen, Research Systems Administrator
Fran Fitzgerald, CU Facilities Project Manager
Role:
Day-to-day responsibility for planning, executing and measuring required by the proposal. The
Columbia University in-house team will perform much of the work.
7
Project Structure
8
Welcome & Opening Remarks
Candace Fleming
Vice President and CIO, Columbia University Information Technology
David Hirsh
Executive Vice President for Research
Nilda Mesa
AVP, Environmental Stewardship
Green DC Program presentation
Alan Crosswell, Assoc VP and Chief Technologist, CUIT
CU Data Center - Current Status
Ian Katz, Facility Manager, CUIT Data Center
NYSERDA Project Overview
Richard Hall, Project Manager, CUIT
Project Objectives
•
•
•
•
•
Become well versed in data center efficiency, design techniques and assessment
metrics such as those espoused by Green Grid, ASHRAE and others (PUE, DCeP,
etc.).
Establish baselines and continuously measure several power and cooling variables to
enable the study of historical trends and produce a data set to aid in further analysis of
data center electrical and heat loading changes over time.
– Implement the recommended and advanced best practice improvements and
measure the degree of efficiency achieved when these techniques are applied in
our data center
Validate claims of the degree of power and cooling efficiency improvements that are
achievable through best practices and advanced data center techniques.
Utilize results to inform subsequent phases of the central data center re-fitting and
design of future data center space
Provide training and guidance for other NYS data center operators, especially peer
higher education institutions
13
Project Scope
•
Include participation and critique from influential skeptics
•
Update the Data Center’s aging infrastructure (modular approach)
– Replacement of thirty servers, each of which is at the end of their useful life
– Implementation of nine high-density equipment racks
• support current and future research computing clusters
• consolidating the University’s server needs for High Density Computing
(HDC).
– Address electrical distribution systems to increase efficiency
– Improve and or replace inefficient cooling systems
– Installation of instrumentation for power and cooling systems to monitor
consumption and performance to establish metrics to measure our progress
toward PlaNYC.
•
Realize energy and environmental gains while demonstrating economic and
operational feasibility in infrastructure improvements, innovative computational
equipment, centralized administrative systems and dispersed research clusters.
14
Project Plan
To achieve the overall project objectives, CUIT has broken down the project into major tasks.
•
Inventory
– Create detailed physical inventory of existing in-scope server racks
– Create data center baseline (formulated from above inventoried equip)
•
Instrument server power consumption
– Install network monitored power monitors for each server
– Perform data collection at 5-min intervals using one or more of the following methods:
•
Instrument server input air temperature and overall data center chilled water heat load
– Install server input ambient air temperature for each server
– Install BTU metering on data center supply and return lines
– Perform data collection at 5-min intervals
•
Establish overall Data Center profile
– Utilize equipment load results to establish baselines
– Develop PUE ratio for entire data center & inventoried servers
•
Implement 9 racks of high power density in-row cooling
– Install 9 server racks outfitted for high power density
– Provide in-row cooling subsystem for 9 server racks
– Implement measurement system for in-row cooling, BTU rejection and electrical consumption
15
Project Plan Cont’d
•
•
•
•
•
Replace 30 “old” servers and measure efficiency improvement.
– Consolidate the replacement servers into high density racks and re-implement the same IT
services
– Take measurements of before-and-after power consumption
– Document expected and actual efficiency improvement
Compare old and new research clusters
– Benchmark applications on new Astronomy/Statistics HPC cluster
– “kWh consumed” comparison with old Astronomy server cluster
Implement server power management
– Install server BIOS/high level power management feature upgrades to servers (identified in
task 2)
– Document benchmarks
Increase chilled water set point for high power density racks and measure
– Document measured before-and-after BTUs
Communicate results
– Share results with key stakeholders
16
Project Schedule
•
•
•
•
•
•
•
•
•
•
•
•
Project Approval
Inventory
Instrument Power
Instrument Cooling
Data Center Profile
Nine Racks
Replace 30 Servers
Compare Clusters
Server Power Mgmt
Increase 5-degrees
Project Review
Project close
4/1/09
5/15/09
7/30/09
9/30/09
9/30/09
1/31/10
5/30/10
10/31/09
9/30/10
3/31/10
9/30/10
10/1/10
17
Closing Comments
Victoria Hamilton, Director of the Office for Research Initiatives (ORI)