NCAR Supercomputing ‘Data
Center’ Project
An NCAR-led computing ‘facility’ for
the study of Earth System Science
Outline
• A Problem
• An Opportunity
– NSF’s Petascale Roadmap
• A Solution
– Facility Proposal: Site and Cost
– Partners
• The Scientific Payoff
• Next Steps
– Schedule and Panels
NCAR Leadership in
Supercomputing…
• One of the founding missions of NCAR was: “… to provide,
or arrange for provision of facilities for the scientific
community as a whole that whose initial cost and upkeep
lie beyond the capability of individual universities or
research groups.” – Preliminary Plans for a National
Institute for Atmospheric Research. 1959 – NCAR Blue
Book
• Note: the wording does not imply physical collocation.
• This mission does confer a responsibility that cannot be
delegated - namely maintaining an complete integrated
cyberinfrastructure (CI) system for modeling and data
analysis that meets our scientific community’s needs.
Examples of NCAR simulation
science today
•
•
•
•
•
•
Global change climate ensembles
Weather Research Forecast
Geophysical Turbulence
Fire storm front modeling
Space weather
More…
A problem
• NCAR Mesa Lab computer facility is
quickly becoming obsolete
• Power, cooling and floor space will be
inadequate beyond the next
procurement
• Science is being restricted by focusing
on capacity ahead of capability
CMOS Trends Continue …
Chips: Faster, Cheaper but Hotter
An Opportunity
NSF’s Petascale Roadmap
“Overarching Recommendation:
Establish a Petascale Collaboratory
for the Geosciences with the mission
to provide leadership-class
computational resources that will
make it possible to address, and
minimize the time to solution of, the
most challenging problems facing
the geosciences.”
www.joss.ucar.edu/joss_psg/meetings/petascale/
NSF Conclusions
• NSF is committed to developing and implementing a
strategic plan for cyberinfrastructure
– Broad based plan involving the university, Federal agencies,
vendors, and International partners
• ATM, OCE, and EAR take different approaches to the
realization of CI for their discipline
– Dependent on the readiness of the community
• Petascale facility is an integrating theme for the
Geosciences community
– High potential for the generation of new knowledge and paradigm
for the conduct of research
– Building and sustaining a petascale facility will be a significant
challenge to budgets and technology
– Consistent with NSF strategic vision for CI
A solution for NCAR
• A new computing facility (not at the Mesa Lab)
• Extensive investigations, working with
consultants and internal needs resulted in a
detailed set of options
• Provides for 5-20 years of computing (capacity
and capability) diversity based on current and
predicted future trends in CMOS technology
• Allows NCAR to reach beyond its current
research scope
The facility needed
• Data Center Expansion Report from NCAR’s
Computing and Information Systems Lab
• 20,000 (initial to 60,000) sq. ft. of (8ft) raised
floor computer room
• 150,000 sq ft. (gross) building area
• 4 (to 13) MW power + generators
• Cooling, etc.
• On 13 acres (20 year lifetime)
• Accommodates computers, staff, open space,
initial and future requirements
Birds Eye View
Architectural View
Phase 2
Addition
Phase 3
Addition
Importance of Site Selection
• Limited selection of sites that meet criteria
– Size (10-15 acres)
– Electrical capacity (up to 24 MW)
– Fiber optic route (dark fiber)
• Investigated
–
–
–
–
Marshall
Louisville
Longmont
Westminster
(Water, Political Complications, Fiber Optics)
(Electrical Capacity)
• New partners and options are now being sought
–
–
–
–
–
IBM
Colorado School of Mines
Colorado State University
University of Colorado
University of Wyoming
Cost Drivers
• Primary Drivers
– Tier III Reliability
• Mechanical Systems
• Electrical Systems
– Engineering
• Secondary Drivers
– Building Size
– Land Site
•
•
•
•
Facility - up to $75M (one time)
Operations - ~ $15M/year (2Xpresent)
Computing increments - $15M/year (2Xpresent)
Computing infrastructure - $5M/year (new)
The Scientific Payoff…
A petascale computer will enable
scientists to …
• Do credible regional climate modeling for decision
support. Requires resolving individual mountain
ranges and ocean western boundary currents.
• Model climate and weather in a fully coupled mode.
• Better understand the marine biogeochemical cycles.
Requires resolving ocean mesoscale eddies.
• Accurately simulate the dynamical, microphysical and
radiative cloud processes.
• Improve seismic predictions and understand the
structure of the inner core as well as the fine
structure of the lower mantle.
A petascale computer will enable
scientists to
• Perform new research in solid earth and
environmental engineering
• Assimilate thousands of earthquakes bringing the fine
structure of the Earth’s mantle and inner core into
focus.
• Study the physical basis of land surface
parameterizations by modeling soils, topography and
vegetation at sub-meter scales.
• More accurately predict the damaging effects of solar
flares on satellites and power distribution systems by
resolving the fine structure of the corona magnetic
field.
• Investigate energy management applications
Science examples …
Hurricane Katrina Reflectivity at Landfall
29 Aug 2005 14 Z
4 km WRF, 62 h forecast
Mobile Radar
Main Points
• Huge scientific discoveries await geoscience modelers at
1 PFLOPS and beyond.
• CMOS continues to get hotter and cheaper. The most
recent acquisition tracks this trend.
• Every center is (or will be) facing facility challenges in
the race to these discoveries. This situation is NOT
unique to NCAR.
• NCAR now has a facility plan, that if successful, uniquely
positions it as a world leader in geoscience simulation.
• The new facility is not a crisis: it is an opportunity.
The Opportunity
• Understanding of fundamental physical
processes in the Sun-Earth system
• Environmental and Energy applications
not yet possible
• NCAR and partners will scope/define
these options
– Such a facility would be a computational equivalent of the
Hubble Telescope for geoscience simulation.
Next Steps
NSF Geosciences Community
Draft Version 1
UCAR/ATM Universities
NSF OCN and EAR Universities
NSF Geosciences Research Programs (ITR,OCI)
The Geosciences
Research Consortium
Geoscience
Domain Partner
Geoscience
Domain Partner
NSF Geosciences
Capability Center
Geoscience
Domain Partner
Geoscience
Domain Partner
NCAR + Facility Partner
Somewhere near Boulder
Geoscience
Domain Partner
Geoscience
Domain Partner
The Schedule
• Formed NCAR project committee
• Forming Blue Ribbon Panel and hold teleconference Oct. 17 2005, meet mid-Nov.
• Project plan development Oct-Dec
• Community engagement - Nov-Jan
• Formalize partnerships - Oct-Dec
• Present initial plan to National Science Foundation,
late-October, 2005
• Forge international collaborations - Nov. 2005
• Complete project plan - Feb. 2005
• Initiate facility - June 2006?
• First electrons - June 2008 - March 2009?
Contacts at NCAR
• Tim Killeen (killeen@ucar.edu) - NCAR
Director
• Lawrence Buja (southern@ucar.edu) and
Peter Fox (pfox@ucar.edu) are co-chairs of
the NCAR project team
• Aaron Anderson (aaron@ucar.edu) is the
computing facilities contact
• Jeff Reaves (jreaves@ucar.edu) is the
financial/ contracts contact
Concluding remarks …