Climate and weather modeling prospects for usage by the nascent
climate-adaption and commercial
communities - a "missing middle"?
Mid-Range Scientific
Computing (and its
barriers to use) are a
particular interest of mine
NCAR’s CAS2K11
September 12, 2011, Annecy
Bill Feiereisen
Intel Corporation
University of New Mexico
I am standing between
you and lunch, so feel
free to heckle if I run
over
Intel is changing …
• We make processors, right?
• The following should be evident, but …
– In reality, very few computing users care to buy processors just for fun
– Most users want an answer to a question or a solution to a problem
– The way to understand their computing needs is to understand their
questions
– And translate those into the implications for the underlying computing
architectures
• To do this, Intel is engaging with technical communities in both science
and business
– Not just as a supplier of silicon
– But to learn from the community
The “Missing Middle”
•
•
You will see this chart in many discussions
of scientific computing, but this version
stems from manufacturing.
Studies from manufacturing: There are
many prospective users of mid-range HPC
who “are missing”
–
–
–
•
•
•
Courtesy of the National Center
for Manufacturing Sciences(NCMS)
Between NCMS* and NAM**, the number
of US Manufacturing small/medium
businesses (SMBs) is 280K-350K.
IDC: nearly half of Mfg SMBs would use
HPC if they could; almost all don’t
why?
Is there an equivalent problem in the
weather/climate community?
If so, does it have a similar computing
character?
And what does this character mean for
the computing infrastructure?
*National Center for Manufacturing Sciences
**National Association of Manufacturers
Tell NCAR and NOAA about HPC?
… I don’t think so
• Immensely sophisticated in
HPC, Modeling and Simulation
• Leader in the climate and
weather enterprises
• Supporter of many
organizations who are carrying
their products to other
agencies and commercial
companies
• Let’s look at a sampling of
organizations who are trying
to base their services on
weather and climate products,
and see if their needs are
similar
Downscaled and integrated with local
privately maintained observation
stations
Customers buy energy, not information,
so much of Xcel’s computing work is
traditional
Accuracy of predictions means big
money in energy contracts.
World Health Organization
Rifat Hosain, Collaboration: Global Public Health and Climatology
Assessing WatSan* Resilience to Build an Early Warning System for Water Borne Disease
Coupling short term climate predictions with disease spread models
Time frames are longer than weather, but shorter than climate
These are needs, but organizations like the WHO do not know how
to go about this
*Water and Sanitation
American National Flood Insurance
Program
US Department of Agriculture
Joint Agricultural Weather Facility
• Mission: To keep the Nation’s
growers, exporters, USDA
commodity analysts, as well as
the Secretary and top staff
informed of world-wide weather
related developments and their
effects on crops and livestock.
A mash-up of climate and
agricultural products
Robert Marshal, Earth Networks CEO
Peaking vastly during the weather events of this year
And dominated by interactive requests from mobile devices
Robert Marshal, Earth Networks CEO
Private observation networks to augment the weather service products
And to measure additional factors of interest to customers
Data acquisition devices
attached to all cars and truck –
connected over the cell network
What do we learn from these … and
others?
•
Well … raw tightly coupled compute horsepower … duh!? – for the simulations
–
–
But need way, way more.
Does this mean in-house computing on a smaller scale like NCAR and the National Labs?
•
•
–
•
Right now for direct weather products, but presumably soon for derivative products
This is introducing an interactivity that has not previously existed
Two types of “derivative users”
–
–
•
Land usage, watershed management, transportation, agriculture etc.
With downscaled local simulations of both weather – and as boundary conditions for watershed, disease etc. simulations
Vastly increased demand for output of products through mobile devices
–
–
•
Publicly accessible satellite and NWS data, aircraft, etc. – but
But there is not enough publicly available observation, thus the growth of private networks to gather
And as boundary/initial conditions for downscaled simulations – not only for weather but air quality, etc
Mash-ups of climate and weather predictions with specific databases
–
–
•
And closely tied to an “enterprise-like” infrastructure
And … Assimilation of observational for data – duh?!
–
–
–
•
Maybe, but maybe not …
Some weather product providers are moving part of the their computing to commercially provided clouds – their customer
demand is episodic.
Sophisticated commercial businesses
Smaller businesses and government agencies “who have plans”
What are the prospects for research funding to address these users?
 With the prospective
decline in public funding
 … and the concurrent rise in
demand for commercialized
weather/climate products
 Should we consider
Public/Private Partnerships?
We propose
to construct consortia that will address these problems
Maybe these become Public/Private Partnerships?
What hindrances must we address
carrying this out?
• In manufacturing we know:
• The organizational investments to learn the techniques and tools have
been too great
• The software has required great expertise in the computer science of high
performance computing, and the mathematics of numerical modeling
– this talent pool is small and expensive.
• The disruption to on-going production is too risky
• Scientists, engineers, and researchers coming into the market know their
specific area of expertise but do not necessarily have the computer
science skills necessary to utilize parallel computing.
– Nor access to the full service laboratory that can teach them
• The initial capital costs for computers have (had) been too large?
• These are hindrances we know from more extensive work in the
manufacturing sector – How applicable are they to this sector?
“Architecture” of a Consortium
Three elements:
1) A group of technology providers,
• Focused on the real needs of a small/medium
business or a government agency.
• each bringing complementary technology expertise
(domain, applications, software, hardware).
2) An outreach program to communicate results
and educate users/students.
3) A hardware/software infrastructure.
We think a Consortium must:
• Provide a reference implementation or “how-to”
that includes steps and procedures, software or
proven recommendations for software/hardware.
• Document the business return on investment with
real case studies.
• By solving a real problem, educate potential users
and help build the scarce talent pool.
• In this process introduce new users to technical
HPC – teach the teachers.
Summary
• To the original question: No, the computing
infrastructure is not the same as we have seen in
manufacturing. There is a missing middle, but it will be
oriented more around data gathering, data mash-ups
and distribution to millions of end-users
• Intel is engaging at all of these levels of technology to
help precipitate this
• But we are not getting into the weather/climate
business 