Proceedings of the WTEC Workshop
INTERNATIONAL ASSESSMENT OF R&D IN
S I M U L A T I O N -B A S E D E N G I N E E R I N G A N D S C I E N C E
National Science Foundation
4201 Wilson Boulevard, Room 375, Arlington, VA 22230
April 25, 2008
WTEC PANEL ON SIMULATION-BASED ENGINEERING AND SCIENCE
Sponsored by the National Science Foundation (NSF), the Department of Defense (DOD), the Department of
Energy (DOE), the National Aeronautics and Space Administration (NASA), the National Institute for Biomedical
Imaging and Bioengineering (NIBIB) and the National Library of Medicine (NLM) of the National Institutes of
Health (NIH), and the National Institute of Standards and Technology (NIST).
Dr. Sharon C. Glotzer (Chair)
Department of Chemical Engineering
University of Michigan
3074 H.H. Dow Building
Ann Arbor, MI 48109-2136
Dr. Sangtae Kim (Vice Chair)
School of Chemical Engineering
Forney Hall of Chemical Engineering
Purdue University
480 Stadium Mall Drive
West Lafayette, IN 47907-2100
Dr. Peter Cummings
303 Olin Hall
Vanderbilt University
VU Station B 351604
Nashville, TN 37235
Dr. Abhi Deshmukh
Department of Industrial and Systems
Engineering
Texas A&M University
235G Zachry Engineering Center
3131 TAMU
College Station, TX 77843-3131
Dr. Martin Head-Gordon
Department of Chemistry
University of California, Berkeley
Berkeley, CA 94720
Dr. George Karniadakis
Division of Applied Mathematics
Box F
Brown University
Providence, RI 02912
Dr. Linda Petzold
Department of Computer Science
Room 3217 Phelps Hall
University of California, Santa
Barbara
Santa Barbara, CA 93106
Dr. Celeste Sagui
Department of Physics
North Carolina State University
Raleigh, NC 27695-8202
Dr. Masanobu Shinozuka
Department of Civil and
Environmental Engineering
University of California, Irvine
E-4150 Engineering Gateway
Irvine, CA 92697-2175
ADVISORY PANEL
Dr. Jack Dongarra
Electrical Engineering and Computer
Science Department
1122 Volunteer Blvd
University of Tennessee
Knoxville, TN 37996-3450
Dr. James J. Duderstadt
Millennium Project
University of Michigan
2001 Duderstadt Center
2281 Bonisteel Blvd.
Ann Arbor, MI 48109-2094
Dr. Tinsley Oden
The University of Texas at Austin
1 University Station C0200
Austin, TX 78712-0227
Gilbert S. Omenn, M.D., Ph.D
Internal Medicine, Human Genetics
and Public Health
Center for Computational Medicine
and Biology
University of Michigan
100 Washtenaw Avenue
2017F Palmer Commons
Ann Arbor, MI 48109-2218
Dr. Tomás Díaz de la Rubia
Lawrence Livermore National
Laboratory
7000 East Avenue
Livermore, CA 94550
Dr. David E. Shaw
The D. E. Shaw Group
39th Floor, Tower 45
120 West Forty-Fifth Street
New York, NY 10036
Professor M. A. Wortman
Industrial & Systems Engineering
237-C Zachry Engineering Center
Texas A&M University
College Station, TX 77843-3131
WTEC Mission
WTEC provides assessments of international research and development in selected technologies under awards from the
National Science Foundation (NSF), the Office of Naval Research (ONR), and other agencies. Formerly part of Loyola
College, WTEC is now a separate nonprofit research institute. Michael Reischman, Deputy Assistant Director for Engineering,
is NSF Program Director for WTEC. Sponsors interested in international technology assessments and related studies can
provide support for the program through NSF or directly through separate grants or GSA task orders to WTEC.
WTEC’s mission is to inform U.S. scientists, engineers, and policymakers of global trends in science and technology. WTEC
assessments cover basic research, advanced development, and applications. Panels of typically six technical experts conduct
WTEC assessments. Panelists are leading authorities in their field, technically active, and knowledgeable about U.S. and
foreign research programs. As part of the assessment process, panels visit and carry out extensive discussions with foreign
scientists and engineers in their labs.
The WTEC staff helps select topics, recruits expert panelists, arranges study visits to foreign laboratories, organizes workshop
presentations, and finally, edits and publishes the final reports. Dr. R. D. Shelton, President, is the WTEC point of contact:
telephone 410-467-9832 or email Shelton@ScienceUS.org.
WTEC International Assessment of Research and Development in
Simulation-Based Engineering & Science
National Science Foundation
4201 Wilson Blvd., Room 375, Arlington, Virginia 22230
April 25, 2008
AGENDA
12:00
LUNCH and Informal Discussions
Opening
1:00
Welcome — Richard Buckius, Assistant Director, NSF/ENG
1:10
Study Process & Executive Summary — Sharon Glotzer, Study Chair,
University of Michigan
Materials Applications
1:40
Peter Cummings, Vanderbilt University
Life Sciences and Medicine
2:00
Linda Petzold, University of California, Santa Barbara
Energy and Sustainability
2:20
Masanobu Shinozuka, University of California, Irvine
2:40
BREAK
Education and Training
2:50
Celeste Sagui, North Carolina State University
Next-Generation Architectures and Algorithms
3:10
George Karniadakis, Brown University
Multiscale Modeling
3:25
Peter Cummings, Vanderbilt University
Simulation Software
3:40
Martin Head-Gordon, University of California, Berkeley
Engineering Simulation
3:55
Abhi Deshmukh, Texas A&M
Verification, Validation, and Uncertainty Quantification
4:10
George Karniadakis, Brown University
4:25
BREAK
Big Data and Data-Driven Simulations
4:35
Sangtae Kim, Purdue University
Open Discussion/Closing
4:50
Open Discussion — Clark Cooper, Ken Chong, Phil Westmoreland, and Celeste
Rohlfing, Program Directors, NSF
5:30
Closing — Sharon Glotzer
WORLD TECHNOLOGY EVALUATION CENTER, INC. (WTEC)
R. D. Shelton, President
Michael DeHaemer, Executive Vice President
Geoffrey M. Holdridge, Vice President for Government Services
David Nelson, Vice President for Development
Ben Benokraitis, Assistant Vice President
Laura Pearson, Project Manager
Patricia M.H. Johnson, Director of Publications
Halyna Paikoush, Event Manager
This document is sponsored by the National Science Foundation (NSF) under grant No. ENG-0423742 to the
World Technology Evaluation Center, Inc. The Government has certain rights in this material. Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the United States Government, the authors’ parent institutions, or WTEC.
As a draft document, any conclusions are subject to revision.
Copyrights are reserved by individual authors or their assignees except as noted herein. Reproduced with
permission.
Some WTEC final reports are distributed by the National Technical Information Service (NTIS) of the U.S.
Department of Commerce. A list of available WTEC reports and information on obtaining them are on the inside
back cover of this report.
WTEC International Assessment of Research and Development
in Simulation-Based Engineering & Science
TABLE OF CONTENTS
Study Process and Executive Summary (Sharon Glotzer)
Presentation ............................................................................................................. 1
Materials Modeling (Peter Cummings)
Presentation ........................................................................................................... 21
Computational Life Sciences and Medicine (Linda Petzold)
Presentation ........................................................................................................... 29
Energy and Sustainability (Masanobu Shinozuka)
Presentation ........................................................................................................... 35
Education and Training (Celeste Sagui)
Presentation ........................................................................................................... 50
Next-Generation Architectures and Algorithms (George Karniadakis)
Presentation ........................................................................................................... 60
Multiscale Modeling (Peter Cummings)
Presentation ........................................................................................................... 67
Simulation Software (Martin Head-Gordon)
Presentation ........................................................................................................... 73
Engineering Simulation (Abhi Deshmukh)
Presentation ........................................................................................................... 79
Verification, Validation, and Uncertainty Quantification (George Karniadakis)
Presentation ........................................................................................................... 89
Big Data and Data-Driven Simulations (Sangtae Kim)
Presentation ........................................................................................................... 98
Sites Visited by the WTEC SBE&S Panel...................................................................... 103
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
International Assessment of Research and
Development in
Simulation-Based
Engineering and Science
Study Process
and Executive Summary
Sharon C. Glotzer
University of Michigan
 The International Assessment of Research and Development
 in Simulation-Based Engineering and Science (SBE&S) is
 co-sponsored by the
ƒ National Science Foundation (NSF)
ƒ Department of Energy (DoE)
ƒ Department of Defense (DoD)
ƒ National Institutes of Health (NIH)
ƒ National Institute of Biomedical Imaging and Bioengineering
(NIBIB)
ƒ National Aeronautics and Space Administration (NASA)
ƒ National Institute of Standards and Technology (NIST)
 WTEC is the leading organization in the U.S. that performs
 international technology assessments via expert review
ƒ WTEC has conducted over 60 such studies since 1989
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
1
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Previous SBES study
 Our study builds
upon previous efforts:
ƒ Workshops run by NSF
Engineering Directorate
ƒ NSF Blue Ribbon Panel
report chaired by J.
Tinsley Oden, May 2006 lays out intellectual
arguments for SBES
ƒ SBES broadened to
SBE&S
ƒ & many previous reports
on computational science
http://www.nsf.gov/pubs/reports/sbes_final_report.pdf
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
SBE&S - A National Priority
 “The Promise: Advances in mathematical modeling,
in computational algorithms, in the speed of
computers, and in the science and technology of data
intensive computing, have brought the field of
computer simulation to the threshold of a new
era, an era in which unprecedented improvements in
the health, security, productivity, and competitiveness
of our nation may be possible. A host of critical
technologies are on the horizon that cannot be
understood, developed, or utilized without simulation
methods.”
--From Oden report
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
2
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBE&S: Why now?
 A tipping point in SBE&S
ƒ Computer simulation is more pervasive today, and
having more impact, than ever before - hardly a field
untouched
ƒ Fields are being transformed by simulation
ƒ Reached a useful level of predictiveness;
complements traditional pillars of science
ƒ “Flattening world” of computer simulation that will
continue to flatten - everyone can do it
ƒ Disruptive multicore technology likely to be
transformative for SBE&S
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
SBE&S: Why now?
 Simulation is key to scientific discovery and engineering
innovation.
ƒ The toughest scientific and technological problems facing society
today are complex and messy, and their solution requires a
partnership among experiment, theory and simulation,
working across disciplines.
ƒ Recent reports argue our nation is at risk at losing of its
competitive edge. Our continued capability as a nation to lead
in simulation-based discovery and innovation is key to our ability
to compete in the 21st century.
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
3
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Overall Scope/Objectives of Study
 Gather key data needed to assess:
ƒ where the next big breakthroughs are likely to come from,
and in what;
ƒ where US is leading, trailing, or in danger of losing
leadership in SBE&S;
ƒ where critical investments in SBE&S are needed to maintain
or gain US leadership, and how those investments will
impact R&D and innovation capabilities in strategic areas for
US.
 Provide sufficient analysis and guidance to inform
and shape development of multi-agency federal
initiative in SBE&S
 Findings, not recommendations
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Structure of Study
 Primary thematic areas
ƒ Materials
ƒ Life sciences and medicine
ƒ Energy and sustainability
 Core cross-cutting issues
ƒ Next-generation algorithms and high performance computing
ƒ Multiscale simulation
ƒ Simulation software
ƒ Validation, verification, and quantifying uncertainty
ƒ Engineering systems
ƒ Big data and data-driven simulations
ƒ Education and training
ƒ Funding, organization, and collaboration
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
4
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
The SBE&S Study Team
Panelists
Sharon Glotzer (Chair)
Sangtae Kim, NAE (Vice Chair)
Peter Cummings
Abhi Deshmukh
Martin Head-Gordon
George Karniadakis
Linda Petzold, NAE
Celeste Sagui
Masanobu Shinozuka, NAE
University of Michigan
Purdue University
Vanderbilt University & ORNL
Texas A&M University
University of California Berkeley
Brown University
University of California Santa Barbara
North Carolina State University
University of California Irvine
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Advisors
 Tomas Diaz de la Rubia, Lawrence Livermore National Lab
 Jack Dongarra, University of Tennessee/ORNL
 James Duderstadt, University of Michigan
 David Shaw, D. E. Shaw & Co. & Columbia Univ.
 Gil Omenn, University of Michigan
 J. Tinsley Oden, University of Texas, Austin
 Martin Wortman, Texas A&M
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
5
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Study Timeline


Kickoff meeting:
Baseline workshop:
Visit to Asia:
Visit to Europe:
Final workshop:
Final report:

Research Directions Workshop in Fall 2008




10 July 2007
1-2 November 2007
3-7 December 2007
25-29 February 2008
25 April 2008
Fall 2008
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Bibliometrics
 SBE&S publications increasing at a rate more than
double that of E&S publications (5% vs. 2.5%).
 In 2007, US dominates world output with 27%, but
China (CN) moves from 6th to 2nd at 13%.
 US output is consistently less than that of EUR-12,
and the difference is increasing.
evaluametrics
Evaluation of research through publication metrics
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
6
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Sites Visited in China – December 2007
Peking Univ./CCSE,
Tsinghua Univ./DEM,
ICCAS, ICMSEC/CAS,
IPE/CAS,
Dalian Univ. of
Technology
SSC, Shanghai
Univ., Fudan Univ.
http://www.lonelyplanet.com/maps/asia/china/
Sites Visited in Japan – December 2007
RIKEN/ACCC
NIMS/CMSC,
RICS/AIST
Kyoto Univ.
CRIEPI, SBI,
Univ. Tokyo
Japan Agency for
Marine-Earth S&T (ESC),
Nissan Research Center,
Mitsubishi Chemicals
Toyota Central R&D Labs.,
IMS
http://www.ease.com/~randyj/japanmap.htm
7
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Sites Visited in Europe – February 2008
Unilever R&D,
Daresbury Lab
Univ. College London,
The Thomas Young
Centre
Univ. Oxford,
Univ. Cambridge,
Unilever Centre
Vrije Univ.
DTU
ZIB
IWM,BASF, ITTPE,
Univ. Karlsruhe
IFP
Paris Simulation Network
Tech. Univ. Munich
IMFT, ENSEEIHT, IRIT
Eni SpA, MOX*
*Remote site visit
CIMNE, ICMAB/CSIC
http://www.europeetravel.com/maps/western-europe-map.htm
“If it’s
Tuesday,
it must be
Geneva…”
57 sites/36 in Europe
Group B Itinerary
Sat 23 – Sun 24 Feb
1. Frankfurt
Mon 25 Feb
2. Ansterdam: Vrije Universitat
(Theoretical Chem, Molecular Cell
Physiology, Biophysics)
Tue 26 Feb
3. Genève
4. Lausanne: EPFL (Math Dept.,
Blue Brain Project)
5. Genève: CERN
Wed 27 Feb
6. Munich: TUM (Informatik,
Bauinformatik)
7. Berlin: Konrad-Zuse-Zentrum
(ZIB)
Thur 28 – Fri 29 Feb
8. Zürich: IBM, ETHZ, ETHH
(Th) Universität Zürich (F)
9. Freiburg: IWM (F)
10. Frankfurt (F)
http://www.europeetravel.com/maps/western-europe-map.htm
8
CERN, EPFL/IACS,
ETH, IBM, Univ.
Zürich
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Virtual Site Visits (in progress)
 India, Italy, Russia,Taiwan, Korea, Israel
 Additional industry
ƒ GM, Intel, Boeing, Ford, Mercedes, BMW, Dow
Chemical, BP
ƒ Airbus, Rolls Royce, DK Wind Energy companies
 National labs, large collaborative programs
ƒ NTI @ ORNL
ƒ Nanohub
ƒ NIEHS
ƒ National Cancer Institute
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
SBE&S: Next big
breakthroughs and opportunities
 Data-intensive applications
 Integration of experimental and observational data
with modeling and simulation to expedite discovery
and engineering solutions
ƒ Including real-time data integration
 Millisecond timescales for proteins and related
systems with molecular resolution
 More fundamental models in engineering simulations
ƒ Increased resolution and inclusion of physics and chemistry
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
9
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBE&S: Next big
breakthroughs and opportunities
 Multiscale, multiphysics + computational optimization
of multiple objectives + UQ and risk management
 Petascale and multicore
ƒ Increased complexity of problem, speed to solution
 Materials, life sciences & medicine, and energy &
sustainability are among the most likely sectors to be
transformed by SBE&S
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Breakthroughs and opportunities:
SBE&S for Materials
 Structural
ƒ New composites for aircraft, virtual aluminum castings (Ford),
crash testing of steel, fracture toughness and strength, nuclear
waste containment
 Functional
ƒ Nonlinear optical materials, porous materials for catalysis/fuel
cells, skin creams and deodorants.
 Smart biomaterials
ƒ Actuating biopolymers, self-assembled lipids
 ICME: the integration of materials information, captured in
computational tools, with engineering product performance analysis
and manufacturing-process simulation. (NAS Report, 2008)
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
10
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Breakthroughs and opportunities:
SBE&S for Life Sciences and
Medicine
 Pharmaceutical
ƒ Binding of drugs to target, ID of disease
mechanisms and intervention, personalized
medicine, …
 Medical
ƒ Surgery, blood flow, biomechanics/physical
therapy…
 Life sciences
ƒ Structure and function of the brain, systems
biology, biomolecular structure and function
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Breakthroughs and opportunities:
SBE&S for Energy & Sustainability
 Oil/gas production and CO2 sequestration
ƒ Reservoir simulations for oil recovery
 Natural disaster prediction with dynamic data-
driven simulation (tsunamis, earthquakes)
 From physical systems modeling to socialscale engineered systems
ƒ Agent-based simulations of behavioral patterns of 6
billion people
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
11
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBE&S beyond our themes:
LHC at CERN
 Particle physics experiments at CERN demonstrate partnership
among theory, experiment, simulation
 The entire operation, from digging the hole in the ground to the
detection of the Higgs, relies on accurate predictions from
computer simulation.
 Sag in detector under gravity required removal of specific
amount of bedrock (Civil engineering)
 Interaction of radiation with detector materials (Spinoffs to
medical diagnostic instrumentation)
 Signatures of elementary particles (Understanding the origin of
mass in the early universe)
 Gold standard for open science collaboration. Data needs led to
the development of WWW. Simulation needs rely on grids. Poster
child for big data, at 100 Pbytes/minute, reduced to 10
Pbytes/year.
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Worldwide threats to sustained
progress in SBE&S
 Investment in algorithm, middleware, software development lags
behind investment in hardware, and is insufficient.
 Anticipated inability to fully exploit multicore/petaflop technology
 Continued lack of support and reward for code development and
maintenance
ƒ Timescale to develop large complex code > hardware lifetime
ƒ The UK, which once led in this, does not provide the support it once did
 Inability to include complexity and to cross disciplinary boundaries
ƒ In most engineering applications, algorithms, software and data
are primary bottlenecks.
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
12
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Threats to sustained progress
in SBE&S (continued)
 Education and training of the next generation of
simulators is a worldwide concern.
ƒ Insufficient training in HPC; major worry for multicore in US.
ƒ Increased topical specialization beginning with grad school
 Europe and US have various interdisciplinary programs (e.g. certificates,
new departments), but not teaching the right skill sets
ƒ Insufficient exposure to simulation and underlying core subjects at
high school and undergraduate level
ƒ Insufficient “continued learning” opportunities related to programming
for performance
ƒ No real training in software engineering.
ƒ Little to no training in UQ, V&V, risk assessment & decision making
(from atoms to enterprise).
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Threats to US leadership
in SBE&S
 Many countries now have and use HPC
 Japan and Germany have world-class resources, faculty and students
and are committed to sustained investment in HPC and SBE&S
 Despite US DOE lab leadership in applied HPC algorithms, fundamental
algorithm development in US lags
 US invented and first used multicore technology in computers, but we
are not training next generation of simulators to use it
 Community code development projects much stronger in EU, with
national strategies and long-term support.
 Many of the most popular codes developed outside US; some cannot be
used by our defense labs, and we are not developing our own
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
13
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Threats to US leadership in
SBE&S (continued)
 US lead slim in integration of UQ, V&V for “atoms to enterprise’
 The transition from physical systems modeling to social-scale
engineered systems in US lags behind the Japanese
 Many of the best students from Latin America, China, elsewhere
in SBE&S now going to EU instead of US, and we’re not growing
enough of our own.
 All but the top academic institutions report increasing difficulty in
finding qualified/interested SBE&S students.
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Threats to US leadership in
SBE&S (China)
 Not currently a strong US competitor in SBE&S, but their “footprint”
in SBE&S is changing rapidly.
 Strategic change towards innovation, and recognition by industry and
State that innovation requires simulation
 Recognition of need to train new generation of “computationallysavvy” students, and new large-scale programs at the Ministry of
Education to do this
 Overall non-uniform quality of SBE&S research, but high quality
examples on par with EU and US
 China contributes 13% of the world’s output in simulation papers,
second to the US at 27% and growing (although they still publish in
lower impact journals and are overall cited less frequently).
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
14
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Threats to US leadership in
SBE&S (Germany)
 Vigorous new initiatives in SBE&S and commitment.
 Longtime strengths in molecular, polymer, structural materials,





catalyst & process simulation
Increased commitment to SBE&S through industry and government
partnerships
Restructuring of universities (centers, curricula, industry/faculty)
Sustained commitment to HPC infrastructure and “big iron”
Distinctive mechanism for code development support at
supercomputer centers
DFG initiatives (Priority Program Initiative by white papers - $3B/6yr
each year)
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Opportunities for US
investment
 IC-X (e.g. ICME) - industry-driven partnerships with universities,
labs to hardwire scientific discovery to engineering innovation
through SBE&S
ƒ Payoff: New better products; development savings in $$ and time.
 Developing standards for interoperability of codes
 New paradigms for education and training of the next generation
(software engineering, V&V, petascale, etc.)
 Long-term support of code development (and maintenance)
projects for targeted problems in science and engineering
 Support to community in preparing for multicore/petascale
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
15
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Anticipated benefits of SBE&S
 SBE&S will change the way materials are designed,
developed, and applied.
 SBE&S will change the way disease is treated, the
way surgery is performed and patients are
rehabilitated, the way we understand the brain.
 SBE&S will aid in the recovery of untapped oil, the
discovery of new energy sources, and the way we
design sustainable infrastructures.
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
The Promise of SBE&S
 Use of computational science & engineering underpins
progress in all areas of science and technology
 We can reinforce our lead in SBE&S through strategic
research and investments: people, education,
cyberinfrastructure, partnerships
 Leadership in SBE&S = technological leadership =
economic leadership
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
16
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Additional Slides
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Site Visits (China)
 CAS Institute of Computational Mathematics and
Scientific/Engineering Computing (ICMSEC)
 Tsinghua University
 Peking University
 CAS Institute of Polymer Science and Physics
 Dalian University of Technology
 Fudan University
 CAS Institute of Atmospheric Physics
 Shanghai University
 Shanghai Supercomputer Center
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
17
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Site Visits (Japan)
 RIKEN Next-Generation Supercomputer R&D
Center
 RIKEN Advanced Center for Computing and
Communication
 Sony Computer Systems Laboratory
 Central Research Institute of the Electric Power
Industry (CRIEPI)
 National Research Grid Initiative (NAREGI)
 University of Kyoto
 Toyota Central R&D Laboratories
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Site Visits (Japan, cont’d)
 Institute for Molecular Science
 University of Tokyo
 Earth Simulator Center
 Mitsubishi Chemical Corporation
 Tokyo Institute of Technology
 National Institute for Materials Science (NIMS)
 Research Institute for Computational Sciences
(RICS)
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
18
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Site Visits (Europe)
 CH: Switzerland
ƒ ATLAS, LHC; European Organization for Nuclear Research (CERN),
Genéve
ƒ Mathematics, Blue Brain Project; Ecole Polytechnique Fédérale de
Lausanne (EPFL)
ƒ Institute of Physical Chemistry, Universität Zürich
ƒ Fluid Dynamics, Computational Science, Seismology &
Geodynamics, Physical Chemistry, Theoretical Physics;
Eidgenössische Technische Hochschule Zürich (ETHZ)
ƒ IBM Zürich Research Laboratory (ZRL), Zürich
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Site Visits (Europe, cont’d)
 DE: Germany
ƒ Konrad-Zuse-Zentrum für Informationstechnik (ZIB), Berlin
ƒ Fraunhofer Institute for Mechanics of Materials (IWM),
Freiburg
ƒ Institute of Physical Chemistry, University of Karlsruhe
ƒ BASF AG, Ludwigshafen am Rhein
ƒ Institut für Informatik, Lehrstuhl für Bauninformatik;
Technische Universität München
ƒ Institute of Thermodynamics and Thermal Process
Engineering, Stuttgart
 DK: Denmark
ƒ Biology, Chemical Engineering, Fluid Mechanics, Physics;
Technical University of Denmark (DTU), Lyngby
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
19
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Site Visits (Europe, cont’d)
 ES: Spain
ƒ Universitat Politécnica de Catalunya, Barcelona
ƒ Universitat Autònoma de Barcelona (UAB), Bellaterra (Barcelona)
 FR: France
ƒ Institute of Fluid Mechanics (IMFT), Toulouse
ƒ Laboratoire MIP, Universitè Paul Sabatier, Toulouse
ƒ ENSEEIHT, Toulouse
ƒ Airbus Industrie, Toulouse
ƒ Institut Français du Pétrole (IFP), Rueil-Malmaison
ƒ École Nationale Supérieure de Chimie de Paris (ENSCP), Paris
 IT: Italy
ƒ EniTecnologie SpA, Milano
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
Site Visits (Europe, cont’d)
 NL: Netherlands
ƒ Biology, Chemistry & Pharmaceutical Sciences, Biophysics; Vrije
Universiteit, Amsterdam
 UK: United Kingdom
ƒ Science Technology Facilities Council (STFC) Daresbury Laboratory,
Warrington
ƒ Unilever Research Laboratory, Bebington
ƒ Biochemistry, Engineering Science, Physics; University of Oxford
ƒ Applied Math & Theoretical Physics, Chemistry, Physics; University
of Cambridge
ƒ Centre for Computational Science, University College London
ƒ Theory and Simulation of Materials, The Thomas Young Centre (Five
London colleges)
WTEC / SBES Workshop 25 April 2008 -- Glotzer: Executive Summary
20
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Materials Modeling
Peter T. Cummings
Chemical Engineering, Vanderbilt University
Center for Nanophase Materials Sciences, Oak Ridge
National Laboratory
SBE&S Report-Out to Sponsors
April 25, 2008
Potential Impacts
 New materials have always been
 Revolutionary
 Changed lives and course of civilization
 Led to whole new industries
 Examples
 Stone age (caves) o bronze age (farmsteads) o iron
age (cities)
 Plastics replacing wood, steel
 Synthetic fibers (e.g., nylon) replacing natural fibers (e.g.,
silk)
 Silicon-based integrated circuits supplanting discrete
transistors
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
21
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Research and Technology Areas
 Smart/functional
 Application domains
 Chemicals
 Construction
 Materials
 Sensors
 Medicine
 Aerospace
 Composites
…
 Physical and chemical
properties sensitive to,
and dependent on,
changes in environment
(T, P, E, B, adsorbed gas
molecules, stress, pH…)
 Structural
 Load-bearing materials
 Properties derived from
structure at nano,
micro, meso scales
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
Integrated Computational
Materials Engineering
 SBE&S study in parallel with ICME
 Integration of computational tools to
 Model complex materials systems
 Engineer (design) systems
 Great promise
 3:1 to 9:1 return on investment
 Improved product, quicker
process development
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
22
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Integrated Computational
Materials Engineering
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
SBE&S Requirements
 Software infrastructure
 Community-based (and preferably open source)
 Integration enabled - interoperable
 Databases for storing and querying results
 High-performance simulation at scales ranging quantum
to mesoscale and beyond
 Tools
 High performance computing (to petascale and beyond)
 Visualization
 Human resources
 Appropriately trained students
 Large, focused multidisciplinary teams
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
23
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Asia
 Mitsubishi Chemical
 Uses (largely commercial) computational fluid dynamics




(CFD), finite-element method (FEM) and process modeling
and simulation (PMS) codes individually and in combination
to design and optimize chemical processes and plants
Primarily (90%) uses commercial software (such as Aspen
Tech and gPROMS for process-level modeling, STAR-CD
and FLUENT for CFD)
Codes coupled (via in-house Fortran code)
Management at MCC has become comfortable with relying on
predictions of simulation studies for design and optimization of
chemical and materials manufacturing processes
Supports internal molecular modeling effort on polymers
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
Regional Highlights - Asia
 Toyota Central Research and Development
Laboratories
 Wide range of modeling activities
 Simulations for design of automobiles
 Structural response of Toyota automobile frames to
impacts and wind resistance of auto bodies
 reached maturity level that they are no longer conducted
at TCRDL but at Toyota Motor Company.
 Others:
 Chemistry of combustion in internal combustion engines
 Multi-scale modeling of fuel cells
 Performance of catalysts in oxidizing carbon monoxide
 Predicting viscosity of synthetic lubricants.
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
24
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Asia
 Institute of Chemistry, Chinese Academy of
Sciences (ICCAS)
 Joint Laboratory of Polymer Science and Materials
 Led by Charles Han (formerly of NIST)
 Multi-scale polymer modeling effort
 Funded with 8M RMB (~$1.15M) for four years
 20 investigators
 Inspired by the Doi project in Japan
 Close integration with experiment as…
 Validation tool
 Needed input and data for models
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
Regional Highlights - Europe
 Materials simulation code development in UK
 CCPs at Daresbury Laboratory
 DL_POLY, GAMESS-UK, CASTEP,…
 Funding no longer assured or sufficient to innovate
 CASTEP
 Cambridge Sequential Total Energy Package
 http://www.castep.org
 Primary developer: Mike Payne, Cambridge
 Available free of charge to UK researchers but not to
U.S. researchers
 Available through commercial vendor Accelrys
 ONETEP – linear scaling
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
25
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 Fraunhofer Institute for the Mechanics of Materials
J. Am. Chem. Soc.; 1988; 110(25) pp 8355 – 8359
Evaluation of the rate constant for the SN2 reaction CH3F
+ H- o CH4 + F- in the gas phase
Angela Merkel, Zdenek Havlas, and Rudolf Zahradnik
Contribution from the Central Institute of Physical Chemistry of Academy of
Sciences of GDR, 1199 Berlin-Adlershof, Rudower Chaussee 5, German
Democratic Republic; Institute of Organic Chemistry and Biochemistry,
Czechoslovak Academy of Sciences, Flemingovo num. 2, 16610 Prague 6,
Czechoslovakia; and J. Heyrovsky Institute of Physical Chemistry and
Electrochemistry, Czechoslovak Academy of Sciences, Dolejskova 3,
18223 Prague 8, Czechoslovakia. Received February 19, 1988.
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
Regional Highlights - Europe
Interim storage of
 Daresbury Laboratory, UK
processed nuclear waste
 Energy applications of materials at Sellafield
 Predicting:
 Structure & composition, thermodynamics & phase
stability, reaction kinetics & dynamics, electronic
structure (correlation), dynamics, bridging length and
time scales
 Applications to radiation damage in nuclear
reactors
 Atomistic simulations of resistance to amorphization by
radiation damage
 Simulation of radiation damage in gadolinium
pyrochlores
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
26
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Findings
 Computational materials science and engineering
is changing how new materials are discovered,
developed, and applied
 World-class research in all of the above-mentioned
areas in the U.S., Europe, and Asia
 However….
 Algorithm innovation primarily in US and Europe, some in
Japan
 Almost none in China (primarily applications)
 Rapid ramping-up of activities in some countries
 China
 Extraordinary revival of science in Germany
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
Findings
 Much greater collaboration among
groups in code development in
Europe
 US tenure process & academic rewards
systems mitigate against collaboration
 How much credit is given to junior
collaborators on large code development?
 Development of simulation tools is
not considered high-impact science
 Numerous counter-examples
 Impact often not clear for decade or more
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
27
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
How does the U.S. compare?
 Quality of leading researchers in U.S. is
comparable to that of leading researchers in
Europe and Japan
 Funding models to support ambitious, visionary,
long-term research projects is better in Europe
and Asia than in the U.S.
 EU frameworks
 In U.S., long-term support of any science is eroding
 NWCHEM was in construction budget of EMSL
 NIRT projects last just long enough to get people working
together
 Commercialization is considered success
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
How does the U.S. compare?
 Funding models and infrastructure to
support multi-investigator collaborations
between academia and industry
Have long tradition in Japan
Are rapidly developing in Europe
IP issues with U.S. universities
WTEC/SBES Workshop 15 April 2008
Cummings: Materials Modeling
28
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Computational Life Sciences
and Medicine
Linda Petzold
QuickTime™ and a
decompressor
are needed to see this picture.
Potential Impacts
 Rapidly growing research areas
 Pharmaceutical – drugs to market better,
faster, cheaper (>$1.2 billion to develop new
drug, >90% failure rate, average 14 years)
 Public health
 Understanding of disease mechanisms
 Pharma: targeted drugs
 Computer-assisted surgery
 Biomechanics and physical therapy
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
29
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Research and Technology Areas
 Medical:
 Pharmaceutical:
 Predictive surgery
 Binding of drugs to
 Blood flow simulation
targets
 Identification of disease
mechanisms and
targeting of interventions
 Targeting of drugs to
specific classes of
patients
 Cancer research and
laser surgery
 Biomechanics and
physical therapy
 Life Sciences
 Systems biology
 Structure and function of
the brain
 Biomolecular structure
and function
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
QuickTime™ and a
decompressor
are needed to see this picture.
SBE&S
 Large, focused multidisciplinary teams: iteration between
model and experiment
 Integrated community-wide software infrastructure: SBML,
SBGN, Cell Designer, Copasi, …
 High-performance simulation at scales ranging from
molecular dynamics to PDEs, and at levels of complexity
ranging from Boolean to discrete stochastic and multiscale
Multiscale models
and techniques in
systems biology (D.
Lauffenburger)
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
30
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBE&S
 Data: data provenance, heterogeneous data, analysis




of data, network inference from data
Sensitivity, uncertainty, model invalidation: biodata is
notoriously noisy and imprecise – what can we
conclude or not conclude about the mechanism?
High performance computing: scalable algorithms for
multicore architectures – petascale will enable MD
simulation of macromolecules on millisecond
timescale
Visualization: massive data and relationships,
network behavior of 10,000 neurons
Appropriately trained students
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Japan
 Kitano Lab (Systems Biology). Kitano is widely
regarded as the ‘father of systems biology’. Research
plan focuses on development of experimental and
software infrastructure to accelerate development of
the research field
 Systems Biology Markup Language (SBML) (together
with Caltech)
 Systems Biology Graphical Notation (SBGN)
 Web 2.0 Biology
 Connection with Riken next-generation supercomputer
effort
 Funding model: Funded for 10 years, $2 million per
year, noncompetitive ‘men in black’ grant
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
31
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 EPFL Blue Brain Project (EPFL/IBM, Henry Markram).
Digital 3D replica of the brain, models in full
experimental detail the cellular infrastructure and
electrophysiological interactions within the cerebral
neocortex.
 10,000 neurons of 340 different types; 30 million
connections - orders of magnitude larger and in more
detail than state of the art in U.S.
 Development of annotated database of
experimental results is a major focus
 Impressive visualization
 35 research/development personnel
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Europe
 EPFL (Quarteroni). Multifaceted program,
strong connections to industry.
 Models and methods for local drug delivery from
nano/microstructured materials : controlled drug
delivery with application to drug-eluting stents.
Mechanical analysis, analysis of drug release,
characterization of material properties
 Computational fluid dynamics in the cardiovascular
system. Multiscale, fluid-structure interaction.
“…Europe is currently acknowledged to be the world leader in a
number of aspects of the Virtual Physiological Human, and we
hope that by taking heed of the roadmap, European research
can gain additional momentum to improve this position further.”
(letter from Europhysiome leaders to EU)
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
32
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 Vrije Universitat Amsterdam.
 Systems biology (Westerhoff, Bakker). ‘To cure a
disease, we must cure the network’.
Silicon Cell effort – makes computer replicas of
chemical pathways available on the web for in silico
experimentation
Network-based
drug design
 Brain imaging
(J. C. deMunck)
EEG and fMRI
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Europe
 Center for Biological Sequence Analysis (CBS), TU
Denmark
 One of the largest bioinformatics centers in EU
 Strong teaching component, many courses, some
transmitted real-time over
Internet
 Highly popular suite of WWW
servers and bioinformatics
codes (>2 million visits per
month)
 Strong publications, citations
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
33
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Findings
 The role of computation in life sciences and
medicine is growing very rapidly, and will require a
substantial and open computational software
infrastructure
 We are aware of world-class research in all of the
above-mentioned areas in the U.S., Europe and
Japan, and in some specific areas (increasing
potency of natural medicines) in China
 The scarcity of appropriately trained students in the
U.S., Europe and Japan is perhaps the biggest
bottleneck to progress
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
QuickTime™ and a
decompressor
are needed to see this picture.
How does the U.S. compare?
 Quality of leading researchers in the U.S. is comparable
to that of leading researchers in Europe and Japan
 Infrastructure (access to computing resources and
software professionals) and funding models to support
ambitious, visionary, long-term research projects are
much better in Europe and Japan than in the U.S.
 Funding models and infrastructure to support multiinvestigator collaborations between academia and
industry is much more developed in Europe than in the
U.S.
 Support for the development of community software is
much stronger in Europe and Japan than in the U.S.
WTEC/SBES Workshop 25 April 2008 -- Petzold:
Computational Life Sciences and Medicine
34
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Energy and Sustainability
Masanobu Shinozuka
QuickTime™ and a
decompressor
are needed to see this picture.
Potential Impacts
The Link between Energy and Sustainability
 Energy
 From all sources
 Energy conservation
 A rapidly growing base of consumers
 Sustainability
 Harvesting natural resources for human needs is
sustainable, if t is done while protecting environmental
quality and the resource base for future development. (Body
of Knowledge, ASCE, 2008)
 A system is sustainable under natural, accidental, and
manmade hazards, if it is designed sufficiently resilient
relative to the return period of the extreme events arising
from these hazards
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
35
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBES Requirements
 Appropriately trained students
 Large, focused multidisciplinary teams
 Integrated community-wide software infrastructure
 Data: data provenance, heterogeneous data, analysis of




data, network inference from data
High-performance simulation at scales ranging from
atomic size to mesoscale to large engineering scales –
multiscale
Sensitivity, uncertainty, model invalidation
High performance computing: scalable algorithms for
multicore architectures – petascale will enable simulation
of energy and sustainability systems.
Visualization
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights – Japan
Y. Shumuta, Central Research Institute for Electric Power Industry (CRIEPI),
Chugoku Electric Power Transmission System, Japan
Others 9%
Disaster Response and Restoration
Support System is in place
Wind
‰ Simulation of consequences Pressure
Typhoon
23%
9119
of typhoon scenario similar to
FEMA HAZUS
‰ Estimation of human, property
Falling Trees 26%
and social losses
Flying
Debris
42%
08/09/03 21:00H
08/09/03 21:00H
08/06/03 21:00H
08/06/03 9:00H
08/05/03 21:00H
08/05/03 9:00H
08/04/03 21:00H
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
36
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 Inst. Francais du Petrole (IFP), Malmaison/Paris
 State owned industrial & commercial establishment.
 Extensive research in five strategic priorities:
 Extended reserves
 Clean refining (design of new catalysts)
 Fuel-efficient vehicles
 Diversified fuels (e.g. bio-fuels)
 Controlled CO2
 Budget
 301.5 MEuros (241.3 MEuros for R&D)
 1735 FTEs (65% are in R&D)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Findings
 Very broad and extensive activities in energy and
sustainability exist in all regions
 Energy
 Very high concerns for future energy source for
transportation – substitution not possible in short-term –
driver for sense of urgency in simulation of oil reservoirs,
refinery processes, design of efficient engines.
 Nuclear recognized as major source for electricity – large
investments and rapid progress in SBES-driven materials
design for nuclear waste containment.
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
37
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Findings
 Distribution Networks
 Significant efforts in simulation techniques to evaluate
performance of large-scale and spatially distributed
systems such as power transmission systems that are
subject to highly uncertain natural hazards (earthquake,
hurricanes, flood)
 Alternative Energy
 Energy density lower but tax incentives for green sources
driving extensive efforts in wind energy and tidal energy.
 Connection of alternative sources to the power grid.
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Findings
 Simulation-based engineering is vital for advancing
the physical infrastructure of Energy, Energy
Production and Energy for Transportation
 An opportunity for creating smart infrastructure
 The scarcity of appropriately trained students in the
U.S. (relative to Europe and Asia) is a related
bottleneck to progress.
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
38
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
How does the U.S. compare?
 U.S. is ahead in modeling of large-scale infrastructure
systems, but gap is closing – Asia in particular.
 France is ahead in nuclear energy and related research
(e.g., containment)
 Oil production and fossil-fuel supply chain
 Traditionally led by U.S.
 Significant activity in Europe, including new funding models and
leveraging strengths in development of community codes.
 Sustainability
 Currently, all regions recognize the need to increase emphasis
in this area
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Examples
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
39
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Simulation of Seismic Performance of Power Systems
Taiwan Power’s & LADWP’s Transmission Systems
Karl Liu, National Center for Research on Earthquake Engineering (NCREE), Taipei, Taiwan
M. Shinozuka, University of California, Irvine, USA for Los Angeles Department of Water and Power
(LADWP)
Annual
Occurrence
Probability
Scenario Earthquake
Model
Ground Motion Intensity
Model
Damageability (Fragility) Model
of Vulnerable Components
Model of Power Flow in Damaged
Network (IP FLOW)
Repair and Restoration Model
USGS
Attenuation
Relationship
USGS
Fragility
Curves
Analytical, Empirical
and Experimental
Inventory
Data
Network Damage Model
Vulnerable components:
Transmission towers,
Transmission lines,
Substation equipments,
Power generation plants
Data Source
Analytical and
Field Experiment
IPFLOW
Code
Considered Highly
Reliable
Repair/
Restoration
Experimental and
Empirical
• Loss of connectivity
total supply
total supply
! 1.05 or
1
total demand
total demand
Vintact Vdamaged
• Abnormal voltage
! 0.1 (node by node)
Vintact
WTEC/SBES Workshop 25 April 2008
• Imbalance of power
Failure
Criteria
QuickTime™ and a
decompressor
are needed to see this picture.
Energy
and Sustainability
•Shinozuka:
Frequency
change
IPFLOW does not check
LADWP’s Transmission System
Taiwan Power’s Transmission System
10
0
Probability that losses will exceed L, p(L)
Annual Exceeding Probability
Before Retrofit
After Retrofit
10
10
-1
-2
Loss, L
-3
10
10
Percent Loss in Power Supply
-4
Good
Percent Loss in Power Supply
agreement
between
simulated
Mw =
and observed
7.2
supply of
power 6 hrs
after
earthquake
shows
validity of the
models
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
0
0.2
0.4
0.6
0.8
1
40
Observed
Simulated
1-17-94
10:00 AM
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Union for the Co-ordination of Transmission of Electricity
The “Union for the Co-ordination of Transmission of Electricity” (UCTE) is the
association of transmission system operators in continental Europe, providing a reliable
market base by efficient and secure electric “power highways”. 50 years of joint activities
laid the basis for a leading position in the world which the UCTE holds in terms of the
quality of synchronous operation of interconnected power systems. Their common
objective is to guarantee the security of operation of the interconnected power system.
Through the networks of the UCTE, about 450 million people are supplied with electric
energy; annual electricity consumption totals approx. 2300 TWh (www.ucte.org)
UCTE collected past
sequences of black-out
events that can be archived
to develop scenario of
black-out events for future
system risk assessment
“Critical Infrastructures at Risk – Securing the European Electric Power System” by A.V.
Gheorghe, M. Masera, M. Weijnen and L. De Vries, Published by Springer, 2006 will provide
good reading on this subject
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Wind
Direction
1. Identification of the optimum locations of wind
towers within a wind park using Monte Carlo
Simulation. (Greece, Marmidis et al. 2008, Renewable
Energy, V. 33, 1455 - 1460)
2. Development of the analytical model
to simulate the real-world wind power
generating system (France, Ph. Delarue et al.
2003, Renewable Energy, V. 28, 1169 - 1185)
2 km
Wind Power Related Simulations
2 km
wind energy conversion system
3. Identification of the most efficient wind power generator by altering
its configuration (such as blade type, radius, # of blades, rotational
speed of rotor) (Colombia, Mejia et al. 2006, Renewable Energy, V. 31, 383 - 399)
4. Calculation of efficiency of
wind power systems connected
to utility grids (Egypt, Tamaly and
Mohammed, 2004, IEEE,
http://ieeexplore.ieee.org/iel5/9457/30014/0
1374624.pdf?arnumber=1374624)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
41
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Solar Power Related Simulations
‰ Simulation of off-grid generation options for
remote villages in Cameroon (Nfah et al. 2008,
Renewable Energy, V. 33, 1064 - 1072)
Map of electricity generation and distribution networks in Cameroon in 2002
‰ Numerical simulation of the solar chimney
power plant systems coupled with turbine in
China (Tingzhen et al. 2008, Renewable Energy, V. 33, 897 905)
‰ Use of TRNSYS for modeling and
simulation of a hybrid PV–thermal solar
system for Cyprus (Soteris A. Kalogirou, 2001,
Schematic drawing of a solar chimney
Renewable Energy, V. 23, 247 - 260)
‰ Two-Dimensional Numerical
Hybrid system construction details
Simulations of High Efficiency Silicon
Solar Cells in Australia (Heiser et al. 1993, Simulation
of Semiconductor Devices and Processes, V. 5, 389 - 392)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
The UNSW PERL Si solar cell
QuickTime™ and a
decompressor
are needed to see this picture.
Sustainability of a Sea Port under Earthquake
U. Na, Pusan Port Authority, Korea and M. Shinozuka, University of California, Irvine, USA
Kobe Port subjected to Kobe Earthquake (Mw = 6.8)
‰ Submerged pier
‰ Deformed outwards ‰ Liquefaction
3.41m
5.33 m
PL 12 berth
PL13 berth
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
42
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Digital Simulation of Stochastic Field (Muti-Dimensional and
Multi-Variate) by Spectral Representation Method
16m
Liquefiable Backfill
16m
Bedrock
80 m
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Time histories of horizontal displacement for 130 cases
of backfill soil property realizations
Horizontal displacement time histories
1
Mean
Mean+/-SD
Horizontal displacement (m)
0
-1
-2
-3
-4
-5
-6
0
5
10
15
20
25
30
35
Time (sec)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
43
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Decreased TEU
(based on Simulation)
Actual throughput 19.620E5 (TEU)
Container traffic
Restoration Timeframes
In Kobe
TEU = Twenty-foot Equivalent Unit
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Simulation of Tsunamis and Their Consequences
Shunichi Koshimura, Disaster Control Research Center, Tohoku University, Japan
The 2004 Indian Ocean Tsunami
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
44
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Model Validation with Altimetry Data
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
45
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Simulation of Pollutant Dispersion in Population Center
Institute of industrial Science, University of TOKYO, Japan
Field and Wind Tunnel Experiments
Dispersion of pollutants emitted by vehicles particularly in traffic congestion in a dense urban area
must be traced to alert and possibly controlled to protect residents from its hazardous effect. The
research is to develop an analytical model of dispersion and simulate the fate of the pollutants. An
ideal study was performed to verify and calibrate numerical simulation by University of Tokyo
researchers in which field and wind tunnel tests of pollutant dispersion in a built-up urban center
under various meteorological conditions.
M.F. Yassina, S. Kato, R. Ooka, T. Takahashi, R. Kouno, “Field and wind-tunnel study of pollutant
dispersion in a built-up area under various meteorological conditions”, Journal of Wind Engineering
WTEC/SBES Workshop
25 AprilAerodynamics,
2008
and Industrial
93 (2005) pp. 361–382
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Simulation of Pollutant Dispersion under Wind
M. Zako, Osaka University, Osaka, Japan
Diffusion of gas
Wind
direction
Leakage
- Flash evaporation
- Evaporation on
ground surface
Outflow into dike
Wind
velocity
t1
t2
Time(sec.)
3.0m
3.0m
min.
11 min
3.0m
2 min.
2 min
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
46
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Seismic Performance of Highway Network
Veneto Region’s and Caltrans’ Highway Transportation Networks
M. Shinozuka, University of California, Irvine, USA
C. Modena and P. Franchetti, Veneto Region Highway Network
Scenario Earthquake Model
Ground Motion Intensity Model
Annual
Occurrence
Probability
Attenuation
Relationship
PGA,PGV
ShakeMap
Fragility
Curves
Bridge Damageability Model (not Retrofitted)
Bridge Damageability Model (Retrofitted)*
http://www.trinet.org/
shake/index.html
Model for Remaining Traffic Capacity of Damaged Highway Network
*Retrofitted by
steel jacketing of
columns
Traffic Flow Analysis
OD (OriginDestination)
Data
Drivers’ Delay & Opportunity Loss
Economic Analysis & Benefit Cost Ratio
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Highway Network in Veneto Region, Italy (April 2008)
Researchers at Padova University is working on the
problem of optimal retrofit under a budget constraint.
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
47
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Caltrans’ Highway Network
PGA Distribution & Bridge Damage State (1)
- Elysian Park M7.1 (Before Retrofit)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Caltrans’ Highway Network
PGA Distribution & Bridge Damage State (2)
- Elysian Park M7.1 (After Retrofit)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
48
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Effect of Bridge Retrofit on System Restoration Curve
140
No Retrof it
120
Social cost in hour * average
hourly wages (~$21.77/h for
Los Angeles in 2005 DOL)
100% Retrof it
100
80
60
40
20
0
0
50
100
150
200
250
300
1
350
Tim e a fte r Ea rthqua ke (da ys)
Annual Freqency of Exceedance
Daily Drivers' Delay and Opportunity Cost
(million dollars)
Social Cost Avoided = Shaded Area
160
(Shinozuka’s Bridge Restoration Model,
Low Residual Link Capacity Case)
Mean
0.1
mean V
0.01
0.001
mean V
0.0001
0
2
4
6
8
10
12
14
16
18
Social Cost ($ billion)
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
QuickTime™ and a
decompressor
are needed to see this picture.
Cost-effectiveness
Discount Rate
Benefit/Cost
Ratio
CostEffectiveness
3%
4.39
Yes
5%
3.12
Yes
7%
2.36
Moderate
Benefit-Cost
Definitions Depend
on Stakeholders
Total Benefit/Cost Ratio
= [social and repair cost
avoided]/[Retrofit Cost] = 4.39
* Evaluation is based on
discount rate= 3%; Low link residual capacity;
remaining life of retrofitted bridges T =50 years.
R=Benefit/Cost Ratio
No: R<1.5 Moderate: 1.5 <=R<2.5
WTEC/SBES Workshop 25 April 2008
Shinozuka: Energy and Sustainability
Yes:R>=2.5
QuickTime™ and a
decompressor
are needed to see this picture.
49
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Education and Training
Celeste Sagui
Department of Physics
North Carolina State University
QuickTime™ and a
decompressor
are needed to see this picture.
Why is the education of SBE&S important?
 Successful computer simulations in most diverse fields is one of most
important science & engineering developments in the last 30 years.

Proper teaching of computer modeling and simulation methods and
tools in colleges and universities becomes of paramount importance if
the United States is going to remain competitive in the Sciences and
Engineering Sciences.
 “Our educational institutions must be prepared to equip
tomorrow’s scientists and engineers with the foundations of
modeling and simulation and the intellectual tools and
background to compete in a world where simulation and the
powerful predictive power and insight it can provide will be a
cornerstone to many of the breakthroughs in the future.”
(Tinsley Oden, US baseline workshop)
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
50
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Problems that the US is facing
 Less funding allocation to education: Funding limitations and less industrial
involvement affect resources allocated to education.
 Loss of international students: visa constraints and loss of attraction by US.
 Loss of PhD students and Faculty: The scientific field is no longer attractive
to US students; Faculty and PIs in national labs are being lost to industry and other
markets.
 Lack of adequate training of students in SBE&S: students knowledgeable
in running existing codes (with visualization) but unable to actually write code or
formulate a mathematical framework.
 Rigid “silo” structure: due to budget, courses, curricula, promotion and tenure,
etc. The system promotes departmental loyalty and highly discourages
interdisciplinary work.
 Serious danger of compromising creativity and innovation: In present
funding environment, many alliances are made just for the purpose of
seeking funding; independently of real scientific value of research.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
US used to lead the way in SBE&S education
 Certificate programs in CSE: These were among the first in the
world for SBE&S. However they haven’t embraced new developments
in the last decade.
 Many new efforts throughout the country to fill this vacuum:
especially in the form of Summer programs: e.g, schools associated
with main national codes such as AMBER, CHARMM, NAMD; efforts
under NSF and DOE grants, the Ralph Regula School of CS in Ohio
(K-12, undergrads), the GLCPC (Great Lakes Consortium for Petascale
Computation) Virtual School of CSE, etc.
 Some US institutions are being restructured: For instance the
Institute for Computational Engineering and Sciences at the University
of Texas, Austin, offers both education and infrastructure for
interdisciplinary programs in CSE and IT. Faculty from 17 departments
and 4 schools and colleges. Independent PhD program, with
independent Faculty evaluations (promotion & tenure).
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
51
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
First-time graduate enrollment in S&E, by citizenship
and field:2002-06
Solid : US citizens & permanent residents
Dotted: Visa holders
Source: NSF
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
 Percentage of undergrads with S&E degree: South Korea, 38%;
France, 47%; China, 50%; Singapore, 67%. US, 15%
 PhDs awarded to foreign born students: 34% in natural sciences
and 56% in engineering.
 38% of US Science & Technology workforce is foreign-born.
 More S&P 500 CEOs obtained their undergrads in engineering
than in any other field.
Source:
Rising Above The Gathering Storm: Energizing and
Employing America for a Brighter Economic Future
Committee on Prospering in the Global Economy of the21st Century:
An Agenda for American Science andTechnology, National Academy of
Sciences, National Academy of Engineering, Institute of Medicine, 2007
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
52
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
S&E PhDs by region/country: 1989 and 2003
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
How do the other countries compare?
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
53
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: Increasing Asian leadership due to funding
allocation and industrial participation in education
 Japan Earth Simulation Center: focused on developing new algorithms, specially
multiscale and multiphysics.
 Systems Biology Institute (Japan): funded by Japanese government for 10 years.




Software infrastructure: Systems Biology Markup Language (SBML), Systems
Biology Graphical Notation (SBGN), CellDesigner, and Web 2.0 Biology. Difficult to
publish software, the merit system in this lab values software contributions as well as
publications.
University of Tokyo: “21st Century Center of Excellence (COE) Program” 28
worldclass research and education center in Japanese Universities o Global COE
Institute of Process Engineering (P.R. China): 50% of research funding comes
from industry (domestic and international; significant funding from the petrochemical industry). Significant government funding through the National Natural
Science Foundation of China and the Ministry of Science and Technology (main
focus: multiscale simulations for multiphase reactors ).
Tsinghua University Department of Engineering Mechanics: Strong
interaction of R&D centers with industry and multinational companies.
Fudan University, Shanghai: strong emphasis on education, first analytical work
then computational. Prof. Yang is director of leading computational polymer physics
group and Vice Minister of Education; has allocated funding for SBE&S and for 2000
students/year to study abroad.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Increasing EU leadership due to funding
allocation and industrial participation in education
 Center for Biological Sequence Analysis (Bio-Centrum-DTU, Denmark): Danish
Research Foundation, the Danish Center for Scientific Computing, the Villum Kann
Rasmussen Foundation and the Novo Nordisk Foundation (US$100M), other
institutions in European Union, industry and the American NIH (bioinformatics,
systems biology).
 CIMNE –– International Center for Numerical Methods in Engineering
(Barcelona, Spain): independent research center, now as a consortium between
Polytechnic University of Catalonia, the government of Catalonia, and the federal
government; annual funding 10M€ from external sources, focused on SBE&S
research, training activities and technology transfer.
 Germany: German research foundation (DFG) has provided support for collaborative
research centers (SBF), transregion projects (TR), transfer units (TBF), research units
(FOR), Priority programs, and “Excellence Initiatives”. Many of these are based on
or have major components in SBE&S (Stuttgart, Karlsruhe, Munich) and strong
connections with industry.
 Fraunhofer Institute for the Mechanics of Materials (Germany): 15.5M€/year,
44% from industry and 25-30% from government. Significant growth recently (10%
per year). Fully 50% of funding goes to SBE&S (up from 30% 5 years ago) (applied
materials modeling), 50,000 euro projects awarded to PhDs to work in the institute in
topic of their choice.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
54
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: New centers and programs for education
and training in SBES ––all of interdisciplinary nature
 CBS (BioCentrum-DTU): MSc in Systems Biology and in Bioinformatics loosely
structured, not linked to any department in particular. Real-time internet training (all
lectures, exercises and exams), with typically 50:50 students onsite:internet.
International exchange highly encouraged, students can take their salary and move
anywhere in the globe for half a year.
 CIMNE (Barcelona): main especiality is courses and seminars on the theory and
application of numerical methods in engineering. In last 20 years, CIMNE has
organized 100 courses, 300 seminars, 80 national and international conferences,
published 101 books, 15 educational software + 100s of research and technical reports
and journal papers.
 ETH Zurich: pioneering CSE program (MSc and BSc) combining several
departments, successful with grads and postdocs taking the senior level course.
 Institut für Informatik and Leibnitz Supercomputing Center (Technical
University of Munich): Many CSE programs (i) BGCE, a Bavaria-wide MSc honors
program; (ii) IGSSE postgraduate school; (iii) Center for Simulation Technology in
Engineering; (iv) Centre for Computational and Visual Data exploration; (v)
International CSE Master program multidisciplinary involving 7 departments; also
allows for industrial internship; (iv) Software project promotes development of
software for HPC/CSE as an educational goal; (v) many, many other programs with
other universities and industry.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Pitfall of interdisciplinary education:
breadth vs depth
 Educational breadth comes at the expense of educational depth. e.g., in ETH Zurich
the CSE faculty choose physics students when dealing with research issues and CS
majors for software development. CSE students can spend too much time on the
“format” of the program, without really thinking the underlying science beneath.
Finding: Demand exceeds supply: academia vs
industry
 Huge demand for qualified SBE&S students who get hired immediately after MSc,
don’t go into PhDs. Good to maintain a dynamical market force but academia would
like to see more students that continue a tradition of “free” research.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
55
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: EU Centers are attracting more international
students from all over the world (including US)
 CBS (BioCentrum-DTU): The internet courses are used to attract international
students (cost 20% more effort but bring lots of money, always oversubscribed).
 CIMNE (Barcelona): (i) introduced an international course for masters in





computational mechanics for non-European students. This is 1st year with 30 students.
Four universities involved in this course (Barcelona, Stuttgart, Swansea and Nantes).
(ii) Web environment for distance learning, also hosting a Master Course in
Numerical methods in Engineering and other postgraduate courses. (iii) the
“classrooms”: physical spaces for cooperation in education, research and technology
located in Barcelona, Spain, Mexico, Argentina, Colombia, Cuba, Chile, Brazil,
Venezuela and Iran.
ETH Zurich: number of international students has increased dramatically (Asian,
Russian).
Vrije University Amsterdam: 50% graduate students come from outside the
Netherlands (mainly Eastern Europe).
LRZ in TUM Munich: 80% SBE&S students in MSc programs come from abroad:
Near East, Asia, Eastern Europe, Central and South America.
Also Japan: International Center for Young Scientists (Comp. Mat. Science Center &
Nat. Inst. Mat. Sc.); English, interdisciplinary, independent research, high salary,
research grant support (5M yen/year).
Spain, Germany and Italy among others are capturing more and more of the latin
american student market, which has shifted its traditional preference for the US in favor of
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
Europe.
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Difficulty in finding students and postdocs
qualified in algorithmic and program development
 Most students in US and outside are trained primarily to run existing codes to
solve applied problems rather than learning the skills necessary for developing
new codes.
 Almost universal complaint: RICS (Japan); Vrije Universiteit Amsterdam,
Theory of Condensed Matter group (Cambridge University); IBM Zurich;
University College London; CMSC and NIMS (Japan); etc.
Still, algorithm developments are centered in Europe
 Problem: the expectation of US funding agencies and university administrators
for “sound-bite” science published in high profile journals. Software engineering
is incremental, and these expectations undermine it.
 Code development via collaboration is widespread and well-funded in Europe. In
contrast, in the US the pressure of tenure and promotion actively discourages
research until a faculty member reaches a senior level, by which time he has to
consider the future careers of students/post-docs, again undermining software
development.
 Still strong US atomistic biomolecular codes: AMBER, CHARMM, GAUSSIAN
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
56
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Case study: University of Stuttgart
Institute of Thermodynamics and Thermal Process Engineering

Departments: Civil and Environmental Engineering; Chemistry; Energy technology,
Process Engineering and Biological Engineering; Computer Science, Electrical
Engineering and Information Technology; Aerospace Engineering and Geodesy;
Engineering Design, Production Engineering and Automotive Engineering; Mathematics
and Physics.
 Key research areas: modeling and simulation, complex systems, communications,
materials, technology concepts and assessment, energy and the environment, mobility,
construction and living, integrated product and product design.
 Collaborations: inter-departmental research structure, with the different faculties linked
through transfer and research centers interacting with national (Max Planck Institutes,
Fraunhofer Institutes, German Aerospace Center) and international institutions, and with
industry (the university sits in one of Europe’s strongest economic regions: BOSCH,
FISCHER, DAIMLER, HP, IBM, FESTO, Pilz Deutschland, PORSCHE, TRUMPF,
STIHL, ZUBLIN, BASF).
 Rankings and Funding: number 3 in Germany in total amount of external funding;
number 1 in Germany in external funding per professor (average of 400,000 € per
professor); one of top grant university recipients in DFG’s grant ranking (2006); in the
top 3 for a range of Engineering study programs (ranked by CHE, Spiegel, Focus
2007); 5th place in the CHE 2006 research ranking; most successful German
University in the 6th EU Framework Program, particularly in the fields of simulation
technology, energy, and e-health; etc.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
University of Stuttgart: Funding
 DFG Collaborative Research Centers (SBF) and Transregions (TR): The budget
of these centers is about 2M –3M €/year for 12 years, in 3 four-year terms. Dynamic
Simulation of Systems with Large Particle Numbers; Control of Quantum Correlations in Tailored
Matter (Transregion involving Stuttgart/Tübingen/Ulm), etc..
 DFG Transfer Units (TFB): Execute transfer from university to industry. Simulation
and Active Control of Hydroacoustics in Flexible Piping Systems; Transformability in Multi-variant
Serial Production; Rapid Prototyping; Computer Aided Modelling and Simulation for Analysis,
Synthesis and Operation in Process Engineering, etc. Execute transfer from university to industry.
 DFG Research Units (FOR): Smaller than the centers (4-5 researchers). Development
of Concepts and Methods for the Determination of Reliability of Mechatronics Systems in Early Stages
of Development; Noise Generation in Turbulent Flow; Multiscale Methods in Computational
Mechanics; Positioning of Single Nanostructures –Single Quantum Devices, etc.
 DFG Priority programs: These programs bring 1.2-2M €/year (which are spread
over 10-20 projects). Ten new priority programs are opened per year. Molecular
Modeling in Chemical Process Engineering; Nanowires and Nanotubes: from Controled Synthesis to
function, etc.
 Excellence Initiative: To foster excellence in science and research, and to raise the
profile of top performers in the academic and research community by means of three
lines of funding: strategies for the future; excellence clusters and graduate schools.
The Universität Stuttgart has been successful in two of the Excellence Initiative’s
three lines of funding.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
57
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Stuttgart: Simulation Technology Excellence Cluster
 The SimTech Excellence Cluster is coordinated by Dr.-Ing. Wolfgang Ehlers, and the
initiative covers topics from isolated numerical approaches to an integrative systems
science. Interestingly, in order to obtain funding, the group gave arguments partly
based on the US NSF Blue Ribbon Panel Report in February 2006 “… challenges
in SBES … involve …multi-scale and multi-physics modelling, real-time
integration of simulation methods with measurement systems, model validation and
verification, handling large data, and visualisation.” “… one of those challenges is
education of the next generation engineers and scientists in the theory and
practices of SBES.” DFG has agreed wholeheartedly with this report and provided
funding accordingly. SimTech Excellence cluster brings 7M€/year for 5 years. This
plus other sources of funding allow for long-term sustained agenda.
 New positions are being created: (i) three new professorial positions on
Mathematical Systems Theory, Modelling of Uncertain Systems, and Human-System
Interaction and Cognitive Systems; (ii) 7 post-doctoral positions and (iii) 13 new
junior professorships with up to two research associates each, tenure-track options for
4 of the junior professors and a total of 72 scientific projects.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
Stuttgart: Long-Term Sustainability
 Three new structural elements elements have been founded at the university: Research
Centres, the Transfer Centre and the Stuttgart School of Science and Technology.
Compared to the traditional departments with their teaching-oriented "vertical"
structure, the Research Centres are "horizontally" oriented, thus comprising
researchers and their institutions from various departments under the common roof of
a research goal.
 Stuttgart Research Centre of Simulation Technology: opened in April 2007, first
one at the university and represents both a scientific research unit and a new structural
element acting as a Research Department with its own organizational and
administrational structure including financial resources (240,000 EUR/year) and
personnel.
 The SimTech Transfer Unit: bundles all activities of the cluster that require uni- or
bidirectional communication with internal and external institutions and industy. It will
be embedded in the Stuttgart Transfer Centre, whose role is to transfer research
results into application, bundle exchange activities with industrial partners, and
provide a basis for all future fundraising activities of individual research centres.
 Graduate School of Simulation Technology : To promote common research
interests, part of the Stuttgart School of Science and Technology.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
58
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Case Study: Stuttgart & Education
 BSc/MSc Elite Study Program
 BSc & MSc theses in different SimTech Research Areas
 one term abroad during MSc studies is required
 flexible study regulation (more than 1 supervisor, and at least 1 from
abroad)
 e-learning, tele-teaching
 Graduate School in Simulation Technology
 Stuttgart School of Science and Technology
 Concept similar to DFG programs (ENWAT, NUPUS)
 international exchange program
 optional: short course, summer schools, software skills program
 joint internal/external and international supervision
 no more separation in Faculties (departments), new interdisciplinary
flexibility
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
QuickTime™ and a
decompressor
are needed to see this picture.
Conclusions

Increasing Asian and especially EU leadership in SBE&S education due to increased
funding and industrial participation.

New EU centers and programs for education and training in SBE&S ––all of
interdisciplinary nature. Sometimes with complete restructuring of University.

In US and abroad it’s difficult to find students and postdocs qualified in the
development of algorithms and programs (at any level).

In spite of this, EU is the center of algorithms development due to well-funded
collaboration efforts; students thrive in these environments.

Huge demand in EU for qualified SBE&S students who get hired immediately after
MSc by industry/finance: collaboration & competition between industry & academia

European Centers are attracting more international students from all over the world
(including the US); special programs for international students in EU and Japan.
WTEC/SBES Workshop 25 April 2008 -- Sagui: Education & Training
59
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Next-Generation
Architectures and
Algorithms
George Em Karniadakis
QuickTime™ and a
decompressor
are needed to see this picture.
(Oden Report)
•Increase in “effective” Gflops
due to new algorithms versus
Moore’s law
D. Keyes et al, A Science-Based
Case for Large-Scale Simulation,
DOE Report (2004)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
60
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Recent Gordon-Bell Awards:
Diverse Application Areas – Common Solvers
 8 x Partial differential equations, deterministic
 Climate, fluids, seismology, structures
 2 x Other mesh-based, with Monte Carlo aspect
 Boltzmann, quantum chromodynamics
 4 x N-body dynamics
 gravitation
 3 x Molecular dynamics
 electronic structure, magnetism, solidification
 1 x Integral equations
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
61
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
On the Horizon - Petaflop Era: 2008-
• LANL’s RoadRunner
 1 PTf/s in 2008
13,000 Cell HPC chips
x 1.33 PetaFlop/s (from Cell)
7,000 dual-core Opterons
• Japanese’s Life Simulator
 10 PTf/s 2011 2010
• ANL
 IBM BG/P early 2009, 1 PTf/s
Up to 884,736 cores
• ORNL/UTK
 Cray XT late 2008/early 2009, 1 PTf/s
>80,000 cores
• NCSA
• NCSA: Blue Waters 10PTF/s; 2011; > 200,000
Power7 multicores
5
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Japanese government (MEXT) is committed
to sustain supercomputing funding
National
Leadership
Sustained Performance (FLOPS)
100P
1P
10T
Government
Investment
National
National
Leadership
Next-next- Infrastructure
National
Institute,
next
Leadership
University
Generation
（The Next Generation Supercomputer）
Project
The Next Generation
Supercomputer Project
National
Next-next
Generation Infrastructure
Institute,
National
Project
University
Leadership
National
(Earth Simulator)
Earth Simulator
Infrastructure
Project
Institute,
Enterprise
National
University
Company,
Leadership
Laboratory
CP-PACS,
Enterprise
NWT
Company,
National
Infrastructure
Institute
University
100G
National
Infrastructure
Institute,
University
1990
Enterprise
Company,
Laboratory
2000
100P
1P
10T
Laboratory
Enterprise
Company,
Laboratory
Personal
Personal
Entertainment
Personal
Entertainment
PC, Home Server
Entertainment
PC, Home Server
Workstation,
PC, Home Server
Workstation,
Game Machine, Digital
Workstation,
Game Machine, Digital
TV
Game Machine, Digital
TV
TV
2010
2015
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
62
2020
100G
2025
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
•Finding: New large multi-core systems present another
disruptive technology – a challenge greater than
cluster computing and message passing
•Rethink and rewrite the applications, algorithms, and software
•Numerical libraries will change, e.g. both LAPACK and ScaLAPACK
will undergo major changes to accommodate multicore/multithread
computing
•
Example – (Dongarra)
All Large Core
•
•
•
•
•
128 cores per socket
32 sockets per node
128 nodes per system
System = 128*32*128 = 524,288
cores!
4 threads of exec per core
Mixed Large
and
Small Core
Many Small Cores
All Small Core
Many FloatingPoint Cores
• Total: 2M threads to
+
3D Stacked
Memory
SRAM
manage
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
63
7
QuickTime™ and a
decompressor
are needed to see this picture.
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
QuickTime™ and a
decompressor
are needed to see this picture.
University of Zurich: Faster MD by splitting the atom
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
64
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: Many orders-of-magnitude speed-up will come from
simultaneous advances in linear solvers, spatio-temporal
discretization, adaptivity, domain decomposition algorithms,
and computer speed-up
Example: The Jardin-Keyes roadmap for ITER (Magnetic Fusion Energy)
•Factor of 1.5X
•
1.5X
•
4X
•
1X
•
1X
•
3X
-- increased parallelism
-- greater processor speed
-- adaptive gridding
-- high-order elements
-- field-line coordinates
-- implicit time-stepping
•TOTAL 12X – Over a 10-year period!
Finding: Dynamic Adaptivity & Implicit Treatment yield 7X but require:
- new effective & scalable preconditioners and new
- new domain-decomposition/graph partitioning methods
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: New scalable multi-level effective preconditioners
are required for 100,000s cores
•Typically, effectiveness of PC and scalability
are in conflict.
•Geometric MG is not scalable but
algebraic MG with “clever” coarsening is.
•Weak scaling on BG/L of AMG
using the default and a new aggressive
coarsening algorithm (C-new) from
15.6K DOFs to 2.05B DOFs.
(U.M. Yang – LNNL/D. Keyes)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
65
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: US/DOE-Labs lead in both computer architectures
(multicores, interconnect) & applied algorithms (BerkeleyBarcelona-Beijing) but aggressive initiatives overseas may
undermine this position
e.g., Japanese N-N-N Generation program; DEISA/PRACE European FP,
BlueGene/P built in Germany, Deep Computing (Zurich); Blue Brain (EPFL),
O(N) algorithms in Europe,…
Finding: Investments in theoretical algorithm development
are lagging behind of hardware funding
 Create families of algorithms that adapt to memory and concurrency;
develop O(N) scalable (controlled accuracy) algorithms
 Look at non-traditional architectures GPGPUs, Low Power, Routers,
FPGAs (e.g., NCSA activities)
 Different operation balance, much greater concurrency, emphasize
asynchronous operations
 Reconsider problem decompositions (W. Gropp, UIUC)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Next-Generation Architectures and Algorithms
66
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Multiscale Modeling
Peter T. Cummings
Chemical Engineering, Vanderbilt University
Center for Nanophase Materials Sciences, Oak Ridge
National Laboratory
SBE&S Report-Out to Sponsors
April 25, 2008
Potential Impacts
Astrophysics
 Multiscale modeling is pervasive concern in
SBE&S
Materials,
Chemistry,
Biochemistry
Global
Population
models
(ecology,
Individual
medicine)
(biology,
Finite element/
medicine) macroscale
Mesoscale MD
DPD
United-atom MD
Explicit-atom MD
ab initio
Mechanics
(structural,
fluid)
HighEnergy
Physics
WTEC/SBES Workshop 15 April 2008
4/14/08
10:08Modeling
AM
Cummings:
Multiscale
67
2
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Multiscale in Materials
Finite element/
macroscale
methods
Mesoscale MD
DPD
United-atom MD
Explicit-atom MD
Ab initio (QM)
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
Multiscale in Biology
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
68
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Potential Impacts
 True multiscale modeling….
 Automatic upscaling (hard) and downscaling
(harder)
 General (e.g., homogenization theory, equationfree method)
 Orders-of-magnitude time saving
 Revolutionary
 ….remains holy grail of SBE&S
 “Full physics” simulations
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
Research and Technology Areas
 Engineering design
 Materials modeling
 Crack propagation
 Polymers
and operations
 Smart manufacturing
 “In the future, smart
 Biology
plants will be
developed, designed,
and operated using
molecularly informed
engineering.”
 Design and
operations from
global supply chain to
molecule and vice
versa
 Energy
 Global climate
 Astrophysics
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
69
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
SBES Requirements
 Multiscale modeling
 Cross-cutting enabling capability
 SBE&S needs
 Standards for interoperability of codes (e.g.,
CAPE-OPEN)
 New methodologies and validation/application of
existing techniques
 Multidisciplinary research teams
 Development of MM frameworks
 Training of students in use of MM frameworks
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
Regional Highlights - Japan
 Mitsubishi Chemical
Polymer modeling (beyond Doi project)
Multiscale model from atoms to CFD based on
SC-PRISM
Mitsubishi
Chemical USA,
Head Office
Mitsubishi Chemical
(U.K.) PLC
Mitsubishi Chemical
Europe GmbH
Mitsubishi Chemical
(Thailand)Co., Ltd.
QuickTime™ and a
decompressor
are needed to see this picture.
Mitsubishi Chemical Singapore Pte Ltd.
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
70
Mitsubishi
Chemical USA,
Virginia Office
Dalian University of
Technology
Mitsubishi Chemical
Hong Kong Ltd.
Kyoto
University
AIST
Tokyo Institute of
Technology
UC Santa Barbara,
Mitsubishi Chemical - Center for
Advanced Materials
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 Mike Payne, Cambridge
Crack propagation in graphene sheets
Example of multiscale in specific problem
domain
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
Findings
 Multiscale modeling is exceptionally important
 Examples exist within narrow disciplinary
boundaries
 E.g., crack propagation within materials
 Industry attempts to do this because they
must
 Lack of standards-based interoperability of codes
is major impediment
 Cited by several companies
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
71
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
How does the U.S. compare?
 U.S. research is on par with Japan and Europe
 However, U.S. research is distributed, lacks focus and integration
 Japanese and European approach is to fund large interdisciplinary
teams
 Japan: Large industrial component (Doi project)
 Petascale and exascale computing are needed to validate
multiscale approaches
 Can anyone get petascale resources for validation of MM
approach?
 Tradition of interdisciplinary collaboration leading to
community software is much stronger in Europe and
Japan than in the U.S.
 MM may be solved & MM frameworks developed, outside the U.S.
WTEC/SBES Workshop 15 April 2008
Cummings: Multiscale Modeling
72
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Simulation Software
Martin Head-Gordon
Dept. of Chemistry, University of California,
Chemical Sciences, Lawrence Berkeley Lab,
Berkeley, CA.
QuickTime™ and a
decompressor
are needed to see this picture.
Simulation software
is central to SBE&S
 Tools for development & research
 Community resources
 Knowledge creation
 Training next generation developers
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
73
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
(Just a few) American
successes
 Computational quantum chemistry
 104-105 users: 1998 Chemistry Nobel Prize
(Pople, Kohn)
 Process design
 AspenTech -- 1 of 9 “Heroes of American
manufacturing” -- Fortune Magazine, 1999
 Numerical linear algebra
John A. Pople, 1998
 LAPack -- backbone of scientific computing
 Biomolecular dynamics simulations
 American-originated (eg. CHARMM, Amber)
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Oden report: new vistas in
simulation software
 Much current software is inadequate for dealing
with multifaceted SBE&S applications
 New tools are needed so that software is more
transferable between fields and not wastefully
duplicated.
 Experienced software developers should work
closely with engineering scientists to develop
tomorrow’s SBE&S software.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
74
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
How does the US compare?
 20 years ago, the US had a long lead in nearly all areas
 In many disciplines, the US is still leading.
 Standard numerical libraries, system software
 Ease of commercialization. Co-operation with industry
 US advantage has eroded:
 Europe leads in some fields. Better long-term support for
community code development.
 Japan challenging in HPC applications software
 China largely using existing tools for the moment.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Case 1: Condensed matter
electronic structure
Europe dominates
CASTEP (commercial plane-wave DFT: UK)
ONETEP (commercial local orbital DFT: UK)
VASP (commercial plane-wave DFT: Austria)
CPMD (plane-wave DFT dynamics: Switzerland)
Quickstep (local orbital dynamics: Switzerland)
Siesta (local orbital dynamics: Spain)
CASINO (quantum Monte Carlo: UK)
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
75
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Resulting access limitations
 US Air Force Labs cannot purchase the
Siesta code, due to de-facto policy on not
distributing to military organizations.
US research becomes dependent on foreign codes.
 In general, erosion of support for methods
and code development weakens our
competitive position in SBE&S.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Case 2: Quantum chemistry
 Commercial codes:
 Gaussian (US)
 Spartan/Q-Chem (US)
 Jaguar (US)
 DMol (Switzerland)
 Turbomole (Germany)
 ADF (Netherlands)
 MolPro (Germany/UK)
 MolCas (Sweden)
 Crystal (Italy)
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
76
 Public domain codes:
 GAMESS (US)
 NWChem (US)
 Psi3 (US)
 MPQC (US)
 Dalton (EU)
 GAMESS/UK (UK)
 UTChem (Japan)
 DeMon (Canada/Mx)
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Case 3: HPC software; Japan
 RIKEN-led program for petascale supercomputing
 7 applications targeted for > 1 petaflop (10% of peak)
 LAPack targeted for 10 petaflop performance
 Limited work on system tools (compilers, languages)
 Revolutionary simulation software: RSS21 (Tokyo)
 Multi-institution (120 researchers). $12 million/year
 Biosimulations: atoms - tissues / Multiscale nanoscience
 Middleware / optimization
RSS21: nano-worm-gear
 21st Century Center of Excellence: Mechanical Systems Innovation
 “Hypermodeling”: multiscale / multiphysics
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Finding 1: Academia’s role
 State-of-the-art simulation software development
relies on a healthy university and national lab
infrastructure to obtain new advances.
 More accurate and realistic simulation models
 Algorithms
 Software engineering: languages, libraries
 Direct interactions with industry if appropriate
 New graduates to make the new developments
 Widely recognized by relevant industries
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
77
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding 2: Life cycle
 Simulation software typically has a life-cycle
longer than computing platforms and the length of
many funding initiatives. This reflects:
 Challenges of algorithmic and modeling advances
 Dealing with complexity
 Tens to hundreds of man-years of development
 Often millions of lines of code
 Funding initiatives should reflect these realities.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Finding 3: 1 size won’t fit all
 Simulation software is too rich and too diverse to
suit a single paradigm for progress, across diverse
applications disciplines. Consider:
 Desktop computers versus supercomputers
 Public domain software versus commercial software
 New algorithms versus new architectures
 Focused teams vs long-term loosely coupled groups
 Best outcomes arise from permitting viable
alternatives to competitively co-exist.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
78
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding 4: Funding trends
 In the U.S., there are challenges in funding
simulation software development
 Budgetary challenges vs long life-cycle.
 Disciplinary emphasis on phenomena vs methods
 General stress on research funding levels
 Consequences for future trained scientists & engineers
 Europe and Japan lead in some fields, due to
changing trends in funding. China is fast
developing, and will soon be a major competitor.
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
QuickTime™ and a
decompressor
are needed to see this picture.
Finding 5: New opportunities
 US-driven advances in computer power continue
 Multicore & graphics processors are here today.
 Massive processor-level parallelism lies ahead
 New development tools needed: languages, libraries, etc
 Cost of entry at the low end is near zero
 Means widespread adoption, world-wide competition.
 New application needs can be addressed
AMD “Spider” GPU/CPU
 Complexity / multiscale / emergent phenomena
 Resolution / accuracy / entirely new algorithms
WTEC/SBES Workshop 25 April 2008
Head-Gordon: Simulation Software
79
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Engineering Simulation
Abhi Deshmukh
Industrial and Systems Engineering
Texas A&M University
QuickTime™ and a
decompressor
are needed to see this picture.
Use of Simulation in Engineering
 Computation vs. Prediction
 Analysis vs. Synthesis
forward vs. inverse
 Physical Processes vs.
Systems
continuous vs. event-driven
 Off-line vs. Real-time
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
80
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Application Areas
 CFD, turbulence, aero-thermal-acoustics
 Structural analysis, FEM, FEA, deformation
and failure models
 Physical process simulations, continuum
models, bulk transformation models
 System dynamics models, kinematics,
vibrations
 Enterprise and supply chain models
 Transportation, packet flow, networks
 Training, games, situation assessment
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Japan
 RIKEN, home of the10 PFlp NGSC project, uses a multi-
architecture (scalar, vector and accelerators) approach to
provide appropriate computational resources for wide
range of engineering applications
 Digital production
 Computational bioengineering
 Benchmark engineering applications for NSCG are:
Cavitation
Computation of Unsteady Cavitation Flow by the Finite
Difference Method
LANS
Computation of compressible fluid in aircraft and spacecraft
analysis
FrontSTR
Structural Calculation by Finite Element Method (FEM)
FrontFlow/Blue
Unsteady Flow Analysis based on Large Eddy Simulation
(LES)
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
81
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Japan

Toyota Central R&D Labs

Total HUman Model for Safety
(THUMS) is a suite of detailed
finite element models of the
human body that can be
subjected to impacts to assess
injury thresholds.
 Mill-Plan is designed to
determine the optimal number of
machining processes, and the
tool form and cutting conditions
for each process.

Engineering grand challenge

First-principles molecular
dynamics simulations of fuel
cells
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - China
 Dalian Institute of Technology – has two
key labs in SBES domain and maintains
significant interactions with industry
‡Simulation of ascending process
‡Sheet metal stamping simulation
‡Stability analysis
‡Identifies forming defects
‡Design improvements
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
82
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - China
 Shanghai University – High Performance
Computing Center
Significant efforts on numerical methods
Significant use of HPC for engineering
applications
Separated flow at HAA (SU27)
Wind load on “Bird Nest” (Beijing 2008)
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Denmark
 TU Denmark and Riso - Wind Energy
Research
 Focus on design of optimum airfoils; dynamic stall,
especially 3-D stall; tip flows and yaw; heavily loaded
motors; and interference effects
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
83
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Germany
 University of Stuttgart – Simulation Technology
Excellence Cluster (funded by DFG)
 multi-scale, multi-physics simulations under uncertainty
 interactive context-adapted optimization of systems
influenced by sensor data streams in real time, and
 interaction between developers and simulations for
management, optimization, control and automation
 One of the most comprehensive education
programs for simulation and modeling
 BSc/MSc elite program of study in simulation
 Graduate school (PhD) in simulation technology
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Spain
 International Center for Numerical Methods
in Engineering (CIMNE), Barcelona
 Focused on finite element methods
 Strong educational and industry programs throughout
the world
Combination of finite element and
particle methods in solids mechanics
Development of graphical visualization techniques for large scale
simulation problems (www.gidhome.com)
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
84
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Findings
 Interoperability of software and data are major hurdles
 Commercial vendors are defining de facto standards
 Very little effort beyond syntactic compatibility
 Use of simulation software by non-simulation experts is
limited
 Codes are too complicated to permit any user customization
 Effective workflow methods need to be developed to aid in
developing simulations for complex systems
 In most engineering applications, algorithms, software
and data are primary bottlenecks
 Computational resources (flops and bytes) were not limiting
factors at most sites
 Lifecycle of algorithms is in 10-20 years range, whereas
hardware lifecycle is in the 2-3 years range
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Findings
 Visualization of simulation outputs remains a challenge
 Use of HPC and high-bandwidth networks has exasperated the
problem
 Uncertainty is not being addressed adequately in many
of the applications
 Most engineering analyses are conducted under deterministic
settings
 Links between physical and system level simulations are
weak
 Very little evidence of atom-to-enterprise models
 Engineers are not being trained adequately in academia
to address simulation and modeling needs
 Combination of domain, modeling, mathematical, computational
and decision-making skills is rare
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
85
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Comparison with US Baseline
 On average, US academia and industry are ahead
(marginally) of their European and Asian counterparts
 Pockets of excellence exist in Europe and Asia that are more
advanced than US groups (Toyota, Airbus, U Stuttgart)
 European and Asian researchers rely on US to develop
the common middleware tools
 Their focus is on application-specific software
 The transition from physical systems modeling to social-
scale engineered systems in US lags behind the
Japanese
 Modeling behavioral patterns of 6 billion people using the Life
Simulator
 European universities are leading the world in developing
curricula to train the next generation of engineering
simulation experts
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
Research Examples
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
86
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
QuickTime™ and a
decompressor
are needed to see this picture.
HPC activities at RIKEN: Digital production
 Voxel based simulation
 Multi physics:
Flow/heat/structure/noise
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
87
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
HPC activities at RIKEN:
Computational Bioengineering
Coupled simulation with
flow and structure
X–Z
section
Workshop 25 April 2008
X –WTEC/SBES
Y
Deshmukh: Engineering Simulation
section
QuickTime™ and a
decompressor
are needed to see this picture.
Fracturing prediction of a part made in Tianjin
Auto Die Maker
WTEC/SBES Workshop 25 April 2008
Deshmukh: Engineering Simulation
88
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Verification, Validation and
Uncertainty Quantification:
A cross-cutting theme in SBES
George Em Karniadakis
(Oden Report)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Verification, Validation and Uncertainty Quantification:
A cross-cutting theme in SBES
• First addressed rigorously by the
Society for Computer Simulation
(1979)
(2007)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
89
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Basic Definitions
•US Defense Modeling and Simulation Office (DMSO) has
been the leader in developing fundamental
concepts and terminology for V&V
•Specific definitions (1994); also AIAA/ASME
 Verification
 The process of determining that a computational software
implementation correctly represents a model of a physical process
 The process of determining that the equations are solved correctly
 Validation
 The process of assessing the degree to which a computer model is
an accurate representation of the real world from the perspective of
the models intended applications
 The process of determining that we are using the correct equations
 Debate: Invalidation of a theory…K. Popper, S. Hawkins,…
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Uncertainty versus Error
•Uncertainty: A potential deficiency in any phase or activity
of a modeling process that is due to lack of knowledge.
•Aleatory (irreducible) Uncertainty
•Epistemic (reducible) Uncertainty
•Error: A recognizable deficiency in any phase or activity
of modeling and simulation that is not due to lack of knowledge.
•Sources of Uncertainty: Initial and Boundary Conditions,
Thermo-physical/Structural Properties, Geometric Roughness,
Interaction Forces, Background Noise, Statistical Potentials…
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
90
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Validation Hierarchy for a Hypersonic Cruise Missile
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Industry needs more accurate Validation Pyramids
Case study: Failure of the A-380 wing (February 14, 2006)
Babuska et al., 2007
•The wing had to endure 150% of the load for
3 sec but it broke at 147%.
•The wing failed because a key validation
experiment was not included in the Pyramid
Design of a new Validation Pyramid
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
91
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: Modeling explicitly uncertainties in engineering
design enlarges its scope to include economies and politics
•Paradigm shift in system design: From risk-based (bad side of PDF)
to uncertainty-based (both sides of PDF)
•A broader objective - Reframe the design problem:
•From one that meets specifications to one that provides
the best performance over a range of scenarios
DoD CREATE
™Concept of Strategic Engineering
(de Neuvfville et al, 2004)
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: Management of uncertainties reformulates
and expands the approach to engineering, planning, design
and implementation
(Table describes specific example)
MIT
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
92
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: Uncertainty in thermo-physical properties is
of major concern to chemical industries
•Site visits: Mitsubishi Chemicals, BASF, ENI
•NAS/ICME April 2008 report
 Properties required for process screening
 Physical properties frequently estimated by engineering correlations
or measured by simple, low-cost experiment (e.g., infinite dilution
activity coefficients by gas chromatography)
 Accuracy of ±25% acceptable in cost estimates
 Demands for data accuracy vary (Larsen, 1986)
 20% error in density ---> 16% error in equipment size/cost
 20% error in diffusivity ---> 4% error in equipment size/cost
 Errors in density usually small for liquids, errors in diffusivity frequently large
(factor of two or more)
 10% error in activity coefficient results in negligible error in equipment
size/cost for easily separated mixtures, but for close-boiling mixtures
(relative volatility <1.1)
10% error can result in equipment sizes off by a
factor of 2 or more!
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: UQ in Molecular Systems is just starting
GGA-DFT ensemble results for solids and molecules
Technical University of Denmark
QuickTime™ and a
decompressor
are needed to see this picture.
Mortensen et al., Physical Review Letters (2005) vol. 95 (21) pp. 216401
93
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: New uncertainty quantification methods make
validation in Systems Biology more meaningful
Dongbin Xiu, Department of Mathematics, Purdue University, USA
Genetic toggle Switch – Nature, vol. 403, 2000
Mathematical Model:
• Differential-algebraic system
• 6 uncertain parameters
1
0.8
6
nh ( y ) =
nh +
å
wˆ k y k + H .O .T .
k= 1
wˆ 1 = - 0 .0 4 , wˆ 2 = 0 .0 4 , wˆ 3 = 0 .9 ,
wˆ 4 = - 0 .9 , wˆ 5 = 0 .3, wˆ 6 = - 0 .3
Normalized GFP expression
Observable: Hill coefficient n
(measures “slope” of the switch)0.6
0.4
0.2
→ Highly sensitive to 2 parameters in red.
0
6
5.5
5
4.5
4
3.5
3
2.5
2
log10(IPTG)
Numerical error bars (red) vs experimental error bars (blue)
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Finding: New stochastic modeling methods are required for
time-dependent and high-dimensional problems
•Uncertainty increases in time
•Curse of dimensionality
•New sensitivity methods are required for large perturbations
QuickTime™ and a
decompressor
are needed to see this picture.
94
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Finding: US leads V&V and UQ (in volume) activities
at this juncture but Germany follows closely behind…
•Multi-level MC methods developed in Germany/UK
•Europe leads on theoretical developments
•Validation: a lot of “simulation-meets experiment” projects worldwide
•Sponsorship of UQ chair by industry at University of Stuttgart; Munich
•Toyota CRDL organize their own UQ workshops for complex dynamical
systems
•MUNA: (Minimization and Management of Uncertainty in Numerical
Aerodynamics) A collaborative effort among DLR, 8 Universities,
Airbus Germany, Eurocopter, EADS-MAS
•USA activities & funding
•DOE ASCI and Predictive Science programs
•NSF (AM-SS) ; AFOSR; ONR (individual PIs unlike Germany/DFG programme)
•Sandia’s DACOTA code implements uncertainty quantification with sampling,
reliability, and stochastic finite element methods, parameter estimation with
nonlinear least squares methods, and sensitivity/variance analysis with design
of experiments and parameter study capabilities
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Examples:
Sources of Uncertainty and Effects
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
95
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
DNS vs. Experiment
• X. Ma, G. Karniadakis – JFM, 2000
Cross-Flow Velocity: Energy spectrum at X/D=7
100 M DOF
Intrinsic stochasticity
-5/3
Energy input
Black: DNS
Blue: Experiment
By Ong & Wallace
WTEC/SBES Workshop 25 April 2008
-- Karniadakis:
(wave
number)
Verification, Validation, and Uncertainty Quantification
DNS vs. Experiment
QuickTime™ and a
decompressor
are needed to see this picture.
• X. Ma, G. Karniadakis – JFM, 2000
Cross-Flow Velocity: Energy spectrum at X/D=7
100 M DOF
Intrinsic stochasticity
-5/3
Energy input
Black: DNS
Blue: Experiment
By Ong & Wallace
WTEC/SBES Workshop 25 April 2008
-- Karniadakis:
(wave
number)
Verification, Validation, and Uncertainty Quantification
96
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Arterial Wall Mechanics
(Zohar Yosibash, Ben Gurion)
Load
cell
Micromete
Micromete
rr
Stress vs stretch – 5 ITA’
ITA’s segments
produce different response.
Typical stretch is about 1.2 to 1.4
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
QuickTime™ and a
decompressor
are needed to see this picture.
Stochastic Gene Expression in a Single Cell
Eblowitz et al., Science, 2002
•Bacterial cells expressing two different
fluorescent proteins (red and green) from
identical promoters.
•Because of stochasticity (noise) in the
process of gene expression, even two
nearly identical genes often produce unequal
amounts of protein.
•The resulting color variation shows how noise fundamentally limits
the accuracy of gene regulation.
WTEC/SBES Workshop 25 April 2008 -- Karniadakis:
Verification, Validation, and Uncertainty Quantification
97
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Big Data and Data-Driven
Simulations
Sangtae Kim
QuickTime™ and a
decompressor
are needed to see this picture.
Potential Impacts
 Correlation of HPC (Giga-, Tera-, Peta-,…) of
speed in FLOPS and data size in Bytes
 Data generation rate exceeds storage
capacity (for the first time in history)
 “Big data” is the link between HPC/SBES and
smart environments: sensor-nets, u-IT and
pervasive computing
 And …
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
98
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Dynamic, Data-Driven Simulations:
$15B development* of the Khurais mega-field
Massive, seawater injection;
2,700 sq. miles Khurais is
ringed by 100 injection wells
2.8 million 3D images of the
field’s underground strata,
shot over 20 months.
Successful SBES outcome:
non-shocking prices at the
gas pump.
*Aramco hired: Haliburton (driller)
SNC Lavalin Ltd, Eni SpA, and Foster
Wheeler Ltd (project manager)
Wall Street J. 4/22/2008 Neil King, Jr.
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
QuickTime™ and a
decompressor
are needed to see this picture.
SBES Big Data Issues
 Appropriately trained students (and researchers)
 Data provenance, heterogeneity, inference
 Distributed resources and users: data (web) services
and middleware, e.g., caBIGTM
 High-speed networks
 Workflows linking high-performance simulation and data
analytics and visualization
 Integrated community-wide data infrastructure (funding)
 Communities differ in strategic view of data issues
 Inter-disciplinary issues: transfer and collaborations
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
99
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Japan
 Kitano Lab (Systems Biology): data software
infrastructure
 Systems Biology Markup Language (SBML) (together
with Caltech)
 Systems Biology Graphical Notation (SBGN)
 Web 2.0 Biology
 Connection with Riken next-generation supercomputer
effort
 Funding model: Funded for 10 years, $2 million per
year, noncompetitive grant
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
QuickTime™ and a
decompressor
are needed to see this picture.
Regional Highlights - Europe
 EPFL Blue Brain Project
 Development of annotated database of experimental
results is a major focus
 Impressive visualization
 35 research/development personnel
 DL/STFC Big Data in Structural Biology
 e-HPTX management of workflow
 DNA data collection
 PIMS laboratory information management system
 Funded by U.K. e-Science initiative
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
100
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Regional Highlights - Europe
 CERN Large Hadron Collider
 (global collaboration including USA)
 Extreme high end of data generation (100 PB/min),
collection, transmission, storage and analytics (1 PB/yr)
 Multi-tier (Tier0, Tier1, Tier2, Tier3)
 Informatics for the Atlas and CMS detectors includes
 DTU, Lyngby, Center for Systems Biology Dept.
 Data integration: “let data speak to data”
 Disease interactomes
 Future convergence of PDE and data branches of Systems
Biology (data-driven SB)
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
QuickTime™ and a
decompressor
are needed to see this picture.
“Big Data” Findings
 An appreciation of the role of “big data” varies
primarily by field, with the biological sciences
(bioinformatics) and particle physics (CERN)
leading the way.
 The gold standard for open science is the CERN
data infrastructure for the global particle physics
community, from 100 PB/min to 1 PB/yr.
 “Big data” appreciated by industry (bio/pharma, oil
production, … ). Universities lag in recognizing
strategic role of data management.
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
101
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
How does the U.S. compare?
 Funding models and infrastructure to support multi-
investigator collaborations between academia and
industry is more developed in Europe (FP7).
 Primary correlation is by discipline, not region.
 Thus for U.S., significant opportunity for taking
leadership with new investments in education and
training in “big data”, data analytics and workflow.
 Data and dynamic data-driven applications of special
significance in energy production and sustainability,
thus high impact on U.S. competitiveness (U.S.
economy is more leveraged to energy costs).
WTEC/SBES Workshop 25 April 2008 -- Kim: Big Data
102
QuickTime™ and a
decompressor
are needed to see this picture.
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Sites Visited
by the WTEC Panel on International R&D
in Systems-Based Engineering & Science
Asia
Central Research Institute of Electric Power Industry (CRIEPI)
Computational Materials Science Center (Japan)
Dalian University of Technology Department of Vehicle Engineering and Mechanics
Fudan University
Institute for Molecular Science (IMS)
Institute of Chemistry, Chinese Academy of Sciences (ICCAS)
Institute of Computational Mathematics and Scientific/Engineering Computing (ICMSEC/CAS)
Institute of Process Engineering, Chinese Academy of Sciences
Japan Agency for Marine-Earth Science and Technology Earth Simulator Center (ESC)
Kyoto University
Mitsubishi Chemical Group Science and Technology Research Center
Nissan Research Center, Fuel Cell Laboratory
Peking University Center for Computational Science and Engineering
Research Institute for Computational Sciences (RICS/AIST)
RIKEN – The Institute of Physical and Chemical Research Advanced Center
for Computing and Communication (ACCC)
Shanghai Supercomputer Center
Shanghai University
The Systems Biology Institute (Japan)
Toyota Central R&D Labs, Inc.
Tsinghua University Department of Engineering Mechanics
University of Tokyo (Workshop on R&D in Simulation-Based Engineering and Science)
Europe
Autonomous University of Barcelona and Materials Science Institute of Barcelona
BASF – The Chemical Company
Center for Atomic-Scale Materials Design (CAMD), Technical University of Denmark
CERN (European Organization for Nuclear Research)
CIMNE (International Center for Numerical Methods in Engineering)
Ecole Polytechnique Fédérale de Lausanne (EPFL)
Ecole Polytechnique Fédérale de Lausanne (EPFL), Blue Brain Project
Eni SpA
ETH (Swiss Federal Institute of Technology) Zürich, Zentrum and Hönggerberg
Fraunhofer Institute for the Mechanics of Materials (IWM)
IBM Zurich Laboratory, Deep Computing
Imperial College London and Thomas Young Centre
Institute Français du Petrol (French Petroleum Institute)
Institute of Fluid Mechanics of Toulouse (IMFT)
IRIT (Institut de Recherche en Informatique de Toulouse), and ENSEEIHT (Ecole Nationale Supérieure
d´Electrotechnique, d´Electronique, d´Informatique, d´Hydraulique et des Télécommunications)
Paris Simulation Network
Politecnico di Milano (Polimi) (remote site report)
Science and Technology Facilities Council (STFC) Daresbury Laboratory (DL)
Technical University of Denmark Department of Chemical and Biochemical Engineering
Technical University of Denmark Wind Engineering
Technical University of Denmark Center for Biological Sequence Analysis
Technical University of Munich (Institut für Informatik), and Leibniz Supercomputing Centre
Unilever Centre for Molecular Informatics, University of Cambridge
103
WTEC 25 April 2008 Workshop Draft Proceedings, International R&D in Simulation-Based Engineering & Science
Unilever R&D Port Sunlight
University College London
University of Cambridge Centre for Computational Chemistry
University of Cambridge Dept. of Applied Mathematics and Theoretical Physics (DAMTP)
University of Cambridge Theory of Condensed Matter Group
University of Karlsruhe , Karlsruhe Research Center, and Karlsruhe Institute of Technology
University of Oxford Condensed Matter Theory Group
University of Oxford Department of Engineering Science
University of Oxford Theoretical Chemistry Group
University of Oxford, Structural Bioinformatics and Computational Biochemistry Group
University of Stuttgart Institute of Thermodynamics and Thermal Process Engineering
University of Zurich Physical Chemistry Institute
Vrije University Amsterdam and BioCentrum Amsterdam
Vrije University Theoretical Chemistry Section
Zuse Institute Berlin (ZIB)
104
JTEC/WTEC reports and information are available on the Web at http://www.wtec.org or from the National Technical Information
Service (NTIS), 5285 Port Royal Road, Springfield, VA 22161. Call for prices/availability, 703-487-4650, or Fax 703-321-8547.
JTEC Panel Report on High Temperature Superconductivity in Japan
(11/89) PB90-123126
JTEC Panel Report on Space Propulsion in Japan (8/90)
PB90-215732
JTEC Panel Report on Nuclear Power in Japan (10/90) PB90-215724
JTEC Panel Report on Advanced Computing in Japan (10/90)
PB90-215765
JTEC Panel Report on Space Robotics in Japan (1/91) PB91-100040
JTEC Panel Report on High Definition Systems in Japan (2/91)
PB91-100032
JTEC Panel Report on Advanced Composites in Japan (3/91)
PB90-215740
JTEC Panel Report on Construction Technologies in Japan (6/91)
PB91-100057
JTEC Panel Report on X-Ray Lithography in Japan (10/91)
PB92-100205
WTEC Panel Report on European Nuclear Instrumentation and
Controls (12/91) PB92-100197
JTEC Panel Report on Machine Translation in Japan (1/92)
PB92-100239
JTEC Panel Report on Database Use and Technology in Japan (4/92)
PB92-100221
JTEC Panel Report on Bioprocess Engineering in Japan (5/92)
PB92-100213
JTEC Panel Report on Display Technologies in Japan (6/92)
PB92-100247
JTEC Panel Report on Material Handling Technologies in Japan
(2/93) PB93-128197
JTEC Panel Report on Separation Technology in Japan (3/93)
PB93-159564
JTEC Panel Report on Knowledge-Based Systems in Japan (5/93)
PB93-170124
NASA/NSF Panel Report on Satellite Communications Systems &
Technology (7/93): Executive Summary (PB93-231116);
Analytical Chapters (PB93-209815); Site Reports (PB94-100187)
WTEC Monograph on Instrumentation, Control & Safety Systems of
Canadian Nuclear Facilities (7/93) PB93-218295
JTEC/WTEC Annual Report and Program Summary 1993/94 (3/94)
PB94-155702
JTEC Panel Report on Advanced Manufacturing Technology for
Polymer Composite Structures in Japan (4/94) PB94-161403
ITRI Monograph on Benchmark Technologies Abroad: Findings from
40 Assessments, 1984-94 (4/94) PB94-136637
WTEC Panel Report on Research Submersibles and Undersea
Technologies (6/94) PB94-184843
JTEC Panel Report on Microelectromechanical Systems in Japan
(9/94) PB95-100244
WTEC Panel Report on Display Technologies in Russia, Ukraine, and
Belarus (12/94) PB95-144390k,
JTEC Panel Report on Electronic Manufacturing and Packaging
in Japan (2/95) PB95-188116
JTEC Monograph on Biodegradable Polymers and Plastics in Japan
(3/95) PB95-199071
JTEC Panel Report on Optoelectronics in Japan and the United States
(2/96) PB96-152202
JTEC Panel Report on Human-Computer Interaction Technologies
in Japan (3/96) PB96-157490
WTEC Panel Report on Submersibles and Marine Technologies in
Russia’s Far East and Siberia (8/96) PB96-199526
JTEC Panel Report on Japan’s ERATO and PRESTO Basic Research
Programs (9/96) PB96-199591
JTEC/WTEC Panel Report on Rapid Prototyping in Europe and
Japan: Vol. I. Analytical Chapters (3/97) PB97-162564
Vol. II. Site Reports (9/96) PB96-199583
WTEC Panel Report on Advanced Casting Technologies in Japan and
Europe (3/97) PB97-156160
WTEC Panel Report on Electronics Manufacturing in the Pacific Rim
(5/97) PB97-167076
WTEC Panel Report on Power Applications of Superconductivity
in Japan and Germany (9/97) PB98-103161
WTEC Workshop Report on R&D Status and Trends in
Nanoparticles, Nanostructured Materials, and Nanodevices in the
United States (1/98) PB98-117914
WTEC Panel Report on Electronic Applications of Superconductivity
in Japan (7/98) PB98-150139
WTEC Monograph on Use of Composite Materials in Civil
Infrastructure in Japan (8/98) PB98-158215
WTEC Panel Report on Nanostructure Science and Technology
(12/98 — available from Kluwer Academic Publishers)
WTEC Panel Report on Global Satellite Communications Technology
and Systems (12/98) PB99-117954
WTEC Panel Report on Digital Information Organization in Japan
(2/99) PB99-128019
WTEC Panel Report on the Future of Data Storage Technologies
(6/99) PB99-144214
WTEC Panel Report on Japan’s Key Technology Center Program
(9/99) PB99-142424
WTEC Workshop on Russian R&D Activities on Nanoparticles and
Nanostructured Materials (12/99) PB99-150518
WTEC/MCC Strategic Technology Tour Report on MEMS and
Microsystems in Europe (1/2000) PB2000-104945
WTEC Panel Report on Wireless Technologies and Information
Networks (7/2000) PB2000-105895
WTEC Panel Report on Environmentally Benign Manufacturing
(4/2001) PB2001-104339
WTEC Panel Report on Tissue Engineering Research (1/2002;
available from Academic Press)
WTEC Panel Report on Applications of Molecular and Materials
Modeling (1/2002; available from Kluwer Academic Publishers)
WTEC Panel Report on Spin Electronics (8/2003; available from
Kluwer Academic Publishers)
WTEC Panel Report on Microsystems Research in Japan (9/2003)
WTEC Panel Report on International Research and Development in
Biosensing (8/2004) (5/2006; available from Springer)
WTEC Panel Report on Additive/Subtractive Manufacturing Research
and Development in Europe (11/2004)
WTEC Panel Report on High-End Computing Research and
Development in Japan (12/2004)
WTEC Panel Report on Operations Research for Health Care
Delivery Systems (5/2005)
WTEC Panel Report on Systems Biology (10/2005) (1/2007 available
from Springer)
WTEC Panel Report on Micromanufacturing (10/2005) (3/2007
available from Springer)
WTEC Panel Report on Robotics (1/2006)
WTEC Panel Report on Research and Development in Carbon
Nanotube Manufacturing and Applications (6/2007)
WTEC Panel Report on Research and Development in BrainComputer Interfaces (10/2007) (in press at Springer)
WTEC Panel Report on Research and Development in Rapid Vaccine
Manufacturing (12/2007)