The DoD Computational Research and Engineering Acquisition Tools and Environments (CREATE) Program Dr. Douglass Post Chief Scientist and CREATE Program Manager DoD High Performance Computing Modernization Program NDIA Workshop, 1 Feb 2011, Arlington, VA NDIA CREATE-D.Post 2/1/2011 Page-1 Distribution Statement A: Approved for public release; distribution is unlimited. DoD HPC Modernization Program NDIA CREATE-D.Post 2/1/2011 Page-2 Distribution Statement A: Approved for public release; distribution is unlimited. DoD HPC Modernization Program In 2011, DoD HPCMP will have 5 centers, with 2x1015 FLOPs & 200,000 processors Software Development Supports 5 Institutes, CREATE and Education High Bandwidth, Low Latency Network Linking 284 sites Gather Service and Agency Requirements and Allocate Resources NDIA CREATE-D.Post 2/1/2011 Page-3 Distribution Statement A: Approved for public release; distribution is unlimited. What Do We Want To Accomplish Today? • Describe the DoD CREATE program and paradigm • Describe how CREATE and the paradigm of multiphysics-based computational engineering can help the US defense industry design, deliver and sustain DoD weapons systems in less time with lower risks • Start engaging with US defense industries to ensure that CREATE will help enable US defense industries to directly support DoD acquisition • Describe the current and future technical capabilities of CREATE • Begin to use NDIA as a vehicle to connect CREATE and the HPCMP computer centers to the US Defense Industry NDIA CREATE-D.Post 2/1/2011 Page-4 Distribution Statement A: Approved for public release; distribution is unlimited. Acquisition Takes Too Long! Major root causes: • Rapidly changing requirements • Continuing pressure to deliver complex systems faster • A build and test methodology that results in problems being discovered late in the acquisition process Root causes lead to: • Extensive testing to identify flaws and integration problems • Extensive rework to fix design flaws and integration problems Impact: • Delays to fix problems • Large technical, schedule and budget risks NDIA CREATE-D.Post 2/1/2011 Page-5 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE Programmatic Objectives and Concept • Objective: Sustain the US military competitive advantage by developing and deploying scalable multiphysics-based computational engineering tools to design and analyze DoD weapons system performance – Produce optimized designs with fewer defects and design flaws – Reduce testing and rework to enable faster system development and deployment • How? – Supplement physical testing with multiphysics-based computational performance predictions specifically targeted to complex DoD systems • Ancillary benefit – Higher productivity for the shrinking number of US defense community engineers and scientists NDIA CREATE-D.Post 2/1/2011 Page-6 Distribution Statement A: Approved for public release; distribution is unlimited. Testing Conundrum • Weapons systems are increasing in complexity – Need more complex testing in face of pressure to reduce testing facilities and test time for all phases of acquisition • Multiphysics-based performance predictions can improve testing effectiveness and efficiency – Supplement and partially replace real prototypes with virtual prototypes – Frees testing community to identify and understand the phenomena that determine system behavior • Example1: For wind tunnels, physics-based computational engineering models can REDUCE: 1. 2. 3. “Overall testing workload through identifying the most important data to take rather than “One Factor at a Time” approaches and using condensing calculated performance and accessing it with non-linear systems identification (SID) methods Down stream effects of late defect discovery on total development cycle time Amount of testing required to determine support interference or wind tunnel wall effects or to better understand scaling issues from wind tunnel to flight conditions” 1Kraft, NDIA CREATE-D.Post 2/1/2011 Page-7 E., “Integrating Computational Science and Engineering with Testing to Re-engineer the Aeronautical Development Process.” AIAA 2010-139. CREATE AV Board of Directors chair Distribution Statement A: Approved for public release; distribution is unlimited. Present Product Development Process Iterated Design Build Test Cycles Requirements Design Build Physical Product (Many) Design iterations Manufacture, Test Physical Product Sustain, and Modify F-22 Flight Test • Long time to market – Requires many lengthy and expensive design/build/test iteration loops • Process converges slowly – Process is rigid, not responsive to new requirements – Design flaws discovered late in process leading to rework – Systems Integration happens late in process NDIA CREATE-D.Post 2/1/2011 Page-8 Distribution Statement A: Approved for public release; distribution is unlimited. MultiPhysics-Based Performance Analysis Increases Productivity for Complex Systems Requirements Design on Computer Build Mesh Analyze Performance (Many) Design iterations Groundbased and Flight Tests • Reduced design and development time – Highly scalable computational performance analysis of virtual prototypes reduces the need to test real prototypes • Process converges much faster – Process is flexible, very responsive to new requirements – Design flaws early in process reducing rework – Systems Integration happens at every step of the process NDIA CREATE-D.Post 2/1/2011 Page-9 Distribution Statement A: Approved for public release; distribution is unlimited. Manufacture, Sustain, and Modify This Paradigm Has Been Proven Successful in Industry Flight Radial Requirements • Reduced time to market from 3 years to less than 1 year • In 2009, Goodyear introduced 62 new products using this process Design and Mesh Virtual Product Analyze and Test Build and Test Virtual Product Physical Product Market Design iterations Fortera TripleTred Miller, L.K., “Simulation-Based Engineering for Industrial Competitive Advantage.” Computing in Science and Engineering, 12, pp. 14–21, 2010. NDIA CREATE-D.Post 2/1/2011 Page-10 Distribution Statement A: Approved for public release; distribution is unlimited. Performance Analysis of Virtual Prototypes Is the Key Concept Development Engineering Development A Mission Need Determination Material Solution Analysis AOA B Technology Development Post Development C Engineering and Manufacturing Development IOC Production and Deployment FOC Operation and Support Virtual Integrated Prototyping Environment Experimental Sub-System Prototypes Experimental System Prototypes Inject physics into design early and all through the process! NDIA CREATE-D.Post 2/1/2011 Page-11 Distribution Statement A: Approved for public release; distribution is unlimited. The 1015 increase in computer power since 1945 can enable us to develop and deploy codes during the next decade that are much more powerful than past tools: – Utilize accurate solution methods – Include all the effects we know to be important – Model a complete system – Complete parameter surveys in hours rather than days to weeks to months • • In ~ 10 years, workstations will be as powerful as today’s supercomputers Greatest opportunities for 2020 (and 2010) include large-scale codes that integrate many multi-scale effects to model a complete system NDIA CREATE-D.Post 2/1/2011 Page-12 Computing Power For The World's Fastest Computer Floating Point Operations/sec 108 106 Supercomputers 104 Cores • Performance (GFLOPs/sec) Enabling Technology 100 1 0.01 Workstation Performance 0.0001 10-61940 1950 1960 1970 1980 1990 2000 2010 2020 Year Moore’s “Law” Distribution Statement A: Approved for public release; distribution is unlimited. What is CREATE? • CREATE is a DoD program to develop and deploy multiphysics-based software for engineering design and analysis of: • Air Vehicles (AV) – Aerodynamics, structural mechanics, propulsion, control, … • Ships – Shock vulnerability, hydrodynamics, concept design • Radio Frequency (RF) Antennas – RF Antenna electromagnetics and integration with platforms • Mesh and Geometry (MG) Generation – Rapid generation of mesh and geometry representations CREATE tools support all stages of acquisition from rapid early stage design to full life-cycle sustainment NDIA CREATE-D.Post 2/1/2011 Page-13 Distribution Statement A: Approved for public release; distribution is unlimited. F-35 CREATE-AV Product Development and Deployment DaVinci targets early-phase acquisition engineering (conceptual design) Kestrel is a high-fidelity, full-vehicle, multi-physics analysis tool for arbitrary fixed-wing aircraft Helios Helios is a high-fidelity, full-vehicle, multi-physics analysis tool for arbitrary rotary-wing aircraft Firebolt is a module for propulsion systems in fixed and rotary-wing air vehicles. Firebolt A CREATE-AV Software Product NDIA CREATE-D.Post 2/1/2011 Page-14 Module includes engineering models of propulsion systems suitable for conceptual studies and also high-fidelity, multi-physics simulation. Distribution Statement A: Approved for public release; distribution is unlimited. CREATE – Ships Product Components Rapid Design and Integration – Rapid development, assessment, and integration of candidate ship designs to avoid cost versus capability mismatches – Partnership with ONR & NAVSEA Ship Shock & Damage – Provide analysis of shock and damage effects and reduce need for tests to assess ship shock and damage effects Ship Hydrodynamics – Accelerate and improve all stages of ship hydrodynamic design (seaway loads, sea keeping, resistance…) NDIA CREATE-D.Post 2/1/2011 Page-15 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE – Radio Frequency (RF) Product Component SENTRI enables antenna design integrated with platforms – Able to calculate multiple antenna arrays for small- and large-scale systems (small UAVs ships) ● Staged capability – Rapid design – coarse resolution with fast, first-order codes for small antenna systems – High fidelity detailed design, high-resolution complex antenna systems integrated with large platforms – Enabled by improved computational algorithms (n3 n log n scaling improvement) Small Less Complex NDIA CREATE-D.Post 2/1/2011 Page-16 Larger Highly Complex Distribution Statement A: Approved for public release; distribution is unlimited. CAPSTONE – CREATE-MG Product Overview CAPSTONE provides geometry and meshing needs for all phases of acquisition engineering (conceptual-, preliminary-, detailed-design and operational-support) CAPSTONE: Core CAPSTONE: Frontend Produce analyzable representations for complex and detailed analysis NDIA CREATE-D.Post 2/1/2011 Page-17 Enable parametric, associative geometry and meshes in AV:DaVinci, Ships:RDI; geometry-based mesh adaptivity Distribution Statement A: Approved for public release; distribution is unlimited. CREATE Is a Multi-Service, Multi-institutional Program HPCMP Executing Agent HPCMP Director Official HPCMP Advisory Panel CREATE Program Ships Project NSWCCarderock Air Vehicles Project NAVAIR, Patuxent River NAVYRANS NSWCCarderock, Kestrel Eglin AFB Integrated Hydro Design Environment NSWC-Carderock Shadow-Ops NAVAIR, Patuxent River Navy Enhanced Sierra Mechanics NSWC-Carderock Rapid Design NAVSEA 05D NDIA CREATE-D.Post 2/1/2011 Page-18 RF Antennas Project AFRL, WPAFB SENTRI Helios Army, Ames DaVinci ASC & AFRL Firebolt AEDC Distribution Statement A: Approved for public release; distribution is unlimited. Mesh & Geometry Project NRL Capstone CREATE is a Distributed Set of Teams at DoD Engineering Centers AV U of Wy AMES ETI SPAWAR NDIA CREATE-D.Post 2/1/2011 Page-19 SNL AFRL SNL SNL Ships RF U of Iowa MG U of Mich ASC AFRL HPCMP Penn State CERDEC Pax River AEDC 46th Test Wing Distribution Statement A: Approved for public release; distribution is unlimited. HPCMP Carderock NAVSEA Indian Head ONR NRL The CREATE Approach • Software is being built by government-led teams • Each product is released following a roadmap – Each year there is a release of a usable application – Each release builds on the previous release and adds the increased capability called for in the roadmap – Each release is beta-tested by targeted user communities before a broader release – Each release goes through a rigorous V&V process • Releases are progressively more scalable for massively parallel computers and responsive to user requirements • CREATE Program is guided by DoD service acquisition engineering organizations and their senior leadership and US defense industry (where possible) NDIA CREATE-D.Post 2/1/2011 Page-20 Distribution Statement A: Approved for public release; distribution is unlimited. Software Engineering Focused on Producing High Quality Engineering Software Tools • Goal is maintainable, extensible, portable and reliable software products developed with the right balance of agility and flexibility and short and long term planning based Use Software Engineering Processes Specifically Designed for Technical Software Documents: ICD +…. • • – • Manuals: Technical, Developer, User (+ training, tutorials, sample input and output,…) V&V is a key priority – – – • Verification at every level Validation as much as feasible (large potential role for defense industry) Beta testing a key opportunity for defense industry input User support and training NDIA CREATE-D.Post 2/1/2011 Page-21 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE is Guided by Senior Science and Technology Leaders From the Services HPCMP Executing Agent Official HPCMP Advisory Panel Cray Henry (Director) CREATE Program Douglass Post, IPA (PM) Luanne Handley-Blair (Dep PM) Ships Air Vehicles Myles Hurwitz, IPA (PM) Robert Keane (con.) Robert Meakin, IPA (PM) Chris Atwood (Dep) John D’Angelo (PM) Board of Directors Board of Directors Board of Directors *RADM Thomas Eccles, NAVSEA Chief Engineer Dr. Walter Jones, SES, Executive Director, ONR Mr. C.F. Snyder, SES, 33Technical Director, NSWC, Carderock Division Mr. Glen Sturtevant, Director, S&T, PEO, SHIPS Mr. Cray Henry, Director, HPCMP *Board of Director Chairs NDIA CREATE-D.Post 2/1/2011 Page-22 *Dr. Ed Kraft, ST, Chief Technologist, AEDC/CZ Samuel T Crews, U.S. Army Aviation and Missile Research, Development, and Engineering Center Mr. Richard Gilpin, Director, AIR-4.3 NAVAIR Mr. Tom Rudowsky, Director, NAVAIR), (IBST) Dr. Michael Scully, ST, Aeroflightdynamics Directorate (AMRDEC), U.S. Army RDECOM Mr. Cray Henry, Director, HPCMP MG RF Antennas *Dr. Frederick Tokarz, SES, Senior Leader, AF ASC Engineering Directorate Mr. Dennis DeCarlo, Chief Antenna Systems Engineer, NAVAIR Dr. Betsy Delong, ONR RF Antenna contact PEO (SHIPS) Dr. Janet Fender, SES, Chief Scientist, AF Air Combat Command Dr. Mahbub Hoque, Chief Scientist (STCD), CERDEC Dr. Robert Peterkin, ST, Chief Scientist, AFRL Directed Energy Directorate Mr. Cray Henry, Director, HPCMP Distribution Statement A: Approved for public release; distribution is unlimited. Saikat Dey (PM) Board of Directors *Dr. Robert Meakin, PM CREATE Air Vehicles Mr. Myles Hurwitz, CREATE Ships Project Manager Dr. Sudip Sodanjh, Computational Engineering Leader, SNL Dr. John D’Angelo, CREATE RF Antenna Project Manager Mr. Cray Henry, Director, HPCMP 36 Months After Program Start, CREATE Has Delivered Usable Software • 5 beta releases in FY09/10 with V&V and ~150 beta testers: – – – – – • • 10 releases planned for FY10/11 and each succeeding year Helios 2.0, Kestrel 2.0, SENTRI 2.0, NESM 1.0 and IHDE 2.0 plus: – – – • Helios 1.0—Accurate calculation of rotorcraft vortex shedding Kestrel 1.0—Rigid body CFD fixed wing AV with preliminary aeroelastics SENTRI 1.0 and 1.5—Initial RF antenna design and analysis with V&V NESM 0.1—Initial ship shock vulnerability analysis for underwater explosions IHDE 1.0—Initial user interface for ship hydrodynamics Rapid design: RDI 1.0 for Ships, DaVinci 1.0 for Aircraft plus SENTRI 2.0 Components: CAPSTONE 1.0 for geometry and mesh, Firebolt 1.0 for gas turbines Detailed ship hydrodynamics NavyFoam 1.0 (seakeeping, drag, resistance,…) Developing approaches to improve scalability Kestrel vs. F-16 Flight Data M=0.95, Alt.=10,000 ft More accurate vortex shedding Antenna Radiation Near Fields Free Space NDIA CREATE-D.Post 2/1/2011 Page-23 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE-AV Developed An Architecture to Facilitate Multi-physics Integration “Light-Weight” Infrastructure – Highly Scalable and Modular (available as executable) Structural Dynamics Solvers Comprehensive Fluid Structure Interaction Control Laws FEM Aircraft Trim Autopilot On-the-Fly Data Harvesting … Well defined APIs available for use by industry for proprietary modules OR for DoD to leverage innovations from research community Industry Tools Event-Based Infrastructure … Cartesian Propulsion Strand Unstruc Curvilinear Aerodynamics Solvers 6-DOF Research Tools Force/Moment Calculator Stubs represent Application Program Interfaces (APIs)s NDIA CREATE-D.Post 2/1/2011 Page-24 Distribution Statement A: Approved for public release; distribution is unlimited. Early Success Rapid Deployment of EP-3E • • Shadow-Ops: CREATE staff use computational tools to support acquisition programs provide experience and establish connections and value Performed CFD analysis of impact of electronic countermeasure pod for EP-3E flight clearance–Not sufficient time for conventional process (flight tests) – – – • • • Eliminated construction cost of wind tunnel model and tests and need for contractor flying quality report. Provided aircraft flying qualities characteristics within required time frame. Provided data required to issue flight clearance in time for direct deployment. Reduced overall program cost and time Only 1 flying qualities flight test required – Saving between 3–4 flight tests System was deployed in the forward theatre in less than four months instead of twelve New Forward POC: Ms. Ryan Fitzgerald, FQ Engineer NAVAIR 4.3.2.5 Component NDIA CREATE-D.Post 2/1/2011 Page-25 Distribution Statement A: Approved for public release; distribution is unlimited. HPCMP Computer Centers • CREATE is part of the DoD High Performance Computing Modernization Program (HPCMP) • The HPCMP has five large computer centers with 200,000 cores and 2 Peta-FLOPS, and upgrades this capability annually • CREATE is being linked to these centers • HPCMP has a vision to open up their computer centers to the US defense acquisition community (both government and industry) NDIA CREATE-D.Post 2/1/2011 Page-26 Distribution Statement A: Approved for public release; distribution is unlimited. What the HPCMP Offers • • • • • Approximately a dozen large HPC systems from a variety of vendors at five DoD Shared Resource Centers Connectivity to these centers via a capable wide-area network – the Defense Research and Engineering Network Expert help in all aspects of setting up accounts and using HPC systems through the Consolidated Customer Assistance Center and the User Productivity Enhancement, Technology Transfer and Training program Access to many commercial applications software packages for analysis HPCMP is seeking descriptions of industry use cases for high performance computer usage – Problem domains, engineering workflows, user interfaces, software tools, entry and access processes, software application requirements, etc. questionnaire in back of room All members of the DoD RDT&E community are eligible to access and use these capabilities. NDIA CREATE-D.Post 2/1/2011 Page-27 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE Is For Government and US Defense Industry Engineers • CREATE can reduce the acquisition time and risks for complex DoD air vehicles, ships and RF antennas • CREATE adoption by DoD engineering community is important for US defense competitiveness • CREATE, through the NDIA, is seeking your guidance and input to develop tools that will be useful to you and will be adopted and used by you NDIA CREATE-D.Post 2/1/2011 Page-28 Distribution Statement A: Approved for public release; distribution is unlimited. But…. • Constraints exist – Need to maintain US competitive advantage US government control of distribution (e.g. export control), and retention of unlimited use rights – Must provide “Equal and fair access” for all the US defense companies • Support needs can overwhelm the small CREATE development teams (~ 5 to 7 developers per team) – 10 teams for a total program staff ~ 75 professionals (government, industry and academia) – Software is maturing rapidly but is generally not yet ready for heavy usage – CREATE will need to build up the capability to support their user community – Support effort needs to be balanced with development , and we likely need more resources • Tools are not yet mature, but are maturing – Early CY2011, ~ 5 products are releasing version 2 – Late CY2011, other 5 will release either version 1 or 2 – CY2012, repeat with version n+1 NDIA CREATE-D.Post 2/1/2011 Page-29 Distribution Statement A: Approved for public release; distribution is unlimited. CREATE Needs Your Help to Ensure That You Can Use The CREATE Tools • CREATE needs beta testers – Get feedback from your engineers to ensure that we meet your needs and don’t spend years producing tools that are not useful • CREATE needs information on your workflows and design processes – Need to know what our targeted user community needs • CREATE needs product models and validation data – Codes are only a model of nature and must be validated – Need good initial conditions and validation data – CREATE team has security clearances and access to secure computers (export controlled classified computing at all levels) • CREATE needs your recommendations for the technical capability required for your problems – NDIA Participation in technical reviews and discussions of the CREATE tools – Tell us “Have we got it right?” and “How we can make it better.” NDIA CREATE-D.Post 2/1/2011 Page-30 Distribution Statement A: Approved for public release; distribution is unlimited. Additional References NDIA CREATE-D.Post 2/1/2011 Page-31 Distribution Statement A: Approved for public release; distribution is unlimited. Physics-based Engineering Software Helped The US Win Cold War. • Nuclear weapons are complex, expensive, and hard to test • ~ 5 to 10 tests per system • DOE NNSA uses computational tools for: • Design development, optimization, & analysis. • DOE NNSA labs own the biggest supercomputers Testing 1,000,000 GigaFlops/s Computational Design 2010 Weapon Capability Improved safety Improved robustness NIF Test ban Underground Air Tests NDIA CREATE-D.Post 2/1/2011 Page-32 Computer Power Improved yield to weight Increasing Computational Design Capability MIRV (even lighter, smaller) SLBM Improvements over time: (even lighter, • Solution methods smaller) • Spatial resolution • Temporal resolution ICBM (Lighter,smaller) • Geometric fidelity •1-D to 2-D to 3-D Heavy • Physics models Hydrogen Bombs 0.000000001 • ……. Atomic Bombs GigaFlops/s 1945 Distribution Statement A: Approved for public release; distribution is unlimited. It Takes a Village! • Sponsors Provide mission, resources and support − Designers—End-users Engineers to use the tools to design products − Codes Takes a good team ~ 10 years and ~ $100M to develop a complex code − V&V Dedicated experiments and tests − Computers Capability to develop codes and run the problems quickly and conveniently NDIA CREATE-D.Post 2/1/2011 Page-33 Distribution Statement A: Approved for public release; distribution is unlimited. Developing These Codes Takes A Large Team A Long Time 2003 ~20 From: D. Post, R. Kendall and E. Whitney, “Case Study of the Falcon Code Project”, Proceedings of the Workshop on Software Engineering for High Performance Computing, International Conference on Software Engineering, May 15, 2005, St. Louis, Missouri. NDIA CREATE-D.Post 2/1/2011 Page-34 Distribution Statement A: Approved for public release; distribution is unlimited. Development of Science-Based Software is Complex! NDIA CREATE-D.Post 2/1/2011 Page-35 Distribution Statement A: Approved for public release; distribution is unlimited. Development of Science-Based Software is Risky! NDIA CREATE-D.Post 2/1/2011 Page-36 Distribution Statement A: Approved for public release; distribution is unlimited. Incremental Development and Deployment in Annual Releases 10-12 years Fielding Capability,… Rel 11 Rel 10 ExaFLOP (1018 FLOPs) computers Rel 9 Stages 6-9 years Rel 8 Rel 7 PetaFlop (1015 FLOPs) computers Rel 6 Rel 5 Rel 4 Improve scalability,… Rel 2 years 1-5 Rel 3 Rel 1 1 NDIA CREATE-D.Post 2/1/2011 Page-37 Physics integration tests,… Develop Initial Requirements and Plans 4 Year 8 Distribution Statement A: Approved for public release; distribution is unlimited. 12 Kestrel SE Docs NDIA CREATE-D.Post 2/1/2011 Page-38 Distribution Statement A: Approved for public release; distribution is unlimited. 38 Helios SE Docs NDIA CREATE-D.Post 2/1/2011 Page-39 Distribution Statement A: Approved for public release; distribution is unlimited. 39 Firebolt SE Docs NDIA CREATE-D.Post 2/1/2011 Page-40 Distribution Statement A: Approved for public release; distribution is unlimited. 40 DaVinci SE Docs NDIA CREATE-D.Post 2/1/2011 Page-41 Distribution Statement A: Approved for public release; distribution is unlimited. 41