Advanced Engineering
Simulation and its
future at EPCC
Mark Parsons, EPCC
Dr Mark Parsons
Commercial Director, EPCC
m.parsons@epcc.ed.ac.uk
+44 131 650 5022
Outline
• Introduction to the EPCC Industry Hub
• Recent advanced engineering simulation projects
• Challenges facing simulation in the next decade
eDIKT 2010
EPCC in 2010
• Supercomputing Centre at The University of Edinburgh
• 20 years old
• 75 staff
Facilities
– highly experienced
– wide range of skills
Technology
Transfer
• Multidisciplinary
• Multi-funded
– turnover £4.5 million
– 95% from external sources
– 50% of turnover involves industry
European
Coordination
• Provides National HPC services
– HECToR is hosted at Advanced Computing Facility
eDIKT 2010
HPC
Research
Training
Visitor
Programme
EPCC Industry Hub
• Cycles, Software and Skills
• One stop shop for industry
supercomputing
• Unique range of services
– Cycles – provision of compute and data resources
– Software – provision of simulation and
bespoke applications to meet industry challenges
– Skills – understanding industry problems and finding solutions
• Benefits
–
–
–
–
Scotland – supporting local industry – creating jobs and growth
Inward investment – attracting companies eg. Xilinx, Cray
International leadership – worldwide research collaborations
Creating an important facility where we work together with end-user
companies and leading HPC hardware and software vendors
eDIKT 2010
HPC Adopter Programme
• Too few companies benefit from modelling and simulation
• Big hurdles to overcome for the new HPC user
– Even for high-tech companies
• HPC Adopter Programme is designed to tackle this
– Initial small 2-4 month projects designed to introduce companies to
benefits of HPC
– Initial access to HECToR and other HPC systems donated by EPCC
– Companies pay for staff effort only
– For Scottish companies Scottish Enterprise may also support staff
costs through State Aid to company
– To date SE have supported 3 HPC Adopter Programme pilots
– Other companies have already benefitted from programme
• Long term virtuous circle of repeat business
eDIKT 2010
Deep Casing Tools – Turbocaser
•
•
•
Deep Casing Tools is a privately owned
Deep Casing Tools Turbocaser uses a
SME in Aberdeen.
motor powered by drilling-mud to ream oil
Company focus is on innovative design
wells prior to pipe installation.
combined with precision engineering that
•
enable development and manufacture of
the next generation of casing and
Important to understand and optimise the
performance of the multi-stage motor.
•
Drilling mud is a non-Newtonian fluid with
completion tools for the oil and gas
Reynold's Numbers between 4,000 and
industry.
20,000 and densities typically between
1.1 and 1.4 g/cm 3.
•
Very computationally expensive to
performa multi-stage design studies.
•
Ideal application for HPC: project
modelled mud flow through various
Turbocaser turbine designs using
HECToR and
OpenFOAM
eDIKT 2010
OpenFOAM-based design study
• Built mesh from existing
Deep Casing Tools'
AutoCAD designs
• Initial single stage
simulations
• Multistage design study
OpenFOAM
eDIKT 2010
•
Free, open source CFD software
package
•
Large user base across commercial
and academic organisations.
•
Solves complex fluid flows involving
chemical reactions, turbulence and
heat transfer, solid dynamics and
electromagnetics.
•
Already ported to HECToR, also
runs on commodity resources
Project outcome
• Impact for company
• Impact for EPCC
– Validation of basic design
– Better understanding of product
behaviour
– Optimisation of pressure drop
versus torque across multistage
turbine
– Support for future product design
and development
eDIKT 2010
– Demonstration of value of HPC
to real world problem
– Increased staff experience of
Computational Fluid Dynamics
– Extended use of OpenFOAM on
complex problem
– Potential for further use of
HECToR eg. rotating mesh
Prospect – Wind & Wave Modelling
The Company:
Prospect is an engineering design and
analysis provider to the world energy
industries. Headquartered in Aberdeen,
Prospect was founded by in 1999 and has
grown rapidly since to become part of a 300
strong Group with operations spanning six
continents
The Problem:
•
Simulations exist of blades, turbine, wind on
tower and waves on tower.
•
Combining these simulations, using different
software packages, is very difficult
•
Massive computational resources required to
keep all simulations sychronised
•
Ideal application for HPC: EPCC asked to
couple simulations and run result on HECToR
© Copyright Davagh and licensed for reuse under this Creative Commons Licence
eDIKT 2010
HPC solution: Coupled Simulation
•
Create polyhedral / tetrahedral
mesh to represent physical
objects
•
Simulate action of waves, ocean
current and wind simultaneously
•
Understand combined complex
stresses on structure
Waves
Tools:
Simulia-Abaqus Finite
Element Analysis
Current
StarCCM Computational
Fluid Dynamics
eDIKT 2010
Tower
Wind
Prospect Summary
Impact for Company
Impact for EPCC
• First full simulation of entire deep-
• Breakthroughs in coupled
sea wind turbine
• In-house capacity to run 1 variation
per night – HECToR can run 500
variations simultaneously
• Understanding impact of design
changes, highlights efficiency
savings
simulations – FEA and CFD
• Gained experience of two
industry codes, ported to HPC
platforms
• Potential to perform further work
for client and software vendors
© Copyright Davagh and licensed for reuse under this Creative Commons Licence
eDIKT 2010
Challenges facing modelling & simulation
• We are at a complex juncture in the history of
supercomputing
• For the past 20 years supercomputing has “hitched a lift” on
the microprocessor revolution driven by the PC
• Hardware has been surprisingly stable
• EPCC in 1994 had the 512 processor Cray T3D system
– 0.0768 TFlops peak
• EPCC in 2010 retired the 2,560 processor IBM HPCx system
– 15.36 TFlops peak – 200 x faster but only 5 x more processors ...
• But now we have a problem ...
eDIKT 2010
0%
eDIKT 2010
TNC
Milipeia
Galera
HERA
Jump
hww SX-8
murska.csc.fi
ZAHIR
Huygens
BCX
HLRB II
HPCx
Neolith
Platine
Stallo
MareNostrum
Palu
HECToR
Legion
XC5
Louhi
Jubl
Jugene
Average job size in Europe today
100%
90%
80%
70%
60%
>2048
513-2048
50%
129-512
33-128
<32
40%
30%
20%
10%
Data from PRACE Project
Multicore and Moore’s Law
• In 2005 microprocessor clock speeds peaked around 4GHz
• Exponential increase in individual core performance has
ceased – in fact individual performance is reducing
– Serial codes are now running slower on your new system
• Theoretical performance of multi-core microprocessors
continues to rise
• Moore’s Law is alive and well
– Relates to number of transistors on an area of silicon not clock speed
• But – today’s microprocessors are more and more difficult to
use
– This can only get worse in coming years
eDIKT 2010
Meeting user needs?
• Hardware is leaving many HPC users and codes behind
• Majority of codes scale to less than 512 cores
– these will soon be desk-side systems
• Less than 10 codes in EU today will scale on capability
systems with 100,000+ cores
– Soon HECToR will have 70,000+ cores
– Germany’s Jugene system already has 294,912 cores
• Many industrial codes scale very poorly – some codes will
soon find a laptop processor a challenge!
• Much hope is pinned on accelerator technology
– But this has its own set of parallelism and programming challenges
eDIKT 2010
The hardware software disconnect
• We have lived through a golden age of easy programmability
and relaxed parallelism
• In the future codes will have to be highly scalable
– But many were designed in a different age
• Main parallel programming paradigms were settled on 15-20
years ago
– Many codes are written in Fortran, C or C++
– Most use either MPI or OpenMP
– They use mathematical algorithms developed for 10s or 100s of
processors – not hundreds of thousands
• Exascale systems now being planned will have many
millions of cores
eDIKT 2010
Reconnecting
• We need to focus much more effort on applications
programming
• Rethinking the mathematics behind modelling on massively
parallel systems
• Redesigning and re-implementing simulation and modelling
codes
• Building new languages and methods of expressing and
harnessing parallelism
• Unless we do this the justification for buying larger and more
powerful systems will become increasingly false
eDIKT 2010
Final comments
• The software area is huge
• We have challenges in
–
–
–
–
–
Scaling
Languages
Tools
Power consumption
Accelerators
–
–
–
–
–
ISVs
Algorithms
Programming models
Memory management
Communications etc
• The next few years are going to be very exciting in HPC
eDIKT 2010
Questions?
eDIKT 2010