Computational Science at Edinburgh
From Excellence to Enterprise
Dr Arthur Trew
Director
on the shoulders of giants
 for centuries science has relied on experiment and theory, but
– theory is best suited to a reductionist approach
– experiment is limited in its range of applicability
 there is a need for a complementary approach: simulation 
study of emergent phenomena
Theory- Experiment Greece
Italy
400 BC
1,500 AD
Simulation Edinburgh
1,980 AD
For problems which are:
- too large/small
- too fast/slow
- too complex
- too expensive ...
arth
52.8
becoming the best1.0 D
 in 1990 Edinburgh established EPCC as its focus for HPC
simulation
– following a decade of research in Physics
 in 1992 EPCC set itself a mission “to be Europe’s top HPC
centre within 5 years”
– we succeeded
– today, that success is benefiting Scottish research and industry
.4 Ea
1.68
2.3 e
1.0 S
2.3 N
3 GC
0.2 G
0.35
 in 2001 Edinburgh & Glasgow created NeSC to extend that
vision
– over the past 18 months EPCC & NeSC have brought in £65M of
research grants/contracts
• of which £2.3M from SHEFC for eDIKT
– Edinburgh & Glasgow backed this with £3M
– … and foundation departments lead major UK Grid projects
0.6 G
arth
UoE
UoG
(AK’
what’s that to me?
 as computer performance improves, the range of applications
increases
100
90
Tflops capability
80
70
60
50
HPCx
£53M: 3 machines
40
30
eddy
cells
20
resolution materials
oceans
design
whole
10
protein drug design
nanostructures
aircraft
structures
0
0
1
2
3
4
5
year
whole earth
climate
organs
solar weather
astroplasmas
6
7
8
linking to data
 but simulation is only part of the story …
 linking computer modelling to experiment is vital to
the interpretation of many large experiments, eg LHC
 … and with big experiments set to generate
Pbyte/year data management is critical
 … but we have the web, don’t
we?
EPCC overview
Technology Transfer
Academic:
o National HPC facilities
o Research
o Support
Industry:
o Projects
o Consultancy
o Standards
Training:
o Academia
o Industry
European leadership:
o Visitor programmes
o Technology Transfer
o Strategic Planning
 70 staff, £3M turnover
 … possible through the “win-win-win” model
HPCx rationale
UK academic research is increasingly
dependent upon high-end compute facilities
recent technological and Grid advances
highlighted the need to upgrade UK resources
HPCx objectives are thus:
– aim “to deliver optimum service resulting in worldleading science”
– address “the problems involved in scaling to the
capability levels required”
What is HPCx?
 Consortium of leading UK organisations committed to
creating and managing the new resource for the next
6 years
– led by University of Edinburgh
 multi-stage project to deliver a world-class academic
computing resource, the largest in Europe
 £54M/$100M budget
 Grid-enabled, a key component in the UK e-Science
programme
HPCx Consortium Members
 UoE HPCx Ltd – wholly-owned subsidiary of the
University of Edinburgh and lead contractor
 EPCC, University of Edinburgh
 Daresbury Laboratory, CCLRC
 IBM
 the University of Edinburgh is one of the top 5 research
universities in the UK
 EPCC is the leading computer centre in Europe, bridging
the gap between academia and industry
 … and provides both HPC and novel computing solutions
to a wide range of problems and users
 long experience of providing national HPC services
including:
– Meiko Computing Surfaces
– Thinking Machines CM200
– Cray T3D/T3E
Daresbury Laboratory
 A multi disciplinary research lab with
over 500 people
 Provides large-scale research
facilities both for UK academic and
industrial research communities
 Runs the UK’s Collaborative
Computational Projects
 Daresbury hosts the HPCx
hardware
 IBM provides the technology for HPCx
 Long standing involvement in HPC including the
development of a number of ASCI machines and 4 of the
top dozen machines in the 21st TOP500 list:
– ASCI White: Rmax = 7.3 TFlop/s
– SP Power3 (6656 Processors): Rmax = 7.3 TFlop/s
– xSeries (1920 Processors): Rmax = 6.6 TFlop/s
– HPCx (1280 processors): Rmax = 3.2 TFlop/s
 IBM has the long term technology road map essential to
a 6 year project such as HPCx
HPCx in Place
HPCx: Phase 1
 System will be commissioned in three main stages
Phase 1 covering 2002-2004 consists of:
– 40 Regatta-H SMP nodes, 1280 processors
– Peak performance 6.6 TFlop/s, with 3.4 Tflop/s sustained
Linpack
• currently 16th in the Top500
– 1.28 TB total memory capacity
– Over 50 TB of storage capacity
– Double plane Colony switch with total
peak bandwidth of 250 MB/s per
processor
The e-Science Grid
CeSC
(Cambridge)
HPCx Phases 2 & 3
 Phase 2 (2004-2006)
– aiming for 6 TFlop/s sustained on Linpack and 2.5 TFlop/s on
sPPM
– O(48) Regatta-H+ SMP nodes
– interconnect upgraded to Federation switch
– doubling of I/O and storage
• already built a cluster with 8 Regatta-H+ frames and a prerelease Federation switch
• undertaking a phased upgrade during 1H2004
 Phase 3 (2006-2008)
– target of 12 TFlop/s sustained on Linpack
– may be additional nodes or alternative technology
HPCx Science Support
18 staff in 5 dual-centre functional support teams
Outreach
Life sciences
New applications
Applications Support
Users
Helpdesk
Training
Liaising with users
Terascaling
Capability applications
Scalable algorithms
Performance optimisation
Software Engineering
Underpinning technology
Grid/e-Science
Systems & Networking
Flexible and responsive capability computing service
Smooth transitions between phases
Technology
CPU Usage by Job Size
80%
128
Jan-03
Feb-03
Mar-03
Apr-03
May-03
Jun-03
Jul-03
Aug-03
Sep-03
Oct-03
Nov-03
Dec-03
256
1000000
>1024
2000000
40%
512
1500000
0%
AUs
HPCx Status: Usage
Utilisation of Capability Region
100%
60%
1024
20%
64
500000
32
0
8
Jan-03
Feb-03
Mar-03
Apr-03
May-03
Jun-03
Jul-03
Aug-03
Sep-03
Oct-03
Nov-03
Dec-03
16
HPCx Status: Application Areas
 currently 28 active consortia, and over 450 users
Usage by Application Area
Atomic &
Molecular Physics
Science Support
Materials
Chemistry
Engineering
Environmental
Science
 Life Sciences outreach activity supported by IBM
Atomic and Molecular Physics
 The UK Multiphoton, Electron Collision and Bose Einstein
Condensates (MECBEC) HPC Consortium
 Two flagship projects model two-electron atoms (helium)
and molecules (hydrogen) exposed to intense, ultra-short
laser pulses
 Modelling involves the grid solution of multidimensional, time-dependent partial differential
equations)
 Requires large amounts of computing power
 Visualisation techniques crucial in
extracting information
Simulation of Double Ionization of laser-driven helium performed at
Queen’s University Belfast
Environmental Science: POLCOMS
 POLCOMS is a multi-disciplinary model developed at the
Proudman Oceanographic Laboratory
 … a 3-D hydrodynamic model
integrating coasts and oceans using a
wide range of associated models
 POLCOMS is a step towards real
time modelling of coastal zones,
enabling better analysis of impacts
to, and sustainability of, the marine
environment
A simulation of Chlorophyll density in UK
waters using the POLCOMS model
Material Science: Crystal
 CRYSTAL computes electronic structure and related properties of
periodic systems
 Developed jointly by Daresbury and the University of Turin
 A Fortran90 and MPI program that performs Hartree-Fock, density
functional and other approximation calculations
 On HPCx, CRYSTAL used to calculate the structure of the Crambin
molecule, the largest Hartree-Fock calculation ever converged (1284
atoms)
– next, the Rusticyanin molecule (6284 atoms)
Engineering: UKTC
UK Turbulence Consortium is developing world
leading turbulence simulation codes using HPCx
essential that they can easily facilitate the
scientific usage of the resulting data
the calculated data are transferred from HPCx to a
remote site for analysis
so, the Grid is becoming increasingly important
Chemistry: RealityGrid
 RealityGrid is a UK collaboration
– which aims to grid-enable the realistic modelling and
simulation of complex condensed matter structures at
the meso and nanoscale levels
 HPCx terascaling team has worked on parallelisation &
optmisation of RealityGrid codes, such as LB3D
 RealityGrid also uses the Grid for data transfer,
computational steering, and remote visualisation
– using Globus on HPCx
– TeraGyroid won award at SC2003
The Vision
we want Edinburgh to lead the e-science
revolution
– to become the European equivalent of one of the big US centres, eg
San Diego Supercomputer Center

