HPCx: Multi-Teraflops in the UK
A World-Class Service for World-Class Research
Dr Arthur Trew
Director
HPCx rationale
UK academic research is increasingly
dependent upon high-end compute facilities
recent technological and Grid advances
highlighted the need to upgrade UK resources
HPCx objectives are thus:
– aim “to deliver optimum service resulting in worldleading science”
– address “the problems involved in scaling to the
capability levels required”
What is HPCx?
 Consortium of leading UK organisations committed to
creating and managing the new resource for the next
6 years
– led by University of Edinburgh
 multi-stage project to deliver a world-class academic
computing resource, the largest in Europe
 £54M/$100M budget
 Grid-enabled, a key component in the UK e-Science
programme
HPCx Consortium Members
 UoE HPCx Ltd – wholly-owned subsidiary of the
University of Edinburgh and lead contractor
 EPCC, University of Edinburgh
 Daresbury Laboratory, CCLRC
 IBM
 the University of Edinburgh is one of the top 5 research
universities in the UK
 EPCC is the leading computer centre in Europe, bridging
the gap between academia and industry
 … and provides both HPC and novel computing solutions
to a wide range of problems and users
 long experience of providing national HPC services
including:
– Meiko Computing Surfaces
– Thinking Machines CM200
– Cray T3D/T3E
EPCC overview
Technology Transfer
Academic:
o National HPC facilities
o Research
o Support
Industry:
o Projects
o Consultancy
o Standards
Training:
o Academia
o Industry
European leadership:
o Visitor programmes
o Technology Transfer
o Strategic Planning
Daresbury Laboratory
 A multi disciplinary research lab with
over 500 people
 Provides large-scale research
facilities both for UK academic and
industrial research communities
 Runs the UK’s Collaborative
Computational Projects
 Daresbury hosts the HPCx
hardware
 IBM provides the technology for HPCx
 Long standing involvement in HPC including the
development of a number of ASCI machines and 4 of the
top dozen machines in the 21st TOP500 list:
– ASCI White: Rmax = 7.3 TFlop/s
– SP Power3 (6656 Processors): Rmax = 7.3 TFlop/s
– xSeries (1920 Processors): Rmax = 6.6 TFlop/s
– HPCx (1280 processors): Rmax = 3.2 TFlop/s
 IBM has the long term technology road map essential to
a 6 year project such as HPCx
HPCx in Place
HPCx: Phase 1
 System will be commissioned in three main stages
Phase 1 covering 2002-2004 consists of:
– 40 Regatta-H SMP nodes, 1280 processors
– Peak performance 6.6 TFlop/s, with 3.4 Tflop/s sustained
Linpack
• currently 16th in the Top500
– 1.28 TB total memory capacity
– Over 50 TB of storage capacity
– Double plane Colony switch with total
peak bandwidth of 250 MB/s per
processor
The e-Science Grid
CeSC
(Cambridge)
HPCx Phases 2 & 3
 Phase 2 (2004-2006)
– aiming for 6 TFlop/s sustained on Linpack and 2.5 TFlop/s on
sPPM
– O(48) Regatta-H+ SMP nodes
– interconnect upgraded to Federation switch
– doubling of I/O and storage
• already built a cluster with 8 Regatta-H+ frames and a prerelease Federation switch
• undertaking a phased upgrade during 1H2004
 Phase 3 (2006-2008)
– target of 12 TFlop/s sustained on Linpack
– may be additional nodes or alternative technology
HPCx Science Support
18 staff in 5 dual-centre functional support teams
Outreach
Life sciences
New applications
Applications Support
Users
Helpdesk
Training
Liaising with users
Terascaling
Capability applications
Scalable algorithms
Performance optimisation
Software Engineering
Underpinning technology
Grid/e-Science
Systems & Networking
Flexible and responsive capability computing service
Smooth transitions between phases
Technology
CPU Usage by Job Size
80%
128
Jan-03
Feb-03
Mar-03
Apr-03
May-03
Jun-03
Jul-03
Aug-03
Sep-03
Oct-03
Nov-03
Dec-03
256
1000000
>1024
2000000
40%
512
1500000
0%
AUs
HPCx Status: Usage
Utilisation of Capability Region
100%
60%
1024
20%
64
500000
32
0
8
Jan-03
Feb-03
Mar-03
Apr-03
May-03
Jun-03
Jul-03
Aug-03
Sep-03
Oct-03
Nov-03
Dec-03
16
HPCx Status: Application
Areas
 currently 28 active consortia, and over 450 users
Usage by Application Area
Atomic &
Molecular Physics
Science Support
Materials
Chemistry
Engineering
Environmental
Science
 Life Sciences outreach activity supported by IBM
Atomic and Molecular
Physics
 The UK Multiphoton, Electron Collision and Bose Einstein
Condensates (MECBEC) HPC Consortium
 Two flagship projects model two-electron atoms (helium)
and molecules (hydrogen) exposed to intense, ultra-short
laser pulses
 Modelling involves the grid solution of multidimensional, time-dependent partial differential
equations)
 Requires large amounts of computing power
 Visualisation techniques crucial in
extracting information
Simulation of Double Ionization of laser-driven helium performed at
Queen’s University Belfast
Environmental Science:
POLCOMS
 POLCOMS is a multi-disciplinary model developed at the
Proudman Oceanographic Laboratory
 … a 3-D hydrodynamic model
integrating coasts and oceans using a
wide range of associated models
 POLCOMS is a step towards real
time modelling of coastal zones,
enabling better analysis of impacts
to, and sustainability of, the marine
environment
A simulation of Chlorophyll density in UK
waters using the POLCOMS model
Material Science: Crystal
 CRYSTAL computes electronic structure and related properties of
periodic systems
 Developed jointly by Daresbury and the University of Turin
 A Fortran90 and MPI program that performs Hartree-Fock, density
functional and other approximation calculations
 On HPCx, CRYSTAL used to calculate the structure of the Crambin
molecule, the largest Hartree-Fock calculation ever converged (1284
atoms)
– next, the Rusticyanin molecule (6284 atoms)
Engineering: UKTC
UK Turbulence Consortium is developing world
leading turbulence simulation codes using HPCx
essential that they can easily facilitate the
scientific usage of the resulting data
the calculated data are transferred from HPCx to a
remote site for analysis
so, the Grid is becoming increasingly important
Chemistry: RealityGrid
 RealityGrid is a UK collaboration
– which aims to grid-enable the realistic modelling and
simulation of complex condensed matter structures at
the meso and nanoscale levels
 HPCx terascaling team has worked on parallelisation &
optmisation of RealityGrid codes, such as LB3D
 RealityGrid also uses the Grid for data transfer,
computational steering, and remote visualisation
– using Globus on HPCx
– TeraGyroid won award at SC2003
Summary
HPCx is the new UK HPC resource
– capability computing for world-leading science
– largest academic computer in Europe
Series of IBM pSeries clusters
– 6.6TF  12TF  22TF
– remaining very competitive until 2008+
Very successful first year
HPCx: Supporting the
Future of UK Research
Summary: Website
 For further information:
http://www.hpcx.ac.uk
The front page of
the HPCx website