Mesoscopic
Lattice Boltzmann methods
for Complex Fluids
Computational Steering
• The lattice Boltzmann method is a
mesoscale method for simulating complex
fluids. This is something traditional CFD
cannot do.
• We have a parallel and efficient
implementation of the lattice Boltzmann
method.
• Simulations that require greater computational resources also require
increasingly sophisticated and complex tools for the analysis and
management of the output of the simulations.
A
► Computational Steering enables the user to influence and interact with
the otherwise sequential simulation and analysis process.
• As a minimum, Computational Steering improves utilization of
computational resources and enhances a scientist's productivity.
HPCx Gold medal
on 1024 processors. Can run 10243 cells.
Gold medal will allow us to make full use of
HPCx’s Capability Computing Initiative
B
C
Computational Steering –
Lattice Boltzmann
• Steering has proved useful for detecting and studying topological
changes in vortex cores
– Once a change is detected we can return to the last checkpoint and improve
either the spatial or temporal resolution of the simulation
• The figure below shows how steering through parameter space
allows a computational scientist to uncover different binary phases
• A central theme of RealityGrid is the facilitation of distributed and
interactive exploration of physical models and systems through
computational steering of parallel simulation codes and simultaneous
on-line, high-end visualisation.
• Can simulate
– flow in porous media: industrial applications
e.g. hydrocarbon recovery process (A).
Using a 10243 simulation we can now model
~1cm3 of rock
– non-equilibrium process of self-assembly of
amphiphilic fluids into equilibrium liquidcrystalline cubic mesophases.(e.g. gyroid
phase B).
– sheared equilibrium mesophases (C).
• For more details on computational steering in computational science:
– J Chin, J Harting, S Jha, P V Coveney, A R Porter & S M Pickles, Contemporary
Physics (2003), in press
Mesoscopic
• Both codes are fully parallelised using MPI and use both the VTK
graphics library and the RealityGrid steering library
Evolution of a
(2,3) torus knot
using the MTLB
model on a 1003
grid with a
viscosity of
0.0002 lattice
units.
• Centre for Computational Science, UCL (established 2003)
• Research centre headed by Professor Peter V. Coveney (PI for RealityGrid)
– Also involved in the new EPSRC e-Science Pilot Project in Integrative Biology
– Approximately 15 full time members
– Range of problems: theoretical & computational science, computer science,
distributed computing
• Our different computational techniques span time and length-scales from the macro-,
through the meso- and to the nano- and microscales. We are committed to studying
new approaches (e.g. the Grid) and techniques that bridge these scales.
• For more information, please see
These and many other of our
simulations are done on the
Pittsburgh Supercomputer Centre’s
LeMieux, a 3,000 processor
machine via 200,000SU grant
end to end distance
Microscopic
Macroscopic to Microscopic
– Explicit solvent
– Polymer made of generic spherical
monomers held together by non-linear
springs
– see R.Delgado-Buscalioni & P.V.Coveney
J. Chem. Phys. 119 2 978-987 (2003)
– Our hybrid scheme and loosely coupled
models in general are good candidates for
solving on the grid
Running applications on the
UK Level 2 Grid
• We have used the following resources on the L2G
– CSAR SGI Origins
– Viking LeSC Linux cluster
RealityGrid deployed
on the Level 2 Grid
SGI
Op
enG
L Vi
Simulation
Data
GLOBUS-IO
• Systems under investigation:
– MHC complex (Immune Response)
– DHPS (Drug resistance)
– DNA (Drug binding)
HPCx Bronze medal
on 256 processors
can run 100k atom
systems
• NAMD2 has been successfully interfaced with the
RealityGrid steering library
– Our own SGI Onyx2 (Dirac)
http://www.realitygrid.org/L2G
Visualization
SGI Onyx
Vtk + VizServer
www.realitygrid.org
• An atomistic description of biological molecules is necessary to model their
dynamics and thermodynamics
• We use NAMD2 on large, tightly-coupled parallel machines to investigate
several biomolecular problems
We have developed and tested a new
hybrid algorithm comprising
This hybrid algorithm models a single
polymer tethered to a wall undergoing
shear flow.
Lamellar phase:
surfactant bilayers
between water layers.
• A Grid infrastructure that permits the coordination of
heterogeneous and distributed computing resources provides
a natural testbed for demonstrating the effectiveness of
computational steering.
www.chem.ucl.ac.uk/ccs
Biomolecular modelling
A Hybrid Multiscale Modelling Scheme –
Interfacing the Macro and Micro
– Molecular Dynamics (MD)
– Computational Fluid Dynamics (CFD)
– Buffering region: swap fluxes of mass,
momentum and energy
– Validated hybrid model results against
established MD (results below)
Self-organization
starts.
CCS
With Professor B. Boghosian’s group at Tufts University.
– Implemented a multiple time-scale relaxation lattice Boltzmann (MTLB)
model for a single phase fluid
– Developed a pseudospectral Navier-Stokes solver
Initial condition:
Random water/
surfactant mixture.
Cubic micellar phase,
low surfactant density
gradient.
Rewind and
restart from
checkpoint.
Lattice Boltzmann Simulations
of Vortex Knot Evolution
• Vorticity is the curl of the hydrodynamic velocity, and is strongest
at the core of a swirling region of fluid
• At high Reynolds number, regions of high vorticity tend to form
filamentary structures
• We study the dynamical behaviour of vortex knots and links
Cubic micellar phase,
high surfactant density
gradient.
u
mm
Co
program lbe
use
lbe_init_module
use
lbe_steer_module
use
lbe_invasion_module
Application
with RealityGrid
Steering API
zSe
rve
ng
eri
e
e
t
S
gR
n
i
us
on rary
i
t
a
lib
nic
r
t ee
GS
g
rin
Laptop
Vizserver Client
Steering GUI
GLOBUS used to
launch jobs
ReG steering GUI
• However, there is currently little incentive to run
applications on the L2G.
• Access to L2G resources is still far from transparent
(the key feature of a grid)
– It is hard to get answers to simple questions such
as which resources are available?
– Support is limited because most sysadmins do not
have much experience with GLOBUS
• The system is still in prototype stage.
• Need to foster much more involvement from the user
community if the grid is to take off.
“As you run into bumps in the road, remember that you are
a Grid pioneer. Do not expect all the roads to be paved (do
not expect roads). Grids do not yet run smoothly.”
From the Globus Quickstart Guide