John Brooke (University of Manchester)
Peter Coveney PI RealityGrid (University College London)
Stephen Pickles (University of Manchester)
Thanks also to the other RealityGrid co-Investigators
John Darlington (Imperial College)
Steve Kenny and Roy Kalawsky (Loughborough University)
John Gurd (University of Manchester)
Mike Cates (University of Edinburgh)
Adrian Sutton (University of Oxford)

2
Mission: “Using Grid technology to closely couple high performance computing, high throughput experiment and visualization, RealityGrid will move the bottleneck out of the hardware and back into the human mind.”
Scientific aims:
ƒ to predict the realistic behavior of matter using diverse simulation methods (Lattice Boltzmann, Molecular
Dynamics and Monte Carlo) spanning many time and length scales
ƒ to discover new materials through integrated experiments.
http://www.realitygrid.org

Academic
ƒ University College London
ƒ Queen Mary, University of London
ƒ Imperial College
ƒ University of Manchester
ƒ University of Edinburgh
ƒ University of Oxford
ƒ University of Loughborough
Industrial
ƒ Schlumberger
ƒ Edward Jenner Institute for
Vaccine Research
ƒ Silicon Graphics Inc
ƒ Computation for Science
Consortium
ƒ Advanced Visual Systems
ƒ Fujitsu
3 http://www.realitygrid.org

“Instruments”:
XMT devices, LUSI,…
Storage devices
Grid infrastructure
(Globus, Unicore,…)
HPC resources
Scalable MD, MC, mesoscale modelling
User with laptop/PDA
(web based portal)
Steering
ReG steering API
Performance control/monitoring
4
Visualization engines VR and/or AG nodes
Moving the bottleneck out of the hardware and into the human min d…

5
ƒ Grid-enabled (Globus, UNICORE)
ƒ Component-based, service-oriented
ƒ Steering is central
– Computational steering
– On-line visualisation of large, complex datasets
– Feedback-based performance control
– Remote control of novel, grid-enabled, instruments (LUSI)
ƒ Advanced Human-Computer Interfaces (Loughborough)
ƒ Everything is (or should be) distributed and collaborative
ƒ High performance computing, visualization and networks
ƒ All in a materials science domain
– multiple length scales, many "legacy" codes (Fortran90, C, C++, mostly parallel) http://www.realitygrid.org

6
ƒ Code (LB3D) written in Fortran90 and parallelized using MPI.
ƒ Scales linearly on all available resources.
ƒ Fully steerable.
ƒ Future plans include move to parallel data format PHDF5.
ƒ Data produced during a single large scale simulation can exceed hundreds of gigabytes or even terabytes.
ƒ Simulations require supercomputers
ƒ High end visualization hardware and parallel rendering software (e.g. VTK) needed for data analysis.
3D datasets showing snapshots from a simulation of spinodal decomposition: A binary mixture of water and oil phase separates. ‘Blue’ areas denote high water densities and
‘red’ visualizes the interface between both fluids.
http://www.realitygrid.org

7
Cubic micellar phase, high surfactant density gradient.
Initial condition:
Random water/ surfactant mixture.
Self-assembly starts.
Rewind and restart from checkpoint.
Cubic micellar phase, low surfactant density gradient.
Lamellar phase: surfactant bilayers between water layers.
http://www.realitygrid.org

8
ƒ Terascale simulations can generate in days data that takes months to understand
ƒ Problem: to efficiently explore and understand the parameter spaces of materials science simulations
ƒ Computational steering aims to short circuit post facto analysis
– Brute force parameter sweeps create a huge data-mining problem
– Instead, we use computational steering to navigate to interesting regions of parameter space
– Simultaneous on-line visualization develops and engages scientist's intuition
– thus avoiding wasted cycles exploring barren regions, or even doing the wrong calculation http://www.realitygrid.org

9
ƒ We instrument (add "knobs" and "dials" to) simulation codes through a steering library
ƒ Library provides:
– Pause/resume
– Checkpoint and windback
– Set values of steerable parameters
– Report values of monitored (read-only) parameters
– Emit "samples" to remote systems for e.g
. on-line visualization
– Consume "samples" from remote systems for e.g.
resetting boundary conditions
ƒ Images can be displayed at sites remote from visualization system, using e.g.
SGI OpenGL VizServer, or Chromium
ƒ Implemented in 5+ independent parallel simulation codes, F90, C, C++ http://www.realitygrid.org

ƒ Provide right level of steering functionality to application developer
ƒ Instrumentation of existing code for steering
– should be easy
– should not bifurcate development tree
ƒ Hide details of implementation and supporting infrastructure
– eg. application should not be aware of whether communication with visualisation system is through filesystem, sockets or something else
– permits multiple implementations
– application source code is proof against evolution of implementation and infrastructure
10 http://www.realitygrid.org

11
Client
Steering library
Simulation
Steering library data transfer
Steering library
Display
Display
Display http://www.realitygrid.org

Communication modes:
• Shared file system
• Files moved by UNICORE daemon
• GLOBUS-IO
• SOAP over http/https
Client
Steering library
Data mostly flows from simulation to visualization.
Reverse direction is being exploited to integrate NAMD&VMD into RealityGrid framework.
12
Simulation
Steering library data transfer
Steering library http://www.realitygrid.org

Steering
GS bind publish
Client
Steering client find
Registry publish bind
Steering
GS
Simulation data transfer
Steering library
13 http://www.realitygrid.org

ƒ Each application has an associated OGSI-compliant “Steering Grid Service”
(SGS)
ƒ SGS provides public interface to application
– Use standard grid service technology to do steering
– Easy to publish our protocol
– Good for interoperability with other steering clients/portals
– Future-proofed next step to move away from file-based steering or Modular Visualisation
Environments with steering capabilities
ƒ SGSs used to bootstrap direct inter-component connections for large data transfers
ƒ Early working prototype of OGSA Steering Grid Service exists
– Based on light-weight Perl hosting environment OGSI::Lite
– Lets us use OGSI on a GT2 Grid such as UK e-Science Grid today
14 http://www.realitygrid.org

15
ƒ Built using C++ and Qt library – currently have execs. for Linux and IRIX
ƒ Attaches to any steerable RealityGrid application
ƒ Discovers what commands are supported
ƒ Discovers steerable & monitored parameters
ƒ Constructs appropriate widgets on the fly
ƒ Web client (portal) under development http://www.realitygrid.org

Visualization
SGI Onyx
Vtk + VizServer
SGI Op enGL V izSe rver
Simulation
Data
GLOBUS-IO program lbe use lbe_init_module use lbe_steer_module use lbe_invasion_module
Simulation
LB3D with RealityGrid
Steering API sed
Ste
co
File
ba file sys tun
: erin g (X
M
L) mm uni erin cat g G tem
Ste nel led ion
via
s har ed s
ba ck
UI
X o usi ng utp ut i ssh
.
ReG steering GUI
Laptop
Vizserver Client
Steering GUI
GLOBUS used to launch jobs
16 http://www.realitygrid.org

application performance steerer application component performance steerer component
1 component performance steerer component
2 component performance steerer component
3
17 http://www.realitygrid.org

ƒ Computational steering + remote, on-line visualization demand:
– co-allocation of HPC (processors) and visualization (graphics pipes and processors) resources
– at times to suit the humans in the loop
• advanced reservation
ƒ For medium to large datasets, Network QoS is important
– between simulation and visualization,
– visualisation and display
ƒ Integration with Access Grid
– want to book rooms and operators too
ƒ Cannot assume that all resources are owned by same VO
ƒ Want programmable interfaces that we can rely on
– must be ubiquitous, standard, and robust
ƒ Reservations (agreements) should be re-negotiable
ƒ Hard to change attitudes of sysadmins and (some) vendors
18 http://www.realitygrid.org

ƒ Steering adds extra channels of information and control to
Grid services.
ƒ Steering and steered components must be state-aware, underlying mechanisms in OS and lower-level schedulers, monitors, brokers must be continually updated with changing state.
ƒ How do we store and restore the metadata for the state of the parameter space search?
ƒ Human factors are built into our architecture, humans continually interact with orchestrated services. What implications for workflow languages?
19 http://www.realitygrid.org

ƒ Multiple groups exploring multiple regions of parameter space.
ƒ How to record and restore the state of the collaboration?
ƒ How to extend the collaboration over multiple sessions?
ƒ What are the services and abstractions necessary to bootstrap collaborative sessions?
ƒ How do we reliably recreate the resources required by the services, in terms of computation, visualization, instrumentation and networking.
20 http://www.realitygrid.org

Visualization Workflow
Workflows saved from
Previous sessions or
Created in this session
Participants location and access rights
Virtual Venues Server
Multicast addressing
Bridges
Simulation Workflow
Workflows saved from
Previous sessions or
Created in this session
Data Source Workflow
Workflows saved from
Previous sessions or
Created in this session
Service for Bootstrapping session
Contains “just enough”
Information to start other
Services, red arrows indicate bootstrapping
Application data, computation and visualization requirements
Who participates?
Process Repository
Collaborative processes
Captured using ontology
Can be enacted by
Workflow engines
What do they use?
Application Repository
Uses application specific ontology to describe what in silico processes need
To be utilised for the session
21 http://www.realitygrid.org

Linking US Extended Terascale Facilities and UK HPC resources via a Trans-
Atlantic Grid
ƒ We used these combined resources as the basis for an exciting project
– to perform scientific research on a hitherto unprecedented scale
ƒ Computational steering, spawning, migrating of massive simulations for study of defect dynamics in gyroid cubic mesophases
ƒ Visualisation output was streamed to distributed collaborating sites via the Access
Grid
ƒ Workshop presentation with FZ Juelich and HLRS, Stuttgart on the theme of computational steering.
ƒ At Supercomputing , Phoenix, USA, November 2003 TRICEPS entry won “Most
Innovative Data-Intensive Application”
22 http://www.realitygrid.org

ƒ All our workflow concepts are built around the idea of
Steerable Grid Services.
ƒ Resources used by services have complex state, may migrate, may be reshaped.
ƒ Collaborative aspects of “Humans in the loops” are becoming more and more important.
ƒ The problems of allocating and managing the resources necessary for realistic modelling are very hard, they require
(at present) getting below the Grid abstractions.
ƒ Clearly the Grid abstractions are not yet sufficiently comprehensive and in particular lack support for expression of synchronicity.
23 http://www.realitygrid.org

LUSI is located at and developed by Queen Mary College, University of London
Aim: Find ceramics (e.g. rare earth metal oxides) with interesting / valuable properties (e.g. high temperature superconductivity)
Motivation: theory cannot indicate how to construct a compound with a particular property.
Established methodology in pharmaceutical industry uses automated sample generation and testing. Let's apply the same idea in materials science, exploring properties that are difficult to predict: superconductivity, luminescence, dielectric response…
Furnace XY Table Instruments Printer
24 http://www.realitygrid.org

c
Robot
New materials
Neural network
c
Predictions
25
Measured data Database http://www.realitygrid.org