National e-Science Core Programme
&
Grid Highlights
BiGUM1 Meeting @ eSI
30th October 2001
Contents
Welcome
NeSC and e-Science Support
Grid Definitions
Grid Examples
Grid Architectures
e-Science Programme
DG Research Councils
E-Science
Steering Committee
Director’s
Awareness and Co-ordination Role
Academic Application Support
Programme
Research Councils (£74m), DTI (£5m)
PPARC (£26m)
BBSRC (£8m)
MRC (£8m)
NERC (£7m)
£80m
ESRC (£3m)
EPSRC (£17m)
CLRC (£5m)
Grid TAG
Director
Director’s
Management Role
Generic Challenges
EPSRC (£15m), DTI (£15m)
Collaborative projects
Industrial Collaboration (£40m)
From Tony Hey 27 July 01
UK Grid Network
Edinburgh
Glasgow
DL
Belfast
Newcastle
Manchester
Oxford
Cardiff
RAL
Cambridge
London
Hinxton
Soton
From Tony Hey 27 July 01
Key Elements of UK Grid
Development Plan
1.
2.
3.
4.
5.
6.
Network of Grid Core Programme eScience Centres
Development of Generic Grid
Middleware
Grid IRC Grand Challenge Project
Support for e-Science Testbeds
International Involvement via GGF
Grid Network Team
From Tony Hey 27 July 01
NeSC’s context
Coordination
e-Science Centres Application Pilots
IRCs …
e-Scientists, Grid users, Grid services & Grid Developers
TAG
GNT
DBTF
ATF
NeSC
GSC
UK Core Programme Team
eSI
CS Research
Global Grid Forum …
NeSC — The Team
Director
Malcolm Atkinson (Universities of Glasgow & Edinburgh)
Deputy Director
Arthur Trew (Director EPCC)
Commercial Director
Mark Parsons (EPCC)
Regional Director
Stuart Anderson (Edinburgh Informatics)
Chairman
Richard Kenway (Edinburgh Physics & Astronomy)
Initial Board Members
Muffy Calder (Glasgow Computing Science)
Tony Doyle (Glasgow Physics & Astronomy)
Centre Manager
Anna Kenway
NeSC’s Roles
Stimulation of Grid & e-Science Activity
Users, developers, researchers
Education, Training, Support
Think Tank & Research
Coordination of Grid & e-Science Activity
Regional Centres, Task Forces, Pilots & IRCs
Technical and Managerial Fora
Support for training, travel, participation
Developing a High-Profile Institute
Meetings
Visiting Researchers
Regional Support
Portfolio of Industrial Research Projects
eSI Highlights
Report given yesterday
History
X
X
X
3 workshops week 1: DF1, GUM1 & DBAG1
HEC
preGGF3 & DF2
October
X
X
X
Steve Tuecke Globus tutorial (oversubscribed)
4-day workshop Getting Going with Globus (G3)
– Reports on DataGrid & GridPP experience
Biologist Grid Users’ Meeting 1 (BiGUM1)
November
X
X
GridPP
Configuration management
December
X
AstroGrid
eSI Highlights cont.
2002 & 2003
January
X
Steve Tuecke 4 day Globus Developers’ Workshop
February
X
UKOLN
March
X
Protein folding Workshop 14th to 17th IBM sponsor
May
X
Mind and Brain Workshop
22nd to 26th July GGF5 & HPDC 11 EICC
August Research Festival
4 juxtaposed 1-week in-depth workshops
Topics under consideration
X
X
X
X
X
Dependability and Security for the Grid
Metadata and the Grid
Provenance, Annotation and Archiving
The Knowledge Grid
Programming Models for the Grid
14th to 16th April 2003 Dependability
¾Motivation for IPG
Large-scale science and engineering are
done through the interaction of people,
heterogeneous computing resources,
information systems, and instruments, all of
which are geographically and
organizationally dispersed.
The overall motivation for “Grids” is to
facilitate the routine interactions of these
resources in order to support large-scale
science and engineering.
From Bill Johnston 27 July 01
Why Grids?
A biochemist exploits 10,000 computers to screen
100,000 compounds in an hour
1,000 physicists worldwide pool resources for
petaop analyses of petabytes of data
Civil engineers collaborate to design, execute, &
analyze shake table experiments
Climate scientists visualize, annotate, & analyze
terabyte simulation datasets
An emergency response team couples real time
data, weather model, population data
From Steve Tuecke 12 Oct. 01
Why Grids? (contd.)
A multidisciplinary analysis in aerospace couples
code and data in four companies
A home user invokes architectural design
functions at an application service provider
An application service provider purchases cycles
from compute cycle providers
Scientists working for a multinational soap
company design a new product
A community group pools members’ PCs to
analyze alternative designs for a local road
From Steve Tuecke 12 Oct. 01
The Grid Problem
Flexible, secure, coordinated resource sharing
among dynamic collections of individuals,
institutions, and resource
From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”
Enable communities (“virtual organizations”) to
share geographically distributed resources as they
pursue common goals -- assuming the absence
of…
central location,
central control,
omniscience,
existing trust relationships.
From Steve Tuecke 12 Oct. 01
Elements of the Problem
Resource sharing
Computers, storage, sensors, networks, …
Sharing always conditional: issues of trust,
policy, negotiation, payment, …
Coordinated problem solving
Beyond client-server: distributed data analysis,
computation, collaboration, …
Dynamic, multi-institutional virtual
organisations
Community overlays on classic org structures
Large or small, static or dynamic
From Steve Tuecke 12 Oct. 01
Why Now?
Moore’s law improvements in computing
produce highly functional endsystems
The Internet and burgeoning wired and
wireless provide universal connectivity
Changing modes of working and problem
solving emphasize teamwork,
computation
Network exponentials produce dramatic
changes in geometry and geography
From Steve Tuecke 12 Oct. 01
Network Exponentials
Network vs. computer performance
Computer speed doubles every 18 months
Network speed doubles every 9 months
Difference = order of magnitude per 5 years
1986 to 2000
Computers: x 500
Networks: x 340,000
2001 to 2010
Computers: x 60
Networks: x 4000
Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.
From Steve Tuecke 12 Oct. 01
Broader Context
“Grid Computing” has much in common
with major industrial thrusts
Business-to-business, Peer-to-peer, Application
Service Providers, Storage Service Providers,
Distributed Computing, Internet Computing…
Sharing issues not adequately addressed by
existing technologies
Complicated requirements: “run program X at
site Y subject to community policy P, providing
access to data at Z according to policy Q”
High performance: unique demands of
advanced & high-performance systems
From Steve Tuecke 12 Oct. 01
The Globus Project™
Making Grid computing a reality
Close collaboration with real Grid projects in science and
industry
Development and promotion of standard Grid protocols to
enable interoperability and shared infrastructure
Development and promotion of standard Grid software
APIs and SDKs to enable portability and code sharing
The Globus Toolkit™: Open source, reference software
base for building grid infrastructure and applications
Global Grid Forum: Development of standard protocols
and APIs for Grid computing
From Steve Tuecke 12 Oct. 01
Online Access to
Scientific Instruments
Advanced Photon Source
wide-area
dissemination
real-time
collection
archival
storage
desktop & VR clients
with shared controls
tomographic reconstruction
DOE X-ray grand challenge: ANL, USC/ISI, NIST, U.Chicago
From Steve Tuecke 12 Oct. 01
Complex,
Widely Distributed Workflow
Management
Mathematicians Solve NUG30
Looking for the solution to the
NUG30 quadratic assignment
problem
An informal collaboration of
mathematicians and
computer scientists
Condor-G delivered 3.46E8
CPU seconds in 7 days (peak
1009 processors) in U.S. and
Italy (8 sites)
14,5,28,24,1,3,16,15,
10,9,21,2,4,29,25,22,
13,26,17,30,6,20,19,
8,18,7,27,12,11,23
MetaNEOS: Argonne, Iowa, Northwestern, Wisconsin
From Miron Livny 7 Aug. 01
Network for Earthquake
Engineering Simulation
NEESgrid: national
infrastructure to couple
earthquake engineers with
experimental facilities,
databases, computers, &
each other
On-demand access to
experiments, data streams,
computing, archives,
collaboration
NEESgrid: Argonne, Michigan, NCSA, UIUC, USC
From Steve Tuecke 12 Oct. 01
Home Computers
Evaluate AIDS Drugs
Community =
1000s of home
computer users
Philanthropic
computing vendor
(Entropia)
Research group
(Scripps)
Common goal=
advance AIDS research
From Steve Tuecke 12 Oct. 01
Layered Grid Architecture
(By Analogy to Internet Architecture)
“Coordinating multiple resources”:
ubiquitous infrastructure services,
app-specific distributed services
“Sharing single resources”:
negotiating access, controlling use
Collective
Application
Resource
“Talking to things”: communication
(Internet protocols) & security
Connectivity
Transport
Internet
“Controlling things locally”: Access
to, & control of, resources
Fabric
Link
Internet Protocol Architecture
Application
From Steve Tuecke 12 Oct. 01
Architecture of a Grid
Discipline Specific Portals and
Scientific Workflow Management Systems
clusters
Distributed
national supercomputer
facilities
Condor pools
Fault
Management
Monitoring
= Globus services
Resources
tertiary storage
Auditing
Security
Services
Authentication
Authorization
Communication
Services
Network
Cache
Collaboration
and Remote
Instrument
Services
Uniform Data
Access
Data
Cataloguing
Global Event
Services
CoScheduling
Global
Queuing
Brokering
Uniform
Resource
Access
Grid
Information
Service
Applications: Simulations, Data Analysis, etc.
Toolkits: Visualization, Data Publication/Subscription, etc.
Grid Common Services: Standardized Services and Resources Interfaces
national user facilities
network
caches
High-speed Networks and Communications Services
Architecture of a Grid – upper layers
•Knowledge based query
Problem
Solving
Environments
•Tools to implement the human interfaces, e.g. SciRun, ECCE, WebFlow, .....
•Mechanisms to express, organize, and manage the workflow of problem
solutions (“frameworks”)
data publish
and
subscribe
toolkits
instrument
management
toolkits
collaboration
toolkits
visualization
toolkits
Applications
and
Supporting
Tools
application
codes
•Access control
Grid Common Services
Distributed Resources
DCOM
Java/
Jini
CondorG
CORBA
Application
Development and
Execution Support
Globus
MPI
Grid enabled libraries (security, communication services, data
access, global event management, etc.)
From Steve Tuecke 12 Oct. 01
Three Layer GRID Abstraction
Knowledge Grid
Data
Data to
to
Knowledge
Knowledge
Control
Control
Information Grid
Computation/
Data Grid
From Tony Hey 12 Sep. 01
Data, Information and Knowledge
Data
Uninterpreted bits and bytes
Information
Data equipped with meaning
Knowledge
Information applied to achieve a
goal, solve a problem or enact a
decision
From Tony Hey 12 Sep. 01
Biological Grid Users
Are they different?
Do they have different collaborations?
Do they have different data?
Do they have different computations?
Do they have the same shared
“instruments”?
Can they be supported using the same
Infrastructure
Architecture
Policies?