A WORLDWIDE E-INFRASTRUCTURE FOR COMPUTATIONAL NEUROSCIENTISTS
Innovative ICT Solutions for Multi-centric
European Research Networks
2011-10-27 ITU Telecom World – WHO eHealth Pavilion
Palexpo – Geneva, Switzerland
David MANSET
outGRID Technical Coordinator
From the Internet to the Grid
Internet
Grid
The worldwide network
of networks...
The environment for sharing
computing resources...
Web
The protocol for sharing
Information over the
Internet...
*Courtesy of EGI Project
neuGRID e-Infrastructure
3 participating clinical centers: FBF
(Brescia, Italy), VUmc (Amsterdam, The
Netherlands), KI (Stockholm, Sweden),
600 CPU cores (+ EGI elastic resources),
Unique bridge to transatlantic initiatives
CBRAIN (Canada) and LONI (USA),
Expands to support clinical decision
making in neuro-degenerative diseases
High-capacity Network
Sim-e-Child e-Infrastructure
Leverages on FP6-funded Health-e-Child
project and excellence network,
5 participating clinical centers:
OPBG(Rome, Italy), IGG (Genova, Italy),
NECKER (Paris, France), GOSH (London,
UK), JHU (Baltimore, USA)
300 CPU cores (+ EGI elastic resources),
Unique bridge to transatlantic initiatives
GenTAC and COAST clinical trials and
databases (USA),
High-capacity Network
Disease Markers Development,
Test and Validation
neuGRID
A Grid-brained e-Infrastructure for
Neurosciences
Imaging Markers for Alzheimer’s
Gray Matter Loss
Isolated
Memory
Problems
Early
Disability
Consolidated
Disability
Recipe to Develop Markers for Alzheimer’s:
1. Large Databases (‘0,000s)
…
T0
US
Australian
Japanese
European
T6
T12
ADNI
ADNI (AIBL)
ADNI
ADNI (IMI Pharma-cog)
T18
≈10,000 images
≈ 4,000 images
(≈ 3,000 images)
(≈ 1,500 images)
…
Recipe to Develop Markers for Alzheimer’s:
2. Sophisticated Algorithms for Image Analysis
Cortical thickness
at month 0
1.50 mm
Cortical thickness
at month 30
4.50 mm
Difference analysis
+.50
-.50
Recipe to Develop Markers for Alzheimer’s:
3.Sophisticated Statistical Models
Gray matter change in
Healthy elders
Very early
Alzheimer’s
-3 | -2
| -1 | 0 | 1 |
Years from dementia
2
|
3
TOMORROW
neuGRID
neuGRID Infrastructure
Scalable Robust Distributed
Grid
SOA
Workflow
Provenance
Pipeline
neuGRID DACS are connected to GEANT2 Network
neuGRID Infrastructure integrates EGI Grid resources
LEVEL 0
Thousands of CPUs
Petabytes
of storage
Data
LORIS
Grid Coordination
Center
20 Mb/s
Coordination
Strong liaison with BiomedVO/LSVRC
Center
LEVEL 1
Slave
LORIS
DACS1
100 Mb/s
USERS
Slave
LORIS
DACS2
100 Mb/s
Slave
LORIS
DACS3
1 Gb/s
neuGRID Platform
CoreLab
•Data Acq. (Imaging +
Clinical Variables)
•Data Quality Control
•Data Anonymization
Pipelining
•Pipeline Authoring
•Pipeline Optimization
•Pipeline Enactment
•Multiple Toolkits
(CIVET, BrainVISA,
FreeSurfer, ITK/VTK,
R etc)
Collaboration
•Document Sharing
•Pipeline Sharing
•Provenance Sharing
Pipeline Output Visualization
Augmented Reality Service
LEVEL 0
PHP
JSON
MyProxy
CAS
Grid
Coordination
Service
Service
GW
Center
Auth
Service
GSI
LORIS
Data
Coordination
Center
LEVEL 1
Augmented Reality Spot
AR Service
Slave
LORIS
DesktopFusion
DACS2
Service
Slave
LORIS
DACS3
Pipeline authoring
and debugging
AC/DC2 “Highway to Hell” - Data Challenge
LEVEL 0
Scalable Robust Distributed
Grid
SOA
Workflow
Provenance
Pipeline
Fast transfers through GEANT2
US-ADNI
Database
GCC
US-ADNI Dataset
• 6’235 MRData
Scans LORIS
• 70
GB
Coordination
Center
LEVEL 1
Slave
LORIS
DACS1
CE
WN1
WN2
WN3
WNn
Slave
LORIS
Slave
LORIS
DACS3
DACS2
SE
CIVET
Pipelines
Execution
AC/DC2 “Highway to Hell” - Data Challenge
EGEE’09 Best Live
Demonstration Award!
Expected Results
Experiment duration on the Grid
< 2 Weeks
Experiment duration on single computer
Analyzed data
Patients
MR
Scans
Images
Voxels
Total mining operations
Max # of processing cores in parallel
Number of countries involved
Volume of output data produced
> 5 Years
715
6’235
~1’300’000
~9’352’500’000
286’810
184
4
1 TB
AC/DC3 “Thunderstruck” - Data Challenge
Alzheimer’s Disease Neuroimaging Initiative
- The ADNI is a multisite, multiyear program which began in October 2004,
- More than 800 subjects recruited, more than 7’500 MR scans
- CIVET, FreeSurfer and BrainVISA pipelines executed over the 7’500
MR scans,
- Exercised 3 DACS sites and the EGEE Seed Resources
Expected Results
AC/DC3 “Thunderstruck” - Data Challenge (3/3)
Experiment duration on the Grid
< 3 Months
Experiment duration on single computer
> 100 Years
Analyzed data
Patients
MR Scans
Images
Voxels
Total mining operations
Max # of processing cores in parallel
Number of countries involved
Volume of output data produced
800
7’500
~1’563’750
~10’791’500’000
~700’000
1’300
6
2.2 TB
Research Infrastructures
Interoperability
In A Nutshell
To promote interoperability among three e-infrastructures for computational
neuroscience to converge into one unique worldwide facility
Home to US-ADNI/IDA
LONI Pipeline
Cluster Facility
CIVET Pipeline
HPC Facilities
NIH-funded
http://www.loni.ucla.edu/
outGRID Facts
- FP7-funded
- Started 11/09
- 24 Months
http://cbrain.mcgill.ca/
Access to large image datasets
(future home to EU-ADNI)
Image Analysis Pipelines
Grid Facilities
http://www.neuGRID.eu/
Developing Interoperability
Current HPC Integration
Syntactic Interoperability
Syntactic Interoperability
Semantic Interoperability
(~13,500 cores 1,000,000+ core hours)
JUROPA – Julich
(26304 cores)
CLUMEQ – U. Laval
(7600 cores)
Westgrid – UBC
(3090 cores)
RQCHP - U.Sherbrooke
(2464 cores)
SHARCNET - UWO
(3082 cores)
•
•
Dedicated HPC cluster f acility at UCLA/LONI,
306-node, dual-processor SUN Microsy stems V20z cluster.
Each V20z node has dual 64-bit 2.4 GHz AMD Opteron 250
CPUs with 8 GB of memory
64-node Dell development cluster , w ith each node using
dual 64-bit 3.6 G Hz Intel EM 64T pr o cesso rs and 4 G B o f
m em ory
64-pr ocessor SGI Or igin 3800 SMP super co mputer w ith 32
G B of m em ory
•
Dedicated Grid nodes combined with
public computing resources from EGI
•
EGI LSGC VRC
•
10’000 CPU Cores
LSGC
•
•
Semantic Interoperability
**SciNET - UoT
(30240 cores)
CLUMEQ - McGill
(350 - 20000? cores)
Sunday, March 14, 2010
•
Shared HPC resources from public
facilities
•
7 HPC Centers sharing computing
resources, 6 across Canada and 1
in Europe
•
13’500 CPU Cores
Interoperability Thinking Framework
• Focus on interoperability
across DCIs and user
communities
• Formulation of general
interoperability guidelines
•
Syntactic
•
•
•
•
Communication
protocols/standards
Security
Networks
Semantic
•
•
Workflow
standards/technologies
Provenance, Data and
Meta-data
standards/technologies
Interoperability Impact
Preliminary Impact:
unified space for developing new biomarkers
Current HPC Integration
Algorithms/Pipelines
(~13,500 cores 1,000,000+ core hours)
JUROPA – Julich
(26304 cores)
CLUMEQ – U. Laval
(7600 cores)
Westgrid – UBC
(3090 cores)
Data
RQCHP - U.Sherbrooke
(2464 cores)
SHARCNET - UWO
(3082 cores)
**SciNET - UoT
(30240 cores)
CLUMEQ - McGill
(350 - 20000? cores)
Sunday, March 14, 2010
•
•
Dedicated HPC cluster f acility at UCLA/LONI,
306-node, dual-processor SUN Microsy stems V20z cluster.
Each V20z node has dual 64-bit 2.4 GHz AMD Opteron 250
CPUs with 8 GB of memory
64-node Dell development cluster , w ith each node using
dual 64-bit 3.6 G Hz Intel EM 64T pr o cesso rs and 4 G B o f
m em ory
64-pr ocessor SGI Or igin 3800 SMP super co mputer w ith 32
G B of m em ory
•
•
Dedicated Grid nodes combined with
public computing resources from EGI
•
EGI LSGC VRC
•
Shared HPC resources from public
facilities
•
7 HPC Centers sharing computing
Frisoni, Manset,
et 6al.,
resources,
across Canada and 1
in Europe
10’000 CPU Cores
Nature Reviews •Neurology
2011
13’500 CPU Cores
LSGC
•
•
in press
Super Infrastructure
Continental Access for Communities
CBRAIN
oGEP
LONI
oGEP
McGILL
neuGRID
oGEP
UCLA
FBF
outGRID
LInked Neuroscientific Grand chAllenge (LINGA)
Scientific Objective
DATA: ≅ 2’000 subjects acquired in different time points at different ages
neuGRID
AD – MCI - CTR
ICBM
CTR
LONI
ANM
CBRAIN
T12
T24
T36
T48
missing
Longitudinal
CIVET/CLASP CORTICAL THICKNESS
ADNI
T6
AD – MCI - CTR
Baseline
LINGA – LInked Neuroscientific Grand chAllenge
Technical Objective
International Web Portal
Shared Workflow Authoring Space
- Objective: Validating interoperability
demonstrator and architecture,
- “1 variable” approach, i.e. “same
application with different data”,
- Developing a “super workflow”
involving all 3 infrastructures,
- CIVET Pipeline to be run in all 3
infrastructures,
- Different datasets to be processed in
each infrastructure
- CBRAIN:
ICBM
- LONI:
US-ADNI
- neuGRID:
EU-ADNI
AddNeuroMed
Grand challenge in numbers…
CIVET @ CBRAIN
CIVET @ LONI
CIVET @ neuGRID
 11’000 MR Scans
 100’000
CPU Hours Post-processing
Post-processing
 1.5 TB of Scientific Data
 2 weeks runtime
Quality Control &
Statistical Analysis
neuGRID
LONI
CBRAIN
Interoperability Demonstrator
In Collaboration with
LINGA – LInked Neuroscientific Grand chAllenge
Political Objective
International Web Portal
Shared Workflow Authoring Space
- Objective: Interconnecting major
international/national initiatives and having
them collaborating towards one common goal
- Neuroscience Infrastructures
- CBRAIN
CA
- LONI
US
- neuGRID
EU
- NeuroLOG
FR
- DCI Infrastructures
- HPC
- EGI
- SHIWA
CA
EU
EU
CIVET @ CBRAIN
CIVET @ LONI
Post-processing
CIVET @ neuGRID
Post-processing
- Research Infrastructures
- GéANT
EU
- CAnet
CA
- Internet2.0
US
Quality Control &
Statistical Analysis
neuGRID
LONI
CBRAIN
Interoperability Demonstrator
LINGA Challenge
EGI’11 Best Live
Demonstration Award!
MetaWorkflow
Management
Sub-Workflows
17/09/11
SSC & Virtual Laboratory
Users
Pipelines &
Algorithms
Analyses
New
Datasets
Markers
FBF
VUmc
N4U Tier 1
GRID
KI
LSVRC
N4U - IS
N4U Science Gateway
NSPIN
UNIG
E
GéANT
N4U Tier 2
Grid
Coordination
Center
HPC
GBRAIN
LONI
CAnet
Data
Coordination
Center
INTERNET2
N4U Tier 3
CLOUD
From Markers to
Diagnostic Tools
Production quality e-Infrastructure
to service Diagnostic Tools
•
•
•
•
•
•
•
•
•
•
•
•
Interoperability
•
LifeRay-based Web portal
•
•
Diagnostic services as Portlets
User-friendly access
•
•
•
Simple login + password
• Single Sign On + Shibboleth
Multiple IdP security architecture
Robot-certificate mapping
DIAGNOSIS OF ALZHEIMER’S AT THE MCI STAGE
Structure:
Gray matter
volumetry
Metabolism:
FDG PET
Alzheimer
CSF total tau
Amyloid deposits:
PET
Biochemistry:
CSF
tau/Abeta
Normal
CSF Abeta42
Neural activity.
EEG
From Research Infrastructures
to Translational Medicine
Sim-e-Child
Grid-Enabled Platform for Simulations in Paediatric Cardiology
Toward the Personalized Virtual Child Heart
– ITU Telecom World 2011 –
36
October 2011
??
Sim-e-Child
Sim-e-Child
First Trans-Atlantic platform towards personalized & predictive
modeling of congenital heart disease
•
•
•
37
FP7 STREP
– Virtual Physiological Human (VPH) Objective
– Building upon FP6 IP Health-e-Child
– January 2010 to June 2012
Technical focus
– Computational modeling & simulation of cardiovascular anatomy, function and hemodynamics
– Grid-enabled infrastructure for data management
& distributed high-performance computation
Clinical focus
– Model validation
– Multi-scale quantification of disease extend
– Planning, simulation & assessment of efficacy
and safety of therapies
October 2011
??
Simulation Platform
Sim-e-Child
Web-Portal
Grid-Enabled, Distributed Data
Management & High-Performance
Computing
Local Simulation Application
in Hospital
http://sec-portal.maatg.fr
38
October 2011
??
Sim-e-Child Network
Sim-e-Child
Health-e-Child
Sim-e-Child
COAST and
genTAC
(>1000 patients)
HeC and OPBG
(>1200 patients)
Siemens
SCR
TUM
JHU
MAAT
ACC
OPBG
Lynkeus
US Grid
Resource
Point
39
EU Grid
Resource
Point
October 2011
??
Simulation Platform: Web-Portal
Sim-e-Child
Information System
PHP
1. Create and export
proxy to MyProxy
(done only once)
JSON
MyProxy
CAS
Grid Coordination
Service Center
Service
GW
Auth
Service
2. Standard CAS
Authentication
through Web
portal
40
GSI
3. Access all facilities
connected to the SSO
October 2011
??
General Modelling Approach
Sim-e-Child
Image Data (CT, MRI, US)
Anatomical Model Fitting and
Clinical Parameter Estimation
Haemodynamics
Simulation
*Courtesy of SIEMENS
41
October 2011
??
 High-level seminar at the International Telecommunication
Union (ITU) next Feb 2012
• To start forming the global structure
• To discuss coordinated calls between funding agencies
• To define seed initiatives in low-income countries
 GLOBIOS project preparation to structure the global
initiative
Conclusion