Crystal25
Hunter Valley, Australia, 11 April 2007
JAINIS (JCU and Indiana Instrument Services):
A Grid Enabled Remote Access and Data Management System for X-Ray
Diffraction
I. M. Atkinson1, M. D. Wyatt1, T. King1, N. G. Sim1, P. Turner, R.
Quilici2, D. F. McMullen3
A collaboration between:
James Cook University, the University of Sydney and Indiana
University.
Crystal25
Hunter Valley, Australia, 11 April 2007
Goal: Develop a Grid enabled and Web services based portal system
for collaborative remote access to scientific instruments and their
data.
Why?
Maximise resource utilization and return on investment/s
Outreach, diffusion, training and education
Grid?
Access to compute cycles for processing and modeling and
flexible distributed storage management that may further facilitate
remote access
Multi-Site Collaboration
Goal for production level service: Significant collaboration
between USyd, JCU, IU
(and many others: UAdel, Monash, UQ, DART…)
Deployment and user focus issues (this is hard)
Crystal25
Hunter Valley, Australia, 11 April 2007
Common Instrument Middleware Architecture (CIMA)
•
Integrate instruments and sensors
(e.g. real-time data sources) into a grid
environment with Web Services interfaces
•
Abstract instrument capabilities and functions
to reduce data acquisition and analysis
applications dependence on specialized
knowledge about particular instruments
•
Move production of metadata as close to
instruments as possible and facilitate the
automatic production of metadata
•
Develop a standard, reusable methodology for
grid enabling instruments
QuickTime™ and a
TIFF (LZW) decompressor
are neededQuickTime™
to seeandthis
picture.
a
TIFF (LZW) decompressor
are needed to see this picture.
Rick McMullen,
Knowledge Acquisition and
Projection Laboratory,
Indiana University
Crystal25
Hunter Valley, Australia, 11 April 2007
Crystal25
Hunter Valley, Australia, 11 April 2007
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Crystal25
Hunter Valley, Australia, 11 April 2007
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Crystal25
Hunter Valley, Australia, 11 April 2007
IU CIMA X-Ray system
●
●
●
4 Components
i. Instrument Representative,
ii. My Manager,
iii. Data Manager,
iv. Portal Interface
Instrument Rep,
My Manager, Data Manager Service
implemented in c++, gSoap, Perl
Storage – ext3 file-system, MySQL database
Portal – jrs168 portlets, gridsphere
Issues – My Manager & Portal
implementations domain specific
i.e. Crystallography vs
oceanographic monitoring
Crystal25
Hunter Valley, Australia, 11 April 2007
Data Manager
SOAP
Instruments
Live
data
feeds
Interface
Instrument
Representative
CIMA
SOAP
CIMA
Workflow & comp
pipelines.
Automatic metadata
generation.
Video feeds
GridSphere Portal
PGL: SRB access
& schemas
Data Processing
Applications,
Annontation and
visualisation
CIMA core code from IU
DART/JCU/UQ/Usyd/MU
Implementation
SRB
HSM
Storage
MCAT
SRB data
repository (data
and metadata)
Crystal25
Hunter Valley, Australia, 11 April 2007
Data Management




NFS and MySQL replaced with Storage Resource Broker
(SRB)
Kepler workflow used for MyManager
Ability to customise
data storage via a
workflow system
Highly extensible,
not restricted to SRB
as repository
Crystal25
Hunter Valley, Australia, 11 April 2007
Storage Resource Broker (SRB)
Data Grid middleware
from SDSC (San Diego
Supercomputing Centre)
QuickTime™ and a TIFF (Uncomp resse d) de com press or are nee ded to s ee this picture.
•Uniform interface to heterogeneous resources over
a network.
•Logical View of data
•User defined metadata structures
•Works in conjunction with MCAT.
http://www.sdsc.edu/srb
Crystal25
Hunter Valley, Australia, 11 April 2007
User Facing Portal





Use of Personal Grid Library (PGL) for SRB data
manipulation / metadata display
Metadata schema definitions applied to
experimental data
Stored experimental data is able to be easily
retrieved, secured and annotated
Visual diagnostics for sensors
(e.g. Graphs, Visual Indicators)
Live CCD image analysis (experimental status)
Crystal25
Hunter Valley, Australia, 11 April 2007
Crystal25
Hunter Valley, Australia, 11 April 2007
QuickTime™ and a
TIFF (U ncompressed) decompressor
are needed to see this picture.
Crystal25
Hunter Valley, Australia, 11 April 2007
A. Beitz, et al. Monash
Crystal25
Hunter Valley, Australia, 11 April 2007
J. Hunter, et al. UQ
Crystal25
Hunter Valley, Australia, 11 April 2007
Data Federations

Initial Instrument Representatives
deployed at JCU, USyd, Monash & UQ
Goal

Each site having their
own Data Manager and SRB
storage facility which is
federated across sites

Shibbolith based AAA and
Virtual Organisations

Federate/Replicate data into a
a national SRB store…
Crystal25
Hunter Valley, Australia, 11 April 2007
Future Developments
– Data federations (SRB)
– Security and Rights Management
• Shibboleth
– Analysis tools (Portals/portlets)
– Workflows
• Goal is an end-to-end workbench
• Extensions to larger and smaller instruments
(e.g. sensors)
Crystal25
Hunter Valley, Australia, 11 April 2007
Acknowledgements:
JCU
S. Ness
M. Morgan
F. Eilert
Sydney Uni
D. du Boulay
C. Chee
Indiana Uni
K. Chiu (SUNNY)
Adelaide Uni
A. Wendelborn
P. Coddington
D. Zhang
Department of Education Science and Training: Dataset Acquisition,
Accessibility, and Annotation e-Research Technologies (DART) project
Australian Research Council: e-Research Seed Funding Programme and the
Research Networks Programme
(MMSN: Molecular and Materials Structure Network).
Crystal25
Hunter Valley, Australia, 11 April 2007