Applied CyberInfrastructure Concepts
ISTA 420/520 Fall 2013
Will Computers Crash Genomics? Science Vol 331 Feb 2011
Nirav Merchant (nirav@email.arizona.edu)
Bio Computing & iPlant Collaborative
Eric Lyons (ericlyons@email.arizona.edu)
Plant Sciences & iPlant Collaborative
University of Arizona
1
http://goo.gl/p4j3m
or https://sites.google.com/site/appliedciconcepts/
1
Topic Coverage
 Some pointers for your midterm
 What are my choices for data transfer methods
 Journey of data and gotcha
 Data transfer != Data Management != Storage
 iRODS hands on
Mid term
 Consider Tyson as your client/customer and you are
trying to win his business (and confidence)
 Make it easy for him to use it, provide the instructions
(user manual) for him to use it.
 Include things like faq, gotcha. Expect that he is a new
user with no idea about futuregrid , UA HPC etc. so be
through and minimize jargon, explain things as step
 The wiki page should be laid out such that he would
want to hire you as his data team !
 Your report should be well laid out and to the point
 To run your job across distributed systems is good and
recommended (use high port numbers 50000 and above
)
How to transfer ..and gotcha
 http://moo.nac.uci.edu/~hjm/HOWTO_move_d
ata.html
 Network, disks, CPU and such
 Some tools for bench marking and estimating
– Measurement lab (NDT)
http://www.measurementlab.net/
– DDCopy http://monalisa.cern.ch/FDT/
 Bits and Bytes (Mb V/s MB)
 http://www.wu.ece.ufl.edu/links/dataRate/Data
MeasurementChart.html
iPlant Data Store
Free Your Data
Now with:
Java
Python
REST API
Different Users,
Different Access Needs:
One Data Store
The journey of data to iDS
and challenges along the way !
Hard Drive
Internet
Network
card
UA/FG
network
Building
network
iDS
Campus
network
Network
card
http://en.wikipedia.org/wiki/List_of_device_bandwidths
Check: USB, HDD, Network capabilities
Internet
Hard
Drive
Basic Exercise
 Lets learn about icommands
 https://pods.iplantcollaborative.org/wiki/display/DS/Usi
ng+iCommands
 Lets connect to UA HPC and use irods (hint module load)
 TRANSFERING LARGE DATA FROM LAPTOP ON
WIRELESS IS A BAD IDEA (steps below are from UA HPC
after you ssh there)
 Create a 100MB file (hint use dd and search for it) on UA
HPC
 Put it into iPlant data store (hint what icommand to use
?)
 Log into http://de.iplantcollaborative.org and browse
you data, share it with someone
 Connect to future grid machine and get the same file
Basic exercise
 Exploring special commands like imeta inc
 (you will be better of getting icommands from
iPlants wiki)