Applied CyberInfrastructure Concepts
ISTA 420/520 Fall 2012
Will Computers Crash Genomics? Science Vol 331 Feb 2011
Nirav Merchant (nirav@email.arizona.edu)
Bio Computing & iPlant Collaborative
Eric Lyons (ericlyons@email.arizona.edu)
Plant Sciences & iPlant Collaborative
University of Arizona
http://goo.gl/p4j3m or https://sites.google.com/site/appliedciconcepts/
1
Topic Coverage:
HPC
Racks, Nodes, sockets, chips &
cores !
Accessing UA HPC
Resource and Job Management
systems
2
Docker Vs. Git
 Discussion
3
What is an HPC
 Discussion
4
HPC: racks of servers
5
HPC: miles of cables
6
Look inside…
Quad-core, quad-socket Opteron compute node
Learn more at:
https://computing.llnl.gov/tutorials/linux_clusters/
7
Node, socket, chips/processor, cores
For simple english explanation visit
http://kb.iu.edu/data/avfb.html
8
Etiquettes for working on HPC
 Respect the login node
 Understanding the queue and cores+ram
limits
 Quota
 Scratch and disc space
 Password less entry via ssh and security
ramifications
9
Getting started with UA HPC
Check course wiki under Getting started ..feel free to add new ones !
10
Hands on:
 Login to: login.hpc.arizona.edu
 Big data: http://snap.stanford.edu/data/webMovies.html
 Defensive data downloading
 Find the size of data before downloading
 Decide where you want to keep it
 Plan space for uncompressed version
 Make some accommodations for exploring
data, planning
11
Hints
 Work as a team for this hands on part
 http://www.thegeekstuff.com/2012/04/curl-examples/
 Estimate the size using curl and “content
length”
 Quota check
 Get space using xdisk
 Use alias command to remember (cd to xdisk
location by typing cds)
 Download using curl
 Give me the first 1Mb of data (checksum it for
me and count number of userID’s)
12
Git
 Create a dir called src/gitfun
(hint mkdir –p)
 git init
 git status try with -s
 Create a text file (fun.txt) edit and add text
into it
 git add fun.txt
 git comitt –m “Adding my first file”
 git show HEAD^^:fun.txt
 git diff HEAD^^:fun.txt ./fun.txt
13
General Architecture
CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH
PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)
15
Resource & Job Mgmt System
CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH
PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)
16
Why are you torturing us




This is absolutely boring
I have no use for it
I own my HPC, HTC (and it does what I tell it)
I <3 Amazon and this is never needed on the
cloud
 Etc etc.
17
Resource utilization on HPC systems
per parallel job
CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH
PERFORMANCE COMPUTING - Yiannis Georgiou 2010 (thesis)
18
Principal tasks (sequence dia.)
19
Scheduling Policy
20
Popular Options
(commercial and opensource)
 SLURM
 CONDOR
 TORQUE
 OAR
 SGE
 MAUI
 MOAB
 LSF
 PBSPro
 LoadLeveler
21
Resource and Job Mangement
CONTRIBUTIONS FOR RESOURCE AND JOB MANAGEMENT IN HIGH PERFORMANCE COMPUTING - Yiannis Georgiou
2010 (thesis)
22
Cloud is no different for RJMS
23
Comparison of RJMS
24
Some common concepts
 Login nodes
 Compute nodes
 Storage systems
 Time: CPU, Wall etc
 RJMS systems
 Understanding queues
 Understanding scheduling
 Understanding monitoring
 Clever tricks specific to RJMS !
25
PBSpro
26