The Campus Cluster
What is the Campus Cluster?
•
•
•
•
Batch job system
High throughput
High latency
Available resources:
–
–
–
–
–
~450 nodes
12 Cores/node
24-96 GB memory
Shared high performance filesystem
High speed multinode message passing
What isn’t the Campus Cluster?
• Not: Instantly available computation resource
– Can wait up to 4 hours for a node
• Not: High I/O Friendly
– Network disk access can hurt performance
• Not: ….
Getting Set Up
Getting started
• Request an account:
https://campuscluster.illinois.edu/invest/user_form.html
• Connecting:
ssh to taub.campuscluster.illinois.edu
Use netid and AD password
Where to put data
• Home Directory ~/
– Backed up, currently no quota (in future 10’s of GB)
• Use /scratch for temporary data - ~10TB
– Scratch data is currently deleted after ~3 months
– Available on all nodes
– No backup
• /scratch.local - ~100GB
– Local to each node, not shared across network
– Beware that other users may fill disk
• /projects/VisionLanguage/ - ~15TB
– Keep things tidy by creating a directory for your netid
– Backed up
• Current Filesystem best practices (Should improve for Cluster v. 2):
– Try to do batch writes to one large file
– Avoid many little writes to many little files
Backup = Snapshots
(Just learned this yesterday)
• Snapshots taken daily
• Not intended for disaster recovery
– Stored on same disk as data
• Intended for accidental deletes/overwrites, etc.
– Backed up data can be accessed at:
/gpfs/ddn_snapshot/.snapshots/<date>/<path>
e.g. recover accidentally deleted file in home directory:
/gpfs/ddn_snapshot/.snapshots/2012-1224/home/iendres2/christmas_list
Moving data to/from cluster
• Only option right now is sftp/scp
• SSHFS lets you mount a directory from remote
machines
– Haven’t tried this, but might be useful
Modules
[iendres2 ~]$ modules load <modulename>
Manages environment, typically used to add
software to path:
– To get the latest version of matlab:
[iendres2 ~]$ modules load matlab/7.14
– To find modules such as vim, svn:
[iendres2 ~]$ modules avail
Useful Startup Options
Appended to the end of my bashrc:
– Make default permissions the same for user and
group, useful when working on a joint project
• umask u=rwx,g=rwx
– Safer alternative – don’t allow writing
• umask u=rwx,g=rx
– Load common modules
• module load vim
• module load svn
• module load matlab
Submitting Jobs
Queues
– Primary (VisionLanguage)
• Nodes we own (Currently 8)
• Jobs can last 72 hours
• We have priority access
– Secondary (secondary)
• Anyone else’s idle nodes (~500)
• Jobs can only last 4 hours, automatically killed
• Not unusual to wait 12 hours for job to begin runing
Scheduler
• Typically behaves as first come first serve
• Claims of priority scheduling, we don’t know
how it works…
Types of job
– Batch job
• No graphics, runs and completes without user
interaction
– Interactive Jobs
• Brings remote shell to your terminal
• X-forwarding available for graphics
• Both wait in queue the same way
Scheduling jobs
– Batch job
• [iendres2 ~]$ qsub <job_script>
• job_script defines parameters of job and the actual
command to run
• Details on job scripts to follow
– Interactive Jobs
• [iendres2 ~]$ qsub -q <queuename> -I -l
walltime=00:30:00,nodes=1:ppn=12
• Include –X for X-forwarding
• Details on –l parameters to follow
Configuring Jobs
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
…
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
Queue to use:
VisionLanguage or
secondary
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
…
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
• Number of nodes – 1, unless using MPI
or other distributed programming
• Processors per node – Always 12,
smallest computation unit is a physical
node, which has 12 cores (with current
hardware)*
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
…
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
*Some queues are configured to allow
multiple concurrent jobs per node, but
this is uncommon
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
#PBS -q VisionLanguage
• Maximum time job will run for – it is
#PBS -l nodes=1:ppn=12
killed if it exceeds this
#PBS -l walltime=04:00:00
• 72:00:00 hours for primary queue
…
• 04:00:00 hours for secondary queue
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
…
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
Bash comands are allowed anywhere in
the script and will be executed on the
scheduled worker node after all PBS
commands are handled
Basics
• Parameters of jobs are defined by a bash
script which contains “PBS commands”
followed by script to execute
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
…
cd ~/workdir/
echo “This is job number ${PBS_JOBID}”
There are some reserved variables that the scheduler
will fill in once the job is scheduled (see `man qsub` for
more variables)
Basics
Scheduler variables (From manpage)
PBS_O_HOST
the name of the host upon which the qsub command is running.
PBS_SERVER
the hostname of the pbs_server which qsub submits the job to.
PBS_O_QUEUE
the name of the original queue to which the job was submitted.
PBS_O_WORKDIR
the absolute path of the current working directory of the qsub command.
PBS_ARRAYID
each member of a job array is assigned a unique identifier (see -t)
PBS_ENVIRONMENT
set to PBS_BATCH to indicate the job is a batch job, or to PBS_INTERACTIVE to indicate the job is a PBS interactive job, see -I option.
PBS_JOBID
the job identifier assigned to the job by the batch system.
PBS_JOBNAME
the job name supplied by the user.
PBS_NODEFILE
the name of the file contain the list of nodes assigned to the job (for parallel and cluster systems).
PBS_QUEUE
the name of the queue from which the job is executed.
There are some reserved variables that the scheduler
will fill in once the job is scheduled (see `man qsub` for
more variables)
Monitoring Jobs
grep is your friend for finding specific jobs
[iendres2 ~]$ qstat
Sample output:
JOBID
333885[].taubm1
333899.taubm1
333900.taubm1
333901.taubm1
333902.taubm1
333903.taubm1
333904.taubm1
333905.taubm1
333906.taubm1
333907.taubm1
333908.taubm1
333914.taubm1
333915.taubm1
333916.taubm1
(e.g. qstat –u iendres2 | grep “ R ” gives all of
my running jobs)
JOBNAME
r-afm-average
test6
cgfb-a
cgfb-b
cgfb-c
cgfb-d
cgfb-e
cgfb-f
cgfb-g
cgfb-h
...conp5_38.namd
ktao3.kpt.12
ktao3.kpt.14
joblammps
USER
hzheng8
lee263
dcyang2
dcyang2
dcyang2
dcyang2
dcyang2
dcyang2
dcyang2
dcyang2
harpole2
chandini
chandini
daoud2
WALLTIME
0
03:33:33
09:22:44
09:31:14
09:28:28
09:12:44
09:27:45
09:30:55
09:06:51
09:01:07
0
03:05:36
03:32:26
03:57:06
STATE
QUEUE
Q secondary
R secondary
R secondary
R secondary
R secondary
R secondary
R secondary
R secondary
R secondary
R secondary
H cse
C secondary
R secondary
R cse
States:
Q – Queued, waiting to run
R – Running
H – Held, by user or admin, won’t run until released (see qhold, qrls)
C – Closed – finished running
E – Error – this usually doesn’t happen, indicates a problem with the cluster
Managing Jobs
qalter, qdel, qhold, qmove, qmsg,
qrerun, qrls, qselect, qsig, qstat
Each takes a jobid + some arguments
Problem: I want to run the same job
with multiple parameters
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
cd ~/workdir/
./script <param1> <param2>
Solution: Create wrapper script to iterate over params
Problem 2: I can’t pass parameters into
my job script
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
cd ~/workdir/
./script <param1> <param2>
Solution 2: Hack it!
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
Problem 2: I can’t pass parameters into
my job script
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
# Pass parameters via jobname:
export IFS="-"
i=1
for word in ${PBS_JOBNAME}; do
echo $word
arr[i]=$word
((i++))
done
# Stuff to execute
echo Jobname: ${arr[1]}
cd ~/workdir/
echo ${arr[2]} ${arr[3]}
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
We can pass parameters via the
jobname, and delimit them using
the ‘-’ character (or whatever
you want)
Problem 2: I can’t pass parameters into
my job script
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
# Pass parameters via jobname:
export IFS="-"
i=1
for word in ${PBS_JOBNAME}; do
echo $word
arr[i]=$word
((i++))
done
# Stuff to execute
echo Jobname: ${arr[1]}
cd ~/workdir/
echo ${arr[2]} ${arr[3]}
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
qsub –N job-param1-param2 job_script
qsub’s -N parameter sets the job
name
Problem 2: I can’t pass parameters into
my job script
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
# Pass parameters via jobname:
export IFS="-"
i=1
for word in ${PBS_JOBNAME}; do
echo $word
arr[i]=$word
((i++))
done
# Stuff to execute
echo Jobname: ${arr[1]}
cd ~/workdir/
echo ${arr[2]} ${arr[3]}
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
qsub –N job-param1-param2 job_script
Output would be:
Jobname: job
param1 param2
Problem: I want to run the same job
with multiple parameters
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
# Pass parameters via jobname:
export IFS="-"
i=1
for word in ${PBS_JOBNAME}; do
echo $word
arr[i]=$word
((i++))
done
# Stuff to execute
echo Jobname: ${arr[1]}
cd ~/workdir/
echo ${arr[2]} ${arr[3]}
Where:
param1 = {a, b, c}
param2 = {1, 2, 3}
#!/bin/bash
param1=({a,b,c})
param2=({1,2,3}) # or {1..3}
for p1 in ${param1[@]}; do
for p2 in ${param2[@]}; do
qsub –N job-${p1}-${p2} job_script
done
done
Now Loop!
Problem 3: My job isn’t multithreaded,
but needs to run many times
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
cd ~/workdir/
./script ${idx}
Solution: Run 12 independent
processes on the same node so 11
CPU’s don’t sit idle
Problem 3: My job isn’t multithreaded,
but needs to run many times
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
Solution: Run 12 independent
processes on the same node so 11
CPU’s don’t sit idle
cd ~/workdir/
# Run 12 jobs in the background
for idx in {1..12}; do
./script ${idx} & # Your job goes here (keep the ampersand)
pid[idx]=$!
# Record the PID
done
# Wait for all the processes to finish
for idx in {1..12}; do
echo waiting on ${pid[idx]}
wait ${pid[idx]}
done
Matlab and The Cluster
Simple Matlab Sample
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
cd ~/workdir/
matlab -nodisplay -r “matlab_func(); exit;”
Matlab Sample: Passing Parameters
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
cd ~/workdir/
param = 1
param2 = \’string\’ # Escape string parameters
matlab -nodisplay -r “matlab_func(${param}); exit;”
Simple Matlab Sample
#PBS -q VisionLanguage
#PBS -l nodes=1:ppn=12
#PBS -l walltime=04:00:00
You may use too many licenses especially Distributed Computing
Toolbox (e.g. parfor)
cd ~/workdir/
matlab -nodisplay -r “matlab_func(); exit;”
Running more than a few matlab jobs
(thinking about using the secondary
queue) ?
Compiling Matlab Code
Doesn’t use any matlab licenses once compiled
Compiles matlab code into a standalone executable
Constraints:
– Code can’t call addpath
– Functions called by eval, str2func, or other implicit methods must be
explicitly identified
•
e.g. for eval(‘do_this’) to work, must also include %#function do_this
To compile (within matlab):
>> addpath(‘everything that should be included’)
>> mcc –m function_to_compile.m
is useful for modifying behavior for compiled applications
(returns true if code is running the compiled version)
isdeployed()
Running Compiled Matlab Code
• Requires Matlab compiler runtime
>> mcrinstaller
% This will point you to the installer and help install it
% make note of the installed path MCRPATH (e.g. …/mcr/v716/)
• Compiled code generates two files:
– function_to_compile and run_function_to_compile.sh
• To run:
– [iendres2 ~]$ ./run_function_to_compile.sh MCRPATH param1 param2 … paramk
– Params will be passed into matlab function as usual, except they will always be strings
– Useful trick:
function function_to_compile(param1, param2, …, paramk)
if(isdeployed)
param1 = str2num(param1);
%param2 expects a string
paramk = str2num(paramk);
end
Parallel For Loops on the Cluster
• Not designed for multiple nodes on shared
filesystem:
– Race condition from concurrent writes to:
~/.matlab/local_scheduler_data/
• Easy fix: redirect directory to /scratch.local
Parallel For Loops on the Cluster
1. Setup (done once, before submitting jobs):
[iendres2 ~]$ ln –sv
/scratch.local/tmp/USER/matlab/local_scheduler_data
~/.matlab/local_scheduler_data
(Replace USER with your netid)
Parallel For Loops on the Cluster
2. Wrap matlabpool function to make sure tmp data exists:
function matlabpool_robust(varargin)
if(matlabpool('size')>0)
matlabpool close
end
% make sure the directories exist and are empty for good measure
system('rm -rf /scratch.local/tmp/USER/matlab/local_scheduler_data');
system(sprintf('mkdir -p
/scratch.local/tmp/USER/matlab/local_scheduler_data/R%s', version('release')));
% Run it:
matlabpool (varargin{:});
Warning:
/scratch.local may get filled up by other users, in which case this
will fail.
Best Practices
• Interactive Sessions
– Don’t leave idle sessions open, it ties up the nodes
• Job arrays
– Still working on kinks in the scheduler, I managed
to kill the whole cluster
• Disk I/O
– Minimize I/O for best performance
– Avoid small reads and writes due to metadata
overhead
Maintenance
• “Preventive maintenance (PM) on the cluster is
generally scheduled on a monthly basis on the third
Wednesday of each month from 8 a.m. to 8 p.m.
Central Time. The cluster will be returned to service
earlier if maintenance is completed before schedule.”
Resources
• Beginner’s guide:
https://campuscluster.illinois.edu/user_info/doc/beginner.html
• More comprehensive user’s guide:
http://campuscluster.illinois.edu/user_info/doc/index.html
• Cluster Monitor:
http://clustat.ncsa.illinois.edu/taub/
• Simple sample job scripts
/projects/consult/pbs/
• Forum
https://campuscluster.illinois.edu/forum/