Using the NCSA Supercluster
for Cactus
NT Cluster Group
Computing and Communications Division
NCSA
Mike Showerman
mshow@ncsa.uiuc.edu
University of Illinois at Urbana-Champaign
NCSA NT 320 Pentium® CPU Cluster
256 CPUs - Parallel MPI
64 HP Kayak XU systems
Dual 550 MHz
Pentium III Xeon
1 GB RAM
64 HP Kayak XU systems
Dual 300MHz
Pentium II
512 MB memory
64 CPUs - Serial
32 Compaq PWS 6000
Dual 333 MHz Pentium II
512 MB memory
64 Dual 550 MHz Pentium III Xeon HP Kayaks back-to-back
University of Illinois at Urbana-Champaign
NT Cluster Software
• Microsoft NT 4.0 Server
• LSF from Platform for queuing system
• Cygnus toolkit
– UNIX tools for porting
• MPI
– HPVM from Chien’s CSAG group
• Microsoft and Digital compilers
– C, C++, F77, F90
• OpenMP
– This does not span hosts. 2 processor only
University of Illinois at Urbana-Champaign
Cactus Performance on 32 Processors
Elapsed Time (seconds)
2000
1800
R12000
1600
NT 550 Dual
1400
NT 550 Single
1200
1000
800
600
400
200
0
0
10
20
30
40
50
60
70
Grid Size
Source: http://origin.aei-potsdam.mpg.de/projects/cluster by O. Wehrens,
NCSA NT group for NT 550 results
University of Illinois at Urbana-Champaign
Obtaining an Account
• Cluster is in “friendly user” mode
– No production level support
– System upgrades are common
– Your research must be tolerant of this environment
– Allocation data collected,
but not currently billed
• Contact NCSA consulting to get an account
– Details at http://www.ncsa.uiuc.edu/SCD/Hardware/NTCluster/
University of Illinois at Urbana-Champaign
Accessing the Cluster
• Windows Terminal Server Interactive Nodes
– Multiuser form of Windows NT
– Surprisingly good performance
– Access Methods
– Windows RDP Client from Microsoft
– Available at http://ntsc-ts1.ncsa.uiuc.edu
– Citrix ICA client
– Available for most platforms
– http://www.citrix .com to download the clients
– A java applet client is available at http://ntsc-ts1.ncsa.uiuc.edu
– X Windows
– Rsh ntsc-ts1.ncsa.uiuc.edu wincenter
– Option: -resolution ex: -resolution 800x600
– Option: -depth for color depth ex: -depth 8
University of Illinois at Urbana-Champaign
Windows NT on a Web Page
University of Illinois at Urbana-Champaign
Compiling Your Executable
• Choose the desired Terminal server client
– Connect to ntsc-ts1.ncsa.uiuc.edu
• Open a command prompt
– Select “run” from the START menu.
– Enter cmd in the box Hit OK
• Start bash
– Type “cygnus” at the prompt
– This is the Cygnus Windows toolkit
– Tcsh is available for csh scripts
• Run make
– Equivalent to gmake
• Unix style commands
– tar, make, awk, grep, ar, ranlib, cp, rm etc.
University of Illinois at Urbana-Champaign
Modifying your Makefile
• Editors available
– Windows style – notepad and write
– Unix style – vi and emacs
• Compilers
– C/C++ = cl
– FORTRAN = df -> f77 and f90
• Linker
– LINK or use the compiler
• Options available
Cl /? Df /? or link /? to list options
University of Illinois at Urbana-Champaign
Building an Application
• Makefile differences
– Set compiler to be cl or df
– Change extension on object files (.obj by default)
– For MPI set libs
– mpi.lib fm.lib wsock32.lib kernel32.lib advapi32.lib
– LIB and include paths are already set in environment
– May also need dfport for FORTRAN applications
• Suffix rules for makefiles
– .c.o : ; $(CC) -c $(CCFLAGS) $(CDEFS) $< -Fo$*.o
– .f.o : ; $(F77) -c $(F77FLAGS) $< -Fo$*.o
• Link
– Link objects.o /OUT:example.exe
University of Illinois at Urbana-Champaign
bsub Command
• Bsub
– Current working directory is passed through LSF
– Your current working directory will be the UNC path of your network
drive ex: bsub from z:\bob will have a cwd of \\ntsc-file1\home\bob
– Arguments
– -q qname
– Current queues are
• 48NoMPI - no myrinet, single processor or machine applications
• 128hp - 300 MHz HP systems for parallel work, up to 128p
• 128hp550 - 550 MHz HP systems for >32p applications, up to
128p
– -n is for LSF to allocate N machines
– -np is for MPI to use N machines
– -o output file (for standard out)
University of Illinois at Urbana-Champaign
bsub Command (cont.)
• Use prun to start the processes
– \\ntsc-file1\home\lsf\prun runs the trailing command on all
of the hosts allocated
• Example Command
– bsub -q 128hp -o out.txt -n 32 \lsf\prun helloworld -np 32 -key UniqueText
• If you use standard input
– Wrap your command with cmd /c “type file|cmd“
– Example
–
Z:\Bsub –n 32 –o out –q 128hp \lsf\prun cmd /c“type
input.txt | helloworld –np 32 –key test32”
• if using bash, escape all \ with another
– Bash$ bsub -q 128hp -o out -n N \\lsf\\prun exec args -np N -key ABC
University of Illinois at Urbana-Champaign
Interacting with jobs
• Your jobs can be listed by bjobs
– Bjobs –u all shows all jobs by all users
– Bjobs –l provides detailed job information
• Bkill <jobid>
– Sends a control-c to your processes
• Bpeek <jobid>
– Outputs the current job output (stdout)
• Bqueues
– Information about the queues (pending jobs…)
• Lsload
– Provides current information about the hosts
• LSF DOCS are on http://ntsc-file1.ncsa.uiuc.edu
University of Illinois at Urbana-Champaign
NT Cluster Monitoring
Available at:
\\ntsc-file1\glmon\glmon
(this can be entered from
the command line or “run”
Box)
Shows in real time:
•machine status
•systems status
•current load
•load by user name
•Load by jobid
•All running/pending jobs
University of Illinois at Urbana-Champaign
Advanced Topics
• Changing passwords
– Log on and select “Windows security” from the start menu.
– Select “change password”
– Also, You must change your password for LSF
• Run lspasswd from a command prompt
• You must type in the same password as your NT password
• Storage
– Currently there are no quotas in place
– Your Windows Desktop
– Please do not store data here
– Your home directory
– Located in \\ntsc-file1\home and mapped to z:\ on the terminal server
– This area is backed up and is used to store semi-permanent data
– Scratch
– Located at \\ntsc-file2\scratch Larger data sets can be stored here
University of Illinois at Urbana-Champaign
Advanced Topics (cont.)
• Using scratch
– Map a drive letter for scratch
– Net use y: \\ntsc-file2\scratch
• net use y: \\\\ntsc-file2\\scratch under bash
– Move to the y: drive by typing y: at a command prompt
• Cd //y using bash
– Create a directory and move data here
– Submit the job from this drive
• Getting your files
– The terminal servers have both ftp and Kerberos ftp clients
– See http://www.ncsa.uiuc.edu/SCD/Hardware/NTCluster/
University of Illinois at Urbana-Champaign
Cactus on the cluster
• Getting the source
– From a command prompt type the following
– Cvs –d :pserver:cvs_anon@cvs.cactuscode.org:/cactus checkout
• Source Modification
– Due to our MPI (fix is on the way)
– In Cactus/src/main/ProcessCommandLine.c
– Line 85 remove CCTKi_CommandLineHelp();
• Open a bash shell in the cactus directory
– Make checkout to get thorns
– Make mpi MPI=CUSTOM MPI_LIBS=“mpi.lib fm.lib advapi32.lib
kernel32.lib wsock32.lib” MPI_INC_DIRS=“d:/apps/hpvm/include”
– Make mpi
– done
University of Illinois at Urbana-Champaign
Future Directions
–
–
–
–
–
Better integration with the mass storage system
High performance shared filesystems
Improved reliability and process management
Advanced user support
Advancements in interconnects
– Better scaling
– Better performance
University of Illinois at Urbana-Champaign