Step-by-Step Guide for Job Submission in
Windows HPC Server 2008 Beta 2
Microsoft Corporation
Published: May 2008
Abstract
This guide describes the various methods that you can use to submit jobs to an HPC cluster in
Windows® HPC Server 2008 Beta 2. Jobs are the means by which users make use of the
computation resources provided by their Windows HPC Server 2008 cluster. Once a user submits
jobs, the jobs remain in a queue until the allocated recourses become available, and then the
tasks in the jobs are run by those resources.
Copyright
This document supports a preliminary release of a software product that may be changed
substantially prior to final commercial release, and is the confidential and proprietary information
of Microsoft Corporation. It is disclosed pursuant to a non-disclosure agreement between the
recipient and Microsoft. This document is provided for informational purposes only and Microsoft
makes no warranties, either express or implied, in this document. Information in this document,
including URL and other Internet Web site references, is subject to change without notice. The
entire risk of the use or the results from the use of this document remains with the user. Unless
otherwise noted, the example companies, organizations, products, domain names, e-mail
addresses, logos, people, places, and events depicted herein are fictitious, and no association
with any real company, organization, product, domain name, e-mail address, logo, person, place,
or event is intended or should be inferred. Complying with all applicable copyright laws is the
responsibility of the user. Without limiting the rights under copyright, no part of this document may
be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by
any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose,
without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual
property rights covering subject matter in this document. Except as expressly provided in any
written license agreement from Microsoft, the furnishing of this document does not give you any
license to these patents, trademarks, copyrights, or other intellectual property.
© 2008 Microsoft Corporation. All rights reserved.
Microsoft, Windows, and Windows PowerShell are either registered trademarks or trademarks of
Microsoft Corporation in the United States and/or other countries.
All other trademarks are property of their respective owners.
Contents
Step-by-Step Guide for Job Submission in Windows HPC Server 2008 Beta 2 ............................. 4
Jobs and Tasks Overview ............................................................................................................ 4
Submitting a Job .............................................................................................................................. 5
Submitting a Job using the HPC Job Manager ............................................................................ 5
Step 1: Create a New Job ......................................................................................................... 5
Step 2: Add Tasks to the Job .................................................................................................... 6
Step 3: Submit the Job ............................................................................................................. 7
Submitting a Job using the Command Line or PowerShell .......................................................... 7
Step 1: Create a New Job ......................................................................................................... 8
Step 2: Add Tasks to the Job .................................................................................................... 8
Step 3: Submit the Job ............................................................................................................. 9
Checking the Status of a Job or Task.............................................................................................. 9
Viewing a Job or Task in the HPC Job Manager ......................................................................... 9
View a Job or Task from the Command Line or Windows PowerShell ...................................... 10
Appendix 1: Job and Task Properties ........................................................................................... 10
Appendix 2: Sample Commands ................................................................................................... 12
[This is prerelease documentation and is subject to change in future releases.]
Step-by-Step Guide for Job Submission in
Windows HPC Server 2008 Beta 2
Jobs are the means by which users can make use of the computation resources provided by their
Windows® HPC Server 2008 cluster. Jobs can be submitted to the system through a variety of different
interfaces. Once submitted, jobs are placed in the queue to await execution. When resources become
available, jobs are allocated resources and their tasks are executed.
Jobs and Tasks Overview
An HPC Server 2008 job is basically a resource request. A job says “Give me X number of Y resources
for Z time.” For example, a job might ask for 8 nodes in the “Have DDR Interconnects” group for 2 hours.
When a job is submitted to the scheduler, it is put in the queue where it waits until the necessary
resources become available and are allocated for the job’s use. Thus how long a job waits in the queue
depends on a number of factors, including:

What resources the job needs

What jobs are currently running

What other jobs are in the queue

The scheduling policies configured by the cluster’s administrator
The amount of time a job must wait in the queue can be decreased by:

Submitting the job with a higher priority

Specifying an accurate maximum run time for the job

Requesting as small of a minimum number of resources as possible

Being as unspecific as possible about the types of resources required
Once resources have been allocated to a job, the scheduler uses those resources to execute the job’s
Tasks. A task is basically a command line to execute, along with some associated information about how
that command line needs to run.
A job may contain one task, many tasks, or even no tasks at all (“Go reserve some resources for me and
I’ll figure out what to do with them later). The tasks within a job may depend on one another and need to
complete in a certain order, or they may be entirely independent.
Tasks themselves come in two different types:

Basic tasks are simply a command line with some meta-data about how to execute that command
line. When running an MPI application, generally a basic task is created where the command line is
“mpiexec . . .” and the task requests a number of resources spanning many nodes.

Parametric tasks are a command line including the wildcard character “*”. The command is run
multiple times, with each instance using a different value to replace the “*”.
[This is prerelease documentation and is subject to change in future releases.]
Submitting a Job
This example creates a job which:

Uses the “Default” job template

Runs for no more than 10 minutes

Runs on no more than 4 processors

Runs “processfile.exe” on 200 different input files (input1.dat, input2.dat, . . . input200.dat)
There are three basic steps for creating a job:
1. Create a New Job.
a. Specify the appropriate Job Template for the job.
b. Provide any necessary details on the jobs resources requirements.
2. Add the task(s) to the job.
3. Submit the job.
The procedure for creating a job is described for the three primary user interfaces:

HPC Job Manager

Command prompt

PowerShell
Submitting a Job using the HPC Job Manager
Step 1: Create a New Job
To create a new job
1. Open the HPC Job Manager (or, if you are an admin, open the HPC Cluster Manager and navigate
to the Job Management tab).
2. In the Action pane, select New Job.
3. Set the necessary details about the job on the Job Details tab:
a. Provide a Job name, e.g. “Data File Processing.”
b. Select the Default job template from the drop down list of job templates.
c.
Set the maximum runtime of the job to 10 minutes:
i.
Check the box labeled Do not run this job for more than.
ii.
Set the run time to 0 days, 0 hours, and 10 minutes.
d. Set the Maximum size of the job to 4 cores:
i.
Set the type of resource for the job to “Core.”
ii.
Use the radio button to disable Auto calculate on the job’s Maximum.
iii.
Set the Maximum to 4.
[This is prerelease documentation and is subject to change in future releases.]
Step 2: Add Tasks to the Job
Now that all of the job settings are correctly set, the next step is to add the required tasks to the job:
To add tasks to the job
1. In the Create New Job dialog, navigate to the Task List tab.
2. Click the Add button to get a drop-down list of task add options, and select Parametric Task (A
parametric task is a single task which will be run many times over a different sets of data. The same
goal could be achieved by creating many individual tasks).
3. Set the sweep to run from 1 – 200, with an increment of 1 (this will create 200 steps, with the indices
1, 2, 3, 4 . . . 200):
a. Set the Start Value to 1.
b. Set the End Value to 200.
c.
Set the Increment Value to 1.
4. Set the command line for the sweep to “ProcessFile.exe”.
5. Set the Standard input for the sweep to read the 200 data files:
a. In the Standard input text box, enter “input*.dat” (this command will be run 200 times, with the
numbers 1 through 200 replacing the * character for each step in the sweep).
6. Add the task to the job by clicking the Save button.
[This is prerelease documentation and is subject to change in future releases.]
Figure 1 Creating a Parametric Sweep Task
Step 3: Submit the Job
Once the job properties are set and any tasks have been added, submit the job to the cluster by clicking
the Submit button in the Create New Job dialog.
Submitting a Job using the Command Line or
PowerShell
Users can also use the Command Line or PowerShell interfaces to submit jobs to the cluster:
[This is prerelease documentation and is subject to change in future releases.]
Step 1: Create a New Job
Using the command line
To create a new job

Run the following command:
job new /jobtemplate:"Default" /jobname:"Data File Processing" /runtime:0:0:10 /numcores:*-4
This command produces the following output.
Created job, ID: 1
Using PowerShell
To create a new job

Run the following command:
New-HpcJob -TemplateName "Default" -Name "Data File Processing" -RunTime "0:0:10" -NumCores
*-4
This command produces the following output.
Id
Name
State
Owner
Priority
NumberOfTasks
--
----
-----
-----
--------
-------------
2
Data Fi... Configuring
REDMOND\jbarnard
Normal
0
Step 2: Add Tasks to the Job
Using the command line
To add tasks to a job

Run the following command:
job add 1 /parametric:200 /stdin:"input*.dat" ProcessFile.exe
This command produces the following output.
Task 1.1 added.
Using PowerShell
To add tasks to a job

Run the following command:
Add-HpcTask -JobId 2 -Parametric -End 200 -Stdin "input*.dat" -CommandLine "ProcessFile.exe"
This command produces the following output.
Id
Name
State
Owner
Priority
NumberOfTasks
--
----
-----
-----
--------
-------------
[This is prerelease documentation and is subject to change in future releases.]
2
Data Fi... Configuring
REDMOND\jbarnard
Normal
200
Step 3: Submit the Job
Using the command line
To submit a job

Run the following command:
job submit /id:1
This command produces the following output.
Job has been submitted. ID: 1.
Using PowerShell
To submit a job

Run the following command:
Submit-HpcJob -Id 2
This command produces the following output.
Id
Name
State
Owner
Priority
NumberOfTasks
--
----
-----
-----
--------
-------------
2
Data Fi... Queued
REDMOND\jbarnard
Normal
200
Checking the Status of a Job or Task
Once your job is submitted, you’ll probably want to see what it is up to. This can be done using the HPC
Job Manager, the Command Line Interface, or the HPC Pack PowerShell Cmdlets.
Viewing a Job or Task in the HPC Job Manager
To view the status of a job or task in the HPC Job Manager
1. Select a set of jobs in the left-hand navigation pane.
a. Select All Jobs -> Active to display all of the jobs in the system that are not yet completed.
b. Select My Jobs to show all jobs that you have submitted.
2. The list of jobs now appears in the center list pane.
3. The controls at the top of this pane allow you to filter the displayed list of jobs.
a. Enter “Data File Processing” in the Job Name filter box.
b. Click the Magnifying Glass icon to filter the list.
4. Single-click a job to highlight it. Notice that the tasks in that job appear in the details pane.
5. Double-click a job to view details and results for the task.
[This is prerelease documentation and is subject to change in future releases.]
6. Click the check-box labeled Expand parametric tasks have the system display the 200 steps in your
parametric sweep as 200 separate commands.
7. Double-click one of the tasks to view details and results for the task:
a. For the job you submitted earlier, an error message should be visible explaining that the input file
could not be found.
View a Job or Task from the Command Line or
Windows PowerShell
To view the status of a job or task from the command line or PowerShellTM, run the appropriate command
from the following table.
Action
Command Line
PowerShell
View a list of my active jobs
Job list
Get-HpcJob
View a list of all jobs
Job list /all
Get-HpcJob –Owner * -State All
View a list of all jobs
submitted in the last X days
Job list /submittime:<X>
Get-HpcJob –BeginSubmitDate <Date X
Days Ago>
View the details of a job
Job view <JobID>
Get-HpcJob –Id <JobID>
View the list of tasks in a job Job listtasks <JobID>
Get-HpcTask –JobId <JobID>
View the details of a task
Get-HpcTask –JobId <JobID> -TaskId
<TaskID>
Task view
<JobID>.<TaskID>
Appendix 1: Job and Task Properties
The tables below contain a list of all user-settable job and task properties which can be used to control
the allocation and execution of jobs and tasks.
Job Property
Description
Job ID
The numeric ID of this job.
Job Name
The name of this job.
Project
The name of the project of which this job is a part.
Template
The name of the Job Template against which this job is being submitted.
Priority
The priority of the job; it can be one of the five values (Lowest, Below Normal,
Normal, Above Normal, Highest).
Run Time
The hard limit on the amount of time this job will be allowed to run. If the task is
still running after the specified Run Time has been reached, it will be
automatically canceled by the scheduler. (dd:hh:mm)
Number of Processors
The number of processors required by this job; the job will run on at least the
Minimum and no more than the Maximum. If set to *, the scheduler will
automatically calculate these values based on the job's tasks.
[This is prerelease documentation and is subject to change in future releases.]
Job Property
Description
Number of Sockets
The number of sockets required by this job; the job will run on at least the
Minimum and no more than the Maximum. If set to *, the scheduler will
automatically calculate these values based on the job's tasks.
Number of Nodes
The number of nodes required by this job; the job will run on at least the
Minimum and no more than the Maximum. If set to *, the scheduler will
automatically calculate these values based on the job's tasks.
Exclusive
If True, no other jobs can be run on a compute node at the same time as this
job.
Requested Nodes
A list of nodes; the job can only be run on nodes that are in this list.
Node Groups
A list of node groups; the job can only be run on nodes that are in all of these
groups.
Memory
The Minimum amount of memory (in MB) that must be present on any nodes
that this job is run on.
Processors Per Node
The Minimum number of processors that must be present on any nodes that this
job is run on.
Node Ordering
The ordering to use when selecting nodes for the job. For example, ordering by
"More Memory" will allocate the available nodes with the largest amount of
memory.
Preemptable
If True, this job can be preempted by a higher priority job. If False, the job
cannot be preempted.
Fail On Task Failure
If True, then failure of any task in the job will cause the entire job to fail
immediately.
Run Until Cancelled
If True, this job will run until it is cancelled or its runtime is expired. It will not
stop when there are no tasks left within it.
Licenses
A list of licenses that are required for the job. This list can be interpreted by a
Job Activation Filter, but it is not used by the Windows HPC Job Scheduler.
Table 1: Job Properties
Task Property
Description
Task ID
The numeric ID of this task.
Task Name
The name of this task.
Command Line
The command line that will be executed by this task.
Working Directory
The working directory to be used during execution for this task.
Standard Input
The path (relative to the working directory) to the file from which the stdin of
this task should be read.
Standard Output
The path (relative to the working directory) to the file to which the stdout of
this task should be written.
Standard Error
The path (relative to the working directory) to the file to which the stderr of
this task should be written.
Number of Processors
The number of processors required by this task; the task will run on at least
the Minimum and no more than the Maximum.
Number of Sockets
The number of sockets required by this task; the task will run on at least the
Minimum and no more than the Maximum.
Number of Nodes
The number of nodes required by this task; the task will run on at least the
[This is prerelease documentation and is subject to change in future releases.]
Task Property
Description
Minimum and no more than the Maximum.
Required Nodes
Lists the nodes that must be assigned to this task and its job in order to run.
Run Time
The hard limit on the amount of time this task will be allowed to run. If the
task is still running after the specified Run Time has been reached, it will be
automatically canceled by the scheduler. (dd:hh:mm)
Exclusive
If True, no other tasks can be run on a compute node at the same time as
this task.
Rerunnable
If the task runs and fails, setting Rerunnable to True means the scheduler
will attempt to rerun the task. If Rerunnable is False, the task will fail after
the first run attempt fails.
Environment Variables
Specifies the environment variables to set in the task's runtime environment.
Environment variables must be separated by commas in the format:
name1=value1
Sweep Start Index*
The starting index for the sweep.
Sweep End Index*
The ending index for the sweep.
Sweep Increment*
The amount to increment the sweep index at each step of the sweep.
Table 2: Task Properties
Appendix 2: Sample Commands
The tables below provide some sample commands which can be used to submit a variety of different
jobs, as well as text explaining exactly how these jobs will be run by the scheduler.
Command Line
Result
Job submit myapp.exe
Runs myapp.exe on a single processor. The undirected output will be
available in the Output property of the job’s task.
Job submit
Runs myapp.exe on a single processor using a headnode fileshare as
/workdir:\\headnode\MyFolder the working directory. The input and output are redirected to/from files
/stdout:data.out /stdin:data.in in that directory.
myapp.exe
Job submit /numcores:4-8
/parametric:100
/stdin:input*.dat myapp.exe
Runs between 4 and 8 simultaneous instances of the myapp.exe
command. Myapp.exe will be executed 100 times, on files input1.dat,
input2.dat, input3.dat, through input4.dat.
Job submit /exclusive:true
/requestednodes:MyNode01,
MyNode02 myapp.exe
Runs the myapp.exe on either MyNode01 or MyNode02. No other
command may execute on that node will this command is running.
Job submit
/NodeGroups:TestGroup
myapp.exe
Runs myapp.exe on a node in the node group “TestGroup.”
Table 3: Simple and Parametric Jobs
[This is prerelease documentation and is subject to change in future releases.]
Command Line
Result
job submit /numcores:6
/stdout:myOutput.txt mpiexec
MyApp.exe
Runs MyApp.exe on 6 processors with the standard output directed to
a file myOutput.txt in the user’s home directory on the head node (as
defined in the %USERPROFILE% environment variable on the head
node).
job submit /numnodes:2
mpiexec MyApp.exe
Runs one MyApp.exe process on each of two compute nodes.
STDOUT will be stored in the scheduler database in the task’s
“output” field.
job submit /numnodes:2
Runs one MyApp.exe process on each of two specific compute nodes/requestednodes:Node1,Node2 Node1 and Node2
mpiexec MyApp.exe
Job submit /numnodes:16
mpiexec -cores 4 MyApp.exe
Runs four MyApp.exe processes on each of the 16 nodes assigned to
the job – regardless of the number of physical cores on each node.
job submit /numcores:24
/workdir:\\headnode\MyFolder
mpiexec MyApp.exe
Specifies a single, shared working directory for all 24 MyApp.exe
processes in the job.
job submit /numcores:128
mpiexec -env
MPICH_NETMASK
157.59.0.0/255.255.0.0
MyApp.exe
Runs 128 MyApp.exe processes where the MPI data traffic is routed
to a specific network in the cluster (157.59.x.x/255.255.0.0 in this
example) by setting an environment variable (MPICH_NETMASK) on
the MPI processes.
job submit /numcores:16
mpiexec –affinity MyApp.exe
The Windows Operating System optimizes use of multi-core systems
by dynamically shifting workload to under-utilized cores. However,
this refinement can be detrimental to the performance of some HPC
applications. The -affinity flag prevents the OS from moving the MPI
processes between cores- they will run exclusively on the core upon
which they were started.
Table 4: MPI Jobs