High Productivity Computing:
Taking HPC Mainstream
Lee Grant
Technical Solutions Professional
High Performance Computing
leegrant@microsoft.com
Challenge: High Productivity Computing
“Make high-end computing easier and more productive to use.
Emphasis should be placed on time to solution, the major
metric of value to high-end computing users…
A common software environment for scientific computation encompassing
desktop to high-end systems will enhance productivity gains by promoting
ease of use and manageability of systems.”
2004 High-End Computing Revitalization Task Force
Office of Science and Technology Policy, Executive Office of the President
X64 Server
The Data Pipeline
Data Gathering
Discovery and
Browsing
Science
Exploration
Domain specific
analyses
Scientific Output
“Raw” data includes
sensor output, data
downloaded from
agency or collaboration
web sites, papers
(especially for ancillary
data
“Raw” data browsing for
discovery (do I have
enough data in the right
places?), cleaning (does
the data look obviously
wrong?), and light weight
science via browsing
“Science variables” and
data summaries for early
science exploration and
hypothesis testing.
Similar to discovery and
browsing, but with
science variables
computed via gap filling,
units conversions, or
simple equation.
“Science variables”
combined with models,
other specialized code,
or statistics for deep
science understanding.
Scientific results via
packages such as
MatLab or R2. Special
rendering package such
as ArcGIS.
Paper preparation.
Operations per second for serial code
for traditional software
Free Lunch
Free Lunch Is Over For Traditional Software
24 GHz
1 Core
12 GHz
1 Core
3 GHz
2 Cores
6 GHz
1 Core
3 GHz
4 Cores
3 GHz
8 Cores
3 GHz
1 Cor 3 GHz
1 Cores
Additional operations per second if code can take advantage of concurrency
No Free Lunch for traditional software
(Without highly concurrent software it won’t get any faster!)
Microsoft’s Vision for HPC
“Provide the platform, tools and broad ecosystem to reduce the complexity of HPC by
making parallelism more accessible to address future computational needs.”
Reduced Complexity
Mainstream HPC
Developer Ecosystem
Ease deployment for
larger scale clusters
Address needs of traditional
supercomputing
Increase number of parallel
applications and codes
Simplify management for
clusters of all scale
Address emerging
cross-industry
computation trends
Offer choice of parallel
development tools,
languages and libraries
Integrate with
existing infrastructure
Enable non-technical users to
harness the power of HPC
Drive larger universe of
developers and ISVs
Microsoft HPC++ Solution
Application Benefits
The most productive distributed
application development
environment
Cluster Benefits
Complete HPC cluster platform
integrated with the enterprise
infrastructure
System Benefits
Cost-effective, reliable and high
performance server operating
system
Windows HPC Server 2008
Integrated security via Active
Directory
Support for batch, interactive and
service-oriented applications
High availability scheduling
Interoperability via OGF’s HPC
Basic Profile
Rapid large scale deployment and
built-in diagnostics suite
Integrated monitoring,
management and reporting
Familiar UI and rich scripting
interface
Systems
Management
Storage
Access to SQL, Windows and Unix
file servers
Key parallel file server vendor
support (GPFS, Lustre, Panasas)
In-memory caching options
Job
Scheduling
MPI
MS-MPI stack based on MPICH2
reference implementation
Performance improvements for
RDMA networking and multi-core
shared memory
MS-MPI integrated with Windows
Event Tracing
List or Heat Map view cluster at a
glance
Group compute nodes based on hardware, software and
custom attributes; Act on groupings.
Receive alerts for failures
Track long running operations and
access operation history
Pivoting enables correlating
nodes and jobs together
Integrated Job Scheduling
Services oriented HPC apps
Expanded Job Policies
Support for Job Templates
Improve interoperability
with mixed IT infrastructure
Skip/Demo
Skip/Demo
Node/Socket/Core Allocation
Windows HPC Server can help your application make the
best use of multi-core systems
Node 1
Node 2
P1
P0
S0
S1
J1
P2
P1
P0
P3
P0
P2
P2
P3
P1
S1
S0
J1
P0
P1
P3
P2
P3
P0
P1
J2
P1
P0
S2
P2
P0
J3
P1
J3
P0
S3
J1
P3
P2
P1
S3
S2
J3
P3
J3
J1: /numsockets:3 /exclusive: false
J3: /numcores:4 /exclusive: false
P2
P3
J2: /numnodes:1
P2
P3
Job submission: 3 methods
• Command line
–
–
–
–
–
•
Programmatic
•
•
Job submit /headnode:Clus1 /Numprocessors:124 /nodegroup:Matlab
Job submit /corespernode:8 /numnodes:24
Job submit /failontaskfailure:true /requestednodes:N1,N2,N3,N4
Job submit /numprocessors:256 mpiexec \\share\mpiapp.exe
[Completel Powershell system mgmt commands are available as well]
Support for C++ & .Net
languages
Web Interface
•
Open Grid Forum: “HPC
Basic Profile”
using Microsoft.Hpc.Scheduler;
class Program
{
static void Main()
{
IScheduler store = new Scheduler();
store.Connect(“localhost”);
ISchedulerJob job = store.CreateJob();
job.AutoCalculateMax = true;
job.AutoCalculateMin = true;
ISchedulerTask task = job.CreateTask();
task.CommandLine = "ping 127.0.0.1 -n *";
task.IsParametric = true;
task.StartValue = 1;
task.EndValue = 10000;
task.IncrementValue = 1;
task.MinimumNumberOfCores = 1;
task.MaximumNumberOfCores = 1;
job.AddTask(task);
store.SubmitJob(job, @"hpc\user“,
"p@ssw0rd");
}
}
Scheduling MPI jobs
•Job Submit /numprocessors:7800 mpiexec hostname
•Start time: 1 second, Completion time: 27 seconds
NetworkDirect
A new RDMA networking interface built for speed and stability
– 2 usec latency, 2 GB/sec
bandwidth on ConnectX
• OpenFabrics driver for
Windows includes support for
Network Direct, Winsock Direct
and IPoIB protocols
MPI App
MS-MPI
Windows Sockets
(Winsock + WSD)
RDMA
Networking
Networking
Networking
WinSock
Direct
Hardware
Hardware
Provider
TCP/Ethernet
Networking
Networking
Networking
NetworkDirect
Hardware
Hardware
Provider
Networking Hardware
Hardware
Networking
User
Mode Access Layer
TCP
User
Mode
Kernel By-Pass
• Verbs-based design for close fit
with native, high-perf
networking interfaces
• Equal to Hardware-Optimized
stacks for MPI microbenchmarks
Socket-Based
App
IP
NDIS
Networking
Networking
Mini-port
Hardware
Hardware
Driver
Kernel
Mode
Networking Hardware
Hardware
Networking
Hardware Driver
Networking Hardware
Hardware
Networking
Networking
Hardware
(ISV) App
CCP
Component
OS
Component
IHV
Component
Spring 2008, NCSA, #23
9472 cores, 68.5 TF, 77.7%
Spring 2008, Umea, #40
5376 cores, 46 TF, 85.5%
Spring 2008, Aachen, #100
2096 cores, 18.8 TF, 76.5%
Fall 2007, Microsoft, #116
2048 cores, 11.8 TF, 77.1%
30% efficiency
improvement
Windows HPC Server 2008
Windows Compute Cluster 2003
Spring 2007, Microsoft, #106
2048 cores, 9 TF, 58.8%
Spring 2006, NCSA, #130
896 cores, 4.1 TF
November 2008 Top500
Customers
“It is important that our IT environment is easy to use and support.
Windows HPC is improving our performance and manageability.”
-- Dr. J.S. Hurley, Senior Manager, Head Distributed Computing, Networked Systems
Technology, The Boeing Company
“Ferrari is always looking for the most advanced technological solutions and, of course, the same
applies for software and engineering. To achieve industry leading power-to-weight ratios,
reduction in gear change times, and revolutionary aerodynamics, we can rely on Windows HPC
Server 2008. It provides a fast, familiar, high performance computing platform for our users,
engineers and administrators.”
-- Antonio Calabrese, Responsabile Sistemi Informativi (Head of Information Systems), Ferrari
“Our goal is to broaden HPC availability to a wider audience than just power users. We believe that
Windows HPC will make HPC accessible to more people, including engineers, scientists, financial
analysts, and others, which will help us design and test products faster and reduce costs.”
-- Kevin Wilson, HPC Architect, Procter & Gamble
“We are very excited about utilizing the Cray CX1 to support our research activities,” said Rico
Magsipoc, Chief Technology Officer for the Laboratory of Neuro Imaging. “The work that we do in
brain research is computationally intensive but will ultimately have a huge impact on our
understanding of the relationship between brain structure and function, in both health and disease.
Having the power of a Cray supercomputer that is simple and compact is very attractive and
necessary, considering the physical constraints we face in our data centers today.”
Porting Unix Applications
• Windows Subsystem for Unix applications
– Complete SVR-5 and BSD UNIX environment with 300
commands, utilizes, shell scripts, compilers
– Visual Studio extensions for debugging POSIX applications
– Support for 32 and 64-bit applications
• Recent port of WRF weather model
– 350K lines, Fortran 90 and C using MPI, OpenMP
– Traditionally developed for Unix HPC systems
– Two dynamical cores, full range of physics options
• Porting experience
– Fewer than 750 lines of code changed in makefiles/scripts
– Level of effort similar to port to any new version of UNIX
– Performance on par with the Linux systems
• India Interoperability Lab, MTC Bangalore
– Industry Solutions for Interop jointly with partners
– HPC Utility Computing Architecture
– Open Source Applications on HPC Server 2008
(NAMD, PL_POLY, GROMACS)
High Productivity Modeling
Languages/Runtimes
C++, C#, VB
F#, Python, Ruby, Jscript
Fortran (Intel, PGI)
OpenMP, MPI
.Net Framework
LINQ: language
integrated query
Dynamic Language
Runtime
Fx/JIT/GC improvements
Native support for Web
Services
Team Development
Team portal: version
control, scheduled build,
bug tracking
Test and stress
generation
Code analysis, Code
coverage
Performance analysis
IDE
Rapid application
development
Parallel debugging
Multiprocessor builds
Work flow design
MSFT || Computing Technologies
Task Concurrency
IFx / CCR
•Robotics-based
manufacturing assembly
Maestro
line
•Silverlight
TPL / Olympics
PPL
viewer
Local
Computing
•Automotive
control system
WCF
•Internet –based photo
services
WF
Cluster-TPL
•Ultrasound imaging
equipment
•Media encode/decode
PLINQ
•Image processing/
enhancement
OpenMP
TPL / PPL
•Data visualization
CDS
MPI / MPI.Net
•Enterprise search, OLTP,
Cluster SOA
collab
•Animation / CGI rendering
Cluster-PLINQ
•Weather forecasting
•Seismic monitoring
•Oil exploration
Data Parallelism
Distributed/
Cloud Computing
UDF
UDF
UDF
UDF
UDF
UDF
UDF
Head Nodes
Supports SOA
functionality
WCF Brokers.
UDF
Compute Nodes
Each performs UDF
Tasks as called
From WCF Broker
SOA Broker Performance
Low latency
Messages/sec (25 ms compute time)
Round Trip Latency ( ms )
1.6
High throughput
1.4
1.2
1
0.8
0.6
0.4
0.2
0
6000
5000
4000
3000
2000
1000
0
0
50
100
150
Number of clients
Message Size ( bytes )
WSD
IPoIB
Gige
0k pingpong
1k pingpong
4k pingpong
16k pingpong
200
MPI.NET
• Supports all .NET languages
(C#, C++, F#, ..., even Visual Basic!)
• Natural expression of MPI in C#
if (world.Rank == 0)
world.Send(“Hello, World!”, 1, 0);
else
string msg = world.Receive<string>(0, 0);
string[] hostnames =
comm.Gather(MPI.Environment.ProcessorName, 0);
double pi = 4.0*comm.Reduce(dartsInCircle,(x, y) =>
return x + y, 0) / totalDartsThrown;
• Negligible overhead (relative to C) over TCP
Allinea DDT VS Debugger Add-in
Skip/Demo
NetPIPE Performance
Throughput (Mbps)
100
10
C (Native )
C# (Prim itive )
C# (Se rialize d)
1
1.E+00
1.E+01
1.E+02
1.E+03
1.E+04
0.1
0.01
Message Size (Bytes)
1.E+05
1.E+06
1.E+07
Parallel Extensions to .NET
• Declarative data parallelism (PLINQ)
var q = from n in names.AsParallel()
where n.Name == queryInfo.Name &&
n.State == queryInfo.State &&
n.Year >= yearStart &&
n.Year <= yearEnd
orderby n.Year ascending
select n;
• Imperative data and task parallelism (TPL)
Parallel.For(0, n, i=> {
result[i] = compute(i);
});
• Data structures and coordination constructs
Example: Tree Walk
Sequential
static void ProcessNode<T>(Tree<T>
tree, Action<T> action) {
if (tree == null) return;
Thread Pool
static void ProcessNode<T>(Tree<T> tree, Action<T> action) {
if (tree == null) return;
Stack<Tree<T>> nodes = new Stack<Tree<T>>();
Queue<T> data = new Queue<T>();
nodes.Push(tree);
while (nodes.Count > 0) {
Tree<T> node = nodes.Pop();
data.Enqueue(node.Data);
if (node.Left != null) nodes.Push(node.Left);
if (node.Right != null) nodes.Push(node.Right);
}
ProcessNode(tree.Left, action);
ProcessNode(tree.Right, action);
action(tree.Data);
}
using (ManualResetEvent mre = new ManualResetEvent(false)) {
int waitCount = Environment.ProcessorCount;
WaitCallback wc = delegate {
bool gotItem;
do {
T item = default(T);
lock (data) {
if (data.Count > 0) {
item = data.Dequeue();
gotItem = true;
}
else gotItem = false;
}
if (gotItem) action(item);
} while (gotItem);
if (Interlocked.Decrement(ref waitCount) == 0) mre.Set();
};
for (int i = 0; i < Environment.ProcessorCount - 1; i++) {
ThreadPool.QueueUserWorkItem(wc);
}
wc(null);
mre.WaitOne();
}
}
Example: Tree Walk
Parallel Extensions (with Task)
static void ProcessNode<T>(Tree<T> tree, Action<T> action) {
if (tree == null) return;
Task t = Task.Create(delegate { ProcessNode(tree.Left, action); });
ProcessNode(tree.Right, action);
action(tree.Data);
t.Wait();
}
Parallel Extensions (with Parallel)
static void ProcessNode<T>(Tree<T> tree, Action<T> action) {
if (tree == null) return;
Parallel.Do(
() => ProcessNode(tree.Left, action),
() => ProcessNode(tree.Right, action),
() => action(tree.Data) );
}
Parallel Extensions (with PLINQ)
static void ProcessNode<T>(Tree<T> tree, Action<T> action) {
tree.AsParallel().ForAll(action);
}
F# is...
...a functional, object-oriented,
imperative and explorative
programming language for .NET
Libraries
Scalable
Explorative
Succinct
Strongly
Typed
Interoperable
F#
Efficient
Interactive F# Shell
C:\fsharpv2>bin\fsi
MSR F# Interactive, (c) Microsoft Corporation, All Rights Reserved
F# Version 1.9.2.9, compiling for .NET Framework Version v2.0.50727
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
NOTE:
See 'fsi --help' for flags
Commands: #r <string>;; reference (dynamically load) the given DLL.
#I <string>;; add the given search path for referenced DLLs.
#use <string>;; accept input from the given file.
#load <string> ...<string>;;
load the given file(s) as a compilation unit.
#time;; toggle timing on/off.
#types;; toggle display of types on/off.
#quit;; exit.
Visit the F# website at http://research.microsoft.com/fsharp.
Bug reports to fsbugs@microsoft.com. Enjoy!
> let rec f x = (if x < 2 then x else f (x-1) + f (x-2));;
val f : int -> int
> f 6;;
val it = 8
val it : int
Example: Taming Asynchronous I/O
using System;
using System.IO;
using System.Threading;
public static void ReadInImageCallback(IAsyncResult asyncResult)
{
public static void ProcessImagesInBulk()
ImageStateObject state = (ImageStateObject)asyncResult.AsyncState;
{
public class BulkImageProcAsync
Stream stream = state.fs;
Console.WriteLine("Processing images... ");
int bytesRead = stream.EndRead(asyncResult);
{
long t0 = Environment.TickCount;
if
(bytesRead
!=
numPixels)
public const String ImageBaseName = "tmpImage-";
NumImagesToFinish = numImages;
throw
new
Exception(String.Format
public const int numImages = 200;
AsyncCallback readImageCallback = new
("In ReadInImageCallback, got the wrong number of " + AsyncCallback(ReadInImageCallback);
public const int numPixels = 512 * 512;
"bytes from the image: {0}.", bytesRead));
for (int i = 0; i < numImages; i++)
ProcessImage(state.pixels, state.imageNum);
{
// ProcessImage has a simple O(N) loop,
and
you
can
vary
the
number
stream.Close();
ImageStateObject state = new ImageStateObject();
// of times you repeat that loop to make the application more CPUstate.pixels = new byte[numPixels];
// Now write out the image.
// bound or more IO-bound.
state.imageNum = i;
// =Using
public static int processImageRepeats
20; asynchronous I/O here appears not to be best practice.// Very large items are read only once, so you can make the
// It ends up swamping the threadpool, because the threadpool
// buffer on the FileStream very small to save memory.
// threads are blocked on I/O requests that were just queued toFileStream fs = new FileStream(ImageBaseName + i + ".tmp",
// Threads must decrement NumImagesToFinish,
and protect
// the threadpool.
FileMode.Open, FileAccess.Read, FileShare.Read, 1, true);
// their access to it through a mutex.
FileStream fs = new FileStream(ImageBaseName + state.imageNum +state.fs = fs;
public static int NumImagesToFinish = numImages;
".done", FileMode.Create, FileAccess.Write, FileShare.None,fs.BeginRead(state.pixels, 0, numPixels, readImageCallback,
public static Object[] NumImagesMutex = 4096,
new Object[0];
false);
state);
0, numPixels);
// WaitObject is signalled when all fs.Write(state.pixels,
image processing is done.
}
public static Object[] WaitObject = fs.Close();
new Object[0];
public class ImageStateObject
// Determine whether all images are done being processed.
// This application model uses too much memory.
// If not, block until all are finished.
{
// Releasing memory as soon as possible is a good idea,
bool mustBlock = false;
public byte[] pixels;
// especially global state.
lock (NumImagesMutex)
public int imageNum;
state.pixels = null;
{
public FileStream fs;
fs = null;
if (NumImagesToFinish > 0)
// Record that an image is finished now.
}
mustBlock = true;
Processing
200 images in
parallel
lock (NumImagesMutex)
{
NumImagesToFinish--;
if (NumImagesToFinish == 0)
{
Monitor.Enter(WaitObject);
Monitor.Pulse(WaitObject);
Monitor.Exit(WaitObject);
}
}
}
if (mustBlock)
{
Console.WriteLine("All worker threads are queued. " +
" Blocking until they complete. numLeft: {0}",
NumImagesToFinish);
Monitor.Enter(WaitObject);
Monitor.Wait(WaitObject);
Monitor.Exit(WaitObject);
}
long t1 = Environment.TickCount;
Console.WriteLine("Total time processing images: {0}ms",
(t1 - t0));
}
}
}
Example: Taming Asynchronous I/O
Equivalent F#
code
(same perf)
Open the file
synchronously
Read from the
file,
asynchronously
let ProcessImageAsync(i) =
async { let inStream = File.OpenRead(sprintf "source%d.jpg" i)
let! pixels
= inStream.ReadAsync(numPixels)
let pixels'
= TransformImage(pixels,i)
let outStream = File.OpenWrite(sprintf "result%d.jpg" i)
do! outStream.WriteAsync(pixels')
do
Console.WriteLine "done!" }
Write the result
asynchronously
let ProcessImagesAsync() =
Async.Run (Async.Parallel
[ for i in 1 .. numImages -> ProcessImageAsync(i) ])
Generate the
tasks and queue
them in parallel
The Coming of Accelerators
Current Offerings
Microsoft
AMD
nVidia
Intel
Apple
Accelerator
Brook+
RapidMind
Ct
Grand
Central
D3DX,
DaVinci, FFT,
Scan
ACML-GPU
cuFFT,
cuBLAS, cuPP
MKL++
CoreImage
CoreAnim
Compute
Shader
CAL
CUDA
LRB Native
OpenCL
Any Processor
AMD CPU or
GPU
nVidia GPU
Intel CPU
Larrabee
Any Processor
DirectX11 Compute Shader
• A new processing model for GPUs
–
–
–
–
Integrated with Direct3D
Supports more general constructs
Enables more general data structures
Enables more general algorithms
• Image/Post processing:
– Image Reduction, Histogram, Convolution, FFT
– Video transcode, superResolution, etc.
• Effect physics
– Particles, smoke, water, cloth, etc.
• Ray-tracing, radiosity, etc.
• Gameplay physics, AI
FFT Performance Example
• Complex 1024x1024 2-D FFT:
– Software
– Direct3D9
– CUFFT
– Prototype DX11
– Latest chips
42ms
15ms
8ms
6ms
3ms
6 GFlops
17 GFlops 3x
32 GFlops 5x
42 GFlops 6x
100 GFlops
• Shared register space and random access
writes enable ~2x speedups
IMSL .NET Numerical Library
•
•
•
•
•
•
Linear Algebra
Eigensystems
Interpolation and
Approximation
Quadrature
Differential Equations
Transforms
•
•
•
•
•
•
Nonlinear Equations
Optimization
Basic Statistics
Nonparametric Tests
Goodness of Fit
Regression
•
•
•
•
•
•
Variances, Covariances and
Correlations
Multivariate Analysis
Analysis of Variance
Time Series and Forecasting
Distribution Functions
Random Number Generation
Research
Integrate


Data acquisition from
source systems and
integration
Data transformation
and synthesis
Analyze


Data enrichment, with
business logic,
hierarchical views
Data discovery via
data mining
Report


Data presentation
and distribution
Data access for
the masses
Data Browsing with Excel
Annual
Mean
Monthly
Mean
Weekly
Mean
Courtesy Catherine van Ingen, MSR
Datamining with Excel
Integrated algorithms
• Text Mining
• Neural Nets
• Naïve Bayes
• Time Series
• Sequent Clustering
• Decision Trees
• Association Rules
Workflow Design for Sharepoint
Microsoft HPC++ Labs:
Academic Computational Finance Service
Taking HPC Mainstream
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or
trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation
as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of
Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.