MATRIX MULTIPLY WITH
DRYAD
B649 Course Project Introduction
Matrix Multiplication
• Fundamental kernel algorithm used by many applications
• Examples: Graph Theory, Physics, Electronics
Scalability Issues:
• Run on single machine:
• Memory overhead increase in terms of N^2
• CPU overhead increase in terms of N^3
• Run on multiple machines:
• Communication overhead increase in terms of N^2
memory overhead
CPU overhead
1E+10
100000000
1000000
10000
100
0
500
1000
1500
Matrix Multiply Approaches
Programming
Mdoel
Algorithm
Customized
Libraries
User
Implementation
Sequential
Naïve approach,
tiles matrix
multiply,
Blas_dgemm
Vendor supplied
package (ie, Intel,
AMD Blas),
ATLAS
Fortran, C, C++,
C#, Java
Shared memory
parallelism
Row Partition
ATLAS
Multi Threads,
TPL, PLINQ,
OpenMP
Distributed
memory
parallelism
Row Column
Partition,
Fox Algorithm
ScalePack
OpenMPI, Twister,
Dryad
Why DryadLINQ?
• Dryad is a general purpose runtime that supports the
processing of data intensive application in Windows
• DryadLINQ is a high level programming language and
compiler for Dryad
• Applicability:
• Dryad transparently deal with the parallelism, scheduling, fault.
tolerance, messaging, and workload balancing issues.
• SQL-like interface, based on .NET platform, easy to have code.
• Performance:
• Intelligent job execution engine, optimized execution plan.
• Scale out for thousands of machines.
Parallel Algorithms for Matrix Multiplication
• MM algorithms can deal with matrices distributed on
rectangular grids
• No single algorithm always achieves best performance on
different matrix and grid shapes.
• MM Algorithms can be classified into categories according
to the communication primitives
• Row Partition
• Row Column Partition
• Fox Algorithm (BMR) – broadcast, multiply, roll up
Row Partition
• Heavy communication overhead
• Large memory usage per node
• The full Matrix B is copied to every node
• The Matrix A row blocks are distributed to each node
Pseudo Code sample:
Partition matrix A by rows
Broadcast matrix B
Distributed matrix A row blocks
Compute the matrix C row blocks
Row Column Partition
• Heavy communication overhead
• Scheduling overhead for each iteration
Column Block 1
Column Block 2
Column Block 3
...
Column Block n
• Moderate memory usage
1
Row Block 1
Block
(1,1)
Block
(1,2)
Block
(1,3)
...
Block
(1,n)
2
Row Block 2
Block
(2,1)
Block
(2,2)
Block
(1,3)
...
Block
(2,n)
3
Row Block 3
Block
(m,0)
Block
(m,1)
Block
(m,3)
...
Block
(m,n)
…
...
m
Row Block m
...
Block
(m,n)
B
Matrix
Pseudo Code sample:
Partitioned matrix A by rows
Partitioned matrix B by columns
For each iteration i:
broadcast matrix A row block i
distributed matrix B column blocks
compute matrix C row blocks
Iterations
A
Matrix
C
Matrix
...
Block
(m,0)
Block
(m,1)
Block
(m,3)
...
Node 1
Node 2
Node 3
Node n
Fox Algorithm
Stage One
Stage Two
Fox algorithm
• Less communication overhead than other approach
• Scale well for large matrices sizes
Pseudo Code sample:
Partitioned matrix A, B to blocks
For each iteration i:
1) broadcast matrix A block (i%N,
i%N) to row i
2) compute matrix C blocks add the
to the previous result
3) roll-up matrix B block
Performance Analysis on Fox algorithm
Cache Size Turning Point
Relative Parallel Efficiency
• Cache Issue
• Cache miss (size),
pollution, confliction
• Tiles matrix multiply
• Memory Issue
• Size (memory paging)
• Bandwidth, latency
OpenMPI/Threads/Cblas on 16 nodes
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Grain Size Per Node
1node_8cores
16nodes_8cores
• Absolute performance
degrade as problem size
increase for both cases
• Single node performance
worse than multiple nodes
due to memory issue.
10^3Mflops
100
10
1
MPI/Threads/Cblas with Various Problem Sizes
Multicore level parallelism
• To use every core on a compute node for Dryad Job, the
task must be programmed with multicore technology. (i.e.
Task Parallel Library<TPL>, Thread, PLINQ)
• For each thread, it will compute one row in matrix C or
several rows in matrix C depends on the implementation.
• By using TPL or PLINQ, the optimization for threads is
implicit and easier to use.
Timeline for term long project
• Stage One
• Familiar with HPC cluster
• Sequential MM with C#
• Multithreaded MM with C#
• Performance comparison of above two approaches
• Stage Two
• Familiar with DryadLINQ Interface
• Implement Row Partition algorithm with DryadLINQ
• Performance study
• Stage Three
• Refinement experiments results
• Report and presentation
Backup slides
Dryad Job Submission
Client machine
ToTable
HPC Cluster
DryadLINQ
.NET
program
Query Expr
Distributed
query plan
Invoke
Query Vertex
code
JM
foreach
Output
.Net Objects DryadTable
(11)
Results
Input
Tables
Dryad
Execution
Output Tables
Input: C# and LINQ data objects DryadLINQ distributed data objects.
DryadLINQ translates LINQ programs into distributed Dryad computations:
C# methods become code running on the vertices of a Dryad job.
Output: DryadLINQ distributed data objects .Net objects
Dryad Job Execution Flow
Performance on one Node
Performance on Multiple Node
Analysis for three algorithms
Performance for three algorithms
• Test done on 16 nodes of Tempest, using one core per
node.
Performance for Multithreaded MM
• Test done on one node of Tempest, 24 cores
25
20
Speed-up
15
10
5
TPL
Thread
PLINQ
0
2400
4800
7200
9600
12000
Scale of Square Matrix
14400
16800
19200
0
