5636 - Parallel Numerical Algorithms
Mike Peardon — mjp@maths.tcd.ie
http://www.maths.tcd.ie/~mjp/5636
School of Mathematics
Trinity College Dublin
Hilary Term 2016
Mike Peardon — mjp@maths.tcd.ie (TCD)
5636
Hilary Term 2016
1/5
Data-parallel vectors and
matrices
Mike Peardon — mjp@maths.tcd.ie (TCD)
5636
Hilary Term 2016
2/5
A data-parallel representation of a vector (1)
A vector in n-dimensions is represented naturally as a flat,
length-n 1-d array.
A good data-parallel representation on p processes would
try to distribute the entries evenly across nodes. Unless
n mod p = 0, some nodes will get more data than others.
Example: length-10 vector on 4 MPI processes
At least one process will need to store d pn e = 3 numbers.
10 = 3 + 3 + 2 + 2
= 3+3+3+1
These two partitionings have the same largest data size (and so are
essentially equivalent for most cases). Best choice depends on how
we want to lay out and access the data.
Mike Peardon — mjp@maths.tcd.ie (TCD)
5636
Hilary Term 2016
3/5
A data-parallel representation of a vector (2)
Data-parallel: each entry is stored on a single MPI process.
Can map data to processes in many ways
Best choice depends on what algorithm will act on the data.
Common examples: block and cyclic decompositions:
0
1
2
3
4
5
6
7
1
2
3
4
5
6
7
8
9
Cyclic
Nloc=(3,3,2,2)
Proc 0
Proc 2
Proc 1
Proc 3
Mike Peardon — mjp@maths.tcd.ie (TCD)
9
Block
Nloc=(3,3,3,1)
0
8
5636
Hilary Term 2016
4/5
A data-parallel representation of a vector (3)
Inherently parallel operations (such as vector-vector
addition) are simple to implement and parallelise efficiently.
To change one entry, each process needs to determine if it
holds the data, and if so, where.
To access one entry, the process that is storing the data needs
to MPI_Bcast to all others.
Global reduction operations, such as evaluating a · b need
MPI_Reduce or MPI_Allreduce.
Finite-precision means code using different data-layouts
evaluate sums in different orders which can yield different
answers (so eg if this is an issue, try Kahan summation).
Mike Peardon — mjp@maths.tcd.ie (TCD)
5636
Hilary Term 2016
5/5