Parallelizing and Optimizing Programs
for GPU Acceleration using CUDA
Martin Burtscher
Department of Computer Science
CUDA Optimization Tutorial
 Martin Burtscher
 burtscher@txstate.edu
 http://www.cs.txstate.edu/~burtscher/
 Tutorial slides
 http://www.cs.txstate.edu/~burtscher/tutorials/COT5/slides.pptx
CUDA Optimization Tutorial
2
High-end CPU-GPU Comparison
Cores
Active threads
Frequency
Peak performance (SP)
Peak mem. bandwidth
Maximum power
Price
Xeon E5-2687W
8 (superscalar)
2 per core
3.1 GHz
397 GFlop/s
51 GB/s
150 W
$1900
Kepler GTX 680
1536 (simple)
~11 per core
1.0 GHz
3090 GFlop/s
192 GB/s
195 W*
$500*
Release dates
Xeon: March 2012
Kepler: March 2012
*entire card
CUDA Optimization Tutorial
Intel
Nvidia
3
GPU Advantages
 Performance
 8x as many operations executed per second
 Main memory bandwidth
 4x as many bytes transferred per second
 Cost-, energy-, and size-efficiency
 29x as much performance per dollar
 6x as much performance per watt
 11x as much performance per area
(based on peak values)
CUDA Optimization Tutorial
4
GPU Disadvantages
 Clearly, we should be using GPUs all the time
 So why aren’t we?
 GPUs can only execute some types of code fast
 Need lots of data parallelism, data reuse, & regularity
 GPUs are harder to program and tune than CPUs
 In part because of poor tool support
 In part because of their architecture
 In part because of poor support for irregular codes
CUDA Optimization Tutorial
5
Outline (Part I)
 Introduction
 GPU programming
 N-body example
 Porting and tuning
 Other considerations
Hightechreview.com
 Conclusions
CUDA Optimization Tutorial
6
CUDA Programming Model
 Non-graphics
 C/C++ with extensions
 Function launch
programming
 Uses GPU as massively
parallel co-processor
CPU
PCI-Express
bus
GPU
 SIMT (single-instruction
multiple-threads) model
 Thousands of threads
needed for full efficiency
CUDA Optimization Tutorial

Calling functions on GPU
 Memory management

GPU memory allocation,
copying data to/from GPU
 Declaration qualifiers

Device, shared, local, etc.
 Special instructions

Barriers, fences, etc.
 Keywords

threadIdx, blockIdx
7
Calling GPU Kernels
 Kernels are functions that run on the GPU
 Callable by CPU code
 CPU can continue processing while GPU runs kernel
KernelName<<<m, n>>>(arg1, arg2, ...);
 Launch configuration (programmer selectable)
 GPU spawns m blocks with n threads (i.e., m*n
threads total) that run a copy of the same function
 Normal function parameters: passed conventionally

Different address space, should never pass CPU pointers
CUDA Optimization Tutorial
8
GPU Architecture
 GPUs consist of Streaming Multiprocessors (SMs)
 1 to 30 SMs per chip (run blocks)
 SMs contain Processing Elements (PEs)
 8, 32, or 192 PEs per SM (run threads)
Shared
Memory
Shared
Memory
Adapted from NVIDIA
CUDA Optimization Tutorial
Shared
Memory
Shared
Memory
Shared
Memory
Shared
Memory
Shared
Memory
Shared
Memory
Global Memory
9
Block Scalability
 Hardware can assign blocks to SMs in any order
 A kernel with enough blocks scales across GPUs
 Not all blocks may be resident at the same time
GPU with 2 SMs
Kernel
GPU with 4 SMs
Block 0 Block 1
Block 2 Block 3
Block 0
Block 1
Block 4 Block 5
Block 6 Block 7
Block 2
Block 3
Block 4
Block 5
Block 6
Block 7
CUDA Optimization Tutorial
time
Block 0
Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
Block 7
Adapted from NVIDIA
10
GPU Memories
GPU
 Separate from CPU memory
Block (0, 0)
 CPU can access GPU’s global
& constant mem. via PCIe bus
 Requires slow explicit transfer
Block (1, 0)
Shared Memory (SRAM)
Registers
Registers
Shared Memory (SRAM)
Registers
Registers
 Visible GPU memory types
Thread (0, 0) Thread (1, 0)
 Registers (per thread)
 Local mem. (per thread)
 Shared mem. (per block)

Software-controlled cache
 Global mem. (per kernel)
 Constant mem. (read only)
CUDA Optimization Tutorial
C
P
U
Thread (0, 0) Thread (1, 0)
Global + Local Memory (DRAM)
Constant Memory (DRAM, cached)
Adapted from NVIDIA
 Slow communic. between blocks
11
SM Internals (Fermi and Kepler)
 Caches
 Software-controlled shared memory
 Hardware-controlled incoherent L1 data cache
 64 kB combined size, can be split 16/48, 32/32, 48/16
 Synchronization support
 Fast hardware barrier within block (__syncthreads())
 Fence instructions: memory consistency & coherency
 Special operations
 Thread voting (warp-based reduction operations)
CUDA Optimization Tutorial
12
Block and Thread Allocation Limits
 Blocks assigned to SMs
 Until first limit reached
 Threads assigned to PEs
t0 t1 t2 … tm
SM 0 SM 1
MT IU
MT IU
PE
PE
Blocks
t0 t1 t2 … tm
Shared
Memory
 8/16 active blocks/SM
 1024, 1536, or 2048


Blocks
Shared
Memory
 Hardware limits


resident threads/SM
512 or 1024 threads/blk
16k, 32k, or 64k regs/SM
16 kB or 48 kB shared
memory per SM
216-1 or 231-1 blks/kernel
Adapted from NVIDIA
CUDA Optimization Tutorial
13
Warp-based Execution
 32 contiguous threads form a warp
 Execute same instruction in same cycle (or disabled)
 Warps are scheduled out-of-order with respect to
each other to hide latencies
 Thread divergence
 Some threads in warp jump to different PC than others
 Hardware runs subsets of warp until they re-converge
 Results in reduction of parallelism (performance loss)
CUDA Optimization Tutorial
14
Thread Divergence
 Non-divergent code
if (threadID >= 32) {
some_code;
} else {
other_code;
}
Thread ID:
0123…
31
 Divergent code
if (threadID >= 13) {
some_code;
} else {
other_code;
}
Thread ID:
0123…
31
disabled
disabled
Adapted from NVIDIA
CUDA Optimization Tutorial
Adapted from NVIDIA
15
Parallel Memory Accesses
 Coalesced main memory access (16/32x faster)
 Under some conditions, HW combines multiple (half)
warp memory accesses into a single coalesced access
CC 1.3: 64-byte aligned 64-byte line (any permutation)
 CC 2.x+3.0: 128-byte aligned 128-byte line (cached)

 Bank-conflict-free shared memory access (16/32)
 No superword alignment or contiguity requirements
CC 1.3: 16 different banks per half warp or same word
 CC 2.x+3.0 : 32 different banks + 1-word broadcast each

CUDA Optimization Tutorial
16
Coalesced Main Memory Accesses
single coalesced access
NVIDIA
CUDA Optimization Tutorial
one and two coalesced accesses*
NVIDIA
17
Outline
 Introduction
 GPU programming
 N-body example
 Porting and tuning
 Other considerations
NASA/JPL-Caltech/SSC
 Conclusions
CUDA Optimization Tutorial
18
N-body Simulation
 Time evolution of physical system
 System consists of bodies
 “n” is the number of bodies
 Bodies interact via pair-wise forces
RUG
 Many systems can be modeled in this way
 Star/galaxy clusters (gravitational force)
 Particles (electric force, magnetic force)
Cornell
CUDA Optimization Tutorial
19
Simple N-body Algorithm
 Algorithm
Initialize body masses, positions, and velocities
Iterate over time steps {
Accumulate forces acting on each body
Update body positions and velocities based on force
}
Output result
 More sophisticated n-body algorithms exist
 Barnes Hut algorithm (covered in Part II)
 Fast Multipole Method (FMM)
CUDA Optimization Tutorial
20
Key Loops (Pseudo Code)
bodySet = ...; // input
for timestep do { // sequential
foreach Body b1 in bodySet { // O(n2) parallel
foreach Body b2 in bodySet {
if (b1 != b2) {
b1.addInteractionForce(b2);
}
}
}
foreach Body b in bodySet {
// O(n) parallel
b.Advance();
}
}
// output result
CUDA Optimization Tutorial
21
Force Calculation C Code
struct Body {
float mass, posx, posy, posz; // mass and 3D position
float velx, vely, velz, accx, accy, accz; // 3D velocity & accel
} *body;
for (i = 0; i < nbodies; i++) {
. . .
for (j = 0; j < nbodies; j++) {
if (i != j) {
dx = body[j].posx - px; // delta x
dy = body[j].posy - py; // delta y
dz = body[j].posz - pz; // delta z
dsq = dx*dx + dy*dy + dz*dz; // distance squared
dinv = 1.0f / sqrtf(dsq + epssq); // inverse distance
scale = body[j].mass * dinv * dinv * dinv; // scaled force
ax += dx * scale; // accumulate x contribution of accel
ay += dy * scale; az += dz * scale; // ditto for y and z
}
}
. . .
CUDA Optimization Tutorial
22
Outline
 Introduction
 GPU programming
 N-body example
 Porting and tuning
 Other considerations
 Conclusions
CUDA Optimization Tutorial
23
N-body Algorithm Suitability for GPU
 Lots of data parallelism
 Force calculations are independent
 Should be able to keep SMs and PEs busy
 Sufficient memory access regularity
 All force calculations access body data in same order*
 Should have lots of coalesced memory accesses
 Sufficient code regularity
 All force calculations are identical*
 There should be little thread divergence
 Plenty of data reuse
 O(n2) operations on O(n) data
 CPU/GPU transfer time is insignificant
CUDA Optimization Tutorial
24
C to CUDA Conversion
 Two CUDA kernels
 Force calculation
 Advance position and velocity
 Benefits
 Force calculation requires over 99.9% of runtime

Primary target for acceleration
 Advancing kernel unimportant to runtime
But allows to keep data on GPU during entire simulation
 Minimizes GPU/CPU transfers

CUDA Optimization Tutorial
25
C to CUDA Conversion
__global__ void ForceCalcKernel(int nbodies, struct Body *body, ...) {
. . .
}
__global__ void AdvancingKernel(int nbodies, struct Body *body, ...) {
. . .
}
Indicates GPU kernel that CPU can call
int main(...) {
Separate address spaces, need two pointers
Body *body, *bodyl;
Allocate memory on GPU
. . .
cudaMalloc((void**)&bodyl, sizeof(Body)*nbodies);
cudaMemcpy(bodyl, body, sizeof(Body)*nbodies, cuda…HostToDevice);
for (timestep = ...) {
ForceCalcKernel<<<1, 1>>>(nbodies, bodyl, ...); Copy CPU data to GPU
AdvancingKernel<<<1, 1>>>(nbodies, bodyl, ...);
}
cudaMemcpy(body, bodyl, sizeof(Body)*nbodies, cuda…DeviceToHost);
cudaFree(bodyl);
Copy GPU data back to CPU
. . .
Call GPU kernel with 1 block
and 1 thread per block
}
CUDA Optimization Tutorial
26
Evaluation Methodology
 Systems and compilers
 CC 1.3: Quadro FX 5800, nvcc 3.2

30 SMs, 240 PEs, 1.3 GHz, 30720 resident threads
 CC 2.0: Tesla C2050, nvcc 3.2

14 SMs, 448 PEs, 1.15 GHz, 21504 resident threads
 CC 3.0: GeForce GTX 680, nvcc 4.2

8 SMs, 1536 PEs, 1.0 GHz, 16384 resident threads
 Inputs and metric
 1k, 10k, or 100k star clusters (Plummer model)
 Median runtime of three experiments, excluding I/O
CUDA Optimization Tutorial
27
1-Thread Performance
 Problem size
 n=10000, step=1
 n=10000, step=1
 n=3000, step=1
 Slowdown rel. to CPU
 CC 1.3: 72.4
 CC 2.0: 36.7
 CC 3.0: 68.1
(Note: comparing different GPUs to
different CPUs)
CUDA Optimization Tutorial
 Performance
 1 thread is one to two
orders of magnitude
slower on GPU than CPU
 Reasons
 No caches (CC 1.3)
 Not superscalar
 Slower clock frequency
 No SMT latency hiding
28
Using N Threads
 Approach
 Eliminate outer loop
 Instantiate n copies of inner loop, one per body
 Threading
 Blocks can only hold 512 or 1024 threads
Up to 8/16 blocks can be resident in an SM at a time
 SM can hold 1024, 1536, or 2048 threads
 We use 256 threads per block (works for all of our GPUs)

 Need multiple blocks

Last block may not need all of its threads
CUDA Optimization Tutorial
29
Using N Threads
__global__ void ForceCalcKernel(int nbodies, struct Body *body, ...) {
for (i = 0; i < nbodies; i++) {
i = threadIdx.x + blockIdx.x * blockDim.x; // compute i
if (i < nbodies) { // in case last block is only partially used
for (j = ...) {
. . .
}
}
}
__global__ void AdvancingKernel(int nbodies,...) // same changes
#define threads 256
int main(...) {
. . .
int blocks = (nbodies + threads - 1) / threads; // compute block cnt
for (timestep = ...) {
ForceCalcKernel<<<1, 1blocks, threads>>>(nbodies, bodyl, ...);
AdvancingKernel<<<1, 1blocks, threads>>>(nbodies, bodyl, ...);
}
}
CUDA Optimization Tutorial
30
N Thread Speedup
 Relative to 1 GPU thread  Performance
 CC 1.3: 7781 (240 PEs)
 Speedup much higher
 CC 2.0: 6495 (448 PEs)
than number of PEs
(32, 14.5, and 7.9 times)
 Due to SMT latency hiding
 CC 3.0: 12150 (1536 PEs)
 Relative to 1 CPU thread
 CC 1.3: 107.5
 CC 2.0: 176.7
 CC 3.0: 176.2
CUDA Optimization Tutorial
 Per-core performance
 CPU core delivers up to
4.4, 5, and 8.7 times as
much performance as a
GPU core (PE)
31
Using Scalar Arrays
 Data structure conversion
 Arrays of structs are bad for coalescing
 Bodies’ elements (e.g., mass fields) are not adjacent
 Optimize data structure
 Use multiple scalar arrays, one per field (need 10)
 Results in code bloat but often much better speed
structs in array
scalar arrays
CUDA Optimization Tutorial
32
Using Scalar Arrays
__global__ void
// change all
}
__global__ void
// change all
}
ForceCalcKernel(int nbodies, float *mass, ...) {
“body[k].blah” to “blah[k]”
AdvancingKernel(int nbodies, float *mass, ...) {
“body[k].blah” to “blah[k]”
int main(...) {
float *mass, *posx, *posy, *posz, *velx, *vely, *velz, *accx, *accy,*accz;
float *massl, *posxl, *posyl, *poszl, *velxl, *velyl, *velzl, ...;
mass = (float *)malloc(sizeof(float) * nbodies); // etc
. . .
cudaMalloc((void**)&massl, sizeof(float)*nbodies); // etc
cudaMemcpy(massl, mass, sizeof(float)*nbodies, cuda…HostToDevice); // etc
for (timestep = ...) {
ForceCalcKernel<<<blocks, threads>>>(nbodies, massl, posxl, ...);
AdvancingKernel<<<blocks, threads>>>(nbodies, massl, posxl, ...);
}
cudaMemcpy(mass, massl, sizeof(float)*nbodies, cuda…DeviceToHost); // etc
. . .
}
CUDA Optimization Tutorial
33
Scalar Array Speedup
 Problem size
 n=100000, step=1
 n=100000, step=1
 n=300000, step=1
 Performance
 Threads access same
memory locations, not
adjacent ones

Always combined but not
really coalesced access
 Relative to struct
 Slowdowns may be due to
 CC 1.3: 0.83
DRAM page/TLB misses
 CC 2.0: 0.96
 CC 3.0: 0.82
CUDA Optimization Tutorial
 Scalar arrays
 Still needed (see later)
34
Constant Kernel Parameters
 Kernel parameters
 Lots of parameters due to scalar arrays
 All but one parameter never change their value
 Constant memory
 “Pass” parameters only once
 Copy them into GPU’s constant memory
 Performance implications
 Reduced parameter passing overhead
 Constant memory has hardware cache
CUDA Optimization Tutorial
35
Constant Kernel Parameters
__constant__ int nbodiesd;
__constant__ float dthfd, epssqd, float *massd, *posxd, ...;
__global__ void ForceCalcKernel(int step) {
// rename affected variables (add “d” to name)
}
__global__ void AdvancingKernel() {
// rename affected variables (add “d” to name)
}
int main(...) {
. . .
cudaMemcpyToSymbol(massd, &massl, sizeof(void *)); // etc
. . .
for (timestep = ...) {
ForceCalcKernel<<<1, 1>>>(step);
AdvancingKernel<<<1, 1>>>();
}
. . .
}
CUDA Optimization Tutorial
36
Constant Mem. Parameter Speedup
 Problem size
 Performance
 n=1000, step=10000
 Minimal perf. impact
 n=1000, step=10000
 May be useful for very
 n=3000, step=10000
 Speedup
 CC 1.3: 1.015
 CC 2.0: 1.016
 CC 3.0: 0.971
CUDA Optimization Tutorial
short kernels that are
often invoked
 Benefit
 Less shared memory
used on CC 1.3 devices
37
Using the RSQRTF Instruction
 Slowest kernel operation
 Computing one over the square root is very slow
 GPU has slightly imprecise but fast 1/sqrt instruction
(frequently used in graphics code to calculate inverse
of distance to a point)
 IEEE floating-point accuracy compliance
 CC 1.x is not entirely compliant
 CC 2.x and above are compliant but also offer faster
non-compliant instructions
CUDA Optimization Tutorial
38
Using the RSQRT Instruction
for (i = 0; i < nbodies; i++) {
. . .
for (j = 0; j < nbodies; j++) {
if (i != j) {
dx = body[j].posx - px;
dy = body[j].posy - py;
dz = body[j].posz - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = 1.0f / sqrtf(dsq + epssq);
dinv = rsqrtf(dsq + epssq);
scale = body[j].mass * dinv * dinv * dinv;
ax += dx * scale;
ay += dy * scale;
az += dz * scale;
}
}
. . .
}
CUDA Optimization Tutorial
39
RSQRT Speedup
 Problem size
 n=100000, step=1
 n=100000, step=1
 Performance
 No change for CC 1.3

 n=300000, step=1
 Speedup
 CC 1.3: 0.99
 CC 2.0: 1.83
 CC 3.0: 1.64
CUDA Optimization Tutorial
Compiler automatically
uses less precise RSQRTF
as most FP ops are not
fully precise anyhow
 83% speedup for CC 2.0



Over entire application
Compiler defaults to
precise instructions
Explicit use of RSQRTF
indicates imprecision okay
40
Using 2 Loops to Avoid If Statement
 “if (i != j)” causes thread divergence
 Break loop into two loops to avoid if statement
for (j = 0; j < nbodies; j++) {
if (i != j) {
dx = body[j].posx - px;
dy = body[j].posy - py;
dz = body[j].posz - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssq);
scale = body[j].mass * dinv * dinv * dinv;
ax += dx * scale;
ay += dy * scale;
az += dz * scale;
}
}
CUDA Optimization Tutorial
41
Using 2 Loops to Avoid If Statement
for (j = 0; j < i; j++) {
dx = body[j].posx - px;
dy = body[j].posy - py;
dz = body[j].posz - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssq);
scale = body[j].mass * dinv * dinv * dinv;
ax += dx * scale;
ay += dy * scale;
az += dz * scale;
}
for (j = i+1; j < nbodies; j++) {
dx = body[j].posx - px;
dy = body[j].posy - py;
dz = body[j].posz - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssq);
scale = body[j].mass * dinv * dinv * dinv;
ax += dx * scale;
ay += dy * scale;
az += dz * scale;
}
CUDA Optimization Tutorial
42
Loop Duplication Speedup
 Problem size
 n=100000, step=1
 n=100000, step=1
 n=300000, step=1
 Performance
 No change for 2.0 & 3.0

 45% slowdown for CC 1.3

 Speedup
 CC 1.3: 0.55
 CC 2.0: 1.00
 CC 3.0: 1.00
CUDA Optimization Tutorial
Divergence moved to loop
Unclear why
 Discussion
 Not a useful optimization
 Code bloat
 A little divergence is okay
(only 1 in 3125 iterations)
43
Blocking using Shared Memory
 Code is memory bound
 Each warp streams in all bodies’ masses and positions
 Block inner loop
 Read block of mass & position info into shared mem
 Requires barriers (fast hardware barrier within SM)
 Advantage
 A lot fewer main memory accesses
 Remaining main memory accesses are fully coalesced
(due to usage of scalar arrays)
CUDA Optimization Tutorial
44
Blocking using Shared Memory
__shared__ float posxs[threads], posys[threads], poszs[…], masss[…];
j = 0;
for (j1 = 0; j1 < nbodiesd; j1 += THREADS) { // first part of loop
idx = tid + j1;
if (idx < nbodiesd) { // each thread copies 4 words (fully coalesced)
posxs[id] = posxd[idx]; posys[id] = posyd[idx];
poszs[id] = poszd[idx]; masss[id] = massd[idx];
}
__syncthreads(); // wait for all copying to be done
bound = min(nbodiesd - j1, THREADS);
for (j2 = 0; j2 < bound; j2++, j++) { // second part of loop
if (i != j) {
dx = posxs[j2] – px; dy = posys[j2] – py; dz = poszs[j2] - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssqd);
scale = masss[j2] * dinv * dinv * dinv;
ax += dx * scale; ay += dy * scale; az += dz * scale;
}
}
__syncthreads(); // wait for all force calculations to be done
}
CUDA Optimization Tutorial
45
Blocking Speedup
 Problem size
 Performance
 n=100000, step=1
 Great speedup for CC 1.3
 n=100000, step=1
 Some speedup for others
 n=300000, step=1
 Speedup
 CC 1.3: 3.7
 CC 2.0: 1.1
 CC 3.0: 1.6
CUDA Optimization Tutorial

Has hardware data cache
 Discussion
 Very important
optimization for memory
bound code
 Even with L1 cache
46
Loop Unrolling
 CUDA compiler
 Generally good at unrolling loops with fixed bounds
 Does not unroll inner loop of our example code
 Use pragma to unroll (and pad arrays)
#pragma unroll 8
for (j2 = 0; j2 < bound; j2++, j++) {
if (i != j) {
dx = posxs[j2] – px; dy = posys[j2] – py; dz = poszs[j2] - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssqd);
scale = masss[j2] * dinv * dinv * dinv;
ax += dx * scale; ay += dy * scale; az += dz * scale;
}
}
CUDA Optimization Tutorial
47
Loop Unrolling Speedup
 Problem size
 Performance
 n=100000, step=1
 Noticeable speedup
 n=100000, step=1
 All three GPUs
 n=300000, step=1
 Discussion
 Can be useful
 Speedup
 CC 1.3: 1.07
 CC 2.0: 1.16
 CC 3.0: 1.07
CUDA Optimization Tutorial
 May increase register
usage, which may lower
maximum number of
threads per block and
result in slowdown
48
CC 2.0 Absolute Performance
 Problem size
 n=100000, step=1
 Runtime
 612 ms
 FP operations
 326.7 GFlop/s
 Main mem throughput
 1.035 GB/s
 Not peak performance
 Only 32% of 1030 GFlop/s

Peak assumes FMA every cycle
 3 sub (1c), 3 fma (1c), 1 rsqrt
(8c), 3 mul (1c), 3 fma (1c) =
20c for 20 Flop
 63% of realistic peak of 515.2
GFlop/s

Assumes no non-FP operations
 With int ops = 31c for 20 Flop
 99% of actual peak of 330.45
GFlop/s
CUDA Optimization Tutorial
49
Eliminating the If Statement
 Algorithmic optimization
 Potential softening parameter avoids division by zero
 If-statement is not necessary and can be removed

Eliminates thread divergence
for (j2 = 0; j2 < bound; j2++, j++) {
if (i != j) {
dx = posxs[j2] – px; dy = posys[j2] – py; dz = poszs[j2] - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssqd);
scale = masss[j2] * dinv * dinv * dinv;
ax += dx * scale; ay += dy * scale; az += dz * scale;
}
}
CUDA Optimization Tutorial
50
If Elimination Speedup
 Problem size
 Performance
 n=100000, step=1
 Large speedup
 n=100000, step=1
 All three GPUs
 n=300000, step=1
 Discussion
 Speedup
 CC 1.3: 1.38
 CC 2.0: 1.54
 CC 3.0: 1.64
CUDA Optimization Tutorial
 No thread divergence
 Allows compiler to
schedule code much
better
51
Rearranging Terms
 Generated code is suboptimal
 Compiler does not emit as many fused multiply-add
(FMA) instructions as it could
 Rearrange terms in expressions to help compiler

Need to check generated assembly code
for (j2 = 0; j2 < bound; j2++, j++) {
dx = posxs[j2] – px; dy = posys[j2] – py; dz = poszs[j2] - pz;
dsq = dx*dx + dy*dy + dz*dz;
dinv = rsqrtf(dsq + epssqd);
dsq = dx*dx + (dy*dy + (dz*dz + epssqd));
dinv = rsqrtf(dsq);
scale = masss[j2] * dinv * dinv * dinv;
ax += dx * scale; ay += dy * scale; az += dz * scale;
}
CUDA Optimization Tutorial
52
FMA Speedup
 Problem size
 Performance
 n=100000, step=1
 Small speedup
 n=100000, step=1
 All three GPUs
 n=300000, step=1
 Discussion
 Speedup
 CC 1.3: 1.03
 CC 2.0: 1.05
 Seemingly needless
transformations can
make a difference
 CC 3.0: 1.06
CUDA Optimization Tutorial
53
Higher Unroll Factor
 Problem size
 Unroll 128 times
 n=100000, step=1
 Avoid looping overhead
 n=100000, step=1
 Now that there are no ifs
 n=300000, step=1
 Performance
 Speedup
 CC 1.3: 1.01
 CC 2.0: 1.04
 CC 3.0: 0.93
CUDA Optimization Tutorial
 Little speedup/slowdown
 Discussion
 Carefully choose unroll
factor (manually tune)
54
Compiler Flags
 Problem size
 -use_fast_math
 n=100000, step=1
 “-ftz=true” suffices
 n=100000, step=1
(flush denormals to zero)
 Makes SP FP operations
faster except on CC 1.3
 n=300000, step=1
 Speedup
 CC 1.3: 1.00
 CC 2.0: 1.18
 CC 3.0: 1.15
 Performance
 Significant speedup
 Discussion
 Use faster but less precise
operations when prudent
CUDA Optimization Tutorial
55
Final Absolute Performance
 CC 2.0 Fermi GTX 480
 Problem size

n=100000, step=1
 Runtime

296.1 ms
 FP operations



675.6 GFlop/s (SP)
66% of peak performance
261.1 GFlops/s (DP)
 Main mem throughput

2.139 GB/s
CUDA Optimization Tutorial
 CC 3.0 Kepler GTX 680
 Problem size

n=300000, step=1
 Runtime

1073 ms
 FP operations



1677.6 GFlop/s (SP)
54% of peak performance
88.7 GFlops/s (DP)
 Main mem throughput

5.266 GB/s
56
Outline
 Introduction
 GPU programming
 N-body example
 Porting and tuning
 Other considerations
 Conclusions
CUDA Optimization Tutorial
gamedsforum.ca
Thepcreport.net
57
Things to Consider
 Minimize PCIe transfers
 Implementing entire algorithm on GPU, even some
slow serial code sections, might be overall win
 Can stream data to/from GPU while computing
 Locks and synchronization
 Lightweight locks & fast barriers possible within SM
 Slow across different SMs
 L1 data caches are not coherent
 Use volatile & fences to avoid deadlocks
CUDA Optimization Tutorial
58
Warp-based Execution
// wrong on GPU, correct on CPU
do {
cnt = 0;
if (ready[i] != 0) cnt++;
if (ready[j] != 0) cnt++;
} while (cnt < 2);
ready[k] = 1;
// correct
do {
cnt = 0;
if (ready[i] != 0) cnt++;
if (ready[j] != 0) cnt++;
if (cnt == 2) ready[k] = 1;
} while (cnt < 2);
CUDA Optimization Tutorial
 Problem
 Thread divergence
 Loop exiting threads
wait for other threads in
warp to also exit
 “ready[k] = 1” is not
executed until all
threads in warp are
done with loop
 Possible deadlock
59
Hybrid Execution
 CPU always needed for program launch and I/O
 CPU much faster on serial program segments
 GPU 10 times faster than CPU on parallel code
 Running 10% of problem on CPU is hardly worthwhile
 Complicates programming and requires data transfer

Best CPU data structure is often not best for GPU
 PCIe bandwidth much lower than GPU bandwidth
 1.6 to 6.5 GB/s versus 192 GB/s
 But can send data while CPU & GPU are computing
 Merging CPU and GPU on same die (e.g., AMD’s
Fusion APU) makes finer grain switching possible
CUDA Optimization Tutorial
60
Outline
 Introduction
 GPU programming
 N-body example
 Porting and tuning
 Other considerations
 Conclusions
CUDA Optimization Tutorial
gamedsforum.ca
61
Summary and Conclusions (Part I)
 Step-by-step porting and tuning of CUDA code
 Example: n-body simulation
 GPUs have very powerful hardware
 But only exploitable with some codes
 Even harder to program and optimize than CPU hardware
CUDA Optimization Tutorial
62
Parallelizing and Optimizing Programs for
GPU Acceleration using CUDA (Part II)
Martin Burtscher
Department of Computer Science
Mapping Regular Code to GPUs
 Regular codes
 Operate on array- and matrix-based data structures
 Exhibit mostly strided memory access patterns
 Have relatively predictable control flow (control flow
behavior is mainly determined by input size)
 Largely independent computations
 Many regular codes are easy to port to GPUs
 E.g., matrix codes executing many ops/word
Dense matrix operations (level 2 and 3 BLAS)
 Stencil codes (PDE solvers)

CUDA Optimization Tutorial
LLNL
64
Mapping Irregular Code to GPUs
 Irregular codes
 Build, traverse, and update dynamic data structures
FSU
(trees, graphs, linked lists, priority queues, etc.)
 Exhibit pointer-chasing memory access patterns
 Have complex control flow (control flow behavior
depends on input values and changes dynamically)
 Many important scientific programs are irregular
 E.g., n-body simulation, data clustering, SAT solving,
social networks, discrete-event simulation, meshing, …
 Need case studies on how to best map irreg codes
CUDA Optimization Tutorial
65
Example: N-body Simulation
 Irregular Barnes Hut algorithm
 Repeatedly builds unbalanced tree and performs
complex traversals on it
 Our implementation
 Designed for GPUs (not just port of CPU code)
 First GPU implementation of entire BH algorithm
 Results
 GPU is 21 times faster than CPU (6 cores) on this code
CUDA Optimization Tutorial
66
Outline
 Introduction
 Barnes Hut algorithm
 CUDA implementation
 Experimental results
 Conclusions
CUDA Optimization Tutorial
NASA/JPL-Caltech/SSC
67
Barnes Hut Idea
 Precise force calculation
 Requires O(n2) operations (O(n2) body pairs)
 Computationally intractable for large n
 Barnes and Hut (1986)
 Algorithm to approximately compute forces

Bodies’ initial position & velocity are also approximate
 Requires only O(n log n) operations
 Idea is to “combine” far away bodies
 Error should be small because force  1/distance2
CUDA Optimization Tutorial
68
Barnes Hut Algorithm
 Set bodies’ initial position and velocity
 Iterate over time steps
1. Compute bounding box around bodies
2. Subdivide space until at most one body per cell

Record this spatial hierarchy in an octree
3. Compute mass and center of mass of each cell
4. Compute force on bodies by traversing octree

Stop traversal path when encountering a leaf (body) or an
internal node (cell) that is far enough away
5. Update each body’s position and velocity
CUDA Optimization Tutorial
69
Build Tree (Level 1)
*
o
*
*
*
*
*
* *
* *
*
*
*
*
*
*
*
*
*
*
*
*
*
Compute bounding box around all bodies → tree root
CUDA Optimization Tutorial
70
Build Tree (Level 2)
*
o
*
*
*
o
o
o
o
*
*
* *
* *
*
*
*
*
*
*
*
*
*
*
*
*
*
Subdivide space until at most one body per cell
CUDA Optimization Tutorial
71
Build Tree (Level 3)
*
o
*
*
*
o
o
o
o
*
* *
* *
*
o
o
o
o
o
o
o
o
o
o
o
o
*
*
*
*
*
*
*
*
*
*
*
*
*
Subdivide space until at most one body per cell
CUDA Optimization Tutorial
72
Build Tree (Level 4)
*
o
*
*
*
o
o
o
o
*
* *
* *
*
o
o
o
o
o
o o o o
o
o
o
o o o o
o
o
o
o
*
*
*
*
*
o o o o
*
*
*
*
*
*
*
*
Subdivide space until at most one body per cell
CUDA Optimization Tutorial
73
Build Tree (Level 5)
*
o
*
*
*
o
o
o
o
*
* *
* *
*
o
o
o
o
o
o o o o
o
o
o
o o o o
o
o
o
o
*
*
*
*
*
o o o o
*
*
oooo
*
*
*
*
*
*
Subdivide space until at most one body per cell
CUDA Optimization Tutorial
74
Compute Cells’ Center of Mass
*
o
*
*
*
*
o
o
o
o
* *
* *
*
*
o
o
o
o
o
o
o
o
o
o
o
*
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
For each internal cell, compute sum of mass and weighted average
of position of all bodies in subtree; example shows two cells only
CUDA Optimization Tutorial
75
Compute Forces
*
o
*
*
*
*
o
o
o
o
* *
* *
*
*
o
o
o
o
o
o
o
o
o
o
o
*
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
Compute force, for example, acting upon green body
CUDA Optimization Tutorial
76
Compute Force (short distance)
*
o
*
*
*
*
o
o
o
o
*
*
o
o
o
o
o
o
o
o
o
o
o
*
* *
* *
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
Scan tree depth first from left to right; green portion already completed
CUDA Optimization Tutorial
77
Compute Force (down one level)
*
o
*
*
*
*
o
o
o
o
*
* *
* *
*
*
o
o
o
o
o
o
o
o
o
o
o
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
Red center of mass is too close, need to go down one level
CUDA Optimization Tutorial
78
Compute Force (long distance)
*
o
*
*
*
*
o
o
o
o
*
*
o
o
o
o
o
o
o
o
o
o
o
*
* *
* *
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
Blue center of mass is far enough away
CUDA Optimization Tutorial
79
Compute Force (skip subtree)
*
o
*
*
*
*
o
o
o
o
*
* *
* *
*
*
o
o
o
o
o
o
o
o
o
o
o
*
*
o
o
*
*
*
o o o o
o o o o
o o o o
*
*
o
oooo
*
*
*
*
Therefore, entire subtree rooted in the blue cell can be skipped
CUDA Optimization Tutorial
80
Pseudocode
bodySet = ...
foreach timestep do {
bounding_box = new Bounding_Box();
foreach Body b in bodySet {
bounding_box.include(b);
}
octree = new Octree(bounding_box);
foreach Body b in bodySet {
octree.Insert(b);
}
cellList = octree.CellsByLevel();
foreach Cell c in cellList {
c.Summarize();
}
foreach Body b in bodySet {
b.ComputeForce(octree);
}
foreach Body b in bodySet {
b.Advance();
}
}
CUDA Optimization Tutorial
*
*
*
*
*
*
*
*
*
*
*
* *
* *
*
*
*
* *
* *
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
o
o
o
o
o
o
o
o
o
o
o
o
o o o o
o
o
o
o o o o
o
o
o o o o
oooo
*
*
*
*
*
*
*
*
*
*
*
* *
* *
*
*
*
*
*
*
* *
* *
*
*
*
o
*
o
*
*
*
*
*
*
*
o
*
*
*
*
o
*
*
*
*
*
*
81
Complexity and Parallelism
bodySet = ...
foreach timestep do {
bounding_box = new Bounding_Box();
foreach Body b in bodySet {
bounding_box.include(b);
}
octree = new Octree(bounding_box);
foreach Body b in bodySet {
octree.Insert(b);
}
cellList = octree.CellsByLevel();
foreach Cell c in cellList {
c.Summarize();
}
foreach Body b in bodySet {
b.ComputeForce(octree);
}
foreach Body b in bodySet {
b.Advance();
}
}
CUDA Optimization Tutorial
// O(n log n) + ordered sequential
// O(n) parallel reduction
// O(n log n) top-down tree building
// O(n) + ordered bottom-up traversal
// O(n log n) fully parallel
// O(n) fully parallel
82
Outline
 Introduction
 Barnes Hut algorithm
 CUDA implementation
 Experimental results
 Conclusions
CUDA Optimization Tutorial
83
Efficient GPU Code
 Large amounts of data parallelism
 Coalesced main memory accesses
 Little thread divergence
 Relatively little synchronization between blocks
 Little CPU/GPU data transfer
 Efficient use of shared memory
Thepcreport.net
CUDA Optimization Tutorial
84
Main BH Implementation Challenges
 Uses irregular tree-based data structure
 Initially little parallelism
 Little coalescing
o
o
 Load imbalance
 Complex recursive traversals
o
o
o
o
o
o
o
o
o
o
o o o o
o
o o o o
o
o
o
o o o o
oooo
 Recursion not well supported
 Lots of thread divergence
 Memory-bound pointer-chasing operations
 Not enough computation to hide latency
CUDA Optimization Tutorial
85
o
Six GPU Kernels
Read initial data and transfer to GPU
for each timestep do {
1.
2.
3.
4.
5.
6.
Compute bounding box around bodies (not irregular)
Build hierarchical decomposition, i.e., octree
Summarize body information in internal octree nodes
Approximately sort bodies by spatial location (optional)
Compute forces acting on each body with help of octree
Update body positions and velocities (not irregular)
}
Transfer result from GPU and output
CUDA Optimization Tutorial
86
Global Optimizations
 Make code iterative (recursion not supported*)
 Keep data on GPU between kernel calls
 Use array elements instead of heap nodes
 One aligned array per field for coalesced accesses
objects in array
objects on heap
CUDA Optimization Tutorial
fields in arrays
87
Global Optimizations (cont.)
 Maximize thread count (round down to warp size)
 Maximize resident block count (all SMs filled)
 Pass kernel parameters through constant memory
 Use special allocation order
c0
 Alias arrays (56 B/node)
 Use index arithmetic
 Persistent blocks & threads
 Unroll loops over children
CUDA Optimization Tutorial
c2
b5
ba
c4
c3
b3
b6
b2
c1
b7
b1
bodies (fixed)
b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba
b0
c5
b8
b4
b9
cell allocation direction
...
c5 c4 c3 c2 c1 c0
88
Kernel 1: Bounding Box (Regular)
main memory
...
 Optimizations
threads
warp 1
warp 2
warp 3
warp 4
 Fully coalesced
shared memory
barrier
threads
warp 1
warp 2
 No bank conflicts
shared memory
barrier
threads
warp 1
barrier
t1 t2
shared memory
barrier
threads
shared memory
 Minimal divergence
 Built-in min and max
shared memory
threads
 Fully cached
t1
Reduction operation
CUDA Optimization Tutorial
 2 red/mem, 6 red/bar
 Bodies load balanced
 512*3 threads per SM
89
Kernel 2: Build Octree (Irregular)
 Optimizations
 Only lock leaf “pointers”
Top-down tree building
 Lock-free fast path
 Light-weight lock release
 No re-traverse after lock




acquire failure
Combined memory fence
Re-compute position
during traversal
Separate init kernels
512*3 threads per SM
CUDA Optimization Tutorial
*
90
Kernel 2: Build Octree (cont.)
// initialize
cell = find_insertion_point(body); // no locks, cache cell
child = get_insertion_index(cell, body);
if (child != locked) { // skip atomic if already locked
if (child == null) { // fast path (frequent)
if (null == atomicCAS(&cell[child], null, body)) { // lock-free insertion
// move on to next body
}
} else {
if (child == atomicCAS(&cell[child], child, lock)) { // acquire lock
// build subtree with new and existing body
flag = true;
}
}
}
__syncthreads(); // optional barrier
__threadfence(); // make data visible
if (flag) {
cell[child] = new_subtree; // insert subtree and releases lock
// move on to next body
}
CUDA Optimization Tutorial
91
Kernel 3: Summarize Subtrees (Irreg.)
 Optimizations
Bottom-up tree traversal
 Scan avoids deadlock
 Use mass as flag + fence
4

3
1
2
3
No locks, no atomics
 Use wait-free first pass
4
 Cache the ready info
 Piggyback on traversal
allocation direction
...
3 4 1 2 3 4
scan direction
CUDA Optimization Tutorial

Count bodies in subtrees
 No parent “pointers”
 128*6 threads per SM
92
Kernel 4: Sort Bodies (Irregular)
 Optimizations
Top-down tree traversal
4
3
1
2
3
 (Similar to Kernel 3)
 Scan avoids deadlock
 Use data field as flag

4
 Use counts from Kernel 3
 Piggyback on traversal

allocation direction
...
3 4 1 2 3 4
scan direction
CUDA Optimization Tutorial
No locks, no atomics
Move nulls to back
 Throttle warps with
optional barrier
 64*6 threads per SM
93
Kernel 5: Force Calculation (Irregular)
 Optimizations
Multiple prefix traversals
 Group similar work together
 Uses sorting to minimize size of
prefix union in each warp
 Early out (nulls in back)
 Traverse whole union to avoid





CUDA Optimization Tutorial
divergence (warp voting)
Lane 0 controls iteration stack
for entire warp (fits in shmem)
Minimize volatile accesses
Use fast 1/sqrtf instruction
Cache tree-level-based data
256*5 threads per SM
94
Architectural Support
 Coalesced memory accesses & lockstep execution
 All threads in warp read same tree node at same time
 Only one mem access per warp instead of 32 accesses
 Warp-based execution
 Enables data sharing in warps w/o synchronization
 RSQRTF instruction
 Quickly computes good approximation of 1/sqrtf(x)
 Warp voting instructions
 Quickly perform reduction operations within a warp
CUDA Optimization Tutorial
95
Kernel 6: Advance Bodies (Regular)
 Optimizations
 Fully coalesced, no divergence
 Load balanced, 1024*1 threads per SM
Straightforward streaming
main memory
threads
...
warp 1
main memory
CUDA Optimization Tutorial
warp 2
warp 3
warp 4
warp 1
warp 2
...
warp 2
...
96
Outline
 Introduction
 Barnes Hut algorithm
 Experimental results
 Conclusions
10000.0
1000.0
runtime per timestep [s]
 CUDA implementation
CPUbh
GPUbh
100.0
GPUsq
10.0
1.0
10,000
100,000
1,000,000 10,000,000
0.1
0.0
number of bodies
CUDA Optimization Tutorial
97
Evaluation Methodology
 Implementations
 CUDA/GPU: Barnes Hut and O(n2) algorithms
 OpenMP/CPU: Barnes Hut algorithm (derived from CUDA)
 Pthreads/CPU: Barnes Hut algorithm (SPLASH-2 suite)
 Systems and compilers




nvcc 4.0 (-O3 -arch=sm_20 -ftz=true*)
GeForce GTX 480, 1.4 GHz, 15 SMs, 32 cores per SM
gcc 4.1.2 (-O3 -fopenmp* -ffast-math*)
Xeon X5690, 3.46 GHz, 6 cores, 2 threads per core
 Inputs and metric
 5k, 50k, 500k, and 5M star clusters (Plummer model)
 Best runtime of three experiments, excluding I/O
CUDA Optimization Tutorial
98
Nodes Touched per Activity (5M Input)
 Kernel “activities”
 K1: pair reduction
 K2: tree insertion
 K3: bottom-up step
 K4: top-down step
 K5: prefix traversal
 K6: integration step
 Max tree depth ≤ 22
 Cells have 3.1 children
CUDA Optimization Tutorial
kernel 1
kernel 2
kernel 3
kernel 4
kernel 5
kernel 6
neighborhood size
min
avg
max
1
2.0
2
2
13.2
22
2
4.1
9
2
4.1
9
818 4,117.0
6,315
1
1.0
1
 Prefix ≤ 6,315 nodes
(≤ 0.1% of 7.4 million)
 BH algorithm & sorting
to min. union work well
99
CUDA Optimization Tutorial
100
10
kernel and round
 Almost every “round” has lots of activities without
data dependencies that can be processed in parallel
100
k6.1
1,000
force calc.
integration
summarization
k5.1
100,000
tree building
k4.1
k4.2
k4.3
k4.4
k4.5
k4.6
k4.7
k4.8
k4.9
k4.10
k4.11
k4.12
k4.13
k4.14
k4.15
k4.16
k4.17
k4.18
k4.19
k4.20
bounding box
k3.1
k3.2
k3.3
k3.4
k3.5
k3.6
k3.7
k3.8
k3.9
k3.10
k3.11
k3.12
k3.13
k3.14
k3.15
k3.16
k3.17
k3.18
k3.19
k3.20
1,000,000
k2.1
k2.2
k2.3
k2.4
k2.5
k2.6
k2.7
k2.8
k2.9
k2.10
k2.11
k2.12
k2.13
k2.14
k2.15
k2.16
k2.17
k2.18
k2.19
k1.1
k1.2
k1.3
k1.4
k1.5
k1.6
k1.7
k1.8
k1.9
k1.10
k1.11
k1.12
k1.13
k1.14
k1.15
k1.16
k1.17
k1.18
k1.19
k1.20
k1.21
k1.22
k1.23
available parallelism
Available Amorphous Data Parallelism
10,000,000
5,000 bodies
sorting
50,000 bodies
500,000 bodies
5,000,000 bodies
10,000
1
Runtime Comparison
 GPU BH inefficiency
 5k input too small for
100,000
GPU CUDA
GPU O(n^2)
10,000
5,760 to 23,040 threads
algorithm
 O(n2) faster with fewer
than about 15k bodies
 GPU (5M input)
 21.1x faster than OpenMP
 23.2x faster than Pthreads
CPU Pthreads
runtime per timestep [ms]
 BH vs.
O(n2)
CPU OpenMP
1,000
100
10
1
5,000
50,000
500,000
5,000,000
0
number of bodies
CUDA Optimization Tutorial
101
Kernel Performance for 5M Input
 $200 GPU delivers 228 GFlops/s on irregular code
2
kernel 1 kernel 2 kernel 3 kernel 4 kernel 5 kernel 6 BarnesHut O(n )
Gflops/s
71.6
5.8
2.5
n/a
240.6
33.5
228.4
897.0
GB/s
142.9
26.8
10.6
12.8
8.0
133.9
8.8
2.8
runtime [ms]
0.4
44.6
28.0
14.2
1641.2
2.2
1730.6 557421.5
 GPU chip is 2.7 to 23.5 times faster than CPU chip
non-compliant fast single-precision version
IEEE 754-compliant double-precision version
kernel 1 kernel 2 kernel 3 kernel 4 kernel 5 kernel 6 kernel 1 kernel 2 kernel 3 kernel 4 kernel 5 kernel 6
X5690 CPU
5.5 185.7
75.8
52.1 38,540.3
16.4
10.3 193.1 101.0
51.6 47,706.4
33.1
GTX 480 GPU
0.4
44.6
28.0
14.2 1,641.2
2.2
0.8
46.7
31.0
14.2 5,177.1
4.2
CPU/GPU
13.1
4.2
2.7
3.7
23.5
7.3
12.7
4.1
3.3
3.6
9.2
7.9
 GPU hardware is better suited for BH than CPU hw
 But difficult and very time consuming to program
CUDA Optimization Tutorial
102
Kernel Speedups
 Optimizations that are generally applicable
avoid
volatile
50,000
1.14x
500,000
1.19x
5,000,000
1.18x
rsqrtf
instr.
1.43x
1.47x
1.46x
recalc.
data
0.99x
1.32x
1.69x
thread full multivoting threading
2.04x
20.80x
2.49x
27.99x
2.47x
28.85x
 Optimizations for irregular kernels
throttling waitfree combined sorting of sync'ed
barrier pre-pass mem fence bodies execution
50,000
0.97x
1.02x
1.54x
3.60x
6.23x
500,000
1.03x
1.21x
1.57x
6.28x
8.04x
5,000,000
1.04x
1.31x
1.50x
8.21x
8.60x
CUDA Optimization Tutorial
103
Outline
 Introduction
 Barnes Hut algorithm
 CUDA implementation
 Experimental results
 Conclusions
CUDA Optimization Tutorial
104
Optimization Summary
 Reduce main memory accesses
 Share data within warp, combine memory fences &
traversals, re-compute data, avoid volatile accesses
 Minimize thread divergence
 Group similar work together, force synchronicity
 Implement entire algorithm on and for GPU
 Avoid data transfers & data structure inefficiencies,
wait-free pre-pass, scan entire prefix union
CUDA Optimization Tutorial
105
Optimization Summary (cont.)
 Exploit hardware features
 Fast synchronization & thread startup, special instrs.,
coalesced memory accesses, even lockstep execution
 Use light-weight locking and synchronization
 Minimize locks, reuse fields, and use fence + store ops
 Maximize parallelism
 Parallelize every step within and across SMs
CUDA Optimization Tutorial
106
CPU/GPU Implementation Comparison
 Irregular CPU code
 Dynamically




(incrementally) allocated
shared data structures
Structure-based shared
data structures
Logical lock-based
implementation
Global/local worklists
Recursive or iterative
implementation
CUDA Optimization Tutorial
 Irregular GPU code
 Statically (wholly)




allocated shared data
structures
Multiple-array-based
shared data structures
Lock-free
implementation
(Implicit) local worklists
Iterative implementation
107
Useful GPU Hardware Features
 Wide parallelism
 Great for exploiting large
amounts of parallelism
 Massive multithreading
 Ideal for hiding latency of
irregular mem. accesses
 Fast thread startup
 Essential when launching
thousands of threads
 Shared memory
 Fast data sharing
 Useful for local worklists
CUDA Optimization Tutorial
 HW support for reduction
and synchronization
 Makes otherwise costly
operations very fast
 Coalesced accesses
 Memory access combining
is useful in irregular codes
 Lockstep execution
 Can share data without
explicit synchronization
 Allows to consolidate
iteration stacks
108
Challenges with GPUs
 Warp-based execution
 Often requires sorting of
work or algorithm change
 Data structure layout
 Best layout for CPU differs
from best layout for GPU
 SoA can be tedious to
code and deal with
(parameter passing)
 Separate memory space
 Slow transfers
 Pack/unpack data
CUDA Optimization Tutorial
 Incoherent L1 caches
 May need to explicitly
manage data (fences)
 Poor recursion support
 Need to make code
iterative and maintain
explicit iteration stacks
 Thread and block counts
 Hierarchy complicates
implementation
 Optimal counts have to
be (auto-)tuned
109
Running Irregular Algorithms on GPUs
 Mandatory
 Important
 Need vast amounts of data
 Scheduling is independent
parallelism
 Can do large chunks of
computation on GPU
of previous activities
 Easy to sort activities by
similarity (if needed)
 Very Important
 Beneficial
 Cautious implementation
 DS can be expressed
 Easy to express iteratively
 Has statically known range
through fixed arrays
 Uses local worklists that
can be statically populated
of neighborhoods
 DS size (or bound) can be
determined based on input
CUDA Optimization Tutorial
110
Conclusions
 Irregularity does not necessarily prevent high-
performance on GPUs
 Entire Barnes Hut algorithm implemented on GPU
 Builds and traverses unbalanced octree
 GPU is 21.1 times (float) and 9.1 times (double)
faster than high-end 6-core Xeon
 Code directly for GPU, do not merely adjust CPU code
 Requires different data and code structures
 Benefits from different algorithmic modifications
CUDA Optimization Tutorial
111
Acknowledgments
 Hardware
 NVIDIA Corp. and Intel Corp.
 Funding
 NVIDIA Corp. and Texas State University
 OpenMP code
 Ricardo Alves (Universidade do Minho, Portugal)
 Collaborator
 Keshav Pingali (University of Texas at Austin)
CUDA Optimization Tutorial
112
CUDA Optimization Tutorial
 Martin Burtscher
 burtscher@txstate.edu
 http://www.cs.txstate.edu/~burtscher/
 Barnes Hut CUDA code
 http://www.gpucomputing.net/?q=node/1314
 Tutorial slides
 http://www.cs.txstate.edu/~burtscher/tutorials/COT5/slides.pptx
CUDA Optimization Tutorial
113