Threading Hardware in G80
1
Sources
• Slides by ECE 498 AL : Programming Massively
Parallel Processors : Wen-Mei Hwu
• John Nickolls, NVIDIA
2
Single-Program Multiple-Data (SPMD)
• CUDA integrated CPU + GPU application C program
– Serial C code executes on CPU
– Parallel Kernel C code executes on GPU thread blocks
CPU Serial Code
Grid 0
GPU Parallel Kernel
KernelA<<< nBlk, nTid >>>(args);
...
CPU Serial Code
Grid 1
GPU Parallel Kernel
KernelB<<< nBlk, nTid >>>(args);
...
3
Grids and Blocks
•
A kernel is executed as a grid
of thread blocks
–
•
All threads share global memory
space
Host
Grid 1
Kernel
1
A thread block is a batch of
threads that can cooperate with
each other by:
–
–
–
Synchronizing their execution
using barrier
Efficiently sharing data through
a low latency shared memory
Two threads from two different
blocks cannot cooperate
Device
Block
(0, 0)
Block
(1, 0)
Block
(0, 1)
Block
(1, 1)
Grid 2
Kernel
2
Block (1, 1)
(0,0,1) (1,0,1) (2,0,1) (3,0,1)
Thread Thread Thread Thread
(0,0,0) (1,0,0) (2,0,0) (3,0,0)
Thread Thread Thread Thread
(0,1,0) (1,1,0) (2,1,0) (3,1,0)
Courtesy: NDVIA
4
Figure 3.2. An Example of CUDA Thread Org
CUDA Thread Block: Review
•
Programmer declares (Thread) Block:
–
–
–
•
•
•
•
Block size 1 to 512 concurrent threads
Block shape 1D, 2D, or 3D
Block dimensions in threads
CUDA Thread Block
All threads in a Block execute the same
thread program
Threads share data and synchronize while
doing their share of the work
Threads have thread id numbers within
Block
Thread program uses thread id to select
work and address shared data
Thread Id #:
0123…
m
Thread program
Courtesy: John Nickolls, NVIDIA
5
GeForce-8 Series HW Overview
Streaming Processor Array
TPC
TPC
TPC
Texture Processor Cluster
…
TPC
TPC
Streaming Multiprocessor
Instruction L1
SM
TPC
Data L1
Instruction Fetch/Dispatch
Shared Memory
TEX
SP
SM
SP
SP
SP
SFU
SFU
SP
SP
SP
SP
6
CUDA Processor Terminology
•
SPA
–
•
TPC
–
•
Texture Processor Cluster (2 SM + TEX)
SM
–
–
–
•
Streaming Processor Array (variable across GeForce 8-series, 8 in
GeForce8800)
Streaming Multiprocessor (8 SP)
Multi-threaded processor core
Fundamental processing unit for CUDA thread block
SP
–
–
Streaming Processor
Scalar ALU for a single CUDA thread
7
Streaming Multiprocessor (SM)
•
Streaming Multiprocessor (SM)
–
–
•
Multi-threaded instruction dispatch
–
–
–
•
•
•
8 Streaming Processors (SP)
2 Super Function Units (SFU)
1 to 512 threads active
Shared instruction fetch per 32 threads
Cover latency of texture/memory loads
20+ GFLOPS
16 KB shared memory
texture and global memory access
Streaming Multiprocessor
Instruction L1
Data L1
Instruction Fetch/Dispatch
Shared Memory
SP
SP
SP
SP
SFU
SFU
SP
SP
SP
SP
8
G80 Thread Computing Pipeline
Processors
execute
threadsprocessing
• The
future of
GPUscomputing
is programmable
Alternative
operating
modearound
specifically
for computing
• So
– build the
architecture
the processor
Generates Thread
grids based on
kernel calls
Input
Input Assembler
Assembler
Thread Execution Manager
Vtx Thread Issue
SP
SP
SP
SP
SP
SP
Setup / Rstr / ZCull
Geom Thread Issue
SP
SP
SP
SP
Pixel Thread Issue
SP
SP
SP
SP
SP
SP
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Parallel
Data
TF
Cache
Texture
Texture
L1
Texture
L1
Texture
L1
Texture
L1
Texture
L1
Texture
L1
Texture
L1
Texture
L1
Load/store
L2
Load/storeL2
FB
FB
Load/store
L2
Load/store
L2
FB Global Memory
FB
Thread Processor
Host
Host
Load/store
L2
Load/store
L2
FB
FB
9
Thread Life Cycle in HW
•
•
Grid is launched on the SPA
Thread Blocks are serially
distributed to all the SM’s
–
•
•
Device
Grid 1
Kernel
1
Potentially >1 Thread Block per
SM
Each SM launches Warps of
Threads
–
•
Host
2 levels of parallelism
SM schedules and executes
Warps that are ready to run
As Warps and Thread Blocks
complete, resources are freed
–
SPA can distribute more Thread
Blocks
Block
(0, 0)
Block
(1, 0)
Block
(2, 0)
Block
(0, 1)
Block
(1, 1)
Block
(2, 1)
Grid 2
Kernel
2
Block (1, 1)
Thread Thread Thread Thread Thread
(0, 0)
(1, 0)
(2, 0)
(3, 0)
(4, 0)
Thread Thread Thread Thread Thread
(0, 1)
(1, 1)
(2, 1)
(3, 1)
(4, 1)
Thread Thread Thread Thread Thread
(0, 2)
(1, 2)
(2, 2)
(3, 2)
(4, 2)
10
SM Executes Blocks
t0 t1 t2 … tm
SM 0 SM 1
MT IU
SP
t0 t1 t2 … tm
MT IU
Blocks
SP
•
Blocks
Threads are assigned to SMs in
Block granularity
–
Shared
Memory
Shared
Memory
–
Up to 8 Blocks to each SM as
resource allows
SM in G80 can take up to 768 threads
•
TF
Texture L1
•
L2
Memory
•
Could be 256 (threads/block) * 3
blocks
Or 128 (threads/block) * 6 blocks,
etc.
Threads run concurrently
–
–
SM assigns/maintains thread id #s
11
SM manages/schedules thread
execution
Thread Scheduling/Execution
•
Each Thread Blocks is divided in 32thread Warps
–
•
•
This is an implementation decision, not
part of the CUDA programming model
Warps are scheduling units in SM
If 3 blocks are assigned to an SM and each
Block has 256 threads, how many Warps
are there in an SM?
–
–
–
Each Block is divided into 256/32 = 8
Warps
There are 8 * 3 = 24 Warps
At any point in time, only one of the 24
Warps will be selected for instruction
fetch and execution.
Block 1 Warps
…
t0 t1 t2 … t31
…
Block 2 Warps
…
t0 t1 t2 … t31
…
Streaming Multiprocessor
Instruction L1
Data L1
Instruction Fetch/Dispatch
Shared Memory
SP
SP
SP
SP
SFU
SFU
SP
SP
SP
SP
12
SM Warp Scheduling
•
SM hardware implements zerooverhead Warp scheduling
–
SM multithreaded
Warp scheduler
–
time
–
warp 8 instruction 11
warp 1 instruction 42
warp 3 instruction 95
..
.
warp 8 instruction 12
warp 3 instruction 96
•
Warps whose next instruction has its
operands ready for consumption are
eligible for execution
Eligible Warps are selected for execution
on a prioritized scheduling policy
All threads in a Warp execute the same
instruction when selected
4 clock cycles needed to dispatch the
same instruction for all threads in a
Warp in G80
–
–
If one global memory access is needed
for every 4 instructions
A minimal of 13 Warps are needed to
fully tolerate 200-cycle memory latency13
SM Instruction Buffer – Warp Scheduling
•
Fetch one warp instruction/cycle
–
–
•
Issue one “ready-to-go” warp instruction/cycle
–
–
•
•
from instruction L1 cache
into any instruction buffer slot
from any warp - instruction buffer slot
operand scoreboarding used to prevent hazards
Issue selection based on round-robin/age of
warp
SM broadcasts the same instruction to 32
Threads of a Warp
I$
L1
Multithreaded
Instruction Buffer
R
F
C$
L1
Shared
Mem
Operand Select
MAD
SFU
14
Scoreboarding
•
All register operands of all instructions in the Instruction
Buffer are scoreboarded
–
–
–
•
Instruction becomes ready after the needed values are deposited
prevents hazards
cleared instructions are eligible for issue
Decoupled Memory/Processor pipelines
–
–
any thread can continue to issue instructions until scoreboarding
prevents issue
allows Memory/Processor ops to proceed in shadow of other waiting
Memory/Processor ops
TB1, W1 stall
TB2, W1 stall
Instruction:
TB1
W1
1
2
Time
3
4
TB2
W1
5
6
1
2
TB3
W1
1
2
TB3, W2 stall
TB3
W2
1
2
TB2
W1
3
4
TB1
W1
7
8
TB1
W2
1
2
TB1
W3
1
2
TB3
W2
3
4
TB = Thread Block, W = Warp
15
Granularity Considerations
•
For Matrix Multiplication, should I use 4X4, 8X8, 16X16 or 32X32 tiles?
– For 4X4, we have 16 threads per block, Since each SM can take up to 768 threads, the
thread capacity allows 48 blocks. However, each SM can only take up to 8 blocks, thus
there will be only 128 threads in each SM!
• There are 8 warps but each warp is only half full.
– For 8X8, we have 64 threads per Block. Since each SM can take up to 768 threads, it
could take up to 12 Blocks. However, each SM can only take up to 8 Blocks, only 512
threads will go into each SM!
• There are 16 warps available for scheduling in each SM
• Each warp spans four slices in the y dimension
– For 16X16, we have 256 threads per Block. Since each SM can take up to 768 threads, it
can take up to 3 Blocks and achieve full capacity unless other resource considerations
overrule.
• There are 24 warps available for scheduling in each SM
• Each warp spans two slices in the y dimension
– For 32X32, we have 1024 threads per Block. Not even one can fit into an SM!
16
Memory Hardware in G80
17
CUDA Device Memory Space: Review
• Each thread can:
–
–
–
–
–
–
(Device) Grid
R/W per-thread registers
R/W per-thread local memory
R/W per-block shared memory
R/W per-grid global memory
Read only per-grid constant
memory
Read only per-grid texture memory
• The host can R/W
global, constant, and
texture memories
Host
Block (0, 0)
Block (1, 0)
Shared Memory
Registers
Registers
Shared Memory
Registers
Registers
Thread (0, 0) Thread (1, 0)
Thread (0, 0) Thread (1, 0)
Local
Memory
Local
Memory
Local
Memory
Local
Memory
Global
Memory
Constant
Memory
Texture
Memory
18
Parallel Memory Sharing
•
Thread
Local Memory:
–
–
Local Memory
•
Block
–
•
Shared by threads of the same
block
Inter-thread communication
Global Memory: per-application
–
–
Grid 0
Private per thread
Auto variables, register spill
Shared Memory: per-Block
–
Shared
Memory
per-thread
Shared by all threads
Inter-Grid communication
...
Global
Memory
Grid 1
Sequential
Grids
in Time
...
19
SM Memory Architecture
t0 t1 t2 … tm
SM 0 SM 1
MT IU
SP
t0 t1 t2 … tm
MT IU
Blocks
SP
Blocks
•
Shared
Memory
–
–
Shared
Memory
•
TF
Texture L1
Courtesy:
John Nicols, NVIDIA
L2
Memory
Threads in a block share data &
results
In Memory and Shared Memory
Synchronize at barrier instruction
Per-Block Shared Memory
Allocation
–
–
–
Keeps data close to processor
Minimize trips to global Memory
Shared Memory is dynamically
allocated to blocks, one of the
20
limiting resources
SM Register File
•
Register File (RF)
–
•
•
32 KB (8K entries) for each SM in G80
TEX pipe can also read/write RF
–
I$
L1
2 SMs share 1 TEX
Load/Store pipe can also read/write RF
Multithreaded
Instruction Buffer
R
F
C$
L1
Shared
Mem
Operand Select
MAD
SFU
21
Programmer View of Register File
• There are 8192 registers in
each SM in G80
4 blocks
3 blocks
– This is an implementation
decision, not part of CUDA
– Registers are dynamically
partitioned across all blocks
assigned to the SM
– Once assigned to a block, the
register is NOT accessible by
threads in other blocks
– Each thread in the same block
only access registers assigned
to itself
22
Matrix Multiplication Example
• If each Block has 16X16 threads and each thread uses
10 registers, how many thread can run on each SM?
– Each block requires 10*256 = 2560 registers
– 8192 = 3 * 2560 + change
– So, three blocks can run on an SM as far as registers are
concerned
• How about if each thread increases the use of registers
by 1?
– Each Block now requires 11*256 = 2816 registers
– 8192 < 2816 *3
– Only two Blocks can run on an SM, 1/3 reduction of
parallelism!!!
23
More on Dynamic Partitioning
• Dynamic partitioning gives more flexibility to
compilers/programmers
– One can run a smaller number of threads that require many
registers each or a large number of threads that require few
registers each
• This allows for finer grain threading than traditional CPU threading
models.
– The compiler can tradeoff between instruction-level
parallelism and thread level parallelism
24