Parallelism
Lecture notes from MKP and S. Yalamanchili
Introduction
• Goal: Higher performance through parallelism
• Job-level (process-level) parallelism
 High throughput for independent jobs
• Application-level parallelism
 Single program run on multiple processors
• Multicore microprocessors
 Chips with multiple processors (cores)
 Support for both job level and application-level
parallelism
(2)
Core Count Roadmap: AMD
From anandtech.com
(3)
Core Count: NVIDIA
1536 cores at 1GHz
• All cores are not created equal
• Need to understand the programming model
(4)
Hardware and Software
• Hardware
 Serial: e.g., Pentium 4
 Parallel: e.g., quad-core Xeon e5345
• Software
 Sequential: e.g., matrix multiplication
 Concurrent: e.g., operating system
• Sequential/concurrent software can run on
serial/parallel hardware
 Challenge: making effective use of parallel hardware
(5)
Parallel Programming
• Parallel software is the problem
• Need to get significant performance
improvement
 Otherwise, just use a faster uniprocessor, since it’s
easier!
• Difficulties
 Partitioning
 Coordination
 Communications overhead
(6)
Amdahl’s Law
• Sequential part can limit speedup
• Example: 100 processors, 90× speedup?
 Tnew = Tparallelizable/100 + Tsequential

1
Speedup 
 90
(1 Fparalleliz able )  Fparalleliz able /100
 Solving: Fparallelizable = 0.999
• Need sequential part to be 0.1% of original
time
(7)
Scaling Example
• Workload: sum of 10 scalars, and 10 × 10
matrix sum
 Speed up from 10 to 100 processors
• Single processor: Time = (10 + 100) × tadd
• 10 processors
 Time = 10 × tadd + 100/10 × tadd = 20 × tadd
 Speedup = 110/20 = 5.5 (55% of potential)
• 100 processors
 Time = 10 × tadd + 100/100 × tadd = 11 × tadd
 Speedup = 110/11 = 10 (10% of potential)
• Idealized model
 Assumes load can be balanced across processors
(8)
Scaling Example (cont)
• What if matrix size is 100 × 100?
• Single processor: Time = (10 + 10000) × tadd
• 10 processors
 Time = 10 × tadd + 10000/10 × tadd = 1010 × tadd
 Speedup = 10010/1010 = 9.9 (99% of potential)
• 100 processors
 Time = 10 × tadd + 10000/100 × tadd = 110 × tadd
 Speedup = 10010/110 = 91 (91% of potential)
• Idealized model
 Assuming load balanced
(9)
Strong vs Weak Scaling
• Strong scaling: problem size fixed
 As in example
• Weak scaling: problem size proportional to
number of processors
 10 processors, 10 × 10 matrix
o
Time = 20 × tadd
 100 processors, 32 × 32 matrix
o
Time = 10 × tadd + 1000/100 × tadd = 20 × tadd
 Constant performance in this example
 For a fixed size system grow the number of
processors to improve performance
(10)
What We Have Seen
• §3.6: Parallelism and Computer Arithmetic
 Associativity and bit level parallelism
• §4.10: Parallelism and Advanced InstructionLevel Parallelism
 Recall multi-instruction issue
• §6.9: Parallelism and I/O:
 Redundant Arrays of Inexpensive Disks
• Now we will look at categories in computation
 classification
(11)
Concurrency and Parallelism
• Each core can operate concurrently
and in parallel
• Multiple threads may operate in a
time sliced fashion on a single core
• Concurrent access to shared data must be
controlled for correctness
• Programming models?
Image from futurelooks.com
(12)
Instruction Level Parallelism (ILP)
Multiple instructions in
EX at the same time
IF
ID
MEM WB
• Single (program) thread of execution
• Issue multiple instructions from the same
instruction stream
• Average CPI<1
• Often called out of order (OOO) cores
(13)
The P4 Microarchitecture
From, “The Microarchitecture of the Pentium 4 Processor 1,” G. Hinton et.al, Intel Technology Journal Q1, 2001
(14)
ILP Wall - Past the Knee of the Curve?
Performance
Made sense to go
Superscalar/OOO:
good ROI
Very little gain for
substantial effort
Scalar
In-Order
Moderate-Pipe
Superscalar/OOO
Very-Deep-Pipe
Aggressive
Superscalar/OOO
“Effort”
Source: G. Loh
(15)
Thread Level Parallelism (TLP)
• Multiple threads of execution
• Exploit ILP in each thread
• Exploit concurrent execution across threads
(16)
Instruction and Data Streams
• Taxonomy due to M. Flynn
Data Streams
Single
Instruction Single
Streams
Multiple

Multiple
SISD:
Intel Pentium 4
SIMD: SSE
instructions of x86
MISD:
No examples today
MIMD:
Intel Xeon e5345
SPMD: Single Program Multiple Data


A parallel program on a MIMD computer where each
instruction stream is identical
Conditional code for different processors
(17)
Programming Model: Multithreading
• Performing multiple threads of execution in
parallel
 Replicate registers, PC, etc.
 Fast switching between threads
• Fine-grain multithreading
 Switch threads after each cycle
 Interleave instruction execution
 If one thread stalls, others are executed
• Coarse-grain multithreading
 Only switch on long stall (e.g., L2-cache miss)
 Simplifies hardware, but doesn’t hide short stalls
(eg, data hazards)
(18)
Conventional Multithreading
• Zero-overhead context switch
• Duplicated contexts for threads
0:r0
0:r7
1:r0
CtxtPtr
Memory
(shared by
threads)
1:r7
2:r0
2:r7
3:r0
3:r7
Register file
(19)
Simultaneous Multithreading
• In multiple-issue dynamically scheduled
processor
 Schedule instructions from multiple threads
 Instructions from independent threads execute when
function units are available
 Within threads, dependencies handled by scheduling
and register renaming
• Example: Intel Pentium-4 HT
 Two threads: duplicated registers, shared function
units and caches
 Known as Hyperthreading in Intel terminology
(20)
Hyper-threading
2 CPU Without Hyper-threading
Arch State
Arch State
Processor
Execution
Resources
Processor
Execution
Resources
2 CPU With Hyper-threading
Arch State
Arch State
Processor
Execution
Resources
Arch State
Arch State
Processor
Execution
Resources
• Implementation of Hyper-threading adds less
that 5% to the chip area
• Principle: share major logic components by
adding or partitioning buffering logic
21
(21)
Multithreading Example
(22)
Shared Memory
• SMP: shared memory multiprocessor
 Hardware provides single physical
address space for all processors
 Synchronize shared variables using locks
 Memory access time
o UMA (uniform) vs. NUMA (nonuniform)
(23)
Example: Communicating Threads
Producer
Consumer
The Producer calls
while (1) {
while (count == BUFFER_SIZE)
; // do nothing
// add an item to the buffer
++count;
buffer[in] = item;
in = (in + 1) % BUFFER_SIZE;
}
(24)
Example: Communicating Threads
Producer
Consumer
The Consumer calls
while (1) {
while (count == 0)
; // do nothing
// remove an item from the buffer
--count;
item = buffer[out];
out = (out + 1) % BUFFER_SIZE;
}
(25)
Uniprocessor Implementation
•
count++ could be implemented as
register1 = count;
register1 = register1 + 1;
count = register1;
•
count-- could be implemented as
register2 = count;
register2 = register2 – 1;
count = register2;
•
Consider this execution interleaving:
S0:
S1:
S2:
S3:
S4:
S5:
producer execute
producer execute
consumer execute
consumer execute
producer execute
consumer execute
register1 = count
register1 = register1 + 1
register2 = count
register2 = register2 - 1
count = register1
count = register2
{register1 = 5}
{register1 = 6}
{register2 = 5}
{register2 = 4}
{count = 6 }
{count = 4}
(26)
Synchronization
• We need to prevent certain instruction
interleavings
 Or at least be able to detect violations!
• Some sequence of operations (instructions)
must happen atomically
 E.g.,
register1 = count;
register1 = register1 + 1;
count = register1;
 atomic operations/instructions
(27)
Synchronization
• Two processors sharing an area of memory
 P1 writes, then P2 reads
 Data race if P1 and P2 don’t synchronize
o Result depends of order of accesses
• Hardware support required
 Atomic read/write memory operation
 No other access to the location allowed between the
read and write
• Could be a single instruction
 E.g., atomic swap of register ↔ memory
 Or an atomic pair of instructions
(28)
Synchronization in MIPS
• Load linked: ll rt, offset(rs)
• Store conditional: sc rt, offset(rs)
 Succeeds if location not changed since the ll
o Returns 1 in rt
 Fails if location is changed
o Returns 0 in rt
• Example: atomic swap (to test/set lock
variable)
try: add
ll
sc
beq
add
$t0,$zero,$s4
$t1,0($s1)
$t0,0($s1)
$t0,$zero,try
$s4,$zero,$t1
;copy exchange value
;load linked
;store conditional
;branch store fails
;put load value in $s4
(29)
Cache Coherence
• A shared variable may exist in multiple caches
• Multiple copies to improve latency
• This is a really a synchronization problem
(30)
Cache Coherence Problem
• Suppose two CPU cores share a physical
address space
 Write-through caches
Time Event
step
CPU A’s
cache
CPU B’s
cache
0
Memory
0
1
CPU A reads X
0
0
2
CPU B reads X
0
0
0
3
CPU A writes 1 to X
1
0
1
(31)
Example (Writeback Cache)
P
P
Rd?
Cache
Rd?
Cache
X= -100
Memory
Courtesy H. H. Lee
P
Cache
X= -100
505
X=
X= -100
X= -100
(32)
Coherence Defined
• Informally: Reads return most recently written
value
• Formally:
 P writes X; P reads X (no intervening writes)
 read returns written value
 P1 writes X; P2 reads X (sufficiently later)
 read returns written value
o
c.f. CPU B reading X after step 3 in example
 P1 writes X, P2 writes X
 all processors see writes in the same order
o
End up with the same final value for X
(33)
Cache Coherence Protocols
• Operations performed by caches in
multiprocessors to ensure coherence
 Migration of data to local caches
o
Reduces bandwidth for shared memory
 Replication of read-shared data
o
Reduces contention for access
• Snooping protocols
 Each cache monitors bus reads/writes
• Directory-based protocols
 Caches and memory record sharing status of blocks
in a directory
(34)
Invalidating Snooping Protocols
• Cache gets exclusive access to a block when it is
to be written
 Broadcasts an invalidate message on the bus
 Subsequent read in another cache misses
o
Owning cache supplies updated value
CPU activity
Bus activity
CPU A’s
cache
CPU B’s
cache
Memory
0
CPU A reads X
Cache miss for X
0
0
CPU B reads X
Cache miss for X
0
0
0
CPU A writes 1 to X
Invalidate for X
1
invalid
0
CPU B read X
Cache miss for X
1
1
1
(35)
Programming Model: Message Passing
• Each processor has private physical address
space
• Hardware sends/receives messages between
processors
(36)
Parallelism
• Write message-passing programs
• Explicit send and receive of data
 Rather than accessing data in shared memory
Process 2
Process 2
send()
receive()
receive()
send()
(37)
Loosely Coupled Clusters
• Network of independent computers
 Each has private memory and OS
 Connected using I/O system
o E.g., Ethernet/switch, Internet
• Suitable for applications with independent
tasks
 Web servers, databases, simulations, …
• High availability, scalable, affordable
• Problems
 Administration cost (prefer virtual machines)
 Low interconnect bandwidth
o c.f. processor/memory bandwidth on an SMP
SMP – Shared Memory Processor
(38)
High Performance Computing
theregister.co.uk
zdnet.com
• The dominant programming model is message
passing
• Scales well but requires programmer effort
• Science problems have fit this model well to
date (e.g., Weather Prediction)
(39)
Grid Computing
• Separate computers interconnected by longhaul networks (2 dimensional grid)
 E.g., Internet connections
 Work units farmed out, results sent back
• Can make use of idle time on PCs
 E.g., SETI@home, World Community Grid
(40)
Programming Model: SIMD
• Operate element wise on vectors of data
 E.g., MMX and SSE instructions in x86
o
Multiple data elements in 128-bit wide registers
• All processors execute the same instruction at
the same time
 Each with different data address, etc.
• Simplifies synchronization
• Reduced instruction control hardware
• Works best for highly data-parallel applications
• Data Level Parallelism
Single Instruction – Multiple Data
(41)
SIMD Co-Processor
• Graphics and media processing operates on
vectors of 8-bit and 16-bit data
 Use 64-bit adder, with partitioned carry chain
o
Operate on 8×8-bit, 4×16-bit, or 2×32-bit vectors
 SIMD (single-instruction, multiple-data)
4x16-bit
2x32-bit
(42)
History of GPUs
• Early video cards
 Frame buffer memory with address generation for
video output
• 3D graphics processing
 Originally high-end computers (e.g., SGI)
 Moore’s Law  lower cost, higher density
 3D graphics cards now for PCs and game consoles
• Graphics Processing Units
 Processors oriented to 3D graphics tasks
 Vertex/pixel processing, shading, texture mapping,
rasterization
(43)
Graphics in the System
(44)
GPU Architectures
• Processing is highly data-parallel
 GPUs are highly multithreaded
 Use thread switching to hide memory latency
o Less reliance on multi-level caches
 Graphics memory is wide and high-bandwidth
• Trend toward general purpose GPUs
 Heterogeneous CPU/GPU systems
 CPU for sequential code, GPU for parallel code
• Programming languages/APIs
 DirectX, OpenGL
 C for Graphics (Cg), High Level Shader Language
(HLSL)
 Compute Unified Device Architecture (CUDA)
(45)
Example: NVIDIA Tesla
Streaming
multiprocessor
8 × Streaming
processors
(46)
Compute Unified Device Architecture
Bulk synchronous
processing (BSP)
execution model
 For
access to CUDA tutorials
http://developer.nvidia.com/cuda-education-training
(47)
47
Example: NVIDIA Tesla
• Streaming Processors
 Single-precision FP and integer units
 Each SP is fine-grained multithreaded
• Warp: group of 32 threads
 Executed in parallel,
SIMD style
o
8 SPs
× 4 clock cycles
 Hardware contexts
for 24 warps
o
Registers, PCs, …
(48)
Classifying GPUs
• Does not fit nicely into SIMD/MIMD model
 Conditional execution in a thread allows an illusion of
MIMD
o
o
But with performance degradation
Need to write general purpose code with care
Instruction-Level
Parallelism
Data-Level
Parallelism
Static: Discovered
at Compile Time
Dynamic: Discovered
at Runtime
VLIW
Superscalar
SIMD or Vector
Tesla Multiprocessor
Really Single Instruction Multiple Thread (SIMT)
(49)
Vector Processors
• Highly pipelined function units
• Stream data from/to vector registers to units
 Data collected from memory into registers
 Results stored from registers to memory
• Example: Vector extension to MIPS
 32 × 64-element registers (64-bit elements)
 Vector instructions
o lv, sv: load/store vector
o addv.d: add vectors of double
o addvs.d: add scalar to each element of vector of
double
• Significantly reduces instruction-fetch bandwidth
(50)
Cray-1: Vector Machine
• Mid 70’s – principles
have not changed
• Aggregate operations
defined on vectors
• Vector ISAs operating
on vector instruction
sets
• Load/store ISA ala MIPS
• Often confused with
SIMD
From pg-server.csc.ncsu.edu
(51)
Vector vs. Scalar
• Vector architectures and compilers
 Simplify data-parallel programming
 Explicit statement of the absence of loop-carried
dependences
o
Reduced checking in hardware
 Regular access patterns benefit from interleaved and
burst memory
 Avoid control hazards by avoiding loops
• More general than ad-hoc media extensions
(such as MMX, SSE)
 Better match with compiler technology
(52)
Interconnection Networks
• Network topologies
 Arrangements of processors, switches, and
links
Bus
Ring
N-cube (N = 3)
2D Mesh
Fully connected
(53)
Network Characteristics
• Performance
 Latency per message (unloaded network)
 Throughput
o
o
o
Link bandwidth
Total network bandwidth
Bisection bandwidth
 Congestion delays (depending on traffic)
• Cost
• Power
• Routability in silicon
(54)
Modeling Performance
• Assume performance metric of interest is
achievable GFLOPs/sec
 Measured using computational kernels from Berkeley
Design Patterns
• Arithmetic intensity of a kernel
 FLOPs per byte of memory accessed
• For a given computer, determine
 Peak GFLOPS (from data sheet)
 Peak memory bytes/sec (using Stream benchmark)
(55)
Roofline Diagram
Attainable GPLOPs/sec
= Max ( Peak Memory BW × Arithmetic Intensity, Peak FP Performance )
(56)
Comparing Systems
• Example: Opteron X2 vs. Opteron X4
 2-core vs. 4-core, 2× FP performance/core, 2.2GHz vs.
2.3GHz
 Same memory system

To get higher
performance on X4 than
X2


Need high arithmetic
intensity
Or working set must fit in
X4’s 2MB L-3 cache
(57)
Optimizing Performance
• Optimize FP performance
 Balance adds & multiplies
 Improve superscalar ILP and
use of SIMD instructions
• Optimize memory usage
 Software prefetch
o
Avoid load stalls
 Memory affinity
o
Avoid non-local data accesses
(58)
Optimizing Performance
• Choice of optimization depends on arithmetic
intensity of code

Arithmetic intensity is not
always fixed


May scale with problem size
Caching reduces memory
accesses
 Increases arithmetic
intensity
(59)
Four Example Systems
2 × quad-core
Intel Xeon e5345
(Clovertown)
2 × quad-core
AMD Opteron X4 2356
(Barcelona)
(60)
Four Example Systems
2 × oct-core
Sun UltraSPARC
T2 5140 (Niagara 2)
2 × oct-core
IBM Cell QS20
(61)
And Their Rooflines
•Kernels
SpMV (left)
LBHMD (right)
•Some optimizations
change arithmetic
intensity
•x86 systems have
higher peak GFLOPs
But harder to achieve,
given memory bandwidth
(62)
Pitfalls
• Not developing the software to take account of
a multiprocessor architecture
 Example: using a single lock for a shared composite
resource
o
o
Serializes accesses, even if they could be done in
parallel
Use finer-granularity locking
(63)
Concluding Remarks
• Goal: higher performance by using multiple
processors
• Difficulties
 Developing parallel software
 Devising appropriate architectures
• Many reasons for optimism
 Changing software and application environment
 Chip-level multiprocessors with lower latency, higher
bandwidth interconnect
• An ongoing challenge for computer architects!
(64)
Study Guide
• Be able to explain the following concepts ILP,
MT, SMT, TLP, DLP, MIMD, SIMD, SISD*
• Explain the roofline model of performance
• Use of Amdahl’s Law in demonstrating the
limits of scaling
• What is the impact of a sequence of read/write
operations on shared data?
 Cache coherence
• How does ILP differ from SMT
• How does SIMD differ from vector?
• What is the difference between weak vs. strong
scaling?
*Instruction Level Parallelism, Multi-Scaler Techn., Shared Memory Techn.,
Thread LP, Data LP, Multiple Instruction-Multiple Data, Single-Instruct Single Data
(65)