CS 152 Computer Architecture and
Engineering
Lecture 11 - Out-of-Order Issue,
Register Renaming,
& Branch Prediction
Krste Asanovic
Electrical Engineering and Computer Sciences
University of California at Berkeley
http://www.eecs.berkeley.edu/~krste
http://inst.eecs.berkeley.edu/~cs152
March 2, 2011
CS152, Spring 2011
Last time in Lecture 12
• Pipelining is complicated by multiple and/or variable
latency functional units
• Out-of-order and/or pipelined execution requires
tracking of dependencies
– RAW
– WAR
– WAW
• Dynamic issue logic can support out-of-order
execution to improve performance
– Last time, looked at simple scoreboard to track out-of-order
completion
• Hardware register renaming can further improve
performance by removing hazards.
March 2, 2011
CS152, Spring 2011
2
Register Renaming
ALU
IF
ID
Mem
Issue
WB
Fadd
Fmul
• Decode does register renaming and adds instructions to
the issue-stage instruction reorder buffer (ROB)
 renaming makes WAR or WAW hazards impossible
• Any instruction in ROB whose RAW hazards have been
satisfied can be issued.
 Out-of-order or dataflow execution
March 2, 2011
CS152, Spring 2011
3
Renaming Structures
Renaming
table &
regfile
Ins# use exec
op
p1
src1
p2
src2
Reorder
buffer
Replacing the
tag by its value
is an expensive
operation
Load
Unit
FU
FU
t1
t2
.
.
tn
Store
Unit
< t, result >
• Instruction template (i.e., tag t) is allocated by the
Decode stage, which also associates tag with register in regfile
• When an instruction completes, its tag is deallocated
March 2, 2011
CS152, Spring 2011
4
Reorder Buffer Management
Ins#
use exec
op
p1
src1
p2
src2
t1
t2
.
. Destination registers
. are renamed to the
ptr2
next to
deallocate
instruction’s slot tag
ptr1
next
available
tn
ROB managed circularly
•“exec” bit is set when instruction begins execution
•When an instruction completes its “use” bit is marked free
• ptr2 is incremented only if the “use” bit is marked free
Instruction slot is candidate for execution when:
• It holds a valid instruction (“use” bit is set)
• It has not already started execution (“exec” bit is clear)
• Both operands are available (p1 and p2 are set)
March 2, 2011
CS152, Spring 2011
5
Renaming & Out-of-order Issue
An example
Renaming table
v1
F1
F2
F3
F4
F5
F6
F7
F8
p
data
v1
t1
t5
t2
Reorder buffer
Ins# use exec
op p1
src1
p2 src2
0
1
1
0
LD
LD
3
0
1
10
1
0
MUL
10
v2
t2
11
v1
v1
4
5
10
1
1
0
0
SUB
DIV
1
1
v1
v1
1
0
1
v1
t4
v4
1
22
t3
t1
t2
t3
t4
t5
.
.
v4
t4
data / ti
1
2
3
4
5
6
LD
LD
MULTD
SUBD
DIVD
ADDD
March 2, 2011
F2,
F4,
F6,
F8,
F4,
F10,
34(R2)
45(R3)
F4,
F2,
F2,
F6,
F2
F2
F8
F4
• When are tags in sources
replaced by data?
Whenever an FU produces data
• When can a name be reused?
Whenever an instruction completes
CS152, Spring 2011
6
IBM 360/91 Floating-Point Unit
R. M. Tomasulo, 1967
1
2
3
4
5
6
p
p
p
p
p
p
Distribute
instruction
templates
by
functional
units
tag/data
tag/data
tag/data
tag/data
tag/data
tag/data
load
buffers
(from
memory)
instructions
...
1
2
3
4
p
p
p
p
tag/data
tag/data
tag/data
tag/data
FloatingPoint
Regfile
1 p tag/data p tag/data
2 p tag/data p tag/data 1 p tag/data p tag/data
3 p tag/data p tag/data 2 p tag/data p tag/data
Adder
Mult
< tag, result >
store buffers
(to memory)
March 2, 2011
p tag/data
p tag/data
p tag/data
Common bus ensures that data is made available
immediately to all the instructions waiting for it.
Match tag, if equal, copy value & set presence “p”.
CS152, Spring 2011
7
Effectiveness?
Renaming and Out-of-order execution was first
implemented in 1969 in IBM 360/91 but did not
show up in the subsequent models until midNineties.
Why ?
Reasons
1. Effective on a very small class of programs
2. Memory latency a much bigger problem
3. Exceptions not precise!
One more problem needed to be solved
Control Hazards!
March 2, 2011
CS152, Spring 2011
8
Precise Interrupts
It must appear as if an interrupt is taken between
two instructions (say Ii and Ii+1)
• the effect of all instructions up to and including Ii is
totally complete
• no effect of any instruction after Ii has taken place
The interrupt handler either aborts the program or
restarts it at Ii+1 .
March 2, 2011
CS152, Spring 2011
9
Effect on Interrupts
Out-of-order Completion
I1
I2
I3
I4
I5
I6
out-of-order comp
DIVD
LD
MULTD
DIVD
SUBD
ADDD
1
2
f6,
f2,
f0,
f8,
f10,
f6,
2
3
f6,
45(r3)
f2,
f6,
f0,
f8,
1 4 3 5
restore f2
f4
f4
f2
f6
f2
5
4 6 6
restore f10
Consider interrupts
Precise interrupts are difficult to implement at high speed
- want to start execution of later instructions before
exception checks finished on earlier instructions
March 2, 2011
CS152, Spring 2011
10
Exception Handling
Commit
Point
(In-Order Five-Stage Pipeline)
Inst.
Mem
PC
Select
Handler
PC
PC Address
Exceptions
Kill F
Stage
•
•
•
•
D
Decode
E
Illegal
Opcode
+
M
Overflow
Data
Mem
Data Addr
Except
W
Kill
Writeback
Exc
D
Exc
E
Exc
M
Cause
PC
D
PC
E
PC
M
EPC
Kill D
Stage
Kill E
Stage
Asynchronous
Interrupts
Hold exception flags in pipeline until commit point (M stage)
Exceptions in earlier pipe stages override later exceptions
Inject external interrupts at commit point (override others)
If exception at commit: update Cause and EPC registers, kill
all stages, inject handler PC into fetch stage
March 2, 2011
CS152, Spring 2011
11
Phases of Instruction Execution
PC
I-cache
Fetch: Instruction bits retrieved
from cache.
Fetch Buffer
Decode/Rename Decode: Instructions dispatched to
appropriate issue-stage buffer
Issue Buffer
Execute: Instructions and operands issued
to execution units.
Functional Units
When execution completes, all results and
exception flags are available.
Result Buffer
Commit
Commit: Instruction irrevocably updates
architectural state (aka “graduation”).
Architectural
State
March 2, 2011
CS152, Spring 2011
12
In-Order Commit for Precise Exceptions
In-order
Fetch
Out-of-order
Reorder Buffer
Decode
Kill
In-order
Commit
Kill
Kill
Execute
Inject handler PC
Exception?
• Instructions fetched and decoded into instruction
reorder buffer in-order
• Execution is out-of-order (  out-of-order completion)
• Commit (write-back to architectural state, i.e., regfile &
memory, is in-order
Temporary storage needed to hold results before commit
(shadow registers and store buffers)
March 2, 2011
CS152, Spring 2011
13
Extensions for Precise Exceptions
Inst# use exec
op
p1
src1 p2 src2
pd dest
data cause
ptr2
next to
commit
ptr1
next
available
Reorder buffer
• add <pd, dest, data, cause> fields in the instruction template
• commit instructions to reg file and memory in program
order  buffers can be maintained circularly
• on exception, clear reorder buffer by resetting ptr1=ptr2
(stores must wait for commit before updating memory)
March 2, 2011
CS152, Spring 2011
14
Rollback and Renaming
Register File
(now holds only
committed state)
Ins# use exec
op
p1
src1
p2
src2
pd dest
t1
t2
.
.
tn
data
Reorder
buffer
Load
Unit
FU
FU
FU
Store
Unit
Commit
< t, result >
Register file does not contain renaming tags any more.
How does the decode stage find the tag of a source register?
Search the “dest” field in the reorder buffer
March 2, 2011
CS152, Spring 2011
15
Renaming Table
Rename
Table
r1
r2
t
tag
valid bit
v
Ins# use exec
op
p1
Register
File
src1
p2
src2
pd dest
t1
t2
.
.
tn
data
Reorder
buffer
Load
Unit
FU
FU
FU
Store
Unit
Commit
< t, result >
Renaming table is a cache to speed up register name look up.
It needs to be cleared after each exception taken.
When else are valid bits cleared?
Control transfers
March 2, 2011
CS152, Spring 2011
16
Control Flow Penalty
Next fetch
started
PC
I-cache
Modern processors may have
> 10 pipeline stages between
next PC calculation and branch
resolution !
Fetch
Buffer
Fetch
Decode
Issue
Buffer
How much work is lost if
pipeline doesn’t follow
correct instruction flow?
Func.
Units
Execute
~ Loop length x pipeline width
Branch
executed
Result
Buffer
Commit
Arch.
State
March 2, 2011
CS152, Spring 2011
17
MIPS Branches and Jumps
Each instruction fetch depends on one or two pieces
of information from the preceding instruction:
1) Is the preceding instruction a taken branch?
2) If so, what is the target address?
Instruction
Taken known?
Target known?
J
After Inst. Decode
After Inst. Decode
JR
After Inst. Decode
After Reg. Fetch
BEQZ/BNEZ
After Reg. Fetch*
After Inst. Decode
*Assuming
March 2, 2011
zero detect on register read
CS152, Spring 2011
18
Branch Penalties in Modern Pipelines
UltraSPARC-III instruction fetch pipeline stages
(in-order issue, 4-way superscalar, 750MHz, 2000)
Branch
Target
Address
Known
Branch
Direction &
Jump
Register
Target
Known
March 2, 2011
A
P
F
B
I
J
R
E
PC Generation/Mux
Instruction Fetch Stage 1
Instruction Fetch Stage 2
Branch Address Calc/Begin Decode
Complete Decode
Steer Instructions to Functional units
Register File Read
Integer Execute
Remainder of execute pipeline
(+ another 6 stages)
CS152, Spring 2011
19
Reducing Control Flow Penalty
Software solutions
• Eliminate branches - loop unrolling
Increases the run length
• Reduce resolution time - instruction scheduling
Compute the branch condition as early
as possible (of limited value)
Hardware solutions
• Find something else to do - delay slots
Replaces pipeline bubbles with useful work
(requires software cooperation)
• Speculate - branch prediction
Speculative execution of instructions beyond
the branch
March 2, 2011
CS152, Spring 2011
20
Branch Prediction
Motivation:
Branch penalties limit performance of deeply pipelined
processors
Modern branch predictors have high accuracy
(>95%) and can reduce branch penalties significantly
Required hardware support:
Prediction structures:
• Branch history tables, branch target buffers, etc.
Mispredict recovery mechanisms:
• Keep result computation separate from commit
• Kill instructions following branch in pipeline
• Restore state to state following branch
March 2, 2011
CS152, Spring 2011
21
Static Branch Prediction
Overall probability a branch is taken is ~60-70% but:
backward
90%
forward
50%
JZ
JZ
ISA can attach preferred direction semantics to branches,
e.g., Motorola MC88110
bne0 (preferred taken) beq0 (not taken)
ISA can allow arbitrary choice of statically predicted direction,
e.g., HP PA-RISC, Intel IA-64
typically reported as ~80% accurate
March 2, 2011
CS152, Spring 2011
22
Dynamic Branch Prediction
learning based on past behavior
Temporal correlation
The way a branch resolves may be a good
predictor of the way it will resolve at the next
execution
Spatial correlation
Several branches may resolve in a highly
correlated manner (a preferred path of
execution)
March 2, 2011
CS152, Spring 2011
23
Branch Prediction Bits
• Assume 2 BP bits per instruction
• Change the prediction after two consecutive mistakes!
taken
taken
take
right
¬ taken
¬take
wrong
taken
¬ taken
¬take
right
¬ taken
taken
take
wrong
¬ taken
BP state:
(predict take/¬take) x (last prediction right/wrong)
March 2, 2011
CS152, Spring 2011
24
Branch History Table
Fetch PC
00
k
I-Cache
Instruction
Opcode
BHT Index
2k-entry
BHT,
2 bits/entry
offset
+
Branch?
Target PC
Taken/¬Taken?
4K-entry BHT, 2 bits/entry, ~80-90% correct predictions
March 2, 2011
CS152, Spring 2011
25
Exploiting Spatial Correlation
Yeh and Patt, 1992
if (x[i] <
y +=
if (x[i] <
c -=
7) then
1;
5) then
4;
If first condition false, second condition also false
History register, H, records the direction of the last
N branches executed by the processor
March 2, 2011
CS152, Spring 2011
26
Two-Level Branch Predictor
Pentium Pro uses the result from the last two branches
to select one of the four sets of BHT bits (~95% correct)
00
Fetch PC
k
2-bit global branch
history shift register
Shift in
Taken/¬Taken
results of each
branch
Taken/¬Taken?
March 2, 2011
CS152, Spring 2011
27
Speculating Both Directions
An alternative to branch prediction is to execute
both directions of a branch speculatively
• resource requirement is proportional to the
number of concurrent speculative executions
• only half the resources engage in useful work
when both directions of a branch are executed
speculatively
• branch prediction takes less resources
than speculative execution of both paths
With accurate branch prediction, it is more cost
effective to dedicate all resources to the predicted
direction
March 2, 2011
CS152, Spring 2011
28
Limitations of BHTs
Only predicts branch direction. Therefore, cannot redirect
fetch stream until after branch target is determined.
Correctly
predicted
taken branch
penalty
Jump Register
penalty
A
P
F
B
I
J
R
E
PC Generation/Mux
Instruction Fetch Stage 1
Instruction Fetch Stage 2
Branch Address Calc/Begin Decode
Complete Decode
Steer Instructions to Functional units
Register File Read
Integer Execute
Remainder of execute pipeline
(+ another 6 stages)
UltraSPARC-III fetch pipeline
March 2, 2011
CS152, Spring 2011
29
CS152 Administrivia
March 2, 2011
CS152, Spring 2011
30
Branch Target Buffer
predicted
target
BPb
Branch
Target
Buffer
(2k entries)
IMEM
k
PC
target
BP
BP bits are stored with the predicted target address.
IF stage: If (BP=taken) then nPC=target else nPC=PC+4
later:
check prediction, if wrong then kill the instruction
and update BTB & BPb else update BPb
March 2, 2011
CS152, Spring 2011
31
Address Collisions
132 Jump 100
Assume a
128-entry
BTB
1028 Add .....
target
236
BPb
take
Instruction
What will be fetched after the instruction at 1028? Memory
BTB prediction
Correct target
= 236
= 1032
 kill PC=236 and fetch PC=1032
Is this a common occurrence?
Can we avoid these bubbles?
March 2, 2011
CS152, Spring 2011
32
BTB is only for Control Instructions
BTB contains useful information for branch and
jump instructions only
 Do not update it for other instructions
For all other instructions the next PC is PC+4 !
How to achieve this effect without decoding the
instruction?
March 2, 2011
CS152, Spring 2011
33
Branch Target Buffer (BTB)
I-Cache
2k-entry direct-mapped BTB
PC
(can also be associative)
Entry PC
Valid
predicted
target PC
valid
target
k
=
match
•
•
•
•
Keep both the branch PC and target PC in the BTB
PC+4 is fetched if match fails
Only taken branches and jumps held in BTB
Next PC determined before branch fetched and decoded
March 2, 2011
CS152, Spring 2011
34
Combining BTB and BHT
• BTB entries are considerably more expensive than BHT, but can
redirect fetches at earlier stage in pipeline and can accelerate
indirect branches (JR)
• BHT can hold many more entries and is more accurate
BTB
BHT in later
pipeline stage
corrects when
BTB misses a
predicted
taken branch
BHT
A
P
F
B
I
J
R
E
PC Generation/Mux
Instruction Fetch Stage 1
Instruction Fetch Stage 2
Branch Address Calc/Begin Decode
Complete Decode
Steer Instructions to Functional units
Register File Read
Integer Execute
BTB/BHT only updated after branch resolves in E stage
March 2, 2011
CS152, Spring 2011
35
Uses of Jump Register (JR)
• Switch statements (jump to address of matching case)
BTB works well if same case used repeatedly
• Dynamic function call (jump to run-time function address)
BTB works well if same function usually called, (e.g., in
C++ programming, when objects have same type in
virtual function call)
• Subroutine returns (jump to return address)
BTB works well if usually return to the same place
 Often one function called from many distinct call sites!
How well does BTB work for each of these cases?
March 2, 2011
CS152, Spring 2011
36
Subroutine Return Stack
Small structure to accelerate JR for subroutine returns,
typically much more accurate than BTBs.
fa() { fb(); }
fb() { fc(); }
fc() { fd(); }
Pop return address
when subroutine
return decoded
Push call address when
function call executed
&fd()
&fc()
&fb()
March 2, 2011
CS152, Spring 2011
k entries
(typically k=8-16)
37
Mispredict Recovery
In-order execution machines:
– Assume no instruction issued after branch can write-back before
branch resolves
– Kill all instructions in pipeline behind mispredicted branch
Out-of-order execution?
– Multiple instructions following branch in program
order can complete before branch resolves
March 2, 2011
CS152, Spring 2011
38
In-Order Commit for Precise Exceptions
In-order
Fetch
Out-of-order
Reorder Buffer
Decode
Kill
In-order
Commit
Kill
Kill
Execute
Inject handler PC
Exception?
• Instructions fetched and decoded into instruction
reorder buffer in-order
• Execution is out-of-order (  out-of-order completion)
• Commit (write-back to architectural state, i.e., regfile &
memory, is in-order
Temporary storage needed in ROB to hold results before commit
March 2, 2011
CS152, Spring 2011
39
Branch Misprediction in Pipeline
Inject correct PC
Branch
Prediction
Kill
Kill
PC
Fetch
Decode
Branch
Resolution
Kill
Reorder Buffer
Commit
Complete
Execute
• Can have multiple unresolved branches in ROB
• Can resolve branches out-of-order by killing all the
instructions in ROB that follow a mispredicted branch
March 2, 2011
CS152, Spring 2011
40
Recovering ROB/Renaming Table
Rename
Table r1
t vv
t
t t vv
Rename
Snapshots
Registe
r File
r2
Ptr2
next to commit Ins# use exec
op
p1
src1
p2
src2
pd dest
data
t1
t2
.
.
tn
rollback
next available
Ptr1
next available
Reorder
buffer
Load
Unit
FU
FU
FU
Store
Unit
Commit
< t, result >
Take snapshot of register rename table at each predicted
branch, recover earlier snapshot if branch mispredicted
March 2, 2011
CS152, Spring 2011
41
“Data-in-ROB” Design
(HP PA8000, Pentium Pro, Core2Duo, Nehalem)
Register File
holds only
committed state
Ins# use exec
op
p1
src1
p2
src2
pd dest
t1
t2
.
.
tn
data
Reorder
buffer
Load
Unit
FU
FU
FU
Store
Unit
Commit
< t, result >
• On dispatch into ROB, ready sources can be in regfile or in ROB
dest (copied into src1/src2 if ready before dispatch)
• On completion, write to dest field and broadcast to src fields.
• On issue, read from ROB src fields
March 2, 2011
CS152, Spring 2011
42
Acknowledgements
• These slides contain material developed and
copyright by:
–
–
–
–
–
–
Arvind (MIT)
Krste Asanovic (MIT/UCB)
Joel Emer (Intel/MIT)
James Hoe (CMU)
John Kubiatowicz (UCB)
David Patterson (UCB)
• MIT material derived from course 6.823
• UCB material derived from course CS252
March 2, 2011
CS152, Spring 2011
43