18-447 Computer Architecture Lecture 7: Pipelining Prof. Onur Mutlu Carnegie Mellon University Spring 2014, 1/29/2014 Can We Do Better? What limitations do you see with the multi-cycle design? Limited concurrency Some hardware resources are idle during different phases of instruction processing cycle “Fetch” logic is idle when an instruction is being “decoded” or “executed” Most of the datapath is idle when a memory access is happening 2 Can We Use the Idle Hardware to Improve Concurrency? Goal: Concurrency throughput (more “work” completed in one cycle) Idea: When an instruction is using some resources in its processing phase, process other instructions on idle resources not needed by that instruction E.g., when an instruction is being decoded, fetch the next instruction E.g., when an instruction is being executed, decode another instruction E.g., when an instruction is accessing data memory (ld/st), execute the next instruction E.g., when an instruction is writing its result into the register file, access data memory for the next instruction 3 Pipelining: Basic Idea More systematically: Pipeline the execution of multiple instructions Analogy: “Assembly line processing” of instructions Idea: Divide the instruction processing cycle into distinct “stages” of processing Ensure there are enough hardware resources to process one instruction in each stage Process a different instruction in each stage Instructions consecutive in program order are processed in consecutive stages Benefit: Increases instruction processing throughput (1/CPI) Downside: Start thinking about this… 4 Example: Execution of Four Independent ADDs Multi-cycle: 4 cycles per instruction F D E W F D E W F D E W F D E W Time Pipelined: 4 cycles per 4 instructions (steady state) F D E W F D E W F D E W F D E W Time 5 The Laundry Analogy Time 6 PM 7 8 9 10 11 12 1 2 AM Task order A B C D “place one dirty load of clothes in the washer” “when the washer is finished, place the wet load in the dryer” 1 6 PM 8 9 10 11 dry 12 2 AM “when the dryer is7 finished, take out the load and fold” Time “when folding is finished, ask your roommate (??) to put the clothes Task away” order A B C - steps to do a load are sequentially dependent - no dependence between different loads - different steps do not share resources D Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 6 Pipelining Multiple Loads of Laundry 6 PM 7 8 9 10 11 12 1 2 AM 6 PM TimeA 7 8 9 10 11 12 1 2 AM 7 8 9 10 11 12 1 2 AM 7 8 9 10 11 12 1 2 AM Time Task order Task B order A C D B C D 6 PM Time Task order 6 PM Time A Task order B A C B D C D Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] - 4 loads of laundry in parallel - no additional resources - throughput increased by 4 - latency per load is the same 7 Pipelining Multiple Loads of Laundry: In Practice Time 6 PM 7 8 9 10 11 12 1 2 AM 6 PM 7 8 9 10 11 12 1 2 AM 6 PM 7 8 9 10 11 12 1 2 AM 6 PM 7 8 9 10 11 12 1 2 AM Task order A Time B Task order C A D B C D Time Task order TimeA Task B order C A D B C the slowest step decides throughput D Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 8 Pipelining Multiple Loads of Laundry: In Practice 6 PM 7 8 9 10 11 12 1 2 AM A 6 PM Time 7 8 9 10 11 12 1 2 AM 6 PM 7 8 9 10 11 12 1 2 AM Task order 6 PM Time A 7 8 9 10 11 12 1 2 AM Time Task order Task B order C A D B C D Time Task B order C A B D A B A B C D Throughput restored (2 loads per hour) using 2 dryers Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 9 An Ideal Pipeline Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing) Repetition of identical operations Repetition of independent operations No dependencies between repeated operations Uniformly partitionable suboperations The same operation is repeated on a large number of different inputs Processing can be evenly divided into uniform-latency suboperations (that do not share resources) Fitting examples: automobile assembly line, doing laundry What about the instruction processing “cycle”? 10 Ideal Pipelining combinational logic (F,D,E,M,W) T psec T/2 ps (F,D,E) T/3 ps (F,D) BW=~(1/T) BW=~(2/T) T/2 ps (M,W) T/3 ps (E,M) T/3 ps (M,W) BW=~(3/T) 11 More Realistic Pipeline: Throughput Nonpipelined version with delay T BW = 1/(T+S) where S = latch delay T ps k-stage pipelined version BWk-stage = 1 / (T/k +S ) BWmax = 1 / (1 gate delay + S ) T/k ps T/k ps 12 More Realistic Pipeline: Cost Nonpipelined version with combinational cost G Cost = G+L where L = latch cost G gates k-stage pipelined version Costk-stage = G + Lk G/k G/k 13 Pipelining Instruction Processing 14 Remember: The Instruction Processing Cycle Fetch fetch (IF) 1. Instruction Decodedecode and 2. Instruction register operand fetch (ID/RF) Evaluate Address 3. Execute/Evaluate memory address (EX/AG) Fetch Operands 4. Memory operand fetch (MEM) Execute 5. Store/writeback result (WB) Store Result 15 Remember the Single-Cycle Uarch Instruction [25– 0] 26 Shift left 2 Jump address [31– 0] 28 PC+4 [31– 28] ALU Add result Add 4 Instruction [31– 26] Control Instruction [25– 21] PC Read address Instruction memory Instruction [15– 11] M u x M u x 1 0 Shift left 2 RegDst Jump Branch MemRead MemtoReg ALUOp MemWrite ALUSrc RegWrite PCSrc2=Br Taken Read register 1 Instruction [20– 16] Instruction [31– 0] PCSrc 1 1=Jump 0 0 M u x 1 Read data 1 Read register 2 Registers Read Write data 2 register 0 M u x 1 Write data Zero ALU ALU result Read data Address Write Data memory 1 M u x 0 bcond data Instruction [15– 0] 16 Sign extend 32 ALU control Instruction [5– 0] ALU operation T Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] BW=~(1/T) 16 Dividing Into Stages 200ps 100ps 200ps IF: Instruction fetch ID: Instruction decode/ register file read EX: Execute/ address calculation 200ps 100ps MEM: Memory access WB: Write back 0 M u x 1 ignore for now Add 4 Add Add result Shift left 2 PC Read register 1 Address Instruction Instruction memory Read data 1 Read register 2 Registers Read Write data 2 register Write data 0 M u x 1 Zero ALU ALU result Address Data memory Write data 16 Sign extend Read data 1 M u x 0 RF write 32 Is this the correct partitioning? Why not 4 or 6 stages? Why not different boundaries? Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 17 Instruction Pipeline Throughput Program execution Time order (in instructions) lw $1, 100($0) 2004 2 Instruction Reg fetch lw $2, 200($0) 400 6 ALU 600 8 800 Data access 101000 12 1200 14 1400 16 1600 ALU Data access Reg 18 1800 Reg Instruction Reg fetch 8 ns 800ps lw $3, 300($0) Instruction fetch 8800ps ns ... 8 ns 800ps Program execution Time order (in instructions) 2 lw $1, 100($0) Instruction fetch lw $2, 200($0) 2 ns 200ps lw $3, 300($0) 200 4 Reg Instruction fetch 2 ns 200ps 400 6 600 ALU Reg Instruction fetch 2 ns 200ps 8800 Data access ALU Reg 2 ns 200ps 1000 10 1200 12 1400 14 Reg Data access Reg ALU Data access 2200ps ns 2 ns 200ps Reg 2 ns 200ps 5-stage speedup is 4, not 5 as predicted by the ideal model. Why? 18 Enabling Pipelined Processing: Pipeline Registers IF: Instruction fetch ID: Instruction decode/ register file read WB: Write back PCE+4 EX/MEM MEM/WB nPCM ID/EX PCD+4 IF/ID 44 MEM: Memory access No resource is used by more than 1 stage! 00 MM uu x x 11 Add Add EX: Execute/ address calculation Add Add Add result Add result 1616 Sign Sign extend extend 3232 T Write Write data data Read Read data data MDRW AoutM AE Data Data memory memory T/k ps Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] Address Address 11 MM uu xx 00 AoutW Write Write data data 00 MM uu xx 11 Zero Zero ALU ALU ALU ALU result result BM Instruction memory Read Read data data 11 Read Read register register 22 Registers Read Registers Read Write Write data data 22 register register BE Instruction Instruction memory Read Read register register 11 ImmE Address Address Instruction PCPC IRD PCF Shift Shift leftleft 22 T/k ps 19 0 Write data Write data 16 32 32Sign Sign extend extend Pipelined Operation Example 16 lw Instruction fetch All instruction classes must follow the same path and timing through the Any performance impact? lw pipeline stages. 0 00 00 M M M uuu x xxx 111 Instruction decode IF/ID IF/ID IF/ID IF/ID IF/ID lw lw Execution Memory ID/EX ID/EX ID/EX ID/EX lw Write back EX/MEM EX/MEM EX/MEM EX/MEM MEM/WB MEM/WB MEM/WB MEM/WB Add Add Add Add Add Add Add Add Add result Add Add Add result result result 4 444 PC PC PC Address Address Address Instruction Instruction Instruction memory memory memory Instruction Instruction Instruction Instruction Instruction Shift Shift Shift Shift left left222 left left Read Read Read Read register register111 register Read Read Read data data111 data Read data Read Read Read Read register register register222 Registers Registers Read Registers Read Read Write Write data Write data222 data data register register register register Write Write Write data data data 16 16 16 000 M M M uuu x xx 111 Zero Zero Zero Zero ALU ALU ALU ALU ALU ALU ALU ALU ALU result result result result Address Address Address Address Data Data Data Data memory memory memory memory memory Write Write Write Write data data data Read Read Read Read data data data data data 11 11 M M M M uuu xxxx 0000 32 32 32 Sign Sign Sign extend extend extend lw 0 0 M u M x u Based on original figure from [P&H CO&D, x 1 COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] Instruction decode lw Write back 20 data register 1M uu xx 11 Write Write data data Pipelined Operation Example 16 16 16 Sign extend Sign Sign extend extend 32 Write data M 0M uu xx 00 Data memory memory Write Write data data 32 32 Clock 1 Clock Clock 5 3 lw $10, sub $11,20($1) $2, $3 Instruction fetch lw $10, sub $11,20($1) $2, $3 lw $10, 20($1) Instruction decode Execution sub $11, $2, $3 0 00 M M M u uu xxx 1 11 lw $10, sub $11,20($1) $2, $3 Memory Memory Execution IF/ID IF/ID IF/ID ID/EX ID/EX ID/EX EX/MEM EX/MEM EX/MEM sub $11,20($1) $2, $3 lw $10, Write back back Write MEM/WB MEM/WB MEM/WB Add Add Add Add Add Add Add Add Add result result result 444 PC PC PC Address Address Address Instruction Instruction Instruction memory memory memory Instruction Instruction Shift Shift Shift left 22 2 left left Read Read Read register 11 1 register register Read Read Read data 11 1 data data Read Read Read register 22 2 register register Registers Read Registers Registers Read Read Write Write 2 Write data 22 data register register register Write Write Write data data 16 16 16 0 00 M M uuu xxx 1 11 Zero Zero Zero ALU ALU ALU ALU ALU ALU result result result Address Address Address Read Read Read data data data Data Data Data Data memory memory memory Write Write Write data data data 1 11 M M u uu xxx 0 00 32 32 Sign 32 Sign Sign extend extend extend Clock Clock 56 21 43 Clock Clock Clock sub $11, $2, $3 21 lw $10, 20($1) Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] sub $11, $2, $3 lw $10, 20($1) sub $11, $2, $3 Illustrating Pipeline Operation: Operation View t0 t1 t2 t3 t4 Inst0 IF Inst1 Inst2 Inst3 Inst4 ID IF EX ID IF MEM EX ID IF WB MEM EX ID IF t5 WB MEM EX ID IF WB MEM EX ID IF WB MEM EX ID IF 22 Illustrating Pipeline Operation: Resource View t0 IF ID EX MEM WB I0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I0 I1 I2 I3 I4 I5 I6 I7 I8 I9 I0 I1 I2 I3 I4 I5 I6 I7 I8 I0 I1 I2 I3 I4 I5 I6 I7 I0 I1 I2 I3 I4 I5 I6 23 Control Points in a Pipeline PCSrc 0 M u x 1 IF/ID ID/EX EX/MEM MEM/WB Add Add result Add 4 Branch Shift left 2 PC Address Instruction memory Instruction RegWrite Read register 1 MemWrite Read data 1 Read register 2 Registers Read Write data 2 register Write data ALUSrc Zero Zero ALU ALU result 0 M u x 1 MemtoReg Address Data memory Write Read data 1 M u x 0 data Instruction 16 [15– 0] Sign extend 32 6 ALU control MemRead Instruction [20– 16] Instruction [15– 11] Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 0 M u x 1 ALUOp RegDst Identical set of control points as the single-cycle datapath!! 24 Control Signals in a Pipeline For a given instruction same control signals as single-cycle, but control signals required at different cycles, depending on stage decode once using the same logic as single-cycle and buffer control signals until consumed WB Instruction IF/ID Control M WB EX M WB ID/EX EX/MEM MEM/WB or carry relevant “instruction word/field” down the pipeline and decode locally within each or in a previous stage Which one is better? 25 Pipelined Control Signals PCSrc ID/EX 0 M u x 1 WB Control IF/ID EX/MEM M WB EX M MEM/WB WB Add Add Add result Instruction memory ALUSrc Read register 1 Read data 1 Read register 2 Registers Read Write data 2 register Write data Zero ALU ALU result 0 M u x 1 MemtoReg Address Branch Shift left 2 MemWrite PC Instruction RegWrite 4 Address Data memory Read data Write data Instruction 16 [15– 0] Instruction [20– 16] Instruction [15– 11] Sign extend 32 6 ALU control 0 M u x 1 1 M u x 0 MemRead ALUOp RegDst Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 26 An Ideal Pipeline Goal: Increase throughput with little increase in cost (hardware cost, in case of instruction processing) Repetition of identical operations Repetition of independent operations No dependencies between repeated operations Uniformly partitionable suboperations The same operation is repeated on a large number of different inputs Processing an be evenly divided into uniform-latency suboperations (that do not share resources) Fitting examples: automobile assembly line, doing laundry What about the instruction processing “cycle”? 27 Instruction Pipeline: Not An Ideal Pipeline Identical operations ... NOT! different instructions do not need all stages - Forcing different instructions to go through the same multi-function pipe external fragmentation (some pipe stages idle for some instructions) Uniform suboperations ... NOT! difficult to balance the different pipeline stages - Not all pipeline stages do the same amount of work internal fragmentation (some pipe stages are too-fast but take the same clock cycle time) Independent operations ... NOT! instructions are not independent of each other - Need to detect and resolve inter-instruction dependencies to ensure the pipeline operates correctly Pipeline is not always moving (it stalls) 28 Issues in Pipeline Design Balancing work in pipeline stages How many stages and what is done in each stage Keeping the pipeline correct, moving, and full in the presence of events that disrupt pipeline flow Handling dependences Data Control Handling resource contention Handling long-latency (multi-cycle) operations Handling exceptions, interrupts Advanced: Improving pipeline throughput Minimizing stalls 29 Causes of Pipeline Stalls Resource contention Dependences (between instructions) Data Control Long-latency (multi-cycle) operations 30 Dependences and Their Types Also called “dependency” or less desirably “hazard” Dependencies dictate ordering requirements between instructions Two types Data dependence Control dependence Resource contention is sometimes called resource dependence However, this is not fundamental to (dictated by) program semantics, so we will treat it separately 31 Handling Resource Contention Happens when instructions in two pipeline stages need the same resource Solution 1: Eliminate the cause of contention Duplicate the resource or increase its throughput E.g., use separate instruction and data memories (caches) E.g., use multiple ports for memory structures Solution 2: Detect the resource contention and stall one of the contending stages Which stage do you stall? Example: What if you had a single read and write port for the register file? 32 Data Dependences Types of data dependences Flow dependence (true data dependence – read after write) Output dependence (write after write) Anti dependence (write after read) Which ones cause stalls in a pipelined machine? For all of them, we need to ensure semantics of the program are correct Flow dependences always need to be obeyed because they constitute true dependence on a value Anti and output dependences exist due to limited number of architectural registers They are dependence on a name, not a value We will later see what we can do about them 33 Data Dependence Types Flow dependence r3 r1 op r2 r5 r3 op r4 Read-after-Write (RAW) Anti dependence r3 r1 op r2 r1 r4 op r5 Write-after-Read (WAR) Output-dependence r3 r1 op r2 r5 r3 op r4 r3 r6 op r7 Write-after-Write (WAW) 34 How to Handle Data Dependences Anti and output dependences are easier to handle write to the destination in one stage and in program order Flow dependences are more interesting Five fundamental ways of handling flow dependences 35 Readings for Next Few Lectures P&H Chapter 4.9-4.11 Smith and Sohi, “The Microarchitecture of Superscalar Processors,” Proceedings of the IEEE, 1995 More advanced pipelining Interrupt and exception handling Out-of-order and superscalar execution concepts 36 Review: Pipelining: Basic Idea Idea: Divide the instruction processing cycle into distinct “stages” of processing Ensure there are enough hardware resources to process one instruction in each stage Process a different instruction in each stage Instructions consecutive in program order are processed in consecutive stages Benefit: Increases instruction processing throughput (1/CPI) Downside: ??? 37 Review: Execution of Four Independent ADDs Multi-cycle: 4 cycles per instruction F D E W F D E W F D E W F D E W Time Pipelined: 4 cycles per 4 instructions (steady state) F D E W F D E W F D E F D Is life always this beautiful? W E W Time 38 data register 1M uu xx 11 Write Write data data Write data M 0M uu xx 00 Data memory memory Review: Pipelined Operation Example 16 16 16 Sign extend Sign Sign extend extend 32 Write Write data data 32 32 Clock 1 Clock Clock 5 3 lw $10, sub $11,20($1) $2, $3 Instruction fetch lw $10, sub $11,20($1) $2, $3 lw $10, 20($1) Instruction decode Execution sub $11, $2, $3 0 00 M M M u uu xxx 1 11 lw $10, sub $11,20($1) $2, $3 Memory Memory Execution IF/ID IF/ID IF/ID ID/EX ID/EX ID/EX EX/MEM EX/MEM EX/MEM sub $11,20($1) $2, $3 lw $10, Write back back Write MEM/WB MEM/WB MEM/WB Add Add Add Add Add Add Add Add Add result result result 444 PC PC PC Address Address Address Instruction Instruction Instruction memory memory memory Instruction Instruction Shift Shift Shift left 22 2 left left Read Read Read register 11 1 register register Read Read Read data 11 1 data data Read Read Read register 22 2 register register Registers Read Registers Registers Read Read Write Write 2 Write data 22 data register register register Write Write Write data data 0 00 M M uuu xxx 1 11 Zero Zero Zero ALU ALU ALU ALU ALU ALU result result result Is life always this beautiful? 16 16 16 Address Address Address Read Read Read data data data Data Data Data Data memory memory memory Write Write Write data data data 1 11 M M u uu xxx 0 00 32 32 Sign 32 Sign Sign extend extend extend Clock Clock 56 21 43 Clock Clock Clock sub $11, $2, $3 39 lw $10, 20($1) Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] sub $11, $2, $3 lw $10, 20($1) sub $11, $2, $3 Review: Instruction Pipeline: Not An Ideal Pipeline Identical operations ... NOT! different instructions do not need all stages - Forcing different instructions to go through the same multi-function pipe external fragmentation (some pipe stages idle for some instructions) Uniform suboperations ... NOT! difficult to balance the different pipeline stages - Not all pipeline stages do the same amount of work internal fragmentation (some pipe stages are too-fast but take the same clock cycle time) Independent operations ... NOT! instructions are not independent of each other - Need to detect and resolve inter-instruction dependencies to ensure the pipeline operates correctly Pipeline is not always moving (it stalls) 40 Review: Fundamental Issues in Pipeline Design Balancing work in pipeline stages How many stages and what is done in each stage Keeping the pipeline correct, moving, and full in the presence of events that disrupt pipeline flow Handling dependences Data Control Handling resource contention Handling long-latency (multi-cycle) operations Handling exceptions, interrupts Advanced: Improving pipeline throughput Minimizing stalls 41 Review: Data Dependences Types of data dependences Flow dependence (true data dependence – read after write) Output dependence (write after write) Anti dependence (write after read) Which ones cause stalls in a pipelined machine? For all of them, we need to ensure semantics of the program is correct Flow dependences always need to be obeyed because they constitute true dependence on a value Anti and output dependences exist due to limited number of architectural registers They are dependence on a name, not a value We will later see what we can do about them 42 Data Dependence Types Flow dependence r3 r1 op r2 r5 r3 op r4 Read-after-Write (RAW) Anti dependence r3 r1 op r2 r1 r4 op r5 Write-after-Read (WAR) Output-dependence r3 r1 op r2 r5 r3 op r4 r3 r6 op r7 Write-after-Write (WAW) 43 data register 1M uu xx 11 Write Write data data Pipelined Operation Example 16 16 16 Sign extend Sign Sign extend extend 32 Write data M 0M uu xx 00 Data memory memory Write Write data data 32 32 Clock 1 Clock Clock 5 3 lw $10, sub $11,20($1) $2, $3 Instruction fetch lw $10, sub $11,20($1) $2, $3 lw $10, 20($1) Instruction decode Execution sub $11, $2, $3 0 00 M M M u uu xxx 1 11 lw $10, sub $11,20($1) $2, $3 Memory Memory Execution IF/ID IF/ID IF/ID ID/EX ID/EX ID/EX EX/MEM EX/MEM EX/MEM sub $11,20($1) $2, $3 lw $10, Write back back Write MEM/WB MEM/WB MEM/WB Add Add Add Add Add Add Add Add Add result result result 444 PC PC PC Address Address Address Instruction Instruction Instruction memory memory memory Instruction Instruction Shift Shift Shift left 22 2 left left Read Read Read register 11 1 register register Read Read Read data 11 1 data data Read Read Read register 22 2 register register Registers Read Registers Registers Read Read Write Write 2 Write data 22 data register register register Write Write Write data data 0 00 M M uuu xxx 1 11 Zero Zero Zero ALU ALU ALU ALU ALU ALU result result result Address Address Address Read Read Read data data data Data Data Data Data memory memory memory What if the SUB were dependent on LW? 16 16 16 Write Write Write data data data 1 11 M M u uu xxx 0 00 32 32 Sign 32 Sign Sign extend extend extend Clock Clock 56 21 43 Clock Clock Clock sub $11, $2, $3 44 lw $10, 20($1) Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] sub $11, $2, $3 lw $10, 20($1) sub $11, $2, $3 How to Handle Data Dependences Anti and output dependences are easier to handle write to the destination in one stage and in program order Flow dependences are more interesting Five fundamental ways of handling flow dependences Detect and wait until value is available in register file Detect and forward/bypass data to dependent instruction Detect and eliminate the dependence at the software level No need for the hardware to detect dependence Predict the needed value(s), execute “speculatively”, and verify Do something else (fine-grained multithreading) No need to detect 45 Interlocking Detection of dependence between instructions in a pipelined processor to guarantee correct execution Software based interlocking vs. Hardware based interlocking MIPS acronym? 46 Approaches to Dependence Detection (I) Scoreboarding Each register in register file has a Valid bit associated with it An instruction that is writing to the register resets the Valid bit An instruction in Decode stage checks if all its source and destination registers are Valid Advantage: Yes: No need to stall… No dependence No: Stall the instruction Simple. 1 bit per register Disadvantage: Need to stall for all types of dependences, not only flow dep. 47 Not Stalling on Anti and Output Dependences What changes would you make to the scoreboard to enable this? 48 Approaches to Dependence Detection (II) Combinational dependence check logic Advantage: Special logic that checks if any instruction in later stages is supposed to write to any source register of the instruction that is being decoded Yes: stall the instruction/pipeline No: no need to stall… no flow dependence No need to stall on anti and output dependences Disadvantage: Logic is more complex than a scoreboard Logic becomes more complex as we make the pipeline deeper and wider (flash-forward: think superscalar execution) 49 Once You Detect the Dependence in Hardware What do you do afterwards? Observation: Dependence between two instructions is detected before the communicated data value becomes available Option 1: Stall the dependent instruction right away Option 2: Stall the dependent instruction only when necessary data forwarding/bypassing Option 3: … 50 Data Forwarding/Bypassing Problem: A consumer (dependent) instruction has to wait in decode stage until the producer instruction writes its value in the register file Goal: We do not want to stall the pipeline unnecessarily Observation: The data value needed by the consumer instruction can be supplied directly from a later stage in the pipeline (instead of only from the register file) Idea: Add additional dependence check logic and data forwarding paths (buses) to supply the producer’s value to the consumer right after the value is available Benefit: Consumer can move in the pipeline until the point the value can be supplied less stalling 51 A Special Case of Data Dependence Control dependence Data dependence on the Instruction Pointer / Program Counter 52 Control Dependence Question: What should the fetch PC be in the next cycle? Answer: The address of the next instruction If the fetched instruction is a non-control-flow instruction: Next Fetch PC is the address of the next-sequential instruction Easy to determine if we know the size of the fetched instruction If the instruction that is fetched is a control-flow instruction: All instructions are control dependent on previous ones. Why? How do we determine the next Fetch PC? In fact, how do we know whether or not the fetched instruction is a control-flow instruction? 53 Data Dependence Handling: More Depth & Implementation 54 Remember: Data Dependence Types Flow dependence r3 r1 op r2 r5 r3 op r4 Read-after-Write (RAW) Anti dependence r3 r1 op r2 r1 r4 op r5 Write-after-Read (WAR) Output-dependence r3 r1 op r2 r5 r3 op r4 r3 r6 op r7 Write-after-Write (WAW) 55 How to Handle Data Dependences Anti and output dependences are easier to handle write to the destination in one stage and in program order Flow dependences are more interesting Five fundamental ways of handling flow dependences Detect and wait until value is available in register file Detect and forward/bypass data to dependent instruction Detect and eliminate the dependence at the software level No need for the hardware to detect dependence Predict the needed value(s), execute “speculatively”, and verify Do something else (fine-grained multithreading) No need to detect 56 RAW Dependence Handling Following flow dependences lead to conflicts in the 5-stage pipeline addi ra r- - addi r- ra - addi r- ra - addi r- ra - addi r- ra - addi r- ra - IF ID EX MEM WB IF ID EX MEM WB IF ID EX MEM IF ID EX IF ?ID IF 57 Register Data Dependence Analysis R/I-Type LW SW Br read RF read RF read RF read RF write RF write RF J Jr IF ID read RF EX MEM WB For a given pipeline, when is there a potential conflict between 2 data dependent instructions? dependence type: RAW, WAR, WAW? instruction types involved? distance between the two instructions? 58 Safe and Unsafe Movement of Pipeline j:_rk stage X Reg Read iOj i:rk_ j:rk_ Reg Write iAj stage Y Reg Write RAW Dependence i:_rk j:rk_ Reg Write iDj Reg Read WAR Dependence i:rk_ Reg Write WAW Dependence dist(i,j) dist(X,Y) Unsafe ?? to keep j moving dist(i,j) > dist(X,Y) Safe ?? 59 RAW Dependence Analysis Example R/I-Type LW SW Br read RF read RF read RF read RF write RF write RF J Jr IF ID read RF EX MEM WB Instructions IA and IB (where IA comes before IB) have RAW dependence iff IB (R/I, LW, SW, Br or JR) reads a register written by IA (R/I or LW) dist(IA, IB) dist(ID, WB) = 3 What about WAW and WAR dependence? What about memory data dependence? 60 Pipeline Stall: Resolving Data Dependence t0 Insth IF i Insti Instj Instk Instl i: rx _ j: _ rx bubble j: _ rx bubble j: _ rx bubble j: _ rx t1 t2 t3 t4 ID IF ALU ID IF MEM ALU ID IF WB MEM ALU ID ID IF IF j t5 WB MEM ALU ID ALU ID IF ID IF IF ID WB MEM ALU WB MEM ALU MEM ID ALU ID IF WB MEM ALU ALU IF ID IF MEM ALU ID ID IF ALU ID IF IF ID IF dist(i,j)=1 Stall==make the dependent instruction IF dist(i,j)=2wait until its source data value is available dist(i,j)=3 1. stop all up-stream stages dist(i,j)=4 2. drain all down-stream stages 61 How to Implement Stalling PCSrc ID/EX 0 M u x 1 WB Control IF/ID EX/MEM M WB EX M MEM/WB WB Add Add Add result Instruction memory ALUSrc Read register 1 Read data 1 Read register 2 Registers Read Write data 2 register Write data Zero ALU ALU result 0 M u x 1 MemtoReg Address Branch Shift left 2 MemWrite PC Instruction RegWrite 4 Address Data memory Read data Write data Instruction 16 [15– 0] Sign extend Instruction [20– 16] Instruction [15– 11] 32 6 ALU control 0 M u x 1 1 M u x 0 MemRead ALUOp RegDst Stall disable PC and IR latching; ensure stalled instruction stays in its stage Insert “invalid” instructions/nops into the stage following the stalled one Based on original figure from [P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 62 Stall Conditions Instructions IA and IB (where IA comes before IB) have RAW dependence iff IB (R/I, LW, SW, Br or JR) reads a register written by IA (R/I or LW) dist(IA, IB) dist(ID, WB) = 3 In other words, must stall when IB in ID stage wants to read a register to be written by IA in EX, MEM or WB stage 63 Stall Conditions Helper functions Stall when rs(I) returns the rs field of I use_rs(I) returns true if I requires RF[rs] and rs!=r0 (rs(IRID)==destEX) && use_rs(IRID) && RegWriteEX or (rs(IRID)==destMEM) && use_rs(IRID) && RegWriteMEM (rs(IRID)==destWB) && use_rs(IRID) && RegWriteWB or (rt(IRID)==destEX) && use_rt(IRID) && RegWriteEX or (rt(IRID)==destMEM) && use_rt(IRID) && RegWriteMEM (rt(IRID)==destWB) && use_rt(IRID) && RegWriteWB or or It is crucial that the EX, MEM and WB stages continue to advance normally during stall cycles 64 Impact of Stall on Performance Each stall cycle corresponds to 1 lost ALU cycle For a program with N instructions and S stall cycles, Average CPI=(N+S)/N S depends on frequency of RAW dependences exact distance between the dependent instructions distance between dependences suppose i1,i2 and i3 all depend on i0, once i1’s dependence is resolved, i2 and i3 must be okay too 65 Sample Assembly (P&H) for (j=i-1; j>=0 && v[j] > v[j+1]; j-=1) { ...... } for2tst: addi slti bne sll add lw lw slt beq ......... addi j $s1, $s0, -1 $t0, $s1, 0 $t0, $zero, exit2 $t1, $s1, 2 $t2, $a0, $t1 $t3, 0($t2) $t4, 4($t2) $t0, $t4, $t3 $t0, $zero, exit2 3 stalls 3 stalls 3 stalls 3 stalls 3 stalls 3 stalls $s1, $s1, -1 for2tst exit2: 66 Data Forwarding (or Data Bypassing) It is intuitive to think of RF as state But, RF is just a part of a computing abstraction “add rx ry rz” literally means get values from RF[ry] and RF[rz] respectively and put result in RF[rx] “add rx ry rz” means 1. get the results of the last instructions to define the values of RF[ry] and RF[rz], respectively, and 2. until another instruction redefines RF[rx], younger instructions that refers to RF[rx] should use this instruction’s result What matters is to maintain the correct “dataflow” between operations, thus add ra r- r- addi r- ra r- IF ID EX MEM WB IF ID ID EX ID MEM ID WB 67 Resolving RAW Dependence with Forwarding Instructions IA and IB (where IA comes before IB) have RAW dependence iff IB (R/I, LW, SW, Br or JR) reads a register written by IA (R/I or LW) dist(IA, IB) dist(ID, WB) = 3 In other words, if IB in ID stage reads a register written by IA in EX, MEM or WB stage, then the operand required by IB is not yet in RF retrieve operand from datapath instead of the RF retrieve operand from the youngest definition if multiple definitions are outstanding 68 Data Forwarding Paths (v1) a. No forwarding ID/EX EX/MEM MEM/WB dist(i,j)=3 M u x Registers ForwardA M u x internal forward? Rs Rt Rt Rd ALU dist(i,j)=1 dist(i,j)=2 Data memory M u x ForwardB M u x EX/MEM.RegisterRd Forwarding unit MEM/WB.RegisterRd dist(i,j)=3 b. With forwarding [Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 69 Data Forwarding Paths (v2) a. No forwarding ID/EX EX/MEM MEM/WB dist(i,j)=3 M u x Registers ForwardA M u x Rs Rt Rt Rd ALU dist(i,j)=1 dist(i,j)=2 Data memory M u x ForwardB M u x EX/MEM.RegisterRd Forwarding unit MEM/WB.RegisterRd b. With forwarding [Based on original figure from P&H CO&D, COPYRIGHT 2004 Elsevier. ALL RIGHTS RESERVED.] 70 Assumes RF forwards internally Data Forwarding Logic (for v2) if (rsEX!=0) && (rsEX==destMEM) && RegWriteMEM then forward operand from MEM stage // dist=1 else if (rsEX!=0) && (rsEX==destWB) && RegWriteWB then forward operand from WB stage // dist=2 else use AEX (operand from register file) // dist >= 3 Ordering matters!! Must check youngest match first Why doesn’t use_rs( ) appear in the forwarding logic? What does the above not take into account? 71 Data Forwarding (Dependence Analysis) R/I-Type LW SW Br J Jr IF ID EX MEM use use produce use use produce (use) use WB Even with data-forwarding, RAW dependence on an immediately preceding LW instruction requires a stall 72 Sample Assembly, Revisited (P&H) for (j=i-1; j>=0 && v[j] > v[j+1]; j-=1) { ...... } addi $s1, $s0, -1 for2tst: slti $t0, $s1, 0 bne $t0, $zero, exit2 sll $t1, $s1, 2 add $t2, $a0, $t1 lw $t3, 0($t2) lw $t4, 4($t2) nop slt $t0, $t4, $t3 beq $t0, $zero, exit2 ......... addi $s1, $s1, -1 j for2tst exit2: 73 Pipelining the LC-3b 74 Pipelining the LC-3b Let’s remember the single-bus datapath We’ll divide it into 5 stages Fetch Decode/RF Access Address Generation/Execute Memory Store Result Conservative handling of data and control dependences Stall on branch Stall on flow dependence 75 An Example LC-3b Pipeline 77 78 79 80 81 82 Control of the LC-3b Pipeline Three types of control signals Datapath Control Signals Control Store Signals Control signals that control the operation of the datapath Control signals (microinstructions) stored in control store to be used in pipelined datapath (can be propagated to stages later than decode) Stall Signals Ensure the pipeline operates correctly in the presence of dependencies 83 84 Control Store in a Pipelined Machine 85 Stall Signals Pipeline stall: Pipeline does not move because an operation in a stage cannot complete Stall Signals: Ensure the pipeline operates correctly in the presence of such an operation Why could an operation in a stage not complete? 86