Digital System Clocking: High-Performance and Low-Power Aspects Vojin G. Oklobdzija, Vladimir M. Stojanovic, Dejan M. Markovic, Nikola M. Nedovic Chapter 9: Microprocessor Examples Wiley-Interscience and IEEE Press, January 2003 Microprocessor Examples • Clocking for Intel® Microprocessors • IA-32 Pentium® Pro • First IA-64 Microprocessor • Pentium 4 • Sun Microsystems UltraSPARC-III® Clocking • Clocking and CSEs • Alpha® Clocking: A Historical Overview • Clocking and CSEs • IBM® Microprocessors • Level-Sensitive Scan Design • Examples of CSEs Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 2 Microprocessor Examples • Clocking for Intel® Microprocessors • IA-32 Pentium® Pro • First IA-64 Microprocessor • Pentium 4 • Sun Microsystems UltraSPARC-III® Clocking • Clocking and CSEs • Alpha® Clocking: A Historical Overview • Clocking and CSEs • IBM® Microprocessors • Level-Sensitive Scan Design • Examples of CSEs Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 3 Intel® Microprocessor Features Pentium II Pentium III Pentium 4 MPR Issue June 1997 April 2000 Dec 2001 Clock Speed 266 MHz 1GHz 2GHz Pipeline Stages 12/14 12/14 22/24 Transistors 7.5M 24M 42M 16k/16K/- 16K/16K/256K 12K/8K/256K 203mm2 106mm2 217mm2 IC Process 0.28mm, 4M 0.18mm, 6M 0.18mm, 6M Max Power 27W 23W 67W Cache (I/D/L2) Die Size Source: Microprocessor Report Journal Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 4 IA-32 Pentium® Pro Ext Clk CLK Gen FB Clk Delay Line Delay SR Delay Line Deskew Control PD Right Spine Left Spine Core Delay SR Clock distribution network with deskewing circuit (Geannopoulos and Dai 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 5 Adaptive Deskewing Technique • Equalization of two clock distribution spines by compensating for delay mismatch • Delay lines • Phase detector • Controller • Result: global clock skew of only 15ps • 0.25mm technology • 7.5M transistors Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 6 IA-32 Pentium® Pro In Out Load<1:15,2> Delay Line Load<0:14,2> <1:15,2> <0:14,2> Delay Shift Register Delay shift register (Geannopoulos and Dai 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 7 IA-32 Pentium® Pro Right Clk Bandwidth Control Left Clk Delay = Delay = n Left Leads Phase Detector 1 n Right Leads Phase Detector 2 Phase detector (Geannopoulos and Dai 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 8 First IA-64 Microprocessor PLL RCDs Deskew Cluster Core Clock PLL Reference Clock Clock distribution topology (Rusu and Tam 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 9 Programmable Deskew Units • Strategy similar to that in IA-32 • External differential clock • System bus frequency • PLL generates internal clock • 2x frequency • Clock distribution architecture • Balanced global clock tree • Multiple deskew buffers • Multiple local clock buffers Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 10 First IA-64 Microprocessor RCD Deskew Buffer Regional Clock Grid Global Clock TAP Interface Reference Clock Phase Detector Digital Filter Control FSM Deskew Settings RCD Regional Feedback Clock Deskew buffer architecture (Rusu and Tam 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 11 First IA-64 Microprocessor Input Output Enable Delay Control Register Digitally controlled delay line (Rusu and Tam 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 12 First IA-64 Microprocessor Simulated regional clock-grid skew (Rusu and Tam 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 13 First IA-64 Microprocessor Measured regional clock skew (Rusu and Tam 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 14 Pentium® 4 1x-Clk enable clock enable distribution & sync clock enable generator clock enable distribution & sync MACRO addr. bus outbound clocks MACRO Core PLL bus clock 2x-Clk enables core Clk distribution bus clock# I/O PLL I/O data Clk distribution core clock data bus outbound clocks I/O feedback clock divide by 4 core clock outbound deskew state machine data from core D data data clock MSFF data to core Q D inbound buffers MSFF core clock input buffer inbound latching clocks inbound clocks gen state machine Nov. 14, 2003 Q strobe glitch protection and detection input buffers strobes Core and I/O clock generation (Kurd et al. 2001), Copyright © 2001 IEEE Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 15 Multi-GHz Clock Network in Pentium 4 • Three core and three I/O frequencies (total 6 frequencies running concurrently) • Differential off-chip reference clock • PLL synthesizes core and I/O clocks • Global core clock distribution • 47 independent clock domains • Each domain has 5-bit deskew control register • Clock skew < 20ps Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 16 Pentium® 4 To Test Access Port 3 3-stage binary tree of clock repeaters PLL Domain Buffer 1 Local Clock Macro Phase Detector Domain Buffer 2 Local Clock Macro Phase Detector Domain Buffer 3 Local Clock Macro Sequential Elements Sequential Elements Sequential Elements Phase Detector Domain Buffer 46 Local Clock Macro Sequential Elements Local Clock Macro Sequential Elements Phase Detector Domain Buffer 47 Nov. 14, 2003 Logical diagram of core clock distribution (Kurd et al. 2001), Copyright © 2001 IEEE Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 17 Pentium® 4 Stretch 1 Stretch 0 Enable 1 Stretch 1 Stretch 0 Enable 2 Adjustable Delay Buffer Gclk Enable 1 Stretch 1 Stretch 0 Enable 2 ClkBuf Type 1 Gclk Stretch 1 medium freq. pulse clk phase 1 Enable 2 Stretch 0 Enable 1 SlowClkSync Gclk Gclk Stretch 1 Stretch 0 Enable 1 Enable 2 ClkBuf Type 1 Gclk Gclk ClkBuf Type 3 Stretch 0 slow freq. pulse clk phase 1 Enable ClkBuf Type 1 Stretch 1 medium freq. pulse clk phase 2 Enable 1 Adjustable Delay Buffer Enable 1 medium freq. normal clk phase 1 fast freq. pulse clk Enable 2 Gclk ClkBuf Type 2 Example of local clock buffers generating various frequency, phase and types of clocks (Kurd et al. 2001), Copyright © 2001 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 18 Intel Clocking: Summary • Increasing clock speeds and die size • Balancing the clock skew in large designs using simple RC trees is becoming less effective • Insertion delay 7-8FO4 due to increased die • Comparable to the clock period • Clock skew control has been getting harder to due to increased PVT variations • Inductive effects at multi-GHz rates • Use of active deskewing circuits Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 19 Microprocessor Examples • Clocking for Intel® Microprocessors • IA-32 Pentium® Pro • First IA-64 Microprocessor • Pentium 4 • Sun Microsystems UltraSPARC-III® Clocking • Clocking and CSEs • Alpha® Clocking: A Historical Overview • Clocking and CSEs • IBM® Microprocessors • Level-Sensitive Scan Design • Examples of CSEs Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 20 UltraSPARC® Family Characteristics UltraSPARC-I® UltraSPARC-II® UltraSPARC-III® Year 1995 1997 2000 Architecture SPARC V9, 4-issue SPARC V9, 4-issue SPARC V9, 4-issue Die size 17.7x17.8mm2 12.5x12.5mm2 15x15.5mm2 # of transistors 5.2M 5.4M 23M Clock Frequency 167MHz 330MHz 1GHz Supply voltage 3.3V 2.5V 1.6V Process 0.5mm CMOS 0.35mm CMOS 0.15mm CMOS Metal layers 4 (Al) 5 (Al) 7 (Al) Power consumption <30W <30W <80W Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 21 UltraSPARC-III: Clocking • Performance-driven high-power clock distribution • Eight logic gates per cycle • High-speed semi-dynamic flip-flops with logic embedding • Large hold time mandates use of advanced tools for fixing fast-path violations Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 22 UltraSPARC-III®: Clocking Clock distribution delay in UltraSPARC-III (Heald et al. 2000), Copyright © 2000 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 23 UltraSPARC-III: Clock Storage Elements Vdd Vdd MP1 S NAND MN2 MN1 Q Inv4 MN3 D MP2 Q MN5 Clk1 Inv2 Inv5 Inv6 Inv3 Inv1 MN4 Clk Semidynamic flip-flop (Klass 1998), Copyright © 1998 IEEE • Single-ended dynamic structure with use of keepers for static operation and use of clock pulsing • Positive feedback (NAND) improves low-to-high setup time • Fast, at the price of high internal and clock power Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 24 UltraSPARC-III: Clock Storage Elements Vdd M P1 Vdd M P1 M P2 Q Inv 4 M N3 S N AN D S D1 Inv 3 NMOS netw ork DN M N5 M N3 Inv 5 Q N AN D D2 Vdd Vdd Inv 6 M N 2a M N4 C lk Inv 1 D1 Inv 2 D2 M N5 Q Inv 6 M N4 M N1 Inv 1 Logic embedding in a semi-dynamic flip-flop Inv 3 Inv 5 M N 2d D2 M N1 C lk Q M N 2c D1 M N 2b M P2 Inv 4 Inv 2 Two-input XOR function (Klass, 1998), Copyright © 1998 IEEE • A non-inverting logic function can be embedded by replacing the input D transistor with an n-MOS logic network • Necessary for fitting 8 logic stages in cycle time, also used for scan • Complexity of embedded logic limited by the n-MOS stack depth Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 25 UltraSPARC-III: Clock Storage Elements Vdd Vdd M P1 Q S Q Inv 4 M N3 M N6 Inv 3 D C lk Inv 6 Q M N5 Inv 1-2 M N2 M P3 R M N3 N AN D D M P4 Inv 5 Inv 5 S Vdd M P2 M P1 Inv 3-4 M N2 M N4 M N7 D C lk M N1 Inv 1 M N1 Inv 2 Single-ended dynamic SDFF Differential dynamic SDFF (Klass, 1998), Copyright © 1998 IEEE • Dynamic version of SDFF used in dynamic logic paths • Outputs exercise precharge-evaluate sequence to ensure monotonicity Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 26 UltraSPARC-III: Clock Storage Elements Vdd MP3 Vdd D Vdd Vdd MP1 MP4 MP6 Vdd MP2 Inv5 S Q MN3 MN2 MN1 Inv2 Q MN6 NAND D MP7 MP5 MN7 MN4 Inv4 Inv3 Inv1 MN5 Clk UltraSPARC-III flip-flop (Heald et al. 2000), Copyright © 2000 IEEE • Final UltraSPARC-III flip-flop modified by decoupling keepers to increase immunity to -particles • Somewhat degraded speed and logic embedding property Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 27 Microprocessor Examples • Clocking for Intel® Microprocessors • IA-32 Pentium® Pro • First IA-64 Microprocessor • Pentium 4 • Sun Microsystems UltraSPARC-III® Clocking • Clocking and CSEs • Alpha® Clocking: A Historical Overview • Clocking and CSEs • IBM® Microprocessors • Level-Sensitive Scan Design • Examples of CSEs Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 28 Alpha® Microprocessor Features 21064 21164 21264 21364 1.68 9.3 15.2 152 16.8x13.9 18.1x16.5 16.7x18.8 21.1x18.8 0.75mm 0.5mm 0.35mm 0.18mm Supply [V] 3.3 3.3 2.2 1.5 Power [W] 30 50 72 125 Clk Freq. [MHz] 200 300 600 1200 Gates/Cycle 16 14 12 12 # transistors [M] Die Size [mm2] Process Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 29 Alpha® Microprocessors: Clocking clock grid (a) (b) (c) Alpha microprocessor final clock driver location: (a) 21064, (b) 21164, (c) 21264 Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 30 Alpha® Microprocessors: Clocking 21064 clock skew (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 31 Alpha® Microprocessors: Clocking 21164 clock skew (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 32 Alpha® Microprocessors: Clocking D D Clk Clk D local clk GCLK Grid ext. clk Clk D Box Clk Grid PLL Clk local clk D Clk D cond cond. local clk Clk cond cond. local clk 21264 clock hierarchy (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 33 Alpha® Microprocessors: Clocking 21264 clock skew (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 34 Alpha® Microprocessors: Clocking NCLK DLL DLL DLL GCLK grid L2LClk L2RClk 21364 major clock domains (Xanthopoulos et al. 2001), Copyright © 2001 Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 35 Alpha® Microprocessors: Clocking 21364, NCLK clock skew (Xanthopoulos et al. 2001), Copyright © 2001 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 36 Alpha® µP: Clock Storage Elements P1 P5 D P1 D X P2 N3 Clk N4 N1 N2 Q Clk P2 P3 X N1 P4 Q N2 N5 21064 modified TSPC latches (Gronowski et al. 1998), Copyright © 1998 Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 37 Alpha® µP: Clock Storage Elements X D Clk (a) Q X D Q Clk (b) 21164: (a) phase-A latch, (b) phase-B latch (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 38 Alpha® µP: Clock Storage Elements X1 D1 D2 D1 X D2 Clk Q Q Clk D3 D4 X2 Clk (a) (b) Embedding of logic into a latch: (a) 21064 TSPC latch, one level of logic; (b) 21164 latch, two levels of logic. (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 39 Alpha® µP: Clock Storage Elements Q Q Clk D 21264 flip-flop (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 40 Alpha® Microprocessors: Timing Logic D Q D Q R D GCLK Critical Path Definition and Criteria - Identify common clock, D and R - Maximize D - Minimize R D+UR Tcycle Logic D Q D R GCLK cond Race Definition and Criteria - Identify common clock, D and R - Minimize D - Maximize R D R+H Critical-path and race analysis for clock buffering and conditioning (Gronowski et al. 1998), Copyright © 1998 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 41 Microprocessor Examples • Clocking for Intel® Microprocessors • IA-32 Pentium® Pro • First IA-64 Microprocessor • Pentium 4 • Sun Microsystems UltraSPARC-III® Clocking • Clocking and CSEs • Alpha® Clocking: A Historical Overview • Clocking and CSEs • IBM® Microprocessors • Level-Sensitive Scan Design • Examples of CSEs Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 42 Hazard-Free Level-Sensitive Polarity-Hold Latch +Clock Data Out -Clock Eichelberger 1983 Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 43 General LSSD Configuration Inputs (X) Outputs (Y) Combinational Logic Y=Y(X, Sn ) Clocked Storage Elements Scan-Out Clock Present State Sn Scan-In Nov. 14, 2003 Scan-Out Next State Sn+1 Sn+1 = f {Sn , X} Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 44 LSSD Shift Register Latch L1 Latch -Scan_In -L1 +L1 L2 Latch -Data -L2 +L2 +A Clk -C Clk +B Clk Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 45 LSSD Double Latch Design State Sn Primary Outputs Z X1 L1 L2 L1 L2 X2 Primary Inputs X Combinational Logic Sn X3 L1 L2 L1 L2 Xn C1 A Shift Scan In B Shift or Scan In Nov. 14, 2003 Scan Out Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 46 IBM® S/390 Parallel Server Processor B_CLK A_CLK CLKG L1 L2 SCAN_IN CLKL Q CLK_ENABLE (SCAN_OUT) CLKG IN_A SELECT_N IN_B SELECT_A CLKL TEST_DISABLE LSSD SRL with multiplexer used in the IBM S/390 G4 processor (Sigal et al. 1997), reproduced by permission Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 47 IBM® S/390 Parallel Server Processor B_CLK A_CLK Q SCAN_IN mux_A IN_A IN_B IN_C Q (SCAN_OUT) mux_M_N IN_M IN_N CLKL SELECT_N SELECT_A TEST_DISABLE Static multiplexer version of the SRL used in the IBM S/390 G4 (Sigal et al. 1997), reproduced by permission Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 48 A_CLK CLKG SCAN_IN C1 IN L1 L2 Q (SCAN_OUT) IBM® S/390 Parallel Server Processor C2 B_CLK CLKG C2_ENABLE C2 C1_DISABLE C1 A clocked storage element is used in the non-timing-critical timing macros of the IBM S/390 G4 processor (Sigal et al. 1997), reproduced by permission Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 49 IBM® S/390 Parallel Server Processor CLKG C1 C2 B_CLK CLKG C2_ENABLE UNOVERLAP C1_DISABLE C2 C1 The clock-generation element used to detect problems created with fast paths: IBM S/390 G4 processor (Sigal et al. 1997), reproduced by permission Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 50 IBM® PowerPC Processor SCAN_GATE SG SEL_EXTi SELi NCLK CLK (a) OT SEL0 CLK SELn-1 SO D0 Dn-1 CLK True Mux CLK Slave Latch OC SEL0 SELn-1 SR Master Latch Complement Mux (b) The experimental IBM PowerPC processor (Silberman et al. 1998), reproduced by permission Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 51 IBM® PowerPC 603: Master-Slave Latch VDD ACLK SCANin C2 ACLK C1 C2 Dout Din C1 C2 ACLK The PowerPC 603 MSL (Gerosa et al. 1994), Copyright © 1994 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 52 IBM® PowerPC 603: Local Clk Generator C1_FREEZE C1_TEST SCAN_C1 GCLK ACLK C1 WAITCLK OVERRIDE C2 C2_TEST C2_FREEZE The PowerPC 603 local clock regenerator (Gerosa et al. 1994), Copyright © 1994 IEEE Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 53 Summary • Intel® Microprocessors • Active clock deskewing in Pentium® processors • Sun Microsystems® Processors • Semidynamic flip-flop (one of the fastest single-ended flip-flops today, “soft-edge”) • Alpha® Processors • Performance leader in the ‘90s • Incorporating logic into CSEs • IBM® Processors • Design for testability techniques • Low-power champion PowerPC 603 Nov. 14, 2003 Digital System Clocking: Oklobdzija, Stojanovic, Markovic, Nedovic 54