I n t e l ® 5 5 2 0 Ch ipse t : An I / O H u b
Ch ipse t for Se r ve r , W or k st a t ion , a n d
H igh En d D e sk t op
Debendra Das Sharm a
Principal Engineer, Digit al Ent erprise Group
I nt el Corporat ion
1
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Con t r ibu t or s
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
2
Nilesh Bhagat
Rob Blankenship
Celest e Brown
Sam Chiang
Ken Cret a
Debendra Das Sharm a
Hanh Hoang
Siva Gadey Prasad
S. Jayakrishna
Michelle Jen
Daniel Joe
Chandra P Joshi
Lily Looi
Dean Mulla
Sridhar Mut hrasanallur
K. Pat t abhiram an
Guru Raj am ani
Bill Rash
Aquiles Saenz
Raj esh Sankaran
Miles Schwart z
Mark R Swanson
Pat rick Tsui
Cyprian Woo
Robin Zhang
and m any ot hers..
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Age n da
• Plat form Overview
• Feat ure Set
• Micro- archit ect ure and Transact ion Flows
• Perform ance
• Chip St at ist ics
• Sum m ary
3
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Se r ve r Ch ipse t Evolu t ion
CPU
CPU
CPU
CPU
CPU
CPU
FSB
Mem & I/O
Controller
Scalable
Expander Bus
South
Bridge
PXB
PXB
450 GX
(~95)
E8500
(Twincastle*)
(~05)
4
Enterprise
South
Bridge
CPU
CPU
5520 IOH
ESI
PCIe 1.0a*
Legacy I/Os
5000P
(Blackford*)
(~06)
Memory
F BD
Mem & I/O
Controller
ESI
I/O Hub
E8870*
(~01)
CPU
FSB
PCIe 1.0a*
Port
450 NX
(~98)
CPU
Mem & I/O
Controller
South
Bridge
South
Bridge
PCIe 1.0a*
PCIe
to PCI
Bridge
PCI/PCI-X*
PCI2.1*
PCI/PCI-X*
PCI-(X)
Bridge
DMH
PCI/PCI-X*
Memory
Scalable
Node
Controller
Memory
FSB
DMH
Memory
FSB
South
Bridge
Seaburg Clarksboro
(~07)
(~07)
PCIe 2.0*
5520
(~09)
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Plumas
(~02)
E7520
(~04)
Trends:
- Increased integration => fewer chips
- Increased bandwidth for coherent
traffic: FSB -> multiple FSBs -> Links
-Increased memory capacity and
bandwidth (memory buses -> FBD ->
Integrated memory controller)
-Increased I/O connectivity and
bandwidth: PCI bus /bridges to PCIe* links
(FSB: Front Side Bus. *: Published in Hotchips)
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Pla t for m Ove r vie w
CPU 0
CPU 1
• Platform Transition:
QPI
(x20)
– Front Side Bus to Intel® Quick Path
Interconnect Link
QPI
(x20)
(x20)
Intel®
5520
Chipset
x4
x8
ESI
x16 x8
JTAG, SMBus, etc(x4, 2.5G)
12 USB 2.0
6 x1 PCIe*
6 SATA
– Memory controller integrated to CPU
x8
x16
x8
x2
x2
x4
x4
x4
x4
x4
x4
x4
x4
36 PCIe* 2.0 Lanes
QPI
• Intel® 5520 Chipset is an I/O Hub
South Bridge
LPC
Intel® HD Audio
Other
ports
(2 CPU Sockets, Single IOH System)
5
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
– Bridge between QPI and I/O
– First server chipset with PCIe*2.0
– Flexible I/O with 36 PCIe* 2.0 Lanes (3
to 10 Root ports of different widths)
– One or two CPU socket connection
– Server and Workstation platforms
– Customized to High-End Desktop (X58)
– Multi-generational CPU upgrades
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
D u a l I OH Pla t for m
CPU 0
• Increased I/O Connectivity
CPU 1
QPI
– 72 PCIe* 2.0 Lanes
(x20)
– Maintains I/O flexibility
QPI
QPI
(x20)
(x20)
Intel®
5520
Chipset
QPI+
(x20)
JTAG/
SMBus etc..
ESI
6 x1 PCIe*
12 USB 2.0
6 SATA
PCIe* 2.0
(36 Lanes)
South Bridge
– Workstation and server
Intel® • Two IOHs coordinate their
accesses to appear as a
5520
Chipset single IOH to the CPU(s)
PCIe* 2.0
(36 Lanes)
LPC
Intel® HD Audio
Other
ports
(2 Sockets, Dual IOH System)
6
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
– Enhanced QPI protocol in the
IOH-IOH Link (e.g. Recall
cache lines from other IOH to
ensure forward progress)
– Dynamic adjustment of CPU
HOME resources (trackers)
to ensure performance
• One or two CPU sockets
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
I n t e l ® Qu ick Pa t h I n t e r con n e ct ( QPI )
•
•
•
•
•
Two QPI links, each connect ing t o a CPU socket ( or I OH)
Different ial links wit h forwarded clock
Support s up t o 17” channel lengt h wit h t wo connect ors
Each QPI is 20 bit s wide, up t o 6.4 GT/ s Dat a Rat e
51.2 GB/ s raw bandwidt h and 31.5 GB/ s of sust ained real dat a
t ransfer wit h t wo QPI Links at 6.4 GT/ s
• CRC prot ect ion on every 80 bit s along wit h Link level ret ry
• Unordered fabric for perform ance
• Snoopless I OH ensures CPU does not snoop I OH
– QPI connect ing I OH only used for I / O accesses
– I OH does not rem ain in snoop pat h of CPU t hrough upgrades
7
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
I n t e l ® 5 5 2 0 I / O Vir t u a liza t ion Fe a t u r e s
• First server chipset shipping wit h PCI e* I / O Virt ualizat ion
CPU
CPU
• All inbound m em ory request s undergo VT- d ( I OMMU)
address t ranslat ion and access privilege check
5520 IOH
PCIe 2.0*
ESI
South
Bridge
• Translat ion based on t he request or’s BDF < Bus, Device,
Funct ion> and page address
P1
P3
0 1 2
0 1 2
P2
0 1 2
[Assigned I/O for VMs:
VM1: (P1,0), (P2,2)
VM2: (P1, 1), (P2, 0), P3
VM3: (P1, 2), (P2, 1)]
8
– Mult iple VMM vendor ( Xen, KVM, Cit rix, Parallels, VMWare,
et c) product s enabled
– Each BDF m ay belong t o a different virt ual m achine ( VM)
• Cont ext Ent ry ( 2- level) and a 4- level address t ranslat ion
st ruct ure resident in m em ory
• Mult i- level caching st ruct ures inside I OH
– Cont ext ent ry cache, L1/ L2, L3 and I OTLB caches
– I nvalidat ion: Regist er- based and Queue- based from m em ory
• Support s PCI - SI G defined ATS/ ACS for end- point caching
and access right s check
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Re lia bilit y, Ava ila bilit y, Se r vice a bilit y ( RAS)
• High speed int erfaces ( ESI , PCI e* , QPI ) are CRC prot ect ed
wit h Link level ret ry
• Poison support t hroughout ( PCI e* , ESI , QPI , int ernal pat hs)
• I nt ernal dat a pat h m ost ly ECC/ CRC prot ect ed
– Configurat ion regist ers are parit y prot ect ed
• Det ailed error logs in each int erface for each error t ype
• Advanced Error Report ing St ruct ures
– Bot h MSI - based as well as int errupt pin based not ificat ion
• Hot - plug support on all PCI e* Links
• Lane degradat ion and reversal support on PCI e*
• Live Error Recovery for guarant eed error cont ainm ent
– Can program each error t ype in t o t ake t he PCI e* link down
– Exam ple: I f a device can not handle poison dat a, we can t ake
t hat Link down rat her t han propagat e poison t o t he device
9
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
I n t e l ® 5 5 2 0 Ch ipse t Fe a t u r e s
• I soch r on ou s support on ESI for Qualit y of Service
– Separat e Virt ual Channel for HD Audio ( lat ency & bandwidt h)
– Separat e Virt ual Channel for USB ( bandwidt h guarant ee)
• I OAPI C: I / O Advanced Program m able I nt errupt Cont roller
– Convert s legacy int errupt s t o Message Signaled I nt errupt
– Avoids int errupt sharing = > bet t er for perform ance
• Qu ick D a t a Acce le r a t ion for CPU off- load
– DMA Move Engine w/ CRC capabilit y: 5GB/ s of bandwidt h
– Eight funct ions for bet t er virt ualizat ion support
– Direct Cache Access t o CPU cores from PCI e*
• I n t e gr a t e d M a n a ge a bilit y En gin e for Syst em m anagem ent
– Em bedded m icrocont roller wit h encrypt ion engine
– Sideband pat hs t o com ponent s
– I nband PCI port s ( serial port , DMA, em ulat ed I DE, HECI )
– Separat e power well
10
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
I n t e l ® 5 5 2 0 I OH Block D ia gr a m
QPI Link
(x20, up to 6.4 GT/s)
QPI Link
(x20, up to 6.4 GT/s)
QPI AFE
QPI AFE
QPI PL + LL
QPI PL + LL
QPI Protocol Layer with cache
QPI Protocol Layer with cache
JTAG
Cfg ring
IO APIC
Hot Plug
Central Data Path Packet Processing/Routing
VT-d
400 Mhz
(QPI Freq
Div 16)
ESI/PCIe* Transaction Layer Inbound/Outbound Queues
ESI
TL Pkt Proc
ESI
PL + LL
PCIe* PL + LL
ESI/PCIe*
AFE
PCIe* AFE
ESI (x4)
11
PCIe* TL
PCIe* TL
Pkt Processing Pkt Processing
PCIe* x4
(2 x2)
PCIe* PL + LL
PCIe* AFE
PCIe* TL
Pkt Processing
Mgmt
Engine
(267 MHz)
CB3
@400Mhz
250 Mhz
PCIe* PL + LL
PCIe* AFE
PCIe* x16
(2 x8, 4 x4)
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
PCIe* x16
(2 x8, 4 x4)
Hdr Bus + 16B Data Bus +
Check bits + Control
(Internal Datapath B/W: 51.2 GB/s)
[PL: Physical Layer, LL: Link Layer,
TL: Transaction Layer, Pkt: Packet,
AFE: Analog Front End,
QPI: Quick Path Interconnect Link]
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Tr a n sa ct ion Flow Ex a m ple : D M A Re a d
• 1. Mem ory Read ( MRd) loaded t o Queue
6. Snoop Response
CPU 0
QPI
QPI
7. Data
(Tracker 4. Read Request
Release) (Traker consumed)
QPI 0
IOH
CPU 1
QPI
5. Snoop
• 3. Ordering check. Packet broken t o Cache
line( s) . Request sent t o QPI 0 ( hom e in CPU0)
• 4. QPI 0: Conflict check; Check t rackers;
Consum e t racker; Send request t o CPU0
QPI 1
3. Cache Line Request
• 5. QPI 1 sends snoop request t o CPU 1
CDP
8. Data
Completion
PCIe*
Queues
2. VTd Access
• 6. CPU 1 sends snoop response t o CPU0
1. MRd
ESI/ PCIe*
12
• 2. VTd t ranslat ion and access right check
• 7. CPU 0 sends Dat a Ret urn t o I OH. QPI 0
releases t he t racker on receipt of Dat a Ret urn
• 8. Dat a loaded t o out bound PCI e* queue
9. Data
Completion
• 9. Dat a com plet ion sent out on PCI e*
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
D M A W r it e : Re qu e st for Ow n e r sh ip
6. Snoop Response
CPU 0
QPI
CPU 1
QPI
QPI
4. Request for
Ownership (tracker)
QPI 0
• 1. Mem ory Writ e ( MWr) loaded t o queue
• 2. VTd t ranslat ion and access right check.
Page Walk on a m iss.
5. Snoop
• 3. Packet broken t o Cache line( s) . Request
for Ownership ( RFO) sent t o QPI 0 ( hom e in
CPU0) . No Ordering check t o pipeline RFOs
QPI 1
3. Cache Line Request
IOH
CDP
2. VTd Access
PCIe*
Queues
• 5. QPI 1 sends snoop request t o CPU 1
1. MWr
ESI/ PCIe*
13
• 4. QPI 0: Conflict check; Check t rackers;
Consum e t racker; Send request t o CPU0
• 6. CPU 1 sends snoop response t o CPU0
• 7. CPU 0 ret urns t he ( Exclusive) Ownership
of t he Cache Line ( wit hout Dat a) t o I OH
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
D M A W r it e : W r it e ba ck Ph a se
• 8. QPI 0: ownership not ificat ion t o CDP
so t hat it can process DMA Writ e
CPU 0
QPI
CPU 1
QPI
QPI
11. Data and
Relinquish Ownership
QPI 0
8. Cache
ownership
notification
passed
nn to CDP
QPI 1
10. Write Data
(Complete Txn on
CDP PCIe*)
PCIe*
Queues
9. Wait till
Cache line
At Head of
P Queue
IOH
ESI/ PCIe*
14
• 9. CDP wait s t ill t he post ed t ransact ion
get s t o t he t op of t he post ed queue, per
PCI e* Ordering rules
• 10. CDP: Check wit h QPI t o ownership
st ill t here; perform writ e if t here; else
request line again
• 11. QPI 0 perform s Writ eback of Dat a
and relinquishes ownership
• 12. CPU 0 sends com plet ion for t he
Writ eback Transact ion. Tracker released
for subsequent reuse
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Pe r for m a n ce
Configurat ion ( Single I OH)
100% Rd
( GB/ s)
100% Wr
( GB/ s)
50- 50 Rd/
Wr ( GB/ s)
NUMA w/ 3 PCI e 2.0* cards ( 2 x16, 1 x4)
15.9
12.4
15.2
NUMA w/ 4 x8 PCI e 2.0* Cards
13.9
12.3
14.7
I nt erleaved w/ 4 x8 PCI e 2.0* Cards
14.1
12.1
14.5
• Measured result s on single I OH at launch. More t han 2X previous
generat ions due t o QPI as well as PCI e 2.0*
• 100% Rd B/ W PCI e* lim it ed
• 100% Wr and 50- 50 RW B/ W is t racker ent ry lim it ed
– Writ es occupy t racker ent ry longer since t here are t wo round- t rips on QPI Link
– Bandwidt h expect ed t o scale wit h com put e capabilit y in subsequent CPU
generat ions due t o m ore t racker ent ries from CPU
• Ot her det ails: I O Met er benchm ark , 2.93 GHz 5500, QPI at 6.4 GT/ s, 1333MHz
RDI MM DDR3( 6 x2 GB, 2 x8 channel) , Request Size: 4KB, Max payload: 256B
15
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Pow e r
• PCI e* and ESI :
– Act ive St at e Power Managem ent put s idle link t o low power L1 st at e
– PCI * Power Managem ent m echanism s t o allow syst em soft ware t o
m anage Syst em Sleep st at es ( ent ry as well as exit )
• QPI : Support for low power L1 st at e on idle link
• Several power savings m easures in t he design ( e.g., fine- grain
and coarse- grain clock gat ing)
• Syst em wide sleep st at e orchest rat ed wit h Sout h Bridge
• Power num bers:
– TDP: 27.1 W
–
–
–
–
All Links working on full speed ( e.g., QPI at 6.4 GT/ s and PCI e* at 5 GT/ s)
All feat ures and int ernal devices enabled
No act ive st at e power m anagem ent benefit assum ed
Account s for worst possible com binat ion of process, volt age, t em perat ure
– I dle power: 10 W ( t hrough syst em low power st at e)
16
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Ch ip St a t ist ics
• 65 nm process
t echnology
• Die Size: 13.6 m m
X 10.4 m m
• ~ 100 M t ransist ors
– 33x original
Pent ium
• Package: FCBGA
37.5 x 37.5 m m ,
1.067 pin pit ch, 10
layer
• Signal Pins: 570;
t ot al pins on
package: 1295
17
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009
Su m m a r y
• I nt el ® 5520 is first QPI - based chipset wit h PCI e* 2.0
• Leadership feat ures
– I / O bandwidt h wit h flexible I / O Connect ivit y ( 36 or 72 PCI e
2.0* Lanes) for various segm ent s
– I / O Virt ualizat ion
– QuickDat a for I / O Accelerat ion
– Manageabilit y
– I sochrony for Qualit y of Service
• Designed t o last m ult iple CPU generat ions on t he
sam e plat form t o prot ect cust om er invest m ent s
18
* Ot her nam es and brands m ay be claim ed as t he propert y of ot hers.
Copyright © 2009, I nt el Corporat ion. Hot chips 2009