Lab Board
A description of the board you
will do measurements on
12/4/2002
2
Lab Board Description
 Code name is ‘King’s Canyon’
Referred to hereafter as ‘KC’
You will do measurements on this board in the lab
 Background for KC
Low-end server board
Inexpensive board for customers who want
servers
 Server
Computer which serves data to others
Supports more memory, more processing power,
and more I/O (i.e., network bandwidth, hard
drives) than desktop computers
More reliable (crashes less and recovers better)
than desktop computers
Introduction
12/4/2002
3
Server in a network
Shared
Printer
Server
Network Switch
`
`
PC
`
Desktop PC
PC
Introduction
12/4/2002
4
Inside the Server
Hard drive storage
2 CPUs
Plug in memory boards
Gigabytes of memory
MCH
Server Circuit
Board
Introduction
Network (Ethernet)
to network switch
12/4/2002
Circuit Board Block Diagram
CPU 0
CC
CPU 1
SS
P64H2 HI to
PCI bridge
SS
CC
4 PCI-X busses
SS
SS
MCH
Hub Interface
Memory
SS
Front
Side Bus
(FSB)
Memory
SS
CC
ICH
Video
CC
CC
P64H2 HI to
PCI bridge
Low speed 33MHz
PCI bus
12/4/2002
Details to Follow ….
SS – Source Synchronous
CC- Common Clock
Two (2) DDR
(Double Data
Rate) Memory
Busses
Components on KC
‘the chips’
12/4/2002
7
CPU
 CPUs –Central Processing Units
The brains of the computer that ‘run’ your program
Runs the Operating System (e.g. windows) and all the software
Speed of CPU has large effect on speed of system
Currently measured in GHz (e.g., 3.0 GHz)
Consumer pc’s typically have 1 cpu inside the box – called ‘UP’
KC has 2 cpus – this is called ‘DP’, i.e. dual processor
Some boards have 4+ cpus, they are called ‘MP’, i.e. multi-processor
KC uses Intel Xeon CPU’s
Currently at 3GHz
Similar to Pentium 4


Bigger cache (superfast memory inside CPU)
Can operate with other CPUs on same bus
The Silicon die (2) is under this metal heat
spreader (1). The die is significantly smaller
than the heat spreader. A ‘Fansink’ mounts
to the heat spreader to keep die cool.
Introduction
12/4/2002
8
MCH
 MCH – Memory Controller Hub
Helps CPU communicate with rest of system
The ‘gatekeeper’ to the outside world from perspective of
CPU
Talks to memory, CPU, and I/O

I/O means busses and chips and boards which ultimately connect
to things like hard drives, networks, keyboards, monitor, etc..
Maintains memory
tells it when to refresh, etc.. (dynamic memory has to be
refreshed or it forgets the data it holds)
Detects errors in what is stored in memory (ECC) and
periodically corrects these errors
KC uses Intel E7501 MCH
Actual Die: Passive (no fan)
Heatsink mounts on top.
Package decoupling
capacitors. Try to
hold voltage rails
(vcc) steady. Closer
to the die the better.
http://www.intel.com/design/chipsets/e7501/index.htm?iid=ipp_srvr_proc_xeon+e7501&
Introduction
12/4/2002
9
ICH
 ICH – Input/Output Controller Hub
I/O device that deals with critical but slow speed
devices




Video controller (basic 2-d video on servers, not meant for 3d graphics)
Talks to Keyboard, mouse via ‘Super IO’ chip
BIOS (contains software which computer runs when it’s first
powered on. BIOS gets computer up and running and then
passes control to the OS)
IDE Hard drive (slow hard drive that OS boots off of. The
big capacity hard drives do not connect to ICH)
Helps with initial bootup of system
Talks to mch over a slow version of the Hub Interface
bus
KC uses Intel 82801CA ICH
Still uses wire-bond technology.
Doesn’t require heatsink.
Introduction
12/4/2002
10
Memory Modules
 Memory – High speed storage place for code and data used by
CPU
Much faster than hard drive storage, but not as massive in
quantity
2-16 Gigabytes vs. 100’s of Gigabytes
Slower than CPU cache (small amount of memory internal to CPU)
Memory die’s are inside wire-bond packages called ‘DRAMs’
Dynamic Random Access Memory
Many DRAMs solder to a memory board called the ‘DIMM’
Dual In-line Memory Module
Anywhere from 9 to 36 DRAMs on one memory board
2 types of memory boards: Unbuffered and Registered. Unbuffered is
used in desktop. KC (and most servers) only uses Registered DIMMs.
Memory board plugs into KC board via a connector.
KC can support up to 8 DIMMs
Many different DIMM manufacturers
Micron, Samsung, Elpida,
Infineon, …
DRAM
http://www.micron.com/products/modules/ddrsdram/index.html
2 DIMMs
Introduction
12/4/2002
11
PCI-bridge
 P64H2 – PCIX 64-bit to Hub Interface 2 bridge
Gateway to High Speed I/O devices such as hard drive
arrays adapters or ethernet adapters
Talks to MCH on one side
Via high speed variey of ‘Hub Interface’ Bus
Talks to 2 separate pci-busses on other side
Several connectors connect to each pci bus
User can slide a card into each of these connectors
Typically Hard Disk Controller Cards


Each Provides several SCSI or SATA connection to hard drive
Can end up with quite a few hard drives and large storage
capacity
Or Network Adapater Cards


Provide several 100Mb/s ethernet connections, which go to a
switch or router
Servers tend to read data from the hard drive into memory,
process the data with the CPUs, and then send it out to the
requesting computer via the network (e.g., web server)
‘P64H2’ is the code name for the Intel 82870P2
Introduction
P64H2 die, may require
passive heatsink
12/4/2002
12
Identifying parts on the board
 CPUs
there are two of them and they have really huge fansinks.
 Memory
NOTE: If the fansink doesn’t run, and the system is on, the heatsink will get
VERY hot (burn skin) and system will eventually shut itself off.
memory consists of 8 wide connectors all in a row.
 MCH
At least 2 DIMMs must be plugged in (‘populated’) for system to boot up. There
are rules regarding what kind of dimms to use and which order to plug them into
the connectors.
It’s the chip between the cpus and the memory
Probably has a passive heatsink (a small heatsink w/ no fan)
 ICH & P64H2’s
Hard to find, but it looks like the chip pictured in the above slide
There is one ICH and 2 P64H2’s (second one not always populated, i.e.
sometimes it’s missing)
 PCI Slots
Larger connectors on opposite side of board from DIMM connectors.
One slot is from ICH and is slow speed legacy PCI bus (33MHz, 32-bit)
The rest are high speed (66-133MHz, 64-bit – i.e. 64 data bits wide)
Introduction
12/4/2002
Interconnects on KC
‘the wires that connect the
chips together’
12/4/2002
14
Front Side Bus (FSB)
 Function: CPUs talk to each other and to the MCH. This is how
the CPUs get data and code from memory, and communicate
with rest of the system.
 Technology: Source synchronous
533MBit/second/wire (266MHz signals)
This means each data wire on this bus can send 533 million bits of data
in one second
64 data bits wide
So the whole bus can transfer 64*533 million bits of data per second
Topology: Multi-drop bus. CPU1 has to listen to when CPU0
acceses memory because CPU1 may have that memory data in it’s
cache
‘Cache Coherency’ Each cpu has a small superfast memory inside it
called the ‘cpu cache’. The cache holds recently accessed memory
data. If one cpu holds the data for a memory address that another
cpu needs, it has to let it know not to get that data from main memory.
Termination: Parallel termination at both ends of bus (MCH and
CPU0)
Introduction
12/4/2002
FSB Topology
MCH
15
CPU #1
‘End Agent’
CPU #0
‘Middle Agent’
VTT = 1.5V
VTT = 1.5V
50 W
Package
Trace
Board Trace
Board Trace
Package
Trace
Package
Trace
50 W
INSIDE DIE
INSIDE DIE
INSIDE DIE
Note: no 50 ohm
termination
Introduction
12/4/2002
16
FSB Signal Groups
 Data Signal Group
HD[63:0]# – 64 bits of data
DINV[3:0]# - dynamic inversion
HDSTBP[3:0]# - P strobes
HDSTBN[3:0]# - N strobes
Example, HDSTBP[0]# & HDSTBN[0]# are used to strobe in HD[016]# & DINV[0]#

E.g., two strobe signals are used to clock in a set of data bits, more on this
later
 Address Signal Group
Switches at ½ the rate of the FSB Data Signals
HAB[35:3]# - Address Bits. E.g., what memory address is being
requested
HADSTB[1:0]# - Strobes for address signal group
HADSTB[0]# strobes A[16:3]#, HADSTB[1]# strobes the rest
 Common Clock Signals
There a whole lot of common clock signals
HCLKINP, HCLKINN – Common clock for all signals in the ‘Host
Clock Domain’, i.e. all common clock signals, and outer loop timings
for strobes (also called BCLK)
Introduction
12/4/2002
17
FSB Timings – 4X Source Synchronous
Timings not published for 533 MHz FSB,
400 MHz timing diagrams shown
Introduction
12/4/2002
18
FSB Timings – 2X Source Synchronous
Timings not published for 533 MHz FSB,
400 MHz timing diagrams shown
Introduction
12/4/2002
19
DDR (Double Data Rate) Memory Bus
 Function: MCH reads and writes data from memory over this
high speed bus.
 Signaling Technology: Source Synchronous
266Mbit/sec/wire (133Mhz signals)
A DDR bus is 64 data bits wide
KC’s MCH uses 2 DDR busses in parallel, so it likes having a 128-bit
wide bus

Bandwidth is 128*266 Million bits per second
Topology:Multi-Drop Bus.
Each of the two separate instances of this bus on this system go to
four (4) DIMM slots.

Each DIMM can have up to 2GBytes of memory, for a max total of 16GB.
Topology and Timings
The subject of the next class
Introduction
12/4/2002
Memory speed and size within a system
CPU
L1
L2
L2
L3
Faster Retrieval to CPU
CPU
L1
L3
MCH
PCI
BRIDGE
(P64H2)
MAIN
MEMORY
MAIN
MEMORY
MAIN
MEMORY
MAIN
MEMORY
Adapter
Card(s)
Greater Density
When we study
DDR signaling and
timings, it will be
clear the
overhead and
latency that main
memory exhibits,
and thus the need
for local L1-L3
CPU cache.
Hard Drives
Introduction
12/4/2002
20
21
HI (Hub Interface) Bus
 Function: Intel proprietary bus to provide flexible high speed
data transfer from MCH to various downstream components
 Signaling Technology: Source Synchronous
HI1: From MCH to ICH: 266Mbit/sec/wire (133Mhz signals)
8 data bits wide, bandwidth = 8*266 Million bits per second
HI2: From MCH to P64H2’s: 533Mbit/sec/wire (266Mhz signals)
16 data bits wide, bandwidth = 16*533 Million bits per second
Topology: Point to Point.
Termination: There are parallel terminations at both ends of the
bus
Introduction
12/4/2002
22
Hub Interface Topology
VCC=1.2
VCC=1.2
80 W
80 W
Package
Trace
Board Trace
50 W
Package
Trace
50 W
INSIDE DIE
INSIDE DIE
Introduction
12/4/2002
23
Hub Interface Signal Groups
 Data Bits
HI[15:0] – Data Signals
PSTRB_[1:0], PSTRB_[1:0]# – Strobe Signals
PSTRB_[0] & PSTRB_[0]# strobe in HI[0:7]
PSTRB_[1] & PSTRB_[1]# strobe in HI[15:8]
 Other
HI[18:16] – Command Signals
Common Clock signals
 GCLKIN
66MHz common clock for HI[18:16] and outer loops of the
PSTRB’s
 Timings not published for Hub Interface
Introduction
12/4/2002
24
PCI and PCI-X
 Function: Industry standard bus for plug-in peripherals
In servers, quite often plug-in hard drive adapters or network
adapters are connected to the pci bus
 Technology: Common Clock
PCI-X: From P64H2 – 133Mbit/sec/wire, clocks are 133MHz
signals, data is 66MHz
64 data bits wide, bandwidth = 64*133 Million bits per second
PCI: From ICH – 33Mbit/sec/sire, clocks are 33MHz, data is
16MHz
32 data bits wide, bandwidth = 32*33 Million bits per second
This speed and width are common in current desktop pc pci slots
Topology: Multi-Drop Bus.
The bus consists of devices soldered down on the board and slots for
plug-in devices. They all dangle off the PCI bus.
Termination:
Series terminated inside the transmitting agent.
This signaling technolgy depends on a doubling of the waveform at the
end of the bus (due to open-circuit) in order to function.
Introduction
12/4/2002
25
PCIX-133 2-slot Topology
INSIDE DIE
x
INSIDE DIE
x
x
Package
Trace
Package
Trace
Plug-in
Board Trace
Plug-in
Board Trace
Package
Trace
Board Trace
Board Trace
INSIDE DIE
Introduction
12/4/2002
26
PCI-X Signal Groups
 Data and Address use same signals:
AD[63:0]
Common Clocked
 Control Signals
DEVSEL, TRDY, … a bunch more
Common Clocked
 Asynchronous Signals
REQ, GNT – Arbitration signals
Interrupt Signals
 Clocks
PCLK[6:0] – goes to every device that hangs off the pci bus
including the bridge itself
Introduction
12/4/2002
27
PCI-X Data to Common Clock Timings
ns
ns
Introduction
12/4/2002
28
PCI-X Common Clock
Introduction
12/4/2002
29
KC Clock Distribution
14.31818MHz
DDR Bus B
Slots
66MHz
MCH
PCI to Gbit
Ethernet
CPU #1
133MHz
DDR Bus A
Slots
Differential Clock
CPU #2
133MHz
133 MHz
133MHz
Differential Clock
Differential Clock
CK 408
Chip
PCI BUS A
66MHz
P64H2 #1
100 MHz
PCI BUS B
66MHz
Dual
Channel
SCSI
PCI BUS A
P64H2 #2
Miscellaneous Other Clocks
LPC (low pin count) Legacy
Bus
Video Clock
ICH PCI Slot
Other Clocks used by ICH
...
PCI BUS A
PCI “Riser Card”
Introduction
12/4/2002
30
Magic Decoder Ring
 Names for Signals given so far are from the device sheets
 KC Board uses it’s own names
Board’s have to name signals clearly and uniquely across the whole
board
Devices only have to name then uniquely across the device
 Decoder Ring:
FSB: HD[63:0]#  FSB_HD63_N
Rule is append FSB_ in front and replace ‘#’ with ‘_N’ in the device
signal name
HI: HI[15:0]  P64H2_1_HI[15:0]
Rule is append P64H2_1_ before the device signal name
Note that the ‘1’ in P64H2_1_ may be a ‘1’ or a ‘2’ because KC has two
P64H2 devices
PCI-X: AD[63:0]  P64H2_1_PB_AD[63:0]
Rule is append P64H2_1_PB to beginning of device signal name
Note that the ‘1’ in P64H2_1_PD.. Can be a ‘2’ or ‘3’ … depending on
which pci bus the signal belongs too
DDR: Given Later
Introduction
12/4/2002