A look at memory issues
Data-transfers must occur
between system memory and the
network interface controller
Typical Chipset Layout
CPU
Central Processing Unit
Graphics
Controller
AC
Audio Controller
Multimedia
Controller
MCH
Memory Controller Hub
(Northbridge)
ICH
I/O Controller Hub
(Southbridge)
DRAM
Dynamic Random
Access Memory
NIC
Network Interface
Controller
HDC
Hard Disk Controller
USB
controller
Firmware
Hub
Timer
Keyboard
Mouse
Clock
Typical Chipset Layout
CPU
Central Processing Unit
Graphics
Controller
MCH
Memory Controller Hub
(Northbridge)
DRAM
Dynamic Random
Access Memory
DMA
AC
Audio Controller
Multimedia
Controller
ICH
I/O Controller Hub
(Southbridge)
NIC
Network Interface
Controller
HDC
Hard Disk Controller
USB
controller
Firmware
Hub
Timer
Keyboard
Mouse
Clock
PCI Bus Master DMA
82573L i/o-memory
Host’s Dynamic Random Access Memory
on-chip RX descriptors
packet-buffer
on-chip TX descriptors
packet-buffer
Descriptor Queue
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
DMA
RX and TX FIFOs
(32-KB total)
Memory-mapped I/O
• We mentioned that Intel’s x86 architecture
originally was designed with two separate
address-spaces, one for memory and the
other for I/O ports, unlike the designs for
CPUs by many of Intel’s competitors in
which I/O access was “memory-mapped”
• But now the newer Intel processors also
can support memory-mapped I/O as well
Address-bus widths
• The Intel Core-2 Quad processors in our
classroom and Lab machines potentially
could address 236 physical memory cells
(i.e., 64GB), although only 4GB of RAM
actually are installed at the present time
• Some PCI-compliant hardware devices
were designed for a 32-bit address-bus,
thus they must be “mapped” below 4G
Physical-address assignments
Devices’ registers
must be mapped
to addresses in
the bottom 4G
Dynamic Random
Access Memory
The CPU’s physical
address-space
Virtual addresses
• Software running on the x86 processor is
unable to use actual memory addresses,
but instead uses ‘virtual’ addresses that
map to physical addresses by means of
mapping-tables which Linux dynamically
defines for each different process it runs
• This complicates the steps software must
take to arrange for the DMA to take place
Our ‘dram.c’ module
• To help us confirm that our hardware-level
network software is working as we intend,
or to diagnose our ‘bugs’ if it isn’t, we can
use an LKM we’ve written that implements
a character-mode device-driver for system
memory, allowing us to view the contents
of physical memory as if it were a file; for
example, by using our ‘fileview.cpp’ tool
How to view system memory
•
•
•
•
•
Download ‘dram.c’ from course website
Compile it using our ‘mmake’ utility
Install ‘dram.ko’ by using ‘/sbin/insmod
Insure the ‘/dev/dram’ device-node exists
Download ‘fileview.cpp’ from website and
compile it with ‘g++’ (or with ‘make’)
• Execute ‘fileview /dev/dram’ and use the
arrow-keys to navigate (or hit <ENTER>)
“canonical” addresses
00000
00001
00010
00011
00100
00101
00110
00111
01000
01001
01010
01011
01100
01101
01110
01111
10000
10001
10010
10011
10100
10101
10110
10111
11000
11001
11010
11011
11100
11101
11110
11111
0xFFFFFFFFFFFFFFFF
…
0xFFFF800000000000
Analogy
using
5-bit
values
64-bit
“vrtual”
address
space
0x00007FFFFFFFFFFF
…
0x0000000000000000
“canonical” addresses
“non-canonical”
(invalid)
virtual addresses
“canonical” addresses
4-Levels of mapping
63
48 47
sign-extension
39 38
PML4
30 29
PDPT
21 20
PDIR
12 11
PTBL
0
offset
64-bit ‘canonical’ virtual address
Page
Table
Page
Map
Level-4
Table
CR3
Page
Directory
Pointer
Table
Page
Frame
(4KB)
Page
Directory
Each mapping-table contains up to 512 quadword-size entries
4-level address-translation
• The CPU examines any virtual address it
encounters, subdividing it into five fields
63
48 47
signextension
16-bits
39 38
30 29
index into
index into
level 4
pagepage-map
directory
table
pointer table
9-bits
9-bits
21 20
12 11
index into
pagedirectory
index into
page-table
9-bits
9-bits
0
offset into
page-frame
12-bits
Any 48-bit virtual-address
is sign-extended to a
64-bit “canonical” address
Only “canonical” 64-bit virtual-addresses are legal in 64-bit mode
Format of 64-bit table-entries
63 62
E
X
B
52 51
avl
40 39
Page-frame
physical
base-address
[39..32]
Reserved
(must be 0)
31
12 11
Page-frame physical base-address[31..12]
32
9 8 7 6 5 4 3 2 1 0
avl
P P
A C WUWP
D T
Meaning of
these bits varies with the table
Legend:
P = Present (1=yes, 0=no)
W = Writable (1=yes, 0=no)
U = User-page (1=yes, 0=no)
A = Accessed (1=yes, 0=no)
PWT = Page Cache Disable (1=yes, 0=no)
PWT = Page Write-Through (1=yes, 0=no)
avl = available for user-defined purposes
EXB = Execution-disabled Bit (if EFER.NXE=1)
Our ‘mem64.c’ module
• We wrote an LKM to create a pseudo-file
that will let us see how the virtual memory
is being utilized by an application program
• Download this file, compile it with ‘mmake’
and install ‘mem64.ko’ in the Linux kernel
• Then view the virtual-memory mapping
that is being used by the ‘cat’ program:
$ cat /proc/mem64
The NIC’s PCI ‘resources’
16 doublewords
31
0
Status
Register
BIST
Header
Type
Command
Register
Latency
Timer
Cache
Line
Size
31
0
DeviceID
0x109A
VendorID
0x8086
Class Code
Class/SubClass/ProgIF
Revision
ID
Dwords
1- 0
3- 2
Base Address 1
Base Address 0
5- 4
Base Address 3
Base Address 2
7- 6
Base Address 5
Base Address 4
9- 8
CardBus CIS Pointer
11 - 10
Subsystem
Device ID
Subsystem
Vendor ID
reserved
capabilities
pointer
Expansion ROM Base Address
13 - 12
Maximum Minimum Interrupt
Latency
Grant
Pin
Interrupt
Line
reserved
15 - 14
Mechanisms compared
io
NIC i/o-memory
Each NIC register has its
own address in memory
(allows one-step access)
kernel memory-space
user
memory-space
Access to all of the NIC’s
registers is muliplexed
through a pair of I/O-ports
(requires multiple instructions)
addr
CPU’s ‘virtual’ address-space
data
CPU’s ‘I/O’ address-space
‘nicstatus.c’
• Here’s an LKM that creates a pseudo-file
(called ‘/proc/nicstatus’) which will allow a
user to view the current value in our Intel
82573L Network Interface Controller’s
‘DEVICE_STATUS’ register
• It uses the I/O-port interface to the NIC’s
registers, rather than a ‘memory-mapped’
interface to those device-registers
Device Status (0x0008)
31
?
30
29
28
0
0
27
0
26
0
25
24
0
0
23
0
0
22
0
21
20
0
0
19
18
GIO
Master
EN
17
0
16
0
0
some undocumented functionality?
15
0
14
0
13
0
12
0
11
0
10
PHY
RA
9
ASDV
8
7
6
I
S
L
SPEED
L
O
S
U
FD = Full-Duplex
LU = Link Up
TXOFF = Transmission Paused
SPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved)
ASDV = Auto-negotiation Speed Detection Value
PHYRA = PHY Reset Asserted
5
0
4
TX
OFF
3
2
1
0
Function
ID 0
0U
L
F
D
82573L
‘82573.c’
• This is a more elaborate example of an
LKM which not only creates a pseudo-file
(i.e., ‘/proc/82573’) that we can view using
the Linux ‘cat’ command and that lets us
see our NIC’s PCI Configuration Space,
but also implements some device-driver
functions that let us view the NIC’s device
registers by using our ‘fileview.cpp’ tool
Linux PCI helper-functions
#include <linux/pci.h>
struct pci_dev
unsigned int
unsigned int
void
*devp;
mmio_base;
mmio_size;
*io;
devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );
if ( devp == NULL ) return –ENODEV;
mmio_base = pci_resource_start( devp, 0 );
mmio_size = pci_resource_len( devp, 0 );
io = ioremap_nocache( mmio_base, iomm_size );
if ( io == NULL ) return –ENOSPC;
In-class exercise
• Two of the NIC’s 32-bit device registers
are used to hold its 48-bit Ethernet MAC
address – which will be a different value
for each of the hosts in our classroom
• These two registers are located at offsets
0x5400 and 0x5404 in device-memory
• The six bytes occur in network byte-order
• Write code to show the MAC address!