15-213
“The course that gives CMU its Zip!”
Memory System
Case Studies
Oct. 28, 2004
Topics





class18.ppt
P6 address translation
x86-64 extensions
Linux memory management
Linux page fault handling
Memory mapping
Intel P6
(Bob Colwell’s Chip, CMU Alumni)
Internal Designation for Successor to Pentium

Which had internal designation P5
Fundamentally Different from Pentium

Out-of-order, superscalar operation
Resulting Processors


PentiumPro (1996)
Pentium II (1997)
 L2 cache on same chip

Pentium III (1999)
 The Fish machines
Current Systems Pentium 4

–2–
Different operation, but similar memory system
15-213, F’04
Legacy Issue
x86 Processors Support Multiple Addressing Schemes

Segment-based
 Variable sized blocks
 Numerous variants
» Real mode
» Virtual-8086 mode
» Protected mode
 All obsolete

Page-based
 Fixed size blocks
 Virtual address: 32b
 Physical address: 32b, 40b
 Page sizes: 4K, 2M, 4M
What Will We Do?
–3–

Look at the common case
15-213, F’04
P6 Memory System
32 bit address space
4 KB page size
L1, L2, and TLBs
DRAM

external
system bus
(e.g. PCI)
Inst TLB
L2
cache
instruction
fetch unit
processor package
–4–
L1
i-cache

32 entries

8 sets
Data TLB
cache bus
bus interface unit
4-way set
associative
inst
TLB
data
TLB
L1
d-cache

64 entries

16 sets
L1 i-cache and d-cache

16 KB

32 B line size

128 sets
L2 cache

unified

128 KB -- 2 MB
15-213, F’04
Review of Abbreviations
Symbols:

Components of the virtual address (VA)
 TLBI: TLB index
 TLBT: TLB tag
 VPO: virtual page offset
 VPN: virtual page number

Components of the physical address (PA)
 PPO: physical page offset (same as VPO)
 PPN: physical page number
 CO: byte offset within cache line
 CI: cache index
 CT: cache tag
–5–
15-213, F’04
Overview of P6 Address Translation
32
result
CPU
20
VPN
12
virtual address (VA)
VPO
L1 (128 sets, 4 lines/set)
TLB
hit
...
...
TLB (16 sets,
4 entries/set)
10 10
VPN1 VPN2
20
PPN
PDE
PDBR
–6–
L1
miss
L1
hit
16
4
TLBT TLBI
TLB
miss
L2 and DRAM
PTE
Page tables
20
CT
12
PPO
7 5
CI CO
physical
address (PA)
15-213, F’04
P6 2-level Page Table Structure
Page directory




1024 4-byte page directory
entries (PDEs) that point to page
tables
one page directory per process.
page
page directory must be in
directory
memory when its process is
1024
running
PDEs
always pointed to by PDBR
Page tables:


–7–
1024 4-byte page table entries
(PTEs) that point to pages.
page tables can be paged in and
out.
Up to
1024
page
tables
1024
PTEs
...
1024
PTEs
...
1024
PTEs
15-213, F’04
P6 Page Directory Entry (PDE)
31
12 11
Page table physical base addr
9
Avail
8
7
G
PS
6
5
A
4
3
2
1
0
CD WT U/S R/W P=1
Page table physical base address: 20 most significant bits of physical
page table address (forces page tables to be 4KB aligned)
Avail: These bits available for system programmers
G: global page (don’t evict from TLB on task switch)
PS: page size 4K (0) or 4M (1)
A: accessed (set by MMU on reads and writes, cleared by software)
CD: cache disabled (1) or enabled (0)
WT: write-through or write-back cache policy for this page table
U/S: user or supervisor mode access
R/W: read-only or read-write access
P: page table is present in memory (1) or not (0)
31
1
Available for OS (page table location in secondary storage)
–8–
0
P=0
15-213, F’04
P6 Page Table Entry (PTE)
31
12 11
Page physical base address
9
Avail
8
7
6
5
G
0
D
A
4
3
2
1
0
CD WT U/S R/W P=1
Page base address: 20 most significant bits of physical page
address (forces pages to be 4 KB aligned)
Avail: available for system programmers
G: global page (don’t evict from TLB on task switch)
D: dirty (set by MMU on writes)
A: accessed (set by MMU on reads and writes)
CD: cache disabled or enabled
WT: write-through or write-back cache policy for this page
U/S: user/supervisor
R/W: read/write
P: page is present in physical memory (1) or not (0)
31
1
Available for OS (page location in secondary storage)
–9–
0
P=0
15-213, F’04
How P6 Page Tables Map Virtual
Addresses to Physical Ones
10
VPN1
10
VPN2
word offset into
page directory
12
VPO
word offset into
page table
page directory
PDE
physical address
of page table base
(if P=1)
20
PPN
– 10 –
word offset into
physical and virtual
page
page table
PTE
PDBR
physical address
of page directory
Virtual address
physical
address
of page base
(if P=1)
12
PPO
Physical address
15-213, F’04
4Mbyte PDE’s
– 11 –
15-213, F’04
Support for 4Mbyte Pages
– 12 – Page
Directory occupies single 4KB page (1024 4-byte entries) 15-213, F’04
Representation of VM Address Space
PT 3
Page Directory
P=1, M=1
P=1, M=1
P=0, M=0
P=0, M=1
•
•
•
•
P=1, M=1
P=0, M=0
P=1, M=1
P=0, M=1
P=1, M=1
PT 2
PT 0
P=0, M=0
P=1, M=1
P=0, M=1
P=0, M=1
P=0, M=1
P=0, M=0
P=0, M=0
•
•
•
•
•
•
•
•
•
•
•
•
Simplified Example

16 page virtual address space
Page 15
Page 14
Page 13
Page 12
Page 11
Page 10
Page 9
Page 8
Page 7
Page 6
Page 5
Page 4
Mem Addr
Page 3
Disk Addr
Page 2
Flags


– 13 –
P: Is entry in physical memory?
M: Has this part of VA space
been mapped?
In Mem
Page 1
Page 0
On Disk
Unmapped
15-213, F’04
P6 TLB Translation
32
result
CPU
20
VPN
12
virtual address (VA)
VPO
L1 (128 sets, 4 lines/set)
TLB
hit
...
...
TLB (16 sets,
4 entries/set)
10 10
VPN1 VPN2
20
PPN
PDE
PDBR
– 14 –
L1
miss
L1
hit
16
4
TLBT TLBI
TLB
miss
L2 and DRAM
PTE
Page tables
20
CT
12
PPO
7 5
CI CO
physical
address (PA)
15-213, F’04
P6 TLB
TLB entry (not all documented, so this is speculative):




32
16
1
1
PDE/PTE
Tag
PD
V
V: indicates a valid (1) or invalid (0) TLB entry
PD: is this entry a PDE (1) or a PTE (0)?
tag: disambiguates entries cached in the same set
PDE/PTE: page directory or page table entry
 Structure of the data TLB:

– 15 –
16 sets, 4 entries/set
entry
entry
entry
entry
entry
entry
entry
entry
entry
entry
entry
entry
entry
entry
set 0
set 1
set 2
entry
entry
set 15
...
15-213, F’04
Translating with the P6 TLB
1. Partition VPN into
TLBT and TLBI.
CPU
20
VPN
16
4
TLBT TLBI
TLB
miss
2. Is the PTE for VPN
cached in set TLBI?
12 virtual address
VPO

1
2
PDE
...
page table translation
TLB
PTE hit
3
20
PPN
physical
address
4
– 16 –
12
PPO
3. Yes: then
build physical
address.
4. No: then read PTE (and
PDE if not cached)
from memory and
build physical
address.
15-213, F’04
P6 Page Table Translation
32
result
CPU
20
VPN
12
virtual address (VA)
VPO
L1 (128 sets, 4 lines/set)
TLB
hit
...
...
TLB (16 sets,
4 entries/set)
10 10
VPN1 VPN2
20
PPN
PDE
PDBR
– 17 –
L1
miss
L1
hit
16
4
TLBT TLBI
TLB
miss
L2 and DRAM
PTE
Page tables
20
CT
12
PPO
7 5
CI CO
physical
address (PA)
15-213, F’04
Translating with the P6 Page Tables
(case 1/1)
20
VPN
12
VPO
20
PPN
VPN1 VPN2
12
PPO
Case 1/1: page
table and page
present.
MMU Action:

Mem
PDE p=1
PTE p=1
data
Page
directory
Page
table
Data
page
PDBR
MMU builds
physical
address and
fetches data
word.
 OS action
Disk
– 18 –

none
15-213, F’04
Translating with the P6 Page Tables
(case 1/0)
Case 1/0: page table
20
VPN
present but page
missing.
12
VPO
MMU Action:
VPN1 VPN2


PDE p=1
Mem
PTE p=0
 VA that caused fault
 fault caused by
PDBR
Page
directory
Disk
– 19 –
page fault exception
handler receives the
following args:
Page
table
data
Data
page
non-present page
or page-level
protection violation
 read/write
 user/supervisor
15-213, F’04
Translating with the P6 Page Tables
(case 1/0, cont)
OS Action:
20
VPN

12
VPO

20
PPN
VPN1 VPN2
12
PPO

Mem
PDE p=1
PTE p=1
data
Page
directory
Page
table
Data
page
PDBR


Disk
– 20 –
Check for a legal
virtual address.
Read PTE through
PDE.
Find free physical
page (swapping out
current page if
necessary)
Read virtual page
from disk and copy to
virtual page
Restart faulting
instruction by
returning from
exception handler.
15-213, F’04
Translating with the P6 Page Tables
(case 0/1)
Case 0/1: page table
20
VPN
missing but page
present.
12
VPO
Introduces
consistency issue.
VPN1 VPN2

Mem
PDE p=0
data
Page
directory
Data
page
PDBR
Disk
– 21 –
PTE p=1
Page
table
potentially every
page out requires
update of disk page
table.
Linux disallows this

if a page table is
swapped out, then
swap out its data
pages too.
15-213, F’04
Translating with the P6 Page Tables
(case 0/0)
20
VPN
Case 0/0: page
table and page
missing.
12
VPO
MMU Action:
VPN1 VPN2

page fault
exception
PDE p=0
Mem
PDBR
Page
directory
Disk
– 22 –
PTE p=0
data
Page
table
Data
page
15-213, F’04
Translating with the P6 Page Tables
(case 0/0, cont)
20
VPN
12
VPO
OS action:

VPN1 VPN2

Mem
PDE p=1
PTE p=0
Page
directory
Page
table
PDBR
Disk
– 23 –
swap in page
table.
restart faulting
instruction by
returning from
handler.
Like case 0/1 from
here on.
data
Data
page
15-213, F’04
P6 L1 Cache Access
32
result
CPU
20
VPN
12
virtual address (VA)
VPO
L1 (128 sets, 4 lines/set)
TLB
hit
...
...
TLB (16 sets,
4 entries/set)
10 10
VPN1 VPN2
20
PPN
PDE
PDBR
– 24 –
L1
miss
L1
hit
16
4
TLBT TLBI
TLB
miss
L2 and DRAM
PTE
Page tables
20
CT
12
PPO
7 5
CI CO
physical
address (PA)
15-213, F’04
L1 Cache Access
32
data
L2 andDRAM
L1
miss
L1
hit
L1 (128 sets, 4 lines/set)
Partition physical
address into CO, CI,
and CT.
Use CT to determine if
line containing word
at address PA is
cached in set CI.
If no: check L2.
...
20
CT
7 5
CI CO
If yes: extract word at
byte offset CO and
return to processor.
physical
address (PA)
– 25 –
15-213, F’04
Speeding Up L1 Access
Tag Check
20
CT
7 5
CI CO
PPN
PPO
Physical address (PA)
Addr.
Trans.
virtual
address (VA)
No
Change
VPN
VPO
20
12
CI
Observation




– 26 –

Bits that determine CI identical in virtual and physical address
Can index into cache while address translation taking place
Then check with CT from physical address
“Virtually indexed, physically tagged”
Cache carefully sized to make this possible
15-213, F’04
x86-64 Paging
Origin


AMD’s way of extending x86 to 64-bit instruction set
Intel has followed with “EM64T”
Requirements

48 bit virtual address
 256 terabytes (TB)
 Not yet ready for full 64 bits

52 bit physical address
 Requires 64-bit table entries

4KB page size
 Only 512 entries per page
– 27 –
15-213, F’04
x86-64 Paging
Virtual address
9
VPN1
9
VPN2
9
VPN3
9
VPN4
Page Map
Table
Page
Directory
Pointer
Table
Page
Directory
Table
Page
Table
PM4LE
PDPE
PDE
PTE
12
VPO
BR
40
PPN
– 28 –
12
PPO
Physical address
15-213, F’04
Linux Organizes VM as Collection of
“Areas” process virtual memory
task_struct
mm
vm_area_struct
mm_struct
pgd
mmap
vm_end
vm_start
vm_prot
vm_flags
vm_next

pgd:
 page directory address

vm_prot:
 read/write permissions
vm_end
vm_start
vm_prot
vm_flags
vm_flags
 shared with other
processes or private to
this process
– 29 –
0x40000000
data
0x0804a020
vm_next
for this area

shared libraries
text
vm_end
vm_start
vm_prot
vm_flags
vm_next
0x08048000
0
15-213, F’04
Linux Page Fault Handling
process virtual memory
Is the VA legal?
vm_area_struct

vm_end
vm_start
r/o
shared libraries

vm_next
1
read
vm_end
vm_start
r/w
3
read
data
vm_next
vm_end
vm_start
r/o
text
vm_next
0
– 30 –
Is the operation legal?

2
write
i.e. is it in an area
defined by a
vm_area_struct?
if not then signal
segmentation
violation (e.g. (1))

i.e., can the process
read/write this area?
if not then signal
protection violation
(e.g., (2))
If OK, handle fault

e.g., (3)
15-213, F’04
Memory Mapping
Creation of new VM area done via “memory mapping”


create new vm_area_struct and page tables for area
area can be backed by (i.e., get its initial values from) :
 regular file on disk (e.g., an executable object file)
» initial page bytes come from a section of a file
 nothing (e.g., bss)
» initial page bytes are zeros

dirty pages are swapped back and forth between a special
swap file.
Key point: no virtual pages are copied into physical
memory until they are referenced!


– 31 –
known as “demand paging”
crucial for time and space efficiency
15-213, F’04
User-Level Memory Mapping
void *mmap(void *start, int len,
int prot, int flags, int fd, int offset)

map len bytes starting at offset offset of the file specified
by file description fd, preferably at address start (usually 0
for don’t care).
 prot: MAP_READ, MAP_WRITE
 flags: MAP_PRIVATE, MAP_SHARED


return a pointer to the mapped area.
Example: fast file copy
 useful for applications like Web servers that need to quickly
copy files.
 mmap allows file transfers without copying into user space.
– 32 –
15-213, F’04
mmap() Example: Fast File Copy
#include
#include
#include
#include
#include
<unistd.h>
<sys/mman.h>
<sys/types.h>
<sys/stat.h>
<fcntl.h>
int main() {
struct stat stat;
int i, fd, size;
char *bufp;
/* open the file & get its size*/
fd = open("./mmap.c", O_RDONLY);
fstat(fd, &stat);
size = stat.st_size;
/*
* mmap.c - a program that uses mmap
* to copy itself to stdout
*/
/* map the file to a new VM area */
bufp = mmap(0, size, PROT_READ,
MAP_PRIVATE, fd, 0);
/* write the VM area to stdout */
write(1, bufp, size);
}
– 33 –
15-213, F’04
Exec() Revisited
To run a new program p in
the current process
using exec():
process-specific data
structures
(page tables,
task and mm structs)

physical memory
same
for each
process
kernel code/data/stack
0xc0…
%esp
stack
Memory mapped region
for shared libraries
kernel
VM
demand-zero
process
VM

 stack, bss, data, text,
shared libs.
 text and data backed by
ELF executable object file.
 bss and stack initialized to
zero.
.data
.text
libc.so
brk
runtime heap (via malloc)
– 34 –
0
uninitialized data (.bss)
initialized data (.data)
program text (.text)
forbidden
demand-zero
.data
.text
p
free vm_area_struct’s and
page tables for old areas.
create new
vm_area_struct’s and page
tables for new areas.

set PC to entry point in
.text
 Linux will swap in code and
data pages as needed.
15-213, F’04
Fork() Revisited
To create a new process using fork():

How does OS know
when a page can be
deallocated?
make copies of the old process’s mm_struct,
vm_area_struct’s, and page tables.
 at this point the two processes are sharing all of their pages.
 How to get separate spaces without copying all the virtual
pages from one space to another?
» “copy on write” technique.

copy-on-write
 make pages of writeable areas read-only
 flag vm_area_struct’s for these areas as private “copy-on-
write”.
 writes by either process to these pages will cause page faults.
» fault handler recognizes copy-on-write, makes a copy of the
page, and restores write permissions.

Net result:
 copies are deferred until absolutely necessary (i.e., when one of
– 35 –
the processes tries to modify a shared page).
15-213, F’04
Memory System Summary
Cache Memory



Purely a speed-up technique
Behavior invisible to application programmer and (mostly)
OS
Implemented totally in hardware
Virtual Memory

Supports many OS-related functions
 Process creation
» Initial
» Forking children
 Task switching
 Protection

Combination of hardware & software implementation
 Software management of tables, allocations
 Hardware access of tables
– 36 –
 Hardware caching of table entries (TLB)
15-213, F’04