Gerard C. Weatherby
Memory management routines
• ANSI C / C++ routines
• Win32 Virtual memory
• Win32 Heap
• Unix memory functions
Computer Operating Systems
Memory.1
Gerard C. Weatherby
ANSI C routines
• malloc - raw block of memory (aligned for any use)
void *malloc(size_t size);
int *varray = (int *)malloc(100*sizeof(int));
Returns null if memory not allocated
• calloc - array allocation
void *calloc(size_t num, size_t size)
similiar to: malloc(num * size)
memory is zero-filled
• free - release memory
void free(void *mem)
free(varray)
Must free memory to avoid memory leaks. Unused memory
Computer Operating Systems
Memory.2
Gerard C. Weatherby
not being returned to the operating system.
Note: free(NULL) or free(0) is a safe no-operation
free(x) where “x” is not a value returned from a call to malloc, calloc, or realloc is undefined.
Undefined - language standard does not specify
what will happen...
...but it won’t be good. (most often a program
crash)
• realloc - reallocate previous allocated memory
void *realloc(void *previous,size_t newsize)
int *valarray = (int *)realloc(valarray,200 * sizeof(int));
May expand or reduce size of allocated block.
Memory in smaller of old size and new size unchanged.
Computer Operating Systems
Memory.3
Gerard C. Weatherby
Return address may be different from original (block of
memory may be moved and copied)
More likely to be different if new size greater than previous
Returns null if insufficient memory available.
• alloca - variable allocation off runtime stack
void *alloca(size_t size)
void stackAllocate(int size) {
//char block[size];// error C2057: expected constant expression
char * block = (char *)alloca(size * sizeof(char));
exception (Win32) or undefined (Solaris) if insufficient stack
space
freed automatically when function returns
Computer Operating Systems
Memory.4
Gerard C. Weatherby
Memory alignment
• malloc, et. al. return “bytes suitably aligned for any use”
• certain hardware can only process numbers which have specific addresses.
Sparc two-byte quantities should be stored at even locations,
word quantities should be stored at addresses which are
multiples of four.
• Crashes on Solaris:
char memory[13];
double *d = (double *)&memory[2];
*d = 1.234;
Works fine on Win32
Computer Operating Systems
Memory.5
Gerard C. Weatherby
C++ Memory routines
• new - allocates memory for specified type
C++ keyword
semid_ds *buf = new semid_ds;
Note returned pointer is of appropriate type (no cast
required)
For a user defined type (class), special code called constructor automatically executed.
A class may also define a specialized memory allocation routine, a new operator, to provide raw memory
set_new_handler function can be used to customized behavior if insufficient memory behavior
by default, null returned or C++ exception thrown
Computer Operating Systems
Memory.6
Gerard C. Weatherby
• new[] - array new; allocates an array of objects. Constructor
called for each object.
int *array = new int[100]
• delete - releases memory allocated by new.
delete(buf);
A class may define a special function called a destructor
which is automatically executed when object deleted.
• delete[] - release memory allocated by new[]
Must match new/delete and new[]/delete[] calls
Computer Operating Systems
Memory.7
Gerard C. Weatherby
Win32 Virtual Memory
• A set of APIs allow more direct control over memory allocation.
Increased performance
Keeping related data together on nearby pages
Telling operating system to stop swapping pages
(you don’t care about data anymore)
Prohibit operating system from swapping pages
(lock into physical memory)
Set memory to read-only (protect against program errors, security)
Direct mapping of hardware addresses
e.g. Writing a device driver
Computer Operating Systems
Memory.8
Gerard C. Weatherby
• Virtual Alloc - allocate memory
LPVOID VirtualAlloc(
LPVOID lpAddress, // address of region to reserve or commit
DWORD dwSize,
// size of region
DWORD flAllocationType, // type of allocation
DWORD flProtect // type of access protection
);
allocation type:
MEM_COMMIT - allocate
MEM_RESERVE - don’t use for anything else
(not actually allocated)
MEM_RESET - stop maintaining values
MEM_TOP_DOWN - get highest possible address
Computer Operating Systems
Memory.9
Gerard C. Weatherby
protection type:
PAGE_READONLY
PAGE_READWRITE
PAGE_EXECUTE
PAGE_EXECUTE_READ
PAGE_EXECUTE_READWRITE
PAGE_GUARD
PAGE_NOACCESS
PAGE_NOCACHE
Computer Operating Systems
Memory.10
Gerard C. Weatherby
• VirtualFree - release memory
BOOL VirtualFree(
LPVOID lpAddress, // address of region of committed pages
DWORD dwSize,
// size of region
DWORD dwFreeType); // type of free operation
where free type can be
MEM_DECOMMIT
MEM_FREE
• VirtualLock - lock committed memory (keep from swapping)
BOOL VirtualLock(
LPVOID lpAddress, // address of first byte of range to lock
DWORD dwSize );
// number of bytes in range to lock
Maximum of 30 unless changed with SetWorkingSetSize
Computer Operating Systems
Memory.11
Gerard C. Weatherby
• VirtualUnlock - unlocks locked memory
BOOL VirtualUnlock(
LPVOID lpAddress, // address of first byte of range
DWORD dwSize
// number of bytes in range
);
• SetProcessWorkingSetSize - sets number of pages to hold in
physical memory
BOOL SetProcessWorkingSetSize(
HANDLE hProcess,
// handle to the process of interest
DWORD dwMinimumWorkingSetSize, // specifies minimum working set size
DWORD dwMaximumWorkingSetSize // specifies maximum working set size
);
Goal, not a guarantee
Requires appropriate system privileges (Administrator /
Power User)
Computer Operating Systems
Memory.12
Gerard C. Weatherby
• Virtual Protect - change protection level of memory
BOOL VirtualProtect(
LPVOID lpAddress, // address of region of committed pages
DWORD dwSize,
// size of the region
DWORD flNewProtect, // desired access protection
PDWORD lpflOldProtect ); // address of variable to get old protection
DWORD oldProtect;
if (VirtualProtect(myAddress,2000,PAGE_READONLY,&oldProtect))...
options for protect fields same as VirtualAlloc
• Virtual Query - return information about memory range
DWORD VirtualQuery(
LPCVOID lpAddress,
// address of region
PMEMORY_BASIC_INFORMATION lpBuffer, // address of information buffer
DWORD dwLength );
// size of buffer
MEMORY_BASIC_INFORMATION meminfo;
VirtualQuery(memory,&meminfo,sizeof(meminfo));
Computer Operating Systems
Memory.13
Gerard C. Weatherby
MEMORY_BASIC_INFORMATION is defined as:
typedef struct _MEMORY_BASIC_INFORMATION { // mbi
PVOID BaseAddress;
// base address of region
PVOID AllocationBase;
// allocation base address
DWORD AllocationProtect;
DWORD RegionSize;
DWORD State;
// initial access protection
// size, in bytes, of region
// committed, reserved, free
DWORD Protect;
// current access protection
DWORD Type;
// type of pages
} MEMORY_BASIC_INFORMATION;
Type options are image (return from MapDebugInformation), memory mapped, or private
Computer Operating Systems
Memory.14
Gerard C. Weatherby
• GetSystemInfo - returns processor memory information
VOID GetSystemInfo(LPSYSTEM_INFO lpSystemInfo);
SYSTEM_INFO info;
GetSystemInfo(&info);
typedef struct _SYSTEM_INFO {
union { DWORD dwOemId; //obsolete
struct { WORD wProcessorArchitecture; //INTEL, ALPHA, MIPS, PPC
WORD wReserved;
};
};
DWORD dwPageSize; //e.g. 4096 on NT
LPVOID lpMinimumApplicationAddress;
LPVOID lpMaximumApplicationAddress;
DWORD dwActiveProcessorMask; //bit map of active processors
DWORD dwNumberOfProcessors;
DWORD dwProcessorType; //386/486/Pentium -- obsolete
DWORD dwAllocationGranularity; //e.g. 64K
WORD wProcessorLevel;
WORD wProcessorRevision; } SYSTEM_INFO;
Computer Operating Systems
Memory.15
Gerard C. Weatherby
• GlobalMemoryStatus - system’s current physical and virtual
memory usage
VOID GlobalMemoryStatus(LPMEMORYSTATUS lpBuffer );
MEMORYSTATUS mstat;
GlobalMemoryStatus(&mstat);
typedef struct _MEMORYSTATUS {
DWORD dwLength;
// sizeof(MEMORYSTATUS)
DWORD dwMemoryLoad;
// percent of memory in use
DWORD dwTotalPhys;
// bytes of physical memory
DWORD dwAvailPhys;
// free physical memory bytes
DWORD dwTotalPageFile; // bytes of paging file
DWORD dwAvailPageFile; // free bytes of paging file
DWORD dwTotalVirtual; // user bytes of address space
DWORD dwAvailVirtual; // free user bytes
} MEMORYSTATUS, *LPMEMORYSTATUS;
Computer Operating Systems
Memory.16
Gerard C. Weatherby
Virtual memory in other processes
• Previous functions all work in current process
• Variants available to operate in other processes
Have additional process handle argument
• Require appropriate permission
VirtualAllocEx
VirtualFreeEx
VirtualProtectEx
VirtualQueryEx
Computer Operating Systems
Memory.17
Gerard C. Weatherby
• Example program
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <winbase.h>
#include <assert.h>
#include <iostream.h>
const char * decodeProtect(DWORD); //implementation omitted (see web)
const char * decodeState(DWORD);
const char * decodeType(DWORD);
int main( )
{
//find lowest valid address
SYSTEM_INFO info;
GetSystemInfo(&info);
void * address = info.lpMinimumApplicationAddress;
//allocate read write memory
const int length = 100;
Computer Operating Systems
Memory.18
Gerard C. Weatherby
int * memory = static_cast<int *>(
VirtualAlloc(address,
length *sizeof(int),
MEM_COMMIT,
PAGE_READWRITE));
assert(memory);
cout << "allocate request " << address << endl;
cout << "allocate return " << memory << endl;
//build table of squares
for (int i=0;i<length;i++)
memory[i] = i * i;
//protect memory
DWORD oldProtect;
const BOOL protect = VirtualProtect(memory,length*sizeof(int),
PAGE_READONLY,&oldProtect);
assert(protect);
assert(oldProtect == PAGE_READWRITE);
for (int x=5;x<10;x++)
cout << x << " squared: " << memory[x] << endl;
Computer Operating Systems
Memory.19
Gerard C. Weatherby
__try {
//attempt to change memory
cout << "Attempt to write to memory" << endl;
memory[10] = 3;
//never actually executes...
cout << "10 value is " << memory[10] << endl;
}
__except(EXCEPTION_EXECUTE_HANDLER) { cout << "Write failed" <<
endl; }
MEMORY_BASIC_INFORMATION meminfo;
VirtualQuery(memory,&meminfo,sizeof(meminfo));
cout << "base address " << meminfo.BaseAddress << endl;
cout << "allocation address " <<meminfo.AllocationBase << endl;
cout << "region size " << meminfo.RegionSize << endl;
cout << "initial protection " << decodeProtect(meminfo.AllocationProtect) <<
endl;
cout << "current protection " << decodeProtect(meminfo.Protect) << endl;
cout << "state " << decodeState(meminfo.State) << endl;
cout << "type " << decodeType(meminfo.Type) << endl; }
Computer Operating Systems
Memory.20
Gerard C. Weatherby
• Sample output:
allocate request 0x00010000
allocate return 0x00010000
5 squared: 25
6 squared: 36
7 squared: 49
8 squared: 64
9 squared: 81
Attempt to write to memory
Write failed
base address 0x00010000
allocation address 0x00010000
region size 4096
initial protection PAGE_READWRITE
current protection PAGE_READONLY
state MEM_COMMIT
type MEM_PRIVATE
Computer Operating Systems
Memory.21
Gerard C. Weatherby
Windows Heap functions
• malloc/free new/delete operate using a default heap
• Can explicitly operate on a heap
Performance: give each thread its own heap and avoid concurrency checking
Simplicity: create complex memory structures in a dedicated heap. Delete entire heap instead of invidual pointers.
• Following flags are used by several routines
HEAP_GENERATE_EXCEPTIONS - allocation failure
raises operating system exception instead of returning null
HEAP_NO_SERIALIZE - no concurrency checks
HEAP_ZERO_MEMORY - self-explanatory
Computer Operating Systems
Memory.22
Gerard C. Weatherby
• GetProcessHeap - access the default heap
HANDLE GetProcessHeap( );
• HeapCreate - create a new heap
HANDLE HeapCreate(
DWORD flOptions, (exceptions, no serialize)
DWORD dwInitialSize, // initial heap size
DWORD dwMaximumSize); // maximum heap size, (0 for unlimited)
Note: sizes are rounded up to nearest page size
• HeapAlloc - allocate memory from a heap
LPVOID HeapAlloc(
HANDLE hHeap, // handle to heap
DWORD dwFlags, // control flags (exceptions, no serialize, zero memory)
DWORD dwBytes); // number of bytes to allocate
Computer Operating Systems
Memory.23
Gerard C. Weatherby
• HeapReAlloc - reallocate memory
LPVOID HeapReAlloc(
HANDLE hHeap, // handle to heap
DWORD dwFlags, // control flags (exceptions, no serialize, zero memory)
LPVOID lpMem, // pointer to the memory to reallocate
DWORD dwBytes );// number of bytes to reallocate
• HeapFree - release memory
BOOL HeapFree(
HANDLE hHeap, // handle to heap
DWORD dwFlags, // no serialize or 0
LPVOID lpMem // pointer to the memory to free
);
• HeapDestroy - release entire heap
BOOL HeapDestroy(
HANDLE hHeap); // handle to the heap returned from HeapCreate
Computer Operating Systems
Memory.24
Gerard C. Weatherby
• Additonial heap functions
HeapValidate - check integrity of a heap
HeapWalk - step through allocations of a heap
HeapCompact - try to make free blocks bigger
HeapLock - calling thread attempts to gain exclusive access
to heap
HeapUnlock - release lock
HeapSize - get size of heap
Computer Operating Systems
Memory.25
Gerard C. Weatherby
UNIX memory control
memcntl - memory management control
int memcntl(caddr_t addr, //address to alter; must be multiple of pagesize
size_t len, //length to alter
int cmd, //MC_LOCK,MC_LOCKAS,MC_SYNC,MC_UNLOCK,MC_UNLOCKAS
caddr_t arg, //MCL_CURRENT, MCL_FUTURE
int attr, /*SHARED, PRIVATE, PROT_READ,PROT_WRITE,PROT_EXEC,
PROC_TEXT,PROC_DATA */
int mask ); //always 0
If cmd is MCL_LOCKAS
arg - MCL_CURRENT, MCL_FUTURE
If cmd is MC_SYNC
arg - MC_ASYNC,MC_SYNC,MS_INVALIDATE
Only MC_SYNC does not require superuser privilege.
Computer Operating Systems
Memory.26
Gerard C. Weatherby
mlock - lock pages into memory (keep from swapping)
munlock - unlock pages
mlockall - lock all a processes pages
munlockall - unlock all pages
plock - lock based on segment
int mlock(const void * addr, size_t len);
int munlock(const void * addr, size_t len);
int mlockall(int flags); //MCL_CURRENT, MCL_FUTURE
int munlockall( );
int plock(int op); //PROCLOCK (text & data),TXTLOCK,DATLOCK,UNLOCK
Locking pages may degrade system performance; all functions require superuser permission
Computer Operating Systems
Memory.27
Gerard C. Weatherby
• getpagesize - get size of page
int getpagesize( );
Some memory management calls require even pagesize
addresses
• mprotect - change accesses of specified pages
int mprotect(void *addr, //address to protect; must be pagesize interval
size_t len, //number bytes to protect
int prot); //PROT_READ, PROT_WRITE,PROT_EXEC,PROT_NONE
Computer Operating Systems
Memory.28
Gerard C. Weatherby
• mprotect example (simplified)
int main( ){
const int pagesize = getpagesize( );
cout << "pagesize " << pagesize << endl;
const int length= 100;
//must get even page boundary to allow protection
int raw = reinterpret_cast<int>(malloc(2 *pagesize));
int *array = reinterpret_cast<int *>((1 + raw/pagesize)*pagesize);
for (int i=0;i<length;i++)array[i] = i * i;
mprotect(reinterpret_cast<char *>(array),
length * sizeof(int),
PROT_READ);
cout << "5 squared is " << array[5] << endl;
array[5] = 10;
cout << "memory written" << endl;
cout << "5 squared is " << array[5] << endl; }
pagesize 8192
5 squared is 25
bmem[4]: 9516 Segmentation Fault(coredump)
Computer Operating Systems
Memory.29
Gerard C. Weatherby
UNIX shared memory
Can share memory between processes
Fastest means of interprocess communication
Typically use sempahore to guard access
• shmget - get (create or attach to) shared memory identifier
int shmget(key_t key, //IPC_PRIVATE or return from ftok
size_t size, //size of memory to allocate
int shmflg); //IPC_CREAT, permissions
Note size and shmflg ignored if attaching to existing memory
• shmat - create or attach to shared memory region
void * shmat(int shmid, //identifier returned from shmget
const void * shmaddr, //address to allocate (perhaps)
int shmflg); //SHM_SHAR_MMU,SHM_RND,SHM_READONLY
Computer Operating Systems
Memory.30
Gerard C. Weatherby
• shmctl - control operations on shared memory
int shmctl(int shmid, //identifier returned from shmget
int cmd, //IPC_STAT, IPC_SET, IPC_RMID, SHM_LOCK,SHM_LOCK
struct shmid_ds *buf);
shmid_ds struct
filled by IPC_STAT
IPC_SET uses shm_perm
struct ipc_perm shm_perm;
int
/* operation permission struct */
shm_segsz;
struct region
/* size of segment */
*shm_reg;
/* ptr to region structure */
char
pad[4];
/* for swap compatibility */
pid_t
shm_lpid;
/* pid of last operation */
pid_t
shm_cpid;
/* creator pid */
ushort_t
shm_nattch; /* number of current attaches */
ushort_t
shm_cnattch; /* used only for shminfo */
Computer Operating Systems
Memory.31
Gerard C. Weatherby
time_t
shm_atime;
/* last attach time */
time_t
shm_dtime;
/* last detach time */
time_t
shm_ctime;
/* last change time */
SHM_LOCK and SHM_UNLOCK require superuser
IPC_RMID most common.
Shared memory not automatically released when
a process exits -- can degrade system if not
cleaned up.
ipcs / ipcrm command line utilities may be used to
cleanup up semaphores and shared memory
Computer Operating Systems
Memory.32
Gerard C. Weatherby
• Example - first process (simplified)
#include "sem.h"// setSem, semWait
int main( ) {
const int key = ftok("keyfile",'a');
const int shmid = shmget(key,sizeof(int),IPC_CREAT|0644);
void *shared = shmat(shmid,0,0);
cout << "Shared is " << shared << endl;
const int semid = semget(key,1,IPC_CREAT|0666);
setSem(semid,0); //sets semaphore to 0
//wait for other process
semWait(semid);
int *pvalue = static_cast<int*>(shared);
cout << "shared value is " << *pvalue << endl;
shmctl(shmid,IPC_RMID,0);
semctl(semid,IPC_RMID,0);
}
Computer Operating Systems
Memory.33
Gerard C. Weatherby
• Example - second process (simplified)
#include "sem.h" //semSignal
int main(int argc , char *argv[])
{
const int key = ftok("keyfile",'a');
const int shmid = shmget(key,sizeof(int),0644);
void *shared = shmat(shmid,0,0);
int *pvalue = static_cast<int *>(shared);
const int semid = semget(key,1,0666);
const int value = atoi(argv[1]);
cout << "Setting " << shared << " to " << value << endl;
*pvalue = value;
//signal other process
semSignal(semid);
}
Computer Operating Systems
Memory.34
Gerard C. Weatherby
Security
• Displaying windows error messages
• C / C++ approach to coding
• NT security model
Computer Operating Systems
Security.1
Gerard C. Weatherby
Printing Windows error messages
• Unix offers perror for failed system calls -- prints error message to standard error.
• Windows does not have a single call, but it can be replicated:
#include <windows.h>
#include <stdio.h>
win_perror(char *msg)
{
# define BLEN 512
char buffer[BLEN];
long hr;
hr = GetLastError( );
FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM,0,hr,0,buffer,BLEN,0);
fprintf(stderr,"%s: %s\n",msg,buffer);
}
void main( )
Computer Operating Systems
Security.2
Gerard C. Weatherby
{
HANDLE hFile = CreateFile("d:\\notthere.txt", GENERIC_READ |
GENERIC_WRITE,
0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0);
if (hFile == INVALID_HANDLE_VALUE)
win_perror("Create File");
}
produces
Create File: The system cannot find the file specified.
Security.3
Computer Operating Systems
Gerard C. Weatherby
NT model
Access token -- user
Security descriptor -- lock
Security
Descriptor
?
Access token
key
lock
created,
assigned
logs in
User
object
Computer Operating Systems
Security.4
Gerard C. Weatherby
Example program C
#include <windows.h>
#include <iostream.h>
SECURITY_ATTRIBUTES sa;
SECURITY_DESCRIPTOR sd;
BYTE aclbuf[1024], SIDbuf[100];;
PACL pacl=(PACL)&aclbuf;
PSID psid=(PSID) &SIDbuf;
DWORD SIDbufSize = 100;
DWORD domainbufsz = 80;
char domainbuf[80];
SID_NAME_USE snu;
HANDLE hFile;
void main(void)
{
InitializeSecurityDescriptor(&sd, SECURITY_DESCRIPTOR_REVISION);
InitializeAcl(pacl, 1024, ACL_REVISION);
Computer Operating Systems
Security.5
Gerard C. Weatherby
LookupAccountName(0, "guest", psid, &SIDbufSize, domainbuf,
&domainbufsz, &snu);
AddAccessAllowedAce(pacl, ACL_REVISION, GENERIC_READ, psid);
SetSecurityDescriptorDacl(&sd, TRUE, pacl, FALSE);
sa.nLength= sizeof(SECURITY_ATTRIBUTES);
sa.bInheritHandle = FALSE;
sa.lpSecurityDescriptor = &sd;
hFile = CreateFile("d:\\testacl.txt", GENERIC_READ | GENERIC_WRITE,
0, &sa, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, 0);
CloseHandle(hFile);
}
Computer Operating Systems
Security.6
Gerard C. Weatherby
Example program, C++
#include <windows.h>
#include "securitydescriptor.h"
#include "sid.h"
#include "acl.h"
#include "securityattributes.h"
#include "err.h"
void main(void)
{
try {
Sid sid("guest");
Acl acl(sid);
acl.AddAccessAllowed(GENERIC_READ);
SecurityDescriptor sd;
sd.SetDacl(acl);
SecurityAttributes sa(sd);
HANDLE hFile = CreateFile("d:\\testacl.txt", GENERIC_READ |
GENERIC_WRITE,
Computer Operating Systems
Security.7
Gerard C. Weatherby
0, &sa, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, 0);
if (hFile == INVALID_HANDLE_VALUE)
ShowErr("CreateFile");
CloseHandle(hFile);
} catch (const char *msg)
{
ShowErr(msg);
}
}
Computer Operating Systems
Security.8