Security Patterns for
Operating Systems
by Ed Fernandez and Tami Sorgente
Secure Systems Research Group - FAU
Security and operating systems (OS)
• OS act as an intermediary between the user of a
computer and its hardware
• OS supports the execution of all the applications
• The OS architecture is fundamental in the
organization its components and for utilization of
these component services in a given application
• It is the most critical of the software layers because
compromise can affect all applications and
persistent data
• Most reported attacks occur through the OS
• Security Patterns for Operating Systems
Secure Systems Research Group - FAU
Security Patterns for
Operating Systems
• Published Patterns:
–
–
–
–
–
–
–
–
–
File access control
Controlled Virtual Address Space
Reference Monitor
Controlled Execution Environment
Authenticator
Controlled-Process Creator
Controlled-Object Factory
Controlled-Object Monitor
OS Architecture Patterns
• Future Pattern Ideas:
–
–
–
–
–
–
–
–
–
Process structure
Thread structure
Administration roles
Execution Rings
Secure process communication
Protected entry points
Distributed directory
Isolated execution environments
Virtual Address Space Structure
Secure Systems Research Group - FAU
O.S. Access Control Pattern Language
Secure Systems Research Group - FAU
Patterns for Operating Systems
Access Control
• File access control. How do you control access to files in an operating system? Apply the
Authorization pattern to describe access to files by subjects. The protection object is now a file
component that may be a
directory or a file.
• Controlled Virtual Address Space. How to control access by processes to specific areas
of their virtual address space (VAS) according to a set of predefined access types? Divide the VAS
into segments that correspond to logical units in the programs. Use special words (descriptors) to
represent access rights for these segments.
• Reference Monitor. How to enforce authorizations when a process requests access to an
object? Define an abstract process that intercepts all requests for resources and checks them for
compliance with authorizations.
• Controlled Execution Environment. How to define an execution environment for
processes? Attach to each process a set of descriptors that represent the rights of the process. Use
the Reference Monitor to enforce access.
[Fer02] E.B.Fernandez, "Patterns for operating systems access control", Procs. of PLoP
2002, http://jerry.cs.uiuc.edu/~plop/plop2002/proceedings.html
Secure Systems Research Group - FAU
The Authorization pattern
Subject
ID
*
AuthorizationRule
Right
accessType
predicate
copyFlag
checkRights ( )
Secure Systems Research Group - FAU
ProtectionObject
*
ID
The File Authorization pattern
Subject
*
AuthorizedOn
ID
ACLE
accessMode
Authorized
For
*
Workstation
ID
Authorization
priority
privileges
startSession ( )
FileComponent
HasHomeDirectory
File
name
size
createFile ( )
save ( )
Secure Systems Research Group - FAU
*
Directory
name
Virtual Address Space Access
Control pattern
Process
VAS
*
Descriptor
*
base
limit
access_type
Secure Systems Research Group - FAU
*
Segment
Accesses
1 address
size
The Execution Domain pattern
Process
1
*
Domain
Executes In
ID
* ID
create( )
* )
enter(
delete( )
Descriptor
base
limit
access_type
Secure Systems Research Group - FAU
*
Composite
Domain
Simple
Domain
Reference Monitor pattern
Process
MakesRequestTo
*
*
Reference
Monitor
Exists
*
*
Authorization
Rules
Request
resource
access_type
Secure Systems Research Group - FAU
Concrete
Reference
Monitor
*
Authorization
Controlled Execution Environment
pattern
Reference
Monitor
1
1
Request
*
Process
*
1
Executes In
ID
*
*
Domain
Composite
Domain
* ID
create( )
enter* ( )
delete( )
Simple
Domain
Activates
ProtectionObject
Authorization
*
Uses
right
1
0..1 Descriptor
1
User
*
*
ID
create ( )
close( )
delete( )
ID
Resource {A}
name
address
amount
ConcreteResource
Secure Systems Research Group - FAU
More patterns for operating systems
access control
• Authenticator. How to verify that a subject is who it says it is? Use a single point of access to
receive the interactions of a subject with the system and apply a protocol to verify the identity of the
subject.
• Controlled-Process Creator. How to define the rights to be given to a new process?
Define their rights as part of their creation.
• Controlled-Object Factory. How to specify rights of processes with respect to a new
object? When a process creates a new object through a Factory, the request includes the features of
the new object. Among these features include a list of rights to access the object.
• Controlled-Object Monitor. How to control access by a subject to an object? Use a
reference monitor to intercept access requests from processes. The reference monitor checks if the
process has the requested type of access to the object.
[Fer03] E. B. Fernandez and J. C. Sinibaldi, “More patterns for operating system access control”, Proc. of the 8th
European conference on Pattern Languages of Programs, EuroPLoP 2003, http://hillside.net/europlop, 381-398.
Secure Systems Research Group - FAU
Authentication Pattern
Secure Systems Research Group - FAU
Class model for PKI authentication
Secure Systems Research Group - FAU
Controlled-Process Creator
Secure Systems Research Group - FAU
Controlled Object Creator
Secure Systems Research Group - FAU
Controlled Object Monitor
Secure Systems Research Group - FAU
OS architectures
•
Patterns representing an abstract view of
the four basic architectures of operating
systems:
1) Modular Operating System Architecture
2) Layered Operating System Architecture
3) Microkernel Operating System
Architecture
4) Virtual Machine (VM) Operating System
Architecture
Secure Systems Research Group - FAU
Operating System Architectures
Can be combined
Layered Modular
Operating
System Architecture
Modular Operating
System Architecture
Can run
Can be combined
Can be combined
Layered Operating
System Architecture
Layered Microkernel
Operating
System Architecture
Can be combined
Secure Systems Research Group - FAU
Can run
Can run
Microkernel Operating
System Architecture
Virtual Machine
Operating
System Architecture
The Modular Operating System
Architecture pattern
• An object oriented approach is used to
dynamically load and link loadable
modules to the core component of the OS
LoadedModule
Can call
*
LoadableModule
*
Secure Systems Research Group - FAU
*
1
KernelCore
Modular Operating System
Architecture
• Known uses
– Solaris version 10
– ExtremeWare from Extreme Networks
[Ext].
• Related patterns
– The Controlled Execution Environment
pattern [Fer0] can be used to isolate
executing modules.
Secure Systems Research Group - FAU
Layered Operating System
Architecture
• Separate the OS into layers
• OS has more control over separation of
concerns
• Overall features and functionality are
separated into layers
• Clearly defined interfaces between each
kernel section of the OS and between user
applications and the OS functions
Secure Systems Research Group - FAU
Layered Operating System
Architecture Pattern
• Define a hierarchical set of layers and
assign components to each layer. Each
layer presents an abstract machine to the
layer above it.
Secure Systems Research Group - FAU
Layered Operating System
Architecture Pattern
Client
<<uses>>
LayerN
1
LayerN-1
.
.
.
1
Layer2
1
Layer1
Secure Systems Research Group - FAU
Layered Operating System
Architecture Pattern Dynamics
• Sequence diagram for opening and reading a disk file
<<actor>>
aUser:
:OSInterface
:FileManager
openFile(…)
openFile(…)
readDisk(…)
•A user sends an openFile( ) request to the OSInterface
•The OSInterface interprets the openFile( ) request
•The openFile( ) request is sent from the OSInterface to the FileManager
•The FileManager sends readDisk( ) request to the DiskDriver
Secure Systems Research Group - FAU
:DiskDriver
OS Layered Architecture
users
utilities
file system
I/O drives
disk drives
...
UserApplication
Layer 5
Utilities
Layer 4
FileSystem
Layer 3
I/Odrives
Layer 2
Hardware
Layer 1
hardware
Secure Systems Research Group - FAU
Layered Operating System
Architecture
• Known uses
– OS/2 of IBM, Symbian OS [Sym01]
• Related patterns
– specialization of the Layers architectural
pattern [Bus96].
– A security version of the layers pattern is
presented in [Fer02] and in [Som05].
Secure Systems Research Group - FAU
Microkernel Operating System
Architecture
• Move as much of the OS functionality from
the kernel space
• Very basic set of functions in microkernel
• Use external and internal servers
Secure Systems Research Group - FAU
Microkernel Operating System
Architecture pattern
ExternalServer
*
calls
Microkernel
1
receive request
dispatch request
execute service
*
execute mechanism
init communication
find receiver
call internal server
send message
create handle (unique ID)
1
InternalServer
*
1
activates
receive request
dispatch request
execute service
Initializes
communication
1
sends request
1
Adapter
calls service
creates request
Secure Systems Research Group - FAU
Client
1
1
calls service
do task
Microkernel Operating System
Architecture
• Known uses
– PalmOS [PalmOS], QNX [QNX]
• Related patterns
– specialization of the microkernel pattern
[Bus96].
Secure Systems Research Group - FAU
Virtual Machine Operating System
Architecture
• Provides a set of replicas of the hardware
architecture to separate operating
systems
• Strong isolation between each OS
Secure Systems Research Group - FAU
Virtual Machine Operating System
Architecture pattern
• Define an architectural layer that is in control of
the hardware and supervises and coordinates the
execution of each OS environment.
• This extra layer, usually called a Virtual machine
Monitor (VMM) or Hypervisor presents to each
operating system a replica of the hardware.
• The VMM intercepts all system calls and
interprets them according to the OS from where
they came.
Secure Systems Research Group - FAU
Virtual Machine Operating System
Architecture pattern
VMOS
1
*
<<controls>>
VirtualMachineMonitor
VM
*
*
Can run
*
Supports
*
LocalOS
1
Hardware
*
LocalProcess
Secure Systems Research Group - FAU
Virtual Machine Operating System
Architecture example resolved
• Two companies using Unix and Linux in
different virtual machines
UNIX
Linux
VM1
VM2
VMM (virtual machine monitor)
hardware
Secure Systems Research Group - FAU
OS Pattern Ideas/ Future Work
•
•
•
•
•
•
•
•
•
Process structure
Thread structure
Administration roles
Execution Rings
Secure process communication
Protected entry points
Distributed directory
Isolated execution environments
Virtual Address Space Structure
Secure Systems Research Group - FAU
Process structure
• Process Control Block (PCB),
– a data structure containing its id, a reference to its
code, program counter, stack, registers, status,
and references to its resources (domain).
• A process receives a separate address space for
its execution.
– Some architectures, e.g., the Intel X86 series,
have hardware support for the type of PCB used
by common operating systems (this accelerates
context switching, changing execution from one
process to another).
Secure Systems Research Group - FAU
Thread structure
•
•
•
•
Lightweight process
Faster context switching than a process
shares its address space with other threads.
Typically, a thread includes a program counter, a
register set, and a stack.
• Because of its shared address space, an error or
attack from another thread can corrupt its
memory.
• Thread stacks can be protected if they are kept in
the system address space using separated
segments or pages.
Secure Systems Research Group - FAU
Thread structure
• Most modern operating systems, e.g., Solaris,
allow several threads to be bundled in one
process; this protects the thread group as a
whole from other processes.
• User processes and threads can be created with
special packages, e.g., Posix in Unix, or through
the language, as in Java or Ada.
• The operating system defines kernel threads as
units of concurrent execution.
• Kernel threads usually don’t have any protection
against each other.
Secure Systems Research Group - FAU
Administration roles
•
AIX reduces the privileges of the system administrator by defining five
partially-ordered roles: Superuser, Security Administrator, Auditor,
Resource Administrator, and Operator.
•
Windows Windows NT uses four roles for administrative privileges:
standard, administrator, guest, and operator. A User Manager has
procedures for managing user accounts, groups, and authorization
rules.
•
Trusted Solaris ---This is an extension of Solaris 8. RBAC is used to
restrict the rights of administrators, there is no superuser with all power.
•
Argus Pitbull [Arg]—Least privilege applied to all processes, including
the superuser. The superuser is implemented using three roles:
Systems Security Officer, System Administrator, and System Operator.
[Arg] Argus Systems Group, “Trusted OS security: Principles and practice”, http://www.argussystems.com/products/white_paper/pitbull
[Cam90] N.A.Camillone , D.H.Steves, and K.C.Witte, “AIX operating system: A trustworthy computing system”, in
IBM RISC System/6000 Technology, SA23-2619, IBM Corp., 1990, 168-172.
Secure Systems Research Group - FAU
Execution Rings
• Some hardware architectures define a set of
protection rings (typically 4 to 32) with
hierarchical levels of trust.
• Rings are a generalization of the concept of mode
of operation and define finer domains of
execution.
• They have been used in Multics and the Intel
processors among others.
• A combination (process, domain) corresponds to
a row of the access matrix.
• Crossing of rings is done through gates that
check the rights of the crossing process. A
process calling a segment in a higher ring must
go through a gate.
Secure Systems Research Group - FAU
Execution Rings
0 = kernel
1 = OS functions
2 = safe applications
3 = untrusted applications
3
2
1
0
- Calls upward
(higher privilege)
- Data access toward
less privilege
- Gate crossings
- Protected entry points
Secure Systems Research Group - FAU
Execution Rings
• In this case, processes are assigned to rings
based on their level of trust; for example, if we
had four rings we could assign them in
decreasing order of privilege to: supervisor,
utilities, trusted user programs, untrusted user
programs. The Intel X86 architecture applies two
rules:
– Calls are allowed only in a more privileged direction, with
possible restriction of a minimum calling level.
– Data in a segment at level p can be accessed only by a
program executing at a more privileged level (<= p).
Secure Systems Research Group - FAU
Execution Rings
•
The Program Status word of the process indicates its current ring and data
descriptors also indicate their assigned rings. These values are compared
to apply the transfer rules.
•
Ring crossing applies protected entry points. A process calling a higherprivilege process can only enter this process at pre-designed entry points
with controlled parameters.
•
Jumping many rings is suspicious. Multics defined a call bracket, where
calls are allowed only within rings in the bracket. More precisely, for a call
from procedure i to a procedure with bracket (n1, n2, n3) the following
rules apply: if n2<i<=n3 the call is allowed to specific entry points; if i>n3
the call is not allowed, if i < n1 any entry point is valid. This extension only
makes sense for systems that have many rings.
•
Rings don’t need to be strictly hierarchic, partial orders are possible and
convenient for some applications. For example, a system including a
secure database system could assign a level to this database equal but
separated from system utilities; the highest level is for the kernel and the
lowest level is for user programs. This was done in a design involving an
IBM 370 [Fer78].
Secure Systems Research Group - FAU
Secure process communication
• Process communication also has an effect
on security.
• Systems that use explicit message
passing have the possibility of checking
each message to see if it complies with
system policies.
Secure Systems Research Group - FAU
Protected entry points
• Process communication also has an effect on
security.
• Systems that use explicit message passing have
the possibility of checking each message to see if
it complies with system policies.
• For example, a security feature that can be
applied when calling another process is protected
entry points.
• A process calling another process can only enter
this process at pre-designed entry points. This
prevents bypassing entry checks.
• The number and size of arguments in a gate
crossing can also be controlled (this may protect
against some types of buffer overflow attacks).
Secure Systems Research Group - FAU
Distributed directory
• Windows 2000 introduced the concept of Active
Directory (AD) that provides an administrative
structure for distributed domains.
• The AD is a tree that acts as a repository or
registry for all resource information. A domain is
a group of systems that share the same directory;
in this sense the AD is a directory of directories
(the Composite pattern again !).
• Each domain has a unique name and provides
centralized administration for user and group
accounts. Users can cross domains after being
properly authenticated.
Secure Systems Research Group - FAU
Distributed directory
• The AD introduces the concept of dynamic
inheritance of rights along the tree. This is a
direct application of the concept of implied
authorization that we proposed in 1975 [Fer75].
• Implied access rights are very convenient for
system administration because the administrator
can understand better the structure of rights in
the system.
• W2000 also improved and expanded the use of
groups for security and introduced the use of
Kerberos for distributed authentication
Secure Systems Research Group - FAU
Isolated execution environments
•
•
•
•
Virtual machines
Tagged storage
Capabilities
Separation by encryption
[Dal01] C. Dalton and T.H. Choo, “An operating system approach to
securing services”, Comm. of the ACM, vol. 44, No. 2, February 2001,
58-64.
Secure Systems Research Group - FAU