Operating System Security
Andy Wang
COP 5611
Advanced Operating Systems
Outline





Introduction
Threats
Basic security principles
Security on a single machine
Distributed systems security

Data communications security
Introduction




Security is an engineering problem
Always a tradeoff between safety, cost, and
inconvenience
Not much solid theory in the field
Hard to provide any real guarantees


Because making mistakes is easy
And the nature of the problem implies that
mistakes are always exploited
History of Security Problem



Originally, there was no security problem
Later, there was a problem, but nobody cared
Now, there are increasing problems, and
people are beginning to care



Automation
Action at a distance
Technique propagation
Constraints of Practical Computer Security

Security costs




If too much, it won’t be used
If it isn’t easy, it won’t be used
Misuse often makes security measures
useless
Fit the stringency of the measure to the threat
being countered
Security is as Strong as the Weakest Link



Opponents will attack the weakest point
Putting an expensive lock on a cheap door
doesn’t help much
Must look on security problems as part of an
integrated system

Not just a single component
Security Threats





Extremely wide range of threats
From a wide variety of sources
Requiring a wide variety of countermeasures
Generally, countering any threat costs
something
So people counter as few as they can afford
Physical Security



Some threats involve access to the
equipment itself
Such as theft,
destruction
tampering
Physical threats usually require physical
prevention methods
Social Engineering and Security

Computer security easily subverted by bad
human practices



E.g., giving key out over the phone to anyone who
asks
Social engineering attacks tend to be cheap,
easy, effective
So all our work may be for naught
A Classification of Threats


Viewed as types of attacks on normal service
So what is normal service?
Information
Source
Information
Destination
Classification of Threat Types




Secrecy
Integrity
Availability
Exclusivity
Interruption
Information
Source
Information
Destination
Interruption Threats




Denial of service
Prevents source from sending information to
receiver
Or receiver from sending request to source
A threat to availability
How Does an Interruption Threat Occur?



Destruction of HW/SW
Interference with communications channel
Overloading a shared resource
Interception
Information
Source
Information
Destination
Unauthorized
Third Party
Another Type of Interception
Information
Source
Information
Destination
Unauthorized
Third Party
Interception Threats




Data or services provided to unauthorized
party
Either in conjunction with or independent of
authorized access
A threat to secrecy
Also a threat to exclusivity
How Do Interception Threats Occur?




Eavesdropping
Masquerading
Break-ins
Illicit data copying
Modification
Information
Source
Information
Destination
Unauthorized
Third Party
Another Type of Modification Threat
3
2
1
Information
Source
Information
Destination
Unauthorized
Third Party
Modification Threats




Unauthorized parties modify data
Either on the way to the users (inject ads with
legit looking links)
Or permanently at the servers
A threat to integrity
How Do Modification Threats Occur?



Interception of data requests
Masquerading
Illicit access to servers/services
Fabrication
Information
Source
Information
Destination
Unauthorized
Third Party
Fabrication Threats




Unauthorized party inserts counterfeit objects
into the system
Causing improper changes in data
Or improper use of system resources
A threat of integrity
How Do Fabrication Threats Occur?



Masquerading
Bypassing protection measures
Duplication of legitimate requests
Active Threats vs. Passive Threats

Passive threats are forms of eavesdropping




No modifications, injections of requests, etc. occur
Active threats are more aggressive
Passive threats are mostly to secrecy
Active threats are to availability, integrity,
exclusivity
What Are We Protecting





Hardware
Software
Data
Communications lines and networks
Economic values
Basic Security Principles


Terms and concepts
Mechanisms
Security and Protection

Security is a policy


Protection is a mechanism


E.g., “no unauthorized user may access this file”
E.g., “the system checks user identity against
access permissions”
Protection mechanisms implement security
policies
Design Principles for Secure Systems







Economy
Complete mediation
Open design
Least privilege
Least common mechanism
Acceptability
Fail-safe defaults
Economy in Security Design

Economical to develop




And to use
Should add little of no overhead
Should do only what needs to be done
Generally, try to keep it simple and small
Complete Mediation

Apply security on every access to an object
that a mechanism is meant to protect


E.g., each read of a file, not just the open
Does not necessarily require actual checking
on each access

Secure session
Open Design


Don’t rely on “security through obscurity”
Assume all potential intruders know
everything about the design

And completely understand it
Separation of Privileges



Provide mechanisms that separate the
privileges used for one purpose from those
used for another
To allow flexibility in the security system
E.g., separate access control on each file
Least Privilege



Give bare minimum access rights required to
complete a task
Require another request to perform another
type of access
E.g., don’t give write permission if he only
asked for read
Least Common Mechanism

Avoid sharing parts of the security
mechanism among different users


E.g. passwords
Coupling users leads to possibilities for them
to breach the system
Acceptability


Mechanism must be simple to use
Simple enough that people will use it
automatically

Example



Cashier register sticker
“If you don’t get a receipt, your meal is free”
Must rarely or never prevent permissible
accesses
Fail-Safe Designs



Default to lack of access
So if something goes wrong/is forgotten/isn’t
done, no security is lost
If important mistakes are made, you’ll find out
about them

Without loss of security
Sharing Security Spectrum





No protection
Isolation
Share all or nothing
Share with access limitations
Share with dynamic capabilities
Important Security Mechanisms





Encryption
Authentication
Passwords
Other authentication mechanisms
Access control mechanisms
Encryption


Various algorithms can be used to make data
unreadable to intruders
This process is called encryption


Typically, encryption uses a secret key known
only to legitimate users of the data
Without the key, decrypting the data is
computationally infeasible
Encryption Example




M is the plaintext (text to be encrypted)
E is the encryption algorithm
Ke is the key
C is the ciphertext (encrypted text)
C = E(M, Ke)
Decrypting the Ciphertext



C is the ciphertext
D is the decryption algorithm
Kd is the decryption key
M = D(C, Kd)
Symmetrical Encryption

Many common encryption algorithms are
symmetrical


I.e.: E = D and Ke = Kd
Some important encryption algorithms are not
symmetrical, however
Encryption Security Assumptions

Assume that someone trying to break the
encryption knows:



The algorithms E and D
Arbitrary amounts of matching plaintext and
ciphertext M and C
But does not know the keys Ke and Kd
Evaluating Security of Encryption



Given these assumptions, and a new piece of
ciphertext Cn, how hard is it to discover Mn?
Either by figuring out Kd or some other
method
What if Mn matches one of the known pieces
of plaintext?
Practical Security of Encryption



Most encryption algorithms can be broken
Goal is to make breaking them too expensive
to bother
How do we protect our encryption?
Key Issues in Encryption

Security often depends on length of key



Long keys give better security
But slows down encryption
The more data sent with a given key, the
greater the chance of compromise

The more data sent with a given key, the greater
the value of deducing it
Encryptions not Enough


Limited possibilities: E(“Buy”, K), E(“Sell”, K)
Reordering of encrypted blocks

Alice sends Bob some encrypted blocks



Eve intercepts and rearranges blocks
Bob deciphers it


E(“L”, K), E(“I”, K), E(“V”, K), E(“E”, K)
EVIL
Statistical regularities

If plaintext repeats, cipher text may too
Encryption is not Enough

Incorporated with file system

Keys should not be a function of block numbers







Risks key reuse (e.g., archival, flash, content shifting
due to insertions, etc.)
C1 = K xor P1
C2 = K xor P2
C1 xor C2 = P1 xor P2 (with much lower entropy)
Random access issues
Key storage issues
Padding issues
Encryption is not Enough

Incorporated with file system



Caching issues (plaintext and ciphertext)
Swapping and hibernation areas
More info


The Code Book
See when cryptography meets storage
Stream, Block Ciphers


M = B1B2…with Bi of fixed length
Block cipher


Stream cipher



E(M, K) = E(B1, K)E(B2, K)…
K = K1K2…
E(M, K) = E(B1, K1)E(B2, K2)…
DES

Bi = 64 bits, K = 56 bits
One-Time Pads





Theoretically unbreakable security
A symmetrical encryption system
Use one bit of key for each bit of plaintext
Never reuse any key bits
Generate key bits truly randomly
Advantages of One-Time Pads


Proved secure (in information theoretic
sense)
Encryption is computationally cheap


XOR message with key
Required procedures for proper use well
understood
Problems with One-Time Pads




They burn keys like crazy
Need to keep key usage in sync
If the keys aren’t truly random, patterns can
be deduced in the bits
Distribution of pads
Authentication

If a system supports more than one user, it
must be able to tell who’s doing what


I.e.: all requests to the system must be tagged
with user identity
Authentication is required to assure system
that the tags are valid

Based on what one knows, what one has, and
what one is
Passwords


A fundamental authentication mechanism
A user proves his identity by supplying a
secret


Either at login or other critical time
The secret is the password
Password Security



Password selection
Password storage and handling
Password aging
Selecting a Password

Desirable characteristics include:




Unguessable
Easy to remember (and type)
Not in a dictionary
Too long to search exhaustively
Password Storage and Handling


Passwords are secrets, so their security
depends on careful handling
But seemingly the system must store the
password


To compare when users log in
If system storage is compromised, so is all
authentication
Securely Storing Passwords




Store only in encrypted form
To check a password, encrypt it and compare
to the encrypted version
Encrypted version can be stored in a file
But there are tricky issues
Tricky Issues in Storing Encrypted
Passwords

What do I encrypt them with?



If I use single key to encrypt them all, what if the
key is compromised?
That key must be stored in the system
What if two people choose the same
password?
Example: The UNIX Password File


Each password has an associated salt
UNIX encrypts a block of zeros




Key built from password plus 12-bit salt
Encryption done with DES
Stored information = E(zero, salt + password)
To check password, repeat operations
How Does This Help the Problems?

No single key for encryption



So can’t crack that key
And needn’t ever store it
Each encryption (probably) performed with a
different key

So two people with the same password have
different encrypted versions
Does this solve the problem?



Not entirely
Passwords exist in plaintext in process
checking them
Passwords may be transmitted in plaintext


Especially for remote logins
Bluetooth keyboards
Problems with Passwords




People choose bad ones
People forget them
People reuse them
People rarely change them
How to Deal with Bad Passwords





Educate users so they choose good ones
Automatic password generation
Check when changed
Periodically run automated cracker
Any solution must balance user needs,
password security, and resources
Other Authentication Mechanisms






Challenge/response
Smartcards
Other special hardware
Detection of personal characteristics
All have some drawbacks
Some are combined with passwords

Two-factor authentication

Something you know + something you own (card)
Zero-knowledge Proof
(Authentication)



Alice and Bob know prime numbers p and g
Alice knows secret x, Bob knows y = gx % p
To authenticate Alice

For n rounds

Alice generates a random number r


Then sends Bob C = gr % p
Bob either asks for r and computes C to match


Or asks for z = (x + r) % (p – 1), and computes gz % p to
see if it matches with Cy mod p
Note that (p – 1) is a relative prime of p
Zero-knowledge Proof



Alice knows the 3-coloring solution to a graph
Bob wants to prove that Alice knows the
solution
In N rounds



Alice shuffles the colors
Bob asks for two adjacent nodes, Alice reveals
the colors of the two nodes
With enough N, Bob can prove Alice’s
knowledge, without learning the solution
Data Access Control Mechanisms


Methods of specifying who can access what
in which ways when
Based on assumption that the system has
authenticated the user
Access Matrix



Describes permissible accesses for the
system
Subjects access objects with particular
access rights
A theoretical concept, never kept in practice
Access Matrix Example
File 1
File 2 Server X
Segment 57
User A Read, Write None Query
Read
User B Read
Write Update
None
User C None
Read Start, Stop None
User D None
None Query
None
Types of Access Control

Discretionary access control (DAC)


Mandatory access control (MAC)


System mechanism controls access to object
Originator controlled access control
(ORCON)


Individual user sets ACL mechanism
Creator of information control ACL
Role-based access control (RBAC)

Bookkeeper has access to financial records
Methods for Implementing Access Matrix

Access control lists


Decomposition by columns
Capabilities

Decomposition by rows
Access Control Lists

Each object controls who can access it

Using an access control list
Add subjects by adding entries
 Remove subjects by removing entries
+ Easy to determine who can access object
+ Easy to change who can access object
- Hard to tell what someone can access

Access Control List Example

File 1’s ACL


User A: Read, Write
User B: Read

Segment 57’s ACL

User A: Read
Capabilities
Each subject keeps track of what it can
access
 By keeping a capability for each object
 Capabilities are like admission tickets
+ Easy to tell what a subject can access
- Hard to tell who can access an object
- Hard to revoke/control access (someone can
keep an extra copy around)

Capability Example

User A’s Capabilities



File 1: Read, Write
Server X: Query
Segment 57: Read

User B’s Capabilities



File 1: Read
File 2: Write
Server A: Update
Other Models of Access Control



Military model
Information flow models
Lattice model of information flow
Bell-LaPadula Model


An example of confidentiality policy
Clearance categories


Top secret, secret, confidential, unclassified
Users can only create and write top secret
and secret documents


Users cannot read documents > their clearance
Users cannot write documents < their clearance
Bell-LaPadula Model

Rationale

Cannot copy a top secret document over a
unclassified one



And email the unclassified one away
Information flows up
Problems



Blind writes
Classifications cannot change
Interacts with capability-based systems (passing a
capability from high clearance to low clearance)
Biba’s Model


An example of integrity policy
The higher the level, the more confidence




That a program will execute correctly
That data is accurate and/or reliable
Note integrity levels ≠ security levels
Assumption

Integrity and trustworthiness
Biba’s Model

Requirements




Users use only existing programs
Programmers will develop and test programs
Program installations are controlled and audited
Managers and auditors have access to both the
system state and the system logs
Biba’s Model

Goal:

Prevent untrusted software from altering data or
other software




Credibility rating based on estimate of software’s
trustworthiness
Trusted file systems contain software with a single
credibility level
Process has risk level or highest credibility level at which
process can execute
Must use run-untrusted command to run software at
lower credibility level
Chinese Wall Model

Problem



Tony advises American Bank about investments
He is asked to advise Toyland Bank about
investments
Conflict of interest

His advice for either bank would affect his advice
to the other bank
Chinese Wall Model




Organize entities into conflict of interest
classes
Control subject access to each class
Control writing to all classes to ensure info is
not passed along in violation of rules
Allow sanitized data to be viewed by
everyone
Writing




Anthony, Susan work in the same trading
house
Anthony can read Bank 1’s CD, Gas’s CD
Susan can read Bank 2’s CD, Gas’s CD
If Anthony could write to Gas’s CD, Susan
can read it

Hence, indirectly, she can read information from
Bank 1’s CD, a clear conflict of interest
Compare to Bell-LaPadula

Bell-LaPadula cannot track changes over
time

Susan becomes ill, Anna needs to take over