Implementing an Untrusted
Operating System on Trusted
Hardware
Introduction

Tamper-resistant software




Combat piracy
Mobile code
However, SW-based solutions are easier to
attack than HW-based solutions
To implement tamper-resistant hardware

Challenge: use untrusted OS to manage trusted
HW
XOM: eXecute Only Memory

HW support for tamper-resistant software


Does not trust OS and memory content
Many design implications




Time-slicing
Process forking
User-level signal handling
Prototyped via SimOS
The XOM Trust Model


Protect against attackers who have physical
access to the HW
Does not trust OS



OS performs only resource management
Only can perform denial-of-service attacks against
its applications
Does not trust main memory

Values stored in memory are encrypted, along
with their hashes of values and virtual addresses
Implications of the XOM Trust
Model

XOM prevents programs from tampering with
each other by placing them in separate
compartments


Enforced by HW data tagging and cryptography
OS runs in a separate compartment


OS should be able to manage resources without
having to interpret data values in transit
OS cannot read or modify process data
Related Work

Secure booting




CPU has a tamper-resistant secret
Used to authenticate the next layer (e.g.,
firmware)
Which in turn authenticates the next layer
XOM trusts neither OS nor memory


OS bugs cannot undermine the security of
applications running on it
XOM can detect tampering of code or data at all
times
The Original XOM Architecture

XOM uses tamper-resistant HW to protect a
master secret that is unique for each
processor


Which is used to encrypt keys that encrypt various
software
All operations that use the master secret must be
implemented on processor
The Original XOM Architecture

Master secret is used to support
compartments

Each is immune to modification and observation
Implementing Compartments (1)

Use cryptography to implement
compartments




XOM HW holds the private key
Each compartment uses XOM’s public key to
encrypt its compartment key
Which is used to encrypt the compartment content
Compartment keys are stored in an encrypted
program
Implementing Compartments (2)

Implications on SW distributions




A distributor must encrypt its software key with the
public key of a XOM processor
Occurs when a user registers the software
Data generated by a program are isolated in
its compartment
Data are encrypted when they leave the CPU
chip

Cached data are stored in plain text
Implementing Compartments (3)

XOM Key Table



Maps compartment keys and ownerships
If a piece of data is encrypted, it’s in a
compartment
NULL compartment


Not encrypted
Insecure data sharing among programs
Implementing Compartments (4)

To protect against tampering of data in
memory



Use cryptographic hash to verify data integrity
A hash is generated each time a cache line is
written to memory (a function of the cache line
and its address)
Both hash and the cache lines are encrypted
Implementing Compartments (5)

New XOM HW instructions are introduced

enter xom




Decrypt the compartment key
Enter the key into the XOM Key Table
All instructions following enter xom must be decrypted
before execution
exit xom


Compartment key to be unloaded from the XOM Key
Table
Return to the NULL compartment
Implementing Compartments (6)

secure store, secure load


For non-NULL compartments
move from NULL, move to NULL
Implementing Compartments (7)
Unencrypted
Code
Encrypted
Code (sym)
Symmetric
Decryption
Module
Asymmetric
Decryption
Module
Currently
executing
XOM ID
and Key
Encrypted
Symmetric
Key
Main Memory
Executable
Code
Private Key
XOM Processor
XOM Key
Table
Enter
Exit XOM
XOM
Handling Program State (1)

Since OS cannot access registers tagged by
other compartments


Need a new way to save and restore contexts
New XOM instructions

save register



Encrypt the register
Create a hash of the register
Store both to memory
Handling Program State (2)
Hash
Data
Look up
program
key based
on Tag
Tag
Data
Store encrypted
data and hash to
memory
Encrypt
Data
OS XOM
ID
XOM Key Table
Handling Program State (3)

restore register




Takes an encrypted register value and hash
Verifies the hash
Restores them back to the original register, setting their
ownership tags appropriately
To prevent the OS from replaying encrypted
register values/hashes

Revoke the register key used to encrypt and hash
register values each time a XOM compartment is
interrupted
Handling Program State (4)

To protect memory from replay attacks


Store a hash of memory in a register
XOM currently rely on applications to track
memory hashes




Otherwise, too expensive in terms of performance
(25%)
Each virtual address needs a hash tree
Virtual to physical address mapping maintained by
OS
XOM needs to interact with OS closely
Handling Program State (5)
Registers
Caches
Memory
Hash
Data
Tag
Data
Tag
Secure Store:
Tag is copied
from register
to cache
Data
Writeback:
Look up Tag,
Encrypt and
Hash
XOM Key Table
Relocate data and
hashes together
Handling Program State (6)


Incorrectly written program can still leak
secrets
XOM assures that correct programs can
secure their secrets
Supporting an Operating System (1)

Requirements



Good OS should be able to manage resources
efficiently
Bad OS cannot violate the isolation of a
compartment
An OS must adhere to XOM compartments when
moving resources

Need to use XOM instructions
Supporting an Operating System (2)

XOM limitations


External debugging impossible
Shared memory and IPCs have to go through the
NULL compartment

They need to be secured through separate mechanisms
XOM Key Table System Calls (1)

enter xom and exit xom insufficient when
frequently called



Encryption/decryption everytime
If enter xom is privileged, a program has to cross
the kernel boundary every time
If enter xom is not privileged, a malicious
application can allocate all key entries to deny
services
XOM Key Table System Calls (2)

enter xom is splitted into multiple calls

xalloc



Privileged
Allocate a register key
xentr


Not privileged
Enter a compartment
XOM Key Table System Calls (2)

Similarly for exit xom

xinval



Privileged
Mark the register key entry invalid
xexit

Return to the NULL compartment
Saving and Restoring Context



An OS needs xgetid to decrypt the ownership
of a cache line
Exception frame must be enlarged to
accommodate additional XOM and encrypted
states
Certain things are revealed to the OS


TLB misses
Status bits that indicate whether a thread is in
kernel mode
Paging Encrypted Pages



Hashes are stored separately
A bad OS needs to forge both data and the
corresponding hash
Hashes are paged in and out with the
corresponding data
Shared Libraries


Unencrypted for dynamic linking by various
applications
Libraries that perform security sensitive
routines should be statically linked and
encrypted

OpenSSL
Process Creation

Need xom_fork()


Create two sets of register keys
Share the same compartment key
Process Creation
Child
XOM ID
Process
Process
XOM ID
XOM ID XOM ID
Pre-fork:
Allocate a new
XOM Key Table
Entry
Fork: Child uses
one XOM ID, the
Parent uses other
Parent
XOM ID
User Defined Signal Handlers


Need to enlarge the signal context field for
encrypted data
Signal key must be refreshed at each call to
avoid replay attacks by an OS
Implementation Effort


Added 1,900 lines to the IRIX 6.5
Ran applications in the SimOS simulator
Basic Processor Model Parameters

Encryption added 15 cycles to the memory
access time


On cache misses and flushes
xalloc required 400,000 cycles to perform
public key decryption
Operating System Overhead


Larger memory footprint
Additional I/O for storing and retrieving
hashes
End-to-End Application Overhead


Turns out that granularity has little impact on
performance overhead!
Overheads for most applications turn out to
be less than 5%

If cache behavior is similar
Questions….



What if the CPU is dead?
And you want to move a hard drive to another
machine?
Backups are useless unless the original
private key is also duplicated.…in
hardware…