YFS: An Introduction to the next /afs®
Jeffrey Altman, Daria Brashear, Marc Dionne, & Simon Wilkinson
Your File System Inc. and Your File System Ltd.
2014 European AFS and Kerberos Conference

Your File System Inc. (YFSI) is a New York State
Corporation with HQ in Manhattan and
registered as a business entity in Canada
Your File System Ltd is a wholly owned
subsidiary of YFSI with HQ in London
YFSI is privately owned and operated

YFSI is a Red Hat Partner ISV


 Location Transparency: one name space
 User Mobility: access from any device
 Security: Flexible model for authentication, privacy, data







protection and access control
Availability: Temporary loss to small groups for short time
periods
Integrity: No user initiated backups
Heterogeneity: Multiplatform
Self service: Low Help Desk costs
Atomic Publishing: Software, documentation, web sites, ..
Real time collaboration: Distributed File Locking
Distributed Administration
The vision was decades ahead of its time in 1983
The implementation is decades behind in 2014






Limited network throughput
Increased call processing latency
Decreased service reliability and availability
Elevated risk of distributed deadlocks
Inability to use full capability of available hardware
Failure to keep up with competing technologies
That /afs is still in use today is a credit to its vision and
the strength of its architecture.




Major system rewrites are few and far
between
“Contractor Model of Support” leads to many
small and localized changes
A lack of consistent vision and quality control
Few incentives to invest in the next 30 years

Application Transparency
•


Be a Tier One file system on all major OSes
Embrace multi-producer, multi-consumer
work flows
Extended Integrity: Disaster Recovery

Be performance competitive
•



Lustre, GPFS, Panasas, …
Best of breed data security
Improved Ease of Use
Designed for the long term
 Improved
performance with existing hardware
 Cost
reductions due to hardware consolidation
 Zero
data loss as part of a transition
 No
flag day required
• Mixed deployments are encouraged



Performance issues restrict the jobs that
sites are willing to run in /afs
Deploying excessive hardware to solve load
distribution and fairness problems is
expensive
Support for multiple file systems costs
money, requires additional staff, can result in
data duplication and out of sync issues

Reduced contention in the listener thread
•

255 packet window size (per call) without
degradation
•

10 gbit network interface saturation
Order of magnitude faster on high latency links
Dynamic Thread Pools
•
Thread Count limited by OS resources
 64-bit
volume IDs
 96-bit (79 octillion) vnode IDs
 64-bit,100ns granular timestamps
• 2038 ready
 Ubik
databases extensible up to 16 exabytes
 Partitions, volumes and quotas tracked up to
16 zettabytes


Optimized Cache Manager handshakes
Volume Status Information
Reduces number of GetVolumeStatus RPCs
• Permits RW / RO data cache sharing
• Improved caching of RO volume per user permissions
•

Fewer FetchStatus RPCs for RO volumes

Host and callback package rewritten
•
Significantly faster callback breaks

Vnode lock contention dramatically reduced

Distributed writing to shared data sets now
possible




Open mode supported on some OSes
Bypasses VFS cache and AFS cache for both
read and write
No file threshold to tune
Data is copied directly to the caller, or directly
from the caller to the file server

Data breaches and exposures are followed by
a high cost
•
•
•
•
•
Public Relations Nightmare
Costs of Identity Theft Detection Services (in U.S.)
Loss of employment for key staff members
Organizational reorganization
Disruption of core mission when forced to address
security concerns in crisis mode

Multi-layered policies
Flexibility for self service end users
• System administrator controls
•



Network Security
Reduced Information Exposure
Minimal Privilege Services


Self Service Group Management
Per-Object ACLs
•
Cross directory hard links now permitted

Volume ACLs
•
Limits the permissions that end users can grant

Volume Security Policy
•

Per-Volume minimum acceptable rx connection
security properties
File Server Security Policy
Per-server minimum acceptable rx connection
security properties
• Only volumes with weaker or equivalent security
policy can be attached, moved to, or restored to.
•

YFS RXGK Security Class
GSS Kerberos 5 authentication
• AES-256 wire privacy and integrity protection
•



Cell wide key for DB servers
Individual keys for file servers
Per-host keys for BOS Overseer Service

YFS protects the callback channel with AES256 privacy and integrity protection
when rxgk is used for the incoming connection
• Avoids leaking information about volume and file ids
accessed by a client
• Prevents forged messages from invalidate callback
state
•

Server Processes execute under a daemon
account
•
Not Root

Cache Managers can be issued
a Kerberos keytab
• a Protection DB Machine ID
•


Keyed Cache Managers can use privacy for all
connections
Machines IDs are similar to User IDs
Can be placed on ACLs and added to Groups
• But are not included in system:authuser
•
File System Extensions
 Per
File ACLs
 Cross directory hard links
Extensions for Microsoft Windows
 Mandatory Locks
• Advisory locks are not enforced by the file server
 Symlink
Updates
• Reparse Points can be updated without FileID change
 CreateFile with Lock
• Avoids races when simulating OpLock semantics
Command Output Clarity
• Modifications to human readable and machine
readable output
• vos examine, listvol, rxdebug, xstat_fs, …
 Consolidate output
 Introduce consistency across command options
• Machine readable output –format is not human
formatted
 All fields are now separated by single tabs
 Easy to import into spreadsheets and databases
Library Cleanup
• All libraries are thread safe
• Built using libtool
• Intended for use implementing language bindings
libacquire
• A library to obtain tokens
 rxkad
 yfs-rxgk
• aklog is a wrapper
• Can be linked to pam modules
Automated Windows Domain Token
Acquisition
 Triggered
by access denied errors
 Automatic Token acquisition using Logon
Session Kerberos Credentials
 Works with all applications that use
• WNet API: Network Providers
• Shell API: Explorer, Office, anything with an Open
dialog




Simplify Server Configuration
Provide Extensibility for New Features
BOS command lines are limited in length
Permit the construction of flexible test suites
Greatly improved configuration flexibility and
convenience
Custom file layouts are possible
All settings centralized in a single configuration area,
single file or directory
A configuration directory can ease distribution of
custom options
All command line options can be set in configuration
Goal
Provide a test for every new feature, library function,
RPC
Provide a test with every bug fix, if possible
Requirements
Ability to spin up the various servers and provide a test
configuration
All tests must be able to run as a regular user
Must be able to serve test data not necessarily under
/vicep*
Extensive test suite coverage




A complete test cell can spin up in a few
seconds
Many tests spin up a cell and destroy it when
done, maintaining test independence
Client testing through libafscp and fuse client
All new features require tests before merging
Sample systemd yfs-server.service
[Unit]
Description=YFS Server Service
After=syslog.target network.target
[Service]
EnvironmentFile=-/etc/sysconfig/yfs
ExecStart=/usr/local/sbin/bosserver -config
/s/yfs/server/yfs-server.conf -nofork
ExecStop=/usr/local/bin/bos -config
/s/yfs/server/yfs-server.conf shutdown
hurricane.marcdionne.net -wait -localauth
User=yfs
Group=yfs
[Install]
WantedBy=multi-user.target
Sample file layout
[marco@hurricane /s/yfs/server ]$ ls -l
total 60
drwxr-xr-x. 2 yfs yfs 4096 Mar 23 04:00
-rw-r--r--. 1 yfs yfs
526 Jul 15 2013
-rwxr-xr-x. 1 yfs yfs
26 Jul 15 2013
drwxrwxr-x. 6 yfs yfs 4096 Jan 11 15:36
drwxrwx---. 2 yfs yfs 4096 Oct 25 10:47
-rw-r--r--. 1 yfs yfs
4 Jan 6 09:52
-rw-r--r--. 1 yfs yfs
144 Jan 6 09:52
drwxrwx---. 2 yfs yfs 4096 Mar 25 10:28
drwxrwxrwx. 2 yfs yfs 12288 Mar 25 10:29
-rw-r--r--. 1 yfs yfs
15 Sep 12 2013
-rw-r--r--. 1 yfs yfs
114 Dec 19 16:56
-rw-r-----. 1 yfs yfs 2000 Aug 5 2013
drwxrwxr-x. 2 yfs yfs 4096 Mar 26 18:25
bos
bos.keytab
cacheinfo
data
db
KeyFile
KeyFileExt
local
logs
ThisCell
UserList
vl.keytab
yfs-server.conf
[marco@hurricane /s/yfs/server ]$ ls -l yfs-server.conf/
total 8
-rw-r--r--. 1 yfs yfs 645 Mar 26 18:25 cellservdb.conf
-rw-rw-r--. 1 yfs yfs 792 Mar 4 15:48 yfs-server.conf
Sample cellservdb.conf
[cells]
grand.central.org = {
example.com = {
description = "GCO Public CellServDB 23 Apr 2008"
description = "Test cell"
servers = {
servers = {
penn.central.org = {
blizzard.marcdionne.net = {
addr = 128.2.203.61
addr = 192.168.0.113
}
}
grand.mit.edu = {
}
addr = 18.9.48.14
}
}
marcdionne.net = {
andrew.e.kth.se = {
description = "Marc's cell"
addr = 130.237.48.87
servers = {
}
hurricane.marcdionne.net = {
}
addr = 192.168.0.107
}
}
}
}
Sample server configuration
[dirpath]
SERVER_ETC_DIR = /s/yfs/server
SERVER_DB_DIR = /s/yfs/server/db
SERVER_LOGS_DIR = /s/yfs/server/logs
SERVER_BOSCONFIG_DIR = /s/yfs/server/bos
SERVER_LOCAL_DIR = /s/yfs/server/local
SERVER_PART_PREFIX_DIR = /s/yfs/server/data
[vlserver]
keytab = /s/yfs/server/vl.keytab
auditlog = /s/yfs/server/logs/audVl
[fileserver]
d = 125
p = 200
nojumbo =
auditlog = /s/yfs/server/logs/audFile
security = yfs-rxgk:crypt rxkad:clear rxnull rxkad:crypt
[bosserver]
auditlog = /s/yfs/server/logs/audBos
[volserver]
d = 125
auditlog = /s/yfs/server/logs/audVol
[ptserver]
auditlog = /s/yfs/server/logs/audPt
[salvager]
auditlog = /s/yfs/server/logs/audSalv
[salvageserver]
auditlog =
/s/yfs/server/logs/audSalvserv



Installation is the initial experience an end
user has with the product
If the installation process is frustrating, the
end user is likely to be unhappy with the
product
Lack of digital signatures can block the
installation of a package or trigger a
frightening dialog

New installation packages
Windows
• OSX
• Linux
•



Debian
Fedora
RHEL6 and RHEL7

Single installer
64-bit and 32-bit components
• Heimdal Side by Side Assembly
• Heimdal Command Line tools
• Automatic Cache Sizing
•

All components digitally signed
•
Microsoft Cross Signing of Drivers



Flat package
Integral packages for client, server and
development
Digital signatures on the package, the kext
and the binaries using Apple-issued
certificate



New packaging for Debian, Fedora and RHEL
Integral packages for client, db services, and
file service
Digital signatures on installation packages
Dual Protocol Stacks
 Allows
advanced features while maintaining
backwards compatibility with AFS®
 AFS protocol suite has all of the capabilities
and limitations of OpenAFS
 YFS features only available on YFS protocol
suite
• rxgk, file server, vol server, vl server, pt server
enhancements
 Transparent
negotiation of protocol suite
Mixed Mode Cells
 Two
cell types are defined:
• AFS cell deploys OpenAFS or IBM AFS vlservers
• YFS cell deploys the YFS location server
 OpenAFS
and YFS File Servers can be joined to
either cell
YFS Client in AFS Cell
 No
support for RXGK, AES-256
 No support for file server security policies
YFS Server in AFS Cell
 Improved
 No
RX Performance for writes
Rxgk
 Volume IDs restricted to AFS limits
 Security Policies cannot be enforced
 Only AFS compatible capabilities can be
registered
 IPv6 addresses cannot be registered
YFS File Server in AFS cell
vos
AFS
vlserver
AFS volume
format
AFS
fileserver
YFS
fileserver
YFS Server in AFS Cell
 Improved
 No
RX Performance for writes
Rxgk
 Volume IDs restricted to AFS limits
 Security Policies cannot be enforced
 Only AFS compatible capabilities can be
registered
 IPv6 addresses cannot be registered
AFS Client in YFS Cell
 No
support for RXGK, AES-256
 No support for file server security policies
 Volumes with ID above 232-1 inaccessible
 Mandatory locks cannot be requested but will
be enforced
 Volume sizes and quotas >231KB will be faked
 Other restrictions as required to enforce
security policies
AFS File Server in YFS cell
YFS
location
server
vos
AFS volume
format
AFS
fileserver
YFS
fileserver
YFS volume
format
5
9
AFS and YFS Volserver Compatibility
 RW
volumes on YFS server cannot be
replicated to AFS server
 Volumes containing YFS tags cannot be moved
to an AFS server
• ACL Data
• Volume Attributes (ACL or Security Policy)
 Data
transfers protected with Rxkad and Fcrypt
 RX performance improved in YFS to AFS
direction
YFS POSIX attribute backend store
YFS protocol suite
AFS protocol suite
6
1
64 bit Ubik database
YFS protocol suite
AFS protocol suite
6
2
64 bit Ubik database
YFS protocol suite
AFS protocol suite
rxgk
keyserver
6
3
Documentation
 Updated
man pages
 New Quick Start Guides
 Updated Administrator’s Guide
Export Licenses
 The
U.S. Government has classified YFS 1.x as
a mass market product
 Worldwide Export permitted with a few
exceptions
 No export restrictions on distribution by
customers
YFS 1.0 Binary License
A
full suite of clients and servers
• Windows
• OSX
• iOS
• RHEL5, RHEL6, RHEL7
• Fedora
• Debian
• Solaris
• AIX
Support
 Free
updates to new releases (one year)
• Every four month release cadence
 Free
security updates (two years)
 Unlimited e-mail / web support (one year)
 Cell performance evaluation (once per year)
 Remote monitoring service (one year)
Products
Cell (no replication)
1 Server (DB and File)
Base cell (replication)
4 DB Servers
4 File Servers
1000 User or Machine IDs
Unlimited Client devices
Additional Servers
Additional IDs
Annual purchases continue support
Non-redistribution Source code license available
Training (on-site or web)
Availability
 General
 First
Availability End of May 2014
update, September 2014
2014 Road Map
 Feature Priorities
• IPv6 enhancements
• Rapid Partition Relocation
• Extended Volume Names
• New Directory Format
 Unlimited Directory Sizes
 Extended Attributes
 Alternate Data Streams
• Read/write Replication
• Extended Callbacks