Graybox NFS Caching Proxy
By:
Paul Cychosz and
Garrett Kolpin
What are Graybox Techniques?
• First we need to know a little about how
the “graybox” works.
• Inference by observation
• No server or client-side modifications
necessary
Our Problem
• Figure out what the client is caching
• Maintain low overlap between client and
proxy caches
Our Solution
• Treat the client as a graybox
• Monitor NFS traffic
• Infer from NFS traffic the contents of client
cache
Conclusions
Cache overlap decreased by 5% to 50%
when using graybox techniques
Outline
• Approach
• Results
• Conclusions
Approach
NFS specifics:
 Implemented in kernel as another FS type
 Replaces “file_operations” with its own functions
for read, write, open, etc.
 Uses VFS abstraction
VFS
RPC / XDR
NFS kernel code
retrans., data
wrapping
TCP / UDP
Approach
Client cache / Replacement
• Caches on page level
• Not much NFS specific code – Uses global
Linux page cache
• Not a static size, no upper-bound
• Timestamps: “jiffies”, per page.
Approach
Reads typically > ~85% of traffic
NFS / RPC “read” structure
• RPC Packets sent out: access to dir, getattr to
file, access to file, then read, read, read, …
• Client use of getattr to revalidate cached
copies
 Exploit this, proxy explicitly look for
these to determine client cache contents
Approach
Proxy
Sees:
 File handle
 Offset
 size
--becomes our “cache element”
 Store in hash table
 Cache on reads, run our cache element replacement
policy on getattr (proof of cached client copy)
Approach
NFS Client
Proxy
Replacement
Read system call
(possible force-out)
Revalidate data
Cache lookup
OR
Read setup
read
Lookup
OR
Return data
Typical read()
Forward req.
to server
Add
Viewing Client Cache Contents
Test Environment:
• Disable swap space
• Use mincore() to view pages that client
has cached
• Use “balloon” program to keep client
cache contents manageable
Approach
Difficulties
 Logistics: Initial setup, client/server/proxy all on
same PC – doesn’t work
 Client cache size changes
-solved: compare percentages, normalize
 Repeated RPC calls not idempotent
 Not a significant problem, proxy efficiency
degrades linearly with NFS performance if
network traffic is bad
Approach
Difficulties
 Ambiguities:
 Different FS activities DON’T generate unique
RPC patterns
 getattr used for a few things besides reads
-- Null RPC packets
Cannot tell exact blocks client caching,
only which files
Need second, local replacement policy on proxy!
Approach
Cache Replacement
Global Policies
• LRU
Local Policies
• Forcing entire file
• MRU
• MRU
Outline
• Approach
• Results
• Conclusions
What We Tested
• Control Experiments
– Global LRU, no local force-out
– Global MRU, no local force-out
• Combinations
– Global LRU, whole file force-out
– Global LRU, block-level local MRU
– Global MRU, whole file force-out
– Global MRU, block-level local MRU
How We Tested
• Used Postmark benchmark
• 12 files with sizes randomly selected
between 4 and 50 kilobytes
• Gives us more of a “real-world” workload
Results
LRU Global Replacement
120
100
Percent
80
Proxy Cache Fill
60
Overlap
40
20
0
NF
WF
Force-out Method
MRU
Results
MRU Global Replacement
120
100
Percent
80
Proxy Cache Fill
60
Overlap
40
20
0
NF
WF
Force-out Method
MRU
Results
All Combinations
120
100
Percent
80
Proxy Cache Fill
Overlap
60
40
20
0
LRU NF
MRU NF
LRU WF
LRU MRU
MRU WF MRU MRU
Replacement Combination
Outline
• Approach
• Results
• Conclusions
Conclusions
• Decrease in overlap by between 5 and 50
percent as compared to control tests
• MRU global file replacement is good by
itself
• MRU global policy with whole-file local
policy gives highest cache fill and lowest
overlap
Questions? Feedback?
?
?
?