Bypass:
A tool for building
distributed systems
Douglas Thain and Miron Livny
Computer Sciences Department
University of Wisconsin-Madison
{thain|miron}@cs.wisc.edu
http://www.cs.wisc.edu/condor/bypass
Building distributed systems is
hard.
www.cs.wisc.edu/condor
Bypass makes building split execution
systems easy.
Bypass is to split execution systems
as
yacc is to compilers
www.cs.wisc.edu/condor
Problem:
Unfriendly Machines
› Many systems can distribute your jobs
to available machines scattered around
the world. (rsh, Condor, Globus, etc...)
› But... the machines you have access to
may not be properly equipped to run
your job.
www.cs.wisc.edu/condor
Problem:
Unfriendly Machines
› An unfriendly machine…
allows you to login under some identity.
allows you to execute your program.
might not have your files or a shared file
system!
might not have space for your output!
might be a different architecture or OS!
• (If you want to use a lot of machines, you can’t be picky!)
www.cs.wisc.edu/condor
HELP!
core dumped
foreign
machine
home
machine
www.cs.wisc.edu/condor
Solution: Split Execution
› General strategy:
An agent process traps some of the
application's standard library calls.
Some of the calls can be executed at the
foreign machine.
Some of the calls are sent via RPC back to the
home machine.
A shadow process at the home machine
executes the RPCs and sends the results back
to the agent.
www.cs.wisc.edu/condor
Solution: Split Execution
Shadow
Local
system
calls
Remote
system calls
Trapped
system
calls
Agent
Local
system
calls
Application
Kernel
Kernel
Home Machine
Foreign Machine
www.cs.wisc.edu/condor
Split Execution is an
Open Research Topic
› We want to explore many possibilities:
Foreign machine could be partially friendly –
has some needed resources, but not all.
Data may be buffered and cached at both the
agent and the shadow.
What procedure calls to trap depends on the
application and the services needed.
Some procedure calls could be routed to third
parties such as file servers.
…
www.cs.wisc.edu/condor
Problem:
Split Execution is Hard
› One example of many: Trapping stat()
Different data types:
• struct stat, struct stat64
• Depending on system, integer elements are 2->8 bytes
Multiple entry points:
• stat, _stat, __libc_stat
Surprises:
• #define stat(a,b) _fxstat(VERSION,a,b)
www.cs.wisc.edu/condor
Solution: Bypass
› Bypass takes a specification of a split
execution system and produces a matched
shadow and agent.
› Bypass hides all of the ugly details of
trapping, type conversion, and RPCs.
› Bypass lets you:
split any dynamically-linked application.
transparently use heterogeneous systems.
trap calls with minimal overhead.
control execution paths with plain C++.
www.cs.wisc.edu/condor
Just like
home!
foreign
machine
home
machine
www.cs.wisc.edu/condor
Bypass Language
› Declare what procedures to trap in C++
› Annotate pointer types with data flow.
Direction: in, out, or in out
Binary data: give expression yielding the
number of bytes to send/receive.
› Give two function bodies:
agent_action
shadow_action
www.cs.wisc.edu/condor
ssize_t write
(
int fd,
in "length" const void *data,
size_t length
)
agent_action
{{
if( fd==1 ) {
return bypass_shadow_write(fd,data,length);
} else {
return write(fd,data,length);
}
}}
shadow_action
{{
printf("remote data: %s", data );
}}
;
www.cs.wisc.edu/condor
Agent Action
› Any arbitrary C++ code.
› When the program invokes write(), the
agent_action is executed at the home
machine.
› Within the agent_action:
write() - Invoke the original write() at the
foreign machine.
bypass_shadow_write() - Invoke the
shadow_action via RPC.
www.cs.wisc.edu/condor
Shadow Action
› Any arbitrary C++ code.
› If the agent decides to invoke the RPC
to the shadow, the shadow_action is
executed at the home machine.
› Within the shadow_action:
write() - Invoke write() at the home
machine.
www.cs.wisc.edu/condor
Using Bypass
› Run "bypass" to read the specification
and produce C++ source code:
• % bypass -agent -shadow simple.bypass
› The shadow is compiled into a plain
executable.
› The agent is compiled into a shared
library.
www.cs.wisc.edu/condor
Using Bypass
› The dynamic linker is used to force the
agent into an executable at run-time:
• setenv LD_PRELOAD simple_agent.so
› Procedure calls are “trapped” merely by
putting the agent first in the link list.
› This method can be used on any dynamicallylinked program: tcsh, netscape, emacs…
www.cs.wisc.edu/condor
Example Application:
Complete Remote I/O
› Trap all the standard I/O calls, and send
them to the home machine unmodified:
open(in string char *path, int flags, int mode);
close(int fd);
int read(int fd, out “length” void *data, int length );
int write(int fd, in “length” void *data, int length );
int lseek(int fd, off_t offset, int whence );
www.cs.wisc.edu/condor
Complete Remote I/O
Shadow
open, close,
read, write,
lseek
open, close
read, write,
lseek
Trapped
system
calls
Agent
all
other
calls
Application
Kernel
Kernel
Home Machine
Foreign Machine
www.cs.wisc.edu/condor
Example Application:
Remote Console
› Trap only read and write, and send
operations on standard files back to a
single shadow process.
int read( int fd, in “length” void *data, int length )
agent_action {{
if( fd<3 ) {
bypass_remote_read( fd, data,length );
} else {
return read(fd,data,length);
}
}};
www.cs.wisc.edu/condor
Remote Console
Trapped
system
calls
Agent
Shadow
Standard I/O
reads and writes
all
other
calls
Kernel
read,
write
Foreign Machine
Trapped
system
calls
Trapped
system
calls
Agent
Agent
Kernel
Home Machine
Application
all
other
calls
Application
all
other
calls
Kernel
Foreign Machine
www.cs.wisc.edu/condor
Application
Kernel
Foreign Machine
Example Application:
Attach New Filesystem
› Trap standard I/O calls and replace them
with calls to a user-level filesystem library,
such as Globus GASS.
int open( in string const char *path, int flags, int mode )
agent_action {{
return globus_gass_open( path, flags, mode );
}};
int close( int fd )
agent_action {{
return globus_gass_close( fd );
}};
www.cs.wisc.edu/condor
Attach New Filesystem
open
Globus
Library
THE
GRID
close
more
system
calls
Agent
all
other
calls
Kernel
Foreign Machine
www.cs.wisc.edu/condor
Trapped
system
calls
Application
Bypass can be used by
Real Users!
› Bypass works on unmodified executables.
(Real Users are not willing/able to
rewrite/recompile their programs.)
› Bypass requires no special privileges.
(Real Users do not have the root password)
› Thus, Bypass allows a Real User to make
good use of a remote cluster without
begging the administrator to configure it
to his/her needs.
www.cs.wisc.edu/condor
Performance
› Overhead of trapping a system call is
very small: 1-4 us
The "trapping mechanism" simply
interposes a few extra function calls.
Small compared to the expense of a real
system call (about 10-70us)
› Remote procedure calls are, as
expected, much slower: about 1 ms
under the best conditions.
www.cs.wisc.edu/condor
Related Work
› “Classic” RPC and XDR:
Define standard integer sizes,
endianness, etc.
Start by defining external protocol,
then produce programming interface
which is not always convenient:
• struct read_results * read_1( int fd, int length );
www.cs.wisc.edu/condor
Related Work
› Bypass:
We are stuck with existing interfaces, so
annotate them to produce a protocol:
• int read( int fd, out “length” void *data, int length );
Do “best effort” conversion to/from
external data format:
• off_t is 4 bytes on some platforms, 8 bytes on others.
• A conversion might fail!
Define canonical values for source-level
symbols:
• O_CREAT has different values on Linux and Solaris!
www.cs.wisc.edu/condor
Related Work
› Hunt and Brubacher, “Detours”
Trap library calls on NT using binary rewriting
– can be applied to any executable.
Make original procedure available through
special “trampoline” call.
Bypass leaves the original entry point intact,
so subroutines need not be re-written to use
the trampoline.
www.cs.wisc.edu/condor
Related Work
› Alexandrov, et al., “UFO”
Use a kernel-level facility to trap all of a
process’ system calls and translate some of
them into WWW operations.
The kernel mechanism is secure and can be
applied to any process.
But… it has a high (7x) trapping overhead and
cannot be applied to procedures that are not
true system calls.
www.cs.wisc.edu/condor
Related Work
› Bypass:
Trapping overhead is very small and can
be performed on procedures that are
not necessarily system calls.
But… can only be applied to dynamicallylinked executables, and is not suitable as
a security mechanism.
www.cs.wisc.edu/condor
Related/Future Work
› A complete remote execution system
needs both methods:
The program owner provides a
lightweight mechanism for creating a
correct split execution environment.
The machine owner provides a
heavyweight mechanism to defend itself
from a (possibly) malicious program.
www.cs.wisc.edu/condor
Complete System
Sandbox
Shadow
open, close,
read, write,
lseek
open, close
read, write,
lseek
Agent
all
other
calls
Application
Kernel
Kernel
Home Machine
Foreign Machine
www.cs.wisc.edu/condor
Future Work
› Multiple agents applied to one application
How to select and invoke the correct agent
action?
› Signal handling
Flow of control is backwards.
› Other implementations
Binary rewriting.
Build specialized linker that understands
multiple definitions of symbols.
www.cs.wisc.edu/condor
Further Questions?
› Douglas Thain
thain@cs.wisc.edu
› Miron Livny
miron@cs.wisc.edu
› Bypass Web Page
http://www.cs.wisc.edu/condor/bypass
› Questions now?
www.cs.wisc.edu/condor