EtherPIPE: an Ethernet character device
for network scripting
Yohei Kuga (Keio University)
Takeshi Matsuya (Keio University)
Hiroaki Hazeyama (NAIST)
Kenjiro Cho (IIJ)
Osamu Nakamura (Keio University)
HotSDN’13, August 16, 2013, Hong Kong, China
1
Motivation
• Shell scripting is powerful to process files
• We want to use it for network processing
Our Goal
$ cat /dev/port0 > ./capture
Use Basic UNIX
command
Receive packets
Capture
packets
2
Motivation
• Shell scripting is powerful to process files
• We want to use it for network processing
Our Goal
$ cat /dev/port0 >/dev/port1
Use Basic UNIX
command
Receive packets
Send packets
3
Motivation
• Shell scripting is powerful to process files
• We want to use it for network processing
Our Goal
$ ./FW.sh </dev/port0 >/dev/port0
Build a network
application
Receive packets
Send packets
• It’s useful for SDN prototyping and debugging
4
EtherPIPE
• A character device for packet processing
– It accesses the device using open(2)
– And, handles packets using read(2) and write(2)
Then UNIX commands can read/write packets as a file
write()
read()
/dev/ethpipe
send
receive
5
Network Scripting
• Lightweight network programming by shell
scripts
– Combining UNIX commands, redirections, pipes with
EtherPIPE devices
6
EtherPIPE design: Device namespace
• ‘0’ and ‘1’: Shell Interface
– ASCII format. For shell scripting
• ‘r0’ and ‘r1’: Raw Interface
– Binary format. For high-bandwidth and complex applications
7
EtherPIPE design: Shell Interface (ASCII)
12 words
12 words
4 words
2 words
• one packet per line
• Device driver converts raw data to ASCII
• Each byte is separated by “space” character
• To make string-based UNIX commands to process packets
– But the data size is 3 times larger
8
Shell Interface: Applications
• Sending a packet
$ echo “FFFFFFFFFFFF
E3 FE 32 40 00 22 CF
20 20 20 20 20 20 20
20 20 20 20 20 20 20
0022CF63967B 8899 23 9C 15
63 96 7B 00 20 20 20 20 20
20 20 20 20 20 20 20 20 20
20 20 20 20“ >/dev/ethpipe/0
• Port mirroring
$ cat /dev/ethpipe/0 | tee /dev/ethpipe/0 >
/dev/ethpipe/1
• MAC address filtering
$ grep ‘^FFFFFFFFFFFF’ /dev/ethpipe/0 >
/dev/ethpipe/1
• VLAN untagging
$ sed -e 's/8100 00 01 //' < /dev/ethpipe/0 >
/dev/ethpipe/1
9
EtherPIPE design: Raw interface (Binary)
• For simple capturing and
replaying, and for
optimized applications
• Raw format includes fields
for offloading
– Hardware timestamp
• nanosecond RX timestamp
for precise capturing
– Five-tuple hash for flow
processing
• Hash # <src and dst IP addr,
proto num, src and dst port
num>
10
Prototype implementations
1. Device driver (Shell IF) for generic NICs
– For proof-of-concept
2. FPGA card + device driver
–
–
–
–
Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe)
Supports 1000BASE-T full duplex
Shell IF device driver (working)
Raw IF device driver (under development)
• FPGA logic and drivers source code are available
– https://github.com/sora/ethpipe
11
Prototype implementations
1. Device driver (Shell IF) for generic NICs
– For proof-of-concept
2. FPGA card + device driver
–
–
–
–
Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe)
Supports 1000BASE-T full duplex
RX: 1GE line-rate
Shell IF device driver (working)
TX: near 1GE line-rate
Raw IF device driver (under development)
• FPGA logic and drivers source code are available
– https://github.com/sora/ethpipe
12
How to measure the transmit PPS
• Measured the TX PPS of
Shell IF by `wc`, `cat` and
`time`
• Performs the near 1GE
line-rate PPS
1,481,481 pps vs.
1,488,095 pps for 64B pkts
Enough for prototyping
$ head ./4m.pkt
FFFFFFFFFFFF 0022CF63967B 8899 23 ..
FFFFFFFFFFFF 0022CF63967B 8899 23 ..
FFFFFFFFFFFF 0022CF63967B 8899 23 ..
$ wc ./4m.pkt
4000000 196000000 676000000 4m.pkt
$ time cat ./4m.pkt >/dev/ethpipe/0
real 0m2.700s
User 0m0.004s
sys 0m2.661s
$ bc
scale=3
4000000/2.700
1481481.481
13
Summary
• We proposed “Network scripting”, new design of
network IO for packet processing, and showed a
prototype implementation
– Allows shell scripting to handle network devices w/ ‘<, >, |’
– Simple data format for UNIX commands and scripting
languages
• Future work
– Fix Shell and Raw IF formats
• E.g,: More offloading fields (currently: RX timestamp and 5-tuple
hash)
– Finish Raw IF driver and its userland libraries
14
15
FPGA adapter design
Receiving
Sending
16
Performance transmit PPS of FPGA
prototype: data
$ head 4m.pkt
FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00
22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00
22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
…
17
Host operations (tappipe.c)
• To bind EtherPIPE device with OS network stack
by TAP device
– To use existing network commands (arp, dhclient,
tcpdump, etc)
– Build custom layer 2~3 functions by shell script (ping,
traceroute, nmap, etc)
$ ./tappipe pipe0 </dev/ethpipe/0 >/dev/ethpipe/0 &
$ ip addr pipe0
$ tcpdump -i pipe0
$ dhclient –r pipe0 && dhclient pipe0
$ echo ${PING_REQUEST} > /dev/ethpipe/0
18
Performance result of
command-line based packet processing
• Measuring the potential of Shell IF throughput by dummy driver
– Read(): Dummy driver prepare the 64 Byte packet in kernel
– Write(): the driver simply copies the data into a buffer in the kernel
• Result
– ‘Capture’ (memory copy from kernel to userland) performs over 10GE
– Some command is very slow such as `cut`
19
Our focus
• A typical network device file is designed for
communications between hosts
– A network device file has to provide all functions / APIs
from Layer 2 to Layer 4
– An implementation would be as a special device
– Device specific settings and methods are required to
send/receive packets
• We focus on a network device file for packet processing
– Only provide IO of physical ethernet ports and doesn't care
other functions (MTU, reassemble ,FIB, etc)
– An implementation is developed as a common device file
– Basic system calls are enough to send/receive packets
20