Getting ready for day 2
• Yesterday’s tree was moved to NetFPGA10G-live-BACKUP-Day1/
• IF you edited code
cp NetFPGA-10G-live-BACKUPDay1/projects/crypto_nic/hw/ pcores/crypto/
hdl/verilog/crypto.v \
NetFPGA-10G-live/
projects/crypto_nic/hw/pcores/crypto/hdl/verilog/
nf10_crypto_v1_00_a.v
• Edit your nf10_crypto_v1_00_a.v
– Rename the module to nf10_crypto
Spring Camp 2013
1
NetFPGA Spring Camp
Day 2
Presented by:
Andrew W. Moore
with
Marek Michalski, Neelakandan Manihatty-Bojan, Gianni Antichi
Georgina Kalogeridou, Jong Hun Han, Noa Zilberman
aided by
Yury Audzevich, Dimosthenis Pediaditakis
Poznan University of Technology
May 20 – 24, 2013
http://NetFPGA.org
Spring Camp 2013
2
Previously Covered
•
Infrastructure
– Tree
– Build System
– EDK environment
•
The Life of a Packet Through the NetFPGA
– Hardware Datapath
– Interface to software: Exceptions and Host I/O
•
Implementation
– Module Template
– Write Crypto NIC using a static key
•
Simulation and Debug
– Write and Run Simulations for Crypto NIC
Spring Camp 2013
3
Tutorial Outline
•
Registers
–
–
–
–
•
Explain Register System
Use AXI Lite registers modules to implement register
Add register access stimulus to define Crypto NIC encryption key
Update Simulations
Build and Test Hardware
– Build
– Explanation of Hardware Tests
– Write and run Hardware Tests
•
•
•
•
Verify value: 0xFFFFFFFF
Verify value: 0xFF00FF00
Verify value: 0x55555555
Group Discussion
– Project Ideas
– Scope of work that can be accomplished in 2-3 days
•
Team up for Projects
– Team leaders will describe projects
Spring Camp 2013
4
Crypto Module State Diagram
Detect
Packet’s
Header
Header+
Payload
Payload
Spring Camp 2013
5
Section I: Registers
Spring Camp 2013
6
Specifying the Key via a Register
• Can set the key via a register instead
• Need to understand the register system 
• Register system:
– Specify registers stimulus in the reg_stim.axi
file
– Implement registers in module
• Use instances of ipif_regs and axi_lite_ipif_1bar
• Other instances available as well (2bar, 3bar, table)
Spring Camp 2013
7
Register bus
S_AXI_CLK
S_AXI_WREADY
S_AXI_WVALID
S_AXI_BRESP
S_AXI_BVALID
S_AXI_BREADY
S_AXI_AWREADY
S_AXI_AWADDR
S_AXI_ARREADY
Module
S_AXI_WDATA
S_AXI_RRESP
S_AXI_WSTRB
S_AXI_RVALID
S_AXI_ARVALID
S_AXI_RDATA
S_AXI_RREADY
S_AXI_ARADDR
Spring Camp 2013
8
READ
READ
S_AXI_AWVALID
WRITE
WRITE
S_AXI_ARESETN
ipif_regs Module
// -- IPIF REGS
ipif_regs
#(
.C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH),
.C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH),
.NUM_WO_REGS
(NUM_WO_REGS),
.NUM_RW_REGS
(NUM_RW_REGS),
.NUM_RO_REGS
(NUM_RO_REGS)
)
ipif_regs_inst
(
.Bus2IP_Clk
( Bus2IP_Clk
),
.Bus2IP_Resetn ( Bus2IP_Resetn ),
. . .
.wo_regs
( wo_regs ),
.rw_regs
( rw_regs ),
.ro_regs
( ro_regs )
);
Spring Camp 2013
9
axi_lite_ipif_1bar (in every core)
// -- AXILITE IPIF
axi_lite_ipif_1bar
#(
.C_S_AXI_DATA_WIDTH (C_S_AXI_DATA_WIDTH),
.C_S_AXI_ADDR_WIDTH (C_S_AXI_ADDR_WIDTH),
. . .
.C_BAR0_HIGHADDR
(C_HIGHADDR)
)
axi_lite_ipif_inst
(
.S_AXI_ACLK
( S_AXI_ACLK
),
.S_AXI_ARESETN
( S_AXI_ARESETN ),
. . .
.S_AXI_AWREADY
( S_AXI_AWREADY ),
// Controls to the IP/IPIF modules
.Bus2IP_Clk
( Bus2IP_Clk
. . .
.IP2Bus_Error
( IP2Bus_Error
);
Spring Camp 2013
10
),
)
Additional Related Modules
• axi_lite_ipif_2bar
– Same as axi_lite_ipif_1bar but with support
for 2 address ranges
• axi_lite_ipif_3bar
– Same as axi_lite_ipif_1bar but with support
for 3 address ranges
• ipif_table_regs
– Read and write to a table
Spring Camp 2013
11
Adding Registers Logic (1)
• Registers are arranged in memory in the
following order:
– Write only registers
– Read/Write registers
– Read only registers
• Define the number of registers used of each
type, e.g.:
– localparam NUM_RW_REGS = 2;
localparam NUM_RO_REGS
= 1;
• Use these parameters for the ipif_regs module
– No need to define a parameter that is not used
(e.g. there are no WO registers)
Spring Camp 2013
12
Adding Registers Logic (2)
• Ipif_regs exposes 3 registers busses:
– wo_regs, rw_regs, ro_regs
• Each bus is (NUM_REGS*C_S_AXI_DATA_WIDTH) wide
– Each register is assigned C_S_AXI_DATA_WIDTH bits
• Usage examples:
assign ro_regs = {version_reg,counter_reg1,counter_reg2};
always @(posedge Bus2IP_Clk)
if (~Bus2IP_Resetn) begin
dummy_reg <= 'h0;
end
else begin
dummy_reg <= rw_regs [C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) + 31 :
C_S_AXI_DATA_WIDTH*(DUMMY_REG_ADDR) ];
end
Spring Camp 2013
13
Adding Registers Logic (3)
Registers usage:
• RO, WO, RW refers to software access
• Write only registers can be written only by the
software
• Read only registers are set by hardware and
read by software or hardware
• Read/Write registers can set by software and
read by software or hardware
Spring Camp 2013
14
Testing Registers with Simulation
Spring Camp 2013
15
Simulating Register Access
1. Define register
stimulus
2. The testbench executes the stimulus
system_axisim_tb
reg_stim.axi
DUT
compare
4. A script can
compare expected and
actual values
==
And declare success
or failure
PASS
Spring Camp 2013
reg_stim.log
3. Simulation
accesses are
written to a log file
Legend:
!=
- DUT: Design Under Test
- stim: stimulus
- tb: testbench
FAIL - sim: simulation
16
Registers Stimulus (1)
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.axi

An example of write format :
Address
Data
Byte Enable strobe
Spring Camp 2013
17
Registers Stimulus (2)
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.axi

An example read format :
Address
Spring Camp 2013
18
Registers Access Log
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
vim reg_stim.log
WRITE
READ
Time
Spring Camp 2013
19
Replacing Static Key
• In the crypto project, replace the static key
with the key from the registers
– Provide an Enable registers
• Update your simulations to set the key
Spring Camp 2013
20
To execute a register simulation
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
make simreg
Spring Camp 2013
21
Section II: Build and Test Hardware
Spring Camp 2013
22
Synthesis
• To synthesize your project
CHECK your pao file:
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw/pcores/nf10_crypto_v1_00_a/data/
edit nf10_crypto_v2_1_0.pao
For synthesis
uncomment lines 39,40,43
comment out lines 41 and 42
cd ~/NetFPGA-10G-live/projects/crypto_nic/hw
make
Spring Camp 2013
23
Hardware Tests
• Test compiled hardware
• Test infrastructure provided to
– Read/Write registers
– Read/Write tables
– Send Packets
– Check Counters
Spring Camp 2013
24
Python Libraries
• Start packet capture on interfaces
• Clear all tables in hardware
• Create packets
– MAC header
– IP header
– PDU
• Read/Write registers
• Read/Write reference router tables
– Longest Prefix Match
– ARP
– Destination IP Filter
Spring Camp 2013
26
Hardware Test Examples
from ~/NetFPGA-10G-live/<project>
• reference_nic
– Simple crossover test
• test/hw_external_crossover
• older 1Gbps Reference Router examples
– Packet Forwarding
• test/both_packet_forwarding
– Longest Prefix Match
• test/both_lpm_generic
– Send and Receive
• test/hw_send_rec
Spring Camp 2013
27
Creating a Hardware Test (2)
• Your task:
edit ~/.bashrc and change NF_DESIGN_DIR to be crypto_nic
reopen your Terminal (to force this change)
cd ~/NetFPGA-10G-live/projects/
cp --archive reference_nic/test crypto_nic/.
This creates a directory of hardware tests just like the
reference_nic
cd ~/NetFPGA-10G-live/projects/projects/crypto_nic/test
cp hw_external_loopback hw_crypto_encrypt
• Now edit
hw_crypto_encrypt/run.py
to create your tests.
Spring Camp 2013
29
Running Hardware Tests
• Use command nf_test.py
– Required Parameter
• sim hw or both (right now only use hw)
– Optional parameters
• --major <major_name>
• --minor <minor_name>
both_crypto_encrypt
major
minor
• Run the command
nf_test.py hw --major crypto --minor encrypt
Spring Camp 2013
30
Section III: Interface with Software
Spring Camp 2013
31
NetFPGA-Host Interaction (recap)
– Register reads/writes via ioctl system call
with wrapper functions:
• rdaxi(int address, unsigned *rd_data);
• wraxi(int address, unsigned *wr_data);
eg:
rdaxi(0x7d4000000, &val);
– Useful command line utilities
cd ~/NetFPGA-10-live/projects/crypto_nic/sw/host/apps
./rdaxi 0x7d4000000
./wraxi 0x7d4000000 0x1
Spring Camp 2013
32
Recap
Build a complete NetFPGA design
Learn:
• Module creation (Verilog)
• Reference pipeline integration
• Verification via simulation
• Verification via hardware tests
• Interaction with software
Spring Camp 2013
33
Step 1. Program NetFPGA-10G Guidelines
1) Prepare a bit file, nf10.ko driver for a NetFPGA card
cd ~/NetFPGA-10G-live/projects/crypto_nic/sw/host/driver; make
OR, do your make from the DESIGN directory
cd ~/NetFPGA-10G-live/projects/crypto_nic; make
Reference_nic driver is used for most projects.
2) cd ~/NetFPGA-10G-live/projects/crypto_nic/bitfiles
3) Load a bit file for programming FPGA
~/NetFPGA-10G-live/tools/scripts/impact_run.sh <bit_file_name.bit>
eg
~/NetFPGA-10G-live/tools/scripts/impact_run.sh crypto_nic.bit
4) Reboot machine (only required once each machine power-up)
5) repeat step 4)
Spring Camp 2013
34
Step 2. Program NetFPGA-10G Guidelines
If you are developing new DMA systems steps 6 onward apply
6)To find out loaded bitfile image, run
$ lspci –d *:4244 –vxx
Spring Camp 2013
35
Step 3. Program NetFPGA-10G Guidelines
7) ~/NetFPGA-10G-live/tools/scripts/pci_save_restore.sh save dma
Now, when the FPGA needs to be programmed, run only step 4) that
programs FPGA, restores PCIE configuration, and loads nf10.ko
driver.
Spring Camp 2013
36
Step 4. Program NetFPGA-10G Guidelines
8) Go to ./NetFPGA-10G-live/projects/reference_nic/sw/host/driver
9) Compile nf10 driver
$ make
10) Load nf10 driver by run insmod
$ insmod nf10.ko
Spring Camp 2013
37
Step 5. Program NetFPGA-10G Guidelines
11) Run dmesg to find out the kernel driver
$ dmesg
Spring Camp 2013
38
AppendixImpact (the FPGA loader)
• Xilinx impact has a peculiar behaviour
sometimes (this is a known Xilinx impact
bug)
If the batch mode of impact does not work, a
GUI interface appears….
These next slides will guide you through
using the GUI.
Spring Camp 2013
39
Appendix
Step 1. Use of Impact GUI for Program FPGA
Impact GUI can be used for programming FPGA. This steps shows
how programs FPGA using Impact GUI.
1)Source Xilinx tools
$ source /opt/Xilinx/13.4/ISE_DS/settings64.sh
2)Go to NetFPGA-10G directory and run impact
$ impact
Spring Camp 2013
40
Appendix
Step 2. Use of Impact GUI for Program FPGA
3) Tick ‘create a new project’ and click OK.
Spring Camp 2013
41
Appendix
Step 3. Use of Impact GUI for Program FPGA
4) Tick ‘Configure devices using Boundary-Scan(JTAG) and click
OK
Spring Camp 2013
42
Appendix
Step 4. Use of Impact GUI for Program FPGA
5) Click Yes
Spring Camp 2013
43
Appendix
Step 5. Use of Impact GUI for Program FPGA
6) Select a bit file and click OPEN.
Spring Camp 2013
44
Appendix
Step 6. Use of Impact GUI for Program FPGA
7) Click No.
Spring Camp 2013
45
Appendix
Step 7. Use of Impact GUI for Program FPGA
1) Click Cancel.
Spring Camp 2013
46
Appendix
Step 8. Use of Impact GUI for Program FPGA
8) Click Cancel.
Spring Camp 2013
47
Appendix
Step 9. Use of Impact GUI for Program FPGA
9) Double click ‘Program’
10) This process ONLY loads the FPGA.
Spring Camp 2013
48
Section IV: Wrap-up
Spring Camp 2013
49
Thoughts for Developers
• Build Modular components
– Describe shared registers
– Consider how modules would be used in larger systems
• Define functionality clearly
– Through regression tests
– With repeatable results
• Disseminate projects
– Post open-source code
– Document projects on Web, Wiki
• Expand the community of developers
– Answer questions on the Email list
– Collaborate with your peers to build new applications
Spring Camp 2013
50
NetFPGA.org
Spring Camp 2013
51
Stuck for a NetFPGA project?
•
•
•
Build an accurate, fast, line-rate NetDummy/nistnet element
•
Hardware channel bonding reference implementation
Well
I’m not sure about you but here is a list I created:
A flexible home-grown monitoring card
•
TCP sanitizer
Evaluate new packet classifiers
•
•
Prototype a full line-rate next-generation Ethernet-type
•
Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)
•
Demonstrate the wonders of Metarouting in a different implementation (dedicated hardware)
•
Provable hardware (using a C# implementation and kiwi with NetFPGA as target h/w)
•
Hardware supporting Virtual Routers
•
Check that some brave new idea actually works
•
e.g. Rate Control Protocol (RCP), Multipath TCP,
•
toolkit for hardware hashing
•
MOOSE implementation
•
IP address anonymization
•
SSL decoding “bump in the wire”
•
Xen specialist
nic application classifiers, and other neat network apps….)
– (and
•
computational co-processor
•
Distributed computational co-processor
•
IPv6 anything
•
IPv6 – IPv4 gateway (6in4, 4in6, 6over4, 4over6, ….)
•
Netflow v9 reference
•
PSAMP reference
•
IPFIX reference
•
Different driver/buffer interfaces (e.g. PFRING)
•
or “escalators” (from gridprobe) for faster network monitors
•
Firewall reference
•
GPS packet-timestamp things
•
High-Speed Host Bus Adapter reference implementations
•
–
Infiniband
•
–
iSCSI
•
–
Myranet
•
–
Fiber Channel
•
Smart Disk adapter (presuming a direct-disk interface)
•
Software Defined Radio (SDR) directly on the FPGA (probably UWB only)
•
Routing accelerator
•
–
Hardware route-reflector
–
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
(and application classifiers, and other neat network apps….)
Other protocol sanitizer (applications… UDP DCCP, etc.)
Full and complete Crypto NIC
IPSec endpoint/ VPN appliance
VLAN reference implementation
metarouting implementation
virtual <pick-something>
intelligent proxy
application embargo-er
Layer-4 gateway
h/w gateway for VoIP/SIP/skype
h/w gateway for video conference spaces
security pattern/rules matching
Anti-spoof traceback implementations (e.g. BBN stuff)
IPtv multicast controller
Intelligent IP-enabled device controller (e.g. IP cameras or IP powerm
DES breaker
platform for flexible NIC API evaluations
snmp statistics reference implementation
sflow (hp) reference implementation
trajectory sampling (reference implementation)
implementation of zeroconf/netconf configuration language for rout
h/w openflow and (simple) NOX controller in one…
Network RAID (multicast TCP with redundancy)
inline compression
hardware accelorator for TOR
load-balancer
openflow with (netflow, ACL, ….)
reference NAT device
active measurement kit
network discovery tool
passive performance measurement
active sender control (e.g. performance feedback fed to endpoints fo
Prototype platform for NON-Ethernet or near-Ethernet MACs
•
Build an accurate, fast, line-rate NetDummy/nistnet element
•
A flexible home-grown monitoring card
•
Evaluate new packet classifiers
•
Prototype a full line-rate next-generation Ethernet-type
•
Trying any of Jon Crowcrofts’ ideas (Sourceless IP routing for example)
•
Demonstrate the wonders of Metarouting in a different implementation (dedicated
hardware)
•
Provable hardware (using a C# implementation and kiwi with NetFPGA as target
h/w)
•
Hardware supporting Virtual Routers
–
•
Internet exchange route accelerator
Check that
some
brave
52
Spring
Camp
2013 new idea actually works
–
Optical LAN (no buffers)
Project Ideas for the NetFPGA
•
•
•
•
•
•
•
•
•
•
•
NetFPGA-10G Test Harness
Drop 1-in-N packet module
Rate-limited module
Event capture module
Statistics and Counters
Measurement sketch
Advanced OPL
Input / Output scheduler
40G Port
VLAN Tagging
….
• Ideas from NetFPGA-1G
Spring Camp 2013
53
Visit http://NetFPGA.org
Spring Camp 2013
54
• Wednesday restart from 9am
• This evening is time to try hw testing &
synthesis crypto_nic design if you didn’t
manage that
• Make your group, leader&project
• Specific, Realistic
• Time-bounded
• Modest is good…..
Spring Camp 2013
55
Section IX: Conclusion
Spring Camp 2013
56
Acknowledgments
NetFPGA Team at Stanford University (Past and Present):
Nick McKeown, Glen Gibb, Jad Naous, David Erickson,
G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller,
Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng,
Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo
NetFPGA Team at University of Cambridge (Past and Present):
Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik
Matthew Grosvenor, Gianni Antichi, Neelakandan Manihatty-Bojan,
Georgina Kalogeridou, Jong Hun Han, Noa Zilberman
All Community members (including but not limited to):
Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek,
Yahsar Ganjali, Martin Labrecque, Jeff Shafer,
Eric Keller , Tatsuya Yabe, Bilal Anwer,
Yashar Ganjali, Martin Labrecque
Kees Vissers, Michaela Blott, Shep Siegel
Spring Camp 2013
57
Thanks to our Sponsors:
• Support for the NetFPGA project has been provided by the
following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not
necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this
project.
This effort is also sponsored by the Defense Advanced Research Projects Agency (DARPA) and the Air Force
Research Laboratory (AFRL), under contract FA8750-11-C-0249. This material is approved for public release,
distribution unlimited. The views expressed are those of the authors and do not reflect the official policy or
position of the Department of Defense or the U.S. Government.
Spring Camp 2013
58