InfiniBand FPGA
Matthew Wall
Rachel Ayoroa
Xiang Li
Ryan Schwarzkopf
Tim Prince
Alex Burds
Adviser: Dr. Morris Chang
Client: Troy Benjegerdes
What is InfiniBand




Switched fabric

communication link
2.5 Gbit/s signaling
Primarily used in
supercomputers
Specification maintained
by the InfiniBand Trade
Association
Protocol Stack
› Physical Layer
› Link Layer
› Network Layer
› Transport Layer
IP Cores
Reusable hardware cores accelerate
development
 E.g. DSP, communication controllers,
cryptographers, memory, processors,
Ethernet MAC
 Opencores.org

Problem/Need
Currently have to use proprietary
hardware interface (e.g. Mellanox)
 Can’t perform research on internals
 Need a open-source hardware
implementation

Design Concept
Hardware core in
VHDL
 Compatible with
InfiniBand
specifications
 Implementation
on an FPGA or in
simulator

Xilinx FPGA Chip
InfiniBand HDL Implementation
Link Layer
Physical Layer
Xilinx RocketIO
Fabric
Operating Environment

Proprietary Environments
Xilinx ISE (Development Environment)
 ModelSim (Simulator)
 Virtex-5 Development Board

User Interface
No UI in conventional terms
 VHDL Synthesizer
 VHDL Simulator
 FPGA pinout

Functional Requirements
FR01: The system shall provide InfiniBand
compliant input/output signals.
 FR02: The system shall be able to transmit
and receive packets according to
InfiniBand specifications.
 FR03: The system shall be able to handle
errors according to InfiniBand
specifications.

Nonfunctional Requirements
NFR01: The system shall be written in
VHDL.
 NFR02: The system shall be distributed
under an open source license.
 NFR03: The system should use only
standard VHDL libraries.
 NFR04: The system should be compatible
with open source VHDL simulators.

Market Research
InfiniBand Trade Association (Cisco, IBM,
Intel, etc.) maintains specification
 No open-source IP core competitors
 Specification is freely available/open

Deliverables

CD containing:
› Source Code (VHDL)
› Compilation Instructions
› User’s Manual

Submit to OpenCores.org
Project Plan
Work Breakdown
InfiniBand Project
Planning
Documentation
Design
Implementation
Testing
Scheduling
Bound
Report(s)
Top-Level
Design
Switch
Controller
Unit Testing
Layer
Integration
Testing
Resources
Risks
Requirements
User’s Guide
Layer Design
Link Layer
Compilation
Instructions
Layer
Component
Design
Physical
Layer
Device
Integration
Device
Integration
Resources

Team Members (Alex Burds, Tim Prince, Matt
Wall, Ryan Schwarzkopf, Xiang Li)
 Ryan Schwarzkopf (Communications
Coordinator)
Matt Wall (Project Leader)
Jason Boyd (Lab Coordinator)
Dr. Morris Chang (Project Advisor)
 Troy Benjegerdes (Client)
 Lab with Virtex-5 board and compatible
synthesizer (Coover 2041)



Schedule Overview
Implementation begins – 1/12/09
 Layers completed – 4/3/09
 Layers tested – 4/22/09
 System integration completed – 4/26/09
 Integration testing completed – 4/26/09
 System testing completed – 4/29/09
 Documentation completed – 5/4/09
 Bound Report completed – 5/4/09

Identified Risks
Complexity of system may prevent
complete hardware implementation
 Complete testing may require more time
than is available
 May need to obtain additional test
equipment

Detailed Design
System Overview

Two Layers
› Link
› Physical

Top-down design
approach
Xilinx FPGA Chip
InfiniBand HDL Implementation
Link Layer
Physical Layer
Xilinx RocketIO
Fabric
Link Layer

Handles the sending and receiving
of data across the links at the
packet level

Provides addressing, flow control,
and error detection

Virtual Lanes provide buffering

Link State Machine responds to
changes in the link status
Components







Link State Machine
Packet Receiver
Machine
Data Packet Check
Machine
Link Packet Check
Machine
Flow Control
Generator
Virtual Lane
Arbitration
CRC Generators
Link Layer
Virtual Lane 15 (Subnet
Management)
Virtual Lane 0
Flow Control Packet
Generator
Virtual Lane Selector
Packet Priority Selector
Data/Link Packet
Check Machine
Link State
Machine
Packet Reciever
State Machine
To Physical Layer
Virtual Lanes
Buffer packets to and from Link Layer
 Two VLs required, VL0 and VL15
 VL15 reserved for subnet management
packets

Flow Control
Prevent buffer overflow
 Send flow control packets periodically
 Updates credit information from Virtual
Lane
 Stop transmitting if receiver doesn’t have
enough credits

Packet Check Machines
Data Packet Check Machine
 Variable packet headers and payloads
mean more robust error checking
 Error checks performed by DPCM
VCRC and ICRC check
Link version check
Destination Local Identifier check
VL and VL15 checks
Buffer availability check
› Verifies the length of the packet
›
›
›
›
›
Reports errors in correct precedence
and discards packets with errors
 Error free packets passed to Virtual Lane

Packet Check Machines
Link Packet Check Machine
 Only one type of packet (flow control
packet)
 Checks for errors within a link packet
›
›
›
›
Verifies Link Packet CRC
Flow control opcode is flagged
Verifies correct virtual lane is selected and supported
Verifies the length of the flow control packet (6 Bytes)
Reports errors in correct precedence
and discards packets with errors
 Error free packets passed to Virtual Lane

CRC Generators
Two CRC Types
 Variant
› Entire Packet (Including Payload)
› Changes between transmissions

Invariant
› Data payload
› Static
Physical Layer
Maintains physical
link
 Three main
components

› Link/Phy Layer
› RocketIO Adapter
› RocketIO Module
Link Layer Byte Streams
Physical Layer
Link/Phy Layer
Encoded Lanes
RocketIO Adapter
Physical RocketIO Module
Physical Interconnects
Link/Physical Layer
Manages link
training process
 Maintains link
state once
completed
 Inserts control
sequences to
allow clock
synchronization

To Link Layer
Link/Physical Layer
Training
Sequence
Skip
Sequence
Idle
Sequence
TX
Buffer
RX
Buffer
L.T.F.S.M.
To RocketIO Adapter
Error
Status
Control Sequence
Detectors
Link Training FSM
Maintains link
state
 Handles errors

Polling
Received
TS1
Config
Failed
Configuration
Recovery
Failed
Config
OK
Link Up
Recovery
OK
Major Error or
Link initiated
recovery
Recovery
RocketIO Adapter

Manages Xilinx
RocketIO
operation
› Clock generation
and recovery
› 8b/10b Encoding

Converts Link/Phy
control signals into
the equivalent
RocketIO signals
Testing
Simulation Testing
Simulate individual components to
determine if they behave as expected
 Integrate components in a layer to
determine if they work together
 Use ModelSim VHDL simulation

Integration Testing
Once individual layers are simulated, we
will progressively integrate them together
and physically test them
 Phy/Phy testing

› Connect the Physical layer programmed on a
single board through loopback and ensure that
a data stream can be transmitted
Integration Testing Contd.

Phy+Link/Phy+Link
› Add the Link layer and determine if packets can
be transmitted.
› Also check for packet integrity and flow control
behavior
Test Harness
Xilinx ML507 Development Board
Virtex-5 XC5VFX70T
PPC440
CPU Core
PLB
Test Harness
Instance
Test Harness
Instance
Test Harness
Instance
RocketIO
RocketIO
RocketIO
Connector
(e.g., SMA)
Connector
Connector
SMA Cable (loopback)
Connection to an
InfiniBand device or
another test harness
Test Harness
PLB Interface
Instance of the link
and physical layers
 PLB control interface
 Programmable
interconnects
 Inject and extract
data into and out of
the data-path

TX FIFO
Control Register
Control Signals
RX FIFO
Link Layer
TX
Link Layer
RX
Physical
Layer
TX
Physical
Layer
RX
RocketIO
Development
Accomplishments
 Physical
Layer compiles
 Physical Layer simulation
 Physical Layer physical testing
 Link Layer compiles
 Link Layer simulation
 Performed extended testing with
a 5-10 Giga-sample oscilloscope
Results
Results
Challenges
Debugging the serial loop back
 Switching from Vertex-II to Vertex-5 board
 Only having 1 Vertex-5 board
 VHDL intense project
 Lost a team member

Conclusion
Have a springboard for further
development
 Further verification and development is
needed for a robust IP core

Lessons Learned
Evaluate project complexity in greater
detail
 Divide tasks into smaller parts
 Set stricter guidelines for tasks
 Obtain clearer end goal from client
 Define specific skill set required

Future Work
Verification of layers against
specifications in detail
 Potentially port to different transceivers
 Create an open hardware development
platform
 Make more robust for usability

Demo
Questions
?