Presented By: Tal Goihman, Irit Kaufman
Instructor: Mony Orbach
Winter 2012
Goals

Project Goal:
 Design and implement an A/D system using
Xilinx Virtex6 development board for
sampling at highest possible rate.
 Sample to virtex6 development board DDR
memory.
 Transfer the sampled data to PC memory
through PCIe and save the data to disk.
H/W Block Diagram
FMC125
Fast A/D
A/D IC
ML605 development board
FMC
Conn
Virtex6
FPGA
DDR3
PCI-E Connector
PC
ML605 development board
FMC125 Fast A/D
A/D Sampler
The FMC125 is a Quad-Channel ADC that provides
four 8-bit ADC channels enabling simultaneous
sampling of 1, 2, or 4 channels @ 5 , 2.5 , 1.25Gsps
respectively.
 Problems:
 4DSP provides free of charge a reference design only for
4ch @ 1.25Gsps. A reference design for 1ch @ 5Gsps
priced at 2300 EU.
4DSP reference design
Design Block Diagram
UART
ADC
Control
Implemented
MicroBlaze
In XPS
Timer
FMC125
Aggregator
AXI
Master
AXI
Slave
CDMA
Implemented
In ISE project
Navigator
AXI4
Mailbox
AXI4-lite
Memory
Controller
(MIG)
AXI
Slave
AXI
Master
DDR3
PCI
Express
“wormhole”
Block Diagram: Main Data Channel
FMC125
Aggregator
AXI
Master
AXI
Slave
Memory
Controller
(MIG)
DDR3

FMC125 delivers data using 4 128-bit lanes of a proprietary bus
running at 125Mhz into the aggregator.

Aggregator unites and synchronizes the different channels into a
512-bit bus.

AXI Master sends the data over a 256-bit wide AXI bus running at
200Mhz to the AXI Slave interface of the Memory Controller.

Memory controller handles Writes to DDR3.

FIFO’s and H/W Flow control in every component’s input (and some
components output) throughout the channel to achieve highest possible
bandwidth.

Throughput is measured using a timer from the start of the write
operation until assertion of a write done signal from the AXI master
AXI Bus background
Xilinx has adopted AXI bus, which is a
standard bus protocol from ARM used in
modern ARM SoC.
 Characteristics

 Memory mapped, 32-bit addresses
 Write Address, write data, write response,
read address, read data
 Variable width, clock & burst length over a
single bus.
AXI write example
Memory & Memory controller

512MB DDR3 64-bit @ 400Mhz (800MT/s)

Theoretical bandwidth of
800MT/s * 64bit / 8 = 6.25GB/s

Main channel matched to this theoretical bandwidth
(200Mhz * 256bit /8 = 6.25GB/s)

Memory & controller isn’t perfect, has a utilization
factor.

We achieved 5.13GB/s Throughput (82% utilization!)
Design Block Diagram
UART
ADC
Control
Implemented
MicroBlaze
In XPS
Timer
FMC125
Aggregator
AXI
Master
AXI
Slave
CDMA
Implemented
In ISE project
Navigator
AXI4
Mailbox
AXI4-lite
Memory
Controller
(MIG)
AXI
Slave
AXI
Master
DDR3
PCI
Express
“wormhole”
Block Diagram: memory to PCIe

After the data is sampled to memory the system
transfers the first chunk to the Host PC’s DMA buffer.

The subsequent chunks are transferred upon
receiving a command from the Host PC.

Data is read from memory and transferred to PCIe
by the DMA engine
 Throughput was matched to PCIe
 Transferred in 16MB chunks (the chosen DMA buffer size)
 Address translation occurs in the PCIe core
Block Diagram: PCIe

Xilinx Available solutions include:
 A PCIe integrated block with support for up to x8 gen2. Hard to
work with:
○ Need to know inner workings of PCIe
○ Need to implement several proprietary interfaces with many rules
and signals
 A wrapper for memory mapped AXI with support for up to x4
gen1 / x2 gen 2
○ Connects to a standard AXI4 bus
○ Chosen in our design due to high ROI

We chose to use the wrapper due to high ROI
 x4 gen1 configuration, up to 1GB/s
 One 64bit AXI BAR for sample data transfer with Configurable
address translation, one PCIe BAR for mailbox communication.
Block Diagram: MicroBlaze

MicroBlaze is a soft-core processor by Xilinx.

Runs firmware written in C from a dedicated 64KB
BRAM.

Firmware communicates with Host PC’s software
through a PCIe mailbox
 Supports a predefined set of commands to accomplish the
functional use case and provide debug capabilities
 Initializes and configures the entire system per to the Host
PC’s instructions.
 Provide visual status and information to the user through
UART massages.
Block Diagram: ADC ctrl

ADC ctrl is a custom core deigned to enable
MicroBlaze to communicate with other custom cores

Developed using the Create peripharel wizrd in XPS
and implements mamory mapped register access
through AXI4-lite

Provides the following capabilities:
 Translate the wormhole transactions to / from the FMC125
core.
 Provide AXI Master with burst count, write start, test
signals
 Provides the aggregator with enabled ADC channels
Software

Written in C# .NET 4 in VS2012, GUI in WPF

Enables simple interaction with the system

User adjustable burst size,
burst count, active channels,
output file location and
external trigger mode

compatible with Windows
7/8 x64.
Software: Jungo WinDriver

Easy driver creation

Simple interface for DMA buffer allocation,
read and write operations

Slightly problematic from a .NET environment
since it lacks proper documentation, and the
wizard generates a non working solution

Reference for .NET and DMA was found in a
bundled example for PLX chips
Test Environment

Verification of sampled data on DDR3 is
accomplished by firmware on MicroBlaze comparing
read data with predetermined written pattern

FMC125 incoming data is observable through
ChipScope.

Bandwidth was calculated by measuring the time to
write a chunk of data using a Timer.

Data transfer from memory to PCIe is validated
through observing the memory on system and
comparing with the data received from PCIe
manually.
Backup

Documantaion