KnoteBox
Joe Kramer, Leo Ovanesyan, Jimmy Thompson
Project Objectives
• Recognize musical notes from an analog sound input in
real-time and output MIDI data to a computer
MIDI
KnoteBox
System Overview
Power
ctrl
Audio
in
ADC/
filters
FPGA
MCU
External
Power
USB
ctrl
USB
out
Algorithm Overview
Analog
waveform
Time
frequency
data
FPGA
Thresholding/
De-noising
Note
recognition
MCU
MIDI
formatting
Hardware – Audio In
• Audio jack
– 6.5mm connector
• Typical for electrical guitars and professional audio
Hardware – ADC/Filters
• ADC from Texas Instruments
– 12-bit
– 20 KHz sampling rate
• Can still get one harmonic from highest note
Hardware – FPGA
• FPGA breakout board from Dallas Logic
– Altera Cyclone III
• 24,624 logic elements
• 594 Kbits of memory
– Programmer and power included
Hardware – Microcontroller
• ARM M3-Cortex from STMicroelectronics
– 32-bit
– 72 MHz
– 37 I/O pins
– Single-cycle multiplication and division
Hardware – USB Controller
• FTDI Chip
– Serial to USB
– Output seen as MIDI using:
• Virtual COM Port driver
• Roland MIDI driver
Hardware – USB Out
• Mini USB connector from Hirose
– USB 2.0 compliant
– Standard connection used in cameras
System Overview
Power
Ctrl
Audio
in
20 KHz
ADC
ARM M3Cortex
Cyclone III
External
Power
FTDI
chip
Mini
USB
FPGA
• We will use the FPGA for the frequency
extraction portion of our project as well as to
some the peak picking stage.
FPGA
A/D Converter
Frequency
Extraction
Peak
Picking
Microcontroller
FPGA
• Altera Cyclone Series:
– Cyclone II C35 on DE2 board for prototyping
– Cyclone III C25 from Dallas Logic on PCB
Frequency Extraction
• Using Fast Fourier Transform (FFT)
• Done in FPGA
Frequency Extraction
• Altera FFT IP core
– Transform Length – 4K
– Data Precision – 12 bits
– Input – Stream of input data (sound signal)
– Output – Stream of FFT data
Resources Available
EP2C35
EP3C25
Logic Elements (LEs)
33,216
24,624
Memory Bits
420,000
594,000
Testing FFT IP Core
• FFT IP Core test being done in Simulink
Peak Picking
• We will be picking peaks using the “Neighbor
Method”
• Neighbor Method - focuses on a sample and
then compares it to the samples around it (its
“neighbors”)
• For each bin or collection of points we will find
the slope of the line in the bin relative to its
neighbors and the max power in the bin.
Peak Picking
• How this will help us:
– Will be able to see if each bin is going up or down
and with what intensity
– Find local maximums
and minimums
– Then we will
correlate the data to
find actual peaks
opposed to noise
Peak Picking
• What will be done in the FPGA?
– Find slope and local max of each bin
– Then we will send that data to the Microcontroller
– The Microcontroller will use the data to find actual
peaks
FPGA
[ ]
Power
Slope
Microcontroller
MCU
• The microcontroller receives frequency
information from the FPGA over SPI
• It then recognizes musical notes and sends
MIDI events to the USB output
MCU
FPGA SPI
Master
Peak
Picking
Note
Recoginition
USB
MCU
• STMicroelectronics ARM Cortex M3
– Header board from SparkFun for development
– Raisonance R-Link Programmer/Debugger with
RIDE7 IDE based on gcc
Peak Detection
• Pick peaks based on frequency data
– Power relative to some threshold
– Separation from neighbors
• Peaks over some threshold are put in a buffer
Note Detection
• State of notes being played
– Frequency and harmonic content
• State changes trigger MIDI events
• State updated for each time slice
Get peak info
Update
current notes
Find new
notes
Send MIDI
events for note
on/off
Note Detection- Update Notes
• From the frequency data, possible notes are
given a score based on peaks near ~10
harmonics
• First, the notes in the on state are checked to
make sure their score is high enough
• If a note is still on, its peak content is
subtracted from the peak data
• If a note is no longer on, it’s state changes to
off and a MIDI note off event gets sent
Note Detection- New Notes
• Goal: find the set of notes that best fit the
peak data
• Two iterations over peak data
– Generate possible notes based on fundamentals
– Pick combination of notes with highest score
• Notes that turn on are added to the notes
state table and a MIDI note on event is
generated
MIDI over USB
• USB devices can conform to defined “classes”
– i.e. Mass storage, Communications, and Audio
• Conforming to the MIDI USB class allows for
compatibility on PC, Mac, and Linux
• MCU has USB onboard with several drivers,
but no MIDI class
• FTDI chip would simplify USB interface by
providing USB MIDI from serial interface
Budget
Item
Description
Price/Unit
Units
Total Price
ADC
TI Stereo Audio A/D
$7.75
5
$38.75
FPGA
Dallas Logic Cyclone III Board
$249.00
1
$249.00
Microcontroller
STMicro ARM Cortex
$9.60
3
$28.80
PCB
Audio TRS
Connector
4-Layer board from Advanced Circuits
$66.00
4
$264.00
6.35 Female Audio Connector
$2.77
4
$11.08
Casing
$30.00
1
$30.00
Power
$50.00
1
$50.00
Dev Boards
$175.00
1
$175.00
$150.00
1
$150.00
Total
$996.63
Misc Parts
LEDs, OpAmps, Headers,Wires, Resistors,
etc.
Division of Labor
Task
Joe
Leo
Jimmy
Feature Extraction
x
x
x
Pattern Recognition
x
x
x
X
x
PCB Design
X
FPGA Prototyping
x
Microcontroller
X
x
PC Software
x
x
x
Schedule
Task
Joe
Leo
Jimmy
Feature Extraction
x
x
x
Pattern Recognition
x
x
x
X
x
PCB Design
X
FPGA Prototyping
x
Microcontroller
X
x
PC Software
x
x
x
Milestones
• Milestone I
– Have 1st iteration of PCB ordered
– FPGA – Demonstrate the FFT core working on the DE2 Board
– Microcontroller – Be able to send USB data
• Milestone II
– Interface between FPGA and Microcontroller working
– Have 2nd iteration of PCB ordered
• Expo
– Full System Operating Correctly
Risks and Mitigations
• Inaccuracy with picking out notes and peaks
– Perform Matlab testing at the beginning to find and verify
the best method
– Perform FPGA testing on a test board before building our
own hardware
• Speed (Speed vs. Accuracy)
– Consider parallelizing the system
– Store information instead of having the system be realtime
Risks and Mitigations
• Noisy signal (especially at Expo)
– Have filters in place to remove the noise
– Consider having several microphones to create a noise
array for noise cancellation
• Different instruments have different sound signatures
– Have different algorithms in place that can be used
depending on the instrument
– Have a switch on the box to select what sound type the
signal is
Risks and Mitigations
• Time Constraint
– Plan time accordingly
– Stick to the schedule
• Shipping delay / error
– Order parts early
– Ensure the part is available
Risks and Mitigations
• Problems or errors with PCB design and population
– Fix mistakes and order new iteration
– Make room for an error in our timeline
– Do it right the first time
• Problems with project working on PCB
– Get everything working on development boards first
• Someone has an emergency
– Triangle of stability
– Have at least two people working on all the different parts
Questions
In Order of Appearance:
Leo Ovanesyan
Jimmy Thompson
Joe Kramer