KnoteBox
Joe Kramer, Leo Ovanesyan, Jimmy Thompon
Project Objectives
KnoteBox
MIDI
Project Objectives
• Recognize musical notes from an analog sound
input in real-time and output MIDI data to a
computer
• Why use the MIDI protocol?
– It contains the information needed to display
notes
– It is standard in the music industry
– Software already exists to manipulate MIDI data
Project Motivation
• Learning a new song or instrument
can be irritating and discouraging
• Music-based video games can be
used for learning music but have
simplified controllers
• A music-based video game using
real instruments would be
enjoyable and instructional
Project Motivation
• There are systems which detect notes in audio
signals
• LittleBigStar – attempts to recognize chords but
does not have much accuracy
• The KnoteBox is different and better because it
will attempt to detect multiple notes at the same
time, do it in real time, and with more accuracy
KnoteBox >
Current Technology
Applications
• Guitar Hero / Rockband clone
• Use MIDI input to create a track that must be
played as notes scroll by
• Learning / Teaching capabilities
• Entertainment Game
Applications
• Sheet music generator
• Play into the system in order to get sheet
music for the song played
• Can also generate a MIDI file
Applications
• Tuner
• Ability to show frequency information or MIDI
data to tune a specific instument
• Instrument / Song learning tool
• Ability to show what note is actually being
played
Overview
• Analog audio signal in
– Single instrument
– Polyphonic
– Melodic
Audio in
Audio
processing
I/O
Note matching
MIDI to PC
over USB
• MIDI over USB out
– Buy MIDI to USB chip
– MIDI note on->note off
• Box I/O
– Depends on algorithm
MIDI Protocol
• Industry standard protocol defined in 1982
• Defines interface between electronic
instruments and file format
MIDI Note on
Pitch, velocity
• MIDI events
– Note on with velocity
– Note off
– Aftertouch, pitch bend
MIDI Note off
Pitch
• Pitch ranges from 0-127 (~12 octaves)
Music Theory
• Pitch
– Fundamental frequency determines note being played
– Musical scale is logarithmically spaced (equal temperament)
– Linearly spaced overtones
• Envelope
– Time based attack, sustain, decay, release
• Timbre
– Differences in sound of voices or instruments
– Determined by differences in overtones and envelope
Frequency Analysis
• Constraints
– Delay
– Resolution
– Computation
• Time-frequency Algorithms
–
–
–
–
–
Resonators
Short form Fourier Transform
Wavelet analysis
Bilinear frequency analysis
Q-constant and Tonal Centroid
Harmonic Peak Analysis
• Pick peaks from timefrequency image
– Thresholding, noise canceling
– Decreasing power at high notes
• Form tracks of peaks over time
• Match peaks to a combination of notes
• Trigger note on when peaks rise together
• Keep state of notes being played
Software
Analog
waveform
Time
frequency
data
DSP / FPGA
Thresholding/
De-noising
Note
recognition
MCU
MIDI
formatting
Hardware
Power
Audio
jack
ADC/
filters
DSP/
FPGA
MIDI
driver
chip
MCU
JTAG
Power
Power
supply
Mini
USB
out
Hardware – front end
• Audio jack
– 6.5mm connector
• Typical for electrical guitars and professional audio
Hardware – pre-digital processing
• ADC chip
– Must have sampling rate of about 44.1KHz
• Standard audio CD quality
• Filter chip or RC circuit
– Anti-aliasing low-pass filter
• Amplifiers
– Op-amp: BW up to 44.1KHz
Hardware – for heavy processing
• DSP
– Hardware works, just need software
– Difficult to parallelize solution
– Limited to instruction set
• FPGA
– Solution can be parallelized in hardware
– Hardware needs to be verified
Hardware – for heavy processing
• DSP
– TI DaVinci video processor
• DSP and MCU
• Acceleration hardware and lots of RAM
• FPGA
– Xilinx Virtex
• High performance FPGAs
Hardware – for post-processing
• Microcontroller
– MSP430
• Cheap and popular (lots of example code)
• Low power (not an issue in the application)
– ARM
• Used in cell phones and other demanding applications
• 100MHz-1GHz range
Hardware – output
• Ploytech GM5 chip
– Provides a MIDI driver through a
USB interface
– Computer sees a MIDI device
• Hirose mini USB connector
Budget – for development
Item
Approximate cost
Two PCBs
$130
DSP / FPGA
$200 (x2?)
Microcontroller
<$50 (x2?)
Amplifiers, filters, ADCs
<$50
USB to MIDI chip
$10
Miscellaneous
$100
Total
$540-$790
Budget – for production
Item
Approximate cost
One PCB
$50
DSP / FPGA
$150
Microcontroller
<$50
Amplifiers, filters, ADCs
<$50
USB to MIDI chip
$5
Casing
$30
Total
$335
Schedule
Division of Labor
Task
Joe
Leo
Jimmy
Feature Extraction
x
x
x
Pattern Recognition
x
x
x
X
x
PCB Design
X
FPGA Prototyping
x
Microcontroller
X
x
PC Software
x
x
x
Risks and Mitigations
• Inaccuracy with picking out notes and peaks
– Perform Matlab testing at the beginning to find and verify
the best method
– Perform FPGA testing on a test board before building our
own hardware
• Speed (Speed vs. Accuracy)
– Consider parallelizing the system
– Store information instead of having the system be realtime
Risks and Mitigations
• Noisy signal (especially at Expo)
– Have filters in place to remove the noise
– Consider having several microphones to create a noise
array for noise cancellation
• Different instruments have different sound signatures
– Have different algorithms in place that can be used
depending on the instrument
– Have a switch on the box to select what sound type the
signal is
Risks and Mitigations
• Time Constraint
– Plan time accordingly
– Stick to the schedule
• Don’t get funding / Need more funds
–
–
–
–
Do the URAP application right the first time
Find a local sponsor
Get student discounts
Look for other research grants
Risks and Mitigations
• Problems or errors with PCB design and population
– Fix mistakes and order new iteration
– Make room for an error in our timeline
– Do it right the first time
• Shipping delay / error
– Order parts early
– Ensure the part is available
Risks and Mitigations
• Leo gets hits by a bus
– Ol’ well, we don’t need him
– Recruit from another group
• Jimmy or Joe die / have an emergency
– Triangle of stability
– Have at least two people working on all the different parts
Questions / Comments