My Beats
1. Introduction
1.1 Team Members
Michael Smith
Dylan Barrett
Karen Tipping
1.2 Background and Motivation
Beat games such as Guitar Hero and Rockband are becoming increasingly popular.
However, players can't play along to any song they'd like. The tracks to popular songs are
pre-made for Guitar Hero and Rockband. If someone wants to play their favorite song on
Guitar Hero, they have to manually create a track for it. This seems like a lot of work!
Music is such an integral part of our society, especially for younger generations. It
makes sense that people would want to play all of their favorite songs on Guitar Hero or
Rockband.
1.3 Project Abstract
This brings us to the goal of our project. Rather than creating the tracks one at a time,
we will create a tool that can take any song and dynamically generate the playable tracks for
the player. This will allow players to play all of their favorite songs, and not have to worry
about creating the tracks themselves. We will focus this semester on dynamically generating
tracks for Guitar Hero only.
2. Available Tools, Frameworks and Constraints
2.1 Framework Constraints
Frets on Fire is an open source environment to play Guitar Hero tracks on. This will
benefit us, since our goal is not to create a user interface. Once we have dynamically
generated the tracks for Guitar Hero, we will play them on Frets on Fire to test them out.
This, of course, limits us to strictly guitar. It also constrains us in the format of our
tracks. The MIDI file must be formatted to work within Frets on Fire, and the music file must
be in ogg format.
2.2 Existing Tools
One of the most important tools that we will be using this semester is Analyze by the
Echo Nest. Analyze takes in a song, and outputs XML files describing that song.
We will have data on the notes, pitches, loudness, timbre, and more. Using this
information, we can analyze the notes and patterns to create realistic and fun tracks.
The Echo Nest also provides a library of various functions that we can use to work with
the information received from Analyze.
2.2.1 Sample XML file of a segment from Analyze
One of the more important XML files is one that represents the segment. A segment is
essentially a note, and the XML file describes the loudness, timbre, and pitch at different tick
marks throughout the span of the defined note.
To get this XML file, the function get_segments can be called. This method returns
the information for all the segments in the file.
Below is an example of the XML file with one segment. If there were more than one
segments, then the segment tags would be repeated to represent the other segments.
<?xml version="1.0" encoding="UTF-8"?>
<response version="2">
<status>
<code>0</code>
<message>Success</message>
</status>
<thingID>TRLFPPE11C3F10749F</thingID>
<method>segmentslt;/method>
<analysis>
<segment start="2.54562" duration="0.29587">
<loudness>
<dB time="0">-29.707</dB>
<dB type="max" time="0.03483">-25.351</dB>
</loudness>
<pitches>
<pitch>0.009</pitch>
<pitch>0.090</pitch>
<pitch>0.010</pitch>
<pitch>0.018</pitch>
<pitch>1.000</pitch>
<pitch>0.017</pitch>
<pitch>0.005</pitch>
<pitch>0.005</pitch>
<pitch>0.028</pitch>
<pitch>0.383</pitch>
<pitch>0.010</pitch>
<pitch>0.016</pitch>
</pitches>
<timbre>
<coeff>31.132</coeff>
<coeff>-51.574</coeff>
<coeff>92.135</coeff>
<coeff>60.049</coeff>
<coeff>77.614</coeff>
<coeff>-17.629</coeff>
<coeff>-47.819</coeff>
<coeff>6.888</coeff>
<coeff>17.400</coeff>
<coeff>-0.635</coeff>
<coeff>-8.109</coeff>
<coeff>-45.552</coeff>
</timbre>
</segment>
</analysis>
</response>
Other functions such as get_beats, get_key, get_loudness, get_tempo,
get_time_signature, and get_duration can be used to get further information as
needed. A list of all the functions, their descriptions, and examples are available on
TheEchoNest.com under Developer APIs.
2.3 Complexity Constraint
Most music has multiple instruments and/or vocals all playing at once. Since
separating out different instruments is beyond the scope of this semester, we will be dealing
with guitar only songs.
To expand our work, we may use songs complete with numerous instruments, but treat
all notes as guitar sounds for a Guitar Hero track. It won't feel as realistic as playing along to
a guitar only song, but it should still be fun.
3. Project Description
The aim of the project is to create a command line tool that accepts as input a
song file containing only guitar and outputs a dynamically generated playable track in
the format used by Frets on Fire based on that song.
3.1 Development Environment
Our tool will be created within the Java environment to allow for easy portability.
Java offers a wide range of networking protocols and also has support for the MIDI
format within the javax.sound.midi API. Below are some of the classes and methods that we
will be using from the package to create our MIDI file.
Within javax.sound.midi, we will mainly be using the classes: Sequence, Track, and
ShortMessage. The class Sequence represents the MIDI file, and the class
ShortMessage will hold a message. A message gives information about segment
beginnings, endings, or other information the track might need. It gets the information
through events. There can be more than one Track running in a Midi file, possibly
corresponding to different instruments.
We will also be using the methods: sequence.createTrack, track.add, and
shortMessage.setMessage. The method add adds a new message to the track by
creating a MidiEvent for it that takes in the message. The method setMessage sets the
information about the message.
3.2 Project Flow
When run, the tool will proceed through the following stages:
3.2.1 Stage 1
Stage one is very basic. It will first prompt the user for the song file location and
the Frets on Fire song library location. After obtaining this information it will check that
each piece is valid and then upload the song file over the network using standard web
protocols to the Analyze servers.
3.2.2 Stage 2
Stage two becomes more complicated. In this stage the results of the Analyze
program will be retrieved, again using basic web protocols, and set up to be organized. We
will use various functions provided for us by the Echo Nest to work with the data. When
retrieving the information on the song defining where and what notes exist, they
will be placed into a data structure that focuses on identifying over all song structure.
This is the prep work that will be done to allow the next stage to generate playable
tracks by analyzing this created structure. This structure could be arranged as a Suffix Tree,
or analyzed using Curve Sampling.
A suffix tree's structure can highlight common subsequences of a pattern. Suppose the
notes of the song are placed chronologically into a string, using letters to represent the notes
like so: “abcefgabcdgfabceeeabcabc.” Using the Suffix Tree would make it quickly apparent
that “abc” is the most commonly recurring sequence found within the song. This is important
because in a song the more repetitive parts are the parts that are more important to having a
song be identifiable. So the notes representing "abc" would not want to be cut out (at least
not all 3 of them.) This will be expanded on in Stage 3.
We may also use Curve Sampling to analyze the notes to decide which can be
discarded, while still leaving a feel for the song. Curve Sampling is an easy concept that
basically says that if there is a lot of change in the song (the pitches are going up and down
and up and down) then you would need a lot of notes to portray this. However if the song
was simply going up in pitch at a steady rate, then you could remove plenty of notes and still
recognize the song. We hope to use this to remove unnecessary notes when creating our
tracks, which have to be compressed down to only five buttons.
3.2.3 Stage 3
Stage three is where the song and song structure will be evaluated to generate
the playable song tracks at different difficulty levels. This is the most complicated stage.
3.2.3.1 Note Compression
The playable music tracks consist of five notes/buttons. A given guitar piece can
consist of any note A-F all up and down the musical scale. As such, the notes within the
song need to be compressed and mapped to one of the five different game note
options. This mapping needs to be even across the range of notes appearing in the
original song. Using various statistical methods, five equal divisions of the appearing
notes can be defined for each analyzed song.
This will need to be a different mapping for every song. If we have a very low song, it
would not be much fun to be playing the low button a million times. So we need to look at
each song individually, and divide up the pitches that appear in that song.
On the easy and medium tracks, the fifth button, played by stretching the pinky out, is
almost never included since it's hard to play. We will have to consider this while creating the
tracks for the difficulties.
3.2.3.2 Difficulty Levels
Each song can have tracks for any difficulty. Some songs only have tracks for Hard,
and some songs have tracks for all difficulties. Frets on Fire supports four difficulties,
including SuperEasy. We will define our songs to generate tracks for the difficulties Easy,
Medium, and Hard. This is how we will define each difficulty:
Easy: This level will have the lowest note density for the player to play. The
notes in this level will consist only of the minimum needed to keep the song identifiable
and give the player a sense that they are still playing the song. As such, it will want to
focus on the most repeated sections of notes in the song as identified by the Suffix
Tree. We will also use Curve Sampling to remove as many notes as possible to still keep the
feel for the song in the track. This will generate an Easy track, in that the notes density will
not be greatly demanding and it will be a set of repeating patterns for the user to play. There
will also not be a lot of buttons that must be played at the same time, since that is a little bit
more difficult. Lastly, by conforming to the pre-existing game standards, this level will only
utilize four out of the five buttons.
Medium: This level will build off the Easy track. Everything included by the Easy
track will be included in the Medium track allowing for players to develop their skills by
moving up a setting in a song and not having to relearn the whole thing. Medium will
also start to incorporate more of the less frequent musical patterns found within the
song. Medium will also include more of the notes within these patterns to yield a
greater note density. And in conforming to the pre-existing game standards, this level
will utilize all five of the game buttons, although the fifth button should be used sparingly.
Hard: Starting with the Medium track, Hard will include the most notes in all the
already included patterns generating the greatest note density of all the levels. It will
also include the most random sections such as solo's and small clips that may only
appear once. This difficulty should employ the fifth button just as frequently as the other four
buttons.
3.2.3.3 Heuristic Evaluation
In order to use the Suffix Tree to determine which notes to actually place into
the song tracks, a set of heuristics will have to be developed to make the decisions. By
utilizing a set of heuristics that can change and evolve rather than a few set decision
making functions, it will allow the program to become smarter with evaluation. If a
player alerts the program that the song was less than satisfactory, then the program can
try tweaking the heuristic values to generate better tracks. For example, a player might report
that the song had too many buttons for his pinky while on Medium. The heuristics can then
be tweaked to ensure there are less next time.
One likely heuristic that will be developed is one that will look at note density. This
heuristic will look at how close notes are to each other in timing and decide for each difficulty
level if there are too many.
Another heuristic is going to have to attempt to separate the more important notes from
the less important notes in a given sequence. For instance, if the pattern “abcbcdafe” is
repeated many times throughout the song, it will be a section that should be included in all
difficulty levels. However, each and every note within that sequence should not be included
especially for the Easy track. In this case the heuristic will use Curve Sampling and will
attempt to remove notes to create a sequence similar to “a c c a e” where the remaining
pattern is still clearly identifiable as the song.
The heuristics will also ensure that there are the right number of buttons that will to be
played together. On an Easy track, the number of buttons played together will need to be
kept small, however on a Hard track there may be many.
A third heuristic can be the length of the buttons (if you want the player to hold the
button instead of just hit it.) This doesn't play a role in difficulty, but it's still important to get
the right balance.
A fourth possible heuristic could be how quickly the player must switch buttons. This
can be difficult if the different buttons are extremely close together or if there are a lot of them.
So, on Easy, quickly changing buttons will have to be kept to a minimum or even left out
completely. However, on Hard, including a lot of these especially in a row would definitely
make the track harder.
Another heuristic could watch out for extremely long sequences with the same button.
If we divide up the song based on the overall pitch of the song, there might still be a section of
the song that's really low. This may go on for a while, and the player will be stuck playing the
same button over and over again. This doesn't make for a very fun song, and will have to be
tweaked.
Many heuristics will be employed to ensure the best player experience that we can
deliver!
3.2.4 Stage 4
Stage 4 consists solely of track output. This stage will take the difficulty tracks
produced by Stage 3 and embed them within a MIDI file in the same format used by
Frets on Fire. This MIDI file, along with the original song file will then be outputted to a
folder within the Frets on Fire song library, making the song available the next time the
user plays the game.
We can now test drive our dynamically generated songs!
4. Task Break Down And Time Line
4.1 September 17 - October 1
During the first few weeks, some working tools need to be created. Some preliminary
things must be figured out before we can start working on the meat of our project. The below
things can be worked on in parallel since they are separate things that don't depend on one
another.
The first is to reverse engineer the Frets on Fire MIDI file format so that we can output
our own files using the javax.sound.midi API. This is Dylan's responsibility. This is important
because without figuring this out, we won't be able to test out our tracks.
The second is to develop a set of Java functions that can interact with the Analyze
servers to upload a song and retrieve the results. This will include retrieving the segments, or
notes, organizing them, and beginning to understand how it works. This is Mike's
responsibility. This needs to happen before anything else as well because without this data,
we have nothing to work with.
We also need to be able to translate MP3 files into OGG format for Frets on Fire to
play in the background of the tracks. This is Karen's responsibility. This is equally important
because without it, our tracks will not have music and we cannot test the timing of our track
buttons or get a feel for how well the tracks represent the song.
All of these tasks can be done in parallel since they are completely separate entities.
They must also be completed before we can proceed to the next stage. Once we put this all
together, we should be able to run a test track on Frets on Fire and start working on the meat
of the project: to dynamically produce a fun track for any song.
4.2 October 1 - October 15
Naturally, we can't start creating tracks until we can test them out on Frets on Fire. At
this point all of the preliminary tasks should be in place, so we can concentrate on the
algorithms for creating the tracks.
At this point, Mike will be working on creating a Suffix Tree containing the information
about the song. He will use this to define the common structures. Later we will use this
information to decide which parts of the song can not be omitted based on their important
standing (such as the chorus).
In parallel, Dylan can also be analyzing the input, but in a different way. He will be
implementing Curve Sampling to prune the notes for the tracks. Dylan will be condensing the
overall song to make it more manageable to ultimately map everything to just 4 or 5 buttons.
Mike's Suffix Tree will be gathering information on the structures, while Dylan will be
making a compressed version of the song. From that compressed version we can use Mike's
information and further truncate it by maybe omitting uncommon subsequences especially for
the easier levels.
4.3 October 15 - October 22
Once the song has been pruned for unnecessary notes using the Curve Sampling, and
short uncommon subsequences have been removed, Karen can begin to map what we have
to buttons. At this point, we have the basic notes that we want to include in a track. We can
now move on to the next stage of deciding which notes should be mapped to which button.
For this preliminary stage of initial mapping, we will be implementing it for only one
difficulty. This may be the hardest difficulty since then we wouldn't have to further prune the
notes that we include in the song.
We must determine for each song how to divide up the notes into the buttons. This
must be done dynamically for each song, since the pitches in songs can vary greatly. Karen
will work on figuring out how to dynamically divide up the track notes into the buttons, and
make it playable on Frets on Fire.
4.4 October 22 - November 12
Now that the dynamic mappings to buttons has been figured out and a basic song track
has been created for one difficulty, we have to expand this knowledge to also deal with all
difficulties. We will be making tracks for SuperEasy, Easy, Medium, and Hard.
In the easier tracks, we will need to further prune the notes that we have. We can use
the suffix tree to omit more uncommon subsequences, and also use Curve Sampling to
reduce the frequency of notes. Determining just how much we should prune using Curve
Sampling, and how many uncommon subsequences we should include will be decided by the
heuristics that we will implement.
In this stage, we will develop the heuristics that will help us determine the frequency of
notes and other factors that make the dynamically created tracks fun. We will play the songs
and tweak the heuristics so that each difficulty has the right balance of note frequency, note
length, note variation, note concurrency, and notes that are representative of the song.
These heuristics will be developed by Mike, who is the most familiar with them. Once
they have been implemented, endless tweaking for best results can be done by all of us.
4.6 November 12 - End of Semester
At this point, everything should be successfully implemented and integrated. This
project requires constant integration, so we should be integrated at the end of each stage.
If any bugs are discovered, the individuals responsible for that section of code will fix
them, since they know the code the best.
We will also use this time to do quality assurance. We will thoroughly test the other
group members' modules, and create a bug list. We will organize it by priority, and work on
fixing the most crucial bugs first.
Glossary
Analyze
An API provided by the Echo Nest that allows
us to use its functions to gather information
about songs. It outputs the data in the form
of an XML file for us to work with.
Buttons
The five notes in the compressed track that
are played.
Curve Sampling
A mathematical way of distinguishing which
notes can safely be removed for the tracks
while keeping the song's structure
recognizable in the track. If the song was
graphed using the notes, the maxes and mins
would need to remain, but some of the
transitioning notes would not.
Frets on Fire
A user interface that simulates Guitar Hero on
the computer. This will be the interface
where we demonstrate the song tracks that
we have produced.
Guitar Hero
A beat game where players push buttons on
a game controller shaped like a guitar and try
to match the circles scrolling along the screen
to the beat of the song.
Heuristics
A problem solving technique where the best
solution is found at successive stages and
helps to find the best solution to the next
step.
MIDI file
A way of representing musical information in
a file. It includes information on pitch, note
beginnings and endings, loudness, and more.
OGG file
The background song format used in Frets on
Fire.
Rockband
A game similar to Guitar Hero with the
addition of a guitar base, drums, and vocals.
They all must follow instructions from the
track that tell players what to do.
Segment
A section of the song that represents a note.
Analyze's function get_segments returns an
XML file that gives various information about
all the segments in the song.
Song
The original song usually in MP3 or OGG
format. (OGG is the song format for Frets on
Fire.)
Suffix Tree
A tree which is organized to show common
sequences. The tree branches off when a
subsequence is different from another
already found. The nodes with the most
children are the most common.
The Echo Nest
A website that provides our tool, Analyze, and
many functions to work easily with the data.
Timbre
The note quality.
Track
The truncated version of the song where the
notes are mapped to buttons. There will be
three tracks for each song: one for Easy, one
for Medium, and one for Hard.
XML file
A file that can be traversed through its tagged
nodes for information. The nodes are
denoted by tags (< > ... </ >) and mark the
beginnings and endings of the information
sections. An example in given in section
2.2.1 of the document.