Dance Recognition
Closing Report
Travis Simpson
Dr. Bradford Bennett
Motion Analysis & Motor Performance Lab
August 12, 2011
The purpose of this work is to be able to uniquely identify codified dance moves from
motion capture data of a move with distinct start/stop positions.
The first challenge is deciding how to read in the raw data and what parts of the data are
desirable. The Vicon .c3d files are processed using Bodybuilder and then exported as a
text file. The exact parameters to be exported are chosen at the export screen prompt in
Bodybuilder. The processed text files already available contain the following
markers/joints for each side: ankle, heel, toe, sacrum, ASIS, and knee. These will most
likely need to be expanded over time but are currently sufficient.
A script (readtextfile.m) reads in all of the parameters saved to the text file. The problem
that results is choosing a start/stop point for all the data. Since all the markers analyzed
are compared against each other at some point, the same time frame of data needs to be
used across the board. To accommodate this, the starting time frame in which data is
available from each parameter is pulled and compared. The latest starting point between
all the moves is used as the first frame to use. A similar designation is used for the end
point, with the earliest end point used by all markers being the last frame to use.
Furthermore, there will be periodic gaps in the middle of the parameter sets, so the code
is geared to disregard these sections when necessary. From these procedures, the raw data
is now available to be manipulated in MATLAB.
There are multiple methods of pattern recognition that could be used to solve this
problem. I chose two methods, the first of which being classification using dance
characteristics. This is important because it allows not only the development of which
move a dance sequence mimics but what the move physically does. At this stage, there is
a somewhat limited set of name-identifiable moves (100 or so out of 300), but because of
classification, they can still be described in dance terminology. As will be shown later,
adding the rest of the moves will not be difficult to accomplish. First, the methods of
recognition will be described (combined together to form idMove()) and then the details
of how model moves (those that the trial is compared against) are developed will be
explained.
In past versions, the actual classification followed in order of increasing complexity and
specificity based on the moves remaining. If after the most basic analysis is performed
there are still multiple possible move left, specific analyses were chosen to separate them.
By specific it is not only meant that the analysis may not work for all moves but also that
depending on what moves are left, the type of analysis will change. The disadvantage to
this method was that every move must be “hard-coded” into the classification system
after manually tweaking the code for an optimal result, as a move may need a special/new
analysis.
Currently, classification recognition is used up to a point to categorize the move, which
eliminates many possible moves. This looks at three basic characteristics:
jump/rise/grounded, sacral movement, and cumulative hip rotation. Then, the fourth
characteristic depends on whether or not the move is a jump, rise, or neither (grounded).
In the case of neither, this characteristic is null. However, if it is a jump or lift, it
represents the number of jumps or lifts the move contains, respectively. Additionally,
information is noted in the classification description about what type of take-off/set-down
there is (1 foot, 2 feet), and whether or not the jump contains lifts in addition to jumps.
Although some of the classification methods I have developed are not currently used in
the code, they could be useful for a more specific classification analysis. There is a
function called xCross() that searches for markers coming within a certain threshold of
each other, which was previously used to see if the ankles crossed or the ankle crossed
the knee during jump/lift periods. The functions findPos() and xPos() were used to
determine the positions involved during a grande plie. It could be used for other moves,
although it is limited in that it searches from the beginning to end of the move for the
starting position and vice versa for the end position. It chooses the position based on
marker proximity and only needs to encounter the condition for one frame. Another
function, squat(), looks for the sacrum to drop past a certain threshold position based on
the subject’s approximate femur length. The functions onToe() and transCheck() look for
instances where the dancer is en pointe. transCheck() ensures that the en pointe is not
preceding a jump based on a threshold frequency. In addition to these functions, there are
very detailed arrays describing useful parameters, such as time frames for jumps/lifts,
embedded throughout the code. Much of the basic information needed to perform other,
more specific and advanced classification analysis is probably already in place.
Although the classification method is effective because of its description of moves, it can
be bulky, among other things previously discussed. To combat these weaknesses, a
second pattern recognition method was implemented which utilizes cross-correlation.
This analysis follows the basic classification recognition and builds on its results.
Covariance was used which is a built-in MATLAB function for assessing crosscorrelation between equal length data series. This requires time normalizing for all data
that is to be compared. This can be problematic because some trials of a single move are
slower than others but it is also important to remember this work is for codified dance
moves at this point. Although covariance will analyze for time lag and normalization of
the variable set against time, it will not optimize time scaling. This has not been shown to
be very influential in the results of my work with a single subject but with future
implementation and additional subjects it could be. A better cross-correlating function
should be developed which also optimizes time scaling for even better correlations for
differently paced and different length moves.
The code currently looks at a total of three trends: hip minus knee elevation, knee minus
heel elevation, knee to knee proximity in the transverse plane, and heel to heel proximity
in the transverse plane. Note, for the first two trends, they must be correlated for each
side since it is a unilateral analysis. This produces a total of two sets of 6 individual trend
correlation coefficients (6 assuming same side orientation of move and 6 more assuming
opposite). For the combined correlation coefficient, I chose to use the root mean square
as opposed to the average or some other mean measure because it factors in standard
deviations; root mean square tends to create a larger discrepancy between significant and
insignificant coefficients, making the results clearer. The closest matching move is found
by maximizing the combined coefficient for some orientation. In the pursuit of simply
naming moves, this pattern recognition method seems to be superior since each move
tends to have unique and reproducible trends in relative joint motion and elevation. Over
time more characteristics may be added to further separate matches, considering the code
is designed to accommodate such changes.
Now, that the methods for recognition have been briefly discussed it is important to
describe how codified move sets are chosen and compared to the input move. If the
classification system was able to look at moves consistently and accurately it would be
possible to create a learning system, in which hard coding is unnecessary. This is the
premise behind only determining four basic characteristics with classification recognition
to narrow down possible moves for the cross-correlation analysis. The first step in
developing a model move set requires that the text files desired as “models” be placed
into the modelMoves folder and read in by a function called modelMoves(). For each
move in the folder, the pre-described classification analysis is performed, exactly as
would be done in idMove(). Then the series data is scaled and indexed using the spline fit
function spfit() so that it will be ready for cross-correlation. The resulting data is archived
to a *.mat, as well as the names of the moves which are pulled from the text file’s title
(title format: movename XX.txt). This information is ultimately used as a database to
compare the input move against when the actual identification program, idMove(), is
called. Which moves are model moves are initially chosen arbitrarily then manually
analyzed to make sure the expected trends are present and consistent with other trials, as
well as there being full marker data. This script currently takes a few minutes to run with
approximately 100 model moves. Although this may seem to be a long time, it is only
required to be ran once every time new model moves are added or code is changed, since
the data produced is permanently saved for use in idMove().
When performing the cross-correlation within idMove(), only model moves with the
same first four characteristics are compared, since this information is recorded in the
*.mat. Based on the maximum combined correlation coefficient, it is simple to tell what
model move the input file most closely resembles. From this, the script will suggest what
move the input trial most likely is.
In past versions, the classification system was used exclusively, with a lot of complex,
hard-coded logic. Although it worked fairly well, the maintenance required to add new
moves was extensive and too much work to realistically expand the model move database
past 40 or so moves. Using the current combined analysis, close to 100 moves can be
distinguished. The limit for the number of identifiable moves is theoretically endless as
all one has to do is add more files to the modelMoves folder and rerun modelMoves to
archive each move. With more moves, it is expected that more correlation characteristics
be developed and probably more classification characteristics as well.
Currently some of the classification code is developed using thresholds that are userdefined and not necessarily a function of the individual performing the move (as they
should be). Even when the thresholds are subject related, there are often moves which
teeter along a threshold and going one way or the other can cause the sequence to register
as an undesirable move and skew the rest of the analysis. These thresholds need to be
made more accurate or implement some sort of correlation for the classification system
that does not require the use of absolute thresholds.
Even with the progress made thus far, there are still many issues that need to be
addressed. Due to odd start/stop points and gaps in the middle of start/stop, sometimes an
incomplete sequence of the move is caught. Most moves that have this limitation are
currently unidentified; there are a few documented moves with this problem. If a move
such as this is used as a mode, all subsequent correlations involving it will be misguided
and irrelevant. Another resulting problem from gaps in data is determination of number
of rises and the accompanying points at which the foot leaves and re-contacts the ground.
As shown in the MATLAB command window when modelMoves is ran, there are
numerous moves that find foot offs but no foot downs. Eventually, the model moves will
need to be hand-picked to make sure they contain the marker data needed to accurately
and reproducibly portray the move. In some cases, it is obvious which trials are bad due
to built in errors, but others will be more subtle and require human verification.
Each move needs to be tested and verified as a viable template, because of the variability
of quality motion capture data. This task has been started and there is an excel file (Dance
Moves list Upd.xls) which contains all the move names that were recorded during all
dance recordings performed prior to the summer of 2011. In this file, which trial is used
as the model and which other trials do and don’t work with this model are noted. Green
highlighted moves indicate those which have been verified to work with all trials. Those
not highlighted are not perfect but work for most trials. Those highlighted in red have
issues that need to be addressed directly. In the description and non-working trials
columns, information can be found about what is wrong with the file(s). Note, that some
of the columns included in this spreadsheet are obsolete, as they were used with the pure
classification style of recognition.