Technological Investigation of the Physical Movements of Pianists
by
Jesse Latimer
AN HONORS THESIS
for the
HONORS COLLEGE
Submitted to the
Honors College
at Texas Tech University in
partial fulfillment of the
requirement for
the degree designation of
HIGHEST HONORS
DECEMBER 2014
_____________________________________
____________
Dr. Michael Sanfrancisco, Dean, Honors College
Date
_____________________________________
____________
Dr. Keira Williams, Honors College
Date
The author approves the photocopying of this document for educational purposes.
Abstract
An analysis of physical movements of pianists provides a unique application of
biomechanical engineering to piano pedagogy. This research presents a method of analyzing and
comparing joint-center movements when pianists are playing in each of two modes: “correct”
and “enjoyment.” Each subject plays two pieces in the correct mode (i.e., he or she was invited
to play them “as correctly as you can”) and then plays the same two pieces in the enjoyment
mode (i.e., “this time, just think about enjoying yourself – whatever that means to you”). Threedimensional motion capture was used to record the movement of the upper body (i.e., the hands,
wrists, elbows, shoulders, neck, head, vision, and spine) in the two modes. The difference in
forward/backward, vertical up/vertical down, and lateral right/lateral left movement for each
joint center, as well as the right- and left-hand arcs, was analyzed. T-tests were performed on the
movement data and the jerk data (resulting from the third derivative of the position data with
respect to time). Results suggest more movement in the enjoyment mode than in the correct
mode, while the results for the jerk data are more uncertain. Future work will include more
subjects and investigate the mode difference among genders.
Keywords: Motion capture; enjoyment mode; correct mode; joint center; human model.
ii
Acknowledgements
Above all, I would like to thank Almighty God Who has strengthened me throughout my
time in college and my undergraduate research. Without His kind and ever-present assistance,
none of this work would have been possible. “Now unto the King eternal, immortal, invisible,
the only wise God, be honour and glory for ever and ever. Amen.”
I wish to thank Dr. James Yang for having given me this unique research opportunity in
the spring of 2013 when he first took me on as an undergraduate researcher. He has always been
there to offer advice and assistance in my occasional struggles, especially at the beginning of my
research, to understand the concepts of biomechanics, and his inimitable sense of humor has
helped make the work seem lighter. In addition, his advice with regard to the writing of technical
engineering papers has been invaluable not only in my research but also in my other engineering
classes.
I cannot forget to express my deep gratitude to Aimee Cloutier, Dr. Yang’s Ph.D. student
who has helped me in many ways throughout the course of my work with Dr. Yang. Having
started out in Dr. Yang’s lab as an undergraduate researcher herself, she was able time and time
again to understand and thoroughly answer my many questions about biomechanics research,
especially with regard to writing journal articles and working with biomechanics motion-capture
software. Her friendly mentoring has been of great value to me.
Dr. William Westney’s method of teaching piano was the inspiration for this research. To
say that his well-spoken, clear explanations of the musical and pedagogical interpretations of the
results have been invaluable is a gross understatement. Dr. Cynthia Grund has given many
thought-provoking perspectives on and questions about the research as a professor of philosophy.
Dr. Michael O’Boyle, as a professor of neurosciences who himself has reviewed scientific
iii
journal articles in the past, has given valuable insight into the statistical analyses of the project
and has helped anticipate the questions that academic reviewers may ask about the research. I am
grateful for all three of these professors’ help.
I thank my family, including my parents, my grandmother, my five siblings (as well as
my sister-in-law and brother-in-law), and my many nieces and nephews for their continuous love
and support for me during the course of my studies here at Texas Tech University. I owe my
parents Sam and Mary Latimer greatly for their Christian example, the work ethic they have
instilled in me over the years, and the large amount of time and effort that they put into giving
me the best education possible. I could not have asked for better parents.
I would like to specifically thank my sister and best friend Stacey, without whom much
of the humor and French in my life would not exist and who has so often known how to edify
me. Je t’aime beaucoup!
I thank my dear church family and my many friends for always being there with a laugh,
a hug, a song, and a prayer when I needed them. They are all dear to my heart and soul.
iv
Table of Contents
ABSTRACT ............................................................................................................................................................... ii
ACKNOWLEDGEMENTS ................................................................................................................................... iii
TABLE OF CONTENTS ........................................................................................................................................ iv
LIST OF FIGURES (or Figure) ........................................................................................................................... v
LIST OF TABLES.................................................................................................................................................... vi
CHAPTER 1: Introduction and Literature Review.................................................................................... 1
CHAPTER 2: Data Collection and Analysis ................................................................................................. 6
CHAPTER 3: Results ......................................................................................................................................... 13
CHAPTER 4: Conclusion ................................................................................................................................. 32
APPENDIX ............................................................................................................................................................. 33
BIBLIOGRAPHY ................................................................................................................................................... 36
v
List of Figures
Figure No.
Figure 1: Marker placement .............................................................................................................7
Figure 2: Flow chart depicting the effects of each normalization .................................................10
Figure 3: The reference frame........................................................................................................10
Figure 4: Correct mode vs. enjoyment mode—Subject B, Grieg piece, spine end .......................14
Figure 5: Correct mode vs. enjoyment mode—Subject C, Grieg piece, head ...............................15
Figure 6: Correct mode vs. enjoyment mode: Subject A, Hummel piece, neck ............................16
Figure 7: Correct mode vs. enjoyment mode—Subject D, Hummel piece, vision ........................17
Figure 8: Comparative movement for Subject D for the Grieg piece ............................................21
Figure 9: Comparative movement for Subject A ...........................................................................22
Figure 10: Subject C, Grieg piece, example of jaggedness/smoothness........................................23
Figure 11: Subject C, Hummel piece, right-hand arcs ...................................................................26
Figure 12: Subject D, Grieg piece, left-hand arcs ..........................................................................27
Figure 13: All subjects, Grieg piece, vision, vertical up/vertical down ................................... 29-30
vi
List of Tables
Table No.
Table 1: Comparative movement for Subject D, Grieg piece ........................................................18
Table 2: Comparative movement for Subject A, Hummel piece ...................................................19
Table 3: Comparative movement for Subject A, Grieg piece ........................................................20
Table 4: T-test results for position data between modes, Grieg piece ...........................................23
Table 5: T-test results for position data between modes, Hummel piece ......................................23
Table 6: T-test results for jerk data between modes, Grieg piece ..................................................24
Table 7: T-test results for jerk data between modes, Hummel piece .............................................25
vii
Chapter 1
Introduction and Literature Review
Introduction
Body movement of pianists affects both their performance and the audience perception of
their performance; in general, communication to an audience involves body language as well as
sound. The use of the whole body when interpreting a musical piece is viewed by the musical
community as more aesthetically pleasing than simply playing the correct notes in the correct
rhythm (Mark, 2003). Moreover, learning and exercising correct performance techniques can
prevent injuries, which are very common in musical conservatories (Harer and Munden, 2008;
Mark, 2003).
The present experiment seeks to compare the movements of pianists when playing with an
emphasis on correctness to playing with an emphasis on enjoyment. The following questions
guided the present study, as put forward by Dr. William Westney, a professor in the School of
Music: “(1) Do piano students have a comprehensive and distinct neuro-muscular response to
generalized suggestions such as being asked to play ‘correctly’ or ‘with enjoyment’? (2) Are
there any significant differences between the two methods of playing in joint-center movements
and/or arcs of the hands? Does one mode produce more labored or constricted motions than the
other? (3) Are such differences, if any, consistent among pianists? (4) Does playing in the
‘enjoyment mode’ engender movement that is more physically expansive than the gestures
produced in the ‘correct mode’?” (Latimer et al, 2014)
Literature Review
Several studies have examined the relationship between musical performance and body
motion. A compendium of many philosophical and empirical analyses of musical motions, or
1
“gestures,” can be found in Godøy and Leman (2010). Many have sought to analyze what
differences in motion may exist when a pianist plays a piece with a greater or lesser amount of
expression (Clarke and Davidson, 1998; Davidson, 2007; Thompson, 2007; Shoda and Adachi,
2012; Thompson and Luck, 2012; Castellano et al., 2008). Each study concluded that there is a
definite connection between what might be termed concrete physical motion and the abstract
concept of musical expression.
Thompson and Luck (2012) investigated four expressive performance modes: “deadpan,
normal, exaggerated, and immobile.” The researchers used eight subjects and an eight-camera
motion capture system to compare how long each performance mode took as well as how much
movement there was per measure. An analysis of variance was performed to determine if there
was a greater difference in motion in the head and torso between modes than in the wrists and
hands. The results demonstrated that there was a difference in movement for body parts such as
the head for all mode comparisons except between immobile and deadpan; the left wrist
exhibited the least amount of difference in movement among modes. Also, only small
differences in movement were found between the deadpan and immobile modes, suggesting that
playing with little expression is closely related to playing with little movement. A related study
by Thompson (2007) used three expressive performance modes—“minimum, normal, and
maximum”—with three subjects and a musical piece by Brahms. The motion capture results of
the study demonstrated that a pianist’s expressive movements were directly related to the amount
of expression intended for each mode.
A similar study by Davidson (2007) used three modes with a single performer: in deadpan,
normal, and exaggerated modes, in part to analyze the effect that expressive musical
performance has on body movements. The results indicated that the subject used specific,
identifiable movements that went beyond the usual motion level of his or her musical
2
performance. In an earlier study, Davidson (2002) also reported that a pianist who performed the
same piece of music using different types of expression had varying amounts and types of
associated movement. In another study, Clarke and Davidson (1998) had one professional
performer play a piece by Chopin six times, two of which were analyzed. These performances
were recorded with a video camera. A unique aspect of the investigation was that the subject was
never told to play in any particular way. Three types of data were analyzed: musical instrument
digital interface (MIDI) data for expressive timing and dynamics; head position data, sampled
five times per second during each studied performance; and expressive head gestures obtained
from systematic observation of the recordings. Significant body sway was detected which did not
match the piece’s musical structure, and, while there were strong similarities between the two
trials, there was no fixed relationship between movement and the structure of the piece.
In addition, Castellano et al. (2008) studied the effects of playing a piece with differing
emotions. The subject played one piece in five different emotional contexts: sad, allegro, serene,
over-expressive, and “personally felt affect,” the last being a mode of expression that the pianist
thought best interpreted the piece. Upper-body movement and head velocity were analyzed with
respect to time. The authors found that sad movement produced less motion than the serene or
personal modes. It is also interesting to note that overall quantity of motion was not as indicative
of expression in the music as compared to head velocity.
In a motion-capture experiment, Van Zijl and Luck (2013) focused on audience perceptions
of various expressive intentions on the part of four different violinists all playing the same “sad”
passage. In this study, the various modes of performance (not revealed to the audience) were
called technical, expressive, and emotional – in this case “emotional” meant asking the
performers to access a deep personal sadness while they played. Watching motion-capture
3
videos, the audiences discerned clear differences among these three modes of performance, and
their preference was for the artistically “expressive” mode, which they found the most satisfying.
Furthermore, Shoda and Adachi (2012) sought to discover the relationship between upper
body movement and degree of musical expression. Three modes were used—deadpan,
exaggerated, and artistic—while playing two differently structured musical pieces. The artistic
mode was meant to reflect the true nature of the music and was therefore used as an interpretive
backdrop to which the other two modes were compared. The more energetic musical piece
created more movement differences among the modes of expression, while the slower musical
piece produced a motion differences only between the deadpan and artistic modes.
The present study differs from previous work in several important ways. This research aims
to shift the emphasis away from interpretation and expression and instead focus purely on body
motion and how it might change when subjects are given minimal instruction—to play
“correctly” or “with enjoyment.” For this reason, the researchers made sure not to allude
specifically to any body movement, or to any particular behavior, when giving instructions to the
participating pianists. Additionally, the concept of proposing to subjects in a non-directive way
that they play either “correctly” or “with enjoyment” is unique to this study.
Most pianists today would agree that musical expression is an important aesthetic quality of
performance. However, previous studies have restricted themselves to analyzing the physical
manifestations of such expressivity. The present investigation, by contrast, seeks to go one step
further and asks the question: is musical expression more effective, and is body use more
coordinated and pleasing, when the pianist is focused on enjoying the performance of the piece,
as compared to when they are concentrating on playing the piece as correctly as possible? Such a
study has important implications for the field of piano pedagogy, where the traditional method
of tutelage has often been to encourage students to supervise and control certain “correct”
4
positions and movements in a conscious way when playing a piece of music. In contrast to this
method, another approach has been suggested which assumes that the body will naturally fall
into optimal motions, coordinations, and positions when a well-trained student is invited to enjoy
his or her performance. Philosophies of technique that trust the body to possess a sophisticated
and subtle wisdom that surpasses the (possibly reductionistic) concepts of the mind were
articulated as early as 1905 by Friedrich Steinhausen , as quoted in Kochevitsky (1967) and
others (Whiteside, 1961, and Green, 1986).. This study, therefore, seeks to investigate whether
this more holistic approach to piano pedagogy can produce recognizable results in both
movement and aesthetic quality of the musical performance (Westney et al, 2014).
5
Chapter 2:
Data Collection and Analysis
Data Collection
The present motion capture study involved two modes of playing: “correct” and “enjoyment”
mode. Each performer was asked to play both pieces twice, once in each mode. In the correct
mode, the subjects were asked to play the piece as technically correct as possible, as contrasted
with enjoyment mode, in which subjects were asked to simply enjoy themselves while playing
the piece. Thus, there was a total of sixteen motion capture trials, one for each mode for each
piece for each subject.
Subjects for the experiment were chosen based on their skill in piano performance; three are
graduate students of piano in the Texas Tech School of Music, and one is a faculty member at
the same institution. Accordingly, two males and two females were chosen with a mean weight
of 125.45 (17.52) lb, a mean height of 165.375 (6.575) cm, and a mean age of 31 (6) years. This
selection of subjects is different from the experiment performed by Thompson et al. (2011), in
which five female and three male subjects whose musical ability level ranged from professional
musicians to piano hobbyists.
The musical pieces specifically selected were “Scherzo” by Johann Hummel and “Cowherd’s
Song” by Edvard Grieg. Because these pieces are not widely known, it is not likely that the
subjects would be familiar with them already. Furthermore, they are both easily learned and
memorized and require different technical skills and expressivity since the Hummel piece is
faster, brighter, and more technically risky while the Grieg piece is slower, warmer, and more
6
meditative (Latimer et al, 2014, Westney et al, 2014). (See the Appendix for the sheet music of
both of these pieces.)
The motion capture system used for experiments was an eight Eagle 4 camera system
(Motion Analysis ®) (Cloutier et al., 2011). Only the upper body was modeled because the piano
would obstruct lower-body markers and because there is little movement in the lower body
during piano performance. Forty-six surface markers were used, from which twenty-three virtual
markers were created. Marker placement is shown in Figure 1.
(a)
(b)
Figure 1: Marker placement: (a) front, (b) back
Data Analysis
There were several significant initial facts about the motion capture data. First, each subject
had a different seated height, so that the frame-by-frame distance between joint centers was not
equivalent with respect to the global coordinate system across subjects. Second, each subject
played in varying tempos between the correct and enjoyment modes for each piece, i.e., did not
maintain a consistent tempo throughout each piece. Third, each subject played each piece at
different tempos (speeds).
7
Given these facts, in order to graphically compare the data, three different types of
normalization were required (see Figure 2). First, for each subject, the distance traveled by each
joint center from one frame to the next was divided by the distance from that joint center to the
hip joint, thus creating a normalized distance that could easily be compared between subjects.
Second, each piece was segmented into several divisions that began and ended at transitional
moments in the piece; the Hummel piece was divided into fourteen segments, while the shorter
piece by Grieg was divided into nine segments. Both pieces with their corresponding
segmentations are shown in the Appendix. Within each segment, the number of frames was
compared between the two modes. Next, the mode containing the largest number of frames for
each division had a number of evenly spaced frames removed from the data for that segment
until each mode contained the same amount of frames. This solution was not ideal, but it is worth
noting that the number of frames eliminated was small compared to the total number of frames in
each segment and in the piece as a whole. For the second normalization, an average of 2.09% of
the data was eliminated with a standard deviation of 1.79%. After each segment was normalized,
a graph was created that compared the motion of either hand along the piano keys (termed the zdirection). This was done in light of the fact that the hands must stay in roughly the same
position in order to play the right notes. If the graphs corresponded to each other fairly well, the
normalization was deemed to be successful for that division. In the end, each mode would have
the same number of frames for each piece, making a frame-by-frame analysis of the
corresponding motion possible.
Third, the normalization between subjects was similar to the second normalization described
above but performed separately from it. The number of frames for each segment in each piece
was compared among the four subjects, after which evenly spaced frames were removed from
each subject’s data until each had the same number as the subject with the lowest total frames.
8
This method eliminated more data in the third normalization compared to the second, because
the tempo differences across subjects were often quite large. For the third normalization, an
average of 16.5% of the data was eliminated with a standard deviation of 14.0%. Because the
motion capture data collection rate was 120 frames/second, it is unlikely that removing this data
significantly impacted the analysis.
Raw Data
First Normalization
Dimensionless, normalized data
Second Normalization
Comparison between two modes, same subject
Third Normalization
Comparison between multiple subjects, same mode
Figure 2: Flow chart depicting the effects of each normalization: the first normalization reduces
the impact of differences in seated height; the second normalization creates a more equal spacing
for comparison of two modes and one subject; the third normalization creates a more equal
spacing for comparison of multiple subjects and one mode
After performing the second and third normalizations, the movement of each subject’s joint
centers was compared to a reference frame. The same reference frame was used for both the
correct and the enjoyment modes so as to provide a common standard by which the modes could
be compared. The reference frame was selected as the one in which the pianist had his or her
hands on the piano, equally spaced from the center of the body, with the head looking downward
at the keys. Figure 3(b) shows an example of a reference frame, which can be compared to the
9
photo in Figure 3(a) of the pianist at the keyboard. For the graphical joint-center analysis, each
frame’s joint-center positions were subtracted from the corresponding joint-center positions
depicted in the reference frame. Meanwhile, for the statistical analysis, the first frame’s joint
center position was subtracted from the reference frame positions, the next frame’s position was
subtracted from the first frame’s position, and so on. The positions of each joint center in all
three directions (i.e, forward/backward, vertical up/vertical down, and lateral right/lateral left)
were graphed over the entire piece. Corresponding joint center directions for each mode were
plotted on the same graph, creating comparisons like the ones in Figure 4 in the following
chapter. Only one direction for each joint center is shown on each graph because including
multiple directions makes the plots difficult to read.
(a)
(b)
Figure 3: The reference frame: (a) Subject D at the keyboard; (b) Subject D’s 3D “stick figure”
The analysis performed on the data from these experiments was similar to that described in
Van Zijl and Luck’s paper (2013); we preformed a T-test, so that a concrete conclusion regarding
the difference in the amount of motion and jerk between the two modes could be obtained.
A few changes needed to be made before the position T-test could be performed. First, rather
than measuring every frame with respect to the reference frame as in the first set of analyses,
10
only the first frame was compared to the reference frame Every other frame besides the first
frame was measured with respect to the position in the frame immediately preceding in x, y, and
z directions, respectively as  x  y , and  z , thus eliminating the need to normalize the data with
respect to time Next, the change in position was calculated for every frame using Equation 1:
Δ = √Δ2𝑥 + Δ2𝑦 + Δ2𝑧
(1)
Finally, the sum of the distance was calculated for every joint center in each mode, tracking the
total distance that the joint center moved during the experiment. In the end, each joint center had
two numbers, one for each mode, quantifying the distance traveled over the piece. The mean and
the standard deviation of the joint centers’ motion were then calculated for each mode. Using this
mean and standard deviation, a paired one-tailed T-test was performed, comparing each subject’s
correct mode for each piece to the corresponding enjoyment mode.
The jerk, defined as the third derivative of the position data, was calculated numerically in
MATLAB. The same data preparation and the same type of T-test were performed as with the
position data.
The null hypothesis for the T-test was that the population means for the correct and
enjoyment modes are equal, given the sample mean and standard deviation. Results were
evaluated using a p value of 0.05.
11
Chapter 3:
Results
Scatter charts for selected joint centers and subjects are shown in Figures 4 through 7,
depicting the movement in both correct and enjoyment modes for a whole piece. As these graphs
show, there is in generally more movement in the enjoyment mode than in the correct mode.
However, another interesting result emerged: there was a significant correlation between level of
piano aptitude and motion differences between the two modes. For example, Subject A, the most
experienced pianist, exhibits the least amount of difference between modes, as seen in the
example of her scatter plots (Figure 6). Subject C, the next most proficient performer, has a
similar lack of difference between modes, although there is somewhat more of a difference than
Subject A’s (Figure 5). This result can also be seen by a comparison of the cumulative bar graphs
in Figures 8 and 9: in the correct mode, Subject A exhibits a much greater amount of
comparative movement than Subject D. It is speculated that this lack of difference between
modes occurs as the pianist with greater skill allows himself or herself more freedom in either
mode than does the less proficient player.
12
(a)
(b)
(c)
Figure 4: Correct mode vs. enjoyment mode—Subject B, Grieg piece, spine end: (a)
forward/backward; (b) vertical up/vertical down; (c) lateral left/lateral right
13
(a)
(b)
(c)
Figure 5: Correct mode vs. enjoyment mode—Subject C, Grieg piece, head: (a)
forward/backward; (b) vertical up/vertical down; (c) lateral right/lateral left
14
(a)
(b)
(c)
Figure 6: Correct mode vs. enjoyment mode: Subject A, Hummel piece, neck: (a)
forward/backward; (b) vertical up/vertical down; (c) lateral right/lateral left
15
(a)
(b)
(c)
Figure 7: Correct mode vs. enjoyment mode—Subject D, Hummel piece, vision: (a)
forward/backward; (b) vertical up/vertical down; (c) lateral right/lateral left
Additionally, an analysis considering only the magnitude of movement from the reference
frame was conducted. The chosen metric for this comparative movement M is shown here in
Equation 1.
M    xe  xc

In Equation 1, xe and xc represent each frame’s movement from the reference frame for
enjoyment mode and correct mode, respectively. This calculation created one number that
summarized the comparative movement of a joint center in one direction. These numbers were
then grouped by joint center and placed in Tables 1-3 and a corresponding bar graph. As
displayed in Figures 8 and 9, a bar in the negative region indicates comparatively more
movement in the correct mode than in the enjoyment mode and vice versa. (Note: Spine End is
16
located at the top. Spine 2, Spine 3, and Spine 4 are equally spaced between the hip center and
Spine End.)
Table 1: Comparative movement for Subject D, Grieg piece
Joint Centers
Forward/Backward
Vertical Up/Vertical
Down
Lateral
Right/Lateral Left
Spine 1
21.01320416
17.21876077
6.482816503
Spine 2
45.74454338
23.12615859
12.01166237
Spine 3
71.46489589
24.57459007
18.39819496
Spine 4
102.9947432
22.85743442
20.87460463
Spine End
158.4400787
8.562831331
34.66210334
Right Clavicle
156.9709775
10.96205891
33.56511708
Right Shoulder
137.6786339
-17.14504499
29.19659169
Right Elbow
32.32624653
33.01785853
12.30612257
Right Wrist
65.68032261
-20.22048124
-5.107779216
Right Hand
67.53334586
-19.08432439
-11.7056761
Left Clavicle
156.5516032
23.6251402
30.15606051
Left Shoulder
131.6613665
18.61360727
25.15354021
Left Elbow
62.49358062
42.63216337
22.66603355
Left Wrist
76.04462949
-3.608600906
-11.07271678
Left Hand
70.78712801
-2.204459001
-17.28969392
Neck
204.4580366
21.64162439
28.49335749
Head
238.7961993
27.50490245
39.59564939
Forehead
316.5037191
116.0020587
98.62366573
Vision
320.443864
122.1202696
101.7952952
17
Table 2: Comparative movement for Subject A, Hummel piece
Joint Centers
Forward/Backward
Vertical Up/Vertical
Down
Lateral
Right/Lateral Left
Spine 1
-9.939488494
12.6891171
3.375366627
Spine 2
-25.77638535
10.73825284
-5.181233245
Spine 3
-39.5424408
19.94564042
-11.13320195
Spine 4
-83.2635045
28.53123241
-6.574416696
Spine End
-41.04142462
19.78110105
20.50259051
Right Clavicle
-36.21018288
-0.164771576
26.76719528
Right Shoulder
0.310901087
-16.89712099
32.29441926
Right Elbow
-1.920913813
-13.82885791
-52.97698748
Right Wrist
-42.74259858
35.71740456
-68.81761288
Right Hand
-50.9477032
32.01353728
-62.59864417
Left Clavicle
-48.36987139
12.60860777
32.65434864
Left Shoulder
-40.87424183
29.71872048
21.85431073
Left Elbow
-26.99316734
88.7021262
68.97442496
Left Wrist
-25.28617885
31.30106934
91.61528266
Left Hand
-36.02070503
36.6416986
96.82347073
Neck
-22.49550979
18.57649797
41.03231773
Head
13.68082749
-30.27420173
88.56149162
Forehead
58.7517425
-27.96292146
216.1164951
Vision
58.82418215
-28.51621178
218.0052818
18
Table 3: Comparative movement for Subject A, Grieg piece
Joint Centers
Forward/Backward
Vertical Up/Vertical
Down
Lateral
Right/Lateral Left
Spine 1
4.235354713
18.32421249
5.573835653
Spine 2
2.559852231
29.64071217
-7.582247218
Spine 3
3.190950874
43.0873252
-24.46785577
Spine 4
6.615695426
51.54782348
-31.50081975
Spine End
22.74823384
11.48823377
-37.51682754
Right Clavicle
19.07918521
-26.17087367
-33.3844148
Right Shoulder
-13.49009502
-76.26467539
-28.4389457
Right Elbow
30.31834303
-7.354637069
9.062062724
Right Wrist
16.29491656
18.70975349
-2.955440496
Right Hand
11.93294647
17.02207045
0.61289638
Left Clavicle
22.87478759
-11.45800469
-46.6782247
Left Shoulder
30.86665338
-21.38166576
-56.07727726
Left Elbow
63.33328618
38.94832491
-76.2332076
Left Wrist
-0.395584096
34.78813216
-33.10297373
Left Hand
-20.20632518
21.96926949
-34.09204986
Neck
43.9773338
9.357775181
-35.16060032
Head
82.95543209
-62.14069799
-0.330878944
Forehead
114.7085386
-81.06939347
45.54682893
Vision
118.5312078
-82.05090274
44.203697
19
Figure 8: Comparative movement for Subject D for the Grieg piece
20
(a)
(b)
Figure 9: Comparative movement for Subject A: (a) the Hummel piece and (b) the Grieg piece
The results of the statistical analysis of the position for the Grieg piece is shown in Table 4.
21
Table 4: T-test results for position data between modes, Grieg piece
Subject 1
Subject 2
Enjoy
Mode
Correct
Enjoy
Correct
Subject 4
Enjoy
Correct
ment
Mean
Subject 3
Enjoy
Correct
ment
ment
ment
4.54
4.45
4.42
6.36
1.85
2.05
3.05
3.76
0.89
0.84
0.55
0.90
0.20
0.22
0.35
0.45
Standard
Deviation
1-Tailed
0.018
0.00013
0.0012
0.00023
T-Test
Using a p value of 0.05, it can be seen that the population means are indeed significantly
different, as would be expected with the experiment’s hypothesis. Among the subjects, Subjects
2 and 4, the less experienced performers, show smaller p values than Subjects 1 and 3, indicating
larger differences between modes for the less experienced subjects.
Table 5 shows the results for the position data between modes in the Hummel piece.
Table 5: T-test results for position data between modes, Hummel piece
Subject 1
Subject 2
Subject 3
Subject 4
Mode
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Mean
7.01
7.08
7.19
11.14
4.42
5.64
5.35
7.66
Standard
Deviation
1.41
1.36
1.05
1.72
0.49
0.67
0.96
1.16
1-Tailed TTest
0.11
0.00011
0.000084
22
0.000013
Surveying the results in Table 5 with a p-value of 0.05, it is evident that Subjects 2, 3, and 4
exhibit quite different amounts of movement in the enjoyment mode than in correct mode.
However, Subject 1 has a much greater value than 0.05, meaning that the null hypothesis,
namely that the population means for the two modes are the same, cannot be rejected.
It was hypothesized that the correct mode for each piece would exhibit what could be
termed labored, “jagged” movement while the enjoyment mode would exhibit “smooth”
movement. That is, the performer would tend to make more exact, more confined movements in
the correct mode, in an effort to control every detail of the piece. This result contrasted with a
more freely flowing and trusting movement in the enjoyment mode. Figure 10 is an example
from Subject C’s performances of the Grieg piece.
Figure 10: Subject C, Grieg piece, example of jaggedness/smoothness (right clavicle,
forward/backward)
The T-test results for the two modes’ jerk for the Grieg piece are shown in Table 6.
23
Table 6: T-test results for jerk data between modes, Grieg piece
Subject 1
Subject 2
Subject 3
Subject 4
Mode
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Mean
0.095
0.062
0.022
0.027
0.0078
0.0078
6.31
7.58
Standard
Deviation
0.022
0.023
0.0059
0.0046
0.0053
0.0037
0.52
0.57
1-Tailed TTest
0.00037
0.016
0.49
0.000064
These results are somewhat less favorable to the experiment’s hypothesis that there would be less
smooth movement in the correct mode—implying more jerk in that mode—than there would be
in enjoyment mode. Subject 3’s value is much more than the p value of 0.05, while the other
three subjects are still under that p value. However, looking at the sample means, Subjects 2 and
4 appear to exhibit more jerk in the enjoyment mode than in the correct mode, contrary to the
experimental hypothesis. Nevertheless, though it is true that we cannot conclude smoother
movement in enjoyment mode than correct, it is still true for most subjects that there is a
difference between correct and enjoyment modes.
Finally, Table 7 shows the results of the T-test for the jerk data between modes for the
Hummel piece.
24
Table 7: T-test results for the jerk data between modes, Hummel piece
Subject 1
Subject 2
Subject 3
Subject 4
Mode
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Correct
Enjoym
ent
Mean
0.088
0.036
0.056
0.045
8.29
9.25
0.12
13.56
Standard
Deviation
0.021
0.0047
0.024
0.013
0.58
0.65
0.0036
1.08
1-Tailed TTest
0.0025
0.043
0.0014
0.0000049
The results in Table 7 appear to be more favorable for the hypothesis. However, as with the
results in Table 6, the results in reality are mixed: Subjects 3 and 4 have a greater mean in
enjoyment mode than in correct mode, contrary to the experiment’s hypothesis, while Subjects 1
and 2 have the opposite trend.
Another area of interest for this experiment was the hand arcs created by the pianist in each
mode of interpreting the piece. In piano performance, moving in flowing, graceful arcs along the
keys is often viewed as being more aesthetic and less taxing for the performer. Our hypothesis
was that there would be more prominent hand arcs in the enjoyment mode than in the correct
mode since the performers would be giving themselves more freedom in playing. In order to
create these graphs, the data for the right and left hands (still with respect to the reference frame)
was graphed for y vs. z—that is, vertical up/vertical down data on the vertical axis and lateral
right/lateral left data on the horizontal axis. Additionally, 3D depictions of the vertical and lateral
movement with respect to time are shown in Figures 10 and 11. These visual results tend to
confirm the hypothesis that performers in the enjoyment mode generally display more motion
than they do in the correct mode, although more differences were seen for the Hummel piece
than for the Grieg piece. A statistical analysis was attempted, but there was not enough data to
25
form a definitive conclusion. Further experimentation with more subjects is necessary to
definitively confirm these results.
(a)
(b)
Figure 11: Subject C, Hummel piece, right-hand arcs: (a) 2D graph; (b) 3D graph
26
(a)
(b)
Figure 12: Subject D, Grieg piece, left-hand arcs: (a) 2D graph; (b) 3D graph
The next analysis was a comparison between subjects. This required the third normalization
in order to perform a frame-by-frame analysis similar to the first analysis between modes.
Unfortunately, the rather large differences in tempo between subjects caused the segment-bysegmentnormalizations to be less accurate than for the other types of normalization. An example
27
of the vision data comparison between subjects is shown in Figure 13.The correct mode graph,
Figure 19(a), has the same vertical scale as the enjoyment mode, facilitating a general
comparison between the two modes for each subject. The subject-by-subject analysis was largely
confined to creating graphs such as the one in Figure 13.
(a)
28
(b)
Figure 13: All subjects, Grieg piece, vision, vertical up/vertical down: (a) correct mode and (b)
enjoyment mode
29
Chapter 4:
Discussion and Conclusions
In this study, the movement of pianists while performing a piece was analyzed using a 3D
motion capture system. The hypothesis that performers in enjoyment mode would exhibit more
motion than in the correct mode was supported. It appears that a pianist’s skill level plays an
important role in how different the motion in each of the two modes might be, given that Subject
1, the most skilled pianist, showed the least difference between modes while less experienced
subjects showed more difference. However, many improvements can be made for future research
to the experiment and the process of analysis.
One potential weakness of the present research is the reference frame from which all
movement was measured. The definition of more or less movement in the study depends heavily
on an accurate assessment of this quantity. However, there were some cases in which the current
reference frame may not be acceptable for defining the extent of movement. Choosing the
correct reference frame from which to measure movement changes is not a trivial process and
there are several methods of doing so. If the choice of reference frame (i.e. hands on the piano,
head facing the keys) is considered acceptable, a suggestion for future research would be to have
the subjects begin each trial in a specified reference pose, rather than searching through all
motion capture trials until the pianist attains the specified reference position.
Another idea would be to change the experimental setup to have the pianist perform each
piece in three modes: a correct and enjoyment mode as was currently done, and an additional
mode, perhaps referred to as the “performance mode”. Theoretically, the latter would be a mode
in which the experienced pianist combines aspects of both the correct and enjoyment modes to
put forward his or her “best” performance, i.e., a performance that is correct, dynamic, and
30
aesthetically pleasing. The performance mode could then be used as a reference to which the
other modes would be compared. On a cautionary note, it is unclear just how the playing in the
“performance mode” may affect the mindset of the subjects when playing in the other two
modes.
As mentioned in the Data Analysis section of the paper, performing the third normalization
resulted in the removal of a relatively large amount of data—an average of 16.5% (SD= 14.0%)..
While it is unlikely that the removal of this data significantly impacted the results, a better
method for both the second and third normalizations which minimizes the removal of data is
preferable. One possible technique to accomplish this would be to implement the nonlinear
time-warping algorithm developed by Verron (2005) and used previously in the Wanderley et al.
(2005) and Thompson et al. (2012) investigations. Employing this algorithm would manipulate
the data so that the onset of each measure would be aligned, regardless of performer or mode,
and would not remove any data.
With regard to the statistical analyses performed in the paper, the primary experimental
hypothesis, that there would be more movement in the enjoyment mode than in the correct mode,
has been carried out. Moreover, viewing Tables 4 and 5, one can see that Subject 1, the most
experienced subject, has the greatest T-test values, meaning that the movement between modes is
in reality closer to being equal than that of the other subjects. This trend suggests that there may
be an effect of skill level on the motion in the two modes. These statistical results lead to an
affirmative answer to the first question asked at the beginning of the research.
Unfortunately, although the T-tests for the jerk data appear to have favorable results, those
results are in fact mixed. Because of the small number of subjects, no definite trend for or against
the experimental hypothesis that movement in the enjoyment mode would be smoother can be
31
seen. Thus, the second question asked at the beginning of the research cannot be answered
definitively.
As implied from the previous paragraph, other limitations of the study stem from the fact that
the pool of subjects was small. Having more than four would provide a stronger statistical test of
the results and would perhaps shed light on some of the questions unanswered by this study,
including whether swifter, more labored movements are more common in the correct mode than
in the enjoyment mode. It may also illuminate the significance of other factors such as gender.
Moreover, it is likely that the skill level of the pianist has an effect on motion in each of the two
modes and should be further investigated to determine its impact.
It might be mentioned that in addition to the motion-capture study, members of the research
team are investigating brain responses of those watching these motion-capture performances. A
working hypothesis is that more brain activation will occur in those watching performance
during the enjoyment mode than the correct mode. The latter would provide insight into whether
or not the proposed method of piano pedagogy is beneficial in terms of audience perception.
In conclusion, the present study advances the work of the other investigators discussed in the
literature review mostly by its extensive analysis, covering both visual and statistical methods of
analysis as well as studying joint-center position, hand arcs, and jaggedness vs. smoothness, and
its relatability to the field of piano pedagogy. The importance of this study in that field can be
seen especially with regard to the fact that motion in the enjoyment mode has been shown to be
more expansive than that in the correct mode. Based on this preliminary investigation, when
teaching piano, it appears to be more effective when the student has already attained a relatively
high skill level to have them learn by simply enjoying the piece while playing it rather than
stressing correctness.
32
Appendix
Division 1
Division 2
Division 3
Division 5
Division 7
Division 4
Division 6
Division 8
(a)
33
Division 9
Division 10
Division 11
Division 12
Division 13
Division 14
(b)
Figure A1: Hummel piece with divisions used for second and third normalizations marked: (a)
Page 1 and (b) Page 2
34
Division 1
Division 2
Division 3
Division 4
Division 6
Division 5
Division 7
Division 8
Division 9
Figure A2: Grieg piece with divisions used for second and third normalizations marked
35
References
1. Castellano, G., Mortillaro, M., Camurri, A., Volpe, G., and Scherer, K., (2008). Automated
analysis of body movement in emotionally expressive piano performances. Music
Perception, 26(2), 103-119.
2. Clarke, E.F. and Davidson, J.W., (1998). The body in performance. In Thomas, W. (Ed.):
Composition-Performance-Reception, 74-92. Aldershot, UK: Ashgate Press.
3. Cloutier, A., Boothby, R., Yang, J., (2011). Motion capture experiments for validating
optimization-based human models, 14th Conference on Human-Computer Interaction, July
09-14, 2011, Orlando, FL, USA.
4. Davidson, J.W., (2002). Understanding the expressive performance movements of a solo
pianist. Musikpsychologie, 16, 9-31.
5. Davidson, J.W., (2007). Qualitative insights into the use of expressive body movement in
solo piano performance: a case study approach. Psychology of Music, 35(3), 381-401.
6. Green, B., (1986). The Inner Game of Music. New York: Doubleday.
7. Godøy, R. and Leman, M., eds. (2010). Musical Gestures: Sound, Movement, and Meaning.
New York: Routledge.
8. Gallwey, W. Timothy, (1974). The Inner Game of Tennis. New York: Random House.
9. Grund, C.M and Westney, W. (2010) Music, Movement, Performance & Perception:
Perspectives on Cross-Disciplinary Research and Teaching within NNIMIPA - Nordic
Network for the Integration of Music Informatics, Performance and Aesthetics. Online
version at: http://www.nnimipa.org/CM.html.
10. Harer, J. B., and Munden, S., (2009). The Alexander Technique Resource Book: A Reference
Guide. Lanham: Scarecrow Press, Inc.
11. Jensenius, Alexander Refsum (2009). Musikk og bevegelse. Unipub.
12. Jensenius, Alexander Refsum; Kristian Nymoen; Cynthia M. Grund; William Westney; Ståle
A. Skogstad. (2010) “Video Suite – in Three Movements: Jensenius-Westney-Grund on
Motion-capture, Music and Meaning.”. Webpage: http://www.nnimipa.org/JWG.html.
13. Kochevitsky, G., (1967). The Art of Piano Playing. Miami, FL: Summy-Birchard, 13-14.
14. Latimer, J., Cloutier, A., Yang, J., Westney, W., O’Boyle, M., Grund, C. (2014)
Technological investigation of the physical movement of pianists. Unpublished journal
article, Texas Tech University, Human-Centric Design Research Laboratory.
15. Mark, T., (2003). What Every Pianist Needs to Know about the Body. Chicago: GIA
Publications, Inc.
16. Shoda, H., and Adachi, M. (2012). The role of a pianist’s affective and structural
interpretations in his expressive body movement: a single case study. Music Perception,
29(3), 237-254.
17. Thompson, M.R. (2007). Expressive gestures in piano performance. Master’s thesis,
University of Jyväskylä.
18. Thompson, M.R. and Luck, G. (2012). Exploring relationships between pianists’ body
movements, their expressive intentions, and structural elements of the music. Musicae
Scientiae, 16(1), 19-40.
19. Van Zijl, A. G. W., & Luck, G. (2013). Moved through music: The effect of experienced
emotions on performers’ movements characteristics. Psychology of Music, 41(2), 175-197.
20. Verron, C. (2005). Traitement et visualization de données gesturalles captées par Optotrak
[Processing and visualizing gesture data captured by Optotrak]. Unpublished lab report,
McGill University, Input Devices and Music Interaction Laboratory.
36
21. Wanderley, M. M., Vines, B. W., Middleton, N., McKay, C., Katch, W. (2005). The musical
significance of clarinetists’ ancillary gestures: An exploration of the field. Journal of New
Music Research, 34(1), 97-113.
22. Westney, A., Grund, C., Latimer, J., Cloutier, A., Yang, J., O’Boyle, M. (2014) The
pedagogical value of “enjoyment” – A research report on a transdisciplinary study.
Unpublished journal article, Texas Tech University, School of Music.
23. Whiteside, A. (1961). Indispensables of Piano Playing (2nd ed.). Charles Scribners Sons,
New York, 77.
37