Motion Capture – Art, Technology & Collaboration
Dr Peter Twigg and Professor Steve Hawley
1.0 Introduction
Dr Peter Twigg, a scientist/engineer, and artist Professor Steve Hawley, both from
Manchester Metropolitan University, have been collaborating in the past year in the
field of Motion Capture, the process of recording human movement via a suit fitted
with sensors and transforming this into a computer useable form, Motion Capture has
had many applications in recent years, but principally in the areas of video games and
animated films where lifelike movement would be difficult and time consuming to
accurately reproduce with traditional means.
But the field also throws up some other opportunities, for example in dance and
performance as well as in sport and medicine, and the possibility of a scientist/artist
collaboration gives rise to new approaches which would be unlikely from a single
perspective. In fact the history of Motion Capture predates the invention of motion
pictures, and pioneering work was done in the late nineteenth century by figures who
combined the interests of both the scientist and artist, inventing apparatus at the limit
of available technology which look to us remarkably contemporary. The paper
explores the nature of this collaboration in relation to Motion Capture, and looks also
at philosophical differences in approach, especially in relation to concepts of the
Uncanny.
The origin of the collaboration was an approach from the Royal Opera House Covent
Garden new media department, who were looking for ways of using new technologies
to promote dance to a large audience of predominantly young women, engaged in or
interested in classical ballet. The Royal Opera house is also the home of the Royal
Ballet and we looked amongst other themes at some ideas involving motion capture, a
specialism within the Science and Engineering Faculty. At that point we were looking
at how motion captured dancing figures could be used to engage young female dance
students who were also part of the videogame generation, to attract them as new
audiences to the Royal Ballet at Covent Garden. It should be said that the idea of
collaboration between two academics based in different Faculties in different
buildings was unusual within the university, and had come about from informal
contacts looking to see whether the technological resources available could be
explored in an experimental way by artists and designers.
The first experimental results were obtained at the Northern Ballet School, which
happens to be 200 metres from the main site of Manchester Metropolitan University,
where Peter Twigg and his colleagues obtained capture data from a male and a female
dancer engaged in a fairly extreme test of the capabilities of the motion capture suit.
Looking at these results confirmed firstly that the capture suit could indeed cope with
a range of extreme movement, and that this could be applied to other modes of
performance.
Steve Hawley is a video artist who has been making moving image work for
exhibition in galleries and film and video festivals since the early 1980s, sometimes
involving actors and performers. Seeing the ease with which performance could be
captured but also translated in post production to a variety of computer generated
figures, suggested possibilities for an art video. At the same time this melding of art
and science had echoes of the pioneers of motion capture in the latter half of the
nineteenth century.
1.2 History
Étienne-Jules Marey was a physiologist and Edweard Muybridge a photographer, but
as the two key figures in the genesis of this field, their work was both an important
precursor and a pointer to the future motion picture camera and projector.
Marey’s work on human motion was also very influential on the art of the Cubists and
Futurists who came after him. In 1882 he created his chronophotographic gun which
was capable of taking 12 consecutive frames a second, where all the frames were
recorded on the same picture. As well as conducting a famous study about cats always
landing on their feet, he also studied human locomotion. In his earliest experiments
the subjects wore black suits of metal strips or white lines as they passed in front of
the black backdrops , and these have a remarkable similarity to the present day
gyroscopic capture suit.
Figure 1 – Runner wearing Marey’s capture apparatus
More or less at the same time, the British-born photographer, and later scientist
Edweard Muybridge, engaged in a series of photographic experiments, to settle a bet
made by Leland Stanford the Governor of California and also a racehorse owner, as to
whether all four horses hooves were off the ground at the same time during a gallop.
Up until that time most paintings of horses galloping showed the front legs extended
forward and the hind legs extended to the rear.
Muybridge used a series of large cameras with glass plates placed in a line with the
horse triggering the thread. The images were viewed in a machine called a
Zoopraxiscope, which was another stage towards the film camera. Stanford won his
bet with Muybridge’s help, and later at the University of Pennsylvania Muybridge
used banks of cameras to photograph people and animals to study their movement.
Muybridge’s sequence of images have since been highly influential on artists and in
particular possibly the most influential artist of the last century Marcel Duchamp, who
made an early series of paintings that were heavily indebted to Muybridge and Marey
including Nude Descending a Staircase Number Two of 1912 .
1.3 Motion Data Capture
There are primarily two approaches to recording motion capture: electromagnetic
sensors, and optical markers, which each have their benefits and constraints. With
optical marker systems performers only have to wear reflective markers on their
clothes. The optical approach does not provide real-time feedback, however, and the
data from optical systems are error prone and noisy. The locating and extracting of a
figure’s motion directly from video when using optical systems is extremely
problematic for various reasons, particularly due to occlusion of data points. Because
orientation information is not directly generated, more markers are required than with
magnetic trackers. A subject whose motion is to be captured accurately, in real-world
situations and in real time is typically instrumented in some way so that positions of
key feature points can be easily detected and recorded. Electromagnetic tracking uses
sensors placed at the joints that transmit their positions and orientations back to a
central processor. To transmit their information, the sensors have to use either cables
or wireless transmission to communicate with the central processor. The former
requires that the subject be ‘tethered’ with some kind of cabling harness. The latter
requires that the subject also carries a power source such as a battery pack. The
advantage of electromagnetic sensors is that the three-dimensional position and
orientation of each sensor can be recorded and displayed in real time.
Types of motion capture recording and analysis carried out using sensor based
equipment has been generally limited to nondescript everyday movements such as
walking and extracting dominant features of an actor’s movements for application to a
digital character. In this work, motion capture techniques were used to record, classify
and analyse human motion of a more extreme nature: ballet. The motions of a subject
were captured wirelessly by use of the Animazoo IGS-190 equipment which employs
gyroscopic sensors about the body. The motion data captured is used: (i) to drive a
realistic virtual character in real time, (ii) for analysis of motion features, and (iii) for
development of performance design programmes. Conventional configuration of
articulated body system are designed primarily for walking, for example see Coleman
et al (2008) and Jain et al (2009), and application of the motion capture data for this
project required the development of an extended character rig configuration and
motion representation.
Data was collected using the Animazoo IGS 190 system and a virtual skeletal system
defined from front and side photographs of the subject using Animazoo Autocal
software. This produces a skeletal approximation that helps to eliminate positional
drift in the subject when in motion. The skeletal system is a 26 joint system as defined
within figure 1 The data is collected at a frame rate of 60fps or 60Hz. Translational
data is extracted from 12 of the subjects joints for the X, Y and Z axes. The
translational data comprised of the left shoulder, left elbow, left wrist, left hip, left
knee, left ankle, right shoulder, right elbow, right wrist, right hip, right knee and the
right ankle, thus providing a 36 dimensional data structure containing the subject gait
cycles.
Figure 2 – Skeletal system for motion capture application
The motion capture suit is lightweight and flexible, but this was the most extreme
movement that we had attempted to capture with the suit and the initial concern was
whether it would affect the dancer’s movement in some way. After performing some
test exercises though, the dancers reported that the suit was so non-restrictive that
they were not aware of its presence.
Figure 3 – recording motion capture data from one of the dancers, Nana
Figure 4 – Still image from real-time recording of one of the dancers, Nana
1.4 The Uncanny Valley
In 1970 the Japanese roboticist Masahiro Mori stated that as the likeness of a robot
approached a real human being there was a sudden drop-off of the audience’s
acceptance of the humanoid shape and this changed to a powerful negative reaction,
which he termed the Uncanny Valley. Seven years after Duchamp had painted Nude
Descending a Staircase Number Two, Sigmund Freud had published in 1919 his
famous essay on the Uncanny, Das Unheimliche, which has become very influential
on artists in the latter part of the 20th century, although was apparently unknown to
Mori. There is something profoundly uncanny about a humanoid model which
appears in many ways nonhuman, but whose movement is incredibly lifelike, which
motion capture can achieve. The uncanny may be a deep instinctual reaction in the
sense that it steers humans away from other humans who are dead or diseased.
Masahiro Mori’s conjecture was that things that look very nearly human but not quite,
such as a child’s doll or a ventriloquist’s dummy can produce a highly negative
response and for some reason the response is stronger with a moving object.
However this notion of the uncanny is not something that artists are seeking to reject,
but in some ways are actively pursuing. For example in Ron Mueck’s sculpture
Spooning Couple 2005, he created two bodies at a half scale, which appear to be
asleep but are actually profoundly eerie to us and produce a strong aesthetic reaction.
Motion capture has been utilised in many commercial motion pictures over the last 15
years but the uncanny valley effect has been attributed to the failure of some of these,
for example computer animated films like The Polar Express and Beowulf allegedly
evaded success because the viewers found the dead eyed characters unsettling. In
terms of the MMU collaboration, whilst engineers are interested in a perfected reality
that is indistinguishable from real human motion and representational human
appearance, it is precisely the marginal gulf between reality and the represented image
that artists are interested in. It could be argued that up to the late nineteenth century
the dominant viewer response sought by art was one of the sublime, whilst with
contemporary art the desired reaction is the uncanny.
Freud explores the idea of the double - the doppelganger in all its nuances and
manifestations. In Steve Hawley’s video installation Actor, made with Peter Twigg
and Andrew Brownridge, he created a doppelganger by putting an actor in a motion
capture suit and “filming” him whilst he was reading a prepared text. At the same
time an unseen director gives instructions to the actor to correct his performance. The
text is from the final chapter in the UK edition of Manchester born author Anthony
Burgess’s book A Clockwork Orange, subsequently made into a film
Stanley Kubrick based the film A Clockwork Orange on the American edition of the
novel, which only has 20 chapters as opposed to the English edition which has 21. He
omitted the final chapter, which is the redemptive chapter, where Alex the ultraviolent
droog decides that violence is not worth it and looks to the future and his unborn son.
Actor is the film of the unfilmed final chapter.
Figure 5 – Still image from Actor, Steve Hawley video installation
The effect of clothing the actor in the absurd knight in armour model whilst retaining
lifelike human movement, as well as isolating the actor within an invisible grey
setting, reinforces the sense of the uncanny.
1.5 Collaboration
Collaborations between artists and scientists and engineers are not new - the
Wellcome trust in England funded from 1996 a 10 year SciArt programme of
collaborations between scientists and artists which involved 118 projects and £3
million funding. A review of the scheme found a number of lessons. Firstly that artists
initiated and drove the majority of the projects, and that largely the artist produced the
work with scientists in some way acting as patrons. It was true that the scientists and
doctors were fascinated by the way that artists worked but for the scientists
professionally this SciArt scheme was seen as a marginal project - there were no
instances of scientists initiating projects who would then get an artist on board.
It was difficult to make claims for any ongoing benefits to the scientists involved, or
transformative effect on science. Where the collaborations had worked there tended to
have been a lengthy period of familiarization and exchange over some years, and the
MMU collaboration has deepened and grown through Peter Twigg and Steve Hawley
presenting at the CARPE conference together, as well as Steve Hawley’s cosupervision of Peter Twigg’s Ph.D. student..
1.6 Future directions
Possibilities include the classification of a specific vocabulary of ballet movements
for dance choreography. We are looking at developing a 3D skeleton and human
mesh with optimized real-time pipeline suitable for more extreme movement such as
ballet. We could also make a comparison of dancers’ motion data in real time with
exemplar performance data for high level dance tuition.
Artists are already using motion capture in a variety of ways and have been doing
since Duchamp painted Nude Descending A Staircase. For example within dance the
technique has been used for the capturing of fleeting and unique dance or bodily
movement performances e.g. Paul Kaiser and Shelley Escher of Riverbed with their
work with the legendary dancer Bill T Jones in their piece Ghostcatching. Motion
capture can translate human movement into music or abstract visuals, giving rise to
synaesthetic performances: this was first done by the artist Oscar Fischinger with his
Lumigraph in the late 1940s. More recently renowned U.S. choreographer Trisha
Brown premiered “how long does the subject linger on the edge of the volume” in
which music and visuals projected on a transparent screen reacted and responded to
dancers in the piece
In Steve Hawley’s work in collaboration with Peter Twigg he intends to explore the
uncanny by creating humanoid figures which are incredibly lifelike in terms of
movement and yet cause dissonance in the viewer. This will include multiple human
figures, human actors, who will be motion captured but interact in a subtle and
undemonstrative way. But there is also the possibility to explore the idea of moving
from that uncanny valley to a mythical resurrection and bringing to life the real
substantive human body, as in Shakespeare’s the Winter’s tale, where at the end
Hermione is presented as a statue to the court, then “magically” comes to life and
walks down off her pedestal
The MMU collaboration has been very fruitful, and the interesting part is not just the
outcome in terms of new uses for motion capture but also exploration of the nature of
collaboration between an engineer and an artist.
1.7 References

Coleman, P., Bibliowicz, J., Singh, K., Gleicher, M., (2008), Staggered Poses:
A Character Motion Representation for Detail-Preserving Editing of Pose and
Coordinated Timing, Computer Animation ACM SIGGRAPH

Freud, S., (1919), ‘The Uncanny’, pub Imago.

Jain, S., Ye, Y., Liu, C., K., (2009), Optimization-based Interactive Motion
Synthesis, ACM Transactions on Graphics SIGGRAPH.

Mori, M., (1970), ‘The Uncanny valley’, Energy, 7(4).

Glinkowski P, Bamford A. Insight and Exchange: An evaluation of the
Wellcome Trust’s Sciart programme. London: Wellcome Trust; 2009.
www.wellcome.ac.uk/sciartevaluation