Motion Capture – Art, Technology & Collaboration
Peter Twigg, Manchester Metropolitan University, p.twigg@mmu.ac.uk
Steve Hawley, Manchester Metropolitan University, s.hawley@mmu.ac.uk
1. Introduction
Dr Peter Twigg, a scientist/engineer, and artist Professor Steve Hawley, both from
Manchester Metropolitan University (MMU), have been collaborating in the past year
in the field of Motion Capture, the process of recording human movement via a suit
fitted with sensors and transforming this into a computer useable form.
The possibility of a scientist/artist collaboration gives rise to new approaches which
would be unlikely from a single perspective, and the aim of this paper is to explore
these – the very form of the paper, with sections written by the artist and the scientist,
reflects the disjunctions and synergies of the process. In particular the article explores
the nature of the concept of the uncanny from the viewpoint of science and art.
Masahiro Mori, the roboticist, saw the uncanny as a negative characteristic of lifelike
machine forms, whereas artists, influenced by Freud’s writings, on the contrary desire
to exploit this aesthetic response.
Motion Capture has had many applications in recent years, but principally in the areas
of video games and animated films where lifelike movement would be difficult and
time consuming to accurately reproduce with traditional means; the technology also
throws up some other opportunities, for example in dance and performance as well as
in sport and medicine. Interestingly, the history of Motion Capture predates the
invention of motion pictures, and pioneering work was done in the late nineteenth
century by figures who combined the interests of both the scientist and artist,
inventing apparatus at the limit of available technology which look to us remarkably
contemporary.
The origin of the collaboration was an approach from the Royal Opera House Covent
Garden new media department, who were looking for ways of using new technologies
to promote dance to a large audience of predominantly young women, engaged in or
interested in classical ballet. The Royal Opera house is also the home of the Royal
Ballet and we looked amongst other themes at some ideas involving motion capture, a
specialism within the Science and Engineering Faculty. At that point Twigg and
Hawley were looking at how motion captured dancing figures could be used to
engage young female dance students who were also part of the videogame generation,
to attract them as new audiences to the Royal Ballet at Covent Garden. It should be
said that the idea of collaboration between two academics based in different Faculties
in different buildings was unusual within the university, and had come about from
informal contacts looking to see whether the technological resources available could
be explored in an experimental way by artists and designers.
The first experimental results were obtained at the Northern Ballet School, which
happens to be 200 metres from the main site of Manchester Metropolitan University,
where Peter Twigg and his colleagues obtained capture data from a male and a female
dancer engaged in a fairly extreme test of the capabilities of the motion capture suit.
Looking at these results confirmed firstly that the capture suit could indeed cope with
a range of extreme movement, and that this could be applied to other modes of
performance.
Figure 1. Recording motion capture data from one of the dancers, Nana.
Steve Hawley’s own background is as a video artist who has been making moving
image work for exhibition in galleries and film and video festivals since the early
1980s, sometimes involving actors and performers. Seeing the ease with which
performance could be captured but also translated in post-production to a variety of
computer generated figures, suggested possibilities for an art video which used a
performer/actor clad in a capture suit. Whilst artists and choreographers had used
motion capture outside the Hollywood model of animated cinema, no-one else
seemed to be utilising the captured performances of actors to exploit their uncanny
presence in quite this way. At the same time this merging of art and science had
echoes of the pioneers of motion capture in the latter half of the nineteenth century.
2. History
Étienne-Jules Marey was a physiologist and Edweard Muybridge a photographer, but
as the two key figures in the genesis of this field, their work was both an important
precursor and a pointer to the future motion picture camera and projector. Both these
figures combined elements of the scientist/artist, but both had also had a profound
influence on artists in the twentieth century.
Marey’s work on human motion in particular was very influential on the art of the
Cubists and Futurists who came after him. In 1882 he created his chronophotographic
gun which was capable of taking 12 consecutive frames a second, where all the
frames were recorded on the same picture. As well as conducting a famous study
about cats always landing on their feet, he also studied human locomotion. In his
earliest experiments the subjects wore black suits of metal strips or white lines as they
passed in front of the black backdrops, and these have a remarkable similarity to the
present day gyroscopic capture suit.
Figure 2. Runner wearing Marey’s capture apparatus.
More or less at the same time, the British-born photographer, and later scientist
Edweard Muybridge, engaged in a series of photographic experiments, to settle a bet
made by Leland Stanford the Governor of California and also a racehorse owner, as to
whether all four horses’ hooves were off the ground at the same time during a gallop.
Up until that time most paintings of horses galloping showed the front legs extended
forward and the hind legs extended to the rear.
Muybridge used a series of large cameras with glass plates placed in a line with the
horse triggering the thread. The images were viewed in a machine called a
Zoopraxiscope, which was another stage towards the film camera. Stanford won his
bet with Muybridge’s help, and later at the University of Pennsylvania Muybridge
used banks of cameras to photograph people and animals to study their movement.
Muybridge’s sequence of images was highly influential on the most ground-breaking
artist of the last century Marcel Duchamp, who made an early series of paintings that
were heavily indebted to Muybridge and Marey including Nude Descending a
Staircase Number Two of 1912.
The present day technology of motion capture can be realised in several forms, which
vary from commercial systems which cost several hundred thousand dollars, to newer
systems which are much more affordable, and it is these newer systems which have
made the techniques much more accessible to artists. The cheaper systems have
limitations, but Twigg and Hawley’s work with dancers has shown them to be capable
of recording a wider range of movement than was they thought possible.
3. Motion Data Capture
There are primarily two approaches to recording motion capture: electromagnetic
sensors, and optical markers, which each have their benefits and constraints. With
optical marker systems performers only have to wear reflective markers on their
clothes. The optical approach does not provide real-time feedback, however, and the
data from optical systems are error prone and noisy. The locating and extracting of a
figure’s motion directly from video when using optical systems is extremely
problematic for various reasons, particularly due to occlusion of data points. Because
orientation information is not directly generated, more markers are required than with
magnetic trackers. A subject whose motion is to be captured accurately, in real-world
situations and in real time is typically instrumented in some way so that positions of
key feature points can be easily detected and recorded. Electromagnetic tracking uses
sensors placed at the joints that transmit their positions and orientations back to a
central processor. To transmit their information, the sensors have to use either cables
or wireless transmission to communicate with the central processor. The former
requires that the subject be ‘tethered’ with some kind of cabling harness. The latter
requires that the subject also carries a power source such as a battery pack. The
advantage of electromagnetic sensors is that the three-dimensional position and
orientation of each sensor can be recorded and displayed in real time.
Types of motion capture recording and analysis carried out using sensor based
equipment has been generally limited to nondescript everyday movements such as
walking and extracting dominant features of an actor’s movements for application to a
digital character. In this work, motion capture techniques were used to record, classify
and analyse human motion of a more extreme nature: ballet. The motions of a subject
were captured wirelessly by use of the Animazoo IGS-190 equipment which employs
gyroscopic sensors about the body. The motion data captured is used: (i) to drive a
realistic virtual character in real time, (ii) for analysis of motion features, and (iii) for
development of performance design programmes. Conventional configuration of
articulated body system is designed primarily for walking (for example see Coleman,
Bibliowicz, Singh & Gleicher 2008 and Jain, Ye & Liu 2009). Application of the
motion capture data for this project required the development of an extended
character rig configuration and motion representation.
Data was collected using the Animazoo IGS 190 system and a virtual skeletal system
defined from front and side photographs of the subject using Animazoo Autocal
software. This produces a skeletal approximation that helps to eliminate positional
drift in the subject when in motion. The skeletal system is a 26 joint system as defined
within figure 1. The data is collected at a frame rate of 60fps or 60Hz. Translational
data is extracted from 12 of the subjects’ joints for the X, Y and Z axes. The
translational data comprised of the left shoulder, left elbow, left wrist, left hip, left
knee, left ankle, right shoulder, right elbow, right wrist, right hip, right knee and the
right ankle, thus providing a 36 dimensional data structure containing the subject gait
cycles.
Figure 3. Skeletal system for motion capture application.
The motion capture suit is lightweight and flexible, but this was the most extreme
movement that Twigg had attempted to capture with the suit and the initial concern
was whether it would affect the dancer’s movement in some way. After performing
some test exercises though, the dancers reported that the suit was so non-restrictive
that they were not aware of its presence.
Figure 4. Still image from real-time recording of one of the dancers, Nana.
Viewing the results, above, led to Twigg and Hawley discussing the gulf between the
real movement and that of the computer recorded avatar, which for the scientist
Twigg was a problem to be overcome, but for the artist Hawley was an interesting
possibility to be explored. Twigg then introduced Hawley to Mori’s concept of the
uncanny valley.
4. The Uncanny Valley
In 1970 the Japanese roboticist Masahiro Mori stated that as the likeness of a robot
approached a real human being there was a sudden drop-off of the audience’s
acceptance of the humanoid shape and this changed to a powerful negative reaction,
which he termed the Uncanny Valley (Mori 1970). But seven years after Duchamp
had painted Nude Descending a Staircase Number Two, Sigmund Freud had
published in 1919 his famous essay on the Uncanny, Das Unheimliche, which has
become very influential on artists in the latter part of the 20th century, although was
apparently unknown to Mori (Freud 1919).
There is something profoundly uncanny about a humanoid model which appears in
many ways nonhuman, but whose movement is incredibly lifelike, which motion
capture can achieve. The uncanny may be a deep instinctual reaction in the sense that
it steers humans away from other humans who are dead or diseased.
Masahiro Mori’s conjecture was that things that look very nearly human but not quite,
such as a child’s doll or a ventriloquist’s dummy, can produce a highly negative
response and for some reason the response is stronger with a moving object.
However, this notion of the uncanny is not something that artists are seeking to reject,
but in some ways are actively pursuing. For example in Ron Mueck’s sculpture
Spooning Couple 2005, he created two bodies at a half scale, which appear to be
asleep but are actually profoundly eerie to us and produce a strong aesthetic reaction.
Motion capture has been utilised in many commercial motion pictures over the last 15
years, but the uncanny valley effect has been attributed to the failure of some of these,
for example computer animated films like The Polar Express and Beowulf allegedly
evaded success because the viewers found the dead eyed characters unsettling. In
terms of the MMU collaboration, whilst engineers are interested in a perfected reality
that is indistinguishable from real human motion and representational human
appearance, it is precisely the marginal gulf between reality and the represented image
that artists are interested in. It could be argued that up to the late nineteenth century
the dominant viewer response sought by art was one of the sublime, whilst with
contemporary art the desired reaction is the uncanny.
Freud explores the idea of the double – the doppelganger in all its nuances and
manifestations. In Steve Hawley’s video installation Actor, made with Peter Twigg
and Andrew Brownridge, he created a doppelganger by putting an actor in a motion
capture suit and “filming” him whilst he was reading a prepared text. At the same
time an unseen director gives instructions to the actor to correct his performance. The
text is from the final chapter in the UK edition of Manchester born author Anthony
Burgess’s book A Clockwork Orange, subsequently made into a film.
Stanley Kubrick based the film A Clockwork Orange on the American edition of the
novel, which only has 20 chapters as opposed to the English edition which has 21. He
omitted the final chapter, which is the redemptive chapter, where Alex the ultraviolent
droog decides that violence is not worth it and looks to the future and his unborn son.
Actor is the film of the unfilmed final chapter.
Figure 5. Still image from Actor, Steve Hawley video installation.
The effect of clothing the actor in the absurd knight in armour model whilst retaining
lifelike human movement, as well as isolating the actor within an invisible grey
setting, reinforces the sense of the uncanny. Having completed some tests, and one
finished video piece for art exhibition, Twigg and Hawley were able to reflect on the
nature of collaboration itself.
5. Collaboration
Collaborations between artists and scientists and engineers are not new – the
Wellcome trust in England funded from 1996 a 10-year SciArt programme of
collaborations between scientists and artists which involved 118 projects and £3
million funding. A review of the scheme found a number of lessons. Firstly that artists
initiated and drove the majority of the projects, and that largely the artist produced the
work with scientists in some way acting as patrons (Glinkowski & Bamford 2009). It
was true that the scientists and doctors were fascinated by the way that artists worked,
but for the scientists professionally this SciArt scheme was seen as a marginal project
– there were no instances of scientists initiating projects who would then get an artist
on board.
It was difficult to make claims for any ongoing benefits to the scientists involved, or
transformative effect on science, but where the collaborations had worked there
tended to have been a lengthy period of familiarization and exchange over some
years. Similarly the MMU collaboration has deepened and grown through Peter
Twigg and Steve Hawley presenting at the CARPE conference together, as well as
Steve Hawley’s co-supervision of Peter Twigg’s Ph.D. student. Both partners feel
there is scope for future collaboration which explores this technology and which
satisfies the needs of both scientist and artist whilst leaving an interesting space for
exploration which is the liminal field between the two philosophies.
6. Future directions
Possibilities include the classification of a specific vocabulary of ballet movements
for dance choreography. Twigg and Hawley are looking at developing a 3D skeleton
and human mesh with optimized real-time pipeline suitable for more extreme
movement such as ballet. They could also make a comparison of dancers’ motion data
in real time with exemplar performance data for high level dance tuition.
Artists are already using motion capture in a variety of ways and have been doing
since Duchamp painted Nude Descending A Staircase. For example within dance the
technique has been used for the capturing of fleeting and unique dance or bodily
movement performances e.g. Paul Kaiser and Shelley Escher of Riverbed with their
work with the legendary dancer Bill T Jones in their piece Ghostcatching. Motion
capture can translate human movement into music or abstract visuals, giving rise to
synaesthetic performances: this was first done by the artist Oscar Fischinger with his
Lumigraph in the late 1940s. More recently renowned U.S. choreographer Trisha
Brown premiered “how long does the subject linger on the edge of the volume” in
which music and visuals projected on a transparent screen reacted and responded to
dancers in the piece.
In Steve Hawley’s work in collaboration with Peter Twigg he intends to explore the
uncanny by creating humanoid figures which are incredibly lifelike in terms of
movement and yet cause dissonance in the viewer. This will include multiple human
figures, human actors, who will be motion captured but interact in a subtle and
undemonstrative way. But there is also the possibility to explore the idea of moving
from that uncanny valley to a mythical resurrection and bringing to life the real
substantive human body, as in Shakespeare’s The Winter’s tale, where at the end
Hermione is presented as a statue to the court, then “magically” comes to life and
walks down off her pedestal.
7. Conclusions
Collaboration between artist and scientist has been found to favour the former, but
Twigg and Hawley are attempting to find a middle ground in the creative and
technical exploration of motion capture, that is of equal value to both. The key issue
of the uncanny is an interesting paradox, in that it is seen by the scientist as something
to be overcome, but by the artist as a fertile field of aesthetic interest. Artists and
scientists have different languages that often require simultaneous translation, and this
paper is an attempt to meld these languages and still retain coherence.
8. References
Coleman, P., Bibliowicz, J., Singh, K., Gleicher, M., (2008), Staggered Poses:
A Character Motion Representation for Detail-Preserving Editing of Pose and
Coordinated Timing, Computer Animation ACM SIGGRAPH.
Freud, S., (1919), ‘The Uncanny’, pub Imago.
Jain, S., Ye, Y., Liu, C., K., (2009), Optimization-based Interactive Motion
Synthesis, ACM Transactions on Graphics SIGGRAPH.
Mori, M., (1970), ‘The Uncanny valley’, Energy, 7(4).
Glinkowski P, Bamford A. Insight and Exchange: An evaluation of the
Wellcome Trust’s Sciart programme. London: Wellcome Trust; 2009.
www.wellcome.ac.uk/sciartevaluation.