ARtSENSE
ARtSENSE (Augmented RealiTy Supported adaptive and personalized Experience in a museum
based oN processing real-time Sensor Events) was a three-year European co-funded project
implicating seven technological and three museum and cultural heritage partners. The goal of
ARtSENSE was to examine the potential of Augmented Reality for the museum and gallery
experience. The project introduced new wearable technologies for sensing continuously and nonintrusively the user’s context (visual context, eye-tracking, audio tracking, 3D spatialisation,
physiological sensing) in order to determine the user’s current interests. ARtSENSE tackled a very
important problem in the modern usage of ICT in cultural heritage domain: bridging the gap
between the digital world with the physical in a highly flexible way in order to enable a novel,
adaptive cultural experience. The ARtSENSE project aimed to develop an active assistant which
looks over the user’s shoulder (physical world) and react on any change in a visitor’s state of
interests (user’s world) by adapting the guide (digital world) accordingly.
Besides introducing novel wearable technologies for sensing continuously and non-intrusively the
user’s context (visual eye-tracking, audio, physiological biosensing) in order to determine the
user’s current interest (mental engagement), ARtSENSE examined the way on which adaptive
assistance can be realized: using cutting-edge technology (low weight bidirectional see-through
displays) that enables overlaying reality with digital information transparently, including gaze-and
gesture-controlled interaction, so that visitors have the feeling that physical objects are directly
responding to them. In that way artworks become active artefacts that react on users’ attention and
emotions and provide more information about them! This leads to the new generation of mobile
museum guides based on the novel concept we call Adaptive Augmented Reality (A2R).
The visitor can use natural gestures to interact with the multimedia content delivered to the view
through the glasses in the form of virtual overlays (Figure 2). Audio augmentations are also
provided, as well as 3D sound effects, while the acoustic and affective attention of the visitor is
continuously monitored through the use of audio and acoustic sensors. Thus the visitors have the
feeling that physical objects are directly responding to them; the artworks become active artefacts
that react on users’ attention and engagement levels.
Figure 2: Testing ARtSENSE in MNAD, Madrid, Spain
Figure 2: Testing ARtSENSE in MNAD, Madrid, Spain
FACT’s participation in the ARtSENSE project provided an extra opportunity for the investigation of
core questions related to public engagement, gamification and shared experience using
Augmented Reality not only within but also beyond the museum and gallery walls. The project
features a collaboration among a large team of museum professionals from three totally different
yet complementary Cultural Heritage institutions from Spain (Museo Nacional de Artes Decorativas
– MNAD), Paris (Musée des arts et metiers – MAM) and the UK (FACT).
During the first life-circle of the project, the main focus was on encouraging synergy among the
cultural heritage professionals and technological partners so as to identify requirements and needs
that would be then transformed to meaningful augmented visiting experiences. FACT, however,
presented an even more challenging case study since no permanent collection exists, as is the
case with the permanent, well-established and internationally significant collections of MNAD and
MAM. The solution to this challenge was two-fold:
1. Think of the building itself as an artifact – with its own history, context and relationship to its
public;
2. Create a new work of art that interrogates both the problem and the technology.
Both proposals were put forward to the consortium. FACT suggested a number of different
artworks that had been part of the building’s construction as artifacts to “augment.” The starting
point should be an artifact with a deep and rich set of additional materials that would be extensive
enough to give multiple variables for different levels of “personalised” engagement, and be
granular enough to satisfy the expert and the casual visitor alike in an adaptive way.
The VIP signature pillar
The VIP signature Pillar reflects both the context and the history and the relational meaning of the
building to its public (Figure 3 and 4). The Pillar is an ad hoc document of “Very Important Person”
(VIP) visitors to the FACT building, with signatures from film directors, artists, musicians and actors
that have visited and worked with FACT. The pillar provides an indication of the breadth and scope
of practice that goes on in the building. It is a way to document the passage of these guests, and a
fun and “punk” approach for these auteurs to leave their mark on the building. Moreover, it is
generally the case that each of the signatures has been “captured” in some way on camera and in
interview. This enables FACT to underpin each signature with rich media content and make the
user engagement with the proposed ARtSENSE technology a deep and valuable one. The visitor
equipped with the system would be able to get to this rich media content in a novel way and there
was enough content to be able to adapt to the users’ level of engagement and supply them with
appropriate material.
Audio accompaniments to almost all of the signatures on the pillar are available to listen to here.
You can also watch footage of Ken Loach, Apichatpong Weerasethakul and Krzysztof Wodiczko at
FACT.
Figure 3: The FACT VIP Pillar
Commissioning an Artwork
Though FACT is very experienced in commissioning artworks that incorporate technologically
challenging elements and presenting them to the public, ARtSENSE was a little different: here was
a research project where the technology would be constantly under evolution. FACT had to find
artists who were not only proficient with emerging technology in line with the core aims of the
project – developing a biologically integrated adaptive AR system – but who would be willing to be
part of the experiment and the project itself. The vision was that the successful artists would bring
a new approach and by default open up new and exciting avenues that had previously not been
considered.
It was important that the artists understood the concepts behind the work before we undertook to
commission them. To this end, PhD researcher Aneta Krzemien working on her doctoral thesis
“The Role of New Media in Art, Culture and Society at the Turn of the 21st Century”, helped us to
identify a number of artists whose work was a good fit with the curatorial and technical framework
of the research. The consortium decided that the work of artists collective Manifest.AR was the
most appropriate to take forward.
Figure 4: Eye-tracking tests with the public in
FACT
Manifest.AR
Manifest.AR are an artists’ collective based across the world who are working with the medium of
Augmented Reality. Augmented Reality, or AR for short, is a new technology that allows you to use
a smart phone or tablet to alter or enhance your view of the world. When users examine their
environment through their mobile devices, they see information (text, images, 3D objects)
superimposed on the space around them, integrated into physical locations as though they existed
in the real world. Manifest.AR use this technology to create interventions, applications and games
to encourage debate on how we interact with the world and new technology allowing us to become
‘critical citizens’. The group are relatively young as an organization, having endorsed their own AR
Art Manifesto as recently as the 25 January 2011. The group currently comprises eight core
founding members and 20 affiliated members.
In their ‘AR Art Manifesto’ they state that the future ‘is without boundaries’ and that they wish to
‘transform Data’ into something physical and real. Their work in Turning FACT Inside Out could be
used to deliberately “delete” parts of the city, use the sky as a blank canvas for creativity or
populating the entire city with wild growth vegetation. Manifest.AR’s Invisible ARtaffects features
six new games and applications which could be played at FACT or using your own smart
phone/tablet. Watch the short film about Invisible ARtaffects here.
They have shown their work in major international galleries - often uninvited! Manifest.AR have
invaded galleries including MoMA and Tate Modern where they ‘installed’ their AR games and
visualisations completely without the agreement of curators and gallery staff. Their works are
usually created and framed by social contexts, and are always interactive and participatory, often
in a “live” dynamic. In working with FACT they also took on board the FACT-specific interest and
practices of working in and with the community of Liverpool.
“Manifest.AR’s art practice is always relative to site; site as social space; site as architectural sign;
site as political event; site as community memory; site as picturesque landscape; site as global
issue. Augmented reality, a medium which itself suggests inversion and knows no
physical boundaries, renders the institution permeable, spatially and symbolically. The categories
of body-mind, the institution-city, and the virtual-physical are crisscrossed and reconfigured. FACT
is revised as a laboratory medium for experiments in mind, matter and community.” – Press
Release for Turning FACT Inside Out, Will Pappenheimer, 2012
The first iteration of the collaboration between FACT, Manifest.AR and the ARtSENSE consortium
was presented in the “Turning FACT Inside Out” exhibition in June 2013. FACT’s existing and
historic public realm work has demonstrated value in long form projects embedded in a variety of
communities and interest groups and “alternative” spaces, but how can we create a more
adaptable and open public realm, capable of responding to the evolving needs of our public, the
building users and communities? In essence Manifest.AR was asked: “How do we really use the
digital public sphere?”
Biomer Skelters, Manifest.AR (Tamiko Thiel & Will Pappenheimer) (2012)
Click here for a full overview of the work and projects conceived and developed by Manifest.AR as
part of the “Turning Fact Inside Out Exhibition”.