FP7-ICT-2013-C
SEEDIFF
Small or medium-scale focused research project (STREP)
Full proposal
ICT FET Open Call
FP7-ICT-2013-C
Seeing Differently
SEEDIFF
Date of preparation: April 15, 2013
Version number: 31
Name of short proposal this full proposal refers to: Seeing Differently
Type of funding scheme: CP-FP-INFSO
Work programme topics addressed: ICT-2013.9.1
Name of the coordinating person: Risto Ilmoniemi
List of participants:
Participant no.
Participant organisation name
Part. short name
Country
1 (Coordinator)
AALTO-KORKEAKOULUSAATIO
Aalto
Finland
2
AALBORG UNIVERSITET
Aalborg
Denmark
3
Valtion taidemuseo
Ateneum
Finland
4
Senseg Oy
Senseg
Finland
5
SVEUCILISTE U SPLITU (UNIVERSITY OF SPLIT)
UNIST
Croatia
6
FACULTY OF SCIENCE UNIVERSITY OF ZAGREB
UNIZG
Croatia
1
FP7-ICT-2013-C
SEEDIFF
Abstract
SEEDIFF is a ‘neuro-ICT–art’ project that will enhance human sensory experiences through developing and
applying a new generation of mobile “display” solutions to induce alternative channelling of stimuli that expand
or enhance senses. Two products, an affordable haptic display and an innovative 3-D spectral display, target
three distinct outcomes to enable “seeing” by the blind/partially-sighted, improved discernment for the color
blind, and augmented perception for the sighted. Transforming visual content of natural and man-made scenes,
including masterpieces of art, into perceivable tactile-auditory realm will result in innovative experiences and a
foundational step toward augmenting digital inclusion for those with visual dysfunction. The vision of the
project is enhanced experiences of cultural and civilizational achievements, such as visual arts, augmented and
enriched communication, and increased educational opportunities.
Table of Contents
1
SCIENTIFIC AND/OR TECHNICAL QUALITY, RELEVANT TO THE TOPICS
ADDRESSED BY THE CALL
4
1.1
Targeted breakthrough and long-term vision
1.1.1
Concept and vision
1.1.2
Targeted breakthrough
1.1.3
S&T objectives
4
4
6
8
1.2
Novelty and foundational character
1.2.1
State-of-the-art
1.2.2
Technical limitations of existing products, processes and/or services
1.2.3
Main innovations by SEEDIFF participants
1.2.4
Patent search results
9
9
17
18
18
1.3
S/T methodology
1.3.1
Overall strategy of the work plan
1.3.2
Work description
19
22
23
2
IMPLEMENTATION
41
2.1
Management structure and procedures
2.1.1
Management capability of the co-ordinator
2.1.2
Management structure and decision-making structure
Figure 1. SEEDIFF management structure
Coordinator
Steering Committee
Work Package Leaders / Group Leaders
Administrative Contact Persons
2.1.3
Monitoring, reporting progress and documenting results
Project evaluation and other meetings
41
41
41
42
42
43
43
44
44
44
2
FP7-ICT-2013-C
2.2
SEEDIFF
Resolution of conflicts [to be revised]
Risk assessment and management
2.1.4
Meeting agendas
45
46
46
Individual participants
Participant 1: Aalto
Participant 2: Aalborg
http://visualmusic.blogspot.ie/2011/11/tony-brooks-towards-newmultisensory.html?q=Tony+brooks
Participant 4: Senseg
2.2.1
Participant 5: University of Split (UNIST)
2.2.2
Participant 6: University of Zagreb (UNIZG)
47
47
48
50
53
54
55
Consortium as a whole
2.2.3
Consortium overview and role of the participants
2.2.4
Complementarity of participants
2.2.5
Industrial involvement and exploitation of the results
2.2.6
Subcontracting
57
57
57
58
58
2.3
Resources to be committed
2.3.1
Use of the resources
2.3.2
Equipment resources
2.3.3
Other major financial resources
59
59
59
59
3
IMPACT
60
3.1
Expected impacts listed in the work programme
3.1.1
Impact on the competitiveness of the proposers
3.1.2
Strategy for impact achievement
3.1.3
European dimension
3.1.4
Contribution to Community societal objectives
3.1.5
Other relevant European or National funded research
3.1.6
Influence of external factors
60
60
61
61
62
62
63
3.2
Dissemination and/or exploitation of project results, and management of intellectual property
3.2.1
Exploitation and dissemination plan for use of project results
3.2.2
Management of knowledge and intellectual property
63
63
65
4
ETHICAL ISSUES
66
4.1
Benefits of the present study and experiments
66
4.2
Risks of the proposed approach? [are there any?]
66
4.3
Approvals by ethical committees and informed consent
66
4.4
Data protection issues
66
3
FP7-ICT-2013-C
1
SEEDIFF
Scientific and/or technical quality, relevant to the topics
addressed by the call
1.1
Targeted breakthrough and long-term vision
1.1.1
Concept and vision
The SEEDIFF project will develop and apply new technologies and methods to expand human sensory
experiences in the visual domain. Advancements through the work will impact blind, partially sighted,
color-blind, and sighted end-users. Specifically, our long-term vision involves two distinct neuro-ICT–art
convergences and synergies:
I) The first of these goals is that blind and partially sighted end-users will be empowered to interact with
visuo-spatial real and virtual worlds by means of an affordable new generation of haptic1 display
technology that will set the foundation for enhancing their digital inclusion.
II) The second goal is that color-blind end-users will be able to discriminate colors via a process that
enhances color perception from dichromatic to tetrachromatic2. Utilising the same process, our third goal is
that end-users with intact color vision (hereafter termed ‘the sighted’) will be able to access enhanced color
vision through improved color perception augmented from trichromatic to hexachromatic. This will be
achieved via innovative 3D-display technology that doubles the number of distinguishable chromatic bands
of the color spectrum.
Paving the way to our long-term vision will require not only innovative technologies, but also an
understanding of the psychophysical and brain mechanisms of sensory information processing, multisensory integration and convergence as well as brain’s sensory learning, memory, and plasticity.
Fig. 1. Converting and expanding senses: from visual to tactile or haptic. The idea is in representing art by transforming an
original painting to a set a representations that are suitable for tactile and, with SEEDIFF, haptic exploration. Left: Original
painting (Wounded Angel by H. Simberg). Centre: One of the representations of the scene (there are typically set of
representations to code for the entire visual scene, relevant scene details, perspective, light, and color content). Right: The
same scene to be represented on a haptic display during the project using Senseg’s technology, allowing one to feel both
lines and textures of the image as if the screen would have elevations (e.g., the black parts) and different textures.
1
Haptic = relating to the sense of touch; Tactile = perceptible by touch; Di-, tri-, tetra- or hexachromatic = relating to
vision involving 2, 3, 4 or 6 sensory receptor types with different wavelength sensitivities, respectively; phosphene = a
sensation of seeing without retinal illumination.
4
FP7-ICT-2013-C
SEEDIFF
A
B
C
Fig. 2. Expanding senses: from trichromacy to hexachromacy (or dichromacy to tetrachromacy). The basic idea is
that images are viewed stereoscopically, but instead of (or in addition to) presenting depth information, the two
eyes are used independently to enrich color perception. The left eye receives one coloring and the right eye another
coloring of the same scene. Thus, the left eye is presented with three wavelength bands from the original scene
while the right eye is presented with three other bands. This is illustrated via images in panels A, B, and C: A) A
possible division of the visual spectrum into 6 wavelength bands (horizontal scale: wavelength in nanometres).
Hexachromatic vision is obtained if the shaded bands are presented to the left eye and the non-shaded bands to the
right eye. Superimposed, the sensitivity functions of our three cone types are shown in the same image
(normalized). B) Abstract art (Johannes Itten) can also be expanded to the new color format; normally, a 3D
viewing system, such as one in modern TVs, is needed for looking at the hexachromatic image. C) From
dichromacy (color blindness) to tetrachromacy. The left image shows how an idealized red–green color blind sees
the red berries in essentially the same way as the green leaves. The two right images show how the original
trichromatic image can be seen by the color blind when green is emphasized on the left figure and red is
emphasized in the right image. When viewed stereoscopically, the berries can be distinguished from the leaves not
only by their shapes but also by color, as a combination of dark blue (centre image) and light green (right image).
(Frames A and C are from the special assignment of Lari Koponen, Aalto Univ., unpublished; original images at
seediff.tumblr.com).
5
FP7-ICT-2013-C
1.1.2
SEEDIFF
Targeted breakthrough
Our intended breakthrough is threefold.
First, we plan to develop a novel, advanced, haptic-display platform and tools for mapping visual/spatial
information into tactile–auditory modality that will allow interactions, including haptic navigation in
visually based scenes and maps. This allows turning the wealth of visual content of natural scenes, images
or graphical objects, including masterpieces of art, into the tactile–auditory realm to be experienced by
partially sighted and blind people.
Second, we plan to develop methodology for producing and displaying hexachromatic images and video.
Dividing the light spectrum into 6 bands (Fig. 2A) we will present alternating bands to the left eye whilst
synchronous corresponding alternating bands are presented to the right eye via standard 3D display
technology. This will enable the color blind to experience all standard colors (albeit differently from how
they are traditionally sensed), and allow the sighted to expand color vision from trichromacy to
hexachromacy. Expected outcomes include augmenting life quality and day-to-day living for color blind as
well as offering new perceptual experiences for sighted and new opportunities and challenges to artists,
researchers, and media professionals.
Third, through behavioral and neurophysiological studies of the multi-modal and multi-spectral sensory
expansion provided by our new technological platforms, we plan to obtain fresh insights into the ability of
the brain to adapt to novel and enriched sensory inputs, thus enabling its ability to form previously
inexperienced representations of the external world and stimulating its plasticity. These insights will
contribute to the foundations of sensory expansion technology and will help us in integrating and optimizing
tactile and auditory coding of visual information and guide us in developing a breakthrough in
hexachromatic display technology.
The relevance of our project stems from the fact that vision is the sense that most directly defines our
relationship with our environmental and social surroundings. When awake with eyes open, we constantly
maintain a representation of space around us based on what we see. Approximately half of our cortex, with
tens of distinct visual areas, is devoted to vision. Unfortunately, our visual machinery may be compromised
by genetic defects, disease, aging or accidents. There are close to 300 million blind or partially sighted
people in the world; they lack direct access not only to everyday visual scenes and objects that aid and guide
our interactions with the environment, but also to abstract cultural and civilizational achievements such as
visual arts or even educational opportunities. In particular, digital exclusion of blind and partially sighted
people is still growing, thus our venture is timely and fitting considering recent EU actions and priorities
(Post-i2010: priorities for new strategy for European information society (2010-2015), European Blind
Union [EBU] Position Paper2). Included in such prioritized action is the EBU Strategic Plan 2011-2015: “To
map current levels of good practice in giving blind and partially sighted people access to arts and to
cultural events and facilities to provide the basis for recommendations and guidelines that will serve as a
tool for advocacy and lobbying.” In stating the case of low levels of accessibility in museums and places of
cultural interest to blind and partially sighted people, Carol Borowski, previous chair of the EBU’s Culture
and Education Commission posited that “We invite the European Commission to energetically promote the
cultural rights of disabled people in its new Disability Strategy 2020. Equal access to museums and places
of cultural interest is a right of visually impaired people, not a luxury.”
2
http://ec.europa.eu/information_society/eeurope/i2010/docs/post_i2010/additional_contributions/ebu.pdf
6
FP7-ICT-2013-C
SEEDIFF
Ms. Borowski continued, “Of course, there is good practice, but it is exceedingly rare. Across Europe, we
need significantly more opportunities for visually impaired people to experience art and culture through
audio description, authentic objects for handling, tactile models and tactile images. We need to exploit the
empowering potential of new technology. Exhibition design which is inclusive of the needs of disabled
people has to become the norm, not the exception, as it is today.”3
Such statements indicate the hope and anticipation of communities of blind and partially sighted individuals
of how collateral consequences may be alleviated with tailored ICT and assistance. Such expectations are
supported by structural and functional neuroimaging studies that have demonstrated multi-sensory
perceptual mechanisms underlying cortical plasticity and reorganization. These studies confirm the
involvement of most visual areas, including the primary visual cortex, in processing information from other
sensory modalities, thus supporting our approach of a multi-sensory input based restoration and/or
expansion of human visual perception. Consequently, our aim is to use our new technologies and ideas in
combination with advanced neurodynamic brain imaging methods to design and tune tactile/haptic–auditory
stimuli to demonstrate that the blind are capable of an enriched perceptual capacity and to provide them an
opportunity to get a glimpse of what sighted experience when they observe visual scenes or pieces of art.
The new technology will offer new creative possibilities for the blind such as artistic expressions and
interactive haptic-auditory communications, including games.
SEEDIFF also aims at expanding the awareness of the majority of society to the condition of our fellow
citizens such as the blind, the deaf, and other people with sensory limitations. Therefore, we will organize
exhibitions in three major art museums (Finnish National Gallery Ateneum, Helsinki; Modern Gallery,
Zagreb; Sydvestjyske Museer, Ribe, Denmark), in which tactile–auditory stands and prototype haptic–
auditory displays will be placed next to the original paintings and made available to all visitors.4
Although we will focus on making new mobile technologies and visual art available and affordable to the
blind, the concept is applicable more generally, suggesting wide applications also for sighted people
targeting augmenting sensitization. The new technologies may be used in daily life, under extreme
environmental conditions in which visual inputs are minimal or not available, in training and learning events
to expand sensory perception, stimulating creativity and imagination by enriched perceptual processing and
programmable art, or during rehabilitation to restore or compensate for a temporal loss or a profound
sensory deprivation, respectively by enhancing multisensory cortical interactions.
To summarize, the intended breakthrough is to establish methodology that enables the blind to experience
visuo-spatial scenes using novel haptic touch screen technology, the color blind to discriminate colors they
could not discriminate before, and the sighted to expand their perception of objects with new color and
tactile form. We expect to open also a new line of behavioral and neuroscientific research, which, in turn,
will support the realization of our long-term vision where the blind (especially if they learn to use the
technology already in childhood) can “see” by means of affordable “feelable” haptic display technology and
could thereby be included in the digital world. Furthermore, possibilities for using color would be expanded,
giving new opportunities for artists and for the media as well as consumers.
3
http://www.euroblind.org/media/projects/EBU-Culture-Education_plan-of-action-2011-2015.doc
4
We plan to include blind guides in a similar way as in the exhibition Dialogue in the Dark (www.dialogue-in-thedark.com), which had psychological, social, and practical/economic implications/outcomes. An astonishing 100% of sighted
visitors who were questioned five years later remembered the experience, 90% reported feeling sensitized to the world of
the blind, while 52% recommended Dialogue in the Dark to their friends and family.
7
FP7-ICT-2013-C
1.1.3
SEEDIFF
S&T objectives
Our objective is to expand human sensory experience by introducing and developing new haptic and color
display technologies for blind, partially sighted, color-blind, and sighted people.
Our vision and aims can be described as follows:
New Display Technologies
1) SEEDIFF will develop affordable haptic-display platforms specifically aimed at allowing the blind to
better cope in a world built for those with intact vision. The emergence of such a new haptic display
technology will present new opportunities beyond our goals that focus on innovative representations of
art and a mobile-device user interface for the blind.
2) Methods to interpret and map visual scenes, including visual art, into tactile form will be developed.
This complex mapping will be achieved as a stepped approach: initially a designer is guided to convert
visual images into tactile representation using manual processing; in our future vision, the process will
be automated.
3) The expansion of visual experience in color blind as well as in people with intact color vision will be
achieved by widening the brain’s sense of color with 3D display technology that will present
differently colored images to each eye.
Seeing Differently
4) By developing the presentation of visual art to the blind, SEEDIFF will, while expanding the sensory–
artistic experience, also evolve a new art form, both for the blind and for the sighted population.
5) Behavioral and dynamic neuroimaging studies will explore the aspect of visual experience enabled by
tactile/haptic display and/or auditory guidance.
6) Human sensory perception and plasticity will be investigated via neurodynamic functional brain
imaging methods aimed at exploring uni- and multi-sensory processing of the tactile/haptic and
auditory representations of visual objects and scenes including visual art using spatio-temporal source
localization methods and novel approaches to study the dynamics of cross-modal integration.
Science of Art and Color
7) The science of color and art will be enhanced via SEEDIFF, as will our understanding and appreciation
of what color really is and how our color vision depends on the fact that we have trichromacy, i.e., that
each color is formed as a mixture of the activation of three types of light-sensing cells in the retina of
the eye. Our experimentation with hexachromacy is expected to introduce new aspects to color science
and strengthen non-linear relationship between stimulus intensity and brightness, called the Helmholtz–
Kohlrausch (HK) effect, in respect of the color blind.
Education and Public Awareness
8) The teaching of art, art history, and haptic–visual communication for the blind will be advanced via
SEEDIFF. The blind will be able to draw images with the haptic displays and communicate with other
blind people using haptic displays; we will also explore haptic–auditory games for the blind, which is a
largely unexplored territory and would open huge possibilities regarding the inclusion of the blind in
the culture of games. In addition, the technology will also be used to enhance the human–machine
communication interface for the blind and allow novel rehabilitation options for blind and sighted.
Thus, the societal implications are wide, which will result in high attention from the media.
The objectives are described in more detail in Section 1.3.3.3 (Work package descriptions). The measurable
outcomes are the deliverables.
8
FP7-ICT-2013-C
1.2
SEEDIFF
Novelty and foundational character
SEEDIFF will develop novel tools for tactile/haptic visual-scene presentation to the blind. Our new haptic
display technology will revolutionize the user interface of mobile phones or tablets for the blind, allowing
them to browse the web in haptic form. Also, with this technology, it will become possible to present art and
other visual scenes to the blind via the Internet, so as to bring these experiences to millions of people rather
than to only a handful. Furthermore, we will use standard 3D display technology to expand the sensation of
color, which is anticipated to lead to novel types of color content in movies and in photography, helping the
color blind as well people with full color vision to enrich their visual experience when watching TV or other
media. SEEDIFF will use dynamic neuroimaging and advanced localization, time series correlation, and
pattern recognition analysis techniques to demonstrate effectiveness of its approach using multi-sensory
stimulation to expand sensory experience taking advantage of innate multi-sensory capabilities of the
human brain. SEEDIFF’s behavioral and EEG/MEG studies will advance current understanding of
tactile/haptic and auditory stimulation enabled or enriched visual experience and provide novel insights into
evidences and mechanisms of multi-modal integration and cross-sensory plasticity in particular related to
the 2-D representations of 3-D visual world, including visual art.
1.2.1
State-of-the-art
In presenting the state of the art relevant for this project, it is important to review several important themes:
the solutions that aid the blind in their day-to-day encounters with objects that surround them, the more
technologically advanced tactile displays that have recently been developed for tablets and mobile phones,
equally novel solutions for binocular displays that will serve as the basis for the planned color explorations,
cross-modality mappings and art experience/perception, current access that the blind have in relation to
complex and “non-utilitarian” products such as art as well as the latest research results regarding
neuroimaging evidences on multi-sensory processing and cross-modality plasticity of the human brain.
State-of-the-art of visual-restoration or visual-substitution aids
Numerous approaches and potential solutions aimed at overcoming the difficulties that the blind constantly
face have been proposed, such as reading and mobility aids, more advanced sensory substitution devices
(SSDs) and invasive sensory restoration and neuroprosthetic approaches.
Sensory substitution refers to the transformation of the characteristics of one sensory modality into the
stimuli of another modality and in the case of blindness, SSDs represent a non-invasive rehabilitation
approach within which visual information is captured by an external device such as a video camera and
communicated to the blind via a machine–human interface in the form of typically auditory or tactile input.
Specifically, the first targeted modality for substituting vision was touch, due to the ease of transforming
visual into tactile signals as these are both characterized by 2-D spatial representations. Paul Bach-y-Rita
who devised a tactile display, which mapped images from a video camera to a vibro-tactile device worn on
the subject’s back, did pioneering work in this field in the 1970s. Later, smaller tactile devices placed on the
tongue and foreheads of blind individuals characterized by better spatial somatosensory resolution were
developed. The development of both tactile and auditory-based devices was triggered by limitations of
tactile only SSDs, They can be useful in allowing the blind to learn how to interpret and use the transmitted
information in simple visual discrimination and recognition tasks as well as in more demanding contexts
that require constructing mental images of more complex environments.
9
FP7-ICT-2013-C
SEEDIFF
Nevertheless, such devices have shown limited success among blind individuals in allowing them to “see with
the skin”5 or to “see with the brain”6 or to “see with the ears”7. The level of performance achieved through any
of these SSDs remains, however, inferior to any of the perceptual standards as sensory substitution is slower, less
accurate, and inferior in terms of automaticity and effort. Noticeably, compared to visual perception, the number
of objects that can be jointly accessed or available through sensory substitution devices seems to be much lower
as users of SSDs have not been shown to be able to track or identify multiple objects at once. SSD researchers
justify such observations by either a suboptimal resolution of their devices, or suboptimal training of participants.
According to Loomis et al 2010, however, spatial resolution of the visual is much higher than for the tactile
perception so it might be impossible for touch to ever substitute for vision in complex real-world situations.
Therefore, while sensory substitution allows the blind to perform many functions, this functionality is generally
limited and it is not clear whether it will ever be accomplished, as later described in Section 1.2.2.
Given their limitations, some argue for the approach of restoring capability of a visual input to the blind as it
represents an alternative to sensory substitution devices. There are four main types of approaches in the visual
input restoration, targeting the retina, optic nerve, lateral geniculate nucleus (LGN), and the visual cortex.
Devices based on these approaches have so far shown some promising results, as experienced blind users can
utilize visual phosphenes generated by some of these devices to create meaningful visual percepts. Corneal
transplant, for example, restored the brain’s ability to absorb normal visual input8 but it allowed only perception
of simple movements, colors and shapes.
In this project we propose an alternative approach to building a device for the blind. The approach is based on
multisensory and cross-modal stimulations which utilize brain’s innate capability and plasticity related to crossmodal transfers and integration of sensory inputs - such as the one that exists between the visual and tactile
identification of shapes, or between visual and tactile motion.
State-of-the-art of tactile displays for tablets and mobile phones
Today the concept of tactile display usually refers to a Braille display, but it may also refer to a touch screen
with tactile feedback functionality. The name of Senseg is often referred to in the latter context by being the
most promising enabling technology. Braille devices are naturally only equipped to provide text content in
refreshable Braille format and do not lend themselves for art presentation; therefore, the focus here is in the
tactile touch-screen category.
Tactile displays in the context of touch screens take advantage of the input functionality of the screen. The
finger location on the screen is identified; when a finger is over an element of interest, the device provides a
tactile feedback controlled by the host device. In theory, accurately timed tactile events based on finger
location could allow the user to feel the contours of a graphics object on the screen. Unfortunately, currently
available tactile technologies have severe limitations for creating this illusion.
The main technology available for providing tactile feedback is the vibrating motor. This is available in
almost every smartphone and tablet device, and while it is suitable for providing alarm vibration or even
vibrating the device for confirming, e.g., a key press, it cannot really provide tactile dislay illusion. The
effect is non-localized and also the frequency response of the motor is unsuitable. Some more advanced
5
White, B.W., Saunders, F.A., Scadden, L. et al: Seeing with the skin, Percept. Psychophys.7, 23–27. (1970)
Bach-y-Rita,P.,Tyler,M.E.,and Kacz-marek,K.A: Seeing with the brain. Int.J.Hum.Comput.Interact. 2, 285–295. (2003)
7 Levy-Tzedek, S., Hanassy, S., Abboud S., et. al: Fast, accurate reaching movements with a visual-to-auditory sensory
substitution device. Restor Neurol Neurosci.30, 313-323. (2012)
8
Fine et al., Long-term deprivation affects visual perception and cortex, Nat. Naurosci. 6, 915-916, 2003
6
10
FP7-ICT-2013-C
SEEDIFF
actuator solutions have also been produced such as piezo or electroactive polymers, which can have more
favorable frequency responsies, but they produce mechanical movement of the whole device, which does
not yield the localized sensation to the finger, and are therefore more suited to gaming-like events and
cannot create texture-type sensations, which would be essential for describing visual images.
Senseg technology, on the other hand, is specifically suited for providing texture and surface information
rather than shaking the whole device. The basic principle of the technology is modulation of the electric
force between a charged conductive layer within the screen and the adjacent body part. While it is possible
to generate perceivable vibrations to a stationary finger, the much more pronounced effect for the Senseg
technology is the modulation of the surface friction force when sliding the finger along the surface. With
accurate control and timing of these modulations, a great number of different touch sensations and textures
can be generated, and when combined with matching graphics, this provides the basis for a strong illusion
of a ‘’feelable’’ screen.
Our plan is to take the tactile display technology to the next level. Humans have limited capabilities in
perceiving the world via tactile information, especially with surface information alone (i.e., compared to
grasping an object). When tactile information is static, it is limited in scope and weak in experienced
immersion. For example, it is difficult to determine the shape of an object but it will be even much more
difficult to determine information from complex and detailed pieces of art.
Hence, SEEDIFF plans to make groundbreaking work that uses intercative concepts to allow users to have
more immersive experience. Auditory stimulation and narratives will be included to complement the haptic
information to guide the blind user during the use of the device and augment sensory perception.
The interactive concepts may be expanded to vision as well.....??????
State-of-the-art of hexachromatic displays
The idea of hexachromatic vision was conceived independently by at least two people in the early 1990's, by
the project coordinator (who proposed the idea to the Foundation of Finnish Inventions) and earlier by
Raphael Levien (US patent No. 5,218,386, June 8,1993). The idea at that time was to build spectacles with
different comb filters for the left and right eyes so that alternate wavelength bands would be seen in the
different eyes. The SEEDIFF coordinator experimented with such glasses and observed new color
phenomena, but such eye glasses did not seem practical at the time. It seems (based on Google searches)
that Levien has not either continued on this line.
Our renewed interest in hexachromacy stems from recent developments in 3D displays (TV displays,
Cinema, game consoles, mobile phones), which can be used to display different images to the different eyes.
Thus, hexachromatic content can be presented without any extra cost or conscious effort by the user of 3D
display devices. We only need to develop ways to provide the content.
Multi-spectral imaging was originally developed for imaging from satellites. It is being used in mapping
properties of cultivated land, forests or other target areas by taking up to several tens of differently filtered
narrow-band photographs from an airplane, balloon, or satellite. Multi-spectral images are usually analyzed
by computer; their analysis visually would be very difficult if not impossible. But the principle of multispectral imaging is the same as in hexachromatic photography: getting more spectral information from the
scene than is possible with standard trichromatic photography.
To our knowledge, the idea of using 3D display technology in order to achieve hexachromatic vision in
normally sighted people and tetrachromatic vision in dichromatic color-blind people is new. However,
scientific experiments have been performed to study binocular rivalry, which is a phenomenon where
conscious perception alternates between different images presented to the different eyes. Some of the
11
FP7-ICT-2013-C
SEEDIFF
experimentation has been performed with color, but to our knowledge, the idea of expanding vision to
hexachromacy has not been an issue in these considerations.
State-of-the-art cross-modality mappings and art experience/perception
The state of the art of cross-modality mappings9 within contemporary art to permit a common ‘grammar’ as
well as accessible, verifiable, and robust is limited and inconclusive. Auditory–color bidirectional mappings
can be frustrating for artists and scientists who wish to successfully express themselves beyond a single
sensory modality. A similar “mapping” challenge is apparent within SEEDIFF. Thus, by approaching this
complex challenge in collaboration with brain imaging experts, we will determine the affect of multisensory stimulation to augment perception, and maximise potential of creating a new art form by defining a
uniform grammar to communicate the art message and feeling of beauty. To illustrate the mapping
challenge, a brief history of auditory–color mapping is presented.
Sir Isaac Newton in 1704 proposed correspondences between the proportionate widths of the seven
prismatic rays of the color spectrum matched to string lengths required for producing a musical scale. This
has been criticized as unscientific with ungrounded mappings10. Accordingly, since Sir Isaac’s era,
numerous color-to-musical scale mappings have been posited illustrating complexities involved in
establishing a meaningful mapping language between musical tones and the color spectrum (Fig.
3).
Fig. 3. Three Centuries of Color Scales © Fred Callopy RhythmicLight.com
Analogizing beyond auditory–color mappings, e.g., sound frequency to light wavelength, additional sensory
modalities integrated in a meaningful way toward a total art form was the core of German Romanticism and
Richard Wagner’s Gesamtkunstwerk. This is where synergy, integration, and synthesis of different stimuli
target an ultimate, all-encompassing experience. Such experience has been an on-going challenge for artists
to achieve. However, whether this has been reached is questionable as artists continue to explore
technological advances to further their art. In line with this desire for an integrated form, artists and
9
Here, a mapping means the manner in which a feature in one sensory modality corresponds to a feature in another
modality. Examples of the mapping of color to musical scale are presented in Fig. 3.
10
Gerstner, K.: The Forms of Color: The Interaction of Visual Elements , Cambridge, MA: MIT press (1986)
12
FP7-ICT-2013-C
SEEDIFF
scientists are increasingly enquiring into the phenomenon synaesthesia11. This is a condition that occurs
when an individual who receives a stimulus in one sense modality simultaneously experiences an internally
generated sensation in another . This condition is a phenomenon seemingly at odds with the common-sense
worldview of five separate senses that channel impressions to our minds. Intrigued by this apparent anomaly
of the senses, artists, psychologists and other scientists have explored and experimented with synaesthesia in
order to reveal how the senses are interrelated to realise new art. The unusual approach taken in SEEDIFF is
to attempt to understand how this interrelationship can realise new art forms by learning from people with
sensibilities that are heightened through having an impairment that necessitates training of alternative
sensing means to a level for increased survival and improved life quality. We will focus on heightened
nuances of visual and tactile senses to achieve our objectives of offering new artistic experiences and
opportunities for blind and visually impaired. [This section can be streamlined]
State-of-the-art of the visual art creation and access to the visual art of the blinds
One may wonder what a blind person wants from art. Dr. Betsy Zaborowski, former Director of the
National Federation of the Blind's Jernigan Institute explained how blind people are as different from each
other as sighted people–some are even art experts who have studied specific kinds arts that they go to
experience, some are history buffs, some may just be tourists….12 Questioning how blind artists paint it is
stated that only around 10% of all people with blindness can see absolutely nothing at all. As such most
blind people can in fact perceive some level of light and form, and it is by applying this limited vision that
many blind artists create intelligible art. Also, a blind person may once have been fully sighted and yet
simply lost part of their vision through injury or illness. Blind artists are able to offer insight into the study
of blindness and the ways in which the blind can perceive art, in order to better improve art education for
the visually impaired.
However, in discussing the connection between sensory impairments and art, the phenomenon of “blind
artists” is just a tip of the iceberg. Artistically gifted or not, the blind are just like the sighted: they have a
certain amount of interest in arts, visual included, and should have access to such products throughout their
lives. Unfortunately, at this point in time, this access is very limited. There are some books that offer
insights into this field, such as Elisabeth Salzhauer Axel’s book Art beyond sight: a resource guide to art,
creativity, and visual impairment13 and the companion video titled Art Beyond Sight. A Demonstration of
Practical Techniques for Teaching Art to People with Visual Impairments. Also, there are activities such as
the Art Beyond Sight Collaborative and its annual Awareness Month, which is a series of exhibits,
demonstrations, conferences, telephone and online discussion groups aiming at raising awareness and
bringing together professionals, educators, researchers, the media, sighted and blind artists, and art lovers
from around the world. Thus, Awareness Month offers further opportunities for SEEDIFF dissemination
and impact.
In addition, Art History Through Touch and Sound: A Multisensory Guide for the Blind and Visually
Impaired is a series of six multimedia volumes co-published by the American Printing House for the Blind
and Art Education for the Blind. It is an art history series for people who are blind or visually impaired that
is the result of nine years of research, development, and testing. The series spans the history of art, from
prehistoric through contemporary, guiding the reader through a journey that has long been denied to blind
11
http://leonardo.info/isast/spec.projects/synesthesiabib.html
12
http://www.youtube.com/watch?v=yrLj2IGFWw0
13
http://www.afb.org/store/Pages/ShoppingCart/ProductDetails.aspx?ProductId=978-0-89128-850-3
13
FP7-ICT-2013-C
SEEDIFF
and visually impaired audiences. Each volume contains a book of tactile diagrams and an audio narrative.
The diagrams use a lexicon of seven standardized patterns, enabling the reader to acquire a tactile
vocabulary. The narrative guides the reader through the diagrams, providing art historical information and
richly detailed descriptions of major monuments in the history of art. The success of this two-part system
depends on these complimentary components. Professional art historians collaborated with Art Education
for the Blind’s development team to create audio narratives that convey the historical richness and formal
range of some 30,000 years of visual art.
The book Drawing and the Blind: Perceptions to Touch, by Jonathan Harchick, focuses on the ways in
which the blind, both young and old, can perceive pictures and 3D objects. According to Harchick, visually
impaired people are able to feel a 3D object and then create a drawing of it that can be easily recognized by
a sighted individual. Harchick likens the drawings of the average blind-since-birth person to those of a
sighted child. He notices that blind children are much more willing to attempt to draw than blind adults who
have no prior experience. Kennedy [reference?] discusses the fact that the blind can perceive a drawing
made of raised lines, as well as 3D objects that have shape and form.
The ways in which the visually impaired are able to create art are giving new insights into the study of sight
loss. For example, Ann Roughton is a landscape artist suffering from Macular Degeneration. Her paintings
not only include what she can see with her partial vision, but her work also includes the grey haze that she
sees in the centre in her vision as a result of her condition. In doing so, she is literally painting sight, giving
a new perspective on sight loss.
Organizations and charities exist to support blind artists such as BlindArt 14, a British charity established in
2004 to educate the public about the needs of people who are visually impaired and to promote the idea that
lack of sight need not be a barrier to the creation and enjoyment of works of art. BlindArt exhibitions
typically contain paintings, sculptures, installations and other works of art, created by artists who are blind
or partially sighted, which have been designed to engage all the senses. Unlike conventional art exhibitions,
visitors to BlindArt exhibitions are encouraged to touch and interact with the exhibits, with latex and cotton
gloves provided for this purpose.
However, when presenting more traditionally developed art forms for the blind, this is done in a much more
conservative form. Specifically, the blind is currently limited to only a handful of small exhibitions always
placed in museum corners (as in Centre Pompidou, Paris). In most cases, curators of the museum guide blind
visitors; the blind person is a passive participant. Occasionally, visitors are given a small tactile printed diagram,
which is not very useful; the blind do not like to use it (Museum of Modern Art, New York City; National
Gallery, London). In Tate Modern permanent exhibitions in London there is a possibility to explore sculptures
but not pictures. Other galleries have 3D printouts of artefact replicas so that all can enjoy multisensory
exploration via hands-on experiences; however, this is impossible for a painting. Equally accessible novel
method, created as from an art historian perspective, has been developed by UNIZG (The Jovicic DiTacta):
tactile diagrams are placed on specially designed stands where the blind can take in “their hands” a set of
diagrams representing a piece of art and explore them while listening to an audio narrative (Box 1). DiTacta is
the first educational didactical tool created for exploring art and art history in exhibition space such as museums
and galleries, where blind pople usually do not come. This tool is opening new doors to those who are blind and
have no access into all spheres or life , tool that promote integration of blind and sighted people.
14
http://www.blindartistssociety.com
14
FP7-ICT-2013-C
SEEDIFF
We understand that a "picture of a picture" would need to take the form of a tactile diagram.
The process should be monitored by a blind person, who can judge whether the resulting
tactile diagram is sufficiently "legible". We analysed the painting in detail, described the
composition, the elements of form, line and surface, and the details in the picture. Details are
part of the whole, and in describing them, we arrive at a full understanding of the whole
picture. A single tactile diagram would not suffice for a full understanding of the painting.
Instead, we needed to zoom in on different details, like in a film, and depict them in separate
tactile diagrams.The greatest challenge is how to present colour within the monochrome
tactile diagrams. We found solution.
Even precisely drawn tactile diagrams are of little use to a blind person, unless he or she is
shown how to use them. We use an audio guide, similar to GPS technology, which leads a
blind person from one point to the next, from the top of the painting to the base, from left to
right, and from the base to the top. The audio guide gives precise instructions on how to place
one's fingers on the tactile diagram, how to move across the painting, how far to move left or
right, up or down, etc. So a blind person does two things at the same time: he or she listens to
the audio guide carefully, and touches the tactile diagram while following the instructions. This
is how he or she can gain an impression of the painting being studied. All sad is aplicable to
use in museum.
By developing new haptic mobile displays SEEDIFF will make visual art as well as environmental scenes
equally accessible in an interactive fashion to sighted and blind people.
State-of-the-art neuroimaging studies on multi-sensory integration and cross-modal plasticity
Multi-sensory nature and capabilities of the human and non-human brains are well documented by physiological
and neurimaging studies providing evidence of similar cross-modal binding mechanisms that appear to be
distributed across distinct neuronal networks that vary depending on the nature of the shared information
between different sensory cues15.Brain’s remarkable feature of providing the substrates for very early multisensory integration by multiple possible feedback and feedforward circuits for multi-sensory convergence in all
sensory areas and yet sufficient sensory segregation to pomote uni-modal sensory representation and uni-modal
behavioural experience can be taken as a selective advantage in evolutionary terms. Based on decades of of study
of multi-sensory convergence and integration the major functions of multi-sensory convergence and integration
seem aimed at enhancing the detection of behaviourally-relevant stimuli, and of promoting rapidity and accuracy
of behavioural responding and well as cognitive processing16.Understanding the conditions under which the brain
integrates the different sensory streams and the mechanisms supporting multi-sensory convergence at neuronal
and system level and cross-modality plasticity represent one of the most challenging problems at the forefront of
neuroscience.
Multi-sensory interactions can occur shortly after response onset at the lowest cortical processing stages.
Moreover, recent findings in many laboratories reinforce the notion that cortical processing per se represents a
collaboration between new sensory input and ongoing cortical process16,17. The affective prediction hypothesis
15
FP7-ICT-2013-C
SEEDIFF
was proposed18 which implies that responses signalling an object’s salience, relevance or value do not occur as a
separate step after the object is identified. Instead, affective responses support vision from the very moment that
visual stimulation begins.
Lesion and neuroimaging studies provide evidence suggesting that regions of the brain normally associated with
the processing of visual information undergo remarkable dynamic change in response to blindness. These
neuroplastic changes implicate not only processing carried out by the remaining senses but also higher cognitive
functions such as language and memory19.
However, there is a growing evidence that sensory deprivation is associated with cross-modal neuroplastic
changes int eh brain. After visual or auditory deprivation, brain areas that are normally associated wtih te lost
sense are recruited by spared sensory modalities (Marabet 2010). While the DTI study on how the blindness
onset age affects development of brain anatomical networks20 showed large group differences between
congenitally blind and early blind subgroups compared to adolescent-blind and late-blind subgroups specifically,
a reduced connectivity density and a decreased global efficiency compared to the sighted controls, in particular
in the frontal and occipital cortices they also found that some of the network properties were independent of
visual experienc, which may account for the preserved functions of the brain in blind subjects. Recent studies
provided evidences on much greater levels of plasticity in the adult visual cortex than previously suspected
(Spolidoro 2009).
Among the various properties that can be extracted from an object, including size, colour, texture, material,
hardness, etc., shape is the most prominent for human visual object recognition21. Recent fMRI study Kim 2010
demonstrated that visual and haptic sensory inputs converge on common object-selective brain sites in the
occipital, temporal, and parietal cortices to process object shape and found evidence of multisensory integration
in these brain regions. The study demonstrated the effect of “enhanced effectiveness”, i.e., as the effectiveness
of the unisensory component stimuli increased, so did the multisensory gain with the combination stimulus.
Selective visuo-haptic processing of shape and texture were demonstrated on sighted subjects22.
Sighted individuals can recognize basic facial expressions by haptic surprisingly well. However, this year study
23
conducted both psychophysical and fMRI experiments to determine the nature of the neural representation that
subserves the recognition of basic facial expressions in early blind individuals. Their results suggest that the
neural system that underlies the recognition of basic facial expressions develops supramodally even in the
absence of early visual experience.
15
Calvert G A and Thesene T: Multisensory integration: methodological approaches and emerging principles in the human
brain. Jl Physiology, 98, 191-205, 2004
16
Schroeder C E et al: Anatomical mechanisms and functional implications of multisensory convergence in early cortical
processing. Int. J. Psychophysiology, 50, 5-17, 2003
17
Schroeder C E and Foxe J: Multisensory contributions to low-level, ‘’unisensory’’ processing, Current Opinion in
Neurobiology, 15, 454-458, 2005
18
Barrett L. F. and Bar M.: See it with feeling: affective predictions during object perception. Phil. Trans. R. Scoe. B, 364,
1325-1334, 2009
19
Amedi et al. The Occipital cortex in the blind. Current directions in psychological science, 14, 306-422, 2005
20
Li et al. Age of onset of blindness affects brain anatomical networks constructed using diffusion tensor tractography.
Cerebral Cortex, 23, 542-551, 2013
16
FP7-ICT-2013-C
SEEDIFF
21
Marr D (1982): Vision: A Computational Investigation into the Human Representation and Processing of Visual
Information. San Francisco: W.H. Freeman. xvii, 397 pp.
22
Stilla R and Sathian K: Selective visuao-haptic processing of shapte and texture. Human Brain Mapping, 29, 1123-1138,
2009
23
Kitada et al. Early visual experience and the recognition of basic facial expressions: involvement of the middle temporal
and inferior frontal gyri during haptic identification oby the early blind. Frontiers in Human Neuroscience, 7, 1-15,
2013
1.2.2
Technical limitations of existing products, processes and/or services
Generally, sensory substitution devices have shown truly remarkable progress in recent years. However, at this
time they are still not able to provide detailed information needed for fully experiencing our visual world,
especially its more advanced formats. Furthermore, when using these devices the users are basically passive and
unable to use many features of normal visually based exploration of our world. Therefore, although providing the
basic information, they leave the blind deprived of important contents of our visual cultural heritage such as art
and most content of the web. In the case of invasive sensory restoration and neuroprosthetic approaches, there
are still several major issues preventing these from becoming true clinical solutions, primarily their invasiveness,
costs, technical limitations and the inability to offer them to all populations of the blind. Thus, these solutions
still do not provide sight that resembles natural vision and are even more limited in providing the blind access to
our visual cultural heritage such as art.
Today, the visual world is available to the blind via reading and mobility aids, tactile and auditory sensory
substitution devices, and invasive neuroprosthetic approaches. Haptic displays for the blind are currently
expensive, clumsy, and designed for very limited use such as reading in Braille. The blind may also use
text-to-speech technology for content browsing. This is slow and cumbersome even in text-based scenes,
but has no power in describing visual images. Printed embossed maps are also limited to a certain scale, as
well as expensive and scarcely available. Therefore, inexpensive haptic displays with interactive map
functions could significantly enhance the quality of life of the blind and partially sighted in allowing them
to navigate more accurately and provide more freedom for them to visit unfamiliar places. Although
constantly becoming more advanced, sensory substitution devices provide only very rough images of basic
objects and scenes and don’t offer additional information needed for appreciating the richness of our visual
world, especially its more advanced formats such as visual arts. Sensory deprivation affects functional and
structural organization of the brain underlying visual perception and it takes time to “become fluent in
vision”.15
15
Fine et al. Long-term deprivation affects visual perception and cortex. Nat. Neurosci. 6,915-916, 2003
17
FP7-ICT-2013-C
1.2.3
SEEDIFF
Main innovations by SEEDIFF participants [references to articles or web pages to
footnotes, please!]
1) The Jovicic DiTacta diagrams and stand with audio guides allow independent exploration by blind people of
visual scenes in an accessible way. The novelty is in representing visual information in a new format; the
approach involves formal analysis and is practical and didactic.
2) New haptic touch-screen technology providing a widely-accessible and inexpensive means for transforming
visual art and other visual images such as maps to haptic–auditory form on tablets and mobile phones.
www.senseg.com. Senseg has the core granted patents for electrostatic vibration technology: US patents
7,924,144 (granted APR 2011) and 7,982,588, (granted JUL 2011), and 8,174,373 (granted APR 2012), EPO
patent 08805437.4 (allowed MAR 2013), China ZL200880107664.0 (granted MAR 2013).
Public papers (can be in Company chapter as well)
[HAID 2009]
Linjama, J. & Mäkinen, V. E-Sense screen: Novel haptic display with Capacitive Electrosensory Interface.
Proceedings HAID 09, 4th Workshop for Haptic and Audio Interaction Design,Dresden, Germany 10-11 Sept
2009. Vol II, pp 24 -25.
[Eurohaptics 2012]
D. Wijekoon, M.E. Cecchinato, E. Hoggan, and J. Linjama, Electrostatic Modulated Friction as Tactile
Feedback: Intensity Perception. ;In Proceedings of EuroHaptics (1). 2012, 613-624.
3) New art form: haptic drawings for both the blind and the sighted, allowing also the blind to create haptic
“paintings”.
4) Hexachromatic display technology to expand the experience of color for both the color-blind and the normally
sighted.
Brooks, A. PRODUCT FROM PATENT – see below
5) Treasures in Ateneum. Detailed audio descriptions of six gem art works in the Ateneum collections, designed
and produced in cooperation with Celia Library for persons with visual disabilities
http://www.ateneum.fi/fi/ateneumin-aarteita ( in Finnish)
1.2.4
Patent search results
There is great amount of patents in the fields of haptics and user interfaces. However, this does not pose
difficulties nor concerns to the current project due to following reasons:

Senseg has strong IPR portfolio with multiple granted patents

Only commercial applications of technology can infringe patents and therefore they do not pose limits to
research scope of this project.

The field of expanding human senses and transforming art into tactile form are too broad areas to be
covered by patents; only specific implementations could be patented. This can be exemplified with
18
FP7-ICT-2013-C
SEEDIFF
patents search into the US patent data base. For example patents: Tactile reading system for data coming
from a computer and associated communication device US patent 6,639,510, System and method for
integrating tactile language with the visual alphanumeric characters US patent 7,318,195, Tactile graphic
computer screen and input tablet for blind persons using an electrorheological fluid US patent 5,222,895,
are typical patent applications describing a narrow band application of the technology towards the blind,
which have no bearing for this project.
Brooks, A. et al. COMMUNICATION METHOD AND APPARATUS. Patent Family (Pat App 2000): US
6893407 (B1); EP 1279092 (B1); WO 0186406 (A1); DE 60115876 (T2); AU 5822101 (A); AT 313111 (T).
1.3
S/T methodology
SEEDIFF will develop a range of new technologies and techniques related to presenting visual art in a haptic–
auditory form. A new generation of haptic displays and software-controllable surfaces and sensations is expected
to make even visual art equally accessible to sighted and blind. The S/T methodology will include: 1)
electrostatic technology for haptic display on mobile devices; 2) programming interface to transform visual
scenes to haptic–auditory multi-sensory format; 3) behavioral and functional brain imaging studies to design and
optimize the sensory inputs and explore uni- and multi-sensory perceptual pathways and multi-sensory
integration compensating for and/or extending human visual perception. This neuro-ICT–art trans-disciplinary
project will also include the analysis of paintings from the perspective of an art historian, surveys, multi-media
and web presentations; it will showcase the new art and explore its acceptance among sighted and blind, both
adult and children within educational contexts.
The partners include four well-known universities, a major national art gallery, and a fast-growing SME.
UNIZG will analyse paintings to be presented to the blind, describe their composition, the elements of form,
lines and surfaces, the details in the pictures, and their colors in order to present them as multiple tactile
diagrams. The presentation of colors by textures will be developed, which in the electronic format will include
novel vibratory or other time-dependent effects. To help navigate the display and to convey our message, we will
develop audio enhancements and provide an audio guide that, in addition to explaining the content and
background of the painting, leads the blind person from one point to the next in the painting and from one frame
to another. Once developed, other partners in the proposal will test several potential tactile–auditory
combinations representing the visual input behaviorally and with neurodynamic functional brain-imaging studies
with the blind.
19
FP7-ICT-2013-C
SEEDIFF
Box. Novel concepts and technologies under development by the partners
I. Tactile–auditory exhibition of visual art for blind and sighted invented by UNIZG (Fig.1).
Fig. 4. Left: Exhibition by UNIZG where a 5-display representation of Caravaggio’s painting
was presented to the blind in tactile–audio form. Right: One of the details of the painting; the
tactile pattern is overlaid on the corresponding visual image.
II. The haptic display technology based on a recent invention and under development by
Senseg (Fig. 2) will allow one to produce vivid sensations of texture and form on touch-screen
devices such as tablets or smart phones. New forms of tactile interaction will be developed to
represent the richness of visual information to partially sighted people on these displays.
Fig 5. Left: The principle of our haptic display. A transparent positively charged layer under
the surface of a touch screen gives rise to an electric field that makes the fingertip negative
(positive charges are pushed away). A force proportional to the electric field and the charge
displacement is created (proportional to V2). Right: It is possible to create vivid sensations of
shapes or textures on mobile displays.
III. For the color blind with dichromacy, we plan to expand color vision by a novel 3Ddisplay technology that can add spectral dimensions (as if there were 3 or more cone types in
the retina).
Fig 6. A pair of differently colored paintings (Kandinsky). When viewed stereoscopically (left
image to the left eye, right image to the right eye), new color sensations can be created. In
photography, every other wavelength band could be directed to the left image and the other
bands to the right, creating a hexachromatic display.
20
FP7-ICT-2013-C
SEEDIFF
Senseg will develop its haptic technology to allow one to present tactile–auditory representations of art and
other visual scenes to the blind. It will provide software tools to enable other partners to experiment with the
company’s new technology and to design multisensory content for the blind. The software will allow a flexible
pace of exploration; the amount of detail or background information is selectable. The audio presentation can be
synchronized to the tactile exploration without the need to follow a strict order. This complex processing will be
done in steps allowing both the automated conversion of visual information into tactile realm, as well as guided
design of haptic design and new interaction elements. Aalto will experiment with the new color display
technology based on presenting different parts of the color spectrum to different eyes and will work on software
development. Together with UNIZG and UNIST, it will also perform MEG/EEG experiments with tactile and
multi-color visual stimuli, including visual mismatch (MMN) studies.
In collaboration, the university partners will conduct a range of behavioural experiments to study the effects of
different combinations of the simultaneously presented tactile and auditory information, in order to better
understand the coupling between the two sensory inputs and explore their more efficient use in creating the
emerging visual experience. These results will be used for specifying the properties of cross-modal information i
in the final product of this project. The three university partners will also use neurodynamic MEG/EEG methods
to study multi-sensory modulations of human visual processing evoked by visual, tactile, and auditory–tactile
stimulation in subjects with full vision in a search of: 1) an analogue of a retinotopic mapping of the visual field
presented via a tactile/haptic display; 2) neurodynamic networks activated by single and multiple objects
recognition and discrimination tasks (i.e., simple object vs. face) and 3) potential of texture-coded enabled colour
perception; 4) multi-frame tactile-auditory integration in coherent visual art perception; and 5) an objective
evidence of hexachromatic perception. Partially sighted and blind subjects will be used in a subset of studies to
explore non-visual multi-sensory integration and cross-modal plasticity and find conditions for enabling an
enhanced visual experience even in blind.
Ateneum will contribute by focusing on cultural, art historical and practical aspects of the project. It will
participate in designing and/or evaluating multisensory presentations (a haptic display + audio) for the blind or
partially sighted and in designing and/or evaluating multi-colored images. The exhibition will be made available
to collaborator galleries in different parts of Europe to allow a large audience across EU to get in contact with the
project in its different phases. Providing masterpieces of the museum collection as objects for collaborative
research, and offering space and expertise in arranging its own exhibition and exhibitions in other major
museums, the gallery will form one of the hubs of activity in SEEDIFF. Led by Ateneum, SEEDIFF will
arrange an in-depth visitor survey that includes interviews and video recordings of the behavior of the blind and
other visitors of the exhibition. Based on the ideas, methods, contents, and observations of SEEDIFF, Ateneum,
together with others, will develop educational and demonstration material, including exhibition brochures and
web pages to obtain publicity and to spread the results.
Aalborg, which has long-term experience of creative work in bridging science and art as well as in working with
minorities such as the blind, will act as a catalyst to research, develop and showcase the new art offered by the
state-of-the-art advancements resulting from SEEDIFF. This will include a lab-based/museum-based
Participatory Design strategy questioning the impact for the blind. From this will evolve new frameworks for
creative expression and related stimuli-enhanced experiences. The Noldus Observer behavioral analysis tool will
be used to examine responses, which will inform to advance the creation of the SEEDIFF state-of-the-art. The
phenomenon of blind artists will be researched along with prior related cross-modal works. Aalborg will also
lead the concluding dissemination at the SEEDIFF European Cultural Capital exhibitions that will conclude the
project with high impact dissemination when Denmark co-host with Cyprus in 2017.
21
FP7-ICT-2013-C
SEEDIFF
Fig. 8. SEEDIFF’s inter-connectivity network of innovative neuro-art-ICT technologies and approaches (in
yellow), research studies, exibitions and societal impact, as well as project partners and end-users
1.3.1
Overall strategy of the work plan
We will develop two ICT platforms for the presentation of visual art and scenes in new haptic and color forms.
We will divide the necessary steps towards our goals into well-defined Work Packages (WPs) so that when the
WP tasks are done according to the schedule (see Gantt table), we will reach the goals during the project
duration.
WP1 (see also Section 2) is meant for making sure that SEEDIFF proceeds according to the plan. WP2 and WP3
deal with the development of haptic and color displays, i.e., the tools needed for completing work in the other
WPs. WP4, based on previous work on fixed (usually glass) haptic frames and on our invention of
hexachromatic displays, deals with developing methodology to transform visual images to the new forms. In
addition, neurophysiological experiments in WP5 provide information regarding neural foundations of enhanced
sensory perception. WP6 is a bridge from the methodological development to societal applications of the new
technology and serves WP7 by developing dissemination and educational uses for the technology. The solutions
developed in the different WPs will be tested using behavioral studies that will also provide a bridge towards
testing their neural correlates in neurophysiological studies planned in WP5.
ii) Show the timing of the different WPs and their components (Gantt chart or similar).
Note that, whereas the scientific and technological methodology is evaluated under the criteria ‘S/T quality’, the quality of the actual
workplan is evaluated under FET-Open under the criteria ‘Implementation’.
2 A work package is a major sub-division of the proposed project with a verifiable end-point – normally a deliverable or an import ant milestone in the overall project.
1
Version V1.0 55/72 26-02-2013
Information and Communication Technologies Call FP7-ICT-2013-C
Guide for Applicants Collaborative Projects (STREP)
iii) Provide a detailed work description broken down into work packages:
22
FP7-ICT-2013-C
SEEDIFF
1. Work package list (please use Table 1.3a);
2. Deliverables list (please use Table 1.3b);
3. List of milestones (please use Table 1.3c);
4. Description of each work package (please use Table 1.3d);
5. Summary effort table (please use Table 1.3e)
iv) Provide a graphical presentation of the components showing their interdependencies (Pert diagram
or similar)
v) Describe any significant risks, and associated contingency plans
1.3.2
Work description
Table 1.3a: Work package list
WP
Work package title
Type of
activity
Lead
partner #
Lead
partner
Personmonths
Start
month
End
month
1
Management
MGT
1
Aalto
22
1
36
2
Haptic hardware and software
RTD
4
Senseg
65
1
36
3
Hexachromatic technology and
methodology
RTD
1
Aalto
56
1
36
4
Tactile–auditory representations
of visual scenes
RTD
5
UNIST
77
1
36
5
Neuronal foundations of
enhanced sensory perception
RTD
6
UNIZG
79
1
36
6
Social and personal aspects
RTD
2
Aalborg
58
1
36
7
Dissemination
RTD
3
Ateneum
13
1
36
Total
384
Table 1.3b: Deliverables list
#
Deliverable name
WP
Nature
Dissemination
level
Delivery
month
D1.1
12-month progress report 1
1
R
CO
12
D1.2
24-month progress report 2
1
R
CO
24
D1.3
Final report
1
R
CO
36
D2.1
Visuo-tactile rendering concept for touch screens
2
R
PU
9
23
FP7-ICT-2013-C
#
SEEDIFF
Deliverable name
WP
Nature
Dissemination
level
Delivery
month
D2.2
First test implementation in tablet device
2
R
PU
12
D2.3
Pilot implementation in an exhibition setting
2
R
PU
18
D2.4
Complete tactile display functionality in exhibition
2
R
PU
30
D2.5
Solution description for tactile experience display
2
R
PU
36
D3.1
Study of color spectra in natural and man-made
environments
3
R
PU
18
D3.2
Methodology for hexachromatic photography;
results of hexachromatic photography and
hexachromatic art
3
R
PU
30
D3.3
Results on behavioral and neurophysiological
studies of hexachromatic vision
3
R
PU
30
D4.1
Report on existing tactile-auditory coding schemes
giving details per scheme, evaluation results, the
underlying task, a list of annotated phenomena
4
R
PU
6
D4.2
Test results for tactile–auditory display
4
R
PU
24
D5.1
Report on non-visually multi-sensory enabled
demonstration of enhanced sensory perception of
objects
5
R
PU
D5.2
Workshop on multi-sensory integration and crosssensory plasticity role and potential in education,
rehabilitation
5
R
PU
D5.3
Report on enabled and augmented color perception
in sighted and color blind, respectively by
hexachromatic display technology
5
R
PU
D6.1
Educational workshop on tactile–auditory and
hexachromatic presentations of art to adults and
children
6
R
PU
24
FP7-ICT-2013-C
#
SEEDIFF
Deliverable name
WP
Nature
Dissemination
level
Delivery
month
D6.2
Report on suggested strategies for the use of the
new tactile display in education
6
R
PU
34
D7.1
Web page
7
D
PU
6
D7.2
Exhibition of haptic art and hexachromatic color at
Ateneum
7
D
PU
9
D7.3
Exhibition of haptic art at Modern Gallery, Zabreb
7
D
PU
24
D7.4
Exhibition of social aspect of sensory expansion,
Aalborg
7
D
PU
30
D7.5
Exhibition with a video, presenting SEEDIFF
advances , Ateneum
7
D
PU
34
Table 1.3c: Milestones list
Milestones
WP2: Initial version of tactile display and software ready for experimentation with the blind, M12
WP3: Hardware and first version of software tools ready for experimentation with hexachromatic photography
and to transform trichromatic images artificially to hexachromatic form, M6
25
FP7-ICT-2013-C
SEEDIFF
Tables 1.3d: Work package descriptions
Work package number
1
Work package title
Management
Activity Type
MGT
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
5
6
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
17
1
1
1
1
1
Objectives
The purpose of WP1 is to make SEEDIFF succeed. While ambitious and risky, SEEDIFF has the potential to
change the world of the blind as well as to enhance human sensory experience in sighted people, with
possibilities for new art forms and other applications. The goal of WP1 is to maximize the efficiency of
SEEDIFF in obtaining the best possible results according to the plan.
O1.1: Ensuring that the objectives of SEEDIFF are reached with maximum efficiency.
O1.2: Smooth running of the project.
O1.3: Ensuring intellectual property.
O1.4: Resolution of any conflicts.
O1.5: Financial and technical reporting to the Commission.
Description of work
The work is described in detail in Section 2.1. This covers in particular:
Task 1.1: Constitution of the Executive and Steering Committees
This will be initially done in connection when agreeing about the Consortium Agreement
Task 1.2: Day-to-day management, financial and administrative management
The day-to-day management will be done by the Coordinator and Work Package Leaders.
The Executive Committee and Coordinator will be in charge of the administrative and financial management.
Task 1.3: Reporting and assessments
26
FP7-ICT-2013-C
SEEDIFF
The Coordinator will be in charge of collecting the reports given by each Participant and will prepare the reports,
together with the rest of the Executive Committee, in a form corresponding to templates provided by the
European Commission.
The Executive Committee will assess the progress of SEEDIFF in their monthly meetings and, more thoroughly
in the SEEDIFF project meetings.
Task 1.4: Protection of intellectual property
The Coordinator will oversee that any essential IPR is evaluated as to whether patenting is necessary. The
Consortium agreement will describe in detail how the IPR can be transferred from Participants to the industrial
Participants or to other parties.
Deliverables
Each progress report will describe … [Krisztina, can you say what the progress reports should include?]
D1.1: 12-month rogress report, M12
D1.2: 24-month progress report, M24
D1.3: Final report, M36
27
FP7-ICT-2013-C
SEEDIFF
Work package number
2
Work package title
Haptic hardware and software
Activity Type
RTD
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
5
6
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
6
2
1
49
6
1
Objectives
The overall goal of this WP is to establish principles and practices, i.e., a completely new approach, for haptic
technology for the blind as well as to build a hardware and software platform that is suitable for presenting
tangible art.
O2.1: Develop a software approach that is suitable to present tactile interaction in SEEDIFF context and suitable
for combining multimodality such as auditory narratives into tactile display presentations.
O2.2: Develop hardware platforms that can be provided to the whole SEEDIFF consortium.
O2.3: Demonstrate tactile interaction with the visual scenes to provide life-like experiences of the art scenes.
O2.4: Refine the user experience data and iteratively develop the presentation methods of tangible art.
O2.5: Reach an understanding whether tactile displays in museum concept are viable solution to cater the needs
of the visually impaired and other special groups, and whether this concept is suitable to be expanded to
exhibitions all over the world.
Description of work
Task 2.1: An approach and software tools for mapping visual scenes to interactive experiences
-
Collect and synthetize the data and needs of SEEDIFF group for a formulation of target
specification for the software approach
-
Examine and research media presentations techniques for the blind and for children, adapting
those for the needs of tactile display technology of this project.
-
Making the key decisions for the software methodology, repository structure, operating systems
support scope, hardware dependencies of the software
Establish methodology of creating tactile “haptified” images and having software tools to enable
that
28
FP7-ICT-2013-C
SEEDIFF
-
Build multimodal presentation and interaction capabilities into the software.
-
Develop fluent touch screen and device motion based interaction mechanisms for the blind,
which include: zooming to the details and navigating inside content, and real time rendering of visual and
tactile data.
-
Collaborating on the software work with the other consortium members, providing suitable
software interfaces
Task 2.2: Touch screen hardware platform for tactile presentations
-
Evaluate the needs of the SEEDIFF consortium from hardware perspective
Evaluating various off-the-shelf touch screen platforms for technical challenges in respect to
integrating Senseg tactile technology into them
-
Building multiple iterations of the platform and testing the promising candidates
-
Freezing the specification for a device
-
Sourcing the materials needed for multiple units of the selected platform
-
Building multiple hardware units that can be used by different members of the SEEDIFF group
Task 2.3: Developing user experience demonstration using the software on the hardware platform
-
Together with other SEEDIFF parties draft a palette of design concepts for tangible art user
experience demonstration
-
Co-operate in implementing the tangible user experience software concepts
-
Support other consortium members in the use of software platform and hardware platform
-
Perform both technical and the initial user testing for the user experience concepts
-
Refine the most functional user experience concepts to be ready for experiments with
appropriate visually impaired target groups
Task 2.4: User testing and iterative development of the user experience with the consortium
-
Working with the consortium to build user experience test settings with the visually impaired.
Participating in exhibition arrangements, including customizing the hardware platform
according to exhibition needs including addressing mechanics needs of exhibition setting
-
Collect user experience data, improve and develop the user experience demo
Probe the applicability of the tactile experience display in wider exhibition setting, and probe the
needs for platform for customization.
Deliverables
29
FP7-ICT-2013-C
SEEDIFF
D2.1: Visuo-tactile rendering concept for touch screens, M9
D2.2: First test implementation in hardware platform device, M12
D2.3: Pilot implementation in an exhibition setting, M20
D2.4: Complete tactile display functionality in exhibition, M30
D2.5: Solution description for tactile experience display, M36
30
FP7-ICT-2013-C
SEEDIFF
Work package number
3
Work package title
Hexachromatic technology and methodology
Activity Type
RTD
1
2
3
4
5
6
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
35
3
1
6
1
10
Participant number
Participant short name
Month 1
Starting date
Person-months per participant
Objectives
The overall goal of WP3 is to establish principles and practices for hexachromatic vision and the methodology
of realizing such vision.
O3.1: To build a platform for studies of hexachromatic color perception.
O3.2: To develop the methodology of transforming visual scenes to hexachromatic descriptions. These
descriptions will be used as the basis of presenting hexachromatic images on standard 3D display devices such
as 3D TVs.
O3.3: Exploration of hexachromatic technology as a new aspect in visual art
Description of work
Task 3.1: Study of color spectra in natural and man-made environments
The spectrum of reflected visible light will be measured from different objects in selected environments that
have many differently collared surfaces. Such environments include botanical garden; zoo, especially areas for
birds and butterflies; supermarket with multiple man-made items, fruits and vegetables; art museum with
paintings will be also obtained. The data will be used to determine the division of the visible part of the spectrum
into 6 bands so that maximal information of the original spectra is retained (see Fig. 2).
Task 3.2: Hexachromatic photography
A camera setup will be constructed consisting of two high-quality cameras (such as Nikon D7000) mounted next
to each other for simultaneous exposures and fitted with exchangeable notch filters.
Natural scenes (those mentioned above) will be photographed or video is recorded so that two complementary
exposures (call them “left” and “right”) of each scene are obtained with comb filters selected on the basis of the
results of T3.1 so that the left exposure is formed by light in odd-numbered bands of the visible spectrum while
31
FP7-ICT-2013-C
SEEDIFF
the right exposure is obtained by even-numbered bands (see Fig. 2).
Task 3.3: Hexachromatic “art”
Pieces of visual art consisting of initially uniform patches of color (such as those by Piet Mondrian, Vasili
Kandinsky, Johannes Itten, or Joseph Albers) will be divided into left and right “component pictures” so that the
average picture is identical or similar with the original but the two pictures have different colors and/or
intensities in its patches so that the information content of the image is increased.
Task 3.4: Testing perception of hexachromatic displays
Hexachromatic photography and art will be tested behaviorally (sensitivity thresholds to different kinds of color
changes will be determined) and neurophysiologically. An odd-ball paradigm with hexachromatic pictures and
their trichromatic counterparts (averages of left and right images) will be presented and differences in brain
responses to the two kinds of stimuli will be determined.
Both normally sighted and color blind subjects will be recruited.
Task 3.5: Testing perception of hexachromatic displays
Hexachromatic photography and art will be tested behaviorally (sensitivity thresholds to different kinds of color
changes will be determined) and neurophysiologically (odd-ball paradigm: hexachromatic pictures and their
trichromatic counterparts (averages of left and right images) will be presented and differences in brain responses
will be studied.
Deliverables
D3.1: Study of color spectra in natural and man-made environments
D3.2: Methodology for hexachromatic photography; results of hexachromatic photography and hexachromatic
art.
D3.3: Results on behavioral and neurophysiological studies of hexachromatic vision.
32
FP7-ICT-2013-C
SEEDIFF
Work package number
4
Work package title
Tactile–auditory representations of visual scenes
Activity Type
RTD
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
5
6
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
5
1
2
1
36
32
Objectives
The overall goal of WP4 is to develop methodology to transform visual scenes such as visual art, drawings, and
photographs to tactile–auditory form. This can be done using behavioral experiments aimed to test the currently
developed solutions and to evaluate individual features of tactile-auditory elements present in the diagrams.
O4.1: To develop the methodology of transforming visual scenes (including works of art) to haptic descriptions.
needed for developing 2D tactile diagrams of visual paintings.
O4.2: To test the applicability of the translation methodology developed for visual paintings on haptic display
devices.
Description of work
Task 4.1: Develop “grammar” of segmentation of visual scenes to tactile format and tactile labeling. Determine
the relevant tactile and auditory features that allow the translation of presented visual scenes into an
understandable tactile-auditory format (UNIST, UNIZG)
Collect information on existing tactile/haptic-auditory coding schemes developed for pieces of art work or visual
scenes in general, either by interviewing, e-mail correspondence with developers and users or by literature
search. Information on evaluation results as well as tactile act recognition (how the visual scene was segmented
to tactile format and how it was labelled), communication problems, cross-level issues, co-references, perceived
levels of difficulty and prosody will be collected. Analyse these data and suggest a standard framework and a
series of best practice coding schemes.
Task 4.2: Select the best practice coding schemes. Conduct behavioral experiments in order to test
participants’ experience of developed tactile–auditory representations and determine how well they
translate the visual experience to the “viewer”: how well the viewers understand the display, what
33
FP7-ICT-2013-C
SEEDIFF
emotional reactions it triggers, and how the artistic message is conveyed. (UNIST, UNIZG)
By using structured interviews in qualitative studies of up to 10 blind and partially sighted adults: a) test acuity
of perceived visual scene using questionnaires developed for assessing participants’ experience of tactile
diagrams, b) estimate the difficulty/ease in understanding of the tactile-auditory coding schemes by using
structured interviews and collecting information described in T4.1, c) estimate how the artistic message was
conveyed through tactile diagrams. Based on results choose the best practice coding schemes from the set of
T4.1 schemes.
Interview-based qualitative research approach is chosen for this goal as it allows us to probe into depth
participants’ experience with the presented tactile diagrams.
After choosing the best subset of schemes, repeat experiment a)-c) on pieces of art with significantly different
painting characteristics. This approach should test whether schemes are dependent on concept, i.e., should there
be a difference in method of segmenting visual scenes to tactile format for significantly different paintings.
Finally, validate earlier findings on a separate sample of participants and determine differences between
different populations of the blind.
T4.3: Labeling in haptic displays - testing perception of microelements (< 2–3 mm) and macroelements
(UNIST)
Evaluation of electrostatic haptic elements that optimize tactile spatial acuity for simple 2D forms. Identify
tactile micro and macroelements to be presented (i.e., grating orientation, grating ridge width, gap detection, ...).
For each element define two alternative forms, randomly display these forms (each form is presented at least 3
times) and test accuracy of recognition in several experiments using groups of approximately 30 blindfolded
sighted participants and a smaller number of blind subjects.
Deliverables
D4.1: Report on existing tactile-auditory coding schemes giving details per scheme, evaluation results, the
underlying task, a list of annotated phenomena, M6
D4.2: Test results/reportsfor tactile–auditory display, M24
34
FP7-ICT-2013-C
SEEDIFF
Work package number
5
Work package title
Neuronal foundations of enhanced sensory perception
Activity Type
RTD
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
5
6
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
12
1
-
1
14
51
Objectives
The overall objective of WP5 is to learn about brain processes when novel sense-enhancing technologies are
used during the learning phase and in routine use. We will use neurodynamic EEG and MEG methods to obtain
measures of neuronal function related to the visually and non-visually enabled sensory perception in order to
objectively compare brain responses when different stimulation paradigms are used. The results, together with
those from behavioral studies, will be used to decide how the multi-sensory stimulation and new display
technologies should be enhanced and optimized, respectively.
O5.1: To identify whether there is a tactile–spatial analogue of retinotopic representation using MEG
O5.2: To demonstrate that object recognition and multi-object discrimination (simple object vs. face) is
enhanced by combined tactile/haptic and auditory stimulation
O5.3: To demonstrate effectiveness of texture-coded color perception.
O5.4. To demonstrate multi-frame visual-to-tactile–auditory integration in coherent visual art perception
O5.4: To develop and demonstrate an objective measure of hexachromatic perception based on MEG or EEG
Description of work
Task 5.1: Search for tactile analogy of retinotopy (UNIZG, UNIST, Aalto)
Sighted blindfolded, and blind subjects will be studied. Tactile/haptic objects are presented at different
eccentricities from the centre of the screen with and without tone guidance while MEG is being measured.
UNIZG will select blind subjects after behavioral studies in Croatia (UNIZG and UNIST), UNIST will design
haptic stimuli using software developed by Senseg, UNIZG will conduct MEG measurements together with
Aalto at Aalto MEG laboratory, and perform spatio-temporal source localization during the earliest evoked
responses (up to 100ms poststimulus). Aalto will recruit sighted subjects
Task 5.2: Multi-object recognition and discrimination
35
FP7-ICT-2013-C
SEEDIFF
MEG responses will be recorded from sighted blindfolded and blind subjects during tactile/haptic and
tactile/haptic and auditory simple geometrical objects and face recognition and multi-object discrimination tasks.
Task 5.3: Texture-coded color perception
EEG/MEG responses will be recorded from sighted, sighted blindfolded, and blind subjects during visual and
haptic, haptic, and haptic-auditory task after behavioural studies and training sessions on 3 to 5 texture coded
colors (behaviourally tested by WP 4) of a single and multi-object recognition tasks.
UNIZG will select blind subjects after behavioral studies in Croatia (UNIZG and UNIST), Aalto will recruit
sighted subjects, UNIZG and UNIST will design haptic and haptic and auditory stimuli by using software
developed by Senseg , conduct MEG measurements with Aalto at Aalto’s MEG laboratory . UNIZG will
conduct MEG spatio-temporal source localization during the earliest evoked responses (up to 100ms
poststimulus) for Task 5.1, localization and functional connectivity analysis for Task 5.2.; functional
connectivity and detrended cross-correlation time series and and pattern recognition approaches for Tasks 5.3.
and 5.4.
Task 5.4: MEG/EEG recordings of brain activity elicited by hexachromatic displays (Aalto, UNIZG)
Trichromatic and hexachromatic visual images will be presented to subjects in alternate order in such a way that
the average color remains the same but in the hexachromatic displays, the left and right eye receive different
colors. The hypothesis is that the brain reacts differently to these two kinds of stimuli. This will give us an
objective measure of the new kind of color perception so that we will not need to rely on verbal reports of the
subjects alone.
Aalto will design the stimuli, recruit the sighted and color-blind subjects, conduct and analyze MEG
measurements with UNIZG.
Deliverables
D5.1: Report on non-visually multi-sensory enabled demonstration of enhanced sensory perception of objects,
M?
D5.2: Workshop on multi-sensory integration and cross-sensory plasticity role and potential in education,
rehabilitation, M?
D5.3: Report on enabled and augmented color perception in sighted and color blind, respectively by
hexachromatic display technology, M?
36
FP7-ICT-2013-C
SEEDIFF
Work package number
6
Work package title
End-user studies: specifications for the technology and education
Activity Type
RTD
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
1
1
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
2
43
1
2
6
4
Objectives
The overall goal of this work package (WP) is to develop requirements and specifications for multimodal
communication tools for the blind. The focus is on the different types of users who will be influenced by the
implementation of the tools developed in SEEDIFF. Thus, visually impaired as well as end-users representing
the general public are targeted, together with secondary users such as educators, guides, curators, and blind
artists. The outcomes of the requirement analysis and specification will be to capture and document user input
and knowledge to enable conceptual frameworks for description and analysis with regard to the user, to the
haptic–auditory communication tool and its educational practice, and to its effectiveness to expand human
sensory experiences in the visual domain. The conceptual framework will feed into the analysis of actual
educational practices as well as to the design of visual content. Producing these results will require sensitive
planning and organisation, and a deep understanding of the characteristics of the different types of users
(visually impaired as well as sighted end-users and secondary users such as educators, guides, curators, and
blind artists) and educational knowledge practices as idiosynchratic entities whose interests and requirements
will emerge in a variety of different ways.
The activity is iterative and structured around the three phases, including Preparation, Incubation, and
Revelation. ???
The main lead in this WP will be taken by Aalborg with its experience and deep background in this area.
Therefore, Aalborg will also spend more resources in this WP.
Specific objectives:
O6.1: to identify and define requirements and specifications for tools in support of multimodal communication
processes,
O6.2: to describe methods and strategies for cultivating multimodal processes in educational practices taking
into account the different types of users,
O6.3: to provide optimised specifications in support of multimodal processes taking into account the different
types of users.
37
FP7-ICT-2013-C
SEEDIFF
O6.4: To estimate potential of haptic displays in different settings, i.e., in art education of blind and seeing
people and in development of new art form
The activity is iterative and structured around the three phases, including Preparation, Incubation, and
Revelation.
The main lead in this WP will be taken by Aalborg with its experience and deep background in this area.
Therefore, Aalborg will also spend more resources in this WP.
Description of work
Task 6.1: Requirement analysis
Goal: to define a conceptual framework for description and analysis of multimodal communication processes
and, based on that, map out possible requirements so as to place the theoretical and methodological knowledge
into practice. A fundamental component of this task is “inspiration”. This is a reciprocal activity whereby the
users inspire the researchers through demonstrating their interests.
Activities: xxxx
Task 6.2: Incubation and analysis of educational practices
Goal: to define and describe authentic pedagogical and learning activities by enrolling the users and reflecting
on:
(i) user current knowledge and experience,
(ii) technological feasibility,
(iii)
user practice.
This task requires a balance between the freedom to explore any idea and the practice in which the ideas will be
applied. This task will be the most fertile task of the work as the researchers will need to consider how the ideas
might be transferred to methods and strategies for cultivating multimodal communication processes in
educational practices. This will require sensitive consideration of the different types of user characteristics.
Activities: xxxx
Task 6.3: Revelation of specifications
Goal: to define requirements and specifications to allow the different types of users to influence:
(i) visions and requirement proposals,
(ii) impact on the design of multimodal communication processes,
(iii)
the teaching and learning experiences of multimodal communication processes.
(iv)
Activities: xxxx
Task 6.4: Testing haptic display methodology in educational settings (UNIST, Aalborg)
- Presenting SEEDIFF’s products to art teachers at all levels of education, interviewing them and developing
strategies for the use of the new technology in education of blind as well as seeing people.
Deliverables
D6.1: Educational workshop on tactile–auditory and hexachromatic presentations of art to adults and children,
M?
D6.2: Report on suggested strategies for the use of the new tactile display in education, M 34
38
FP7-ICT-2013-C
SEEDIFF
Work package number
7
Work package title
Dissemination and studies of user behavior at exhibitions
Activity Type
RTD
Participant number
Participant short name
Person-months per participant
Month 1
Starting date
1
2
3
4
1
1
Aalto
Aalborg
Ateneum
Senseg
UNIST
UNIZG
4
6
7
1
1
7
Objectives
The overall goal of this WP is to communicate the results of SEEDIFF to the general public and to end-user
groups such as the blind and other visually impaired. While doing this in the form of exhibitions at major art
museums, visitor acceptance and reactions will be studied in order to understand the needs of the end-users and
to get guidance to continued development of the hardware, software, content, navigation, and instructions.
O7.1: To ensure internal and external dissemination of results.
O7.2: To organize exhibitions and to perform behavioral studies of visitors.
O7.3: To organise a workshop opened to a large industrial and research European community.
O7.4: To prepare the exploitation of the results.
Description of work
T7.1: Web page: creation, development, and maintenance (Aalto)
The web page will be targeted to a wide audience: students, researchers, developers, investors, the lay public as
well as the media. Parts of it will be designed to be accessible also to blind people, especially when they have
access to the haptic technology developed in SEEDIFF.
T7.2: Exhibitions
Art exhibitions for blind, partially sighted and sighted visitors will be produced in 3 major art museums. The
average duration of each exhibition will be three months.
The exhibitions will comprise an art work, tactile diagrams, an audio, a haptic display, and/or a hexachromatic
part.
T7.3: Behavioral experiments
39
FP7-ICT-2013-C
SEEDIFF
Behavioral experiments will be conducted, some of them to be documented in video recordings, in order to
collect feedback regarding the visitors’ experience and satisfaction. Experiments will be executed in cooperation
with 2-4 blind or partially sighted partners.
Analysing the survey results, and passing the results on for the use of other members of SEEDIFF, will provide
those participants developing haptic displays with crucial feedback information of the experiences of the
subjects using the haptic and audio devices.
Writing and producing an exhibition brochure (to be expanded to a book in a separate project) in cooperation
with the University of Zagreb.
T7.4: Visitor surveys in art museums (Ateneum)
Both blind and sighted visitors are asked either orally or by forms to describe their experience and satisfaction.
Analysis of the survey results will give feedback to haptic display developers and, when conducted in the final
exhibition in the end of the SEEDIFF project, it will serve as the feedback and summary of the results of the
project concerning the subject visitors; blind, partially sighted and sighted persons.
T7.5: Workshops and training events
Near the end of the project, a series of local Workshops reporting the results of the project will be arranged in
Aalborg, Helsinki, Split, and Zagreb.
T7.6: Collecting user feedback
At exhibitions, volunteer end-users will be tested with basic psychophysiological methods; they will be
interviewed or they will be asked to anser visitor survey questions on their reactions regarding the displays and
on their acceptance of the technology.
Deliverables
D7.1: Web page, M6
D7.2: Exhibition of haptic art and hexachromatic color at Ateneum, Helsinki, M 9
D7.3: Exhibition of haptic art at Modern Gallery, Zagreb, M24
D7.4: Exhibition of social aspect of sensory expansion, Aalborg, M30
D7.5: Exhibition with a video, presenting SEEDIFF advances, Ateneum, Helsinki, M34
SECTION 1.5.
40
FP7-ICT-2013-C
SEEDIFF
2
Implementation
2.1
Management structure and procedures
2.1.1
Management capability of the co-ordinator
The present Coordinator (Prof. Ilmoniemi) has documented ability to lead research groups and
consortia with determination and success. He has successfully coordinated two EU-funded projects, one
with 9 partners16 and the other with 13 partners17. In addition, he has led the Framework Programme
project participation as the head of the BioMag Laboratory18 at the Helsinki University Central Hospital
and as the CEO of the Nexstim Company19. He has also experience as evaluator of European proposals
and projects20.
The Coordinator led the BioMag Laboratory of the Helsinki University Central Hospital from its
beginning in 1994 until 2003. He was the Chairman of the Board of Nexstim Ltd. after he founded the
medical technology company in 2000; he was CEO in 2003–2005. The Coordinator was Head of the
Laboratory of Biomedical Engineering at the Helsinki University of Technology in the beginning of
2007 and the Head or the Department of Biomedical Engineering and Computational Science in 2008.
2.1.2
Management structure and decision-making structure
The highly interdisciplinary nature of, and the wide scope of expertise, from art to neuroscience to ICT,
required in SEEDIFF will demand close coordination between the participants and a clear management
plan. Project Management is considered a separate work package. A Consortium Agreement that will
establish the rules on how the Project is regulated will reinforce a strong partnership.
The European Commission, DG XII, Grant Contract PSS*1046, Joint European project “Imaging of language functions in
the brain”, 1999–2001 (BioMag Laboratory, Coordinator).
16
The European Commission, FP7-HEALTH-2007-A, Collaborative project “MEGMRI - Hybrid MEG-MRI Imaging
System", Contract No. 200859, 2008–2012 (BECS, Coordinator).
17
The European Commission, Joint European project “Objective evaluation of cognitive brain function and dysfunction”,
Contract N.o BMH4-CT96-0819 (DG XII–SSMA), 1996–1998 (BioMag Laboratory, Partner).
18
19
The European Commission, STREP project “Enough Sleep”, Contract No. 518189, 2005–2008 (Nexstim, Partner).
Evaluation of R&D proposals in the field of Neurosciences under the program “Quality of Life and Management of
Living Resources” of the 5th Framework Program, European Commission, DG XII, Brussels, January 17–20, 2000; in IST
Future and Emerging Technologies FET calls, Brussels, May 22–26, 2000; Helsinki, Sept. 11, 2000; Brussels, May 14–15,
2001; Helsinki, November 2001; Brussels, March 19–22, 2002; Brussels, October 26–29, 2004; Helsinki Nov. 2004.
Review of the progress of the ESPRIT project “Adaptive Brain Interfaces” (European Commission, DG XII; project ABI,
No. 28193). Ispra, Italy, October 29, 1999 and Helsinki, May 16–17, 2000; of the European Commission IST project
“Insight 2+”. Leuven, Belgium, December 2, 2002; of FP6 project PRESENCCIA, February 2007. [Flagship 2011–12
ETC.].
20
41
FP7-ICT-2013-C
SEEDIFF
The main objective of Project Management is to make sure that all steps needed for the realization of our
goals will be made in time and within the frame of the budget.
Project Management is the task of the Coordinator, the Vice Coordinator, the Steering Committee,
and the Executive Committee.
Steering Committee
External Advisors
European Commission
WP1
1
Coordinator
Project Officer
Executive Committee
WP 1
WP 2
WP 5
WP 6
WP 7
Figure 1. SEEDIFF management structure [Must be improved]
Coordinator
The Coordinator (Prof. Ilmoniemi) will lead the project and have the ultimate responsibility for its
success. He will employ an experienced, knowledgeable Project Manager with project-management
skills to help perform daily project management, to coordinate project self-assessment, to organize
meetings, to organize the financial and technical bookkeeping, and to collect the deliverables.
The Coordinator will, after consulting the Executive and Steering Committees, direct resources to any
critical bottleneck that may appear. If necessary, resources may be redistributed among the Participants.
The Coordinator will also use every effort to guarantee that the results of the project will be fully
exploited by the Participant organizations. SEEDIFF will licence new technology to third parties or form
start-up companies especially in cases where the present Participant SME is not able or willing to utilize
some of the results.
42
FP7-ICT-2013-C
SEEDIFF
In the course of the project, the fundamental question to ask regularly is, “how does the present work of
each Participant contribute to the objective of the project”. The Coordinator, together with the Steering
and Executive Committees, will help revise the work plan whenever needed and to redirect efforts
correspondingly.
If the Coordinator is not able to perform his tasks, the Vice Coordinator will overtake his duties until the
situation returns to normal. If necessary, a new Coordinator may be selected by the SC by a majority
vote of at least 4/6.
Supported by the Executive and Steering Committees, the Coordinator oversees the progress of the
project, initiates action when problems arise, delays can be foreseen, or when new opportunities for
additional progress are discovered. The Coordinator is the link between the project and the Commission;
he is responsible for the overall success of the project.
Steering Committee
The Steering Committee (SC) is composed of all Group Leaders (see Table 1), including the Coordinator
(Prof. Ilmoniemi) and Vice Coordinator (Prof. Supek). It is the ultimate decision-making body of the
project. The Steering Committee will be the body that makes any strategic decisions (budgetary,
scientific, etc.) for the project. If a Group Leader is unable to attend an SC meeting, he/she can assign a
representative with full voting power to such a meeting. It is also possible for to participate in decision
making at an SC meeting using Skype or other electronic means.
The Steering Committee will meet twice a year at the regular SEEDIFF project meetings. The decisionmaking process within the SC will be the result of a collegial discussion, followed by a vote in case no
unanimous decision can be made (one vote per those SC members who are present / absolute majority).
In the unlikely event when no majority can be reached, the Coordinator (or if he is absent, the Vice
Coordinator) will have the decisive vote. At least 4 members of the SC must be present for a decision to
be made.
The Steering Committee will be in charge of reporting and the revision and updating of the work plan,
on-going risk assessment, and general self-assessment of the project. It prepares the agenda and material
for each Steering Committee meeting.
The work of the Steering Committee will begin already when a positive funding decision has been
obtained. The Steering Committee members may then be asked to participate in the negotiations with the
Commission services. The work of the Steering Committee continues also after the end of the project as
long as there are open issues regarding the project.
Work Package Leaders / Group Leaders
The Work Package Leaders (WPLs, see table below), who are also Group Leaders of the respective
Participants, are in charge of the implementation of the project work plan and reporting to the
Coordinator. The WPLs will be in frequent (at least every two months) contact with the Coordinator to
ensure that the objectives of the individual WPs are met in accordance with (i) milestones and (ii)
deliverables and that (iii) each Participant fulfils its commitment to the work package.
Participant
number
1
Participant
short name
Aalto
Group leader/WP
Leader
WPs to
lead
Risto Ilmoniemi
1, 3
43
FP7-ICT-2013-C
SEEDIFF
2
Aalborg
Tony Brooks
6
3
Ateneum
Anja Olavinen
7
4
Senseg
Ville Mäkinen
2
5
UNIST
Ana Jeroncic
4
6
UNIZG
Selma Supek
5
Administrative Contact Persons
Regarding the administrative management, each Participant will nominate an Administrative Contact
Person in each institute/company in charge of communication all relevant finance and reporting matters
(finance distribution, audit certificates, human resource, legal aspects, etc.). This person should be
distinct from the group leader (see Table 1 for the names of group leaders) to assure that the information
needed will be readily available to the Coordinator.
The Project Manager serves also as Aalto’s Administrative Contact Person. His duties include the
collection of reports and other information from the participants and administrative work at Aalto,
including the sending of reports to the Commission. He works closely together with the Coordinator,
with an office next to that of the Coordinator’s to allow daily communication.
2.1.3
Monitoring, reporting progress and documenting results
Project evaluation and other meetings
The project will be formally monitored and evaluated by the Commission. In addition, the Steering
Committee may ask external evaluators, commentators, consultants, and collaborators to contribute to
the assessment of the project. We have asked Profs. Charles Schröder21, Göte Nyman22, and Ivica
Kostovic23, to act as external advisors. In addition, David Blankett24, member of the British Parlament,
21
Prof. Schröder is pioneer in primate and human studies in cross-sensory integration. His has demonstrated that low-level
auditory sensory area is involved in multisensory integration and that early multi-sensory convergence can occur through
both feedback and feedforward circuits and that evolving concept of the sensory processing hierarchy must encompass
temporal as well as anatomical dimensions.
22
Prof. Nyman is former dean and professor at University of Helsinki. Since Jan. 2013, he has been leading the 4-year,
“MIND programme” Finnish Academy project "What is an image?". He has wide experience in vision science, user
interfaces and interaction with the industry, e.g., with Nokia.
23
Prof. Kostovic is director of the Croatian Institute for Brain Research. He is developmental neuroscientist who discovered
transient synaptic zone (subplate) and its crucial role in the establishment of cortical maps, its modular organization. He
contributed to the understanding of the protracted period of human cortical plasticity and reorganization related to
maturation, socialization, education and cognitive development.
24
David Blankett was educated at schools for the blind in Sheffield and Shrewsbury, attended the Royal National College
44
FP7-ICT-2013-C
SEEDIFF
blind himself, is already advising the project from the perspective of both societal decision making and
blindness.
The progress of SEEDIFF will be reported in progress reports (deliverables 1.1–1.6) as well as in
SEEDIFF project meetings. The main purpose of the meetings is to advance the progress of the project.
When decisions are made or project issues are formally discussed, one of the meeting participants
(Project Manager, when present) is selected to keep the minutes of the meeting.
The Steering Committee may invite additional people to their meetings if that is considered
advantageous for the project.
Aalto will arrange a 2.5-day Kick-off Meeting during the first month of the project. The first half-day of
the meeting will be devoted to the background and the general approach of the project. The second day
will consist of thorough exposition and discussion of the plans of each of the Participants. On the third
day, specific strategic issues and expected challenges or bottlenecks will be discussed and decisions will
be made on what measures should be taken to address these.
The Steering Committee will hold a Review of Progress Meeting twice a year. The SC may propose a
change in the Project Plan in light of the progress made, new discoveries or developments made
elsewhere. Such a proposal will be presented to the Project Officer appointed by the Commission. These
meetings may be combined with review meetings in presence of external experts appointed by the
Commission.
The Coordinator will be in contact with the Participants monthly, usually by email or Skype, to discuss
the progress of the project.
Training events and small seminars are arranged throughout the project; unofficial project meetings can
be arranged between Participants in connection to these events.
Resolution of conflicts [to be revised]
All disputes or differences arising in connection with SEEDIFF that cannot be amicably settled within
the Steering Committee shall be finally settled through arbitration in Brussels under the Rules of the
International Chamber of Commerce (ICC). Arbitration shall be conducted in the English language.
Three Arbitrators (3) shall be selected. Each Party shall select one (1) arbitrator. Each Contractor or
party shall also notify the other Contractor or party/parties in writing of its selection and the two (2)
arbitrators so chosen shall select the third arbitrator. If either Contractor or party fails to appoint an
arbitrator within thirty (30) days after the Demand for Arbitration, then the International Chamber of
Commerce shall make the selection. If the two arbitrators selected by the parties cannot agree on the
third arbitrator within thirty (30) days from the date the last of the two has been selected, then the
International Chamber of Commerce shall make the selection. In deciding matters, the arbitrators shall
be bound by the terms and conditions of the Consortium Agreement, and they will decide ex bon et
aequo.
for the Blind in Shrewsbury. At the University of Sheffield, he gained a BA honours degree in Political Theory and
Institutions; one of his lecturers was Bernard Crick. He entered local politics on graduation, whilst gaining a Post
Graduate Certificate in Education (PGCE) from Huddersfield Holly Bank College of Education.
45
FP7-ICT-2013-C
SEEDIFF
The arbitration award, if providing for damages, shall include interest from the date of any breach or
other violation of the Consortium Agreement.
The arbitration award shall be final and binding upon the Contractors or parties, not subject to appeal,
and honoured by the Contractors or parties without having resort to any court; however, if the award is
not carried out voluntarily and without delay, it shall be referred to and enforced by any court having
jurisdiction over the subject matter or any of the parties or their assets.
Each Contractor or party bears its own expenses incurred in utilizing arbitration and the fees for
arbitration shall be borne equally between the Contractors or parties.
Risk assessment and management
Tables in Section 1.5 [CHECK NUMBERING] describe the main risks forecasted in this project with
their severity and indication of contingency actions.
The Executive Committee is in charge of the risk management, according to the following principles.
1. Based on the risk-analysis tables, estimate any risk that may have increased or is about to realize.
2. Establish and implement counter-measures if appropriate.
3. Set up early measures to minimize foreseeable risks.
4. Follow-up of the effects of the measures that have been taken.
The risks include those in management, financial situation, technical challenges or setbacks, change of
Participants, changes in dissemination and/or exploitation possibilities.
2.1.4
Meeting agendas
The biannual SC meetings and monthly EC meetings will be conducted according to a standard agenda;
minutes of the meetings will be assembled in Deliverables 1.1–6.
2.1.4.1
Standard agenda for Steering Committee meetings
1. Opening of meeting
2. Review of Progress /checkup status
3. Deliverables and milestones
4. Action list
5. Human resources issues
6. Financial issues
7. Monitoring of ethical and gender issues
8. Decisions regardging corrective actions, if needed
9. Other issues
10. Closing of meeting
46
FP7-ICT-2013-C
2.2
SEEDIFF
Individual participants
Participant 1: Aalto
Main tasks: Coordination, hexachromatic color technology, neurophysiological studies.
Previous experience: Coordination of EU-funded and national projects, summer schools, conferences etc.;
R&D on 6D color technology (funded by Runar Bäckström Foundation, 2011–2012); extensive experience on
neurophysiological studies using MEG, EEG, and TMS.
The Department of Biomedical Engineering and Computational Science (BECS) at Aalto University (formerly
Helsinki Univ. of Technology) is a pioneer in several neuroimaging technologies. MEG development started at
Aalto University (until 2010 Helsinki University of Technology) already in the 1970’s. The work led to the spinoff company Neuromag (now part of Elekta), the world leader in MEG devices. MEG research has been
extensive both at Aalto and at the Aalto-initiated joint research centre BioMag. TMS development started at
Aalto in 1992 and continued at BioMag and Nexstim Ltd. (founded by Ilmoniemi in 2000). Nexstim combined
TMS with MRI and EEG to a new imaging modality that can map cortical excitability and functional
connectivity. NIRS development started at Aalto in 1998 and has led to low-noise transportable multichannel
frequency-domain instrumentation. Ongoing studies on adults and neonates include clinical research.
In addition to the front-line instrumentation development, BECS is very strong in computational technology,
including the development of methodology for brain imaging.
BECS has produced a number of innovative technologies also in… The idea of enhancing color vision by using
the now-common 3D display technology was …
Risto Ilmoniemi is Academy Professor of the Academy of Finland (2012–2015) and tenured professor of
Engineering Physics at Aalto University; prior to the Academy Professorship, he was Head of the BECS. He
headed the BioMag Laboratory of the Helsinki University Central Hospital in 1994–2003 and was Chairman and
CEO of Nexstim in 2000–2003 and 2003–2005, respectively. At Aalto, he has lectured “Structure and Operation
of the Human Brain”, “Classical Electromagnetism” and “Functional Imaging in Medicine”. He is author of
about 200 peer-reviewed papers and 10 patents. His work to combine brain imaging and stimulation technologies
has been recognized by several awards25.
Otto Olavinen is a Producer in Aalto University School of Science. Educated in journalism, his expertise is in
visual presentations and in communication of science. Before coming to Aalto, Otto was a Vice Coordinator for
AAVE (Alternative AudioVisual Event) festival. He has also training in EU project management and will act as
the Project Manager and Aalto’s Administrative Contact Person.
Mika Pollari is an expert in image and signal processing. He was project manager in the FP7 MEGMRI project
(2008–2012) and coordinated BECS’s participation in the FP7 project IMPPACT. He will work in WP1 WP3.
Representative publications
P. T. Vesanen et al., “Hybrid ultra-low-field MRI and magnetoencephalography system based on a commercial whole-head
neuromagnetometer”, Magn. Reson. Med. DOI 10.1002/mrm.24413 (2012).
A. Susac, R. J. Ilmoniemi, D. Ranken, and S. Supek, “Sensory-memory-based change detection in face stimuli”, Transl.
Neurosci. 1, 286–291 (2011).
25
New Technology Foundation Innovation Prize 1997; Foundation for Medical Technology First Prize 1997; Innosuomi
2004 Prize by the President of Finland to Nexstim (Ilmoniemi, founder, as CEO); European IST Prize 2006 to Nexstim.
47
FP7-ICT-2013-C
SEEDIFF
Participant 2: Aalborg
Main tasks:
Previous experience:
Aalborg University is …
Tony Brooks is based in Aalborg University Esbjerg, Denmark, where he is an Associate Professor and a
founding member of the Medialogy education. He is Director of the SensoramaLab complex - see
http://medialogy.eu/facilities. Since the mid-eighties, he has developed SoundScapes, a multi-modal sensory
stimulus system for improvised artistic expression through physical interaction by people in the disabled
community. It is targeted at all people no matter ability, age, or preferences and has developed to be considered
as a non-formal rehabilitation training system for supplementing traditional methods of therapy. It is a nonintrusive (i.e. non wearable) system, which is based upon selectively choosing and combining from 'libraries' of
input sensor devices, selectable content feedback, and 'libraries' of delivery methods. Besides traditional camera
techniques used in multiple arrays, invisible 3D wireless sensor technology is used for sourcing natural
movement that is mapped to personalised multimedia feedback. Robotics are also utilised as interactive content.
The interaction is considered as playful activity that is both creative (music & art) and engaging (game playing).
Both the creative and the engaging aspects target joy and fun for the participant (and facilitator). The mapped
data is also routed and archived for post session analysis for session-to-session progress monitoring in therapy.
An interest for the research is the investigation of compensatory sense dynamic that is often evident following a
loss of another sense in the communities of people with different abilities. There are around 150 publications on
the work, which has been presented globally at major events. Workshops, performances and presentations
include the UN/NGO congress 1995; The Inaugural Cultural event for the Paralympics in Atlanta, USA in 1996
and the Millennium event in Sydney Australia (including the parallel scientific congress). It was featured in the
European Cultural Capital events 1996 and was also featured in the NeWave festival in New York USA. Touring
exhibitions at leading Museums for Modern Art have been ongoing since COIL (Circle of Interactive Light),
Brooks' room size interactive installations, from 1998 where the public created the art (audiovisuals) through
body gesture. A position on the co-ordination group of i3net (The European Network for Intelligent Information
Interfaces) is another credit. SoundScapes was at the core of a European Future probe (www.bris.ac.uk/Twiaysi/) and the ensuing FW5 IST project titled CAREHERE (www.bris.ac.uk/carehere) as well as various other
national and international projects. In 1999, Tony Brooks was awarded the European EUREKA award for
SoundScapes as applied multimedia and in 2006 he received the Vanførefondens Forskerpris award as selected
top Danish researcher in his field.
Eva Petersson Brooks is an associate professor; coordinating/managing the Medialogy Bachelor and Master
Education Program; and vice chancellor at Aalborg University Esbjerg in Denmark. She is member of the
research group SensoramaLab (see http://sensoramalab.aaue.dk). She has a background in Education Science
and her PhD is focused on ludic engagement (playfulness as a foundation for engagement) within virtual
environments
and
the
potentials
in
associated
non-formal
learning
(see
http://dspace.mah.se:8080/dspace/handle/2043/2963). Petersson has been coordinating research projects in
inclusive and participatory design, storytelling, and creativity and learning processes through the use of new
technology in education and rehabilitation contexts. She has been leading projects on physical and virtual toys,
the design of toys for children with disabilities, and the use of computer games (interactive environments) in
educational and rehabilitation contexts involving flexible methods of delivery for local, national and
international users. Petersson is member of the International Toy Research Association (ITRA) and the Toys for
Tomorrow Forum. Furthermore, she is a board member of the Pan-European Game Information (PEGI) and
expert member of the Interactive Software Federation of Europe (ISFE). She was one of the founders of
48
FP7-ICT-2013-C
SEEDIFF
International Toy Research Conference recently organised by the International Toy Research Association
(ITRA).
Awards:
(1) EUREKA AWARD Stockholm 1999 –
The European Brokerage Event on Applied Multimedia (BAM) http://vbn.aau.dk/files/14567272/pdf
(2) Vanførefonden research prize Copenhagen 2006,
http://www.vanfoerefonden.dk/index.php/Anthony-Lewis-Brooks/1102/0/
Representative publications [of Aalborg]
Brooks, A.L.: Human Computer Confluence in Rehabilitation: Digital Media Plasticity and Human Performance Plasticity.
HCII2013, Las Vegas, USA (2013)
Brooks, A.L.: Neuroaesthetic Resonance. In: G. De Michelis et al. (Eds.): ArtsIT 2013, LNICST 116, pp. 57--64. Institute
for Computer Sciences, Social Informatics and Telecommunications Engineering, Milan (2013)
Cobb, S., Brooks, A.L., Sharkey, P.: Virtual Reality Technologies and the Creative Arts in the Areas of Disability, Therapy,
Health, and Rehabilitation. Telerehabilitation Health Informatics, pp 239-261 (2013)
Brooks, A.L.: Active and Non-Active Volumetric Information Spaces to Supplement Traditional Rehabilitation. Journal of
Research and Practice in Information Technology (2013)
Brooks, A.L.: TeleAbilitation: GameAbilitation. Telerehabilitation Health Informatics pp. 225-237 (2013)
Brooks, A.L.: Intelligent Decision-Support in Virtual Reality Healthcare & Rehabilitation. In: Studies in Computational
Intelligence, Vol. 326, pp. 143-169 (2011)
Brooks, A.L.: Arts and Technology. Lecture Notes of the Institute for Computer Sciences, Social Informatics and
Telecommunications Engineering. Springer (2011)
Brooks, A.L.: SoundScapes/ArtAbilitation - Evolution of a hybrid human performance concept, method & apparatus where
Digital Interactive Media, The arts, & Entertainment are combined. Handbook of Digital Media in Entertainment and Arts.
Springer Publishing Company, pp. 683-711 (2009)
Brooks, A.L: Interpretations : an inter-sensory stimulation concept targeting inclusive access offering appreciation of
classical music for all ages, standing, & disability. Proc. 7th International Conference on Disability, Virtual Reality &
Associated Technologies (ICDVRAT) with ArtAbilitation. Reading University Press, pp. 15-22 (2008)
Brooks, A.L.: Towards a platform of alternative and adaptive interactive systems for idiosyncratic special needs. 7th
International Conference on Disability, Virtual Reality & Associated Technologies (ICDVRAT) with ArtAbilitation.
Reading University Press, pp. 319-326 (2008)
Hasselblad, S., Petersson, E., Brooks, A.L.: Empowered interaction through creativity. Digital Creativity, 18(2) pp. 89-98
(2007)
Petersson, E., Brooks, A.L.: Non-formal Therapy and Learning Potentials through Human Gesture Synchronised to Robotic
Gesture. Universal Access in the Information Society, 6(2) pp. 167-177 (2007)
Brooks, A.L., Sharkey, P., Rizzo, A. Merrick, J.: Advances in virtual reality therapy and rehabilitation. International Journal
on Disability and Human Development, 5(3) pp. 203-204 (2006)
Williams, C., Petersson, E., Brooks, A.L.: Picturing Sound : an overview of its efficacy, ArtAbilitation, Aalborg
Universitetsforlag, pp. 69-78 (2006)
Brooks, A.L. Petersson Brooks, E.: Humanics 2 : Human Computer Interaction in Acquired Brain Injury Rehabilitation.
11th International Conference on Human Computer Interaction; the 3rd International Conference on Universal Access in
Human-Computer Interaction, Las Vegas, Nevada, USA, Vol. 8 Lawrence Erlbaum Associates, Incorporated (2005)
49
FP7-ICT-2013-C
SEEDIFF
Brooks, A.L.: Interactive painting. An evolving study to facilitate reduced exclusion from classical music concerts for the
deaf community. International Journal on Disability and Human Development, 4(4) pp. 293-299 (2005)
Brooks, A.L. Petersson Brooks, E.: Play Therapy Utilizing the Sony EyeToy®. Presence 2005: The Eight International
Workshop on Presence. Department of Computer Science, Aalborg University (2005)
Brooks, A.L. Petersson, Brooks, E.: Raw emotional signalling, via expressive behavior. The 15th International Conference
on Artificial Reality and Telexistence ICAT2005, pp. 133-141, (2005)
Brooks, A.L. Petersson Brooks, E.: Recursive Reflection and Learning in Raw Data Video Analysis of Interactive ‘Play’
Environments for Special Needs Health Care. HEALTHCOM2005: 7th International Workshop on Enterprise Networking
and Computing in Healthcare Industry. IEEE Signal Processing Society, pp. 83-87 (2005)
Brooks, A.L. Krüger, V. Guglielmi, M.: A real-time tracking environment towards cross modal applications in public
installations and education. Computers in Art, Design, and Education, København, Denmark (2004)
Brooks, A.L: (1999) Virtual Interactive Space (V.I.S.) : As a movement capture interface tool giving multimedia feedback
for treatment and analysis
Eva selected publications – edit/reduce mine above as necessary
Relative 3rd party publications/acknowledgements –
(1) Tony Brooks Towards New Multisensory Spaces and Environments
http://visualmusic.blogspot.ie/2011/11/tony-brooks-towards-new-multisensory.html?q=Tony+brooks
(2) MULTISENSORY SPACES: THE FOUR SENSES PERFORMANCES WITH A SYMPHONY ORCHESTRA:
A TRANSLATION OF SOUND INTO LIGHT, COLOR, AND SMELL.
http://netzspannung.org/cat/servlet/CatServlet?cmd=netzkollektor&subCommand=showEntry&lang=en&entryId=
74131
(3) TV Documentation – Auckland, New Zealand, http://www.youtube.com/watch?v=gTjvCh-XB2o
50
FP7-ICT-2013-C
SEEDIFF
Participant 3: Ateneum
Main tasks: Leader of WP7 (Dissemination). 1) Organizing art exhibitions for the blind and partially sighted at
the Finnish National Gallery Ateneum. 2) Arranging in-depth visitor surveys, to research the experiences of
persons using tactile and haptic devices in the exhibition. 3) Editing a book/ with tentative title “The Wounded
Angel. Expanding human sensory experience”.
Previous experience: Ateneum, located in the centre of Helsinki, has a long term and vast experience in
organizing art exhibitions. When working with visitor groups in the visual field of an art museum, the museum
has been a pioneer in using stimulation of other senses but vision: hearing, touching and smelling. The Ateneum
team has got plenty of experience in working closely with the blind, e.g., producing the program of 12 art works
in the Ateneum collection in detailed tactile images, combined with an audio (Pictures to Listen); exhibiting
sculptures that visitors can touch in the art museum; and developing audio descriptions for 6 key art works in the
collections of the museum (Treasures of Ateneum).
As the National Gallery of Finland, Ateneum Art Museum houses the largest and most significant collection of
art in Finland.*) The number of annual visitors varies from 200.000 to more than 400.000. In producing
exhibitions, Ateneum is working in an active collaboration with major European art museums, such as the
Picasso Museum in Paris and van Gogh Museum in Amsterdam. In 2000, Ateneum was awarded by Nordic
Council of Ministers for excellent accessibility for the disabled visitors.
The SEEDIFF exhibitions will be on display in an exhibition room in the middle of Ateneum’s collection
premises, where they can be experienced with the majority of museum visitors during their three month’s
duration.
Anja Olavinen is an art historian and the Head of Education in the Finnish National Gallery Ateneum. She has
contributed in many ways in organizing art exhibitions in the museum, and has written and produced several
major educational and other publications, books and films. She's an expert of the entire work of Hugo Simberg,
and has written and edited the catalogue for a comprehensive Simberg exhibition held in Ateneum in 2000. She
will provide her expertise in analysing art as well as in arranging exhibitions for the blind and partially sighted,
and will participate in the transformation of art works into tactile/haptic form together with Natasa Jovicic. With
Erica Othman, she will be organizing in-depth visitor surveys, including interviews and video recordings of the
experiences of the blind and other visitors in the haptic / 6D color exhibitions.
Erica Othman is an art educator, working as the Educational Curator in Ateneum, and responsible for
workshops in the museum. She is an expert in teaching art to a great variety of visitor groups, and has created
“Pictures to Listen”, a program for blind and partially sighted. It was her final thesis in 1987 for the University
of Industrial Arts and Design (nowadays Aalto University). She will be arranging in-depth visitor surveys for the
blind and for sighted people.
Riitta Nousiainen is a Project Manager in the Finnish National Gallery Ateneum, having a central role what
comes to the financial and administrative functions in the museum. She’s a member of the museum’s
management team, and she will be the Adminstrative Contact Person in the SEEDIFF project.
Representative publications
Olavinen, Anja, Paloposki, Hanna-Leena: Hugo Simberg 1873 – 1917. Exhibition catalogue, Helsinki 2000.
Olavinen, Anja: Rikas elämä. Helene Schjerfbeck 1862–1946 [A Rich Life. Helene Schjerfbeck 1862–1946]. E-book,
Helsinki 2012.
51
FP7-ICT-2013-C
SEEDIFF
Olavinen, Anja: Light and Shadow. The Life of Helene Schjerfbeck. A filmed biography of an artist, Helsinki 2012.
52
FP7-ICT-2013-C
SEEDIFF
Participant 4: Senseg
Main tasks: (1) Providing and developing a platform where the touchscreen of a selected portable device is
turned into ‘feel screen’ by adding high fidelity tactile sensations with Senseg’s breakthrough electrostatic tactile
technology. (2) Examining and developing an experience demonstration software for making visual art tangible.
Previous experience: Senseg’s technology has the potential to revolutionize the user experience for touch-based
devices. Senseg’s tactile effects are created with electrostatic ‘vibration’ to convey the sense of physical textures,
edges and contours. This technology adds another sense to the interaction with a touch device. Senseg’s
breakthrough technology has received a number of international recognitions such as being the first Finnish
invention to be named among the top 50 inventions by the TIME Magazine. Senseg has eight granted patents.
Senseg was founded 2006; it has its main offices in Espoo Finland, the other main locations with local staff are
Taipei, Taiwan, Seattle USA, and Tokyo Japan, in total having staff of around 25 persons. Senseg’s main
investor is Ambient Sound Investments, which is a fund by the founders of Skype. Senseg’s business focus has
been in bringing the technology to consumer markets and despite great interest from the academic and the
visually impaired community towards Senseg’s technology, there have been resource limitations to support these
interests. Nonetheless, Senseg has co-operated with Finnish Federation of the Visually Impaired and has also
taken part in Haptic perception and interaction in mobile and multimodal contexts (HAPIMM), which is a joint
research project between University of Tampere, Tampere Univ. of Technology and Stanford University, USA.
As Senseg focuses on large consumer electronics customers, a spinoff company may be established that allows
better approaching non-consumer markets and user experience concepts with a wider approach.
Dr. Ville Mäkinen is a physicist, neuroscientist, and entrepreneur. He founded Senseg in June 2006. Having
created Senseg’s initial customer base in the field of medical technology, Dr. Mäkinen went on to develop the
innovations that underlie Senseg’s solution along with the growing of the company. Dr. Mäkinen led the team
that raised Senseg’s initial funding from Ambient Sound Investments, Avera – the Finnish National Seed Fund,
and the Finnish Funding Agency for Technology and Innovations. Dr. Mäkinen carried out his doctoral studies
(Approved with distinction, 2006) in Helsinki University of Technology in the area of signal processing, medical
physics. Dr. Mäkinen has 23 peer-reviewed articles in the field of brain research, an area that he keenly follows
despite pursuing an entrepreneurial career.
Dr. Jukka Linjama has 15 years of technology development experience, followed by 8 years at Nokia as a
haptics specialist and concept designer. He has contributed to a variety of scientific publications ranging from
acoustics to human-computer interaction and has 10 granted patents to his credit. Dr. Linjama devotes his time to
scientific, industrial and artistic projects. Since joining Senseg in 2007, he has been responsible for the humancentric development of Senseg’s offering, a unique haptic solution for touch interfaces.
Representative publications: Being a company, Senseg is better described by its exceptional media visibility
than by scientific publications. Senseg has been featured or awarded in for example the following media:
53
FP7-ICT-2013-C
SEEDIFF
Participant 5: University of Split (UNIST)
Main tasks: 1) Behavioral studies on the perceptual experience of visual art and when blind subjects use
Senseg’s technology. 2) Data analysis in neuroimaging studies. 3) Reaction-time studies linking behavioral and
neurophysiological data. 4) Addressing acceptance of the created art forms in educational settings.
Previous experience: Coordination of EU-funded and national projects, summer schools, conferences; Vast
experience on behaviour studies; Successful development and production of biomedical equipment in clinical
neurophysiology(http://lahen.org/); long-term cooperation with local Association of Blind through community
oriented medical education projects.
University of Split (established in 1974) is the fastest growing university in Croatia, particularly in research in
physics, medicine, and history and art. It has more than 25000 students at 13 faculties and 4 education centres.
The School is the centre of excellence for evidence-based medicine, with the Croatian Branch of the Cochrane
Collaboration, Croatian Centre for Global Health, and Clinical Hospital Centre as its integral parts. Its most
productive research groups are in cancer biology, applied physiology, and neurophysiology. The School has
several neuroscience laboratories: Human and Experimental Neurophysiology, Speech and Hearing Science,
Physiology, Basic Neuroscience, and Clinical Neuroscience with Sleep Lab.
Researchers at the School are actively participating in numerous national and international projects and
collaborations, funded by, e.g., TEMPUS, FP7, LLP, IPA, and COPE.
Ana Jeroncic is Assistant professor and Vice Chair of Department of Research in Biomedicine and Health at the
School of Medicine. Her research interests are in bio- and neuroinformatics. As a Marie Currie fellow, she used
EEG and computational approaches to develop automatic detection of sleep spindles in infants and adults. She
explored topics from molecular to system levels and acquired wide programming and computational skills. SEAEAS, EU PHARE 2005 project that she coordinated (2008) has been elected a Croatia’s success story.
Andreja Bubic is Assistant Professor at the Faculty of Social Sciences and Humanities at UNIST. She received
her PhD at the Max Planck University for Human Cognitive and Brain Sciences and the University in Leipzig,
where she used fMRI and EEG to study visual perception and the function of the premotor cortex. After a
fellowship in the Laboratory for Multisensory research in Jerusalem and a postdoctoral period in Martinos
Centre in Boston, she now lectures in Split and does research in the field of psychology and cognitive science.
Representative publications of UNIST
Rogic, M., Jeroncic A., Bosnjak, M., Sedlar, A., Hren, D., Deletis, V.: A visual object naming task standardized for the
Croatian language: A tool for research and clinical practice, Behav. Res. Meth. 01/2013 Epub: PMID: 23344740 Jan 2013
Boraska, V., Jeroncic, A., et al.: Genome-wide meta-analysis of common variant differences between men and women.
Human Molecular Genetics, 21(21) 4805-4815 (2012)
Striem-Amit, E., Bubić, A., Amedi, A.: Neurophysiological mechanisms underlying plastic changes and rehabilitation following sensory loss. In: Murray, M.M. Wallace, M.T (Eds.) The Neural Bases of Multisensory Processes. CRC Press (2012)
Bubić, A., von Cramon, D.Y., Schubotz, R.I.: Exploring the detection of associatively novel events using fMRI. Human
Brain Mapping, 32, 370-381 (2011)
Bubic, A., Striem-Amit, E., Amedi, A.: Large-scale brain plasticity following blindness and the use of sensory substitution
devices. In: J. Kaiser and M. Naumer (Eds.) Multisensory Object Perception in the Primate Brain. Springer (2010)
Bubić, A., von Cramon, D.Y., Schubotz, R.I.: Prediction, cognition and the brain. Front. Hum. Neurosci., 4 (2010)
Bubić, A., Bendixen, A., Schubotz, R.I., Jacobsen, T., Schröger, E.: Differences in processing violations of sequential and
feature regularities as revealed by visual event-related brain potentials. Brain Research, 1317, 192-202 (2010)
54
FP7-ICT-2013-C
SEEDIFF
Participant 6: University of Zagreb (UNIZG)
Main tasks: 1) Tactile coding and auditory interpretation of works of art; 2) Behavioral and neurophysiological
studies on the tactile and tactile-auditory perception of the blind and blindfolded participants; 3) Novel time
series and pattern recognition analysis of the neurodynamics multi-sensory integration in blind and sighted.
Previous experience: Coordination/participation in EU-funded and other international and national projects,
organization of conferences, summer schools; strong expertise in the human brain development and
neurophysiological studies using MEG, EEG, MRI and fMRI.
The University of Zagreb is the leading Croatian university encompassing all fields of science and art. UNIZG
is strongly committed to building a modern and innovative university through stimulating research excellence
and facilitating the transfer of knowledge into the business sector. In addition to its traditional strengths in
elementary particle physics, nuclear, and solid state physics, Department of Physics increasingly supports
interdisciplinary research in new materials, nanotechnology, biophysics, and cognitive neurodynamics.
Selma Supek is Assistant Professor of physics and biophysics at the Department of Physics, co-founder and codirector of the first UNIZAG international interdisciplinary postgraduate program in Language Communication
and Cognitive Neuroscience (2001–2004), founder of the Mind and Brain series at the InterUniversity Centre
(IUC), Dubrovnik, Croatia (www.brain.com.hr), and a member of the UNIZAG Coordination for Biomedical
Engineering. Since her doctoral thesis research at Los Alamos National Laboratory her research interest is
related to MEG studies of the human perception and cognition.
Natasa Jovicic is an art historian, director of the Holocaust Museum in Jasenovac, Croatia, with a master degree
in art and multicultural education (Columbia College Chicago). In 2009, she opened one of the first tactile
galleries in Europe within the Modern Gallery in Zagreb, based on her inovative tactile-auditory methodology.
Since then, she organized 12 exhibitions and presented her work in major museums (Ateneum, Georges
Pompidou, Tate Modern, Guggenheim, New York, Art Institute, Chicago).
Ana Susac is Senior Lecturer at the Department of Physics with strong research experience in EEG and MEG
studies of the neurodynamics of face processing and research interest in educational neuroscience including
behavioral and neurophysiological studies. She spent research periods at the BioMag Lab in Helsinki, MEG Lab
in Jena, and postdoctoral research at the University of Oxford.
Davor Horvatic is Assistant Professor at the Department of Physics. He obtained PhD in high energy physics
and acquired wide range of theoretical, statistical, and numerical skills. During last few years, he concentrated
his research in the field of complex systems and time series analysis, particularly related to the physiological
signals. He collaborates with H.E. Stanley’s group at Boston University.
Representative publications of UNIZG
Susac A, Ilmoniemi R, Supek S. Face activated neurodynamic cortical networks, Med. Biol. Eng. Comput. 49, 531-43, 2011.
Sušac, Ana; Ilmoniemi, Risto; Pihko, Elina; Nurminen, Jussi; Supek, Selma. Early dissociation of face and object
processing: A magnetoencephalographic study. Human Brain Mapping. 30 (3): 917-927, 2009.
Supek, S., Aine, C., Ranken D., Best E., Flynn E.R., Wood C.C.: Single vs paired visual stimulation: Superposition of early
neuromagnetic responses and retinotopy in extrastriate cortex in humans, Brain Research, 830: 43-55, 1999.
Aine, C.J., Supek, S., George, J.S., Ranken, D., Lewine, J., Sanders, J., Best, E., Tiee, W., Flynn, E.R., and Wood, C.C.:
Retinotopic organization of human visual cortex: Departures from the classical model. Cerebral Cortex, 6:354-361, 1996.
55
FP7-ICT-2013-C
SEEDIFF
D. Horvatic, H. E. Stanley, and B. Podobnik, "Detrended Cross-Correlation Analysis for Non-Stationary Time Series with
Periodic Trends," Europhys. Lett. (EPL) 94, 18007 (2011).
56
FP7-ICT-2013-C
SEEDIFF
2.3
2. 4 Consortium
as a whole
Describe how the participants collectively constitute a consortium capable of achieving the project objectives,
and how they are suited and are committed to the tasks assigned to them. Show the complementarity between
participants. Explain how the composition of the consortium is well balanced in relation to the objectives of the
project. If appropriate, describe the industrial/commercial involvement to ensure exploitation of the results.
2.4.1 Consortium overview and role of the participants
The consortium has been formed in order to reach the main objectives: to develop new methodology and
technology for expanding senses for blind, color-blind and sighted people.
Senseg and Aalto provide the necessary technical expertise. Drs. Mäkinen and Ilmoniemi are both
physicists and their teams include mathematically highly skilled people with the ability to develop and
implement new algorithms and software for the purposes of the project.
UNIZG and Aalto have world-leading expertise and long-standing collaboration in neurophysiological
experimentation, in particular with regard to studies of vision and other sensory systems.
UNIST and UNIZG have the expertise for studies of the blind … [Please continue]. One of the key people
at UNIZG, the art historian Natasa Jovicic, has worked extensively with the blind, developing combined
haptic relief and audio presentation for the blind. She works constantly with the blind; one of the employees
in SEEDIFF is the soon-to-be psychologist Ivica … who is blind himself and who has been working with
Natasa Jovicic on experimenting with the haptic displays.
UNIST and Aalborg have background in behavioral studies and [please continue]
Ateneum and Aalborg as well as UNIZG provide SEEDIFFs major interface to the public, the museum
exhibits. [CONTINUE]
2.4.2 Complementarity of participants
The project partners and advisors have expertise ranging from information and communication
technologies, physics, psychology, and neuroscience to art history, pedagogy/educational science, and
media technologies.
57
FP7-ICT-2013-C
SEEDIFF
2.4.3 Industrial involvement and exploitation of the results
Assuming that the hexachromatic display technology will attract end-user or industrial interest, Aalto will
aim at commercializing the technology. Any IPR will be protected prior to making the information public.
Aalborg [should write something here about their exploitation of the results]
Ateneum will work together with the other partners in order to explore possibilities for making the
exhibitions for the blind suitable for wider distribution (art museums, science centers).
Senseg or a spin-off company will aim at commercializing the haptic display technology for the blind.
UNIST [should write something here about their exploitation of the results]
UNIZG [should write something here about their exploitation of the results]
Croatian partners are included as Ditacta prototype was developed by Natasa Jovicic’s, Croatian art
historian.
[The different efforts to utilize the results should be integrated]
2.4.4 Subcontracting
Subcontracting: If any part of the work is to be subcontracted by the participant responsible for it, describe the
work involved and explain why a subcontract approach has been chosen for it.
Aalto has budgeted 3000 EUR for auditing costs.
Ateneum has budgeted 8000 EUR for xxx.
Senseg has budgeted 44000 EUR for xxx and 3000 EUR for auditing costs.
UNIZG has budgeted 32000 EUR for xxx.
58
FP7-ICT-2013-C
SEEDIFF
2.5 Resources to be committed
In addition to the costs indicated on form A3 of the proposal, and the staff effort shown in section 1.3 above,
please identify any other major costs (e.g. equipment). Describe how the totality of the necessary resources will
be mobilised, including any resources that will complement the EC contribution. Show how the resources will
be integrated in a coherent way, and show how the overall financial plan for the project is adequate.
(Recommended length for Section 2.4 – two pages)
2.5.1 Use of the resources
The progress of SEEDIFF will be based on technical development work by Senseg (haptic display technology,
WP2) and by Aalto (hexachromatic methodology, WP3). A crucial part of this development will be in userinterface, usability, and image transformation (from visual to haptic, from tri- to hexachromatic) and
presentation software. The development of software and the necessary algorithms will be guided by interviews,
subject observation and behavioral studies (WP4, WP6 and WP7), the methodology of image transformation
(WP4) and neurophysiological studies (WP5). Dissemination of results (WP7) will be arranged so that we can
obtain user experience data that we can use as a guide when we design behavioral and neurophysiological
studies in the laboratory or when we design user interfaces and develop image transformation methodology.
2.5.2 Equipment resources
Aalto will make available for the project its (i) professional equipment for photography as well as (i) MEG and
EEG facilities (at the joint research unit BioMag Laboratory of the Helsinki University Central Hospital) for
neurophysiological studies of visual, tactile, and auditory perception.
Aalborg...
Ateneum...
Senseg...
UNIST...
UNIZG...
2.5.3 Other major financial resources
[If none, we can delete this subsection]
59
FP7-ICT-2013-C
3
3.1
SEEDIFF
Impact
Expected impacts listed in the work programme
Describe how your project will contribute towards the expected impacts listed in the work programme in relation
to the topic or topics in question. Mention the steps that will be needed bring about these impacts. Explain why
this contribution requires a European (rather than a national or local) approach. Indicate how account is taken of
other national or international research activities. Mention any assumptions and external factors that may
determine whether the impacts will be achieved.
3.1.1
Impact on the competitiveness of the proposers
As the National Gallery of Finland, Ateneum is one of the leading actors in the cultural and art field in Finland,
and a leading art museum in the country. Also having being awarded by Nordic Council of Ministers for the
exemplary accessibility as an art museum in 2000, we have the responsibility to stay in the lead of development
in the service of all of our visitors.
Croatian partners are included as Natasa Jovicic (UNIZG) introduced her Ditacta method to the blind public in
Croatia and has already gained some experience and partners in Croatia that are important for the project
implementation.
3.1.1.1
Direct applications and market prospects
3.1.1.2
Potentially patentable ideas
3.1.1.3
Benefits and competitive advantages
Equality is in the core of the strategies of the Finnish National Gallery Ateneum, and reaching that goal means
constant effort and improvement for an institution. To stay in a certain level without improving means declining
and falling, when compared with other actors in the art field in Finland, in Europe, and globally. SEEDIFF
partnership will give Ateneum the keys to outstanding technical and cultural development for our customers’
service in the future.
60
FP7-ICT-2013-C
3.1.1.4
Economic justification
3.1.2
Strategy for impact achievement
3.1.3
European dimension
3.1.3.1
European problem to be solved
3.1.3.2
Effects on transnational co-operation
SEEDIFF
The effects of transnational and also interdiciplinary co-operation are crucial, what comes to the effectiveness of
different partners of the SEEDIFF project. The benefits of the co-operation for them lie in the interchange and
collaboration of actors, who have very different kinds of expertises, but at the same time, goals of mutual, shared
interests. The effects of cooperation of SEEDIFF partners for the European community, on the other hand, lie in
the increase of mutual understanding, and foremost, outcomes of the synergy: in the creating of revolutionary
new technical and cultural solutions, which would not be possible, if the partners would be working alone.
3.1.3.3
Implementation and evolution of EU policies
3.1.3.4
Improvement of European social and economic cohesion
61
FP7-ICT-2013-C
SEEDIFF
3.1.4
Contribution to Community societal objectives
3.1.4.1
Quality of life
potential in SEEDIFF outcomes where the realised displays are applied for adaptive uses across alternative
rehabilitation/therapeutic training situations and end-users. So this can be added as a future research potential.
What comes to the art museums, the blind and partially sighted visitors have hitherto been highly dependant of
the museum personnel. There are few possibilities for them to act in a museum like any other visitor. For
instance touching the sculptures needs consultation with the personnel, who will provide the visitor with gloves
and other equipment. The visually impaired visitor also needs an assistant – she/he typically is not allowed to
touch the art works alone. The availability of haptic displays in museums and elsewhere will give the possibility
for the blind to explore art and other visual information without an assistant. Independency is a basic right and
need of any adult person.
3.1.4.2
Health and safety
For a blind or partially sighted person the possibility to use a haptic display when leaving home, when travelling
and working, and when having free-time and holidays, means an experience of freedom and certainty that has
never been in the reach of these people before. One of the most important features of possible outcomes of the
SEEDIFF project will be the benefits it will bring for the visually impaired of the knowledge of one’s location,
and the possibility to read maps in internet pages, then accessible by means of the haptic display.
3.1.4.3
Employment
3.1.4.4
Gender issues
SEEDIFF is gender neutral, in the sense that both men and women may equally participate in any of its
activities and application of the results.
SEEDIFF outputs will be gender neutral. Both female and male subjects and patients are studied.
The Coordinator and the Steering Committee will oversee and ensure that all Participants are aware of harms
caused by gender bias within the project or elsewhere.
At each meeting, the Steering Committee will make an assessment of the gender balance within the project and,
if needed, will implement actions in order to correct any unjustified changes.
3.1.5
Other relevant European or National funded research
62
FP7-ICT-2013-C
3.1.6
3.2
SEEDIFF
Influence of external factors
Dissemination and/or exploitation of project results, and management of
intellectual property
Describe the measures you propose for the dissemination and/or exploitation of project results, and the
management of knowledge, of intellectual property, and of other innovation related activities arising from the
project.
3.2.1
Exploitation and dissemination plan for use of project results
3.2.1.1
c
3.2.1.2
Validation of the technology
3.2.1.3
Dissemination of results and technology transfer
A special task is devoted to the exploitation of the results. All the Participants of the consortium are involved in
this work package.
Exploitation plans given here will be modified according to the results of the market evaluation performed
during the first months of the contract in order to maximise the exploitation of the results.
63
FP7-ICT-2013-C
SEEDIFF
Haptic systems
Senseg has over the years develop and understanding of the perceptual capabilities of humans in tactile realms.
While user experience enhancements are readily achieved with Sense technology, this does not provide the basis
for directly accessing the needs of the visually impaired. The SEEDIFF project enables developing interactive
approaches for making art tangible, which solves the general limitations of traditional haptic approaches in
tactile displays.
The development and research at SEEDIFF can lead to two alternative paths with their different impacts and
business possibilities.
A) Interactive tactile display technology capable of serving any museum or art gallery in the world specifically
targeted to provide more engaging tangible experience to: visually impaired, children and 3 all people looking
for immersive experience.
B) Applications that can be installed to any smartphone/tablet enabled with Senseg technology or haptic
capabilities in general which allow presenting artwork in tangible format.
Regarding the path A) both the commercial and the social potential for creating tangible art is immense, for
example there are over 1500 museums in the world with over 350 daily visitors (Exhibition and museum
attendance figures 2011 (page 2)" (PDF). London: The Art Newspaper. April 2011). To get scope of this scale
only one museum in Finland makes it to this list, and there over 300 professional museums that are maintained
over the year. Each museum further number of installations therefore there is market opportunity to tens or
hundreds of thousands speciality tactile displays in museums and art galleries. With SEEDIFF project, a
platform and understanding can be build whereby all of these museums could be turned in to places which would
readily provide an experience also for the visually impaired, and would also provide an attractions for the
children. The social impact has viral aspect, taking a key hold in few key museums in the world can lead to
general expectation that museums are in general expected to cater also the visually impaired. This viral aspect
will also create excellent commercial possibilities. Each museum tangible display is custom made can be built to
a large size, eg. size of the original art work when ever feasible, and the experience build together with
exhibitors. This model has excelled fit to an an European business combining expert services and high
technology.
Therefore with the successful completion of SEEDIFF it is conceivable to expect that it there is sufficient
knowledge to start distributing a platform that can lead in few years to situation where it is expected that all high
quality museums also have a tangible displays that make the art accessible also for the visually impaired.
The other alternative path B) is to build software applications based on knowledge learned at SEEDIFF that
allows presenting tangible presentations of art at consumer level devices that have Senseg technology enabled.
The size of the display limits the possibilities of the user experience but would provide the museum indepent
access to visually impaired at convenience without the museums. The experience would be more limited then
case A, but should the Senseg technology be readily available to portable devices at the end of SEEDIFF it
would be natural to also follow this path. The arduous task of finding the representations how to make visual
imagery tangible is one key task of SEEDIFF project that would be the basis for the planned application.
Our dissemination strategy relies on four aspects.
64
FP7-ICT-2013-C
SEEDIFF
1) The publication of scientific breakthroughs in high impact journals, communications at international
conferences, etc.
2) A web-based information data base will be elaborated and maintained during the project containing all
relevant information required for the implementation of the Work Plan, at the pre-authorised various levels of
confidentiality: reference documents (contract, Consortium Agreement), meetings announcement and minutes,
planning advancement, financial data (financial distribution & reporting), publications and reports under
preparation/reviewing. For these reasons, a three-level confidentiality structure will be established: Public
information, Project Participants, and Evaluators (Executive Committee members, finance, reviewing, etc.), all
data being protected by firewalls. This platform will be accessible to all members via the internet allowing realtime communication.
3) Exhibitions
4) Media
5) Finally, a workshop at the end of the project will constitute a major dissemination of results.
3.2.2
Management of knowledge and intellectual property
In this project, patent protection will be sought as new IPR emerges. The ownership of any inventions,
techniques, or methodologies will remain at the inventor organization, which is also responsible for applying for
patent. Regarding exploitation of patents with joint inventors, the distribution of ownership will reflect the level
of research contributions to the patented invention(s) of the different inventors and their laboratories. This will
be decided by mutual agreement under supervision of the SEEDIFF Coordinator.
In addition, a basic training on quality and intellectual property management will be proposed during the first
phase of the program to all the project Participants by a person responsible for patent.
An IPR management plan will be included in the Consortium Agreement. Pre-Existing Knowledge to be
included/excluded in/from the project will be identified and documented in an annex to the Consortium
Agreement.
In particular, the Consortium Agreement will describe how the results of the project may be transferred from the
project or its Participants to the industrial Participant and other parties.
65
FP7-ICT-2013-C
4
SEEDIFF
Ethical Issues
High ethical standards will be followed in all phases and aspects of the project.
Care will be taken to ensure that all human studies are safe for the volunteer subjects, whether blind, partially
sighted or sighted, and that the instruments will be designed, built, and used in conformity with the regulations
of European Union.
We strongly believe that the benefits of the present study will justify the cost and effort of the project.
4.1
Benefits of the present study and experiments
A large number of blind or visually impaired people have no or limited access to xxx. This project will help in
xxx and provide xxx to a large number of people.
For example, ….
4.2
Risks of the proposed approach? [are there any?]
.
4.3
Approvals by ethical committees and informed consent
All human studies within the project will be performed only after appropriate ethical approvals have been
obtained. The ethical committees and regulatory organizations in the Participant countries that will need to be
approached during the life of the project are listed below:
Finland: 1) Coordinating Ethics Committee, Hospital District of Helsinki and Uusimaa, 2) Research Ethics
Committee, Aalto University.
Denmark: ??
Croatia: 1) Research Ethics Committee of University of Split School of Medicine. 2) Ethics Committee, School
of Medicine, University of Zagreb
Each subject (blind, partially sighted, or sighted) to be studied will be informed orally or in writing about the
purpose, execution, possible side effects or risks of the study and about the right to leave the study at any time
without any negative consequences.
Each subject who, after being explained orally or in writing about the study as explained above, wishes to
participate in the study will be asked to sign an informed consent form prior to the beginning of the study. No
subject will be studied without informed consent. Both the written information leaflet informing about the study
and the informed consent form will be the ones approved by the ethical committee in question.
4.4
Data protection issues
All subject or patient data will be coded after the study session so that subject identity cannot be determined
from any of the data files. The data files are labeled and the software identifies the subjects and patients only by
the code, which can be related to the individual only by the investigator in question.
ETHICAL ISSUES TABLE
66
FP7-ICT-2013-C
SEEDIFF
YES
Informed Consent
 Does the proposal involve children?
 Does the proposal involve patients or persons not
able to give consent?
 Does the proposal involve adult healthy volunteers?
 Does the proposal involve Human Genetic Material?
 Does the proposal involve Human biological
samples?
 Does the proposal involve Human data collection
Research on Human embryo/foetus
 Does the proposal involve Human Embryos?
 Does the proposal involve Human Foetal Tissue /
Cells?
 Does the proposal involve Human Embryonic Stem
Cells?
Privacy
 Does the proposal involve processing of genetic
information or personal data (e.g. health, sexual
lifestyle, ethnicity, political opinion, religious or
philosophical conviction)?
 Does the proposal involve tracking the location or
observation of people?
Research on Animals
 Does the proposal involve research on animals?
 Are those animals transgenic small laboratory
animals?
 Are those animals transgenic farm animals?
 Are those animals cloned farm animals?
 Are those animals nonhuman primates?
Research Involving Developing Countries
 Use of local resources (genetic, animal, plant etc)
 Benefit to local community (capacity building i.e.
access to healthcare, education etc)
Dual Use
 Research having direct military application
 Research having the potential for terrorist abuse
I CONFIRM THAT NONE OF THE ABOVE ISSUES
APPLY TO MY PROPOSAL
67
X
X
-
-
-
-
-
PAGE
FP7-ICT-2013-C
SEEDIFF
Comments:- Museum showcase and positing an angle of building on prior work that can
sometimes get some add on value
In section 1.1.1 - you ask which art museums. Maybe this should be at least one in each
partner country and we have a strong cooperation with our regional art museums leadership
board who are keep to work with us and I can approach them to ask about a SEEDIFF
exhibition when appropriate. We also have links to other leading museums in Denmark should
a wider dissemination be required. I see no problems in this to showcase in Denmark.
Also, I feel it appropriate to say that I see this work in a way related to the Neuroesthetic
studies/activities being published (Zeki and others). I published (only a position paper) on
Neuroaesthetic Resonance as my previous research enquired to aesthetic resonance
(involving two EU projects and an i3 future probe (briefly outlined below inc URLs) - and in a
way I see SEEDIFF building on some of the embedded concepts from those projects) =
(1) CARESS - CREATING AESTHETICALLY RESONANT ENVIRONMENTS IN SOUND;
The purpose of CARESS was to create technological and educational tools that will motivate
and empower children to develop creativity, imagination and expression, through interactive
acoustic environments.
(2) CAREHERE - CREATING AESTHETICALLY RESONANT ENVIRONMENTS FOR THE
HANDICAPPED, ELDERLY AND REHABILITATION. The objective of our project is to
empower children and adults with special needs, the elderly in long term care and people
undergoing rehabilitation in hospital or at home, following for example stroke or brain injury.
By giving them access to affordable, appealing and readily usable state of the art technology
for the improvement of their physical and cognitive skills using feedback from acoustic and
visual stimuli. We are concerned with the (re-)development of physical and cognitive skills by
interaction with a responsive sound and visual environment: the improvement of motor control
through direct and immediate feedback through the aural and visual senses.
(3) - The i3.net future probe Twi-aysi = The World is as you see it. While CARESS
successfully motivated and empowered children to develop creativity, imagination and
expression, through interactive acoustic environments. The objective of Twi-aysi was to
answer the question: Can immersion in a visual environment hold similar potential for such
children in terms of the aesthetic resonance they might derive from movement within such a
visual space? The success of Twi-aysi led to the CAREHERE project (above).
INSTRUCTIONS
68
FP7-ICT-2013-C
SEEDIFF
Section 1 (20 pages, not including the tables ): Explain the concept of your project. What are the main ideas that
led you to propose this work? Describe in detail the S&T objectives. Show how they relate to the topics
addressed by the call. The objectives should be those achievable within the project, not through subsequent
development. They should be stated in a measurable and verifiable form, including through the milestones that
will be indicated under section 1.3 below.
R = Report, P = Prototype, D = Demonstrator, O = Other
Please indicate the dissemination level using one of the following codes:
PU = Public
PP = Restricted to other programme participants (including the Commission Services).
RE = Restricted to a group specified by the consortium (including the Commission
Services).
CO = Confidential, only for members of the consortium (including the Commission
Services).
Delivery date
Measured in months from the project start date (month 1).
69
FP7-ICT-2013-C
SEEDIFF
Section 1.2. Describe the state-of-the-art in the area concerned, and the advance that the proposed project
would bring about. If applicable, refer to the results of any patent search you might have carried out.
Section 1.3. A detailed work plan should be presented, broken down into work packages* (WPs) which should
follow the logical phases of the implementation of the project, and include consortium management and
assessment of progress and results. (Please note that your overall approach to management will be described
later, in section 2).
Please present your plans as follows:
i) Describe the overall strategy of the work plan.
ii) Show the timing of the different WPs and their components (Gantt chart or similar).
iii) Provide a detailed work description broken down into work packages:
- Work package list (please use table 1.3a);
- Deliverables list (please use table 1.3b);
- Description of each work package, and summary (please use table 1.3c)
- Summary effort table (please use table 1.3d)
- List of (please use table 1.3e)
iv) Provide a graphical presentation of the components showing their interdependencies (Pert diagram or
similar)
Note:
The number of work packages used must be appropriate to the complexity of the work and the overall value of
the proposed project. The planning should be sufficiently detailed to justify the proposed effort and allow
progress monitoring by the Commission.
Any significant risks should be identified, and contingency plans described.
* A work package is a major sub-division of the proposed project with a verifiable end-point - normally a
deliverable or a milestone in the overall project.
Section 3 (Recommended length for the whole of Section 3 – ten pages)
Sectdion 4. Describe any ethical issues that may arise in the project. In particular, you should explain the benefit
and burden of the experiments and the effects it may have on the research subject. Identify the countries where
research will be undertaken and which ethical committees and regulatory organisations will need to be
approached during the life of the project.
70