FP7-ICT-2013-C SEEDIFF Small or medium-scale focused research project (STREP) Full proposal ICT FET Open Call FP7-ICT-2013-C Seeing Differently SEEDIFF Date of preparation: April 15, 2013 Version number: 31 Name of short proposal this full proposal refers to: Seeing Differently Type of funding scheme: CP-FP-INFSO Work programme topics addressed: ICT-2013.9.1 Name of the coordinating person: Risto Ilmoniemi List of participants: Participant no. Participant organisation name Part. short name Country 1 (Coordinator) AALTO-KORKEAKOULUSAATIO Aalto Finland 2 AALBORG UNIVERSITET Aalborg Denmark 3 Valtion taidemuseo Ateneum Finland 4 Senseg Oy Senseg Finland 5 SVEUCILISTE U SPLITU (UNIVERSITY OF SPLIT) UNIST Croatia 6 FACULTY OF SCIENCE UNIVERSITY OF ZAGREB UNIZG Croatia 1 FP7-ICT-2013-C SEEDIFF Abstract SEEDIFF is a ‘neuro-ICT–art’ project that will enhance human sensory experiences through developing and applying a new generation of mobile “display” solutions to induce alternative channelling of stimuli that expand or enhance senses. Two products, an affordable haptic display and an innovative 3-D spectral display, target three distinct outcomes to enable “seeing” by the blind/partially-sighted, improved discernment for the color blind, and augmented perception for the sighted. Transforming visual content of natural and man-made scenes, including masterpieces of art, into perceivable tactile-auditory realm will result in innovative experiences and a foundational step toward augmenting digital inclusion for those with visual dysfunction. The vision of the project is enhanced experiences of cultural and civilizational achievements, such as visual arts, augmented and enriched communication, and increased educational opportunities. Table of Contents 1 SCIENTIFIC AND/OR TECHNICAL QUALITY, RELEVANT TO THE TOPICS ADDRESSED BY THE CALL 4 1.1 Targeted breakthrough and long-term vision 1.1.1 Concept and vision 1.1.2 Targeted breakthrough 1.1.3 S&T objectives 4 4 6 8 1.2 Novelty and foundational character 1.2.1 State-of-the-art 1.2.2 Technical limitations of existing products, processes and/or services 1.2.3 Main innovations by SEEDIFF participants 1.2.4 Patent search results 9 9 17 18 18 1.3 S/T methodology 1.3.1 Overall strategy of the work plan 1.3.2 Work description 19 22 23 2 IMPLEMENTATION 41 2.1 Management structure and procedures 2.1.1 Management capability of the co-ordinator 2.1.2 Management structure and decision-making structure Figure 1. SEEDIFF management structure Coordinator Steering Committee Work Package Leaders / Group Leaders Administrative Contact Persons 2.1.3 Monitoring, reporting progress and documenting results Project evaluation and other meetings 41 41 41 42 42 43 43 44 44 44 2 FP7-ICT-2013-C 2.2 SEEDIFF Resolution of conflicts [to be revised] Risk assessment and management 2.1.4 Meeting agendas 45 46 46 Individual participants Participant 1: Aalto Participant 2: Aalborg http://visualmusic.blogspot.ie/2011/11/tony-brooks-towards-newmultisensory.html?q=Tony+brooks Participant 4: Senseg 2.2.1 Participant 5: University of Split (UNIST) 2.2.2 Participant 6: University of Zagreb (UNIZG) 47 47 48 50 53 54 55 Consortium as a whole 2.2.3 Consortium overview and role of the participants 2.2.4 Complementarity of participants 2.2.5 Industrial involvement and exploitation of the results 2.2.6 Subcontracting 57 57 57 58 58 2.3 Resources to be committed 2.3.1 Use of the resources 2.3.2 Equipment resources 2.3.3 Other major financial resources 59 59 59 59 3 IMPACT 60 3.1 Expected impacts listed in the work programme 3.1.1 Impact on the competitiveness of the proposers 3.1.2 Strategy for impact achievement 3.1.3 European dimension 3.1.4 Contribution to Community societal objectives 3.1.5 Other relevant European or National funded research 3.1.6 Influence of external factors 60 60 61 61 62 62 63 3.2 Dissemination and/or exploitation of project results, and management of intellectual property 3.2.1 Exploitation and dissemination plan for use of project results 3.2.2 Management of knowledge and intellectual property 63 63 65 4 ETHICAL ISSUES 66 4.1 Benefits of the present study and experiments 66 4.2 Risks of the proposed approach? [are there any?] 66 4.3 Approvals by ethical committees and informed consent 66 4.4 Data protection issues 66 3 FP7-ICT-2013-C 1 SEEDIFF Scientific and/or technical quality, relevant to the topics addressed by the call 1.1 Targeted breakthrough and long-term vision 1.1.1 Concept and vision The SEEDIFF project will develop and apply new technologies and methods to expand human sensory experiences in the visual domain. Advancements through the work will impact blind, partially sighted, color-blind, and sighted end-users. Specifically, our long-term vision involves two distinct neuro-ICT–art convergences and synergies: I) The first of these goals is that blind and partially sighted end-users will be empowered to interact with visuo-spatial real and virtual worlds by means of an affordable new generation of haptic1 display technology that will set the foundation for enhancing their digital inclusion. II) The second goal is that color-blind end-users will be able to discriminate colors via a process that enhances color perception from dichromatic to tetrachromatic2. Utilising the same process, our third goal is that end-users with intact color vision (hereafter termed ‘the sighted’) will be able to access enhanced color vision through improved color perception augmented from trichromatic to hexachromatic. This will be achieved via innovative 3D-display technology that doubles the number of distinguishable chromatic bands of the color spectrum. Paving the way to our long-term vision will require not only innovative technologies, but also an understanding of the psychophysical and brain mechanisms of sensory information processing, multisensory integration and convergence as well as brain’s sensory learning, memory, and plasticity. Fig. 1. Converting and expanding senses: from visual to tactile or haptic. The idea is in representing art by transforming an original painting to a set a representations that are suitable for tactile and, with SEEDIFF, haptic exploration. Left: Original painting (Wounded Angel by H. Simberg). Centre: One of the representations of the scene (there are typically set of representations to code for the entire visual scene, relevant scene details, perspective, light, and color content). Right: The same scene to be represented on a haptic display during the project using Senseg’s technology, allowing one to feel both lines and textures of the image as if the screen would have elevations (e.g., the black parts) and different textures. 1 Haptic = relating to the sense of touch; Tactile = perceptible by touch; Di-, tri-, tetra- or hexachromatic = relating to vision involving 2, 3, 4 or 6 sensory receptor types with different wavelength sensitivities, respectively; phosphene = a sensation of seeing without retinal illumination. 4 FP7-ICT-2013-C SEEDIFF A B C Fig. 2. Expanding senses: from trichromacy to hexachromacy (or dichromacy to tetrachromacy). The basic idea is that images are viewed stereoscopically, but instead of (or in addition to) presenting depth information, the two eyes are used independently to enrich color perception. The left eye receives one coloring and the right eye another coloring of the same scene. Thus, the left eye is presented with three wavelength bands from the original scene while the right eye is presented with three other bands. This is illustrated via images in panels A, B, and C: A) A possible division of the visual spectrum into 6 wavelength bands (horizontal scale: wavelength in nanometres). Hexachromatic vision is obtained if the shaded bands are presented to the left eye and the non-shaded bands to the right eye. Superimposed, the sensitivity functions of our three cone types are shown in the same image (normalized). B) Abstract art (Johannes Itten) can also be expanded to the new color format; normally, a 3D viewing system, such as one in modern TVs, is needed for looking at the hexachromatic image. C) From dichromacy (color blindness) to tetrachromacy. The left image shows how an idealized red–green color blind sees the red berries in essentially the same way as the green leaves. The two right images show how the original trichromatic image can be seen by the color blind when green is emphasized on the left figure and red is emphasized in the right image. When viewed stereoscopically, the berries can be distinguished from the leaves not only by their shapes but also by color, as a combination of dark blue (centre image) and light green (right image). (Frames A and C are from the special assignment of Lari Koponen, Aalto Univ., unpublished; original images at seediff.tumblr.com). 5 FP7-ICT-2013-C 1.1.2 SEEDIFF Targeted breakthrough Our intended breakthrough is threefold. First, we plan to develop a novel, advanced, haptic-display platform and tools for mapping visual/spatial information into tactile–auditory modality that will allow interactions, including haptic navigation in visually based scenes and maps. This allows turning the wealth of visual content of natural scenes, images or graphical objects, including masterpieces of art, into the tactile–auditory realm to be experienced by partially sighted and blind people. Second, we plan to develop methodology for producing and displaying hexachromatic images and video. Dividing the light spectrum into 6 bands (Fig. 2A) we will present alternating bands to the left eye whilst synchronous corresponding alternating bands are presented to the right eye via standard 3D display technology. This will enable the color blind to experience all standard colors (albeit differently from how they are traditionally sensed), and allow the sighted to expand color vision from trichromacy to hexachromacy. Expected outcomes include augmenting life quality and day-to-day living for color blind as well as offering new perceptual experiences for sighted and new opportunities and challenges to artists, researchers, and media professionals. Third, through behavioral and neurophysiological studies of the multi-modal and multi-spectral sensory expansion provided by our new technological platforms, we plan to obtain fresh insights into the ability of the brain to adapt to novel and enriched sensory inputs, thus enabling its ability to form previously inexperienced representations of the external world and stimulating its plasticity. These insights will contribute to the foundations of sensory expansion technology and will help us in integrating and optimizing tactile and auditory coding of visual information and guide us in developing a breakthrough in hexachromatic display technology. The relevance of our project stems from the fact that vision is the sense that most directly defines our relationship with our environmental and social surroundings. When awake with eyes open, we constantly maintain a representation of space around us based on what we see. Approximately half of our cortex, with tens of distinct visual areas, is devoted to vision. Unfortunately, our visual machinery may be compromised by genetic defects, disease, aging or accidents. There are close to 300 million blind or partially sighted people in the world; they lack direct access not only to everyday visual scenes and objects that aid and guide our interactions with the environment, but also to abstract cultural and civilizational achievements such as visual arts or even educational opportunities. In particular, digital exclusion of blind and partially sighted people is still growing, thus our venture is timely and fitting considering recent EU actions and priorities (Post-i2010: priorities for new strategy for European information society (2010-2015), European Blind Union [EBU] Position Paper2). Included in such prioritized action is the EBU Strategic Plan 2011-2015: “To map current levels of good practice in giving blind and partially sighted people access to arts and to cultural events and facilities to provide the basis for recommendations and guidelines that will serve as a tool for advocacy and lobbying.” In stating the case of low levels of accessibility in museums and places of cultural interest to blind and partially sighted people, Carol Borowski, previous chair of the EBU’s Culture and Education Commission posited that “We invite the European Commission to energetically promote the cultural rights of disabled people in its new Disability Strategy 2020. Equal access to museums and places of cultural interest is a right of visually impaired people, not a luxury.” 2 http://ec.europa.eu/information_society/eeurope/i2010/docs/post_i2010/additional_contributions/ebu.pdf 6 FP7-ICT-2013-C SEEDIFF Ms. Borowski continued, “Of course, there is good practice, but it is exceedingly rare. Across Europe, we need significantly more opportunities for visually impaired people to experience art and culture through audio description, authentic objects for handling, tactile models and tactile images. We need to exploit the empowering potential of new technology. Exhibition design which is inclusive of the needs of disabled people has to become the norm, not the exception, as it is today.”3 Such statements indicate the hope and anticipation of communities of blind and partially sighted individuals of how collateral consequences may be alleviated with tailored ICT and assistance. Such expectations are supported by structural and functional neuroimaging studies that have demonstrated multi-sensory perceptual mechanisms underlying cortical plasticity and reorganization. These studies confirm the involvement of most visual areas, including the primary visual cortex, in processing information from other sensory modalities, thus supporting our approach of a multi-sensory input based restoration and/or expansion of human visual perception. Consequently, our aim is to use our new technologies and ideas in combination with advanced neurodynamic brain imaging methods to design and tune tactile/haptic–auditory stimuli to demonstrate that the blind are capable of an enriched perceptual capacity and to provide them an opportunity to get a glimpse of what sighted experience when they observe visual scenes or pieces of art. The new technology will offer new creative possibilities for the blind such as artistic expressions and interactive haptic-auditory communications, including games. SEEDIFF also aims at expanding the awareness of the majority of society to the condition of our fellow citizens such as the blind, the deaf, and other people with sensory limitations. Therefore, we will organize exhibitions in three major art museums (Finnish National Gallery Ateneum, Helsinki; Modern Gallery, Zagreb; Sydvestjyske Museer, Ribe, Denmark), in which tactile–auditory stands and prototype haptic– auditory displays will be placed next to the original paintings and made available to all visitors.4 Although we will focus on making new mobile technologies and visual art available and affordable to the blind, the concept is applicable more generally, suggesting wide applications also for sighted people targeting augmenting sensitization. The new technologies may be used in daily life, under extreme environmental conditions in which visual inputs are minimal or not available, in training and learning events to expand sensory perception, stimulating creativity and imagination by enriched perceptual processing and programmable art, or during rehabilitation to restore or compensate for a temporal loss or a profound sensory deprivation, respectively by enhancing multisensory cortical interactions. To summarize, the intended breakthrough is to establish methodology that enables the blind to experience visuo-spatial scenes using novel haptic touch screen technology, the color blind to discriminate colors they could not discriminate before, and the sighted to expand their perception of objects with new color and tactile form. We expect to open also a new line of behavioral and neuroscientific research, which, in turn, will support the realization of our long-term vision where the blind (especially if they learn to use the technology already in childhood) can “see” by means of affordable “feelable” haptic display technology and could thereby be included in the digital world. Furthermore, possibilities for using color would be expanded, giving new opportunities for artists and for the media as well as consumers. 3 http://www.euroblind.org/media/projects/EBU-Culture-Education_plan-of-action-2011-2015.doc 4 We plan to include blind guides in a similar way as in the exhibition Dialogue in the Dark (www.dialogue-in-thedark.com), which had psychological, social, and practical/economic implications/outcomes. An astonishing 100% of sighted visitors who were questioned five years later remembered the experience, 90% reported feeling sensitized to the world of the blind, while 52% recommended Dialogue in the Dark to their friends and family. 7 FP7-ICT-2013-C 1.1.3 SEEDIFF S&T objectives Our objective is to expand human sensory experience by introducing and developing new haptic and color display technologies for blind, partially sighted, color-blind, and sighted people. Our vision and aims can be described as follows: New Display Technologies 1) SEEDIFF will develop affordable haptic-display platforms specifically aimed at allowing the blind to better cope in a world built for those with intact vision. The emergence of such a new haptic display technology will present new opportunities beyond our goals that focus on innovative representations of art and a mobile-device user interface for the blind. 2) Methods to interpret and map visual scenes, including visual art, into tactile form will be developed. This complex mapping will be achieved as a stepped approach: initially a designer is guided to convert visual images into tactile representation using manual processing; in our future vision, the process will be automated. 3) The expansion of visual experience in color blind as well as in people with intact color vision will be achieved by widening the brain’s sense of color with 3D display technology that will present differently colored images to each eye. Seeing Differently 4) By developing the presentation of visual art to the blind, SEEDIFF will, while expanding the sensory– artistic experience, also evolve a new art form, both for the blind and for the sighted population. 5) Behavioral and dynamic neuroimaging studies will explore the aspect of visual experience enabled by tactile/haptic display and/or auditory guidance. 6) Human sensory perception and plasticity will be investigated via neurodynamic functional brain imaging methods aimed at exploring uni- and multi-sensory processing of the tactile/haptic and auditory representations of visual objects and scenes including visual art using spatio-temporal source localization methods and novel approaches to study the dynamics of cross-modal integration. Science of Art and Color 7) The science of color and art will be enhanced via SEEDIFF, as will our understanding and appreciation of what color really is and how our color vision depends on the fact that we have trichromacy, i.e., that each color is formed as a mixture of the activation of three types of light-sensing cells in the retina of the eye. Our experimentation with hexachromacy is expected to introduce new aspects to color science and strengthen non-linear relationship between stimulus intensity and brightness, called the Helmholtz– Kohlrausch (HK) effect, in respect of the color blind. Education and Public Awareness 8) The teaching of art, art history, and haptic–visual communication for the blind will be advanced via SEEDIFF. The blind will be able to draw images with the haptic displays and communicate with other blind people using haptic displays; we will also explore haptic–auditory games for the blind, which is a largely unexplored territory and would open huge possibilities regarding the inclusion of the blind in the culture of games. In addition, the technology will also be used to enhance the human–machine communication interface for the blind and allow novel rehabilitation options for blind and sighted. Thus, the societal implications are wide, which will result in high attention from the media. The objectives are described in more detail in Section 1.3.3.3 (Work package descriptions). The measurable outcomes are the deliverables. 8 FP7-ICT-2013-C 1.2 SEEDIFF Novelty and foundational character SEEDIFF will develop novel tools for tactile/haptic visual-scene presentation to the blind. Our new haptic display technology will revolutionize the user interface of mobile phones or tablets for the blind, allowing them to browse the web in haptic form. Also, with this technology, it will become possible to present art and other visual scenes to the blind via the Internet, so as to bring these experiences to millions of people rather than to only a handful. Furthermore, we will use standard 3D display technology to expand the sensation of color, which is anticipated to lead to novel types of color content in movies and in photography, helping the color blind as well people with full color vision to enrich their visual experience when watching TV or other media. SEEDIFF will use dynamic neuroimaging and advanced localization, time series correlation, and pattern recognition analysis techniques to demonstrate effectiveness of its approach using multi-sensory stimulation to expand sensory experience taking advantage of innate multi-sensory capabilities of the human brain. SEEDIFF’s behavioral and EEG/MEG studies will advance current understanding of tactile/haptic and auditory stimulation enabled or enriched visual experience and provide novel insights into evidences and mechanisms of multi-modal integration and cross-sensory plasticity in particular related to the 2-D representations of 3-D visual world, including visual art. 1.2.1 State-of-the-art In presenting the state of the art relevant for this project, it is important to review several important themes: the solutions that aid the blind in their day-to-day encounters with objects that surround them, the more technologically advanced tactile displays that have recently been developed for tablets and mobile phones, equally novel solutions for binocular displays that will serve as the basis for the planned color explorations, cross-modality mappings and art experience/perception, current access that the blind have in relation to complex and “non-utilitarian” products such as art as well as the latest research results regarding neuroimaging evidences on multi-sensory processing and cross-modality plasticity of the human brain. State-of-the-art of visual-restoration or visual-substitution aids Numerous approaches and potential solutions aimed at overcoming the difficulties that the blind constantly face have been proposed, such as reading and mobility aids, more advanced sensory substitution devices (SSDs) and invasive sensory restoration and neuroprosthetic approaches. Sensory substitution refers to the transformation of the characteristics of one sensory modality into the stimuli of another modality and in the case of blindness, SSDs represent a non-invasive rehabilitation approach within which visual information is captured by an external device such as a video camera and communicated to the blind via a machine–human interface in the form of typically auditory or tactile input. Specifically, the first targeted modality for substituting vision was touch, due to the ease of transforming visual into tactile signals as these are both characterized by 2-D spatial representations. Paul Bach-y-Rita who devised a tactile display, which mapped images from a video camera to a vibro-tactile device worn on the subject’s back, did pioneering work in this field in the 1970s. Later, smaller tactile devices placed on the tongue and foreheads of blind individuals characterized by better spatial somatosensory resolution were developed. The development of both tactile and auditory-based devices was triggered by limitations of tactile only SSDs, They can be useful in allowing the blind to learn how to interpret and use the transmitted information in simple visual discrimination and recognition tasks as well as in more demanding contexts that require constructing mental images of more complex environments. 9 FP7-ICT-2013-C SEEDIFF Nevertheless, such devices have shown limited success among blind individuals in allowing them to “see with the skin”5 or to “see with the brain”6 or to “see with the ears”7. The level of performance achieved through any of these SSDs remains, however, inferior to any of the perceptual standards as sensory substitution is slower, less accurate, and inferior in terms of automaticity and effort. Noticeably, compared to visual perception, the number of objects that can be jointly accessed or available through sensory substitution devices seems to be much lower as users of SSDs have not been shown to be able to track or identify multiple objects at once. SSD researchers justify such observations by either a suboptimal resolution of their devices, or suboptimal training of participants. According to Loomis et al 2010, however, spatial resolution of the visual is much higher than for the tactile perception so it might be impossible for touch to ever substitute for vision in complex real-world situations. Therefore, while sensory substitution allows the blind to perform many functions, this functionality is generally limited and it is not clear whether it will ever be accomplished, as later described in Section 1.2.2. Given their limitations, some argue for the approach of restoring capability of a visual input to the blind as it represents an alternative to sensory substitution devices. There are four main types of approaches in the visual input restoration, targeting the retina, optic nerve, lateral geniculate nucleus (LGN), and the visual cortex. Devices based on these approaches have so far shown some promising results, as experienced blind users can utilize visual phosphenes generated by some of these devices to create meaningful visual percepts. Corneal transplant, for example, restored the brain’s ability to absorb normal visual input8 but it allowed only perception of simple movements, colors and shapes. In this project we propose an alternative approach to building a device for the blind. The approach is based on multisensory and cross-modal stimulations which utilize brain’s innate capability and plasticity related to crossmodal transfers and integration of sensory inputs - such as the one that exists between the visual and tactile identification of shapes, or between visual and tactile motion. State-of-the-art of tactile displays for tablets and mobile phones Today the concept of tactile display usually refers to a Braille display, but it may also refer to a touch screen with tactile feedback functionality. The name of Senseg is often referred to in the latter context by being the most promising enabling technology. Braille devices are naturally only equipped to provide text content in refreshable Braille format and do not lend themselves for art presentation; therefore, the focus here is in the tactile touch-screen category. Tactile displays in the context of touch screens take advantage of the input functionality of the screen. The finger location on the screen is identified; when a finger is over an element of interest, the device provides a tactile feedback controlled by the host device. In theory, accurately timed tactile events based on finger location could allow the user to feel the contours of a graphics object on the screen. Unfortunately, currently available tactile technologies have severe limitations for creating this illusion. The main technology available for providing tactile feedback is the vibrating motor. This is available in almost every smartphone and tablet device, and while it is suitable for providing alarm vibration or even vibrating the device for confirming, e.g., a key press, it cannot really provide tactile dislay illusion. The effect is non-localized and also the frequency response of the motor is unsuitable. Some more advanced 5 White, B.W., Saunders, F.A., Scadden, L. et al: Seeing with the skin, Percept. Psychophys.7, 23–27. (1970) Bach-y-Rita,P.,Tyler,M.E.,and Kacz-marek,K.A: Seeing with the brain. Int.J.Hum.Comput.Interact. 2, 285–295. (2003) 7 Levy-Tzedek, S., Hanassy, S., Abboud S., et. al: Fast, accurate reaching movements with a visual-to-auditory sensory substitution device. Restor Neurol Neurosci.30, 313-323. (2012) 8 Fine et al., Long-term deprivation affects visual perception and cortex, Nat. Naurosci. 6, 915-916, 2003 6 10 FP7-ICT-2013-C SEEDIFF actuator solutions have also been produced such as piezo or electroactive polymers, which can have more favorable frequency responsies, but they produce mechanical movement of the whole device, which does not yield the localized sensation to the finger, and are therefore more suited to gaming-like events and cannot create texture-type sensations, which would be essential for describing visual images. Senseg technology, on the other hand, is specifically suited for providing texture and surface information rather than shaking the whole device. The basic principle of the technology is modulation of the electric force between a charged conductive layer within the screen and the adjacent body part. While it is possible to generate perceivable vibrations to a stationary finger, the much more pronounced effect for the Senseg technology is the modulation of the surface friction force when sliding the finger along the surface. With accurate control and timing of these modulations, a great number of different touch sensations and textures can be generated, and when combined with matching graphics, this provides the basis for a strong illusion of a ‘’feelable’’ screen. Our plan is to take the tactile display technology to the next level. Humans have limited capabilities in perceiving the world via tactile information, especially with surface information alone (i.e., compared to grasping an object). When tactile information is static, it is limited in scope and weak in experienced immersion. For example, it is difficult to determine the shape of an object but it will be even much more difficult to determine information from complex and detailed pieces of art. Hence, SEEDIFF plans to make groundbreaking work that uses intercative concepts to allow users to have more immersive experience. Auditory stimulation and narratives will be included to complement the haptic information to guide the blind user during the use of the device and augment sensory perception. The interactive concepts may be expanded to vision as well.....?????? State-of-the-art of hexachromatic displays The idea of hexachromatic vision was conceived independently by at least two people in the early 1990's, by the project coordinator (who proposed the idea to the Foundation of Finnish Inventions) and earlier by Raphael Levien (US patent No. 5,218,386, June 8,1993). The idea at that time was to build spectacles with different comb filters for the left and right eyes so that alternate wavelength bands would be seen in the different eyes. The SEEDIFF coordinator experimented with such glasses and observed new color phenomena, but such eye glasses did not seem practical at the time. It seems (based on Google searches) that Levien has not either continued on this line. Our renewed interest in hexachromacy stems from recent developments in 3D displays (TV displays, Cinema, game consoles, mobile phones), which can be used to display different images to the different eyes. Thus, hexachromatic content can be presented without any extra cost or conscious effort by the user of 3D display devices. We only need to develop ways to provide the content. Multi-spectral imaging was originally developed for imaging from satellites. It is being used in mapping properties of cultivated land, forests or other target areas by taking up to several tens of differently filtered narrow-band photographs from an airplane, balloon, or satellite. Multi-spectral images are usually analyzed by computer; their analysis visually would be very difficult if not impossible. But the principle of multispectral imaging is the same as in hexachromatic photography: getting more spectral information from the scene than is possible with standard trichromatic photography. To our knowledge, the idea of using 3D display technology in order to achieve hexachromatic vision in normally sighted people and tetrachromatic vision in dichromatic color-blind people is new. However, scientific experiments have been performed to study binocular rivalry, which is a phenomenon where conscious perception alternates between different images presented to the different eyes. Some of the 11 FP7-ICT-2013-C SEEDIFF experimentation has been performed with color, but to our knowledge, the idea of expanding vision to hexachromacy has not been an issue in these considerations. State-of-the-art cross-modality mappings and art experience/perception The state of the art of cross-modality mappings9 within contemporary art to permit a common ‘grammar’ as well as accessible, verifiable, and robust is limited and inconclusive. Auditory–color bidirectional mappings can be frustrating for artists and scientists who wish to successfully express themselves beyond a single sensory modality. A similar “mapping” challenge is apparent within SEEDIFF. Thus, by approaching this complex challenge in collaboration with brain imaging experts, we will determine the affect of multisensory stimulation to augment perception, and maximise potential of creating a new art form by defining a uniform grammar to communicate the art message and feeling of beauty. To illustrate the mapping challenge, a brief history of auditory–color mapping is presented. Sir Isaac Newton in 1704 proposed correspondences between the proportionate widths of the seven prismatic rays of the color spectrum matched to string lengths required for producing a musical scale. This has been criticized as unscientific with ungrounded mappings10. Accordingly, since Sir Isaac’s era, numerous color-to-musical scale mappings have been posited illustrating complexities involved in establishing a meaningful mapping language between musical tones and the color spectrum (Fig. 3). Fig. 3. Three Centuries of Color Scales © Fred Callopy RhythmicLight.com Analogizing beyond auditory–color mappings, e.g., sound frequency to light wavelength, additional sensory modalities integrated in a meaningful way toward a total art form was the core of German Romanticism and Richard Wagner’s Gesamtkunstwerk. This is where synergy, integration, and synthesis of different stimuli target an ultimate, all-encompassing experience. Such experience has been an on-going challenge for artists to achieve. However, whether this has been reached is questionable as artists continue to explore technological advances to further their art. In line with this desire for an integrated form, artists and 9 Here, a mapping means the manner in which a feature in one sensory modality corresponds to a feature in another modality. Examples of the mapping of color to musical scale are presented in Fig. 3. 10 Gerstner, K.: The Forms of Color: The Interaction of Visual Elements , Cambridge, MA: MIT press (1986) 12 FP7-ICT-2013-C SEEDIFF scientists are increasingly enquiring into the phenomenon synaesthesia11. This is a condition that occurs when an individual who receives a stimulus in one sense modality simultaneously experiences an internally generated sensation in another . This condition is a phenomenon seemingly at odds with the common-sense worldview of five separate senses that channel impressions to our minds. Intrigued by this apparent anomaly of the senses, artists, psychologists and other scientists have explored and experimented with synaesthesia in order to reveal how the senses are interrelated to realise new art. The unusual approach taken in SEEDIFF is to attempt to understand how this interrelationship can realise new art forms by learning from people with sensibilities that are heightened through having an impairment that necessitates training of alternative sensing means to a level for increased survival and improved life quality. We will focus on heightened nuances of visual and tactile senses to achieve our objectives of offering new artistic experiences and opportunities for blind and visually impaired. [This section can be streamlined] State-of-the-art of the visual art creation and access to the visual art of the blinds One may wonder what a blind person wants from art. Dr. Betsy Zaborowski, former Director of the National Federation of the Blind's Jernigan Institute explained how blind people are as different from each other as sighted people–some are even art experts who have studied specific kinds arts that they go to experience, some are history buffs, some may just be tourists….12 Questioning how blind artists paint it is stated that only around 10% of all people with blindness can see absolutely nothing at all. As such most blind people can in fact perceive some level of light and form, and it is by applying this limited vision that many blind artists create intelligible art. Also, a blind person may once have been fully sighted and yet simply lost part of their vision through injury or illness. Blind artists are able to offer insight into the study of blindness and the ways in which the blind can perceive art, in order to better improve art education for the visually impaired. However, in discussing the connection between sensory impairments and art, the phenomenon of “blind artists” is just a tip of the iceberg. Artistically gifted or not, the blind are just like the sighted: they have a certain amount of interest in arts, visual included, and should have access to such products throughout their lives. Unfortunately, at this point in time, this access is very limited. There are some books that offer insights into this field, such as Elisabeth Salzhauer Axel’s book Art beyond sight: a resource guide to art, creativity, and visual impairment13 and the companion video titled Art Beyond Sight. A Demonstration of Practical Techniques for Teaching Art to People with Visual Impairments. Also, there are activities such as the Art Beyond Sight Collaborative and its annual Awareness Month, which is a series of exhibits, demonstrations, conferences, telephone and online discussion groups aiming at raising awareness and bringing together professionals, educators, researchers, the media, sighted and blind artists, and art lovers from around the world. Thus, Awareness Month offers further opportunities for SEEDIFF dissemination and impact. In addition, Art History Through Touch and Sound: A Multisensory Guide for the Blind and Visually Impaired is a series of six multimedia volumes co-published by the American Printing House for the Blind and Art Education for the Blind. It is an art history series for people who are blind or visually impaired that is the result of nine years of research, development, and testing. The series spans the history of art, from prehistoric through contemporary, guiding the reader through a journey that has long been denied to blind 11 http://leonardo.info/isast/spec.projects/synesthesiabib.html 12 http://www.youtube.com/watch?v=yrLj2IGFWw0 13 http://www.afb.org/store/Pages/ShoppingCart/ProductDetails.aspx?ProductId=978-0-89128-850-3 13 FP7-ICT-2013-C SEEDIFF and visually impaired audiences. Each volume contains a book of tactile diagrams and an audio narrative. The diagrams use a lexicon of seven standardized patterns, enabling the reader to acquire a tactile vocabulary. The narrative guides the reader through the diagrams, providing art historical information and richly detailed descriptions of major monuments in the history of art. The success of this two-part system depends on these complimentary components. Professional art historians collaborated with Art Education for the Blind’s development team to create audio narratives that convey the historical richness and formal range of some 30,000 years of visual art. The book Drawing and the Blind: Perceptions to Touch, by Jonathan Harchick, focuses on the ways in which the blind, both young and old, can perceive pictures and 3D objects. According to Harchick, visually impaired people are able to feel a 3D object and then create a drawing of it that can be easily recognized by a sighted individual. Harchick likens the drawings of the average blind-since-birth person to those of a sighted child. He notices that blind children are much more willing to attempt to draw than blind adults who have no prior experience. Kennedy [reference?] discusses the fact that the blind can perceive a drawing made of raised lines, as well as 3D objects that have shape and form. The ways in which the visually impaired are able to create art are giving new insights into the study of sight loss. For example, Ann Roughton is a landscape artist suffering from Macular Degeneration. Her paintings not only include what she can see with her partial vision, but her work also includes the grey haze that she sees in the centre in her vision as a result of her condition. In doing so, she is literally painting sight, giving a new perspective on sight loss. Organizations and charities exist to support blind artists such as BlindArt 14, a British charity established in 2004 to educate the public about the needs of people who are visually impaired and to promote the idea that lack of sight need not be a barrier to the creation and enjoyment of works of art. BlindArt exhibitions typically contain paintings, sculptures, installations and other works of art, created by artists who are blind or partially sighted, which have been designed to engage all the senses. Unlike conventional art exhibitions, visitors to BlindArt exhibitions are encouraged to touch and interact with the exhibits, with latex and cotton gloves provided for this purpose. However, when presenting more traditionally developed art forms for the blind, this is done in a much more conservative form. Specifically, the blind is currently limited to only a handful of small exhibitions always placed in museum corners (as in Centre Pompidou, Paris). In most cases, curators of the museum guide blind visitors; the blind person is a passive participant. Occasionally, visitors are given a small tactile printed diagram, which is not very useful; the blind do not like to use it (Museum of Modern Art, New York City; National Gallery, London). In Tate Modern permanent exhibitions in London there is a possibility to explore sculptures but not pictures. Other galleries have 3D printouts of artefact replicas so that all can enjoy multisensory exploration via hands-on experiences; however, this is impossible for a painting. Equally accessible novel method, created as from an art historian perspective, has been developed by UNIZG (The Jovicic DiTacta): tactile diagrams are placed on specially designed stands where the blind can take in “their hands” a set of diagrams representing a piece of art and explore them while listening to an audio narrative (Box 1). DiTacta is the first educational didactical tool created for exploring art and art history in exhibition space such as museums and galleries, where blind pople usually do not come. This tool is opening new doors to those who are blind and have no access into all spheres or life , tool that promote integration of blind and sighted people. 14 http://www.blindartistssociety.com 14 FP7-ICT-2013-C SEEDIFF We understand that a "picture of a picture" would need to take the form of a tactile diagram. The process should be monitored by a blind person, who can judge whether the resulting tactile diagram is sufficiently "legible". We analysed the painting in detail, described the composition, the elements of form, line and surface, and the details in the picture. Details are part of the whole, and in describing them, we arrive at a full understanding of the whole picture. A single tactile diagram would not suffice for a full understanding of the painting. Instead, we needed to zoom in on different details, like in a film, and depict them in separate tactile diagrams.The greatest challenge is how to present colour within the monochrome tactile diagrams. We found solution. Even precisely drawn tactile diagrams are of little use to a blind person, unless he or she is shown how to use them. We use an audio guide, similar to GPS technology, which leads a blind person from one point to the next, from the top of the painting to the base, from left to right, and from the base to the top. The audio guide gives precise instructions on how to place one's fingers on the tactile diagram, how to move across the painting, how far to move left or right, up or down, etc. So a blind person does two things at the same time: he or she listens to the audio guide carefully, and touches the tactile diagram while following the instructions. This is how he or she can gain an impression of the painting being studied. All sad is aplicable to use in museum. By developing new haptic mobile displays SEEDIFF will make visual art as well as environmental scenes equally accessible in an interactive fashion to sighted and blind people. State-of-the-art neuroimaging studies on multi-sensory integration and cross-modal plasticity Multi-sensory nature and capabilities of the human and non-human brains are well documented by physiological and neurimaging studies providing evidence of similar cross-modal binding mechanisms that appear to be distributed across distinct neuronal networks that vary depending on the nature of the shared information between different sensory cues15.Brain’s remarkable feature of providing the substrates for very early multisensory integration by multiple possible feedback and feedforward circuits for multi-sensory convergence in all sensory areas and yet sufficient sensory segregation to pomote uni-modal sensory representation and uni-modal behavioural experience can be taken as a selective advantage in evolutionary terms. Based on decades of of study of multi-sensory convergence and integration the major functions of multi-sensory convergence and integration seem aimed at enhancing the detection of behaviourally-relevant stimuli, and of promoting rapidity and accuracy of behavioural responding and well as cognitive processing16.Understanding the conditions under which the brain integrates the different sensory streams and the mechanisms supporting multi-sensory convergence at neuronal and system level and cross-modality plasticity represent one of the most challenging problems at the forefront of neuroscience. Multi-sensory interactions can occur shortly after response onset at the lowest cortical processing stages. Moreover, recent findings in many laboratories reinforce the notion that cortical processing per se represents a collaboration between new sensory input and ongoing cortical process16,17. The affective prediction hypothesis 15 FP7-ICT-2013-C SEEDIFF was proposed18 which implies that responses signalling an object’s salience, relevance or value do not occur as a separate step after the object is identified. Instead, affective responses support vision from the very moment that visual stimulation begins. Lesion and neuroimaging studies provide evidence suggesting that regions of the brain normally associated with the processing of visual information undergo remarkable dynamic change in response to blindness. These neuroplastic changes implicate not only processing carried out by the remaining senses but also higher cognitive functions such as language and memory19. However, there is a growing evidence that sensory deprivation is associated with cross-modal neuroplastic changes int eh brain. After visual or auditory deprivation, brain areas that are normally associated wtih te lost sense are recruited by spared sensory modalities (Marabet 2010). While the DTI study on how the blindness onset age affects development of brain anatomical networks20 showed large group differences between congenitally blind and early blind subgroups compared to adolescent-blind and late-blind subgroups specifically, a reduced connectivity density and a decreased global efficiency compared to the sighted controls, in particular in the frontal and occipital cortices they also found that some of the network properties were independent of visual experienc, which may account for the preserved functions of the brain in blind subjects. Recent studies provided evidences on much greater levels of plasticity in the adult visual cortex than previously suspected (Spolidoro 2009). Among the various properties that can be extracted from an object, including size, colour, texture, material, hardness, etc., shape is the most prominent for human visual object recognition21. Recent fMRI study Kim 2010 demonstrated that visual and haptic sensory inputs converge on common object-selective brain sites in the occipital, temporal, and parietal cortices to process object shape and found evidence of multisensory integration in these brain regions. The study demonstrated the effect of “enhanced effectiveness”, i.e., as the effectiveness of the unisensory component stimuli increased, so did the multisensory gain with the combination stimulus. Selective visuo-haptic processing of shape and texture were demonstrated on sighted subjects22. Sighted individuals can recognize basic facial expressions by haptic surprisingly well. However, this year study 23 conducted both psychophysical and fMRI experiments to determine the nature of the neural representation that subserves the recognition of basic facial expressions in early blind individuals. Their results suggest that the neural system that underlies the recognition of basic facial expressions develops supramodally even in the absence of early visual experience. 15 Calvert G A and Thesene T: Multisensory integration: methodological approaches and emerging principles in the human brain. Jl Physiology, 98, 191-205, 2004 16 Schroeder C E et al: Anatomical mechanisms and functional implications of multisensory convergence in early cortical processing. Int. J. Psychophysiology, 50, 5-17, 2003 17 Schroeder C E and Foxe J: Multisensory contributions to low-level, ‘’unisensory’’ processing, Current Opinion in Neurobiology, 15, 454-458, 2005 18 Barrett L. F. and Bar M.: See it with feeling: affective predictions during object perception. Phil. Trans. R. Scoe. B, 364, 1325-1334, 2009 19 Amedi et al. The Occipital cortex in the blind. Current directions in psychological science, 14, 306-422, 2005 20 Li et al. Age of onset of blindness affects brain anatomical networks constructed using diffusion tensor tractography. Cerebral Cortex, 23, 542-551, 2013 16 FP7-ICT-2013-C SEEDIFF 21 Marr D (1982): Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. San Francisco: W.H. Freeman. xvii, 397 pp. 22 Stilla R and Sathian K: Selective visuao-haptic processing of shapte and texture. Human Brain Mapping, 29, 1123-1138, 2009 23 Kitada et al. Early visual experience and the recognition of basic facial expressions: involvement of the middle temporal and inferior frontal gyri during haptic identification oby the early blind. Frontiers in Human Neuroscience, 7, 1-15, 2013 1.2.2 Technical limitations of existing products, processes and/or services Generally, sensory substitution devices have shown truly remarkable progress in recent years. However, at this time they are still not able to provide detailed information needed for fully experiencing our visual world, especially its more advanced formats. Furthermore, when using these devices the users are basically passive and unable to use many features of normal visually based exploration of our world. Therefore, although providing the basic information, they leave the blind deprived of important contents of our visual cultural heritage such as art and most content of the web. In the case of invasive sensory restoration and neuroprosthetic approaches, there are still several major issues preventing these from becoming true clinical solutions, primarily their invasiveness, costs, technical limitations and the inability to offer them to all populations of the blind. Thus, these solutions still do not provide sight that resembles natural vision and are even more limited in providing the blind access to our visual cultural heritage such as art. Today, the visual world is available to the blind via reading and mobility aids, tactile and auditory sensory substitution devices, and invasive neuroprosthetic approaches. Haptic displays for the blind are currently expensive, clumsy, and designed for very limited use such as reading in Braille. The blind may also use text-to-speech technology for content browsing. This is slow and cumbersome even in text-based scenes, but has no power in describing visual images. Printed embossed maps are also limited to a certain scale, as well as expensive and scarcely available. Therefore, inexpensive haptic displays with interactive map functions could significantly enhance the quality of life of the blind and partially sighted in allowing them to navigate more accurately and provide more freedom for them to visit unfamiliar places. Although constantly becoming more advanced, sensory substitution devices provide only very rough images of basic objects and scenes and don’t offer additional information needed for appreciating the richness of our visual world, especially its more advanced formats such as visual arts. Sensory deprivation affects functional and structural organization of the brain underlying visual perception and it takes time to “become fluent in vision”.15 15 Fine et al. Long-term deprivation affects visual perception and cortex. Nat. Neurosci. 6,915-916, 2003 17 FP7-ICT-2013-C 1.2.3 SEEDIFF Main innovations by SEEDIFF participants [references to articles or web pages to footnotes, please!] 1) The Jovicic DiTacta diagrams and stand with audio guides allow independent exploration by blind people of visual scenes in an accessible way. The novelty is in representing visual information in a new format; the approach involves formal analysis and is practical and didactic. 2) New haptic touch-screen technology providing a widely-accessible and inexpensive means for transforming visual art and other visual images such as maps to haptic–auditory form on tablets and mobile phones. www.senseg.com. Senseg has the core granted patents for electrostatic vibration technology: US patents 7,924,144 (granted APR 2011) and 7,982,588, (granted JUL 2011), and 8,174,373 (granted APR 2012), EPO patent 08805437.4 (allowed MAR 2013), China ZL200880107664.0 (granted MAR 2013). Public papers (can be in Company chapter as well) [HAID 2009] Linjama, J. & Mäkinen, V. E-Sense screen: Novel haptic display with Capacitive Electrosensory Interface. Proceedings HAID 09, 4th Workshop for Haptic and Audio Interaction Design,Dresden, Germany 10-11 Sept 2009. Vol II, pp 24 -25. [Eurohaptics 2012] D. Wijekoon, M.E. Cecchinato, E. Hoggan, and J. Linjama, Electrostatic Modulated Friction as Tactile Feedback: Intensity Perception. ;In Proceedings of EuroHaptics (1). 2012, 613-624. 3) New art form: haptic drawings for both the blind and the sighted, allowing also the blind to create haptic “paintings”. 4) Hexachromatic display technology to expand the experience of color for both the color-blind and the normally sighted. Brooks, A. PRODUCT FROM PATENT – see below 5) Treasures in Ateneum. Detailed audio descriptions of six gem art works in the Ateneum collections, designed and produced in cooperation with Celia Library for persons with visual disabilities http://www.ateneum.fi/fi/ateneumin-aarteita ( in Finnish) 1.2.4 Patent search results There is great amount of patents in the fields of haptics and user interfaces. However, this does not pose difficulties nor concerns to the current project due to following reasons: Senseg has strong IPR portfolio with multiple granted patents Only commercial applications of technology can infringe patents and therefore they do not pose limits to research scope of this project. The field of expanding human senses and transforming art into tactile form are too broad areas to be covered by patents; only specific implementations could be patented. This can be exemplified with 18 FP7-ICT-2013-C SEEDIFF patents search into the US patent data base. For example patents: Tactile reading system for data coming from a computer and associated communication device US patent 6,639,510, System and method for integrating tactile language with the visual alphanumeric characters US patent 7,318,195, Tactile graphic computer screen and input tablet for blind persons using an electrorheological fluid US patent 5,222,895, are typical patent applications describing a narrow band application of the technology towards the blind, which have no bearing for this project. Brooks, A. et al. COMMUNICATION METHOD AND APPARATUS. Patent Family (Pat App 2000): US 6893407 (B1); EP 1279092 (B1); WO 0186406 (A1); DE 60115876 (T2); AU 5822101 (A); AT 313111 (T). 1.3 S/T methodology SEEDIFF will develop a range of new technologies and techniques related to presenting visual art in a haptic– auditory form. A new generation of haptic displays and software-controllable surfaces and sensations is expected to make even visual art equally accessible to sighted and blind. The S/T methodology will include: 1) electrostatic technology for haptic display on mobile devices; 2) programming interface to transform visual scenes to haptic–auditory multi-sensory format; 3) behavioral and functional brain imaging studies to design and optimize the sensory inputs and explore uni- and multi-sensory perceptual pathways and multi-sensory integration compensating for and/or extending human visual perception. This neuro-ICT–art trans-disciplinary project will also include the analysis of paintings from the perspective of an art historian, surveys, multi-media and web presentations; it will showcase the new art and explore its acceptance among sighted and blind, both adult and children within educational contexts. The partners include four well-known universities, a major national art gallery, and a fast-growing SME. UNIZG will analyse paintings to be presented to the blind, describe their composition, the elements of form, lines and surfaces, the details in the pictures, and their colors in order to present them as multiple tactile diagrams. The presentation of colors by textures will be developed, which in the electronic format will include novel vibratory or other time-dependent effects. To help navigate the display and to convey our message, we will develop audio enhancements and provide an audio guide that, in addition to explaining the content and background of the painting, leads the blind person from one point to the next in the painting and from one frame to another. Once developed, other partners in the proposal will test several potential tactile–auditory combinations representing the visual input behaviorally and with neurodynamic functional brain-imaging studies with the blind. 19 FP7-ICT-2013-C SEEDIFF Box. Novel concepts and technologies under development by the partners I. Tactile–auditory exhibition of visual art for blind and sighted invented by UNIZG (Fig.1). Fig. 4. Left: Exhibition by UNIZG where a 5-display representation of Caravaggio’s painting was presented to the blind in tactile–audio form. Right: One of the details of the painting; the tactile pattern is overlaid on the corresponding visual image. II. The haptic display technology based on a recent invention and under development by Senseg (Fig. 2) will allow one to produce vivid sensations of texture and form on touch-screen devices such as tablets or smart phones. New forms of tactile interaction will be developed to represent the richness of visual information to partially sighted people on these displays. Fig 5. Left: The principle of our haptic display. A transparent positively charged layer under the surface of a touch screen gives rise to an electric field that makes the fingertip negative (positive charges are pushed away). A force proportional to the electric field and the charge displacement is created (proportional to V2). Right: It is possible to create vivid sensations of shapes or textures on mobile displays. III. For the color blind with dichromacy, we plan to expand color vision by a novel 3Ddisplay technology that can add spectral dimensions (as if there were 3 or more cone types in the retina). Fig 6. A pair of differently colored paintings (Kandinsky). When viewed stereoscopically (left image to the left eye, right image to the right eye), new color sensations can be created. In photography, every other wavelength band could be directed to the left image and the other bands to the right, creating a hexachromatic display. 20 FP7-ICT-2013-C SEEDIFF Senseg will develop its haptic technology to allow one to present tactile–auditory representations of art and other visual scenes to the blind. It will provide software tools to enable other partners to experiment with the company’s new technology and to design multisensory content for the blind. The software will allow a flexible pace of exploration; the amount of detail or background information is selectable. The audio presentation can be synchronized to the tactile exploration without the need to follow a strict order. This complex processing will be done in steps allowing both the automated conversion of visual information into tactile realm, as well as guided design of haptic design and new interaction elements. Aalto will experiment with the new color display technology based on presenting different parts of the color spectrum to different eyes and will work on software development. Together with UNIZG and UNIST, it will also perform MEG/EEG experiments with tactile and multi-color visual stimuli, including visual mismatch (MMN) studies. In collaboration, the university partners will conduct a range of behavioural experiments to study the effects of different combinations of the simultaneously presented tactile and auditory information, in order to better understand the coupling between the two sensory inputs and explore their more efficient use in creating the emerging visual experience. These results will be used for specifying the properties of cross-modal information i in the final product of this project. The three university partners will also use neurodynamic MEG/EEG methods to study multi-sensory modulations of human visual processing evoked by visual, tactile, and auditory–tactile stimulation in subjects with full vision in a search of: 1) an analogue of a retinotopic mapping of the visual field presented via a tactile/haptic display; 2) neurodynamic networks activated by single and multiple objects recognition and discrimination tasks (i.e., simple object vs. face) and 3) potential of texture-coded enabled colour perception; 4) multi-frame tactile-auditory integration in coherent visual art perception; and 5) an objective evidence of hexachromatic perception. Partially sighted and blind subjects will be used in a subset of studies to explore non-visual multi-sensory integration and cross-modal plasticity and find conditions for enabling an enhanced visual experience even in blind. Ateneum will contribute by focusing on cultural, art historical and practical aspects of the project. It will participate in designing and/or evaluating multisensory presentations (a haptic display + audio) for the blind or partially sighted and in designing and/or evaluating multi-colored images. The exhibition will be made available to collaborator galleries in different parts of Europe to allow a large audience across EU to get in contact with the project in its different phases. Providing masterpieces of the museum collection as objects for collaborative research, and offering space and expertise in arranging its own exhibition and exhibitions in other major museums, the gallery will form one of the hubs of activity in SEEDIFF. Led by Ateneum, SEEDIFF will arrange an in-depth visitor survey that includes interviews and video recordings of the behavior of the blind and other visitors of the exhibition. Based on the ideas, methods, contents, and observations of SEEDIFF, Ateneum, together with others, will develop educational and demonstration material, including exhibition brochures and web pages to obtain publicity and to spread the results. Aalborg, which has long-term experience of creative work in bridging science and art as well as in working with minorities such as the blind, will act as a catalyst to research, develop and showcase the new art offered by the state-of-the-art advancements resulting from SEEDIFF. This will include a lab-based/museum-based Participatory Design strategy questioning the impact for the blind. From this will evolve new frameworks for creative expression and related stimuli-enhanced experiences. The Noldus Observer behavioral analysis tool will be used to examine responses, which will inform to advance the creation of the SEEDIFF state-of-the-art. The phenomenon of blind artists will be researched along with prior related cross-modal works. Aalborg will also lead the concluding dissemination at the SEEDIFF European Cultural Capital exhibitions that will conclude the project with high impact dissemination when Denmark co-host with Cyprus in 2017. 21 FP7-ICT-2013-C SEEDIFF Fig. 8. SEEDIFF’s inter-connectivity network of innovative neuro-art-ICT technologies and approaches (in yellow), research studies, exibitions and societal impact, as well as project partners and end-users 1.3.1 Overall strategy of the work plan We will develop two ICT platforms for the presentation of visual art and scenes in new haptic and color forms. We will divide the necessary steps towards our goals into well-defined Work Packages (WPs) so that when the WP tasks are done according to the schedule (see Gantt table), we will reach the goals during the project duration. WP1 (see also Section 2) is meant for making sure that SEEDIFF proceeds according to the plan. WP2 and WP3 deal with the development of haptic and color displays, i.e., the tools needed for completing work in the other WPs. WP4, based on previous work on fixed (usually glass) haptic frames and on our invention of hexachromatic displays, deals with developing methodology to transform visual images to the new forms. In addition, neurophysiological experiments in WP5 provide information regarding neural foundations of enhanced sensory perception. WP6 is a bridge from the methodological development to societal applications of the new technology and serves WP7 by developing dissemination and educational uses for the technology. The solutions developed in the different WPs will be tested using behavioral studies that will also provide a bridge towards testing their neural correlates in neurophysiological studies planned in WP5. ii) Show the timing of the different WPs and their components (Gantt chart or similar). Note that, whereas the scientific and technological methodology is evaluated under the criteria ‘S/T quality’, the quality of the actual workplan is evaluated under FET-Open under the criteria ‘Implementation’. 2 A work package is a major sub-division of the proposed project with a verifiable end-point – normally a deliverable or an import ant milestone in the overall project. 1 Version V1.0 55/72 26-02-2013 Information and Communication Technologies Call FP7-ICT-2013-C Guide for Applicants Collaborative Projects (STREP) iii) Provide a detailed work description broken down into work packages: 22 FP7-ICT-2013-C SEEDIFF 1. Work package list (please use Table 1.3a); 2. Deliverables list (please use Table 1.3b); 3. List of milestones (please use Table 1.3c); 4. Description of each work package (please use Table 1.3d); 5. Summary effort table (please use Table 1.3e) iv) Provide a graphical presentation of the components showing their interdependencies (Pert diagram or similar) v) Describe any significant risks, and associated contingency plans 1.3.2 Work description Table 1.3a: Work package list WP Work package title Type of activity Lead partner # Lead partner Personmonths Start month End month 1 Management MGT 1 Aalto 22 1 36 2 Haptic hardware and software RTD 4 Senseg 65 1 36 3 Hexachromatic technology and methodology RTD 1 Aalto 56 1 36 4 Tactile–auditory representations of visual scenes RTD 5 UNIST 77 1 36 5 Neuronal foundations of enhanced sensory perception RTD 6 UNIZG 79 1 36 6 Social and personal aspects RTD 2 Aalborg 58 1 36 7 Dissemination RTD 3 Ateneum 13 1 36 Total 384 Table 1.3b: Deliverables list # Deliverable name WP Nature Dissemination level Delivery month D1.1 12-month progress report 1 1 R CO 12 D1.2 24-month progress report 2 1 R CO 24 D1.3 Final report 1 R CO 36 D2.1 Visuo-tactile rendering concept for touch screens 2 R PU 9 23 FP7-ICT-2013-C # SEEDIFF Deliverable name WP Nature Dissemination level Delivery month D2.2 First test implementation in tablet device 2 R PU 12 D2.3 Pilot implementation in an exhibition setting 2 R PU 18 D2.4 Complete tactile display functionality in exhibition 2 R PU 30 D2.5 Solution description for tactile experience display 2 R PU 36 D3.1 Study of color spectra in natural and man-made environments 3 R PU 18 D3.2 Methodology for hexachromatic photography; results of hexachromatic photography and hexachromatic art 3 R PU 30 D3.3 Results on behavioral and neurophysiological studies of hexachromatic vision 3 R PU 30 D4.1 Report on existing tactile-auditory coding schemes giving details per scheme, evaluation results, the underlying task, a list of annotated phenomena 4 R PU 6 D4.2 Test results for tactile–auditory display 4 R PU 24 D5.1 Report on non-visually multi-sensory enabled demonstration of enhanced sensory perception of objects 5 R PU D5.2 Workshop on multi-sensory integration and crosssensory plasticity role and potential in education, rehabilitation 5 R PU D5.3 Report on enabled and augmented color perception in sighted and color blind, respectively by hexachromatic display technology 5 R PU D6.1 Educational workshop on tactile–auditory and hexachromatic presentations of art to adults and children 6 R PU 24 FP7-ICT-2013-C # SEEDIFF Deliverable name WP Nature Dissemination level Delivery month D6.2 Report on suggested strategies for the use of the new tactile display in education 6 R PU 34 D7.1 Web page 7 D PU 6 D7.2 Exhibition of haptic art and hexachromatic color at Ateneum 7 D PU 9 D7.3 Exhibition of haptic art at Modern Gallery, Zabreb 7 D PU 24 D7.4 Exhibition of social aspect of sensory expansion, Aalborg 7 D PU 30 D7.5 Exhibition with a video, presenting SEEDIFF advances , Ateneum 7 D PU 34 Table 1.3c: Milestones list Milestones WP2: Initial version of tactile display and software ready for experimentation with the blind, M12 WP3: Hardware and first version of software tools ready for experimentation with hexachromatic photography and to transform trichromatic images artificially to hexachromatic form, M6 25 FP7-ICT-2013-C SEEDIFF Tables 1.3d: Work package descriptions Work package number 1 Work package title Management Activity Type MGT Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 5 6 Aalto Aalborg Ateneum Senseg UNIST UNIZG 17 1 1 1 1 1 Objectives The purpose of WP1 is to make SEEDIFF succeed. While ambitious and risky, SEEDIFF has the potential to change the world of the blind as well as to enhance human sensory experience in sighted people, with possibilities for new art forms and other applications. The goal of WP1 is to maximize the efficiency of SEEDIFF in obtaining the best possible results according to the plan. O1.1: Ensuring that the objectives of SEEDIFF are reached with maximum efficiency. O1.2: Smooth running of the project. O1.3: Ensuring intellectual property. O1.4: Resolution of any conflicts. O1.5: Financial and technical reporting to the Commission. Description of work The work is described in detail in Section 2.1. This covers in particular: Task 1.1: Constitution of the Executive and Steering Committees This will be initially done in connection when agreeing about the Consortium Agreement Task 1.2: Day-to-day management, financial and administrative management The day-to-day management will be done by the Coordinator and Work Package Leaders. The Executive Committee and Coordinator will be in charge of the administrative and financial management. Task 1.3: Reporting and assessments 26 FP7-ICT-2013-C SEEDIFF The Coordinator will be in charge of collecting the reports given by each Participant and will prepare the reports, together with the rest of the Executive Committee, in a form corresponding to templates provided by the European Commission. The Executive Committee will assess the progress of SEEDIFF in their monthly meetings and, more thoroughly in the SEEDIFF project meetings. Task 1.4: Protection of intellectual property The Coordinator will oversee that any essential IPR is evaluated as to whether patenting is necessary. The Consortium agreement will describe in detail how the IPR can be transferred from Participants to the industrial Participants or to other parties. Deliverables Each progress report will describe … [Krisztina, can you say what the progress reports should include?] D1.1: 12-month rogress report, M12 D1.2: 24-month progress report, M24 D1.3: Final report, M36 27 FP7-ICT-2013-C SEEDIFF Work package number 2 Work package title Haptic hardware and software Activity Type RTD Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 5 6 Aalto Aalborg Ateneum Senseg UNIST UNIZG 6 2 1 49 6 1 Objectives The overall goal of this WP is to establish principles and practices, i.e., a completely new approach, for haptic technology for the blind as well as to build a hardware and software platform that is suitable for presenting tangible art. O2.1: Develop a software approach that is suitable to present tactile interaction in SEEDIFF context and suitable for combining multimodality such as auditory narratives into tactile display presentations. O2.2: Develop hardware platforms that can be provided to the whole SEEDIFF consortium. O2.3: Demonstrate tactile interaction with the visual scenes to provide life-like experiences of the art scenes. O2.4: Refine the user experience data and iteratively develop the presentation methods of tangible art. O2.5: Reach an understanding whether tactile displays in museum concept are viable solution to cater the needs of the visually impaired and other special groups, and whether this concept is suitable to be expanded to exhibitions all over the world. Description of work Task 2.1: An approach and software tools for mapping visual scenes to interactive experiences - Collect and synthetize the data and needs of SEEDIFF group for a formulation of target specification for the software approach - Examine and research media presentations techniques for the blind and for children, adapting those for the needs of tactile display technology of this project. - Making the key decisions for the software methodology, repository structure, operating systems support scope, hardware dependencies of the software Establish methodology of creating tactile “haptified” images and having software tools to enable that 28 FP7-ICT-2013-C SEEDIFF - Build multimodal presentation and interaction capabilities into the software. - Develop fluent touch screen and device motion based interaction mechanisms for the blind, which include: zooming to the details and navigating inside content, and real time rendering of visual and tactile data. - Collaborating on the software work with the other consortium members, providing suitable software interfaces Task 2.2: Touch screen hardware platform for tactile presentations - Evaluate the needs of the SEEDIFF consortium from hardware perspective Evaluating various off-the-shelf touch screen platforms for technical challenges in respect to integrating Senseg tactile technology into them - Building multiple iterations of the platform and testing the promising candidates - Freezing the specification for a device - Sourcing the materials needed for multiple units of the selected platform - Building multiple hardware units that can be used by different members of the SEEDIFF group Task 2.3: Developing user experience demonstration using the software on the hardware platform - Together with other SEEDIFF parties draft a palette of design concepts for tangible art user experience demonstration - Co-operate in implementing the tangible user experience software concepts - Support other consortium members in the use of software platform and hardware platform - Perform both technical and the initial user testing for the user experience concepts - Refine the most functional user experience concepts to be ready for experiments with appropriate visually impaired target groups Task 2.4: User testing and iterative development of the user experience with the consortium - Working with the consortium to build user experience test settings with the visually impaired. Participating in exhibition arrangements, including customizing the hardware platform according to exhibition needs including addressing mechanics needs of exhibition setting - Collect user experience data, improve and develop the user experience demo Probe the applicability of the tactile experience display in wider exhibition setting, and probe the needs for platform for customization. Deliverables 29 FP7-ICT-2013-C SEEDIFF D2.1: Visuo-tactile rendering concept for touch screens, M9 D2.2: First test implementation in hardware platform device, M12 D2.3: Pilot implementation in an exhibition setting, M20 D2.4: Complete tactile display functionality in exhibition, M30 D2.5: Solution description for tactile experience display, M36 30 FP7-ICT-2013-C SEEDIFF Work package number 3 Work package title Hexachromatic technology and methodology Activity Type RTD 1 2 3 4 5 6 Aalto Aalborg Ateneum Senseg UNIST UNIZG 35 3 1 6 1 10 Participant number Participant short name Month 1 Starting date Person-months per participant Objectives The overall goal of WP3 is to establish principles and practices for hexachromatic vision and the methodology of realizing such vision. O3.1: To build a platform for studies of hexachromatic color perception. O3.2: To develop the methodology of transforming visual scenes to hexachromatic descriptions. These descriptions will be used as the basis of presenting hexachromatic images on standard 3D display devices such as 3D TVs. O3.3: Exploration of hexachromatic technology as a new aspect in visual art Description of work Task 3.1: Study of color spectra in natural and man-made environments The spectrum of reflected visible light will be measured from different objects in selected environments that have many differently collared surfaces. Such environments include botanical garden; zoo, especially areas for birds and butterflies; supermarket with multiple man-made items, fruits and vegetables; art museum with paintings will be also obtained. The data will be used to determine the division of the visible part of the spectrum into 6 bands so that maximal information of the original spectra is retained (see Fig. 2). Task 3.2: Hexachromatic photography A camera setup will be constructed consisting of two high-quality cameras (such as Nikon D7000) mounted next to each other for simultaneous exposures and fitted with exchangeable notch filters. Natural scenes (those mentioned above) will be photographed or video is recorded so that two complementary exposures (call them “left” and “right”) of each scene are obtained with comb filters selected on the basis of the results of T3.1 so that the left exposure is formed by light in odd-numbered bands of the visible spectrum while 31 FP7-ICT-2013-C SEEDIFF the right exposure is obtained by even-numbered bands (see Fig. 2). Task 3.3: Hexachromatic “art” Pieces of visual art consisting of initially uniform patches of color (such as those by Piet Mondrian, Vasili Kandinsky, Johannes Itten, or Joseph Albers) will be divided into left and right “component pictures” so that the average picture is identical or similar with the original but the two pictures have different colors and/or intensities in its patches so that the information content of the image is increased. Task 3.4: Testing perception of hexachromatic displays Hexachromatic photography and art will be tested behaviorally (sensitivity thresholds to different kinds of color changes will be determined) and neurophysiologically. An odd-ball paradigm with hexachromatic pictures and their trichromatic counterparts (averages of left and right images) will be presented and differences in brain responses to the two kinds of stimuli will be determined. Both normally sighted and color blind subjects will be recruited. Task 3.5: Testing perception of hexachromatic displays Hexachromatic photography and art will be tested behaviorally (sensitivity thresholds to different kinds of color changes will be determined) and neurophysiologically (odd-ball paradigm: hexachromatic pictures and their trichromatic counterparts (averages of left and right images) will be presented and differences in brain responses will be studied. Deliverables D3.1: Study of color spectra in natural and man-made environments D3.2: Methodology for hexachromatic photography; results of hexachromatic photography and hexachromatic art. D3.3: Results on behavioral and neurophysiological studies of hexachromatic vision. 32 FP7-ICT-2013-C SEEDIFF Work package number 4 Work package title Tactile–auditory representations of visual scenes Activity Type RTD Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 5 6 Aalto Aalborg Ateneum Senseg UNIST UNIZG 5 1 2 1 36 32 Objectives The overall goal of WP4 is to develop methodology to transform visual scenes such as visual art, drawings, and photographs to tactile–auditory form. This can be done using behavioral experiments aimed to test the currently developed solutions and to evaluate individual features of tactile-auditory elements present in the diagrams. O4.1: To develop the methodology of transforming visual scenes (including works of art) to haptic descriptions. needed for developing 2D tactile diagrams of visual paintings. O4.2: To test the applicability of the translation methodology developed for visual paintings on haptic display devices. Description of work Task 4.1: Develop “grammar” of segmentation of visual scenes to tactile format and tactile labeling. Determine the relevant tactile and auditory features that allow the translation of presented visual scenes into an understandable tactile-auditory format (UNIST, UNIZG) Collect information on existing tactile/haptic-auditory coding schemes developed for pieces of art work or visual scenes in general, either by interviewing, e-mail correspondence with developers and users or by literature search. Information on evaluation results as well as tactile act recognition (how the visual scene was segmented to tactile format and how it was labelled), communication problems, cross-level issues, co-references, perceived levels of difficulty and prosody will be collected. Analyse these data and suggest a standard framework and a series of best practice coding schemes. Task 4.2: Select the best practice coding schemes. Conduct behavioral experiments in order to test participants’ experience of developed tactile–auditory representations and determine how well they translate the visual experience to the “viewer”: how well the viewers understand the display, what 33 FP7-ICT-2013-C SEEDIFF emotional reactions it triggers, and how the artistic message is conveyed. (UNIST, UNIZG) By using structured interviews in qualitative studies of up to 10 blind and partially sighted adults: a) test acuity of perceived visual scene using questionnaires developed for assessing participants’ experience of tactile diagrams, b) estimate the difficulty/ease in understanding of the tactile-auditory coding schemes by using structured interviews and collecting information described in T4.1, c) estimate how the artistic message was conveyed through tactile diagrams. Based on results choose the best practice coding schemes from the set of T4.1 schemes. Interview-based qualitative research approach is chosen for this goal as it allows us to probe into depth participants’ experience with the presented tactile diagrams. After choosing the best subset of schemes, repeat experiment a)-c) on pieces of art with significantly different painting characteristics. This approach should test whether schemes are dependent on concept, i.e., should there be a difference in method of segmenting visual scenes to tactile format for significantly different paintings. Finally, validate earlier findings on a separate sample of participants and determine differences between different populations of the blind. T4.3: Labeling in haptic displays - testing perception of microelements (< 2–3 mm) and macroelements (UNIST) Evaluation of electrostatic haptic elements that optimize tactile spatial acuity for simple 2D forms. Identify tactile micro and macroelements to be presented (i.e., grating orientation, grating ridge width, gap detection, ...). For each element define two alternative forms, randomly display these forms (each form is presented at least 3 times) and test accuracy of recognition in several experiments using groups of approximately 30 blindfolded sighted participants and a smaller number of blind subjects. Deliverables D4.1: Report on existing tactile-auditory coding schemes giving details per scheme, evaluation results, the underlying task, a list of annotated phenomena, M6 D4.2: Test results/reportsfor tactile–auditory display, M24 34 FP7-ICT-2013-C SEEDIFF Work package number 5 Work package title Neuronal foundations of enhanced sensory perception Activity Type RTD Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 5 6 Aalto Aalborg Ateneum Senseg UNIST UNIZG 12 1 - 1 14 51 Objectives The overall objective of WP5 is to learn about brain processes when novel sense-enhancing technologies are used during the learning phase and in routine use. We will use neurodynamic EEG and MEG methods to obtain measures of neuronal function related to the visually and non-visually enabled sensory perception in order to objectively compare brain responses when different stimulation paradigms are used. The results, together with those from behavioral studies, will be used to decide how the multi-sensory stimulation and new display technologies should be enhanced and optimized, respectively. O5.1: To identify whether there is a tactile–spatial analogue of retinotopic representation using MEG O5.2: To demonstrate that object recognition and multi-object discrimination (simple object vs. face) is enhanced by combined tactile/haptic and auditory stimulation O5.3: To demonstrate effectiveness of texture-coded color perception. O5.4. To demonstrate multi-frame visual-to-tactile–auditory integration in coherent visual art perception O5.4: To develop and demonstrate an objective measure of hexachromatic perception based on MEG or EEG Description of work Task 5.1: Search for tactile analogy of retinotopy (UNIZG, UNIST, Aalto) Sighted blindfolded, and blind subjects will be studied. Tactile/haptic objects are presented at different eccentricities from the centre of the screen with and without tone guidance while MEG is being measured. UNIZG will select blind subjects after behavioral studies in Croatia (UNIZG and UNIST), UNIST will design haptic stimuli using software developed by Senseg, UNIZG will conduct MEG measurements together with Aalto at Aalto MEG laboratory, and perform spatio-temporal source localization during the earliest evoked responses (up to 100ms poststimulus). Aalto will recruit sighted subjects Task 5.2: Multi-object recognition and discrimination 35 FP7-ICT-2013-C SEEDIFF MEG responses will be recorded from sighted blindfolded and blind subjects during tactile/haptic and tactile/haptic and auditory simple geometrical objects and face recognition and multi-object discrimination tasks. Task 5.3: Texture-coded color perception EEG/MEG responses will be recorded from sighted, sighted blindfolded, and blind subjects during visual and haptic, haptic, and haptic-auditory task after behavioural studies and training sessions on 3 to 5 texture coded colors (behaviourally tested by WP 4) of a single and multi-object recognition tasks. UNIZG will select blind subjects after behavioral studies in Croatia (UNIZG and UNIST), Aalto will recruit sighted subjects, UNIZG and UNIST will design haptic and haptic and auditory stimuli by using software developed by Senseg , conduct MEG measurements with Aalto at Aalto’s MEG laboratory . UNIZG will conduct MEG spatio-temporal source localization during the earliest evoked responses (up to 100ms poststimulus) for Task 5.1, localization and functional connectivity analysis for Task 5.2.; functional connectivity and detrended cross-correlation time series and and pattern recognition approaches for Tasks 5.3. and 5.4. Task 5.4: MEG/EEG recordings of brain activity elicited by hexachromatic displays (Aalto, UNIZG) Trichromatic and hexachromatic visual images will be presented to subjects in alternate order in such a way that the average color remains the same but in the hexachromatic displays, the left and right eye receive different colors. The hypothesis is that the brain reacts differently to these two kinds of stimuli. This will give us an objective measure of the new kind of color perception so that we will not need to rely on verbal reports of the subjects alone. Aalto will design the stimuli, recruit the sighted and color-blind subjects, conduct and analyze MEG measurements with UNIZG. Deliverables D5.1: Report on non-visually multi-sensory enabled demonstration of enhanced sensory perception of objects, M? D5.2: Workshop on multi-sensory integration and cross-sensory plasticity role and potential in education, rehabilitation, M? D5.3: Report on enabled and augmented color perception in sighted and color blind, respectively by hexachromatic display technology, M? 36 FP7-ICT-2013-C SEEDIFF Work package number 6 Work package title End-user studies: specifications for the technology and education Activity Type RTD Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 1 1 Aalto Aalborg Ateneum Senseg UNIST UNIZG 2 43 1 2 6 4 Objectives The overall goal of this work package (WP) is to develop requirements and specifications for multimodal communication tools for the blind. The focus is on the different types of users who will be influenced by the implementation of the tools developed in SEEDIFF. Thus, visually impaired as well as end-users representing the general public are targeted, together with secondary users such as educators, guides, curators, and blind artists. The outcomes of the requirement analysis and specification will be to capture and document user input and knowledge to enable conceptual frameworks for description and analysis with regard to the user, to the haptic–auditory communication tool and its educational practice, and to its effectiveness to expand human sensory experiences in the visual domain. The conceptual framework will feed into the analysis of actual educational practices as well as to the design of visual content. Producing these results will require sensitive planning and organisation, and a deep understanding of the characteristics of the different types of users (visually impaired as well as sighted end-users and secondary users such as educators, guides, curators, and blind artists) and educational knowledge practices as idiosynchratic entities whose interests and requirements will emerge in a variety of different ways. The activity is iterative and structured around the three phases, including Preparation, Incubation, and Revelation. ??? The main lead in this WP will be taken by Aalborg with its experience and deep background in this area. Therefore, Aalborg will also spend more resources in this WP. Specific objectives: O6.1: to identify and define requirements and specifications for tools in support of multimodal communication processes, O6.2: to describe methods and strategies for cultivating multimodal processes in educational practices taking into account the different types of users, O6.3: to provide optimised specifications in support of multimodal processes taking into account the different types of users. 37 FP7-ICT-2013-C SEEDIFF O6.4: To estimate potential of haptic displays in different settings, i.e., in art education of blind and seeing people and in development of new art form The activity is iterative and structured around the three phases, including Preparation, Incubation, and Revelation. The main lead in this WP will be taken by Aalborg with its experience and deep background in this area. Therefore, Aalborg will also spend more resources in this WP. Description of work Task 6.1: Requirement analysis Goal: to define a conceptual framework for description and analysis of multimodal communication processes and, based on that, map out possible requirements so as to place the theoretical and methodological knowledge into practice. A fundamental component of this task is “inspiration”. This is a reciprocal activity whereby the users inspire the researchers through demonstrating their interests. Activities: xxxx Task 6.2: Incubation and analysis of educational practices Goal: to define and describe authentic pedagogical and learning activities by enrolling the users and reflecting on: (i) user current knowledge and experience, (ii) technological feasibility, (iii) user practice. This task requires a balance between the freedom to explore any idea and the practice in which the ideas will be applied. This task will be the most fertile task of the work as the researchers will need to consider how the ideas might be transferred to methods and strategies for cultivating multimodal communication processes in educational practices. This will require sensitive consideration of the different types of user characteristics. Activities: xxxx Task 6.3: Revelation of specifications Goal: to define requirements and specifications to allow the different types of users to influence: (i) visions and requirement proposals, (ii) impact on the design of multimodal communication processes, (iii) the teaching and learning experiences of multimodal communication processes. (iv) Activities: xxxx Task 6.4: Testing haptic display methodology in educational settings (UNIST, Aalborg) - Presenting SEEDIFF’s products to art teachers at all levels of education, interviewing them and developing strategies for the use of the new technology in education of blind as well as seeing people. Deliverables D6.1: Educational workshop on tactile–auditory and hexachromatic presentations of art to adults and children, M? D6.2: Report on suggested strategies for the use of the new tactile display in education, M 34 38 FP7-ICT-2013-C SEEDIFF Work package number 7 Work package title Dissemination and studies of user behavior at exhibitions Activity Type RTD Participant number Participant short name Person-months per participant Month 1 Starting date 1 2 3 4 1 1 Aalto Aalborg Ateneum Senseg UNIST UNIZG 4 6 7 1 1 7 Objectives The overall goal of this WP is to communicate the results of SEEDIFF to the general public and to end-user groups such as the blind and other visually impaired. While doing this in the form of exhibitions at major art museums, visitor acceptance and reactions will be studied in order to understand the needs of the end-users and to get guidance to continued development of the hardware, software, content, navigation, and instructions. O7.1: To ensure internal and external dissemination of results. O7.2: To organize exhibitions and to perform behavioral studies of visitors. O7.3: To organise a workshop opened to a large industrial and research European community. O7.4: To prepare the exploitation of the results. Description of work T7.1: Web page: creation, development, and maintenance (Aalto) The web page will be targeted to a wide audience: students, researchers, developers, investors, the lay public as well as the media. Parts of it will be designed to be accessible also to blind people, especially when they have access to the haptic technology developed in SEEDIFF. T7.2: Exhibitions Art exhibitions for blind, partially sighted and sighted visitors will be produced in 3 major art museums. The average duration of each exhibition will be three months. The exhibitions will comprise an art work, tactile diagrams, an audio, a haptic display, and/or a hexachromatic part. T7.3: Behavioral experiments 39 FP7-ICT-2013-C SEEDIFF Behavioral experiments will be conducted, some of them to be documented in video recordings, in order to collect feedback regarding the visitors’ experience and satisfaction. Experiments will be executed in cooperation with 2-4 blind or partially sighted partners. Analysing the survey results, and passing the results on for the use of other members of SEEDIFF, will provide those participants developing haptic displays with crucial feedback information of the experiences of the subjects using the haptic and audio devices. Writing and producing an exhibition brochure (to be expanded to a book in a separate project) in cooperation with the University of Zagreb. T7.4: Visitor surveys in art museums (Ateneum) Both blind and sighted visitors are asked either orally or by forms to describe their experience and satisfaction. Analysis of the survey results will give feedback to haptic display developers and, when conducted in the final exhibition in the end of the SEEDIFF project, it will serve as the feedback and summary of the results of the project concerning the subject visitors; blind, partially sighted and sighted persons. T7.5: Workshops and training events Near the end of the project, a series of local Workshops reporting the results of the project will be arranged in Aalborg, Helsinki, Split, and Zagreb. T7.6: Collecting user feedback At exhibitions, volunteer end-users will be tested with basic psychophysiological methods; they will be interviewed or they will be asked to anser visitor survey questions on their reactions regarding the displays and on their acceptance of the technology. Deliverables D7.1: Web page, M6 D7.2: Exhibition of haptic art and hexachromatic color at Ateneum, Helsinki, M 9 D7.3: Exhibition of haptic art at Modern Gallery, Zagreb, M24 D7.4: Exhibition of social aspect of sensory expansion, Aalborg, M30 D7.5: Exhibition with a video, presenting SEEDIFF advances, Ateneum, Helsinki, M34 SECTION 1.5. 40 FP7-ICT-2013-C SEEDIFF 2 Implementation 2.1 Management structure and procedures 2.1.1 Management capability of the co-ordinator The present Coordinator (Prof. Ilmoniemi) has documented ability to lead research groups and consortia with determination and success. He has successfully coordinated two EU-funded projects, one with 9 partners16 and the other with 13 partners17. In addition, he has led the Framework Programme project participation as the head of the BioMag Laboratory18 at the Helsinki University Central Hospital and as the CEO of the Nexstim Company19. He has also experience as evaluator of European proposals and projects20. The Coordinator led the BioMag Laboratory of the Helsinki University Central Hospital from its beginning in 1994 until 2003. He was the Chairman of the Board of Nexstim Ltd. after he founded the medical technology company in 2000; he was CEO in 2003–2005. The Coordinator was Head of the Laboratory of Biomedical Engineering at the Helsinki University of Technology in the beginning of 2007 and the Head or the Department of Biomedical Engineering and Computational Science in 2008. 2.1.2 Management structure and decision-making structure The highly interdisciplinary nature of, and the wide scope of expertise, from art to neuroscience to ICT, required in SEEDIFF will demand close coordination between the participants and a clear management plan. Project Management is considered a separate work package. A Consortium Agreement that will establish the rules on how the Project is regulated will reinforce a strong partnership. The European Commission, DG XII, Grant Contract PSS*1046, Joint European project “Imaging of language functions in the brain”, 1999–2001 (BioMag Laboratory, Coordinator). 16 The European Commission, FP7-HEALTH-2007-A, Collaborative project “MEGMRI - Hybrid MEG-MRI Imaging System", Contract No. 200859, 2008–2012 (BECS, Coordinator). 17 The European Commission, Joint European project “Objective evaluation of cognitive brain function and dysfunction”, Contract N.o BMH4-CT96-0819 (DG XII–SSMA), 1996–1998 (BioMag Laboratory, Partner). 18 19 The European Commission, STREP project “Enough Sleep”, Contract No. 518189, 2005–2008 (Nexstim, Partner). Evaluation of R&D proposals in the field of Neurosciences under the program “Quality of Life and Management of Living Resources” of the 5th Framework Program, European Commission, DG XII, Brussels, January 17–20, 2000; in IST Future and Emerging Technologies FET calls, Brussels, May 22–26, 2000; Helsinki, Sept. 11, 2000; Brussels, May 14–15, 2001; Helsinki, November 2001; Brussels, March 19–22, 2002; Brussels, October 26–29, 2004; Helsinki Nov. 2004. Review of the progress of the ESPRIT project “Adaptive Brain Interfaces” (European Commission, DG XII; project ABI, No. 28193). Ispra, Italy, October 29, 1999 and Helsinki, May 16–17, 2000; of the European Commission IST project “Insight 2+”. Leuven, Belgium, December 2, 2002; of FP6 project PRESENCCIA, February 2007. [Flagship 2011–12 ETC.]. 20 41 FP7-ICT-2013-C SEEDIFF The main objective of Project Management is to make sure that all steps needed for the realization of our goals will be made in time and within the frame of the budget. Project Management is the task of the Coordinator, the Vice Coordinator, the Steering Committee, and the Executive Committee. Steering Committee External Advisors European Commission WP1 1 Coordinator Project Officer Executive Committee WP 1 WP 2 WP 5 WP 6 WP 7 Figure 1. SEEDIFF management structure [Must be improved] Coordinator The Coordinator (Prof. Ilmoniemi) will lead the project and have the ultimate responsibility for its success. He will employ an experienced, knowledgeable Project Manager with project-management skills to help perform daily project management, to coordinate project self-assessment, to organize meetings, to organize the financial and technical bookkeeping, and to collect the deliverables. The Coordinator will, after consulting the Executive and Steering Committees, direct resources to any critical bottleneck that may appear. If necessary, resources may be redistributed among the Participants. The Coordinator will also use every effort to guarantee that the results of the project will be fully exploited by the Participant organizations. SEEDIFF will licence new technology to third parties or form start-up companies especially in cases where the present Participant SME is not able or willing to utilize some of the results. 42 FP7-ICT-2013-C SEEDIFF In the course of the project, the fundamental question to ask regularly is, “how does the present work of each Participant contribute to the objective of the project”. The Coordinator, together with the Steering and Executive Committees, will help revise the work plan whenever needed and to redirect efforts correspondingly. If the Coordinator is not able to perform his tasks, the Vice Coordinator will overtake his duties until the situation returns to normal. If necessary, a new Coordinator may be selected by the SC by a majority vote of at least 4/6. Supported by the Executive and Steering Committees, the Coordinator oversees the progress of the project, initiates action when problems arise, delays can be foreseen, or when new opportunities for additional progress are discovered. The Coordinator is the link between the project and the Commission; he is responsible for the overall success of the project. Steering Committee The Steering Committee (SC) is composed of all Group Leaders (see Table 1), including the Coordinator (Prof. Ilmoniemi) and Vice Coordinator (Prof. Supek). It is the ultimate decision-making body of the project. The Steering Committee will be the body that makes any strategic decisions (budgetary, scientific, etc.) for the project. If a Group Leader is unable to attend an SC meeting, he/she can assign a representative with full voting power to such a meeting. It is also possible for to participate in decision making at an SC meeting using Skype or other electronic means. The Steering Committee will meet twice a year at the regular SEEDIFF project meetings. The decisionmaking process within the SC will be the result of a collegial discussion, followed by a vote in case no unanimous decision can be made (one vote per those SC members who are present / absolute majority). In the unlikely event when no majority can be reached, the Coordinator (or if he is absent, the Vice Coordinator) will have the decisive vote. At least 4 members of the SC must be present for a decision to be made. The Steering Committee will be in charge of reporting and the revision and updating of the work plan, on-going risk assessment, and general self-assessment of the project. It prepares the agenda and material for each Steering Committee meeting. The work of the Steering Committee will begin already when a positive funding decision has been obtained. The Steering Committee members may then be asked to participate in the negotiations with the Commission services. The work of the Steering Committee continues also after the end of the project as long as there are open issues regarding the project. Work Package Leaders / Group Leaders The Work Package Leaders (WPLs, see table below), who are also Group Leaders of the respective Participants, are in charge of the implementation of the project work plan and reporting to the Coordinator. The WPLs will be in frequent (at least every two months) contact with the Coordinator to ensure that the objectives of the individual WPs are met in accordance with (i) milestones and (ii) deliverables and that (iii) each Participant fulfils its commitment to the work package. Participant number 1 Participant short name Aalto Group leader/WP Leader WPs to lead Risto Ilmoniemi 1, 3 43 FP7-ICT-2013-C SEEDIFF 2 Aalborg Tony Brooks 6 3 Ateneum Anja Olavinen 7 4 Senseg Ville Mäkinen 2 5 UNIST Ana Jeroncic 4 6 UNIZG Selma Supek 5 Administrative Contact Persons Regarding the administrative management, each Participant will nominate an Administrative Contact Person in each institute/company in charge of communication all relevant finance and reporting matters (finance distribution, audit certificates, human resource, legal aspects, etc.). This person should be distinct from the group leader (see Table 1 for the names of group leaders) to assure that the information needed will be readily available to the Coordinator. The Project Manager serves also as Aalto’s Administrative Contact Person. His duties include the collection of reports and other information from the participants and administrative work at Aalto, including the sending of reports to the Commission. He works closely together with the Coordinator, with an office next to that of the Coordinator’s to allow daily communication. 2.1.3 Monitoring, reporting progress and documenting results Project evaluation and other meetings The project will be formally monitored and evaluated by the Commission. In addition, the Steering Committee may ask external evaluators, commentators, consultants, and collaborators to contribute to the assessment of the project. We have asked Profs. Charles Schröder21, Göte Nyman22, and Ivica Kostovic23, to act as external advisors. In addition, David Blankett24, member of the British Parlament, 21 Prof. Schröder is pioneer in primate and human studies in cross-sensory integration. His has demonstrated that low-level auditory sensory area is involved in multisensory integration and that early multi-sensory convergence can occur through both feedback and feedforward circuits and that evolving concept of the sensory processing hierarchy must encompass temporal as well as anatomical dimensions. 22 Prof. Nyman is former dean and professor at University of Helsinki. Since Jan. 2013, he has been leading the 4-year, “MIND programme” Finnish Academy project "What is an image?". He has wide experience in vision science, user interfaces and interaction with the industry, e.g., with Nokia. 23 Prof. Kostovic is director of the Croatian Institute for Brain Research. He is developmental neuroscientist who discovered transient synaptic zone (subplate) and its crucial role in the establishment of cortical maps, its modular organization. He contributed to the understanding of the protracted period of human cortical plasticity and reorganization related to maturation, socialization, education and cognitive development. 24 David Blankett was educated at schools for the blind in Sheffield and Shrewsbury, attended the Royal National College 44 FP7-ICT-2013-C SEEDIFF blind himself, is already advising the project from the perspective of both societal decision making and blindness. The progress of SEEDIFF will be reported in progress reports (deliverables 1.1–1.6) as well as in SEEDIFF project meetings. The main purpose of the meetings is to advance the progress of the project. When decisions are made or project issues are formally discussed, one of the meeting participants (Project Manager, when present) is selected to keep the minutes of the meeting. The Steering Committee may invite additional people to their meetings if that is considered advantageous for the project. Aalto will arrange a 2.5-day Kick-off Meeting during the first month of the project. The first half-day of the meeting will be devoted to the background and the general approach of the project. The second day will consist of thorough exposition and discussion of the plans of each of the Participants. On the third day, specific strategic issues and expected challenges or bottlenecks will be discussed and decisions will be made on what measures should be taken to address these. The Steering Committee will hold a Review of Progress Meeting twice a year. The SC may propose a change in the Project Plan in light of the progress made, new discoveries or developments made elsewhere. Such a proposal will be presented to the Project Officer appointed by the Commission. These meetings may be combined with review meetings in presence of external experts appointed by the Commission. The Coordinator will be in contact with the Participants monthly, usually by email or Skype, to discuss the progress of the project. Training events and small seminars are arranged throughout the project; unofficial project meetings can be arranged between Participants in connection to these events. Resolution of conflicts [to be revised] All disputes or differences arising in connection with SEEDIFF that cannot be amicably settled within the Steering Committee shall be finally settled through arbitration in Brussels under the Rules of the International Chamber of Commerce (ICC). Arbitration shall be conducted in the English language. Three Arbitrators (3) shall be selected. Each Party shall select one (1) arbitrator. Each Contractor or party shall also notify the other Contractor or party/parties in writing of its selection and the two (2) arbitrators so chosen shall select the third arbitrator. If either Contractor or party fails to appoint an arbitrator within thirty (30) days after the Demand for Arbitration, then the International Chamber of Commerce shall make the selection. If the two arbitrators selected by the parties cannot agree on the third arbitrator within thirty (30) days from the date the last of the two has been selected, then the International Chamber of Commerce shall make the selection. In deciding matters, the arbitrators shall be bound by the terms and conditions of the Consortium Agreement, and they will decide ex bon et aequo. for the Blind in Shrewsbury. At the University of Sheffield, he gained a BA honours degree in Political Theory and Institutions; one of his lecturers was Bernard Crick. He entered local politics on graduation, whilst gaining a Post Graduate Certificate in Education (PGCE) from Huddersfield Holly Bank College of Education. 45 FP7-ICT-2013-C SEEDIFF The arbitration award, if providing for damages, shall include interest from the date of any breach or other violation of the Consortium Agreement. The arbitration award shall be final and binding upon the Contractors or parties, not subject to appeal, and honoured by the Contractors or parties without having resort to any court; however, if the award is not carried out voluntarily and without delay, it shall be referred to and enforced by any court having jurisdiction over the subject matter or any of the parties or their assets. Each Contractor or party bears its own expenses incurred in utilizing arbitration and the fees for arbitration shall be borne equally between the Contractors or parties. Risk assessment and management Tables in Section 1.5 [CHECK NUMBERING] describe the main risks forecasted in this project with their severity and indication of contingency actions. The Executive Committee is in charge of the risk management, according to the following principles. 1. Based on the risk-analysis tables, estimate any risk that may have increased or is about to realize. 2. Establish and implement counter-measures if appropriate. 3. Set up early measures to minimize foreseeable risks. 4. Follow-up of the effects of the measures that have been taken. The risks include those in management, financial situation, technical challenges or setbacks, change of Participants, changes in dissemination and/or exploitation possibilities. 2.1.4 Meeting agendas The biannual SC meetings and monthly EC meetings will be conducted according to a standard agenda; minutes of the meetings will be assembled in Deliverables 1.1–6. 2.1.4.1 Standard agenda for Steering Committee meetings 1. Opening of meeting 2. Review of Progress /checkup status 3. Deliverables and milestones 4. Action list 5. Human resources issues 6. Financial issues 7. Monitoring of ethical and gender issues 8. Decisions regardging corrective actions, if needed 9. Other issues 10. Closing of meeting 46 FP7-ICT-2013-C 2.2 SEEDIFF Individual participants Participant 1: Aalto Main tasks: Coordination, hexachromatic color technology, neurophysiological studies. Previous experience: Coordination of EU-funded and national projects, summer schools, conferences etc.; R&D on 6D color technology (funded by Runar Bäckström Foundation, 2011–2012); extensive experience on neurophysiological studies using MEG, EEG, and TMS. The Department of Biomedical Engineering and Computational Science (BECS) at Aalto University (formerly Helsinki Univ. of Technology) is a pioneer in several neuroimaging technologies. MEG development started at Aalto University (until 2010 Helsinki University of Technology) already in the 1970’s. The work led to the spinoff company Neuromag (now part of Elekta), the world leader in MEG devices. MEG research has been extensive both at Aalto and at the Aalto-initiated joint research centre BioMag. TMS development started at Aalto in 1992 and continued at BioMag and Nexstim Ltd. (founded by Ilmoniemi in 2000). Nexstim combined TMS with MRI and EEG to a new imaging modality that can map cortical excitability and functional connectivity. NIRS development started at Aalto in 1998 and has led to low-noise transportable multichannel frequency-domain instrumentation. Ongoing studies on adults and neonates include clinical research. In addition to the front-line instrumentation development, BECS is very strong in computational technology, including the development of methodology for brain imaging. BECS has produced a number of innovative technologies also in… The idea of enhancing color vision by using the now-common 3D display technology was … Risto Ilmoniemi is Academy Professor of the Academy of Finland (2012–2015) and tenured professor of Engineering Physics at Aalto University; prior to the Academy Professorship, he was Head of the BECS. He headed the BioMag Laboratory of the Helsinki University Central Hospital in 1994–2003 and was Chairman and CEO of Nexstim in 2000–2003 and 2003–2005, respectively. At Aalto, he has lectured “Structure and Operation of the Human Brain”, “Classical Electromagnetism” and “Functional Imaging in Medicine”. He is author of about 200 peer-reviewed papers and 10 patents. His work to combine brain imaging and stimulation technologies has been recognized by several awards25. Otto Olavinen is a Producer in Aalto University School of Science. Educated in journalism, his expertise is in visual presentations and in communication of science. Before coming to Aalto, Otto was a Vice Coordinator for AAVE (Alternative AudioVisual Event) festival. He has also training in EU project management and will act as the Project Manager and Aalto’s Administrative Contact Person. Mika Pollari is an expert in image and signal processing. He was project manager in the FP7 MEGMRI project (2008–2012) and coordinated BECS’s participation in the FP7 project IMPPACT. He will work in WP1 WP3. Representative publications P. T. Vesanen et al., “Hybrid ultra-low-field MRI and magnetoencephalography system based on a commercial whole-head neuromagnetometer”, Magn. Reson. Med. DOI 10.1002/mrm.24413 (2012). A. Susac, R. J. Ilmoniemi, D. Ranken, and S. Supek, “Sensory-memory-based change detection in face stimuli”, Transl. Neurosci. 1, 286–291 (2011). 25 New Technology Foundation Innovation Prize 1997; Foundation for Medical Technology First Prize 1997; Innosuomi 2004 Prize by the President of Finland to Nexstim (Ilmoniemi, founder, as CEO); European IST Prize 2006 to Nexstim. 47 FP7-ICT-2013-C SEEDIFF Participant 2: Aalborg Main tasks: Previous experience: Aalborg University is … Tony Brooks is based in Aalborg University Esbjerg, Denmark, where he is an Associate Professor and a founding member of the Medialogy education. He is Director of the SensoramaLab complex - see http://medialogy.eu/facilities. Since the mid-eighties, he has developed SoundScapes, a multi-modal sensory stimulus system for improvised artistic expression through physical interaction by people in the disabled community. It is targeted at all people no matter ability, age, or preferences and has developed to be considered as a non-formal rehabilitation training system for supplementing traditional methods of therapy. It is a nonintrusive (i.e. non wearable) system, which is based upon selectively choosing and combining from 'libraries' of input sensor devices, selectable content feedback, and 'libraries' of delivery methods. Besides traditional camera techniques used in multiple arrays, invisible 3D wireless sensor technology is used for sourcing natural movement that is mapped to personalised multimedia feedback. Robotics are also utilised as interactive content. The interaction is considered as playful activity that is both creative (music & art) and engaging (game playing). Both the creative and the engaging aspects target joy and fun for the participant (and facilitator). The mapped data is also routed and archived for post session analysis for session-to-session progress monitoring in therapy. An interest for the research is the investigation of compensatory sense dynamic that is often evident following a loss of another sense in the communities of people with different abilities. There are around 150 publications on the work, which has been presented globally at major events. Workshops, performances and presentations include the UN/NGO congress 1995; The Inaugural Cultural event for the Paralympics in Atlanta, USA in 1996 and the Millennium event in Sydney Australia (including the parallel scientific congress). It was featured in the European Cultural Capital events 1996 and was also featured in the NeWave festival in New York USA. Touring exhibitions at leading Museums for Modern Art have been ongoing since COIL (Circle of Interactive Light), Brooks' room size interactive installations, from 1998 where the public created the art (audiovisuals) through body gesture. A position on the co-ordination group of i3net (The European Network for Intelligent Information Interfaces) is another credit. SoundScapes was at the core of a European Future probe (www.bris.ac.uk/Twiaysi/) and the ensuing FW5 IST project titled CAREHERE (www.bris.ac.uk/carehere) as well as various other national and international projects. In 1999, Tony Brooks was awarded the European EUREKA award for SoundScapes as applied multimedia and in 2006 he received the Vanførefondens Forskerpris award as selected top Danish researcher in his field. Eva Petersson Brooks is an associate professor; coordinating/managing the Medialogy Bachelor and Master Education Program; and vice chancellor at Aalborg University Esbjerg in Denmark. She is member of the research group SensoramaLab (see http://sensoramalab.aaue.dk). She has a background in Education Science and her PhD is focused on ludic engagement (playfulness as a foundation for engagement) within virtual environments and the potentials in associated non-formal learning (see http://dspace.mah.se:8080/dspace/handle/2043/2963). Petersson has been coordinating research projects in inclusive and participatory design, storytelling, and creativity and learning processes through the use of new technology in education and rehabilitation contexts. She has been leading projects on physical and virtual toys, the design of toys for children with disabilities, and the use of computer games (interactive environments) in educational and rehabilitation contexts involving flexible methods of delivery for local, national and international users. Petersson is member of the International Toy Research Association (ITRA) and the Toys for Tomorrow Forum. Furthermore, she is a board member of the Pan-European Game Information (PEGI) and expert member of the Interactive Software Federation of Europe (ISFE). She was one of the founders of 48 FP7-ICT-2013-C SEEDIFF International Toy Research Conference recently organised by the International Toy Research Association (ITRA). Awards: (1) EUREKA AWARD Stockholm 1999 – The European Brokerage Event on Applied Multimedia (BAM) http://vbn.aau.dk/files/14567272/pdf (2) Vanførefonden research prize Copenhagen 2006, http://www.vanfoerefonden.dk/index.php/Anthony-Lewis-Brooks/1102/0/ Representative publications [of Aalborg] Brooks, A.L.: Human Computer Confluence in Rehabilitation: Digital Media Plasticity and Human Performance Plasticity. HCII2013, Las Vegas, USA (2013) Brooks, A.L.: Neuroaesthetic Resonance. In: G. De Michelis et al. (Eds.): ArtsIT 2013, LNICST 116, pp. 57--64. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, Milan (2013) Cobb, S., Brooks, A.L., Sharkey, P.: Virtual Reality Technologies and the Creative Arts in the Areas of Disability, Therapy, Health, and Rehabilitation. Telerehabilitation Health Informatics, pp 239-261 (2013) Brooks, A.L.: Active and Non-Active Volumetric Information Spaces to Supplement Traditional Rehabilitation. Journal of Research and Practice in Information Technology (2013) Brooks, A.L.: TeleAbilitation: GameAbilitation. Telerehabilitation Health Informatics pp. 225-237 (2013) Brooks, A.L.: Intelligent Decision-Support in Virtual Reality Healthcare & Rehabilitation. In: Studies in Computational Intelligence, Vol. 326, pp. 143-169 (2011) Brooks, A.L.: Arts and Technology. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer (2011) Brooks, A.L.: SoundScapes/ArtAbilitation - Evolution of a hybrid human performance concept, method & apparatus where Digital Interactive Media, The arts, & Entertainment are combined. Handbook of Digital Media in Entertainment and Arts. Springer Publishing Company, pp. 683-711 (2009) Brooks, A.L: Interpretations : an inter-sensory stimulation concept targeting inclusive access offering appreciation of classical music for all ages, standing, & disability. Proc. 7th International Conference on Disability, Virtual Reality & Associated Technologies (ICDVRAT) with ArtAbilitation. Reading University Press, pp. 15-22 (2008) Brooks, A.L.: Towards a platform of alternative and adaptive interactive systems for idiosyncratic special needs. 7th International Conference on Disability, Virtual Reality & Associated Technologies (ICDVRAT) with ArtAbilitation. Reading University Press, pp. 319-326 (2008) Hasselblad, S., Petersson, E., Brooks, A.L.: Empowered interaction through creativity. Digital Creativity, 18(2) pp. 89-98 (2007) Petersson, E., Brooks, A.L.: Non-formal Therapy and Learning Potentials through Human Gesture Synchronised to Robotic Gesture. Universal Access in the Information Society, 6(2) pp. 167-177 (2007) Brooks, A.L., Sharkey, P., Rizzo, A. Merrick, J.: Advances in virtual reality therapy and rehabilitation. International Journal on Disability and Human Development, 5(3) pp. 203-204 (2006) Williams, C., Petersson, E., Brooks, A.L.: Picturing Sound : an overview of its efficacy, ArtAbilitation, Aalborg Universitetsforlag, pp. 69-78 (2006) Brooks, A.L. Petersson Brooks, E.: Humanics 2 : Human Computer Interaction in Acquired Brain Injury Rehabilitation. 11th International Conference on Human Computer Interaction; the 3rd International Conference on Universal Access in Human-Computer Interaction, Las Vegas, Nevada, USA, Vol. 8 Lawrence Erlbaum Associates, Incorporated (2005) 49 FP7-ICT-2013-C SEEDIFF Brooks, A.L.: Interactive painting. An evolving study to facilitate reduced exclusion from classical music concerts for the deaf community. International Journal on Disability and Human Development, 4(4) pp. 293-299 (2005) Brooks, A.L. Petersson Brooks, E.: Play Therapy Utilizing the Sony EyeToy®. Presence 2005: The Eight International Workshop on Presence. Department of Computer Science, Aalborg University (2005) Brooks, A.L. Petersson, Brooks, E.: Raw emotional signalling, via expressive behavior. The 15th International Conference on Artificial Reality and Telexistence ICAT2005, pp. 133-141, (2005) Brooks, A.L. Petersson Brooks, E.: Recursive Reflection and Learning in Raw Data Video Analysis of Interactive ‘Play’ Environments for Special Needs Health Care. HEALTHCOM2005: 7th International Workshop on Enterprise Networking and Computing in Healthcare Industry. IEEE Signal Processing Society, pp. 83-87 (2005) Brooks, A.L. Krüger, V. Guglielmi, M.: A real-time tracking environment towards cross modal applications in public installations and education. Computers in Art, Design, and Education, København, Denmark (2004) Brooks, A.L: (1999) Virtual Interactive Space (V.I.S.) : As a movement capture interface tool giving multimedia feedback for treatment and analysis Eva selected publications – edit/reduce mine above as necessary Relative 3rd party publications/acknowledgements – (1) Tony Brooks Towards New Multisensory Spaces and Environments http://visualmusic.blogspot.ie/2011/11/tony-brooks-towards-new-multisensory.html?q=Tony+brooks (2) MULTISENSORY SPACES: THE FOUR SENSES PERFORMANCES WITH A SYMPHONY ORCHESTRA: A TRANSLATION OF SOUND INTO LIGHT, COLOR, AND SMELL. http://netzspannung.org/cat/servlet/CatServlet?cmd=netzkollektor&subCommand=showEntry&lang=en&entryId= 74131 (3) TV Documentation – Auckland, New Zealand, http://www.youtube.com/watch?v=gTjvCh-XB2o 50 FP7-ICT-2013-C SEEDIFF Participant 3: Ateneum Main tasks: Leader of WP7 (Dissemination). 1) Organizing art exhibitions for the blind and partially sighted at the Finnish National Gallery Ateneum. 2) Arranging in-depth visitor surveys, to research the experiences of persons using tactile and haptic devices in the exhibition. 3) Editing a book/ with tentative title “The Wounded Angel. Expanding human sensory experience”. Previous experience: Ateneum, located in the centre of Helsinki, has a long term and vast experience in organizing art exhibitions. When working with visitor groups in the visual field of an art museum, the museum has been a pioneer in using stimulation of other senses but vision: hearing, touching and smelling. The Ateneum team has got plenty of experience in working closely with the blind, e.g., producing the program of 12 art works in the Ateneum collection in detailed tactile images, combined with an audio (Pictures to Listen); exhibiting sculptures that visitors can touch in the art museum; and developing audio descriptions for 6 key art works in the collections of the museum (Treasures of Ateneum). As the National Gallery of Finland, Ateneum Art Museum houses the largest and most significant collection of art in Finland.*) The number of annual visitors varies from 200.000 to more than 400.000. In producing exhibitions, Ateneum is working in an active collaboration with major European art museums, such as the Picasso Museum in Paris and van Gogh Museum in Amsterdam. In 2000, Ateneum was awarded by Nordic Council of Ministers for excellent accessibility for the disabled visitors. The SEEDIFF exhibitions will be on display in an exhibition room in the middle of Ateneum’s collection premises, where they can be experienced with the majority of museum visitors during their three month’s duration. Anja Olavinen is an art historian and the Head of Education in the Finnish National Gallery Ateneum. She has contributed in many ways in organizing art exhibitions in the museum, and has written and produced several major educational and other publications, books and films. She's an expert of the entire work of Hugo Simberg, and has written and edited the catalogue for a comprehensive Simberg exhibition held in Ateneum in 2000. She will provide her expertise in analysing art as well as in arranging exhibitions for the blind and partially sighted, and will participate in the transformation of art works into tactile/haptic form together with Natasa Jovicic. With Erica Othman, she will be organizing in-depth visitor surveys, including interviews and video recordings of the experiences of the blind and other visitors in the haptic / 6D color exhibitions. Erica Othman is an art educator, working as the Educational Curator in Ateneum, and responsible for workshops in the museum. She is an expert in teaching art to a great variety of visitor groups, and has created “Pictures to Listen”, a program for blind and partially sighted. It was her final thesis in 1987 for the University of Industrial Arts and Design (nowadays Aalto University). She will be arranging in-depth visitor surveys for the blind and for sighted people. Riitta Nousiainen is a Project Manager in the Finnish National Gallery Ateneum, having a central role what comes to the financial and administrative functions in the museum. She’s a member of the museum’s management team, and she will be the Adminstrative Contact Person in the SEEDIFF project. Representative publications Olavinen, Anja, Paloposki, Hanna-Leena: Hugo Simberg 1873 – 1917. Exhibition catalogue, Helsinki 2000. Olavinen, Anja: Rikas elämä. Helene Schjerfbeck 1862–1946 [A Rich Life. Helene Schjerfbeck 1862–1946]. E-book, Helsinki 2012. 51 FP7-ICT-2013-C SEEDIFF Olavinen, Anja: Light and Shadow. The Life of Helene Schjerfbeck. A filmed biography of an artist, Helsinki 2012. 52 FP7-ICT-2013-C SEEDIFF Participant 4: Senseg Main tasks: (1) Providing and developing a platform where the touchscreen of a selected portable device is turned into ‘feel screen’ by adding high fidelity tactile sensations with Senseg’s breakthrough electrostatic tactile technology. (2) Examining and developing an experience demonstration software for making visual art tangible. Previous experience: Senseg’s technology has the potential to revolutionize the user experience for touch-based devices. Senseg’s tactile effects are created with electrostatic ‘vibration’ to convey the sense of physical textures, edges and contours. This technology adds another sense to the interaction with a touch device. Senseg’s breakthrough technology has received a number of international recognitions such as being the first Finnish invention to be named among the top 50 inventions by the TIME Magazine. Senseg has eight granted patents. Senseg was founded 2006; it has its main offices in Espoo Finland, the other main locations with local staff are Taipei, Taiwan, Seattle USA, and Tokyo Japan, in total having staff of around 25 persons. Senseg’s main investor is Ambient Sound Investments, which is a fund by the founders of Skype. Senseg’s business focus has been in bringing the technology to consumer markets and despite great interest from the academic and the visually impaired community towards Senseg’s technology, there have been resource limitations to support these interests. Nonetheless, Senseg has co-operated with Finnish Federation of the Visually Impaired and has also taken part in Haptic perception and interaction in mobile and multimodal contexts (HAPIMM), which is a joint research project between University of Tampere, Tampere Univ. of Technology and Stanford University, USA. As Senseg focuses on large consumer electronics customers, a spinoff company may be established that allows better approaching non-consumer markets and user experience concepts with a wider approach. Dr. Ville Mäkinen is a physicist, neuroscientist, and entrepreneur. He founded Senseg in June 2006. Having created Senseg’s initial customer base in the field of medical technology, Dr. Mäkinen went on to develop the innovations that underlie Senseg’s solution along with the growing of the company. Dr. Mäkinen led the team that raised Senseg’s initial funding from Ambient Sound Investments, Avera – the Finnish National Seed Fund, and the Finnish Funding Agency for Technology and Innovations. Dr. Mäkinen carried out his doctoral studies (Approved with distinction, 2006) in Helsinki University of Technology in the area of signal processing, medical physics. Dr. Mäkinen has 23 peer-reviewed articles in the field of brain research, an area that he keenly follows despite pursuing an entrepreneurial career. Dr. Jukka Linjama has 15 years of technology development experience, followed by 8 years at Nokia as a haptics specialist and concept designer. He has contributed to a variety of scientific publications ranging from acoustics to human-computer interaction and has 10 granted patents to his credit. Dr. Linjama devotes his time to scientific, industrial and artistic projects. Since joining Senseg in 2007, he has been responsible for the humancentric development of Senseg’s offering, a unique haptic solution for touch interfaces. Representative publications: Being a company, Senseg is better described by its exceptional media visibility than by scientific publications. Senseg has been featured or awarded in for example the following media: 53 FP7-ICT-2013-C SEEDIFF Participant 5: University of Split (UNIST) Main tasks: 1) Behavioral studies on the perceptual experience of visual art and when blind subjects use Senseg’s technology. 2) Data analysis in neuroimaging studies. 3) Reaction-time studies linking behavioral and neurophysiological data. 4) Addressing acceptance of the created art forms in educational settings. Previous experience: Coordination of EU-funded and national projects, summer schools, conferences; Vast experience on behaviour studies; Successful development and production of biomedical equipment in clinical neurophysiology(http://lahen.org/); long-term cooperation with local Association of Blind through community oriented medical education projects. University of Split (established in 1974) is the fastest growing university in Croatia, particularly in research in physics, medicine, and history and art. It has more than 25000 students at 13 faculties and 4 education centres. The School is the centre of excellence for evidence-based medicine, with the Croatian Branch of the Cochrane Collaboration, Croatian Centre for Global Health, and Clinical Hospital Centre as its integral parts. Its most productive research groups are in cancer biology, applied physiology, and neurophysiology. The School has several neuroscience laboratories: Human and Experimental Neurophysiology, Speech and Hearing Science, Physiology, Basic Neuroscience, and Clinical Neuroscience with Sleep Lab. Researchers at the School are actively participating in numerous national and international projects and collaborations, funded by, e.g., TEMPUS, FP7, LLP, IPA, and COPE. Ana Jeroncic is Assistant professor and Vice Chair of Department of Research in Biomedicine and Health at the School of Medicine. Her research interests are in bio- and neuroinformatics. As a Marie Currie fellow, she used EEG and computational approaches to develop automatic detection of sleep spindles in infants and adults. She explored topics from molecular to system levels and acquired wide programming and computational skills. SEAEAS, EU PHARE 2005 project that she coordinated (2008) has been elected a Croatia’s success story. Andreja Bubic is Assistant Professor at the Faculty of Social Sciences and Humanities at UNIST. She received her PhD at the Max Planck University for Human Cognitive and Brain Sciences and the University in Leipzig, where she used fMRI and EEG to study visual perception and the function of the premotor cortex. After a fellowship in the Laboratory for Multisensory research in Jerusalem and a postdoctoral period in Martinos Centre in Boston, she now lectures in Split and does research in the field of psychology and cognitive science. Representative publications of UNIST Rogic, M., Jeroncic A., Bosnjak, M., Sedlar, A., Hren, D., Deletis, V.: A visual object naming task standardized for the Croatian language: A tool for research and clinical practice, Behav. Res. Meth. 01/2013 Epub: PMID: 23344740 Jan 2013 Boraska, V., Jeroncic, A., et al.: Genome-wide meta-analysis of common variant differences between men and women. Human Molecular Genetics, 21(21) 4805-4815 (2012) Striem-Amit, E., Bubić, A., Amedi, A.: Neurophysiological mechanisms underlying plastic changes and rehabilitation following sensory loss. In: Murray, M.M. Wallace, M.T (Eds.) The Neural Bases of Multisensory Processes. CRC Press (2012) Bubić, A., von Cramon, D.Y., Schubotz, R.I.: Exploring the detection of associatively novel events using fMRI. Human Brain Mapping, 32, 370-381 (2011) Bubic, A., Striem-Amit, E., Amedi, A.: Large-scale brain plasticity following blindness and the use of sensory substitution devices. In: J. Kaiser and M. Naumer (Eds.) Multisensory Object Perception in the Primate Brain. Springer (2010) Bubić, A., von Cramon, D.Y., Schubotz, R.I.: Prediction, cognition and the brain. Front. Hum. Neurosci., 4 (2010) Bubić, A., Bendixen, A., Schubotz, R.I., Jacobsen, T., Schröger, E.: Differences in processing violations of sequential and feature regularities as revealed by visual event-related brain potentials. Brain Research, 1317, 192-202 (2010) 54 FP7-ICT-2013-C SEEDIFF Participant 6: University of Zagreb (UNIZG) Main tasks: 1) Tactile coding and auditory interpretation of works of art; 2) Behavioral and neurophysiological studies on the tactile and tactile-auditory perception of the blind and blindfolded participants; 3) Novel time series and pattern recognition analysis of the neurodynamics multi-sensory integration in blind and sighted. Previous experience: Coordination/participation in EU-funded and other international and national projects, organization of conferences, summer schools; strong expertise in the human brain development and neurophysiological studies using MEG, EEG, MRI and fMRI. The University of Zagreb is the leading Croatian university encompassing all fields of science and art. UNIZG is strongly committed to building a modern and innovative university through stimulating research excellence and facilitating the transfer of knowledge into the business sector. In addition to its traditional strengths in elementary particle physics, nuclear, and solid state physics, Department of Physics increasingly supports interdisciplinary research in new materials, nanotechnology, biophysics, and cognitive neurodynamics. Selma Supek is Assistant Professor of physics and biophysics at the Department of Physics, co-founder and codirector of the first UNIZAG international interdisciplinary postgraduate program in Language Communication and Cognitive Neuroscience (2001–2004), founder of the Mind and Brain series at the InterUniversity Centre (IUC), Dubrovnik, Croatia (www.brain.com.hr), and a member of the UNIZAG Coordination for Biomedical Engineering. Since her doctoral thesis research at Los Alamos National Laboratory her research interest is related to MEG studies of the human perception and cognition. Natasa Jovicic is an art historian, director of the Holocaust Museum in Jasenovac, Croatia, with a master degree in art and multicultural education (Columbia College Chicago). In 2009, she opened one of the first tactile galleries in Europe within the Modern Gallery in Zagreb, based on her inovative tactile-auditory methodology. Since then, she organized 12 exhibitions and presented her work in major museums (Ateneum, Georges Pompidou, Tate Modern, Guggenheim, New York, Art Institute, Chicago). Ana Susac is Senior Lecturer at the Department of Physics with strong research experience in EEG and MEG studies of the neurodynamics of face processing and research interest in educational neuroscience including behavioral and neurophysiological studies. She spent research periods at the BioMag Lab in Helsinki, MEG Lab in Jena, and postdoctoral research at the University of Oxford. Davor Horvatic is Assistant Professor at the Department of Physics. He obtained PhD in high energy physics and acquired wide range of theoretical, statistical, and numerical skills. During last few years, he concentrated his research in the field of complex systems and time series analysis, particularly related to the physiological signals. He collaborates with H.E. Stanley’s group at Boston University. Representative publications of UNIZG Susac A, Ilmoniemi R, Supek S. Face activated neurodynamic cortical networks, Med. Biol. Eng. Comput. 49, 531-43, 2011. Sušac, Ana; Ilmoniemi, Risto; Pihko, Elina; Nurminen, Jussi; Supek, Selma. Early dissociation of face and object processing: A magnetoencephalographic study. Human Brain Mapping. 30 (3): 917-927, 2009. Supek, S., Aine, C., Ranken D., Best E., Flynn E.R., Wood C.C.: Single vs paired visual stimulation: Superposition of early neuromagnetic responses and retinotopy in extrastriate cortex in humans, Brain Research, 830: 43-55, 1999. Aine, C.J., Supek, S., George, J.S., Ranken, D., Lewine, J., Sanders, J., Best, E., Tiee, W., Flynn, E.R., and Wood, C.C.: Retinotopic organization of human visual cortex: Departures from the classical model. Cerebral Cortex, 6:354-361, 1996. 55 FP7-ICT-2013-C SEEDIFF D. Horvatic, H. E. Stanley, and B. Podobnik, "Detrended Cross-Correlation Analysis for Non-Stationary Time Series with Periodic Trends," Europhys. Lett. (EPL) 94, 18007 (2011). 56 FP7-ICT-2013-C SEEDIFF 2.3 2. 4 Consortium as a whole Describe how the participants collectively constitute a consortium capable of achieving the project objectives, and how they are suited and are committed to the tasks assigned to them. Show the complementarity between participants. Explain how the composition of the consortium is well balanced in relation to the objectives of the project. If appropriate, describe the industrial/commercial involvement to ensure exploitation of the results. 2.4.1 Consortium overview and role of the participants The consortium has been formed in order to reach the main objectives: to develop new methodology and technology for expanding senses for blind, color-blind and sighted people. Senseg and Aalto provide the necessary technical expertise. Drs. Mäkinen and Ilmoniemi are both physicists and their teams include mathematically highly skilled people with the ability to develop and implement new algorithms and software for the purposes of the project. UNIZG and Aalto have world-leading expertise and long-standing collaboration in neurophysiological experimentation, in particular with regard to studies of vision and other sensory systems. UNIST and UNIZG have the expertise for studies of the blind … [Please continue]. One of the key people at UNIZG, the art historian Natasa Jovicic, has worked extensively with the blind, developing combined haptic relief and audio presentation for the blind. She works constantly with the blind; one of the employees in SEEDIFF is the soon-to-be psychologist Ivica … who is blind himself and who has been working with Natasa Jovicic on experimenting with the haptic displays. UNIST and Aalborg have background in behavioral studies and [please continue] Ateneum and Aalborg as well as UNIZG provide SEEDIFFs major interface to the public, the museum exhibits. [CONTINUE] 2.4.2 Complementarity of participants The project partners and advisors have expertise ranging from information and communication technologies, physics, psychology, and neuroscience to art history, pedagogy/educational science, and media technologies. 57 FP7-ICT-2013-C SEEDIFF 2.4.3 Industrial involvement and exploitation of the results Assuming that the hexachromatic display technology will attract end-user or industrial interest, Aalto will aim at commercializing the technology. Any IPR will be protected prior to making the information public. Aalborg [should write something here about their exploitation of the results] Ateneum will work together with the other partners in order to explore possibilities for making the exhibitions for the blind suitable for wider distribution (art museums, science centers). Senseg or a spin-off company will aim at commercializing the haptic display technology for the blind. UNIST [should write something here about their exploitation of the results] UNIZG [should write something here about their exploitation of the results] Croatian partners are included as Ditacta prototype was developed by Natasa Jovicic’s, Croatian art historian. [The different efforts to utilize the results should be integrated] 2.4.4 Subcontracting Subcontracting: If any part of the work is to be subcontracted by the participant responsible for it, describe the work involved and explain why a subcontract approach has been chosen for it. Aalto has budgeted 3000 EUR for auditing costs. Ateneum has budgeted 8000 EUR for xxx. Senseg has budgeted 44000 EUR for xxx and 3000 EUR for auditing costs. UNIZG has budgeted 32000 EUR for xxx. 58 FP7-ICT-2013-C SEEDIFF 2.5 Resources to be committed In addition to the costs indicated on form A3 of the proposal, and the staff effort shown in section 1.3 above, please identify any other major costs (e.g. equipment). Describe how the totality of the necessary resources will be mobilised, including any resources that will complement the EC contribution. Show how the resources will be integrated in a coherent way, and show how the overall financial plan for the project is adequate. (Recommended length for Section 2.4 – two pages) 2.5.1 Use of the resources The progress of SEEDIFF will be based on technical development work by Senseg (haptic display technology, WP2) and by Aalto (hexachromatic methodology, WP3). A crucial part of this development will be in userinterface, usability, and image transformation (from visual to haptic, from tri- to hexachromatic) and presentation software. The development of software and the necessary algorithms will be guided by interviews, subject observation and behavioral studies (WP4, WP6 and WP7), the methodology of image transformation (WP4) and neurophysiological studies (WP5). Dissemination of results (WP7) will be arranged so that we can obtain user experience data that we can use as a guide when we design behavioral and neurophysiological studies in the laboratory or when we design user interfaces and develop image transformation methodology. 2.5.2 Equipment resources Aalto will make available for the project its (i) professional equipment for photography as well as (i) MEG and EEG facilities (at the joint research unit BioMag Laboratory of the Helsinki University Central Hospital) for neurophysiological studies of visual, tactile, and auditory perception. Aalborg... Ateneum... Senseg... UNIST... UNIZG... 2.5.3 Other major financial resources [If none, we can delete this subsection] 59 FP7-ICT-2013-C 3 3.1 SEEDIFF Impact Expected impacts listed in the work programme Describe how your project will contribute towards the expected impacts listed in the work programme in relation to the topic or topics in question. Mention the steps that will be needed bring about these impacts. Explain why this contribution requires a European (rather than a national or local) approach. Indicate how account is taken of other national or international research activities. Mention any assumptions and external factors that may determine whether the impacts will be achieved. 3.1.1 Impact on the competitiveness of the proposers As the National Gallery of Finland, Ateneum is one of the leading actors in the cultural and art field in Finland, and a leading art museum in the country. Also having being awarded by Nordic Council of Ministers for the exemplary accessibility as an art museum in 2000, we have the responsibility to stay in the lead of development in the service of all of our visitors. Croatian partners are included as Natasa Jovicic (UNIZG) introduced her Ditacta method to the blind public in Croatia and has already gained some experience and partners in Croatia that are important for the project implementation. 3.1.1.1 Direct applications and market prospects 3.1.1.2 Potentially patentable ideas 3.1.1.3 Benefits and competitive advantages Equality is in the core of the strategies of the Finnish National Gallery Ateneum, and reaching that goal means constant effort and improvement for an institution. To stay in a certain level without improving means declining and falling, when compared with other actors in the art field in Finland, in Europe, and globally. SEEDIFF partnership will give Ateneum the keys to outstanding technical and cultural development for our customers’ service in the future. 60 FP7-ICT-2013-C 3.1.1.4 Economic justification 3.1.2 Strategy for impact achievement 3.1.3 European dimension 3.1.3.1 European problem to be solved 3.1.3.2 Effects on transnational co-operation SEEDIFF The effects of transnational and also interdiciplinary co-operation are crucial, what comes to the effectiveness of different partners of the SEEDIFF project. The benefits of the co-operation for them lie in the interchange and collaboration of actors, who have very different kinds of expertises, but at the same time, goals of mutual, shared interests. The effects of cooperation of SEEDIFF partners for the European community, on the other hand, lie in the increase of mutual understanding, and foremost, outcomes of the synergy: in the creating of revolutionary new technical and cultural solutions, which would not be possible, if the partners would be working alone. 3.1.3.3 Implementation and evolution of EU policies 3.1.3.4 Improvement of European social and economic cohesion 61 FP7-ICT-2013-C SEEDIFF 3.1.4 Contribution to Community societal objectives 3.1.4.1 Quality of life potential in SEEDIFF outcomes where the realised displays are applied for adaptive uses across alternative rehabilitation/therapeutic training situations and end-users. So this can be added as a future research potential. What comes to the art museums, the blind and partially sighted visitors have hitherto been highly dependant of the museum personnel. There are few possibilities for them to act in a museum like any other visitor. For instance touching the sculptures needs consultation with the personnel, who will provide the visitor with gloves and other equipment. The visually impaired visitor also needs an assistant – she/he typically is not allowed to touch the art works alone. The availability of haptic displays in museums and elsewhere will give the possibility for the blind to explore art and other visual information without an assistant. Independency is a basic right and need of any adult person. 3.1.4.2 Health and safety For a blind or partially sighted person the possibility to use a haptic display when leaving home, when travelling and working, and when having free-time and holidays, means an experience of freedom and certainty that has never been in the reach of these people before. One of the most important features of possible outcomes of the SEEDIFF project will be the benefits it will bring for the visually impaired of the knowledge of one’s location, and the possibility to read maps in internet pages, then accessible by means of the haptic display. 3.1.4.3 Employment 3.1.4.4 Gender issues SEEDIFF is gender neutral, in the sense that both men and women may equally participate in any of its activities and application of the results. SEEDIFF outputs will be gender neutral. Both female and male subjects and patients are studied. The Coordinator and the Steering Committee will oversee and ensure that all Participants are aware of harms caused by gender bias within the project or elsewhere. At each meeting, the Steering Committee will make an assessment of the gender balance within the project and, if needed, will implement actions in order to correct any unjustified changes. 3.1.5 Other relevant European or National funded research 62 FP7-ICT-2013-C 3.1.6 3.2 SEEDIFF Influence of external factors Dissemination and/or exploitation of project results, and management of intellectual property Describe the measures you propose for the dissemination and/or exploitation of project results, and the management of knowledge, of intellectual property, and of other innovation related activities arising from the project. 3.2.1 Exploitation and dissemination plan for use of project results 3.2.1.1 c 3.2.1.2 Validation of the technology 3.2.1.3 Dissemination of results and technology transfer A special task is devoted to the exploitation of the results. All the Participants of the consortium are involved in this work package. Exploitation plans given here will be modified according to the results of the market evaluation performed during the first months of the contract in order to maximise the exploitation of the results. 63 FP7-ICT-2013-C SEEDIFF Haptic systems Senseg has over the years develop and understanding of the perceptual capabilities of humans in tactile realms. While user experience enhancements are readily achieved with Sense technology, this does not provide the basis for directly accessing the needs of the visually impaired. The SEEDIFF project enables developing interactive approaches for making art tangible, which solves the general limitations of traditional haptic approaches in tactile displays. The development and research at SEEDIFF can lead to two alternative paths with their different impacts and business possibilities. A) Interactive tactile display technology capable of serving any museum or art gallery in the world specifically targeted to provide more engaging tangible experience to: visually impaired, children and 3 all people looking for immersive experience. B) Applications that can be installed to any smartphone/tablet enabled with Senseg technology or haptic capabilities in general which allow presenting artwork in tangible format. Regarding the path A) both the commercial and the social potential for creating tangible art is immense, for example there are over 1500 museums in the world with over 350 daily visitors (Exhibition and museum attendance figures 2011 (page 2)" (PDF). London: The Art Newspaper. April 2011). To get scope of this scale only one museum in Finland makes it to this list, and there over 300 professional museums that are maintained over the year. Each museum further number of installations therefore there is market opportunity to tens or hundreds of thousands speciality tactile displays in museums and art galleries. With SEEDIFF project, a platform and understanding can be build whereby all of these museums could be turned in to places which would readily provide an experience also for the visually impaired, and would also provide an attractions for the children. The social impact has viral aspect, taking a key hold in few key museums in the world can lead to general expectation that museums are in general expected to cater also the visually impaired. This viral aspect will also create excellent commercial possibilities. Each museum tangible display is custom made can be built to a large size, eg. size of the original art work when ever feasible, and the experience build together with exhibitors. This model has excelled fit to an an European business combining expert services and high technology. Therefore with the successful completion of SEEDIFF it is conceivable to expect that it there is sufficient knowledge to start distributing a platform that can lead in few years to situation where it is expected that all high quality museums also have a tangible displays that make the art accessible also for the visually impaired. The other alternative path B) is to build software applications based on knowledge learned at SEEDIFF that allows presenting tangible presentations of art at consumer level devices that have Senseg technology enabled. The size of the display limits the possibilities of the user experience but would provide the museum indepent access to visually impaired at convenience without the museums. The experience would be more limited then case A, but should the Senseg technology be readily available to portable devices at the end of SEEDIFF it would be natural to also follow this path. The arduous task of finding the representations how to make visual imagery tangible is one key task of SEEDIFF project that would be the basis for the planned application. Our dissemination strategy relies on four aspects. 64 FP7-ICT-2013-C SEEDIFF 1) The publication of scientific breakthroughs in high impact journals, communications at international conferences, etc. 2) A web-based information data base will be elaborated and maintained during the project containing all relevant information required for the implementation of the Work Plan, at the pre-authorised various levels of confidentiality: reference documents (contract, Consortium Agreement), meetings announcement and minutes, planning advancement, financial data (financial distribution & reporting), publications and reports under preparation/reviewing. For these reasons, a three-level confidentiality structure will be established: Public information, Project Participants, and Evaluators (Executive Committee members, finance, reviewing, etc.), all data being protected by firewalls. This platform will be accessible to all members via the internet allowing realtime communication. 3) Exhibitions 4) Media 5) Finally, a workshop at the end of the project will constitute a major dissemination of results. 3.2.2 Management of knowledge and intellectual property In this project, patent protection will be sought as new IPR emerges. The ownership of any inventions, techniques, or methodologies will remain at the inventor organization, which is also responsible for applying for patent. Regarding exploitation of patents with joint inventors, the distribution of ownership will reflect the level of research contributions to the patented invention(s) of the different inventors and their laboratories. This will be decided by mutual agreement under supervision of the SEEDIFF Coordinator. In addition, a basic training on quality and intellectual property management will be proposed during the first phase of the program to all the project Participants by a person responsible for patent. An IPR management plan will be included in the Consortium Agreement. Pre-Existing Knowledge to be included/excluded in/from the project will be identified and documented in an annex to the Consortium Agreement. In particular, the Consortium Agreement will describe how the results of the project may be transferred from the project or its Participants to the industrial Participant and other parties. 65 FP7-ICT-2013-C 4 SEEDIFF Ethical Issues High ethical standards will be followed in all phases and aspects of the project. Care will be taken to ensure that all human studies are safe for the volunteer subjects, whether blind, partially sighted or sighted, and that the instruments will be designed, built, and used in conformity with the regulations of European Union. We strongly believe that the benefits of the present study will justify the cost and effort of the project. 4.1 Benefits of the present study and experiments A large number of blind or visually impaired people have no or limited access to xxx. This project will help in xxx and provide xxx to a large number of people. For example, …. 4.2 Risks of the proposed approach? [are there any?] . 4.3 Approvals by ethical committees and informed consent All human studies within the project will be performed only after appropriate ethical approvals have been obtained. The ethical committees and regulatory organizations in the Participant countries that will need to be approached during the life of the project are listed below: Finland: 1) Coordinating Ethics Committee, Hospital District of Helsinki and Uusimaa, 2) Research Ethics Committee, Aalto University. Denmark: ?? Croatia: 1) Research Ethics Committee of University of Split School of Medicine. 2) Ethics Committee, School of Medicine, University of Zagreb Each subject (blind, partially sighted, or sighted) to be studied will be informed orally or in writing about the purpose, execution, possible side effects or risks of the study and about the right to leave the study at any time without any negative consequences. Each subject who, after being explained orally or in writing about the study as explained above, wishes to participate in the study will be asked to sign an informed consent form prior to the beginning of the study. No subject will be studied without informed consent. Both the written information leaflet informing about the study and the informed consent form will be the ones approved by the ethical committee in question. 4.4 Data protection issues All subject or patient data will be coded after the study session so that subject identity cannot be determined from any of the data files. The data files are labeled and the software identifies the subjects and patients only by the code, which can be related to the individual only by the investigator in question. ETHICAL ISSUES TABLE 66 FP7-ICT-2013-C SEEDIFF YES Informed Consent Does the proposal involve children? Does the proposal involve patients or persons not able to give consent? Does the proposal involve adult healthy volunteers? Does the proposal involve Human Genetic Material? Does the proposal involve Human biological samples? Does the proposal involve Human data collection Research on Human embryo/foetus Does the proposal involve Human Embryos? Does the proposal involve Human Foetal Tissue / Cells? Does the proposal involve Human Embryonic Stem Cells? Privacy Does the proposal involve processing of genetic information or personal data (e.g. health, sexual lifestyle, ethnicity, political opinion, religious or philosophical conviction)? Does the proposal involve tracking the location or observation of people? Research on Animals Does the proposal involve research on animals? Are those animals transgenic small laboratory animals? Are those animals transgenic farm animals? Are those animals cloned farm animals? Are those animals nonhuman primates? Research Involving Developing Countries Use of local resources (genetic, animal, plant etc) Benefit to local community (capacity building i.e. access to healthcare, education etc) Dual Use Research having direct military application Research having the potential for terrorist abuse I CONFIRM THAT NONE OF THE ABOVE ISSUES APPLY TO MY PROPOSAL 67 X X - - - - - PAGE FP7-ICT-2013-C SEEDIFF Comments:- Museum showcase and positing an angle of building on prior work that can sometimes get some add on value In section 1.1.1 - you ask which art museums. Maybe this should be at least one in each partner country and we have a strong cooperation with our regional art museums leadership board who are keep to work with us and I can approach them to ask about a SEEDIFF exhibition when appropriate. We also have links to other leading museums in Denmark should a wider dissemination be required. I see no problems in this to showcase in Denmark. Also, I feel it appropriate to say that I see this work in a way related to the Neuroesthetic studies/activities being published (Zeki and others). I published (only a position paper) on Neuroaesthetic Resonance as my previous research enquired to aesthetic resonance (involving two EU projects and an i3 future probe (briefly outlined below inc URLs) - and in a way I see SEEDIFF building on some of the embedded concepts from those projects) = (1) CARESS - CREATING AESTHETICALLY RESONANT ENVIRONMENTS IN SOUND; The purpose of CARESS was to create technological and educational tools that will motivate and empower children to develop creativity, imagination and expression, through interactive acoustic environments. (2) CAREHERE - CREATING AESTHETICALLY RESONANT ENVIRONMENTS FOR THE HANDICAPPED, ELDERLY AND REHABILITATION. The objective of our project is to empower children and adults with special needs, the elderly in long term care and people undergoing rehabilitation in hospital or at home, following for example stroke or brain injury. By giving them access to affordable, appealing and readily usable state of the art technology for the improvement of their physical and cognitive skills using feedback from acoustic and visual stimuli. We are concerned with the (re-)development of physical and cognitive skills by interaction with a responsive sound and visual environment: the improvement of motor control through direct and immediate feedback through the aural and visual senses. (3) - The i3.net future probe Twi-aysi = The World is as you see it. While CARESS successfully motivated and empowered children to develop creativity, imagination and expression, through interactive acoustic environments. The objective of Twi-aysi was to answer the question: Can immersion in a visual environment hold similar potential for such children in terms of the aesthetic resonance they might derive from movement within such a visual space? The success of Twi-aysi led to the CAREHERE project (above). INSTRUCTIONS 68 FP7-ICT-2013-C SEEDIFF Section 1 (20 pages, not including the tables ): Explain the concept of your project. What are the main ideas that led you to propose this work? Describe in detail the S&T objectives. Show how they relate to the topics addressed by the call. The objectives should be those achievable within the project, not through subsequent development. They should be stated in a measurable and verifiable form, including through the milestones that will be indicated under section 1.3 below. R = Report, P = Prototype, D = Demonstrator, O = Other Please indicate the dissemination level using one of the following codes: PU = Public PP = Restricted to other programme participants (including the Commission Services). RE = Restricted to a group specified by the consortium (including the Commission Services). CO = Confidential, only for members of the consortium (including the Commission Services). Delivery date Measured in months from the project start date (month 1). 69 FP7-ICT-2013-C SEEDIFF Section 1.2. Describe the state-of-the-art in the area concerned, and the advance that the proposed project would bring about. If applicable, refer to the results of any patent search you might have carried out. Section 1.3. A detailed work plan should be presented, broken down into work packages* (WPs) which should follow the logical phases of the implementation of the project, and include consortium management and assessment of progress and results. (Please note that your overall approach to management will be described later, in section 2). Please present your plans as follows: i) Describe the overall strategy of the work plan. ii) Show the timing of the different WPs and their components (Gantt chart or similar). iii) Provide a detailed work description broken down into work packages: - Work package list (please use table 1.3a); - Deliverables list (please use table 1.3b); - Description of each work package, and summary (please use table 1.3c) - Summary effort table (please use table 1.3d) - List of (please use table 1.3e) iv) Provide a graphical presentation of the components showing their interdependencies (Pert diagram or similar) Note: The number of work packages used must be appropriate to the complexity of the work and the overall value of the proposed project. The planning should be sufficiently detailed to justify the proposed effort and allow progress monitoring by the Commission. Any significant risks should be identified, and contingency plans described. * A work package is a major sub-division of the proposed project with a verifiable end-point - normally a deliverable or a milestone in the overall project. Section 3 (Recommended length for the whole of Section 3 – ten pages) Sectdion 4. Describe any ethical issues that may arise in the project. In particular, you should explain the benefit and burden of the experiments and the effects it may have on the research subject. Identify the countries where research will be undertaken and which ethical committees and regulatory organisations will need to be approached during the life of the project. 70