e-Science Collides with the Performing Arts Dr R P Fletcher University of York So what is this e-Science?? At the inception of the UK e-Science programme John Taylor said that: “e-Science is about global collaboration in key areas of science and the next generation of infrastructure that will enable it” Another source says it is: “science increasingly done through distributed global collaborations enabled by the Internet, using very large data collections, terascale computing resources and high performance visualisation “ Malcolm Atkinson has said that: “e-Science is the development and exploitation of advanced informatics methods to support collaborative research in all disciplines “ Thursday, 20 July 2007 Digital Representations of Performing Arts 2 OVERVIEW Academic Background – am I a real scientist? Involvement in Performing Arts – am I a fraud? Involvement with technology – am I a technologist? Technology Bespoke hardware and software Computers Networks Software Engaging with this technology Right people in the right place Future – some ideas Questions – whatever you want to throw at me! Thursday, 20 July 2007 Digital Representations of Performing Arts 3 My Academic background 1968 Degree in Biology General 1971 D.Phil in Neurophysiology Information processing in single sensory nerve cells 1975 Research Fellow in Psychology Risk of schizophrenia Psychophysiology 1981 Computing Service Graphics Database systems Programming Mathematical and statistical systems and libraries Thursday, 20 July 2007 Digital Representations of Performing Arts 4 Academic Background Involvement in Performing Arts My involvement with technology Technology Bespoke hardware Computers Networks Software Engaging with this technology Right people in the right place Future Questions Thursday, 20 July 2007 Digital Representations of Performing Arts 5 My “Performing Arts” background 1965 Staged school bands 1968 Staged University bands 1971 President of the Electronic Arts Society 1972 Recording engineer York Musicians Cooperative 1972 Worked with Trevor Wishart 1973 Recording/mixing engineer campus band 1974 Built own synthesiser system 1976 Programmed FX on PDP-11 1978 Recording/mixing engineer campus band (again) 1991 SGI Indigos bootstrap Music Technology 1996 Chaotic Constructions 1999 RIMM 2005 UKLight for Collaborative Composition/Performance 2006 PEACHES committee of TERENA 2007 Content Creation tools for Digital Cinema? Thursday, 20 July 2007 Digital Representations of Performing Arts 6 Technology Input and Output Specialised software to edit synthetic speech User interfaces for Music Technology Courses in graphics for Music Technology External Examiner MA Digital Arts (Middlesex) Midi I/O systems for Sun Workstations Developer for GIMP (video plugins for Unix) Developer for XMX (collaborative X-Windows) Developer for .ogg audio formats Still ambisonic developments Tracking technology for Chaotic Constructions Graphics system designer/developer for RIMM Real-time Interactive Multiple Media PEACHES committee at TERENA Panel of Experts for Arts, Culture and Humanities E-Science Thursday, 20 July 2007 Digital Representations of Performing Arts 7 Academic Background Involvement in Performing Arts My involvement with technology Technology Bespoke hardware Computers Networks Software Engaging with this technology Right people in the right place Future Questions Thursday, 20 July 2007 Digital Representations of Performing Arts 8 Technologies Aim to lead you through some technologies Ones I have worked on over the last 15 years (or so!) At some point technology overlaps e-Science Need to engage with the new technologies Need to help drive their development Performing Arts need new/different facilities either from the existing infgrastructures or via new facilities But, unless we know what’s possible How do we know what to ask for? Thursday, 20 July 2007 Digital Representations of Performing Arts 9 YORKTALK Worked with the Language and Linguistics staff Their technology allowed about six synthesis runs PER DAY! Data transferred via Kermit file transfer program! Real informed trial and error to fix problems I created a graphical interface to the data Ran on a then high performance system Could interact with the data Synthesis was up to near real time Combined visualisation to “see” errors Used interactive graphics to “fix” the problems Thursday, 20 July 2007 Digital Representations of Performing Arts 10 YORK TALK Research vehicle for speech synthesis DecTalk example (may recognise the voice) YorkTalk examples Same words as DecTalk A sentence (with a cold!) Thursday, 20 July 2007 Digital Representations of Performing Arts 11 Engaging with “artists Next project was with staff in Music Needed help to create interfaces for a system Needed some lateral thinking Definitely moving into the Arts domain High performance computing was needed Multiple tasks by same computer Video Audio Graphics Midi Thursday, 20 July 2007 Digital Representations of Performing Arts 12 Chaotic Constructions - 1994 Certain amount of serendipity Peter Fluck talking on Radio 4 about Chaotic motion in kinetic scuplture Tony Myatt talked a few weeks later about Chaos in Music Phoned each other classic “WE SHOULD TALK” moment By 1996 developed a project to use the chaotic motion of a kinetic sculpture to drive a music composition system PROBLEM – how to capture the motion ANSWER – ask me Thursday, 20 July 2007 Digital Representations of Performing Arts 13 Chaotic Constructions – the Tracking System Request came to me How can we do this? and … It will be installed in Tate Modern in March 1997 and … It was now January 1997! First thoughts were to use shape tracking systems Problems: most systems were proprietary and expensive shapes to be tracked changed because sculpture was 3 dimensional could not use trajectory prediction due to the chaotic motion ANSWER – simply track unique coloured items Thursday, 20 July 2007 Digital Representations of Performing Arts 14 VIDEO EXAMPLE The Making of CHAOTIC CONSTRUCTIONS Thursday, 20 July 2007 Digital Representations of Performing Arts 15 Stage 1 – Define a Colour “Fingerprint” Program creates a set of “unique fingerprints” Thursday, 20 July 2007 Digital Representations of Performing Arts 16 Tuning the “Fingerprints” Check motion does not lose sync with colour Thursday, 20 July 2007 Digital Representations of Performing Arts 17 Peter Fluck Hand Cranking the Sculpture Thursday, 20 July 2007 Digital Representations of Performing Arts 18 Running with Tracking and Music Program “Fingerprint” used by second program Motion tracking is just one element Thursday, 20 July 2007 Digital Representations of Performing Arts 19 Changing a Tracking Point What are the best points to track? Due to chaotic motion some trial and error! Thursday, 20 July 2007 Digital Representations of Performing Arts 20 And finally …the prototype … we get there … a small excerpt Thursday, 20 July 2007 Digital Representations of Performing Arts 21 And the real thing Thursday, 20 July 2007 Digital Representations of Performing Arts 22 Credits and Awards Exhibited to great acclaim in Tate Modern Other UK Art Galleries Travelled to Hong Kong and Corsica Herald Angel Award by the Glasgow Herald for the best piece of visual art at the 1997 Edinburgh Festival International Critics Award as part of the 1997 Demarco European Art Foundation exhibition Thursday, 20 July 2007 Digital Representations of Performing Arts 23 Not really “Performing Arts” I hear you say In St. Ives staff gave children coloured “bats” Switched off the sculpture Children waved the bats around And, of course, “made music” Security was called once to “remove” a visitor Would not leave at closing time Transfixed by the whole thing Some people came 2 or 3 times a week For a “fix” Thursday, 20 July 2007 Digital Representations of Performing Arts 24 How can this Technology be used … … for Performing Arts Mount a camera above a stage Give actors coloured hats Now you have their position on stage Link their movement around the stage To … Orchestrate lighting changes Hotspots Move to one place on stage and make a sound Detect “collisions” Allow actors to fully interact with their environment Thursday, 20 July 2007 Digital Representations of Performing Arts 25 …or … Point the camera at a tropical fish tank Let the fish make music Thursday, 20 July 2007 Digital Representations of Performing Arts 26 Bringing e-Science into the Frame So far York work done on a single modest computer Scale up To multiple specialist computers Networks Wireless devices Complex tracking 3D sound Interactive graphics All controlled by … The PERFORMER Thursday, 20 July 2007 Digital Representations of Performing Arts 27 Real Time Interactive Multiple Media - RIMM “Construction 3” Realised by the Real-time Interactive Multiple Media Project University of York Music Department (UK), IRCAM (France) and SIMPK (Germany) European Commission Information Society Technologies Programme Incorporates a live performer on soprano saxophone Three-dimensional sculpture Computer generated graphics Ambisonic surround sound Thursday, 20 July 2007 Digital Representations of Performing Arts 28 construction 3 is a synergetic composition The musical and graphical elements of the piece have been composed concurrently Both the computer graphics and music are triggered by the saxophonists natural gestures Through infra-red video tracking Use i/r device and i/r filter on camera Electronic sensors Footswitches, accelerometers Machine listening Score following Gesture recognition using neural networks e.g. “throw” sound into the audience Thursday, 20 July 2007 Digital Representations of Performing Arts 29 Block Diagram of System Thursday, 20 July 2007 Digital Representations of Performing Arts 30 System Diagram Thursday, 20 July 2007 Digital Representations of Performing Arts 31 The “Team” Lots of people needed to make this happen Project manager to keep us in line The Composer/Sound stage designer The Graphic Designer The Graphics Programmer The Audio Programmer The Electronics technical wizard The Local site organiser The commercial suppliers And … THE PERFORMER Thursday, 20 July 2007 Digital Representations of Performing Arts 32 Team Photo Thursday, 20 July 2007 Digital Representations of Performing Arts 33 Screens and Performer Thursday, 20 July 2007 Digital Representations of Performing Arts 34 Examples of the RIMM Computer Graphics Thursday, 20 July 2007 Digital Representations of Performing Arts 35 Technology becomes e-Science Clearly a lot of technology to make it work Some is bespoke electronics Majority of tasks were programmed The SGI Origin 2000 8 fast processors Highly parallel jMax 24-bit optical output for audio High speed network interconnect to graphics computer SGI O2 graphics Optimised for OpenGL Texture memory drawn from main memory Thursday, 20 July 2007 Digital Representations of Performing Arts 36 Academic Background Involvement in Performing Arts My involvement with technology Technology Bespoke hardware Computers Networks Software Engaging with this technology Right people in the right place Future Questions Thursday, 20 July 2007 Digital Representations of Performing Arts 37 Technologies for All Many of these technologies are available Many of the ideas are now embodied in software Many are free Pure Data – aka PD (with GEM add-in) Eyesweb VJ-ing is growing Many VJ products around Grid Computing is available Not easily accessible Lacks some functionality for Performing Arts Photonic networks are available High speed links point to point Up to 10 gb/s possible now 1 gb/s across the globe Thursday, 20 July 2007 Digital Representations of Performing Arts 38 jMax, Pure Data/GEM Graphics Programming Lots of modules Connect up inputs and outputs Uses the modular synthesiser metaphor GEM Graphics Environment for Musicians Link audio to graphics Limited by modules unless you write your own Thursday, 20 July 2007 Digital Representations of Performing Arts 39 eyesweb Video programming with modules Huge number of modules Includes “man” tracking capability Can link to rudimentary audio But … Can also link to PD using Open Sound Control protocols - OSC Thursday, 20 July 2007 Digital Representations of Performing Arts 40 REACTABLE Collaborative electronic music instrument Tabletop tangible multi-touch interface Several simultaneous performers Share complete control over the instrument by moving and rotating physical objects on a luminous round table surface. Objects represent components of a classic modular synthesizer can create complex and dynamic sonic topologies a kind of tangible modular synthesizer or graspable flow-controlled programming language Thursday, 20 July 2007 Digital Representations of Performing Arts 41 REACTABLE Thursday, 20 July 2007 Digital Representations of Performing Arts 42 REACTABLE - demo Click on Image to go to site with Videos [will open a browser] Thursday, 20 July 2007 Digital Representations of Performing Arts 43 More Video Tracking and Performance Messa di Voce Tmema,Blonk,La Barbara http://www.tmema.org/messa/messa.html Thursday, 20 July 2007 Digital Representations of Performing Arts 44 Messa di Voce - demo Click image to go to the Host site [will try to play Quicktime in a browser] Thursday, 20 July 2007 Digital Representations of Performing Arts 45 And just for fun … another demo Thursday, 20 July 2007 Digital Representations of Performing Arts 46 Academic Background Involvement in Performing Arts My involvement with technology Technology Bespoke hardware Computers Networks Software Engaging with this technology Right people in the right place Future Questions Thursday, 20 July 2007 Digital Representations of Performing Arts 47 So what is this Grid Computing? The term originated in the early 1990’s Metaphor for making computer power as easy to access as an electric power grid Many years on and still mainly in the domain of “BIG SCIENCE” Most Grids are run like 80’s batch style computing Submit a job description to a scheduler It decides when and where the job runs – more or less This is a computational grid Many computers and clusters are interconnected Can be globally distributed via photonic networks Thursday, 20 July 2007 Digital Representations of Performing Arts 48 What is this Data Grid? A grid computing system that deals with data The controlled sharing and management of large amounts of distributed data These are often, but not always, combined with computational grid computing systems Finding data is not easy Accessing it is not easy Mainly in the domain of “BIG SCIENCE” Some shift to Arts and Humanities We have lots of data too! More later in context of National Grid Service Thursday, 20 July 2007 Digital Representations of Performing Arts 49 What are these Photonics Networks A worldwide network of optical fibres Speeds are 1gbit/s to 40 gbit/s Run by different organisations UK - Janet Europe - GEANT2 USA - StarLight USA - National Lambda Rail Canada – CANARIE etc … A connection is usually point-to-point Very exclusive Lots of bandwidth Low latency and jitter (more of this later) Thursday, 20 July 2007 Digital Representations of Performing Arts 50 What might I use such links? Many examples Interactive dance Use HDTV rather then Lo-res AccessGrid technology Collaborative audio and video Master classes are more commonplace Collaborative tools for distributed, remote teams Digital cinema (4k x 2k) is used for: Entertainment, media, art and culture Science, medicine, education and research and also … Military, intelligence, security and police Thursday, 20 July 2007 Digital Representations of Performing Arts 51 Nagano Winter Olympics 1998 Massive endeavour with live networked music Opening ceremony had a chorus in: Berlin Cape Town Beijing New York Sydney Performed the final movement of Beethoven's 9th Together with the full symphony orchestra in: Nagano Kenmin Cultural Centre Hall. Also … 2000-person chorus in the Olympic stadium Thursday, 20 July 2007 Digital Representations of Performing Arts 52 Nagano Winter Olympics The final result was mixed and broadcast worldwide This setup required: 7 satellites 12 uplinks with encoders and decoders Multiple-delay compensation The longest delay was to and from Berlin Around 4 seconds! Incredibly … It all worked and the result was orchestras and choirs all in time Thursday, 20 July 2007 Digital Representations of Performing Arts 53 Orchestra Rehearsals New World Symphony in Miami rehearsed pieces by French composers The composers participated live from IRCAM in Paris This transatlantic rehearsal enabled live exchanges between musicians in Miami and the composers in Paris Via RENATER-4 in France, GEANT2 in Europe, and Internet2 These advanced research and education networks provided the bandwidth and quality of service to ensure high quality audio and video Thursday, 20 July 2007 Digital Representations of Performing Arts 54 Using High Speed Networks for Arts Cinegrid is a good Example Cinegrid’s Mission Statement: To build an interdisciplinary community that is focused on the research, development, and demonstration of networked collaborative tools to enable the production, use and exchange of very-high-quality digital media over photonic networks See www.cinegrid.org for more details Thursday, 20 July 2007 Digital Representations of Performing Arts 55 Cinegrid Converging on the digital Fast networking Access to shared instruments, computers, storage Collaboration tools Robust security for intellectual property High visual quality Greater speed More distributed applications and, importantly … Next generation of trained professionals Thursday, 20 July 2007 Digital Representations of Performing Arts 56 Cinegrid@AES October 25th 2006 This event was structured in four acts Each demonstrating a different facet of the CineGrid philosophy of networked extreme media In Act 1, a sequence of 4K “digital shorts” at 24 frames per second (fps), together with fully mixed synchronized audio, were pulled in real time from network-connected servers in Los Angeles and San Diego. Thursday, 20 July 2007 Digital Representations of Performing Arts 57 Cinegrid@AES October 25th 2006 In Act 2, 4K tele-presence was used for interactive video-conferencing and ultra-realistic reproduction of a classical music performance from Tokyo Thursday, 20 July 2007 Digital Representations of Performing Arts 58 Cinegrid@AES October 25th 2006 Acts 3 and 4 were designed to prove the concept: networked, remote audio post-production for digital cinema by creative teams spread around the world who demand the highest-quality production values In Act 3 4K motion pictures were sent compressed from Tokyo 24-channel non-compressed digital audio was streamed from San Diego In Act 4 The performance system was re-configured to use uncompressed 2K motion pictures coming from ILM servers in the LDAC facility, synchronized to 24-channel, non-compressed digital audio streaming from San Diego Thursday, 20 July 2007 Digital Representations of Performing Arts 59 Cinegrid@AES October 25th 2006 Was a good test of the CineGrid concept of: Using 4K cameras Multi-channel, non-compressed audio Ultra-realistic “live” experiences of music concerts and … other kinds of performing arts To distant audiences in theatres Connected by high-speed networks Thursday, 20 July 2007 Digital Representations of Performing Arts 60 Cinegrid@AES October 25th 2006 Tomonori Aoyama, Professor at Keio/DMC and Chairman of the Digital Cinema Consortium of Japan (DCCJ) said: "I believe the very high-speed optical networks such as now being deployed by research organizations will become an essential infrastructure for digital cinema production and distribution” "But, we still have to learn how to integrate systems that creative people can use to make beautiful 4K content - picture and sound - in new ways appropriate for the 21st century. Demonstrations such as CineGrid@AES force us, in a good way, to learn by doing." Thursday, 20 July 2007 Digital Representations of Performing Arts 61 UK National Academic Networks UKERNA UK Education and Research Networking Association Now renamed to JANET Operate the standard IP service What most of use use daily With the rollout of SUPERJANET-5 we now have: JANET Lightpath Service Evolved from UKLight See www.uklight.ac.uk for details Various research projects Many coordinated from NeSC and the eSI Thursday, 20 July 2007 Digital Representations of Performing Arts 62 What is a lightpath? End-to-end Network Capacity Point-to-point circuits Capacities from about 50Mbit/s to 10Gbit/s Presentations • Ethernet – 100/1G/10G (WAN/LAN) • SDH – STM-1/4/16/64 Implemented over JANET optical transmission infrastructure including SDH equipment re-deployed from UKLight Thursday, 20 July 2007 Digital Representations of Performing Arts 63 Janet Networks JANET IP Research capacity UKLight Additional Wavelengths JANET Optical Transmisson Thursday, 20 July 2007 Digital Representations of Performing Arts 64 SUPERJANET-5 Integrated with UKLIGHT FaTMAN UHI Glasgow NIRAN Clydenet EaStMAN C&NLMAN AbMAN NorMAN Dublin (HEANET) YHMAN NNW Leeds Warringto n USA StarLight Europe GEANT MidMAN EastNet EMMAN T-City Reading THouse TVN London LMN WREN Kentish MAN LeNSE SWERN Bristol Thursday, 20 July 2007 Digital Representations of Performing Arts 65 Getting a Lightpath - JANET Policy Subject to capacity and budget availability, lightpaths will be free at point of use up to 1Gbit/s Tariffs available for 10Gbit/s Delivered to Institutions at the JANET Access Point Institutions must commit to extending the lightpath across their networks See www.ja.net/lightpath for details Documents shortly First public version in final editorial stage Service description and application lightpath application form Thursday, 20 July 2007 Digital Representations of Performing Arts 66 High Performance Computing The more power we need the less we can afford it Not sensible to have massive systems duplicated Look towards the grid for power But, what if we need it in real time? Current technology geared towards batch computing Some notable exceptions Interactive visualisation Computational steering VLBI Some CERN LHA experiments Data reduction in real time Thursday, 20 July 2007 Digital Representations of Performing Arts 67 Grid Computing in the UK National Grid Service (NGS) NGS-1 4 core sites with compute and TB storage NGS-2 Officially in pre-production as of 20th June Sevices … Job Submission via: Globus, GridSAM, Condor, gLite or via portals and applications repositories “to simplify job submission for users who do not wish to delve into the technical details of grid computing “ Grid Support Centre to help users Thursday, 20 July 2007 Digital Representations of Performing Arts 68 Data Grid in the UK NGS offers: Oracle 10g Storage Resource Broker (SRB) OGSA-DAI VOMS Oracle 10g – high performance database system SRB - data storage service store your data in geographically distributed locations access them all using only the logical filename Reliabilty and efficiency are improved dramatically OGSA-DAI The Open Grid Services Architecture Data Access and Integration middleware that helps users access and integrate data from separate structured data sources e.g. XML or Oracle Thursday, 20 July 2007 Digital Representations of Performing Arts 69 VOMS Virtual Organisation Membership Service To help grids manage the authorization of their users Delegates the approval of users to the to the VO itself Thus removing the onus upon the end user to register with each resource they might use as part of the VO NGS resources are in the process of adopting VOMS to authorize NGS members In the future the number of VOMS-aware resources is likely to increase Thursday, 20 July 2007 Digital Representations of Performing Arts 70 Academic Background Involvement in Performing Arts My involvement with technology Technology Bespoke hardware Computers Networks Software Engaging with this technology Right people in the right place Future Questions Thursday, 20 July 2007 Digital Representations of Performing Arts 71 High Performance Computing Wavefield Synthesis is a good example Computationally demanding Desktop machines and small clusters not capable Can we have real time grid computing please? If not, will someone fund a big lorry: With lots of computers Lots of aircon Plug in 3-phase power at one end Gigabit ethernet comes out the other end! Thursday, 20 July 2007 Digital Representations of Performing Arts 72 Wavefield Synthesis – what, why, when? The limitation of stereo or ambisonic techniques it only works perfectly well for one listener positioned on the so-called sweet spot Thus in common concert environments the intended effect of movement of the sound will in these cases not be heard by a majority of the listeners Wave field synthesis can overcome this limitation Can provide a good perceptual localisation in a relatively large listening area This makes the technique ideal for concert environments Thursday, 20 July 2007 Digital Representations of Performing Arts 73 Wavefield Synthesis – Simplistic overview Sound is waves Next wave can be synthesised by a infinite number of small speakers Need to approximate “infinite” Thursday, 20 July 2007 Digital Representations of Performing Arts 74 Wavefield Synthesis – the reality Various experimental speaker arrays Thursday, 20 July 2007 Digital Representations of Performing Arts 75 Exemplar Venue RECOMBINANT MEDIA LABS, SF Founded to create, research, and portray Spatial Media Synthesis The science of projecting image and sound objects in 3-dimensional space RML cultivates radical methodologies for experiential engineering Exploring performative processes which expand the formal aesthetic and technological boundaries of immersive installation and surround cinema Thursday, 20 July 2007 Digital Representations of Performing Arts 76 RML The San Francisco facility contains: A flexible black box environment that houses: A high definition multi-channel audio-visual system known as Surround Traffic Control Full fidelity array consists of a design specification for: 10 screens in 360 degrees supported by an ultra impact 16.8.2 horizontal and vertical sound diffusion system scalable for all types of rectangular rooms High Speed Networking Thursday, 20 July 2007 Digital Representations of Performing Arts 77 Summary Seen the evolution of some technology The newer technologies require: High performance computing Very high speed networks Massive storage New breeds of creative artists Many different experts in the technologies Networking of experts New theatres Investment and much much more Thursday, 20 July 2007 Digital Representations of Performing Arts 78 ANY QUESTIONS? Thursday, 20 July 2007 Digital Representations of Performing Arts 79