"A computer terminal is not some clunky old television with a typewriter in front of it. It is an interface where the mind and body can connect with the universe and move bits of it about." Douglas Adams, Mostly Harmless Virtual Reality: History Curs Realitate virtuala in robotica DUGULEANĂ Mihai duguleanamihai@yahoo.com mihai.duguleana@unitbv.ro Site: mihai.duguleana.com Virtual Reality: History Conditii de promovare Curs - prezența nu este obligatorie - 1 punct = prezenta la mai mult de 7 cursuri Laborator - prezenta este obligatorie - lucrarile de laborator – 2 puncte - min 50% Proiect - 3 puncte - min 50% Examen - 4 puncte - min 50% Virtual Reality: History Programul modulului Virtual Reality Visual Channel 2 Modelling Real-time Rendering Virtual Humans 1 History Technologies Applications Tools Acoustical Channel 3 3D Audio Virtual Reality: History What is Virtual Reality ? The term Virtual Reality was born in 1988, in an interview to Jaron Lanier “A Portrait of the Young Visionary”. Lanier: “VR is a technology that uses computerized clothing to synthesize shared reality. It recreates our relationship with the physical world in a new plane, no more, no less. It doesn't affect the subjective world; it doesn't have anything to do directly with what's going on inside your brain. It only has to do with what your sense organs perceive. In VR there's no need for a single metaphor, whereas there is a need for a single design metaphor in a computer. We are used to switching contexts in real life. There's simply no need for one unified paradigm for experiencing the physical world, and there's no need for one in Virtual Reality either. “ Virtual Reality: History What is Virtual Reality? - Neo, I imagine you know something about Virtual Reality. - Essentially, it's a hardware system that uses an apparatus to make you feel that you are in a computer program. - If the VR apparatus controlled all of your senses, would you be able to tell the difference between the virtual world and the real world? - You might not, no. - No, you wouldn't. [The Matrix - WACHOSKI 1999] - A show. Then who am I? - You're the star. - Nothing was real. - Nothing was real. That's what made you so good to watch. [The Truman Show – NICCOL 1998] Virtual Reality: History Virtual Reality properties Prezenţa, ca senzaţie mentală de a exista într-un spaţiu, este legată de implicarea utilizatorului, şi se bazează pe îmbinarea eficientă a senzaţiei de imersie cu cea de interacţiune. Astfel, consecinţă a calităţii reprezentării mediului virtual precum şi a interfeţelor om-calculator folosite, se pot atinge nivele ridicate de prezenţă cu următoarele caracteristici: perspectivă dinamică legată de mişcările capului, stereoscopie, păstrarea dimensiunilor obiectelor virtuale în limita celor reale, randarea cu feedback haptic pentru o interacţiune îmbunătăţită, controlul audio, sinteza de stimuli olfactivi etc Senzaţia de imersie se exprimă prin sentimentul operatorului de a fi în interiorul unui spaţiu virtual. Imersia, ca percepţie a lumii virtuale de către operator, se realizează prin intermediul interfeţelor, cu precădere, video şi audio. Interacţiunea în mediile de RV este reprezentată de capacitatea utilizatorului de a modifica mediul virtual şi de a primi feedback din partea acestuia. Capacitatea de interacţiune dă nivelul de realism al unei procesări virtuale. Virtual Reality: History Virtual Reality is many things… Some types of Virtual Reality: Text VR : Can be even strongly interactive, but not immersive Desktop VR: Variable interaction, immersion usually low. Immersive VR: High immersion, interaction strongly depends on the system complexity. Virtual Reality: History Immersive Virtual Reality Un punct slab al aplicaţiilor de RV destinate desktop-ului este slaba senzaţie de imersie => platforme complexe de tip CAVE. Some features commonly available: - Dynamic perspective linked to the head movements. - Stereoscopic vision - Virtual Environment realized with realistic scale and properties - Realistic interaction with the VE through interface for the manipulation, operation, control - Possible audio, haptic and motion feedback - Shared environments Virtual Reality: History The need of real-time To pursue these objectives, and in particular to allow a natural interaction between the user and the VE, we need computers able to execute in real-time all the calculations needed to produce the response to user actions without perceivable delays. The rate at which the feedbacks are generated must be high enough to create the illusion of continuous movement. Splitting feedbacks over different sensorial channels helps, as requirements are different: - Visual channel: images should be recomputed with a frequency of over 24 Hz to be realistic - Haptic channel: forces should be provided with a frequency of about 1 KHz Virtual Reality: History Legatura RA / VA / RV Conform taxonomiei acceptate pe plan mondial, axa realitate-virtualitate este divizată în trei clase distincte de reprezentare (Realitate Augumentată, Virtualitate Augumentată, Realitate Virtuală), cu nivele descrescătoare de aproximare a realităţii: Virtual Reality: History Augmented Reality Real Environment Augmented Reality Augmented Virtuality Virtual Reality Virtual Reality: History VR History In 1956 Morton Heiling launched an attraction called SENSORAMA, a kind of passive simulator of motorvehicle. -Vibrating seat and handles -Stereoscopic movie of Manhattan -Wind feedback with a fan -Scent of car gas Virtual Reality: History VR History “For the price of 25 cents, «Sensorama» would offer multisensorial impressions of a virtual, ten-minute-long motorcycle ride through New York City. ” The experience was not interactivce, therefore it cannot be properly defined as VR. Sensorama was not a big deal (“It was maybe too revolutionary for its times”). Virtual Reality: History First HMDs In 1969, at the University of Utah, Ivan Sutherland, the father of computer graphics, implemented a stereoscopic HMD. These images were displayed on two tiny monitors, one for each eye. The monitors were mounted on a apparatus suspended from the ceiling and strapped to the user’s head. The head movements were detected by the apparatus and relayed to the computer which generated the correct view, that is the view that the person would see if he were in the room, looking in the same direction. Virtual Reality: History First HMDs The project was name “The Damocle sword“ !!! The BOOM (Binocular Omni-Orientation Monitor) from Fakespace is a head-coupled stereoscopic display device. Virtual Reality: History Interaction and Computer Graphics Sutherland himself got his degree nel 1962 at Stanford presenting the Sketchpad system, a graphical system where the user was able to interact with using an optical pen. Virtual Reality: History Artificial Reality There was enough stuff to define a new entity. In 1970 Myron Kreuger coined the term Artificial Reality “A full-body participation in computer events that were so compelling that they would be accepted as real experience.” Artificial reality promise is NOT that of reproducing the conventional reality, but the possibility of creating SYNTHETIC realities that do not have actual corresponding entities. Virtual Reality: History Force-feedback In 1976 P.J..Kilpatrick, at UNC, connected a manipulator (used to handle radioactive materials) to a simple graphic world. The environment consisted in a table and object on the top of it and a replica of a graphic manipulator, seen from a stereo display. It the user tried to move the graphic manipulator through the table, she felt the physical arm resist the motion. If the user tried to pick up a book, she felt the weight of the book as she raised her hands. The manipulator prevented also to close fingers, giving the grasping sensation. Virtual Reality: History 3D Graphics The first hypermedia and virtual reality system was the Aspen Movie Map which was created at MIT in 1977. The program was a crude virtual simulation of Aspen, Colorado in which users could wander the streets in one of three modes: summer, winter, and polygons. The first two were based on photographs -- the researchers actually photographed every possible movement through the city's street grid in both seasons -and the third was a very crude 3-D model of the city. Virtual Reality: History Sensors In 1979 the Polhemus 3SPACE was presented, a system of magnetic sensors used to retrieve the absolute position and orientation of a point in the space. Initially conceived to be used together with a HMD, it had a number of additional uses. Polhemus was far from being perfect and did not work correctly in presence of metal structures. Yet, it allowed to substitute complex mechanical devices to track 6 DOF, which was needed in VR applications. Virtual Reality: History Advanced interfaces Nel 1984 NASA transformed a HMD used as “virtual cockpit” (created in ’66) reducing its cost from $1M to $20K. A magnetic 6-dof sensor was added to determine where the user were looking. Subsequently the helmet display was reduced in size to become reality goggles. NASA contracted also VPL Research (California) to build a hi-res version of that company’s existing bend-sensing glove. NASA added the Polhemus magnetic 6 dof sensor. VPL, went beyond NASA demos using a powerful Silicon Graphics workstation for 3D real-time graphics. Virtual Reality: History Cyberspace Science Fiction was already dealing with similar topics, although VR was not actually born yet. One of the moral fathers of VR is Philip K. Dick, with UBIK (1969). VR is directly created with sensorial stimula on bodies in suspended animation (like what?) In1984 Gibson writes Neuromancer and introduces Cyberspace, the set of information of a computer network. Nowadays, Cyberspace is commonly meant as the VE that Internet forms together with its services and information. Virtual Reality: History Finally, Virtual Reality With the foundation of VPL from Jaron Lernier (1984), VR becomes a commercial reality: systems where all of the technologies previously described are integrated (sensorized gloves and suites,, HMD, 3D graphics), can be sold at affordable prices. In 1989 Lanier himself in an interview (http://www.well.com/user/jaron/vrint.html) finds a name for this set of technologies and for the related experience: Virtual Reality was finally officialy born. Virtual Reality: History VR and Cinema DISCLOSURE (1995, Barry Levinson) Virtual Reality: History VR and Cinema NIRVANA (1997, Gabriele Salvatores) Virtual Reality: History VR and Cinema 13th floor (1999, Josef Rusnak) Virtual Reality: History VR and Cinema MATRIX (1999, Wachoski Bros.) Virtual Reality: History Why VR is not a mass phenomenon? Safety issues: sometimes VR is scaring because of its interfaces and their invasiveness. Moreover, long lasting uses of some devices may induce sickness. Hardware costs: VR requires expensive and dedicated components, not always affordable. Interface components: Resolution, complexity, speed, latencies, space, encumbrancies: hw must be improved Virtual Reality: History Why VR is not a mass phenomenon? It is more difficult to trick users: first movies were perceived as realistic. Todays users are smarter and notice every incoherence with the reality. Involvement: Although perceptual stimula are provided on the main sensorial channels, some stimula are still non sufficient or not existing (full body FF, tactile feedback, olfactory information, ….). Virtual Reality: History Why VR is not a mass phenomenon? Single user: Whilst cinema and TV are multiuser, VR is (mainly) monouser with some notable exceptions. Purposes: In spite of its various applications, it is often not perfectly clear where and how to insert VR in our life and how to proficiently use it. Virtual Reality: History Challenges Natural interaction: Full and correct actions interpretation Realism of representation: Realistic feedback on all sensorial and motor channels. In particular, address the issues related to - Tactile feedback: wearable interfaces - Movement: locomotion interfaces - Olfact: olfactory analysis and synthesis olfattivo (problems: evacuation times etc.) - Direct nerve stimulation: exciting but disturbing (ego suppression?) Virtual Reality: History Prezenta Virtual Reality: History Imersie Virtual Reality Interactiune Prezenta Virtual Reality: History O clasificare hardware a VR Interactiunea Non-interactive Device based interaction Desktop Devices Mouse Keyboard Joystick Touch Screen Natural interaction Wired sensors Desktop Wearable Haptics Haptics Optical MoCap Wireless sensors Wearable MoCap Brain Computer Interfaces No sensors Gesture Recognition Speech Recognition Virtual Reality: History O clasificare hardware a VR Imersia Non-immersive Low Immersion Desktop Devices External devices Wearable devices Visual Monitor Acoustic Desktop Speakers Headphones Haptics Desktop Haptics Wearable Haptics Motion Workbench High Immersion HMD Retinal Display Whole Body Motion Interface Powerwall Panoramic Powerwall CAVE Multichannel Speakers Encountered Haptics Treadmill Real Objects Motion Platform Virtual Reality: History Prezenta in VR Oversimplifying, interaction and immersion can be considered as variables, placed on a two perpendicular axes, which concur to create the sense of user presence. The stronger emotional involvement triggered by immersion helps the establishment of perceptual mechanisms that, by increasing the user’s ability of interfacing with information, facilitate an effective conveying of contents Suspension of disbelief Subjective factors Virtual Reality: History Prezenta in VR Interdependenţa dintre prezenţă, interacţiune şi imersie Virtual Reality: History Prezenta in VR Subjective factors can be (hardly) descrived in quantitative terms (through sensors recording physiological paramters) and qualitative (questionnaires) The Pit experiment The Virtual Milgram Experiment Virtual Reality: History Prezenta in VR An immersive Virtual Environemnt requires a strong presence feeling, in order to make the interaction natural and to improve user’s perception of it. The feeling of presence is determined by three factors: – Quality of sensorial information: Modeling Rendering – Sensors mobility and control – Environment control Virtual Reality: History Presence: low-quality sensorial info Virtual Reality: History Presence: high-quality sensorial info Virtual Reality: History Presence: sensors low comfort Virtual Reality: History Presence: sensors high comfort Virtual Reality: History Presence: low environment control Virtual Reality: History Presence: high environment control Virtual Reality: History Presence in Virtual Environments According to David Zeltzer, a discussion about presence is meaningless without specifying the application domain and task requirements. He also claims that it is not possible to simulate the physical world in all its detail and components. There should be research done to identify the different sensory cues that must be provided to complete a task. Virtual Reality: History Presence in Virtual Environments Zeltzer’s assertion, though almost trivial at a first glance, is indeed very important because it suggests a way to reduce the complexity of the environment to be simulated. Depending on the task to be performed, it is possible to define the optimal level of the factors in stake: – the sensorial channels to be stimulated – the cues to provide on the selected channels – the level of details of the VE, across the selected channels – the relevant information which have to be exchanged between the user and the VE Virtual Reality: History Logical modules of a VE Sampling Synthesis Modeling Behaviours Properties Virtual Environment Management Rendering Interaction USER Virtual Reality: History Virtual Environment Data Flow VIRTUAL ENVIRONMENT VISUAL CHANNEL ACOUSTICAL CHANNEL HAPTIC CHANNEL INERTIAL CHANNEL USER Virtual Reality: History Virtual Environment Components VIRTUAL ENVIRONMENT SW modules of visual modelling & rendering VISUAL CHANNEL ACOUSTICAL Graphical CHANNEL Feedback HAPTIC CHANNEL INERTIAL CHANNEL HW devices of visualization USER Virtual Reality: History Stereo pairs Our eyes see the world in a slightly different way Each eye provides a perpective 2D vision of the world. Combining these two perspectives forms a stereo pair which inherently contains information about the three dimensions Beware! The term “3D” is often misused. What is commonly called 3D graphics is, usually, a 2D perspective representation Virtual Reality: History Depth cues We can perceive the three dimensions thanks to a series of information (depth cues) To effectlively simulate a 3D environemnt, it is necessary to provide as many depth cues as possible, or at least the most significant ones. Virtual Reality: History Monocular Depth cues Some very important depth-cues are: – Occlusion: if an object occludes another object it is perceived as closer – Shading: provides info about the orientationa and the position of surfaces related to a light souce. – Perspective: an object with a known size can provide information about its depth depending on its apparent size – Parallax: objects place at different distances move with different apparent speed These depth-cues are well simulated also from an appropriate 2D image. Anyway, although they are necessary, they cannot be sufficient. Virtual Reality: History Binocular Depth Cues Other important depth-cues : – Dynamic perspective: when we move, even slightly, our point of view, the perspective of the world changes. We need to track position and orientation of our head to correctly update the perspective. – Ocular Separation: our eyes are separated by a certain offset. This implies that perspectives are different: objects are separated from an offset growing when the distance decreases. We need to build two different images for the two eyes. Virtual Reality: History Binocular Depth Cues – Depth of field: objects at different distance are focused differently. – Field of view: human field of view is about 180°x120°. The presence of a border at the edges of a stereo image “destroys” the 3D illusion (monitors have FOV=35°x27°) – Non visual clues: provide additional information, like the ones coming from the vestibular apparatus or kinaesthetic data from the neck etc. Important: if badly implemented, conflicting depth-cues can damage the entire 3D effect and create discomfort in the perception. Virtual Reality: History Stereo pair A system for the stereoscopic visualization must be composed of: – Software able to generate two monoscopic bi-dimensional images, one for each eye, created and synchronized in order to give back the opportune depth cues – Hardware able to let each eye perceive only its correspondent image “Stereo-crosstalk” (or ghosting): – Ghosting is the permanence, on an eye, of the image produced for the other eye. This produce the vision of a ghost silhouette together with the correct image. Virtual Reality: History Stereo image Virtual Reality: History Stereo image Virtual Reality: History Active Stereo For each frame two images are projected sequentially; therefore there’s a continuous hi-frequency (about 120Hz) switching between the images for the right eye and the images for the left eye. Users wear a special active device, shutter glasses, synchronized with the image switcher and able to make lenses opaque or transparent. When the image for the right eye is present, the left lens is completely opaque, otherwise it is transparent. The same happens for the right lens. The human brain, actually, receives a sequence of images but they are so quickly presented that it believes to perceive them at the same time. In other words the brain merges the images and can reconstruct depth from them Virtual Reality: History Passive Stereo Both images are projected at the same time but, thanks to a system of optical filters only the correct image reaches each eye. Different implementations: – Anaglyphs – Polarization filters – Infitec filters There are advantages and disadvantages for both technologies: active stereo is more expensive and requires dedicated hardware, passive stereo presents the problem of ghosting (or stereo crosstalk), which means that one eye perceives also a small fraction of the image presented for the other eye. Virtual Reality: History Anaglyphs Historically, the first stereo pairs The two images are coded with blue (or green or cyan) and red filters. The same filters decode images Cons: Destructive filtering (good for greyscale images, not so good for color images) Pros: does not necessarily need projectors, works on TC/monitors, easy to produce, very cheap Virtual Reality: History Light polarization A lightwave rotates in all the directions The specific orientation, at a given time t, determines its polarization To polarize a light means to let a specific orientation emerge Human eyes are not so sensitive to polarization changes. Two kind of polarization: – Linear (stereo pairs with horizontal/vertical pol.) – Circular (stereo pairs with cw /ccw pol.) Pros: – Limited costs, any graphical hardware Cons: – Sensitiveness to rotations (LP) or to ghosting (CP) – Needs special screens Virtual Reality: History Infitec solution Left Eye Based on spectrum split: Pros: – – – Almost no ghosting Excellent image separation Works on any surface Right Eye Cons: – – Filters/Glasses are expensive Colors are modified, a gamma correction is needed Virtual Reality: History Powerwall A Powerwall is a back-projected screen, usually large sized, able to visualize stereoscopic images. Virtual Reality: History Workbench Similar to Powerwall, but smaller (low immersion) and variable orientation Useful in contexts (es.surgical simulators) where it is not needed the full immersion in a VE but rather is preferrable to insert virtual content in a real context. Rendering CAVE CAVE is a recursive acronym (CAVE Automatic Virtual Environment). A CAVE is a multi-user environment, a cubic room where some or all the walls are powerwall synchronized among them and, optionally, with the user movements. Virtual Reality: History CAVE – 6 screens Virtual Reality: History Autostereoscopic displays These display do not use lenses, glasses, screens etc. Volumetric displays: Display information in a volume. Voxel, not pixels. Emissive: the volume is filled up with a medium able to emit light in specific directions depending on how it is excited. Rotating: a flat screen rotates at 600 rpm. Depending on the angular position of the screen, an optical system projects on it a “slice” of the object corresponding to the perpective related to that angle. Rendering Autostereoscopic displays Parallax displays: Images are separated by means of parallax barriers, realized through two overlapping screens alternated by columns- Cons: there is a limited range of correct positions (swap risk) Rendering Autostereoscopic displays Lenticular Displays: They make use of lenticular sheets, a set of cylindric lenses that allow to focus only a portion of the back image. Rendering Head Mounted Display Virtual Reality: History Head Mounted Display A helmet provided with 2 small LCD displays, wearable from the user. Optionally with headphones and tracker Best image separation (no ghosting, each eye receives only one image) Pros: - Total immersion (no contact with external reality) Cons: - Costs - Resolution - FOV - Encumbrance, weight - Sickness (latencies) Rendering AR HMD: Video See-Through Virtual Reality: History AR HMD: Optical See-Through Virtual Reality: History AR HMD: Retinal Displays Nuova frontiera: retinal-displays Nascono all’Università di Washington nel 1993 Proiettano un fascio di luce direttamente sulla retina L’osservatore ha l’illusione di vedere l’immagine come se fosse a 50 cm da un display 14” Caratteristiche: Alta Risoluzione Luminosità Consumi Virtual Reality: History Virtual Environment Components VIRTUAL ENVIRONMENT VISUAL CHANNEL ACOUSTICAL CHANNEL HAPTIC CHANNEL Audio Feedback INERTIAL SW modules of audio modelling & rendering CHANNEL HW audio devices USER Virtual Reality: History Audio technologies Virtual Reality: History Dispositive acustice STEREOPHONY: Sounds are distributed on two channels. A virtual source can be created, but only on the line connecting the two loudspeakers. MULTICHANNEL SPEAKER ARRAY: Adding more channels enlarges the space where the virtual source is placed. Virtual Reality: History Crosstalk In speakers-based audio system crosstalk a phenomen similar to ghosting in stereoscopy may occurs: a fraction of the sound intended for the right ear arrives to the left ear and vice versa. It is possible to use upstream filters able to precondition the signal so as to cancel crosstalk, provided the user is in a well specified point (sweet spot). Crosstalk is perceived as more as getting far from the sweet spot. Human ear is more sensitive to crosstalk on the LR direction rather than on the FB one. Increasing the number of speakers allows to extend the sweet spot to an area (if the array is planar) or to a volume . Virtual Reality: History Headphones Like the HMD in vision, headphones provide the best acoustical separation: each ear gets only the correct signal. The signal must be pre-processed so as to arrive already trasformed with the opportune Head Related Transfer Function This way a headphone can provide binaural sound, where the directionality is perceived (more LR than FB, as masking effects are simulated and not real) Surround headphones actually virtualize channels and not sound sources (same difference as between precomputed 3D movies and real-time graphics) In order to compute a correct binaural real-time feedback for moving listeners, it is needed to track their 6DOF Virtual Reality: History Virtual Environment Components VIRTUAL ENVIRONMENT VISUAL ACOUSTICAL CHANNEL SW modules of haptic CHANNEL HAPTIC CHANNEL INERTIAL CHANNEL modelling & rendering Haptic Feedback Haptic Interfaces USER Virtual Reality: History Haptic interfaces: desktop Virtual Reality: History Haptic interfaces: antropomorphic Virtual Reality: History Haptic interfaces: antropomorphic Virtual Reality: History Virtual Environment Components VIRTUAL ENVIRONMENT VISUAL CHANNEL ACOUSTICAL CHANNEL HAPTIC CHANNEL INERTIAL CHANNEL Motion Feedback USER Virtual Reality: History Motion Feedback Virtual Reality: History Integrating feedbacks: co-location issues Virtual Reality: History Randare olfactiva VIRTUAL ENVIRONMENT … OLFACTORY CHANNEL CANAL GUST … Utilizator Virtual Reality: History Olfactory channel SMELL GENERATION Differently from the other senses (sensitive to physical stimula), smell is sensitive to chemical stimula. A theory (Amoore 1963) asserts that any smell can be considere as the combination of seven primary smells. Although approximative, the theory can be considered acceptable. Odore primario Esempio Canforaceo Naftalina Muschioso Muschio Floreale Rosa/Lavanda Mentolato Mentolo Etereo Alcool Pungente Aceto Putrido Uova marce Scent Display PARALLEL WITH GRAPHICAL DISPLAYS: Direct distribution in the nose with pipes ↔ HMD Distribution in a space within the rang of nose ↔ HW projection Virtual Reality: History Olfactory rendering ATR Media Information Science Laboratories Dept. of Information Media Technology, Tokai University Japan Virtual Reality: History Taste rendering TASTE GENERATION Also in this case, a taste can be approximated by the combination of five primary tastes. Sapore primario Esempio Salato Sale Acido Aceto Dolce Zucchero Amaro Caffè Umami Glutammato (es. Parmigiano, Salsa di soia etc.) TASTE DISPLAY So far, the only proposal comes from Tsukuba University which developed “food simulator”, a haptic interface returning the force feedback related to chewing and using a combination of primary elements to generate the desired taste. The result is injected as a liquid (0.5 ml) and injected on the tongue through a pipe placed at the border of the HI. Virtual Reality: History Taste Rendering Hiroo Iwata, University of Tsukuba, Japan Virtual Reality: History Virtual Environment Components VIRTUAL ENVIRONMENT SW modules of tracking and data acquisition VISUAL CHANNEL ACOUSTICAL Interaction CHANNEL INERTIAL HAPTIC CHANNEL Tracking and CHANNEL motion capture devices USER Virtual Reality: History