Virtual environment

Virtual Reality Dr. Yan Liu Department of Biomedical, Industrial and Human Factors Engineering Wright State University Introduction  What is Virtual Reality (VR) or Virtual Environment (VE)  A medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more sense, giving the feeling of being mentally immersed or present in the simulation (a virtual world) (Sherman & Craig, 2003) Virtual world   Four Key Elements in Experiencing VR (Sherman & Craig, 2003)   An imaginary space, often (but not necessarily) manifested through a medium Having a sense of presence within an environment; this can be purely a mental state, or can be accomplished through physical means Mental immersion  A state of being deeply engaged, with a suspension of disbelief Physical immersion  Bodily entering a medium 2 Immersion    Introduction (Cont.)  Four Key Elements in Experiencing VR (Sherman & Craig, 2003)  Sensory feedback  Visual/aural/haptic feedback to participants, based on some aspects of their physical positions In a virtual reality experience, participants are able to move around and change their viewpoint, generally through movements of their head  Interactivity   Four Technologies that are Crucial for VR (Brooks, 1999)     The visual (possibly also aural and haptic) displays that immerse the user in the virtual world and that block out contradictory sensory impressions from the real world The graphics rendering system that generates, at 20 - 30 frames/second, the ever-changing images The tracking system that continually reports the position and orientation of the user’s head and limbs The database construction and maintenance system for building and maintaining 3 detailed and realistic models of the virtual world Introduction (Cont.)  Four Technologies that are Important for VR (Brooks, 1999)  Synthesized sound, displayed to the ears, including directional sound and simulated sound fields  Directional sound is a technology that concentrates acoustic energy into a narrow beam so that it can be projected to a discrete area, much as a spotlight focuses light    Display of synthesized forces and other haptic sensations to the kinesthetic senses Devices, such as tracked gloves with pushbuttons, by which the user specifies interactions with virtual objects Interaction techniques that substitute for the real interactions possible with the physical world A Desktop VR http://www.youtube.com/watch?v=Jd3-eiid-Uw A CAVE VR http://www.youtube.com/watch?v=7_nUa4sFHSo&feature=related 4  Examples of VR   Head-Mounted Display (HMD)  Head-Mounted Display (HMD)  A video display device mounted in a helmet, suspended one in front of each eye (in opaque HMDs) or projecting onto half-silvered mirrors in front of each eye (in see-through HMDs) Full immersion HMD http://www.darpa.mil/MTO/Displays/HMD/ Factsheets/immersion.html) 5 CAVETM  CAVETM  Provides the illusion of immersion by projecting stereo images on the walls and floor of a room-sized cube A wide surrounding field of view The ability to provide a shared experience to a small group The cost of multiple image generation systems (although not a serious limitation nowadays) Space requirement for rear projection   Advantages    Disadvantages     4-8 feet or more, depending on the size of the screen Result in scenes of approximately full-moon brightness and hinder color perception An alternative is to use Dome systems in which imagery is projected onto a hemisphere surrounding 6 Brightness limitation due to large screen size  Corner and edge effects that intrude on displayed scenes  CAVE at NCSA (National Center for Supercomputing Applications) at UIUC (http://cave.ncsa.uiuc.edu/about.html) An illustration of DOME 7 Panoramic Displays  Panoramic Displays    One or more screens arranged in a panoramic configuration, or a single, curved screen on which images from multiple projectors are tiled together Especially suit groups; multidisciplinary design reviews commonly use this type of display One person drives the viewpoint Edge blending, viewpoint-dependent distortion correction, viewpoint-dependent gain correction  Issues  CURV™, by Fakespace Lab (http://www.fakespace.com/curv.htm) 8 Workbenches  Workbenches  Flat, rear-projection screens that display images in stereo and can be set up in a horizontal or tilted position Responsive workbench, by Dr. Krueger at Stanford (http://graphics.stanford.edu/projects/RWB/) M1 DeskTM, by Fakespace Lab (http://www.fakespace.com/M1Desk.htm) 9 Boom (Binocular Omni-Orientation Monitor)  Boom  A head-coupled stereoscopic display device that the screen and optical system are housed in a box that is attached to a multi-link arm; the user looks into the box through two holes, sees the virtual world, and can guide the box to any position within the operational volume of the device Boom3C, by Fakespace lab (http://www.fakespacelabs.com/tools.html) 10 Fishtank VR  Fishtank VR   A desktop VR system in which images are displayed on a desktop monitor, usually in stereo, and coupled to the location of the head which is tracked, resulting in the illusion of looking into a “fishtank” Commonly applied in CAD and design areas where immersion is not of much significance Fishtank VR (http://www.faw.uni-linz.ac.at/save/) 11 Properties of VR Displays  Spatial Resolution   The ability of the system to spatially discriminate an object in the field of view; A system with higher resolution can resolve an image with smaller size Because the smallest unit of an image is pixel, the resolution of a display is limited by its pixel size  Temporal Resolution   The time interval between images, or the number of frames captured per second Often there is a tradeoff between spatial resolution and temporal resolution The ratio of the brightest part of an image to the darkest part of the image The perceived amount of light 12   Contrast  Brightness  Properties of VR Displays (Cont.)      Number of Display Channels  e.g. RGB channels, luminance channel Distance from the center of the lens to the point that is in focus The amount of transparency of the display The angular extent of the observable world that is seen at any given moment The amount of space surrounding the user that is filled with the virtual world Focal Distance  Opacity  Field of View  Field of Regard  13 Motion Tracking  Usage  In VR, tracking technology is required to monitor the real-time position and orientation of the user’s head and limb A simple mechanical tracker can take the form of mechanical arm attached to the tracked object Very useful when integrated with a hand-held device   Mechanical Tracker     e.g. Boom3C High accuracy and low latency due to its electromechanical nature Restricted active volume (movement) 14 Motion Tracking (Cont.)  Optical Tracker  Infrared video cameras that record the movement of a person   Attached to the person is a collection of markers in the form of small balls fixed to a critical joints When the moving person is illuminated with infrared light the marker balls are readily detected within the video images    Fast and low latency The system depends on the line-of-sight, so the orientation of the cameras must ensure that the markers are always visible Often prone to interference caused by ambient lighting conditions ARTTrack1 and ARTTrack2, by Advanced Realtime Tracking Inc. (http://www.ar-tracking.de) 15 Motion Tracking (Cont.)  Ultrasonic Tracker     Ultrasonic sound waves are used to locate the user’s position and orientation Usually used for fishtank VR in which the ultrasonic tracker is placed on the top of the monitor and records the user’s head movements Simple and low cost Slow, restricted active volume, sensitive to temperature and depends on the line-of- sight Logitech Ultrasonic Head Tracker (http://www.i-glassesstore.com/logtractracs.html) 16 Motion Tracking (Cont.)  Electromagnetic Tracker  Employ a device called a source that emits an electromagnetic field, and a sensor that detects the radiated field   The source, which can be no bigger than a 2-inch cube, can be placed on a table or fixed to a ceiling The sensor is even smaller and is readily attached to an HMD or fitted within a 3D mouse   Fast and very low latency; no light-of-sight restriction Restricted active volume and are prone to interference of metallic objects miniBIRD, by Ascension Technology Corp. (http://www.ascension-tech.com/ products/minibird.php) 17 Interaction Devices   Usage  Allowing users to interact with virtual objects Hand-held device containing a tracker sensor and some buttons, used for navigating or picking objects within a VE 6 DOF operations  SpaceMouse/SpaceBall    Transitions in X, Y, Z axes and rotations around X (pitch), Y (yaw), and Z (roll) axes Some allow zooming in/out objects SpaceMouseTM, SpaceBallTM 5000, by 3DConnexion Corp. (http://www.vrlogic.com/html/3dconnexion /3d_connexion.html) 18 Interaction Devices (Cont.)  Gloves    Gloves equipped with sensors that track the user’s hand movement Enable natural interaction with objects Modern VR gloves are used to communicate hand gestures (such as pointing and grasping) and in some cases return tactile signals to the user’s hand Pinch Gloves, by Fakespace lab (http://www.fakespacelabs.com/tools.html) 19 Haptic Devices in VR  Usage  A haptic device gives people a sense of touch with computer generated environments, so that when virtual objects are touched, they seem real and tangible  e.g. A medical training simulator in which a doctor can feel a scalpel cut through virtual skin, feel a needle push through virtual tissue, or feel a drill drilling through virtual bone  Current Technologies    Force feedback joystick Virtual styluses  Sensable phantom series, by SensAble Technologies Immersion cyber series, by Immersion Corp. Virtual gloves  20 Force Feedback Joystick  Force Feedback Joystick  A device allowing the users to feel force of magnitude and orientation, aside from measurement of depression and twist of its stick Rumble Pak, by Nintendo (In most console video game systems today) 21 Sensable Phantom Series  By SensAble Technologies (http://www.sensable.com) • Positional sensing: X, Y, Z, pitch, roll, yaw • Force feedback: X, Y, Z • Range of motion: hand movement pivoting at wrist • Maximum force: 1.8 lbs • Intended for use in haptic research and free-form modeling Phantom desktop • Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: hand movement pivoting at wrist • Maximum force: 1.9 lbs Phantom premium 1.0 22 • Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: lower arm movement pivoting at elbow • Maximum force: 1.9 lbs Phantom premium 1.5 • Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: full arm movement pivoting at shoulder • Maximum force: 4.9 lbs Phantom premium 3.0 23 Pros and Cons of Virtual Styluses  Pros     Inexpensive Easy to set up and operate Works on a desktop Well suited for remote manipulation Not immersive Haptic response at a single point only  Cons   24 Virtual Gloves  Immersion cyber series, by Immersion Corp. (http://www.immersion.com/) • Senses position of finger; no force feedback CyberGloveTM • Adds tactile feedback to CyberGlove using vibrations on fingertips or palm • Limited to simple pulses or sustained vibration CyberTouchTM 25 • Force feedback for fingers and hand CyberGraspTM • Force feedback for hand and arm • Can be used together with CyberGrasp CyberForceTM 26 Pros and Cons of Virtual Gloves  Pros   Multiple points of haptic and tactic responses Allows for full immersion with HMDs Expensive Difficult to set up and operate  Cons   27 Production-Stage Applications of VR (Brooks, 1999)    Vehicle Simulation Ergonomics Evaluation and Design Training and Experience 28 Vehicle Simulation   Vehicle Simulation  Was the first application of VR and is still the most advanced 747 simulator at British Airways Merchant ship simulation at Warsash Maritime Center Cases   Ship bridge simulator at Warsash Maritime Center 29 Vehicle Simulation (Cont.)  Lessons Learned  Why are VR vehicle simulators useful?    They are much cheaper to build than the real vehicles They make it possible to thoroughly train operators in extreme situations and emergency procedures where real practice would imperil equipment and lives Scenarios can be easily run and modified, enabling more efficient training Immersion is complete (or nearly complete) The near-field haptics are perfect Realism of the graphics Realism of the sound Realism of the haptics Realism of the motion Realism of the interaction (how does the display respond to the user’s actions?) 30  What aspects of VR make it work so well?    What aspects of VR are critical to success?      Ergonomics Evaluation and Design  Cases    Evaluating ergonomics in cars, at Daimler-Chrysler Technology Center Submarine design at General Dynamics Design review at John Deere View of virtual wind-shield wiper visibility at Daimler-Chrysler’s Technology Center 31 Ergonomics Evaluation and Design (Cont.)  Lessons Learned  Why is VR useful for ergonomics evaluation and design?    Facilitate communication of ideas among team members Save the cost of materials used to develop physical prototypes Speed up design review and change cycles True scales of the modeled objects The farther the design gets from its conceiver, the better the visuals need to be, in order to enable a factory-floor foreman or an operations person to get an accurate, internalized perception of the design  What aspects of VR are critical to success?   32 Training and Experience  Cases   Astronaut training at NASA-Huston Psychiatric treatment at Georgia Tech and Emory University Medical School     Fear of flying Fear of heights Fear of public speaking Post-traumatic stress disorder for Vietnam War veterans 33 NASA-Houston’s “Charlotte” (a haptic simulator) virtual weightless mass lets astronauts practice handling weightless massy objects • (a) The psychologist gently leads to the patient into a simulated Vietnam battle scene • (b) Imagery seen by the patient (a) (b) Vietnam War simulation at the Atlanta Veterans Administration Hospital 34 Training and Experience (Cont.)  Lessons Learned  Why is VR useful for training and experience?   For NASA, it offers the ability to simulate unearthly experiences  e.g. flying about in space using the back-mounted flight unit which is designed principally as an emergency device for use if an astronaut’s tether breaks; moving around on the outside of a space vehicle For psychiatry, it can save cost (both money and time) and offers a “safe” form of exposure to traumatic stimuli Immersion Haptics Sound  What aspects of VR are critical to success?    35 Open Challenges (Brooks, 1999)  Technological     Lowering latency to acceptable levels Rendering massive models in real time Choosing which display best fits each application (HMD, CaveTM, benchmark, or panorama) Producing satisfactory haptic augmentation for VR illusions Interacting more effectively with virtual worlds     Systems  Manipulation Specifying travel Wayfinding Modeling the existing worlds Modeling the non-existing worlds 36  Making models of worlds efficiently    Measuring the illusion of presence and its operational effectiveness References   Brooks Jr., F.P. (1999). What’s Real About Virtual Reality. IEEE Computer Graphics and Applications,19(6), 16-27. Sherman, W.R., & Craig, A.B. (2003). Understanding Virtual Reality. San Francisco, CA: Morgan Kaufmann. 37