Construction and Presentation of a Virtual Environment Using Panoramic Stereo Images of a Real Scene and Computer Graphics Models
Jun Shimamura†, Haruo Takemura‡, Naokazu Yokoya‡, Kazumasa Yamazawa‡ †NTT Cyber Space Laboratory, ‡Nara Institute of Science and Technology JAPAN
Background
Construction of a large-scale virtualized environment ex. urban area, natural scenes, …
Requirements in constructing an immersive virtualized environment
Approach
• Uses a video-rate omni-directional stereo imaging sensor • Constructs a full panoramic 3-D model using a cylindrical panoramic stereo image sequence • Constructs a prototype system of presenting a mixed environment ( virtualized, CG )
Full panoramic representation Depth information from the real world Digitizing and representing dynamic events
Omni-directional stereo imaging sensor
virtual center of lens
Composed of twelve CCD cameras and two hexagonal pyramidal mirrors
upper component lens center pyramidal mirror camera
lower component
• Single viewpoint constraint • High-resolution image acquisition • Omni-directional stereoscopic imaging at video-rate
Appearance
front view
top view
Geometry
Generation of panoramic stereo images i. Elimination of geometric distortion in images, ii. Color adjustment of camera images, iii. Concatenation of six images for completing upper and lower omni-directional images of a stereo pair
A pair of panoramic images
3006 x 330 pixels
Virtualizing a dynamic real scene
• Layered representation of dynamic real scene Static scene image generation
• A panoramic image of a static scene is generated by applying a temporal mode filter to a panoramic image sequence in a time interval
static image
Extraction of moving objects
• Moving objects are extracted by subtracting consecutive image frames in a sequence
subtracting
panoramic image sequence including dynamic event
majority filtering
• Depth estimation from static panoramic stereo images and moving objects
upper panoramic image
Extracted moving object regions
• Stereo matching used to estimate depths of static scene and moving objects
epipolar line
• Generation of 3-D model
lower panoramic image world cylindrical coordinate
Apply Delaunay’s triangulation
Panoramic depth map of static scene
Bird’s-eye view of texture-mapped 3-D static scene model
Prototype system
Immersive mixed reality system i. Has a 330-degree cylindrical screen ii. Merges virtual objects with a virtualized real world scene iii. Superimposes dynamic event layers onto static scene layer i. Hardware configuration of immersive mixed reality system ii. Mixed environment observed from different viewpoints
User’s appearance in mixed environment using CYLINDRA system
Frame rate: 13 frames/sec.
iii. Superimposing dynamic event layers onto a static scene layer
Original viewpoint New higher viewpoint
6144 x 768 pixels resolution
Background: 13,400 polygons CG tree: 41,340 polygons Total: 54,740 polygons