VIEWS: 6 PAGES: 4 POSTED ON: 9/28/2011
3D OBJECT RELIGHTING BASED ON MULTI-VIEW STEREO AND IMAGE BASED LIGHTING TECHNIQUES Guangwei Yang*, Yebin Liu† * College of Computer Science, Beijing University of Technology, Beijing China PR, 100124 † Automation Department, Tsinghua University, China PR, 100084 ABSTRACT time rendering ability. However, light field technique  requires camera array, which limits the view angle. We present a 3D object relighting technique for multi- Interactive control over illumination has also been view-multi-lighting (MVML) image sets. Our relighting addressed in recent works. Debevec  designs the first technique is a fusion of multi-view stereo (MVS) technique light stage to illuminate human faces with an illumination and image based relighting (IBL) technique. The MVML setup that contains dense lighting directions. It then renders dataset consists of multiple camera view with each view complex reflection of the face and hair, by recombining the filmed under multiple time-multiplex illumination modes. basis images according to light in captured lighting A multi-view 3D reconstruction algorithm is first applied environments. However, their investigation is confined to using traditional multi-view stereo algorithm. After this, the still images of the face in static poses. Later, Wenger  reconstructed model is relighted through an image based improves this system by using a sphere of LED light relighting scheme for each camera view, followed with sources to capture actor’s performance in approximately view-independent texture mapping procedure. Interactive 100 lighting conditions every 24th of a second using a high- relighting results demonstrate our high quality speed camera. However, it did not address postproduction reconstruction accuracy, realistic relighting effects and real- control over the viewpoint on the subject and limited the time relighting performance. Moreover, our relighting view to the subjects head and shoulders. technique is suitable for dynamic 3D object relighting. As for researches that aim at both free-viewpoint control and illumination control. The above light stage systems are Index Terms— Multi-view stereo, Image based upgraded to be permission of relighting locomotion actors relighting, relighting, time-multiplex illumination . However, this system cannot deal with free-motion actors. The technique they used is the combination of image 1. INTRODUCTION based lighting and light field. Theobalt  uses sparse-ring camera array and only two light sources to address similar To integrate a specific 3D object into a virtual problem. Because the technique they used is model-based environment is one of the most desirous applications in the MVS and BRDF for surface reflectance modeling, their field of computer graphics. In order to obtain a realistic work cannot cope with non-human objects and it limited to visual effect, illumination-consistency as well as changeable relighting of tight dress persons. viewpoint is required. This means, the object should be Based on the above analysis, in this paper, we rendered with illumination compatible with the environment demonstrate that the multi-view images captured under and permit free-viewpoint control. Although considerable multiple lights can be fused together for 3D reconstruction works have been focused on the postproduction control of and relighting based on multi-view stereo technique  viewpoint and illumination, most proposed systems only and image based lighting (IBL) technique. Such 3D tackle either the viewpoint or the illumination. reconstruction and relighting allows users to render 3D The most efficient way to realize free-viewpoint video of objects under arbitrary changes in lighting and viewing motion 3D object is through the setup of multi-camera array direction. Our multi-view-multi-light (MVML) imagery is and filming of multi-view video from these cameras. captured by a relighting dome [15, 16] which uses LED Model-based methods  or shape-from silhouette methods array to illuminate the subject from a dense sampling of  or multi-view stereo (MVS) methods [1,2,5] are directions of incident illumination and films the lighted extensively used in these system for reconstruction of 3D object from multi-view cameras. Compared with available objects. Light field camera array [6,7] based on Image systems described above that also permits both viewpoint based rendering techniques are also popular for their real- control and illumination control, our relighting dome is a combination of 3D video studio and light stage. *Guangwei Yang is an intern in Tsinghua University Furthermore, our system can be extended to reconstruct and relight motion objects by using multiple high speed cameras matrix used for lighting control is a 31 by 31 matrix S, with and temporal registration technique. each row denotes a lighting mode at one instant and each The rest of this paper proceeds with the acquisition column denotes a cycling pattern of a bunch of light. Each environment used to record the multi-view-multi-light element has a value of either 0 or 1 with 1 indicates on and images in Section 2. The relighting algorithm consists of 0 indicates off. The Hadamard matrix is shown in Fig.2. individual image relighting, 3D reconstruction and 3D The whole cycle is 31 patterns plus 1 full light pattern with model relighting is presented in Section 3. An interactive all lights on. Each row or column has n elements: with viewpoint and lighting control software interface and our (n+1)/2 have the value 1 and (n-1)/2 have the value 0. Thus, relighting results are presented in Section 4. The paper the light energy corresponding to a little more than half of concludes in Section 5 with a discussion of features and the sources is captured in each acquired multiplexed gives an outlook to future work. measurement. Each row of the matrix S is given by a cyclic permutation of the first row vector. 2. MVML IMAGE ACQUISITION Our acquisition setup is designed to efficiently capture 2D images of a person or object from arbitrary viewpoint under arbitrary illumination, for 3D model reconstruction and relighting. Our DOME system is a hemisphere with the diameter of 6m, with 20 low speed FLEA2 cameras (30fps) locating on the ring, and with 310 Luxeon K2 LEDs Fig.2. Hadamard matrix for lighting control spreading over the whole hemisphere. For the cameras, the triggering mode with resolution of 1024×768 is adopted. 3. RELIGHTING ALGORITHM For the LEDs, 10 neighboring LEDs are clustered as an area to provide one beam of light, which is to say, the Our 3D relighting algorithm is realized through the hemisphere is divided into 31 areas, with every area of following three steps: first relight each camera view LEDs give out one beam. In view of controlling these according to the new illumination. Second is to reconstruct cameras and LEDs, a circuit system is built to trigger the the 3D model using the full lighting images. Finally, we cameras and ignite the LEDs at will. The circuit system texture the 3D model using the new illuminated camera consists of 33 microcontroller boards based on Cygnal’s images. C8051F040 running at 22MHz. One of these microcontroller boards is master controller, sending a global 3.1 Individual view relighting sync pulse to other 32 slavery boards via CAN Bus, driving After Hadamard images have been captured, scene the lighting sequence and triggering the cameras. Our illuminated under individual light can be obtained by system is capable of capturing high speed motion by low decoding each image pixel based on the invert matrix S-1 as speed cameras at arbitrary time spot under controlled light. described in ： Fig.1 shows multiple lighting images filmed in 30 kinds of (1) T = S − 1 = [2 /( n + 1)](2 S t − 1n × n ) lighting modes from one of the camera. For each pixel location (x, y) in each camera c, we observe that location illuminated from 31 kinds of Hadamard lighting patterns with index j (j=1, 2…31). Decoded image pixel under single lighting d (d=1, 2…31) can be computed as: I c ,d ( x, y ) = ∑ Td , j I c , j ( x, y ) (2) j =1....31 Here, Ic,d and Ic,j are the single lighting image and the jth input lighting image of camera c respectively. After this, suppose we wish to generate an image of camera c in a novel form of illumination. Since I c ,d ( x, y ) Fig.1. Multiple lighting images filmed by one of camera under represents how much light is reflected toward the camera c multiple lights by pixel (x, y) as a result of illumination from light d, and since light is additive, we can compute an image of camera For reconstruction and relighting of static object, all ˆ image I c ( x, y ) under any combination of the light sources the 20 cameras are synchronized and triggered at the same time. Hadamard pattern based multiplex illumination  is τ (d ) as follows: adopted to increase the signal to noise ratios. The Hadamard I c ( x, y ) = ∑ I c , d ( x, y )τ (d ) . ˆ (3) vertex is potentially visible from several camera views, d pixel must be chosen from these cameras. In our experiment, Here, τ (d ) is obtained through the decomposing of the we use the camera for which the angle between viewing environment map and describes the light intensity of light d. vector and vertex normal is the smallest. Note that for each color channel (R, G and B) is computed using the above equation independently. Fig.3 shows the 4. 3D OBJECT RELIGTING SOFTWARE decomposition of the HDR environment maps according to our 31-light configuration. To demonstrate the interactive real time rendering ability of our relighting algorithm, we have implemented relighting software as shown in figure 5. This software permits loading of HDR environment map and also virtual environment map (LDR images) for rendering of 3D mesh model chose providing real time rendering of the 3D model object for interactive viewpoint and illumination control. (a) (b) (c) (d) Here, the main components on the panel are labeled and Fig.2. Two lighting environments (a, c) from  and their described, where 1 is the viewing window for viewpoint spherical projections (b, d) onto the 31-element lighting basis control, 2 is the environment sphere capable for rotation, 3 is the rectangular map of the 31 lighting and 4 is the global 3.2 Multi-view stereo reconstruction intensity controller. This software is public in our website For multiple view images under constant light, standard  . multi-view stereo algorithm such as graph-cuts based MVS  or patch based MVS  can be applied to reconstruct the 3D model. We use point cloud based 3D reconstruction algorithm [14, 15] for this job. PCMVS  is a model-free 3D reconstruction algorithm and it is accurate and robust to challenges such as restricted image resolution, unfavorable color synchronization, image noises and invisible surface regions. These challenges are commonly present in multi- camera system. Fig. 5. Interactive 3D object relighting software Fig.4. Multi-view stereo reconstruction for multi-view constant lighting image using PMVS in  3.3 3D model relighting After the mesh model of the 3D object is obtained, relighting can be implemented by texturing the mesh facets using the new relighted images. Since reconstructed model is a triangle mesh with dense triangle vertices, we can just set color values of each triangle vertices for appearance rendering. For each camera, vertex visibility is determined Fig.6. Simulation of relighting results using spot light. (a) is the using the direct visibility checking algorithm . This appearance from one viewpoint with changing lights. (b)~(e) are visibility is determined without reconstruction of a surface the appearance from different viewpoints under one of the lighting or estimating surface normal. Moreover, the algorithm is environment in (a) general and can be applied to point clouds at various dimensions, on both sparse and dense point clouds. Since a Fig.6 illustrates the relighting results of our reconstructed  P.Rander, P.J.Narayanan, T.Kanade: Virtualized reality: model. Fig.6(a) simulates the visual appearance by Constructing time-varying virtual worlds from real events. smoothly rotating the single light illuminating on the model. Proceedings of IEEE Visualization (Phoenix, Arizona, 1997), The results demonstrate our consistency relighting pp. 277–283.  S. Momezzi, A. Katkere, D .Y.Kuramura, R.Jain: Reality capability. Fig.6(b) to Fig.6(e) are the multiple view images modeling and visualization from multiple video sequences. from one of the instant of the changing lighting. IEEE Computer Graphics & Applications 16, 6 (Nov. 1996), Fig.7 shows the model appearance under different 58–63. environment maps. The images in the first row and the  W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, L. second row are the relighting results under two of the Mcmillan: Image-based visual hulls. Proc. SIGGRAPH 2000 viewing angles. The last line shows the corresponding (July 2000), pp. 369– 374. environment maps we used. We achieve natural relighting  J.Carranza, C.Theobalt, M.Magnor, H.P.Seidel: Free- result using these environments. viewpoint video of human actors. ACM Transactions on Graphics 22, 3 (July 2003), 569–577.  S. Vedula, S.Baker, T.kanade: Image based spatio-temporal modeling and view interpolation of dynamic events. ACM Transactions on Graphics 24, 2 (Apr. 2005), 240–261.  Yebin Liu, Qionghai Dai, Wenli Xu ， A Real Time Interactive Dynamic Light Field Transmission System, in IEEE international conference on multimedia expo.2006, ICME 2006.  Wilburn B, Joshi N., Vaish V., Talvala E.-V., Antunez E., Barth A., Adams A., Horowitz M., Levoy M.: High performance imaging using large camera arrays. ACM Transactions on Graphics, 24, 3 (Aug. 2005), 765–776.  M.Levoy, and P.Hanrahan, Light field rendering. In SIGGRAPH, 96 (1996), pp. 31–42.  P.Debevec, T.Hawkins, C.Tchou, H.P.Duiker, W.Sarokin, M.Sagar, Acquiring the reflectance field of a human face. Proceedings of SIGGRAPH 2000 (July 2000), 145–156.  Wenger A., Gardner A., Tchou C., Unger J., Hawkins T., Debevec P.: Performance relighting and reflectance transformation with time multiplexed illumination. ACM Fig.7. 3D relighting results under different virtual environment Transactions on Graphics, 24, 3 (Aug. 2005), 756–764. maps. All the environment maps are LDR images.  N.Ahmed, C.Theobalt, M.Magnor, H.P.Seidel, Spatio- Temporal Registration Techniques for Relightable 3D Video, 5. DISCUSSION AND CONCLUSION in Proc. of ICIP 2007.  S.M.Seitz, B.Curless, J.Diebel, D. Scharstein, and R.Szeliski, This paper introduces the technique to relight static 3D A comparison and evaluation of multi-view stereo object using multi-view-multi-lighting images. Our work reconstruction algorithms, in CVPR 2006. combines existing techniques to realize free-viewpoint  G. Vogiatzis, C. Hernández, P. H. S. Torr, and R. Cipolla, relighting of 3D object. We demonstrate that by combining Multi-view Stereo via Volumetric Graph-cuts and Occlusion image based relighting algorithm on 3D model, realistic and Robust Photo-Consistency, IEEE PAMI, Vol. 29, No. 12, pp.2241-2246, 2007. robust relighting results can be obtained. Moreover, we  Yebin. Liu, Qionghai.Dai, Wenli.Xu, “A Point Cloud based demonstrate the real time capability of interactive viewpoint Multi-view Stereo Algorithm for Free-viewpoint Video”, control and lighting control on the reconstructed 3D object. accepted in IEEE Trans on Visualization and Computer See our public website  for our demonstrations of this Graphics, 2009. work.  http://media.au.tsinghua.edu.cn/ fvv.jsp Our 3D reconstruction and relighting technique can be  http://media.au.tsinghua.edu.cn/dome.jsp further improved for relighting of 3D motion objects by  Y.Y.Schechner, S.K.Nayar, and P.N.Belhumeur, upgrading the cameras to high speed cameras. Actually, Multiplexing for Optimal Lighting, in IEEE Transactions on optical flow tracking algorithm can be adopted to register pattern analysis and machine intelligence, Vol.29, no.8, 2007.  Y.Furukawa and J.Ponce. Accurate, dense, and robust high speed lighting information to every low speed key multiview stereopsis. In CVPR, 2007 frames similar to the video relighting system , which  Katz, S., A. Tal, and R. Basri, Direct Visibility of Point Sets, will be our future works. in Siggraph. 2008.  P.Einarsson, C.F.Chabert, A.Jones, W.C.Ma, B.Lamond, 6. REFERENCES T.Hawkins, M.Bolas, S.Sylwan, P.Debevec, Relighting Human Locomotion with Flowed Reflectance Fields, 2006 Eurographics Symposium on Rendering.
"3D OBJECT RELIGHTING BASED ON MULTI-VIEW STEREO AND "