LIGHTING TECHNIQUES

                                                     Guangwei Yang*, Yebin Liu†

                  * College of Computer Science, Beijing University of Technology, Beijing China PR, 100124
                                      Automation Department, Tsinghua University, China PR, 100084

                           ABSTRACT                                  time rendering ability. However, light field technique [8]
                                                                     requires camera array, which limits the view angle.
    We present a 3D object relighting technique for multi-               Interactive control over illumination has also been
view-multi-lighting (MVML) image sets. Our relighting                addressed in recent works. Debevec [9] designs the first
technique is a fusion of multi-view stereo (MVS) technique           light stage to illuminate human faces with an illumination
and image based relighting (IBL) technique. The MVML                 setup that contains dense lighting directions. It then renders
dataset consists of multiple camera view with each view              complex reflection of the face and hair, by recombining the
filmed under multiple time-multiplex illumination modes.             basis images according to light in captured lighting
A multi-view 3D reconstruction algorithm is first applied            environments. However, their investigation is confined to
using traditional multi-view stereo algorithm. After this, the       still images of the face in static poses. Later, Wenger [10]
reconstructed model is relighted through an image based              improves this system by using a sphere of LED light
relighting scheme for each camera view, followed with                sources to capture actor’s performance in approximately
view-independent texture mapping procedure. Interactive              100 lighting conditions every 24th of a second using a high-
relighting results demonstrate our high quality                      speed camera. However, it did not address postproduction
reconstruction accuracy, realistic relighting effects and real-      control over the viewpoint on the subject and limited the
time relighting performance. Moreover, our relighting                view to the subjects head and shoulders.
technique is suitable for dynamic 3D object relighting.                  As for researches that aim at both free-viewpoint control
                                                                     and illumination control. The above light stage systems are
     Index Terms— Multi-view stereo, Image based                     upgraded to be permission of relighting locomotion actors
relighting, relighting, time-multiplex illumination                  [20]. However, this system cannot deal with free-motion
                                                                     actors. The technique they used is the combination of image
                      1. INTRODUCTION                                based lighting and light field. Theobalt [11] uses sparse-ring
                                                                     camera array and only two light sources to address similar
    To integrate a specific 3D object into a virtual                 problem. Because the technique they used is model-based
environment is one of the most desirous applications in the          MVS and BRDF for surface reflectance modeling, their
field of computer graphics. In order to obtain a realistic           work cannot cope with non-human objects and it limited to
visual effect, illumination-consistency as well as changeable        relighting of tight dress persons.
viewpoint is required. This means, the object should be                  Based on the above analysis, in this paper, we
rendered with illumination compatible with the environment           demonstrate that the multi-view images captured under
and permit free-viewpoint control. Although considerable             multiple lights can be fused together for 3D reconstruction
works have been focused on the postproduction control of             and relighting based on multi-view stereo technique [12]
viewpoint and illumination, most proposed systems only               and image based lighting (IBL) technique. Such 3D
tackle either the viewpoint or the illumination.                     reconstruction and relighting allows users to render 3D
    The most efficient way to realize free-viewpoint video of        objects under arbitrary changes in lighting and viewing
motion 3D object is through the setup of multi-camera array          direction. Our multi-view-multi-light (MVML) imagery is
and filming of multi-view video from these cameras.                  captured by a relighting dome [15, 16] which uses LED
Model-based methods [4] or shape-from silhouette methods             array to illuminate the subject from a dense sampling of
[3] or multi-view stereo (MVS) methods [1,2,5] are                   directions of incident illumination and films the lighted
extensively used in these system for reconstruction of 3D            object from multi-view cameras. Compared with available
objects. Light field camera array [6,7] based on Image               systems described above that also permits both viewpoint
based rendering techniques are also popular for their real-          control and illumination control, our relighting dome is a
                                                                     combination of 3D video studio and light stage.
*Guangwei Yang is an intern in Tsinghua University
                                                                     Furthermore, our system can be extended to reconstruct and
relight motion objects by using multiple high speed cameras     matrix used for lighting control is a 31 by 31 matrix S, with
and temporal registration technique.                            each row denotes a lighting mode at one instant and each
    The rest of this paper proceeds with the acquisition        column denotes a cycling pattern of a bunch of light. Each
environment used to record the multi-view-multi-light           element has a value of either 0 or 1 with 1 indicates on and
images in Section 2. The relighting algorithm consists of       0 indicates off. The Hadamard matrix is shown in Fig.2.
individual image relighting, 3D reconstruction and 3D           The whole cycle is 31 patterns plus 1 full light pattern with
model relighting is presented in Section 3. An interactive      all lights on. Each row or column has n elements: with
viewpoint and lighting control software interface and our       (n+1)/2 have the value 1 and (n-1)/2 have the value 0. Thus,
relighting results are presented in Section 4. The paper        the light energy corresponding to a little more than half of
concludes in Section 5 with a discussion of features and        the sources is captured in each acquired multiplexed
gives an outlook to future work.                                measurement. Each row of the matrix S is given by a cyclic
                                                                permutation of the first row vector.

Our acquisition setup is designed to efficiently capture 2D
images of a person or object from arbitrary viewpoint under
arbitrary illumination, for 3D model reconstruction and
relighting. Our DOME system is a hemisphere with the
diameter of 6m, with 20 low speed FLEA2 cameras (30fps)
locating on the ring, and with 310 Luxeon K2 LEDs                      Fig.2. Hadamard matrix for lighting control
spreading over the whole hemisphere. For the cameras, the
triggering mode with resolution of 1024×768 is adopted.                      3. RELIGHTING ALGORITHM
For the LEDs, 10 neighboring LEDs are clustered as an area
to provide one beam of light, which is to say, the                   Our 3D relighting algorithm is realized through the
hemisphere is divided into 31 areas, with every area of         following three steps: first relight each camera view
LEDs give out one beam. In view of controlling these            according to the new illumination. Second is to reconstruct
cameras and LEDs, a circuit system is built to trigger the      the 3D model using the full lighting images. Finally, we
cameras and ignite the LEDs at will. The circuit system         texture the 3D model using the new illuminated camera
consists of 33 microcontroller boards based on Cygnal’s         images.
C8051F040 running at 22MHz. One of these
microcontroller boards is master controller, sending a global   3.1 Individual view relighting
sync pulse to other 32 slavery boards via CAN Bus, driving           After Hadamard images have been captured, scene
the lighting sequence and triggering the cameras. Our           illuminated under individual light can be obtained by
system is capable of capturing high speed motion by low         decoding each image pixel based on the invert matrix S-1 as
speed cameras at arbitrary time spot under controlled light.    described in [9]:
Fig.1 shows multiple lighting images filmed in 30 kinds of                                                               (1)
                                                                              T = S − 1 = [2 /( n + 1)](2 S t − 1n × n )
lighting modes from one of the camera.
                                                                    For each pixel location (x, y) in each camera c, we
                                                                observe that location illuminated from 31 kinds of
                                                                Hadamard lighting patterns with index j (j=1, 2…31).
                                                                Decoded image pixel under single lighting d (d=1, 2…31)
                                                                can be computed as:
                                                                               I c ,d ( x, y ) = ∑ Td , j I c , j ( x, y ) (2)
                                                                                             j =1....31

                                                                Here, Ic,d and Ic,j are the single lighting image and the jth
                                                                input lighting image of camera c respectively.
                                                                    After this, suppose we wish to generate an image of
                                                                camera c in a novel form of illumination. Since I c ,d ( x, y )
Fig.1. Multiple lighting images filmed by one of camera under   represents how much light is reflected toward the camera c
       multiple lights                                          by pixel (x, y) as a result of illumination from light d, and
                                                                since light is additive, we can compute an image of camera
     For reconstruction and relighting of static object, all            ˆ
                                                                image I c ( x, y ) under any combination of the light sources
the 20 cameras are synchronized and triggered at the same
time. Hadamard pattern based multiplex illumination [17] is     τ (d ) as follows:
adopted to increase the signal to noise ratios. The Hadamard
                     I c ( x, y ) = ∑ I c , d ( x, y )τ (d ) .
                     ˆ                                             (3)   vertex is potentially visible from several camera views,
                                     d                                   pixel must be chosen from these cameras. In our experiment,
Here, τ (d ) is obtained through the decomposing of the                  we use the camera for which the angle between viewing
environment map and describes the light intensity of light d.            vector and vertex normal is the smallest.
Note that for each color channel (R, G and B) is computed
using the above equation independently. Fig.3 shows the                        4. 3D OBJECT RELIGTING SOFTWARE
decomposition of the HDR environment maps according to
our 31-light configuration.                                              To demonstrate the interactive real time rendering ability of
                                                                         our relighting algorithm, we have implemented relighting
                                                                         software as shown in figure 5. This software permits
                                                                         loading of HDR environment map and also virtual
                                                                         environment map (LDR images) for rendering of 3D mesh
                                                                         model chose providing real time rendering of the 3D model
                                                                         object for interactive viewpoint and illumination control.
         (a)              (b)              (c)             (d)           Here, the main components on the panel are labeled and
  Fig.2. Two lighting environments (a, c) from [9] and their             described, where 1 is the viewing window for viewpoint
 spherical projections (b, d) onto the 31-element lighting basis         control, 2 is the environment sphere capable for rotation, 3
                                                                         is the rectangular map of the 31 lighting and 4 is the global
3.2 Multi-view stereo reconstruction                                     intensity controller. This software is public in our website
   For multiple view images under constant light, standard               [16] .
multi-view stereo algorithm such as graph-cuts based MVS
[13] or patch based MVS [18] can be applied to reconstruct
the 3D model. We use point cloud based 3D reconstruction
algorithm [14, 15] for this job. PCMVS [14] is a model-free
3D reconstruction algorithm and it is accurate and robust to
challenges such as restricted image resolution, unfavorable
color synchronization, image noises and invisible surface
regions. These challenges are commonly present in multi-
camera system.

                                                                                    Fig. 5. Interactive 3D object relighting software

   Fig.4. Multi-view stereo reconstruction for multi-view constant
               lighting image using PMVS in [14]

3.3 3D model relighting
    After the mesh model of the 3D object is obtained,
relighting can be implemented by texturing the mesh facets
using the new relighted images. Since reconstructed model
is a triangle mesh with dense triangle vertices, we can just
set color values of each triangle vertices for appearance
rendering. For each camera, vertex visibility is determined
                                                                           Fig.6. Simulation of relighting results using spot light. (a) is the
using the direct visibility checking algorithm [19]. This
                                                                          appearance from one viewpoint with changing lights. (b)~(e) are
visibility is determined without reconstruction of a surface             the appearance from different viewpoints under one of the lighting
or estimating surface normal. Moreover, the algorithm is                                          environment in (a)
general and can be applied to point clouds at various
dimensions, on both sparse and dense point clouds. Since a
  Fig.6 illustrates the relighting results of our reconstructed    [1]    P.Rander, P.J.Narayanan, T.Kanade: Virtualized reality:
model. Fig.6(a) simulates the visual appearance by                        Constructing time-varying virtual worlds from real events.
smoothly rotating the single light illuminating on the model.             Proceedings of IEEE Visualization (Phoenix, Arizona, 1997),
The results demonstrate our consistency relighting                        pp. 277–283.
                                                                   [2]    S. Momezzi, A. Katkere, D .Y.Kuramura, R.Jain: Reality
capability. Fig.6(b) to Fig.6(e) are the multiple view images             modeling and visualization from multiple video sequences.
from one of the instant of the changing lighting.                         IEEE Computer Graphics & Applications 16, 6 (Nov. 1996),
  Fig.7 shows the model appearance under different                        58–63.
environment maps. The images in the first row and the              [3]    W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, L.
second row are the relighting results under two of the                    Mcmillan: Image-based visual hulls. Proc. SIGGRAPH 2000
viewing angles. The last line shows the corresponding                     (July 2000), pp. 369– 374.
environment maps we used. We achieve natural relighting            [4]    J.Carranza, C.Theobalt, M.Magnor, H.P.Seidel: Free-
result using these environments.                                          viewpoint video of human actors. ACM Transactions on
                                                                          Graphics 22, 3 (July 2003), 569–577.
                                                                   [5]    S. Vedula, S.Baker, T.kanade: Image based spatio-temporal
                                                                          modeling and view interpolation of dynamic events. ACM
                                                                          Transactions on Graphics 24, 2 (Apr. 2005), 240–261.
                                                                   [6]    Yebin Liu, Qionghai Dai, Wenli Xu , A Real Time
                                                                          Interactive Dynamic Light Field Transmission System, in
                                                                          IEEE international conference on multimedia expo.2006,
                                                                          ICME 2006.
                                                                   [7]    Wilburn B, Joshi N., Vaish V., Talvala E.-V., Antunez E.,
                                                                          Barth A., Adams A., Horowitz M., Levoy M.: High
                                                                          performance imaging using large camera arrays. ACM
                                                                          Transactions on Graphics, 24, 3 (Aug. 2005), 765–776.
                                                                   [8]    M.Levoy, and P.Hanrahan, Light field rendering. In
                                                                          SIGGRAPH, 96 (1996), pp. 31–42.
                                                                   [9]    P.Debevec, T.Hawkins, C.Tchou, H.P.Duiker, W.Sarokin,
                                                                          M.Sagar, Acquiring the reflectance field of a human face.
                                                                          Proceedings of SIGGRAPH 2000 (July 2000), 145–156.
                                                                   [10]   Wenger A., Gardner A., Tchou C., Unger J., Hawkins T.,
                                                                          Debevec P.: Performance relighting and reflectance
                                                                          transformation with time multiplexed illumination. ACM
Fig.7. 3D relighting results under different virtual environment
                                                                          Transactions on Graphics, 24, 3 (Aug. 2005), 756–764.
maps. All the environment maps are LDR images.
                                                                   [11]   N.Ahmed, C.Theobalt, M.Magnor, H.P.Seidel, Spatio-
                                                                          Temporal Registration Techniques for Relightable 3D Video,
          5. DISCUSSION AND CONCLUSION                                    in Proc. of ICIP 2007.
                                                                   [12]   S.M.Seitz, B.Curless, J.Diebel, D. Scharstein, and R.Szeliski,
This paper introduces the technique to relight static 3D                  A comparison and evaluation of multi-view stereo
object using multi-view-multi-lighting images. Our work                   reconstruction algorithms, in CVPR 2006.
combines existing techniques to realize free-viewpoint             [13]   G. Vogiatzis, C. Hernández, P. H. S. Torr, and R. Cipolla,
relighting of 3D object. We demonstrate that by combining                 Multi-view Stereo via Volumetric Graph-cuts and Occlusion
image based relighting algorithm on 3D model, realistic and               Robust Photo-Consistency, IEEE PAMI, Vol. 29, No. 12,
                                                                          pp.2241-2246, 2007.
robust relighting results can be obtained. Moreover, we
                                                                   [14]   Yebin. Liu, Qionghai.Dai, Wenli.Xu, “A Point Cloud based
demonstrate the real time capability of interactive viewpoint             Multi-view Stereo Algorithm for Free-viewpoint Video”,
control and lighting control on the reconstructed 3D object.              accepted in IEEE Trans on Visualization and Computer
See our public website [16] for our demonstrations of this                Graphics, 2009.
work.                                                              [15]   http://media.au.tsinghua.edu.cn/ fvv.jsp
   Our 3D reconstruction and relighting technique can be           [16]   http://media.au.tsinghua.edu.cn/dome.jsp
further improved for relighting of 3D motion objects by            [17]   Y.Y.Schechner,       S.K.Nayar,       and      P.N.Belhumeur,
upgrading the cameras to high speed cameras. Actually,                    Multiplexing for Optimal Lighting, in IEEE Transactions on
optical flow tracking algorithm can be adopted to register                pattern analysis and machine intelligence, Vol.29, no.8, 2007.
                                                                   [18]   Y.Furukawa and J.Ponce. Accurate, dense, and robust
high speed lighting information to every low speed key
                                                                          multiview stereopsis. In CVPR, 2007
frames similar to the video relighting system [10], which          [19]   Katz, S., A. Tal, and R. Basri, Direct Visibility of Point Sets,
will be our future works.                                                 in Siggraph. 2008.
                                                                   [20]   P.Einarsson, C.F.Chabert, A.Jones, W.C.Ma, B.Lamond,
                       6. REFERENCES                                      T.Hawkins, M.Bolas, S.Sylwan, P.Debevec, Relighting
                                                                          Human Locomotion with Flowed Reflectance Fields, 2006
                                                                          Eurographics Symposium on Rendering.

To top