3D OBJECT RELIGHTING BASED ON MULTI-VIEW STEREO AND IMAGE BASED
Guangwei Yang*, Yebin Liu†
* College of Computer Science, Beijing University of Technology, Beijing China PR, 100124
Automation Department, Tsinghua University, China PR, 100084
ABSTRACT time rendering ability. However, light field technique 
requires camera array, which limits the view angle.
We present a 3D object relighting technique for multi- Interactive control over illumination has also been
view-multi-lighting (MVML) image sets. Our relighting addressed in recent works. Debevec  designs the first
technique is a fusion of multi-view stereo (MVS) technique light stage to illuminate human faces with an illumination
and image based relighting (IBL) technique. The MVML setup that contains dense lighting directions. It then renders
dataset consists of multiple camera view with each view complex reflection of the face and hair, by recombining the
filmed under multiple time-multiplex illumination modes. basis images according to light in captured lighting
A multi-view 3D reconstruction algorithm is first applied environments. However, their investigation is confined to
using traditional multi-view stereo algorithm. After this, the still images of the face in static poses. Later, Wenger 
reconstructed model is relighted through an image based improves this system by using a sphere of LED light
relighting scheme for each camera view, followed with sources to capture actor’s performance in approximately
view-independent texture mapping procedure. Interactive 100 lighting conditions every 24th of a second using a high-
relighting results demonstrate our high quality speed camera. However, it did not address postproduction
reconstruction accuracy, realistic relighting effects and real- control over the viewpoint on the subject and limited the
time relighting performance. Moreover, our relighting view to the subjects head and shoulders.
technique is suitable for dynamic 3D object relighting. As for researches that aim at both free-viewpoint control
and illumination control. The above light stage systems are
Index Terms— Multi-view stereo, Image based upgraded to be permission of relighting locomotion actors
relighting, relighting, time-multiplex illumination . However, this system cannot deal with free-motion
actors. The technique they used is the combination of image
1. INTRODUCTION based lighting and light field. Theobalt  uses sparse-ring
camera array and only two light sources to address similar
To integrate a specific 3D object into a virtual problem. Because the technique they used is model-based
environment is one of the most desirous applications in the MVS and BRDF for surface reflectance modeling, their
field of computer graphics. In order to obtain a realistic work cannot cope with non-human objects and it limited to
visual effect, illumination-consistency as well as changeable relighting of tight dress persons.
viewpoint is required. This means, the object should be Based on the above analysis, in this paper, we
rendered with illumination compatible with the environment demonstrate that the multi-view images captured under
and permit free-viewpoint control. Although considerable multiple lights can be fused together for 3D reconstruction
works have been focused on the postproduction control of and relighting based on multi-view stereo technique 
viewpoint and illumination, most proposed systems only and image based lighting (IBL) technique. Such 3D
tackle either the viewpoint or the illumination. reconstruction and relighting allows users to render 3D
The most efficient way to realize free-viewpoint video of objects under arbitrary changes in lighting and viewing
motion 3D object is through the setup of multi-camera array direction. Our multi-view-multi-light (MVML) imagery is
and filming of multi-view video from these cameras. captured by a relighting dome [15, 16] which uses LED
Model-based methods  or shape-from silhouette methods array to illuminate the subject from a dense sampling of
 or multi-view stereo (MVS) methods [1,2,5] are directions of incident illumination and films the lighted
extensively used in these system for reconstruction of 3D object from multi-view cameras. Compared with available
objects. Light field camera array [6,7] based on Image systems described above that also permits both viewpoint
based rendering techniques are also popular for their real- control and illumination control, our relighting dome is a
combination of 3D video studio and light stage.
*Guangwei Yang is an intern in Tsinghua University
Furthermore, our system can be extended to reconstruct and
relight motion objects by using multiple high speed cameras matrix used for lighting control is a 31 by 31 matrix S, with
and temporal registration technique. each row denotes a lighting mode at one instant and each
The rest of this paper proceeds with the acquisition column denotes a cycling pattern of a bunch of light. Each
environment used to record the multi-view-multi-light element has a value of either 0 or 1 with 1 indicates on and
images in Section 2. The relighting algorithm consists of 0 indicates off. The Hadamard matrix is shown in Fig.2.
individual image relighting, 3D reconstruction and 3D The whole cycle is 31 patterns plus 1 full light pattern with
model relighting is presented in Section 3. An interactive all lights on. Each row or column has n elements: with
viewpoint and lighting control software interface and our (n+1)/2 have the value 1 and (n-1)/2 have the value 0. Thus,
relighting results are presented in Section 4. The paper the light energy corresponding to a little more than half of
concludes in Section 5 with a discussion of features and the sources is captured in each acquired multiplexed
gives an outlook to future work. measurement. Each row of the matrix S is given by a cyclic
permutation of the first row vector.
2. MVML IMAGE ACQUISITION
Our acquisition setup is designed to efficiently capture 2D
images of a person or object from arbitrary viewpoint under
arbitrary illumination, for 3D model reconstruction and
relighting. Our DOME system is a hemisphere with the
diameter of 6m, with 20 low speed FLEA2 cameras (30fps)
locating on the ring, and with 310 Luxeon K2 LEDs Fig.2. Hadamard matrix for lighting control
spreading over the whole hemisphere. For the cameras, the
triggering mode with resolution of 1024×768 is adopted. 3. RELIGHTING ALGORITHM
For the LEDs, 10 neighboring LEDs are clustered as an area
to provide one beam of light, which is to say, the Our 3D relighting algorithm is realized through the
hemisphere is divided into 31 areas, with every area of following three steps: first relight each camera view
LEDs give out one beam. In view of controlling these according to the new illumination. Second is to reconstruct
cameras and LEDs, a circuit system is built to trigger the the 3D model using the full lighting images. Finally, we
cameras and ignite the LEDs at will. The circuit system texture the 3D model using the new illuminated camera
consists of 33 microcontroller boards based on Cygnal’s images.
C8051F040 running at 22MHz. One of these
microcontroller boards is master controller, sending a global 3.1 Individual view relighting
sync pulse to other 32 slavery boards via CAN Bus, driving After Hadamard images have been captured, scene
the lighting sequence and triggering the cameras. Our illuminated under individual light can be obtained by
system is capable of capturing high speed motion by low decoding each image pixel based on the invert matrix S-1 as
speed cameras at arbitrary time spot under controlled light. described in ：
Fig.1 shows multiple lighting images filmed in 30 kinds of (1)
T = S − 1 = [2 /( n + 1)](2 S t − 1n × n )
lighting modes from one of the camera.
For each pixel location (x, y) in each camera c, we
observe that location illuminated from 31 kinds of
Hadamard lighting patterns with index j (j=1, 2…31).
Decoded image pixel under single lighting d (d=1, 2…31)
can be computed as:
I c ,d ( x, y ) = ∑ Td , j I c , j ( x, y ) (2)
Here, Ic,d and Ic,j are the single lighting image and the jth
input lighting image of camera c respectively.
After this, suppose we wish to generate an image of
camera c in a novel form of illumination. Since I c ,d ( x, y )
Fig.1. Multiple lighting images filmed by one of camera under represents how much light is reflected toward the camera c
multiple lights by pixel (x, y) as a result of illumination from light d, and
since light is additive, we can compute an image of camera
For reconstruction and relighting of static object, all ˆ
image I c ( x, y ) under any combination of the light sources
the 20 cameras are synchronized and triggered at the same
time. Hadamard pattern based multiplex illumination  is τ (d ) as follows:
adopted to increase the signal to noise ratios. The Hadamard
I c ( x, y ) = ∑ I c , d ( x, y )τ (d ) .
ˆ (3) vertex is potentially visible from several camera views,
d pixel must be chosen from these cameras. In our experiment,
Here, τ (d ) is obtained through the decomposing of the we use the camera for which the angle between viewing
environment map and describes the light intensity of light d. vector and vertex normal is the smallest.
Note that for each color channel (R, G and B) is computed
using the above equation independently. Fig.3 shows the 4. 3D OBJECT RELIGTING SOFTWARE
decomposition of the HDR environment maps according to
our 31-light configuration. To demonstrate the interactive real time rendering ability of
our relighting algorithm, we have implemented relighting
software as shown in figure 5. This software permits
loading of HDR environment map and also virtual
environment map (LDR images) for rendering of 3D mesh
model chose providing real time rendering of the 3D model
object for interactive viewpoint and illumination control.
(a) (b) (c) (d) Here, the main components on the panel are labeled and
Fig.2. Two lighting environments (a, c) from  and their described, where 1 is the viewing window for viewpoint
spherical projections (b, d) onto the 31-element lighting basis control, 2 is the environment sphere capable for rotation, 3
is the rectangular map of the 31 lighting and 4 is the global
3.2 Multi-view stereo reconstruction intensity controller. This software is public in our website
For multiple view images under constant light, standard  .
multi-view stereo algorithm such as graph-cuts based MVS
 or patch based MVS  can be applied to reconstruct
the 3D model. We use point cloud based 3D reconstruction
algorithm [14, 15] for this job. PCMVS  is a model-free
3D reconstruction algorithm and it is accurate and robust to
challenges such as restricted image resolution, unfavorable
color synchronization, image noises and invisible surface
regions. These challenges are commonly present in multi-
Fig. 5. Interactive 3D object relighting software
Fig.4. Multi-view stereo reconstruction for multi-view constant
lighting image using PMVS in 
3.3 3D model relighting
After the mesh model of the 3D object is obtained,
relighting can be implemented by texturing the mesh facets
using the new relighted images. Since reconstructed model
is a triangle mesh with dense triangle vertices, we can just
set color values of each triangle vertices for appearance
rendering. For each camera, vertex visibility is determined
Fig.6. Simulation of relighting results using spot light. (a) is the
using the direct visibility checking algorithm . This
appearance from one viewpoint with changing lights. (b)~(e) are
visibility is determined without reconstruction of a surface the appearance from different viewpoints under one of the lighting
or estimating surface normal. Moreover, the algorithm is environment in (a)
general and can be applied to point clouds at various
dimensions, on both sparse and dense point clouds. Since a
Fig.6 illustrates the relighting results of our reconstructed  P.Rander, P.J.Narayanan, T.Kanade: Virtualized reality:
model. Fig.6(a) simulates the visual appearance by Constructing time-varying virtual worlds from real events.
smoothly rotating the single light illuminating on the model. Proceedings of IEEE Visualization (Phoenix, Arizona, 1997),
The results demonstrate our consistency relighting pp. 277–283.
 S. Momezzi, A. Katkere, D .Y.Kuramura, R.Jain: Reality
capability. Fig.6(b) to Fig.6(e) are the multiple view images modeling and visualization from multiple video sequences.
from one of the instant of the changing lighting. IEEE Computer Graphics & Applications 16, 6 (Nov. 1996),
Fig.7 shows the model appearance under different 58–63.
environment maps. The images in the first row and the  W. Matusik, C. Buehler, R. Raskar, S. J. Gortler, L.
second row are the relighting results under two of the Mcmillan: Image-based visual hulls. Proc. SIGGRAPH 2000
viewing angles. The last line shows the corresponding (July 2000), pp. 369– 374.
environment maps we used. We achieve natural relighting  J.Carranza, C.Theobalt, M.Magnor, H.P.Seidel: Free-
result using these environments. viewpoint video of human actors. ACM Transactions on
Graphics 22, 3 (July 2003), 569–577.
 S. Vedula, S.Baker, T.kanade: Image based spatio-temporal
modeling and view interpolation of dynamic events. ACM
Transactions on Graphics 24, 2 (Apr. 2005), 240–261.
 Yebin Liu, Qionghai Dai, Wenli Xu ， A Real Time
Interactive Dynamic Light Field Transmission System, in
IEEE international conference on multimedia expo.2006,
 Wilburn B, Joshi N., Vaish V., Talvala E.-V., Antunez E.,
Barth A., Adams A., Horowitz M., Levoy M.: High
performance imaging using large camera arrays. ACM
Transactions on Graphics, 24, 3 (Aug. 2005), 765–776.
 M.Levoy, and P.Hanrahan, Light field rendering. In
SIGGRAPH, 96 (1996), pp. 31–42.
 P.Debevec, T.Hawkins, C.Tchou, H.P.Duiker, W.Sarokin,
M.Sagar, Acquiring the reflectance field of a human face.
Proceedings of SIGGRAPH 2000 (July 2000), 145–156.
 Wenger A., Gardner A., Tchou C., Unger J., Hawkins T.,
Debevec P.: Performance relighting and reflectance
transformation with time multiplexed illumination. ACM
Fig.7. 3D relighting results under different virtual environment
Transactions on Graphics, 24, 3 (Aug. 2005), 756–764.
maps. All the environment maps are LDR images.
 N.Ahmed, C.Theobalt, M.Magnor, H.P.Seidel, Spatio-
Temporal Registration Techniques for Relightable 3D Video,
5. DISCUSSION AND CONCLUSION in Proc. of ICIP 2007.
 S.M.Seitz, B.Curless, J.Diebel, D. Scharstein, and R.Szeliski,
This paper introduces the technique to relight static 3D A comparison and evaluation of multi-view stereo
object using multi-view-multi-lighting images. Our work reconstruction algorithms, in CVPR 2006.
combines existing techniques to realize free-viewpoint  G. Vogiatzis, C. Hernández, P. H. S. Torr, and R. Cipolla,
relighting of 3D object. We demonstrate that by combining Multi-view Stereo via Volumetric Graph-cuts and Occlusion
image based relighting algorithm on 3D model, realistic and Robust Photo-Consistency, IEEE PAMI, Vol. 29, No. 12,
robust relighting results can be obtained. Moreover, we
 Yebin. Liu, Qionghai.Dai, Wenli.Xu, “A Point Cloud based
demonstrate the real time capability of interactive viewpoint Multi-view Stereo Algorithm for Free-viewpoint Video”,
control and lighting control on the reconstructed 3D object. accepted in IEEE Trans on Visualization and Computer
See our public website  for our demonstrations of this Graphics, 2009.
work.  http://media.au.tsinghua.edu.cn/ fvv.jsp
Our 3D reconstruction and relighting technique can be  http://media.au.tsinghua.edu.cn/dome.jsp
further improved for relighting of 3D motion objects by  Y.Y.Schechner, S.K.Nayar, and P.N.Belhumeur,
upgrading the cameras to high speed cameras. Actually, Multiplexing for Optimal Lighting, in IEEE Transactions on
optical flow tracking algorithm can be adopted to register pattern analysis and machine intelligence, Vol.29, no.8, 2007.
 Y.Furukawa and J.Ponce. Accurate, dense, and robust
high speed lighting information to every low speed key
multiview stereopsis. In CVPR, 2007
frames similar to the video relighting system , which  Katz, S., A. Tal, and R. Basri, Direct Visibility of Point Sets,
will be our future works. in Siggraph. 2008.
 P.Einarsson, C.F.Chabert, A.Jones, W.C.Ma, B.Lamond,
6. REFERENCES T.Hawkins, M.Bolas, S.Sylwan, P.Debevec, Relighting
Human Locomotion with Flowed Reflectance Fields, 2006
Eurographics Symposium on Rendering.