3D Video for the production of visual effects and

							3D Video for the production of visual
effects and interactive media                                        Production of interactive media
Oliver Grau                                                          A lot of progress has been made in developing methods to
BBC Research & Innovation                                            compute 3D video. Most approaches operate in a studio
oliver.grau@bbc.co.uk                                                environment under controlled conditions. This usually includes
                                                                     controlled lighting and keying facilities, like uni-coloured
This contribution reviews several 3D video techniques that           background for chroma keying.
were investigated by BBC R&I for use in broadcast                    Much progress has been made towards high quality visual
applications. Further it discusses unsolved production issues in     reconstructions of scenes, in particular human beings. However
new applications, like the use of 3D video in interactive media.     the use of these models in interactive applications raises new
                                                                     issues: What kind of interactivity should be provided? If the
3D graphics in the media industry is traditionally used for the      viewer has full control over the (virtual) viewpoint then the
production of visual effects (e.g. animation in post-production      content must provide a high quality level from all sides. Details
for movies) and by the gaming industry. Both have different          like the face of persons should not break up when moving
production approaches that result in very different kinds of         closer.
media: Post-production aims at highest photo-realistic picture       Another important issue is about how interactive content is
quality; the processing is done off-line (rendering alone of a 30    produced. Many approaches demonstrated so far are deploying
sec clip can take hours and days). The gaming industry               only one or a few actors in a (virtual) studio scenario. From a
achieves good visual quality of completely synthetic content at      creative perspective it is very hard to establish interactions
interactive rendering rates. The production cycle is very            between the actor(s) and scene elements that are added as
different to the film or broadcast industry and a production can     virtual objects or actors since they are not visible on set.
last several years.                                                   The lack of visual feedback for actors is also a problem in the
3D video has the potential to allow the production of new             production of special effects and has been addressed by the
visual effects, e.g. free-viewpoint video, relighting, and others.    ORIGAMI project [4]. The key idea is to provide the actor
Further it can also provide new ways to produce interactive           with visual feedback of any (virtual) objects he has to interact
media, that is currently dominated by the gaming industry.            with. This has been achieved by integrating a view-dependent
                                                                      projection system into the 3D capturing system. Fig. 2 shows
3D video for special effects                                          an immersive actor feedback system working in parallel to the
                                                                      real-time modelling system. It is based on data projectors
Special effects are usually making as much as possible use of         driven from rendering PCs that project onto the special retro-
footage that is captured by real cameras, because this is usually     reflective cloth without interference with the chroma-keying
the cheapest option and guarantees 'photo-realistic' results.         facility, since the light from a LED ring mounted on the
Computer generated imagery (CGI) is used in production when           camera lenses dominates over the projector light.
it is impossible or too expensive to capture a scene with a real     The head position of the actor is derived from the captured 3D
camera.                                                              model and used to synthesise view-dependent images of the
3D video can be used in many ways to allow the integration of        virtual scene for each projector.
real and virtual objects. BBC R&I developed a multi-camera           The system is shown during a demo production that was set in
system with 12 cameras in a studio set-up, as depicted in fig. 1.    the entrance hall of the British Natural History Museum in




                                                                     Illustration 2: View-dependent projects gives actor a visual
Illustration 1: Multi-camera studio set-up and application to        queue to interact with virtual objects
insert actor in different environment
                                                                     London. The boy was watching a Pterosaur flying through the
                                                                     entire length of the hall. Four projectors were used to generate
3D video is computed based on the visual hull concept. The           this 180o view of the hall and the animation. The boy was able
use of 3D data was then demonstrated for compositing of              to establish exact eye-line contact with the Pterosaur.
virtual and real scene components, including changing the
lighting conditions [1,2].                                           References
The ability to render the captured content from a different
camera viewpoint is one of the most attractive features of 3D        [1] O. Grau, ,3D in content and post-production", book chapter
video. It allows the production of special effects or camera         in Schreer, Kauff und Sikora, ,3D Communications", Wiley,
moves that are impossible or too expensive to achieve with a         2005.
real camera.                                                         [2] O. Grau, ,Multi-camera Radiometric Surface Modelling for
This concept is very attractive to visualise actions in sport        Image-Based Re-lighting", in Proc. of 28th Annual DAGM
scenes. The recent collaborative 'iview' project, led by the BBC     Symposium, Sept. 12-14 2006, Berlin, Germany.
is developing a free-video system that allows the replay of          [3] O. Grau, A. Hilton, J. Kilner, G. Miller, T. Sargeant, J.
sport scenes, like football or rugby from any camera angle [3].      Starck, 'A free-viewpoint video system for visualisation of
The environment is less controlled than the studio. In particular    sport scenes', Proc. of IBC 2006, 7-11 September 2006,
camera calibration and segmentation are harder to solve.             Amsterdam, NL.
Robust 3D reconstruction techniques are required to overcome         [4] O. Grau, ,A 3D production pipeline for special effects in
this.                                                                TV and film", Proc. of Mirage 2005 conference. March, 1-2
                                                                     2005, INRIA Rocquencourt, France.