Image-Based Implicit Object Modeling Direct and Indirect Multi

Document Sample
Image-Based Implicit Object Modeling Direct and Indirect Multi Powered By Docstoc
					                                               i c h i m 01
C U L T U R A L   H E R I T A G E   a n d   T E C H N OnLt O G IRE SI
                                                       i   h e T H I D M                L L E N N I U M

   Image-Based Implicit Object Modeling:
     Direct and Indirect Multi-Resolution
                     Augusto Sarti            '" and Stefano Tubaro '*'
   (*)   Politecnico di Milano, Dipartimento di Elettronica e Informazione, Milano, Italy
                              E-mail: sarti/

ABSTRACT                                                  appropriate volumetric function.
In this paper we discuss two image-
based 3D modeling methods based on a                      As we may expect, an atlas-based 3D
multi-resolution     evolution     of    a                modeling method deals with topological
volumetric function's levelset. In the                    complexity with a "divide-and-conquer"
former the role of the levelset implosion                 strateg! , which simplifies the local
is to fuse ("sew'' and "stitch") together                 shape estimation process. The price to
several partial reconstructions (depth                    pay for this simplification, however, is
maps) into a closed model. In the latter                  in the complexity of the steps that are
the levelset's implosion is steered                       necessary      to    fuse    the    local
directly by the texture mismatch                          reconstruction into a global closed one
between views. Both solutions share the                   (registration, fusion and hole-mending).
characteristic of operating in an adaptive                An implicit surface representation, on
multi-resolution fashion, in order to                     the other hand, tends to be quite
boost up computational efficiency and                     insensitive to topological complexity, as
robustness.                                               it may accommodate self-occluding
                                                          surfaces, concavities, surfaces of
KEYWORDS: 3D modeling, volumetric                         volumes with holes (e.g. doughnuts,
reconstruction.                                           objects with handles, etc.), or even
                                                          multiple objects.       However, their
INTRODUCTION                                              volumetric nature requires a more
A 3D manifold can be generally defined                    redundant data structure.
and represented either explicitly as an
atlas    (juxtaposition     of    partially                   In this paper we discuss two image-
overlapping local charts), or implicitly as                   based modeling methods that exploit the
the set of points that satisfy a nonlinear                    key features of a levelset-based
constraint in the 3D space (level set of a                    approach to deal with complex
volumetric function). Similarly, image-                       topological structures. The former
based modeling of 3D objects can be                           ("indirect modeling") tries to overcome
envisioned as based on either one of the                      intrinsic topological difficulties related
above two representations. In the former                      to an "atlas-based" approach using a
case, a global object model is obtained                       volumetric approach to surface fusion.
as a complex "patchworking" of simple                         The latter ("direct method") skips the
 local reconstructions (typically depth                       partial modeling step and uses the
 maps), while in the latter the object                        images to steer the implosion of the
 surface is described as a level set of an                    levelset in such a way to obtain the
                                                            i c h i m 01
           ~          ~                                          ~   ----   -~
                                                                            ~-                     . -~
C i J l T L l R A L       I i E R l T A C E   a n d   T E C H N O L O G I E S    in   t h e   T H I R D M I L L E N N I U M

object model in a robust and fast way.                                            the surface

IMPLICIT SURFACE MODELING                                                    distance from 3D data               - which
A closed surface y can be expressed in                                            promotes data fitting
implicit form as
                                                                             inertia - which promotes topological
                                                                                  changes (object      splitting or
                                                                                  generation of holes)
where Y\X) is a volumetric function                                          texture agreement - which maximizes
whose absolute value in x is given the                                            the    similarity    between  the
distance d between x and the surface,                                             appearence of the modeled surface
and its sign depends on whether the                                               and its available views
point x is inside or outside the surface.
Adopting the signed distance as a                                            Besides such terms, we are free to define
volumetric function is known to simplify                                     new velocity terms that attribute the
the computation of the surface's                                             surface evolution some desired behavior.
differential properties of orders 1 and 2:
                                                                             INDIRECT MODELING
        the surface normal can be                                            A common way to build a complete 3D
        computed as the gradient V and                v                      object model consists of combining
        is a unit vector;                                                    several simpler surface patches through
                                                                             a 3D "patchworking" process. In order
        the surface curvature can be                                         to do so, we need a preliminary
        computed as a divergence of the                                      registration phase, in which all the
        form V . V v .                                                       available surface patches are correctly
                                                                             positioned and oriented with respect to a
In order to model a surface in implicit                                      common reference frame; and a fusion
form, we can proceed [4] by defining a                                       process, which consists of merging all
temporally evolving volumetric function                                      surface patches together into one or
whose levelset zero "sweeps" the whole                                       more closed surfaces. One rather
volume of interest until it takes on the                                     standard registration strategy is the
desired shape under thc influence of                                         Iterative Closest Point [2] algorithm,
some properly defined "external action".                                     which consists of minimizing the mean
The levelset evolution is defined by the                                      square distance between overlapping
IIamilton-Jacobi PDE, which can be                                           portions of the surface, using an iterative
discretized into the update equation                                          procedure. As for surface fusion, in this
                                                                              Section we propose and test an approach
                                                                              that is able to seamlessly "sew" the
                                                                              surface    overlaps     together,     and
 where the velocity function F(x) is                                          reasonably "mend" all the holes that
 bound to be orthogonal to the levelset                                       remain after surface assembly (usually
 zero and can be quite arbitrarily defined                                    corresponding to non-visible surface
 in order to steer the front propagation                                      portions).
 toward a desired shape. Terms that may
 appear into its expression are:                                              This "atlas" approach to 3D modeling is
                                                                              suitable for 2 ~ ' / *modeling solutions
 local curvature - which promotes a                                           such as image-based depth estimation
      maximally smooth implosion of                                           techniques, range camcras, and laser-
                                                    i c h i m 01
  .     .~                        ~-          ~                    ~                                 ~

C U I . l U R A I .   H E R I T A G E a n d   T E C H N O L O G I E S    i n   t h e   T H I R D   M I I . I ~ E N N I U M

scanners. The depth maps produced by                             surface; d is the distance (with sign)
such devices could be made of a several                          between the propagating front and the
non-connected surface patches, as                                surface patch; and a is a parameter that
occlusions and self-occlusions tend to                           balances     local   smoothness     and
generate depth discontinuities [I].
                                                                 proximity to the data. Indeed, this
Such surfaces usually need a lengthy                             formulation assumes that only one
assembly process in order to become a                            surface patch is facing the propagating
complete and closed surface.                                     front.

Surface Fusion
As anticipated in the previous Section,
our fusion process is based on the
temporal evolution of the zero level set a
volumetric function [3,4]. The velocity
function that steers the front evolution
accounts for two contrasting needs: that
of following the motion by curvature
and that of honoring the data (registered
surface patches).

A surface is said to follow the motion by
curvature when the velocity field that                                 Figure 1: Motion by curvature: the
describes the surface deformation is                                    surface deflates in a maximally
normal to the surface itself and its                                   smooth fashion until it disappears.
magnitude is proportional to the local
curvature (with sign). Indeed, if the                              Notice that the above definition of
motion were purely by curvature, a                                 velocity holds valid only for the points
surface would tend to deflate completely                           that lie on the propagation front,
and     disappear,     while    becoming                           therefore we need to extend its validity
progressively smoother and smoother                                in the whole volume (or at least in the
(Fig. 1). The need to honor the available                          sorrounding points of the surface). The
range data prevents this complete                                  extension of this f h c t i o n needs to be
implosion from taking place.                                       done consistently with the front
                                                                   propatation, meaning that the levelset
 in order to implement this implosion-                              should evolve with no self-collisions.
 inhibition mechanism, we need to                                  This can be done quite easily [3] as
 redefine the velocity field associated to                          follows: given a generic point x not
 the update equation that describes the                             lying on the surface y, we can search for
 zero level-set propagation. This velocity                          the point y on y that lies the closest to x
 is bound to be orthogonal to the                                   and let F(x)= F(y).
 propagating front, and its amplitude is
 set to                                                            Given a point on the propagating front,
                                                                   the distance from a surface patch is
                                                                   computed       from     the    orthogonal
 F ( x ) = F,(K (x))        +a u F2 (d(x))                         projection of that point onto the surface
                          ~ ( x )                                  patch itself (Fig. 2). If no point on the
 where K is the levelset's local curvature,                        surface patch faces the point on the
 KM is thc local curvature of the facing                           level-set orthogonally, then the distance
                                             ichirn 0 1
C U L T U R A L   AGE        a   n   d   T   n   H   N   0   L   O   ~   tmh e   T H I R D   M l L L E N N m

function d is computed from the closest                      cost. In fact, in order to obtain high
point on the border of the patch (within a                   performance at low computational cost,
pre-assigned range).                                         besides updating the volumetric function
                                                             just in a narrow region around the zero
When more than one surface patches are                        level-set (narrow-band implementation),
facing the propagation front, then the                       we operate in a multiresolution fashion
distance function d is computed using                         (see Fig. 3). This can be achieved by
the distances from the point on the                           starting with a low-resolution voxset
propagating front and all the orthogonal                      (e.g. a voxset with 10 voxels per side)
projections onto the surface patches that                     and letting the front settle down. Then
face it; the surface orientation; the                         we break down the voxels around the
closeness to the border of the patch (the                     propagating front and resume the front
reliability tends to decrease in the                          propagation. The operation continues
proximity of exhemal boundaries); and                         until the final resolution is reached. A
the mutual occlusion between surface                          key aspect of this process is in the fact
patches.                                                      that the velocity field that drives the
                                                               implosion of the level set can be pre-
                                                               computed on the octree data structure
                                                               that best fits the available range data.

                                                             The resulting model is bound to be a set
                                                             of closed surfaces, therefore all the
                                                             modeling "holes" left after mosaicing
                                                             the partial reconstructions are closed in a
                                                             topologically sound fashion. In fact,
                                                             those surface portions that cannot be
                                                             reconstructed because they are not
                                                             visible, can sometimes be "patched up"
                                                             by the fusion process. This ability to
                                                             "mend" the holes can also be exploited
                                                              in order to simplify the 3D acquisition
                                                              session, as it allows us to skip the
     Figure 2: Given a point on the                          retrieval of some depth maps.
 propagating front, its distance from a
                                                              An interesting aspect of our fusion
  surface patch is computed from the
                                                              method is in the possibility to modify
 orthogonal projection onto the target
                                                              the surface characteristics through a
             triangle mesh.
                                                              processing of the volumetric function.
 Key Features
 This approach to surface fusion exhibits                     For example, filtering the volumetric
 a number of desirable characteristics.                       function results in a smoother surface
 One of its most appealing features is the                    model Finally, the method exhibits a
 fact that it is very robust against                          certain robustness against orientation
 topological complexity. In fact, a level-                    errors, as the non perfect matching of
 set of a volumetric function is adequate                     surface borders can be taken care of by
 for describing multiple objects of rather                    the fusion process through a careful
 arbitrary topology. In addition, it                          definition of the distance function used
 involves a fairly modest computational                       in the specification of the volumetric
                                           i c h i m 01
C U L T U R A L   H   E   ~   G   a En d                        t h e   T H I R D   M I L L E N N I U M

motion field.

An Example of Application
In order to test the effectiveness of the
proposed technique, we applied it to a
variety of study cases. A particularly
interesting experiment was conducted on
an object with a particular topology (a
bottle with a handle) that could easily
create problems of ambiguities. Any
traditional surface fusion approach
would, in fact, encounter difficulties in
deciding how to complete the surface in
 the missing regions. Furthermore,
 besides      exhibiting      self-occlusion
 problems, this object puts the multi-
 resolution approach under a severe test.
 We acquired six depth maps and
 assembled them together using an ICP
 algorithm (see Fig 4. The result was an
 incomplete model with some accuracy
 problems in the overlapping regions (at
 the boundaries of the depth maps). The
  front evolution is shown in Fig. 5, which
 results in the (topologically correct) final
  model of Fig. 6.
                                                          Figure 3: Multiresolution progression
                                                           of the voxset where the volumetric
                                                                   function is defined.
                                                           i c hm 0 1
.-- .          ---   -                     --   --   .          .-.
                                                                 -             --       --
C U L T U R A L .        H E R I T A G E    a n d    T E C H N O L O G I E S   i n    t h c   T H I R D   M I L L E N N I U M

                                                                                     Figure 6: Final 3D model.

                                                                         DIRECT MODELING
   Figure 4: One of the original views                                   In alternative to using 3D data for the
    (top left). Six unregistered surface                                 generation of the complete closed object
   patches obtained with stereometric                                    surface, we can use directly the available
  techniques (top right). Two views of                                   images. An image-based 3D modeling
  the assembled surface patches after                                    method       that   uses    an    implicit
      registration (bottom): notice the                                  representation of surfaces was recently
   creases due to a non-perfect model                                    proposed by Faugeras and Keriven [S].
     overlapping, and the presence of                                    This modeling approach is based on the
         holes in the global model.                                      temporal evolution of a volumetric
                                                                         hnction whose zero level-set is a closed
                                                                         surface that reuresents the surface model
                                                                         as it tends to approximate the imaged
                                                                         object. This surface, which initially
                                                                         contains the imaged object, evolves by
                                                                          following a motion that is always locally
                                                                         normal to the surface, with a speed that
                                                                         depends on the local surface curvature
                                                                          and to a measurement of the local
                                                                         "texture mismatch" between imaged and
                                                                          "transferred" textures. Transferring an
                                                                          imaged texture onto another view means
                                                                          back-projecting it onto the model and re-
                                                                          projecting it onto the other view. In
                                                                          order to kccp the computational
                                                                          complexity at a manageable level, the
        Figure 5: Level set implosion.                                    updating of the volumetric function is
                                                                          only performed within a "narrow band"
                                                                          [3] around the current surface. Our
                                                                          solution,       however,     significantly
generalizes this approach, as it operates       pixel rn, in the i-th image. This
in an adaptive multi-resolution fashion,        definition of d o guarantees that the
which boosts up the computational               surface     representation      will    be
efficiency. Multi-resolution, In fact,          independent of the variables (u,w). The
enables us to quickly obtain a rough            surface patch S through which the
approximation of the objects in the             luminance transfer occurs is assumed to
scene at the lowest possible voxset             be a locally planar approximation of the
resolution.     Successive     resolution       propagating front. Indced, in order to
increments allow us to progressively            guarantee that this approximation will
refine the model and add details. In            niantain a constant quality, the size of
order to do so, we introduce "inertia" in       this planar patch will change according
 the level-set evolution, which tends to        to the local curvature of the levelset.
 favor topological changes (e.g. the
 creation of doughnut-like holcs in the         The inner product (correlation) between
 structure).                                    the pair of subimages I; and I, is defined
                                                as follows:
Finally, through a careful control of the
components that steer the level-set
evolution        (hysteresis,       biased
quantization, etc.), we are able to
recuperate details that were lost at lower
resolution levels (surface creases. ridges,
                                                where m , e m 2 are homologous
Definition of the Velocity Function             image points ( i t . image points that
One of the terms that contribute to             correspond to the same point of the
steering the level-set evolution is the         surface model), and
"texture mismatch" betwecn imaged and
"transferred" textures [I], which is a
function of the correlation between
homologous luminance profiles [ 5 ] . The
texture mismatch is

                                                 Although the correlation could be
                                                 computed between all the viewpo~nts
                                                 where there is visibility, only the pair of
                                                 views with the best visibility is
                                                 considered. Visibility can be easily
                                                 checked through a ray-tracing algorithm
                                                 and measured as a function of the angle
                                                 between visual ray and surface normal.
                                                 Notice that normalizing the correlation
                                                 has a twofold purpose: to limit its range
 where      do= 13, n 3 ,,,I LI v (4 w is the    between 0 and 2; and to guarantee that
 infinitesimal area element of the surface       low-energy areas (smooth texture) will
 S, associated to the local surface              have the same range of high-encrgy
 parametrization ( v , w ) induced by the        (rough     tcxture)      areas.    Finally,
 ilnage coordinate chart; n is the surface       subtracting the average from a
 normal; and I,(rni) is the luminance of
                                                      ichim 01
    C U L T U R A L   H E R I T A G E   a n d   T E C H N O L O G I E S   i n   t h e   T H I R D   M I L L E N N I U M

luminance profile tends to compensate a                            Multiresolution
non-lambertian behavior of the imaged                              If the volumetric function that
surfaces.                                                          charactcrizcs the level-set is defined on a
                                                                   static voxset of N X N X N voxels, the
The velocity fimction associated to the                            computational complexity of each front
front propagation is here defined with                             propagation step is proportional to N L    ,
the twofold need of guaranteeing surface                           as it is proportional to the surface of the
smoothess and consistency between                                  level-set (narrow-band computation).
images and final model                                             Furthermore, since the velocity F is
                                                                   multiplied by I v yl (which is equal to
                                                                   the sampling step), the number of
                                                                   iterations turns out to be proportional to
                                                                   N, with a resulting algorithimc
                                                                   complexity that is proportional to N'.
    The first term C (X)V V represents  v
    the texture-curvature action and favors a                      In order to dramatically reduce this
    maximally smooth implosion toward a                            complexity, we developed a multi-
    shape that agrees with the available                           resolution     approach    to   level-set
    textures. The presence of the texture                          evolution. The algorithm starts with a
    mismatch cost C, in fact, tends to slow                        very low resolution level (a voxset of
    down areas with modest cost, and speed                         10-15 voxel per side). When the
    up areas of high cost. Ideally, one would                      propagation      front converges, the
    be lead to think that the first term is                        resolution increases and the front
    sufficient to correctly steer the model's                      resumes its propagation. The process is
    evolution, as correct surface regions                          iterated until we reach the desired
    should have a zero cost, while other                           resolution. A progressive resolution
     regions are left free to evolve. This,                        increment has the desired result of
     however, is not really true as the cost is                    minimizing the amount of changes that
     rarely equal to zero due to a non-perfect                     each propagation step will introduce in
     luminance transferal and a non-                               the model, with the result of achieving a
     lambertian radiometric behavior. This                         better global minimum of the cost
     causes the front propagation to                               fimction. Furthermore, the number of
     "trespass" the correct surface. The                            iterations will be dramatically reduced
     second term of eq. (2) will tend to                           (from N to log N ) with respect to a
     contrast this behavior. In fact, in the                        fixed-resolution approach, with an
     proximity of the actual surface, the local                     algorithmic complexity that turns out to
     cost gradient V@ is almost parallel                            be proportional to N210g N.
     (although oppositely oriented) to the
     propagation front's normal n= V l,V As a                       Indeed, starting from a low-resolution
     consequence, V (Y . V V/ < 0 tends to                          voxset, we need to prevent the algorithm
     discourage the front from propagating                          from losing details at that resolution or
     beyond the actual surface. Finally, the                        to make sure that the algorithm will be
     third element of eq. (2) acts an "inertial"                    able to recover the lost details. In fact,
     term in order to favor concavities in the                      one has to keep in mind that the motion
      final model, provided that a proper                           by curvature tends to dominate over the
     dynamic adaptation of k is performed.                          other terms, therefore some of the details
                                                                    of the object may totally disappear. In
                                                                    order to prevent this from happening, we
                                                  ichim 01
C U L T U R A L   H E R I T A G E   a n d   T E C H N O L O TI E S   ~n   t h e   T H I R D   M I L L E N N I U M

use a method based on thresholding the                        Examples of application
local curvature with a hyperbolic tangent                     We tested our approach on several
function. This guarantees a smoother                          subjects acquired with a camera moving
behavior than a simpler clipping                              around them. The method proved to be
function.                                                     remarkably robust against topological
                                                              complexity and lack of segmentation
In spite of this smooth thresholding
mechanism, in some cases it is not
possible to prevent some of the smaller
details from disappearing. For this
reason, we developed a technique that
enables the recovery of lost details
before the resolution is increased, which
is based on a mechanism of hysteresis in
the surface implosion. The idea is to
keep track of all the voxels on the zero
                                                                Figure 8: Sequence of original views.
 level-set whose cost 0 is below a certain
 threshold. After the "implosion" of the
 level-set, we let the propagation front
 evolve while driven by a different cost
 function that depends on the distance
 between the surface and such points.
 This operation makes the surface
 litterally "climb up" the lost details. As
 an example of applications, see Fig. 7.

  Figure 7: Illustration of the temporal
 evolution of the cost function (texture                         Figure 9: A view of the cost function
  mismatch) and of the model. Notice                             mapped onto the propagation front.
      that the cost value suddenly                              The darker the texture, the heavier the
 increases at every resolution change,                                         mismatch.
  due to the mechanism of recovery of
Figure 10: Temporal evolution of the
     propagation front. The initial
volumetric resolution is very modest
    (in this case the voxset size is
2Ox20x20), and is not able to account
   for some topologically complex
details of the surface (the fifth frame
in lexicographic order is the best one
                                          Figure 12: Two of the original views of
   can do at this resolution). As the
                                                       the subject.
  resolution increases, more details
 begin to appear, such as the stem of
               the apple.

                                           Figure 13: Temporal evolution of the
                                                    propagation front.

 Figure 11: Four views of the final 3D
                                                    i c h i r n 01
           ~      ~                                   -~                                    ~

C U L T U R A L   I H E R I T A G E   a n d   T E C H N O L O G I E S    ~n   t h e   T H I R D   M I L L E N N I U M

                                                                 2.     P.J. Besl, N.D. McKay, "A method
                                                                        for registration of 3-D shapes IEEE
                                                                        Tr. on PAMI. Vol. 14, No. 2, 1992,
                                                                        pp. 239-256.
                                                                 3.     R. Malladi, J.A. Sethian, B.C.
                                                                        Vemuri, "Shape Modeling with
                                                                        Front Propagation: A Level Set
                                                                        Approach". IEEE Tr. on PAMI,
                                                                        Feb. 1995, Vol. 17, No. 2, pp. 158-
                                                                 4.     S. Osher, J.A. Sethian, "Fronts
                                                                        Propagating      with     Curvature
                                                                        Dependent      Speed:   Algorithms
                                                                        Based       on      Hamilton-Jacobi
                                                                        Formulations". J. Comput. Phys.
                                                                        Vol. 79, pp. 12-49, 1988.
                                                                 5.      0.     Faugeras,     R.    Keriven.
                                                                         "Variational Principles, Surface
                                                                         Evolution,    PDE's,    level   set
                                                                         Methods, and the Stereo Problem".
       Figure 14: Final 3D model.                                        IEEE Tr. on Image Processing, Vol.
                                                                         7, No. 3, March 1998.
In this paper we discuss two image-                              6.      P. Pigazzini, F. Pedersini, A. Sarti,
based 3D modeling methods based on a                                     S. Tubaro: "3D Area Matching with
multi-resolution    evolution     of     a                               Arbitrary Multiview Geometry".
volumetric function's levelset. The                                      Signal       Processing:      Image
former consists of fusing ("sewing" and                                  Communications. Vol. 14, Nos. 1-2,
"stitching")      numerous         partial                               1998, pp. 7 1-94.
reconstructions (depth maps) into a
closed model, while the latter consists of                        ABOUT THE AUTHORS
steering the levelset's implosion with                            Augusto Sarti was born in Rovigo,
texture mismatch between views. Both                                 Italy, in 1963. He received a "laurea"
solutions share the characteristic of                                degree (Summa cum Laude) and a Ph.D.
operating in an adaptive multi-resolution                            from the University of Padova, Italy. He
fashion, which boosts up computational                               spent two years at the University of
efficiency and robustness.                                           California at Berkeley doing research on
                                                                     nonlinear system theory. Prof. Sarti is
                                                                     currently a faculty member of the
1.   F. Pedersini, A. Sarti, S. Tubaro:                              Politecnico di Milano, Milan, Italy, and
     "Visible Surface Reconstruction
                                                                     his research interests are mainly in
     with Accurate Localization of                                   digital signal processing and 3D
     Object Boundaries", IEEE Tr. on
     Circuits and Systems for Video
     Technology, Vol. 10, No. 2, March
                                                      ichirn 01
~                       ~                                ~
                                                                                                      -               .
    C U L T U R A L   H E R I T A G E   a n d   T E C H N O L O G I E S   i n   t h e   T H I R D   M I L L E N N I U M

    Stefano Tubaro was born in Novara,                             he has been an Associate Professor of
    Italy, in 1957. He completed his studies                       Electrical Communications at the
    in Electrical Engineering in 1982. In                          Politecnico di Milano. His current
    1986 he joined the Study Center for                            research interests are mainly on signaI
    Space Telecommunications of the Italian                        processing and computer vision.
    National Research Council. Since 1991                          E-mail: