Real-tome Combined 2D+3D Active Appearance Models Jing Xiao by yaofenji


									Real-time Combined 2D+3D
Active Appearance Models
Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade
CVPR 2004

              Presented by Pat Chan

   Introduction
   Active Appearance Models AAMs
   3D Morphable Models 3DMMs
   Representational Power of AAM
   Combined 2D+3D AAMs
   Conclusion
   Active Appearance Models are generative
    models commonly used to model faces
   Another closely related type of face models
    are 3D Morphable Models
   In this paper, it tries to model 3D
    phenomena by using the 2D AAM
   Constrain the AAM with the 3D models to
    achieve a real-time algorithm for fitting the
Active Appearance Models

   2D linear shape is defined by 2D triangulated mesh and in
    particular the vertex locations of the mesh.

   Shape s can be expressed as a base shape s0.

   pi are the shape parameter.
   s0 is the mean shape and the matrices si are the eigenvectors
    corresponding to the m largest eigenvalues
Active Appearance Models

   The appearance of an independent AAM is defined within the
    base mesh s0. A(u) defined over the pixels u ∈ s0

                  A0(u)   A1(u)   A2(u)    A3(u)
   A(u) can be expressed as a base appearance A0(u) plus a
    linear combination of l appearance

   Coefficients λi are the appearance parameters.
Active Appearance Models

   The AAM model instance with shape parameters p and
    appearance parameters λ is then created by warping the
    appearance A from the base mesh s0 to the model shape s.

                                             Piecewise affine
                                             warp W(u; p):
                                             (1) for any pixel u in
                                             s0 find out which
                      M(W(u;p))              triangle it lies in,
                                             (2) warp u with the
                                             affine warp for that
Fitting AAMs
   Minimize the error between I (u) and M(W(u; p)) = A(u).
   If u is a pixel in s0, then the corresponding pixel in the input
    image I is W(u; p).
   At pixel u the AAM has the appearance

   At pixel W(u; p), the input image has the intensity I (W(u; p)).
   Minimize the sum of squares of the difference between these
    two quantities:

                       u             u        u
DEMO Video – 2D AAMs
DEMO Video – 2D AAMs
3D Morphable Models (3DMMS)
   3D shape of 3DMM is defined by 3D triangulated
    mesh and in particular the vertex location of the

   The ŝ can be expressed as a based shape ŝ 0 plus
    a linear combination of m shape matrices ŝ i:
3D Morphable Models (3DMMS)

   The appearance of a 3DMM is defined
    within a 2D triangulated mesh that has the
    same topology as the base mesh ŝ0.
   The appearance Â(u) can be expressed as
    a based appearance Â0(u) plus a linear
    combination of l appearance images Âi(u).
3D Morphable Models (3DMMS)
   To generate a 3DMM model instance, an image
    formation model is used to convert the 3D shape ŝ
    in to 2D mesh.
   The result of the imaging 3D point x = (x, y, z)T is:

   i, j are the projection axes, o is the offset of the
   Given shape parameters pi  compute 3D shape 
    map to 2D  compute appearance  warp onto 2D
    mesh (defined by mapping from 2D vertices in ŝ0 to
    2D vertices for 3D ŝ.)
Representational Power of
Can 2D shape models represent 3D?
  The 2D shape variation of the 3D model:
  The projection matrix can be expressed as:

   Therefore 3D model can be represented as combination of:

   The variation of the 3D model can therefore be represented by
    an appropriate set of 2D shape vectors, such as:

6*(m^+1) 2D shape vectors needed to represent a 3D model
Representational Power of
   Experiments
       Use 3D-cube : ŝ = ŝ0 + p1ŝ1
       Generate 60 sets p1 and P randomly
       Synthesize 2D shapes of 60 3D model instances
       Compute 2D shape model by performing PCA on 60 2D
       Result: 12 shapes vectors for each 2D shape mode
       Confirm: 6*(m^+1) 2D vector is required
   However, 2D models generate invalid cases.
   Constrains is need to add on the model
Combined 2D + 3D AAMs
   At time t, we have
   2D AAM shape vector in all N images into a matrix:

   Represent as a 3D linear shape modes W = MB =
Compute the 3D Model
   Perform singular value decomposition (SVD) on W and
    factorize it into:

   The scaled projection matrix M and the shape vector matrix B
    are given by:

   G is the corrective matrix.
   Additional rotational and basis constrain to compute
    G  M and B can be determined
   Thus, the 3D shape modes can be computed from the 2D
    AMM shape modes and the 2D AAM tracking results.
Calculate the Corrective Matrix
   Rotational constraints and basis constraints are used.
   Rotational constraints (denote GGT by Q):

   where ˜M2*i−1:2*i represents the ith two-row of ˜M
   c is the coefficient and R is rotation matrices
   Due to orthogonormality of rotation matrices and Q is
Calculate the Corrective Matrix
   Basis constraints:
   We find K frames including independent shapes and treat
    those shapes as a set of bases, the bases are determined
    uniquely, we have
Compute the 3D Model

                       AAM shapes

                     AAM appearance

                 First three 3D shapes modes
Constraining an AAM with 3D
   Constraints on the 2D AAM shape parameters p = (p1, … , pm) that
    force the AAM to only move in a way that is consistent with the 3D
    shape modes:

   and the 2D shape variation of the 3D shape modes over all imaging
    condition is:

   Legitimate values of P and p such that the 2D projected 3D shape
    equals the 2D shape of AAM. The constraint is written as:
Fitting with 3D Shape
   AAM fitting is to minimize:

   I.e the error between the appearance and the original image
   Impose the constrains of 2D projected 3D shape equals the 2D
    shape of AAM as soft constrains on the above equation with a large
Fitting with 3D Shape
   Optimize for the AAM shape p, q, and the appearance λ parameters:

   Calculate the square difference between the appearance and the
    original image and project the difference into orthogonal complement
    of the linear subspace spanned by the vectors A1, …, Al.
   It is optimized by using inverse compositional algorithm, I.e.
    iteratively minimizing:

   Then, solve the appearance parameters using the linear closed form
      Experimental Results
                                        Estimated 3D shape
Estimates of the 3D
Pose extracted from
the current estimate
of the camera matrix P

            2D AAM

            After 30 Iterations   Converged
Experimental Results

   Results of using the algorithm to track a face in 180
   frame video sequence by fitting the model in each frame
DEMO Video -- 2D+3D AAMs
2D+3D AAM Model

           Input Image        Tracked result (2D+3D fit result)

 2D+3D model reconstruction   Shows two new view reconstruction
Compare the fitting speed with

   Frames per second of 2D+3D > 2D
   Iteration per second of 2D > 2D+3D,
    but 2D need more iteration for
   2D AAMs can represent any phenomena that
    3DMMs can.
   Showed how to compute the equivalent 3D shape
    models from a 2D AAM with basis constrains,
    rotational constrains.
   Improve the fitting speed of the 2D AAMs with 3D
    shapes constrains
   2D + 3D AAM is the ability to render the 3D model
    from novel viewpoint.

To top