# Real-tome Combined 2D+3D Active Appearance Models Jing Xiao by yaofenji

VIEWS: 7 PAGES: 29

• pg 1
```									Real-time Combined 2D+3D
Active Appearance Models
Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade
CVPR 2004

Presented by Pat Chan
23/11/2004
Outline

   Introduction
   Active Appearance Models AAMs
   3D Morphable Models 3DMMs
   Representational Power of AAM
   Combined 2D+3D AAMs
   Conclusion
Introduction
   Active Appearance Models are generative
models commonly used to model faces
   Another closely related type of face models
are 3D Morphable Models
   In this paper, it tries to model 3D
phenomena by using the 2D AAM
   Constrain the AAM with the 3D models to
achieve a real-time algorithm for fitting the
AAM
Active Appearance Models
(AAMs)

   2D linear shape is defined by 2D triangulated mesh and in
particular the vertex locations of the mesh.

   Shape s can be expressed as a base shape s0.

   pi are the shape parameter.
   s0 is the mean shape and the matrices si are the eigenvectors
corresponding to the m largest eigenvalues
Active Appearance Models
(AAMs)

   The appearance of an independent AAM is defined within the
base mesh s0. A(u) defined over the pixels u ∈ s0

A0(u)   A1(u)   A2(u)    A3(u)
   A(u) can be expressed as a base appearance A0(u) plus a
linear combination of l appearance

   Coefficients λi are the appearance parameters.
Active Appearance Models
(AAMs)

   The AAM model instance with shape parameters p and
appearance parameters λ is then created by warping the
appearance A from the base mesh s0 to the model shape s.

Piecewise affine
warp W(u; p):
(1) for any pixel u in
s0 find out which
M(W(u;p))              triangle it lies in,
(2) warp u with the
affine warp for that
triangle.
Fitting AAMs
   Minimize the error between I (u) and M(W(u; p)) = A(u).
   If u is a pixel in s0, then the corresponding pixel in the input
image I is W(u; p).
   At pixel u the AAM has the appearance

   At pixel W(u; p), the input image has the intensity I (W(u; p)).
   Minimize the sum of squares of the difference between these
two quantities:

u             u        u
u
DEMO Video – 2D AAMs
DEMO Video – 2D AAMs
3D Morphable Models (3DMMS)
   3D shape of 3DMM is defined by 3D triangulated
mesh and in particular the vertex location of the
mesh.

   The ŝ can be expressed as a based shape ŝ 0 plus
a linear combination of m shape matrices ŝ i:
3D Morphable Models (3DMMS)

   The appearance of a 3DMM is defined
within a 2D triangulated mesh that has the
same topology as the base mesh ŝ0.
   The appearance Â(u) can be expressed as
a based appearance Â0(u) plus a linear
combination of l appearance images Âi(u).
3D Morphable Models (3DMMS)
   To generate a 3DMM model instance, an image
formation model is used to convert the 3D shape ŝ
in to 2D mesh.
   The result of the imaging 3D point x = (x, y, z)T is:

   i, j are the projection axes, o is the offset of the
origin
   Given shape parameters pi  compute 3D shape 
map to 2D  compute appearance  warp onto 2D
mesh (defined by mapping from 2D vertices in ŝ0 to
2D vertices for 3D ŝ.)
Representational Power of
AAM
Can 2D shape models represent 3D?
  The 2D shape variation of the 3D model:
  The projection matrix can be expressed as:

   Therefore 3D model can be represented as combination of:

   The variation of the 3D model can therefore be represented by
an appropriate set of 2D shape vectors, such as:

6*(m^+1) 2D shape vectors needed to represent a 3D model
Representational Power of
AAM
   Experiments
   Use 3D-cube : ŝ = ŝ0 + p1ŝ1
   Generate 60 sets p1 and P randomly
   Synthesize 2D shapes of 60 3D model instances
   Compute 2D shape model by performing PCA on 60 2D
shapes
   Result: 12 shapes vectors for each 2D shape mode
   Confirm: 6*(m^+1) 2D vector is required
   However, 2D models generate invalid cases.
   Constrains is need to add on the model
Combined 2D + 3D AAMs
   At time t, we have
   2D AAM shape vector in all N images into a matrix:

   Represent as a 3D linear shape modes W = MB =
Compute the 3D Model
   Perform singular value decomposition (SVD) on W and
factorize it into:

   The scaled projection matrix M and the shape vector matrix B
are given by:

   G is the corrective matrix.
   Additional rotational and basis constrain to compute
G  M and B can be determined
   Thus, the 3D shape modes can be computed from the 2D
AMM shape modes and the 2D AAM tracking results.
Calculate the Corrective Matrix
   Rotational constraints and basis constraints are used.
   Rotational constraints (denote GGT by Q):

   where ˜M2*i−1:2*i represents the ith two-row of ˜M
   c is the coefficient and R is rotation matrices
   Due to orthogonormality of rotation matrices and Q is
symmetric,
Calculate the Corrective Matrix
   Basis constraints:
   We find K frames including independent shapes and treat
those shapes as a set of bases, the bases are determined
uniquely, we have
Compute the 3D Model

AAM shapes

AAM appearance

First three 3D shapes modes
Constraining an AAM with 3D
Shape
   Constraints on the 2D AAM shape parameters p = (p1, … , pm) that
force the AAM to only move in a way that is consistent with the 3D
shape modes:

   and the 2D shape variation of the 3D shape modes over all imaging
condition is:

   Legitimate values of P and p such that the 2D projected 3D shape
equals the 2D shape of AAM. The constraint is written as:
Fitting with 3D Shape
Constraints
   AAM fitting is to minimize:

   I.e the error between the appearance and the original image
   Impose the constrains of 2D projected 3D shape equals the 2D
shape of AAM as soft constrains on the above equation with a large
K:
Fitting with 3D Shape
Constraints
   Optimize for the AAM shape p, q, and the appearance λ parameters:

   Calculate the square difference between the appearance and the
original image and project the difference into orthogonal complement
of the linear subspace spanned by the vectors A1, …, Al.
   It is optimized by using inverse compositional algorithm, I.e.
iteratively minimizing:

   Then, solve the appearance parameters using the linear closed form
solution:
Experimental Results
Estimated 3D shape
Estimates of the 3D
Pose extracted from
the current estimate
of the camera matrix P
Initialization

2D AAM

After 30 Iterations   Converged
Experimental Results

Results of using the algorithm to track a face in 180
frame video sequence by fitting the model in each frame
DEMO Video -- 2D+3D AAMs
2D+3D AAM Model
Reconstruction

Input Image        Tracked result (2D+3D fit result)

2D+3D model reconstruction   Shows two new view reconstruction
Compare the fitting speed with
2D AAMs

   Frames per second of 2D+3D > 2D
AAM
   Iteration per second of 2D > 2D+3D,
but 2D need more iteration for
convergence
Conclusion
   2D AAMs can represent any phenomena that
3DMMs can.
   Showed how to compute the equivalent 3D shape
models from a 2D AAM with basis constrains,
rotational constrains.
   Improve the fitting speed of the 2D AAMs with 3D
shapes constrains
   2D + 3D AAM is the ability to render the 3D model
from novel viewpoint.
Q&A

```
To top