Feature-Based Surface Parameterization and Texture Mapping

Document Sample
Feature-Based Surface Parameterization and Texture Mapping Powered By Docstoc
					Feature-Based Surface Parameterization and
Texture Mapping
EUGENE ZHANG, KONSTANTIN MISCHAIKOW, and GREG TURK
Georgia Institute of Technology


Surface parameterization is necessary for many graphics tasks: texture-preserving simplification, remeshing, surface painting,
and precomputation of solid textures. The stretch caused by a given parameterization determines the sampling rate on the
surface. In this article, we present an automatic parameterization method for segmenting a surface into patches that are then
flattened with little stretch.
   Many objects consist of regions of relatively simple shapes, each of which has a natural parameterization. Based on this
observation, we describe a three-stage feature-based patch creation method for manifold surfaces. The first two stages, genus
reduction and feature identification, are performed with the help of distance-based surface functions. In the last stage, we create
one or two patches for each feature region based on a covariance matrix of the feature’s surface points.
   To reduce stretch during patch unfolding, we notice that stretch is a 2 × 2 tensor, which in ideal situations is the identity.
Therefore, we use the Green-Lagrange tensor to measure and to guide the optimization process. Furthermore, we allow the
boundary vertices of a patch to be optimized by adding scaffold triangles. We demonstrate our feature-based patch creation and
patch unfolding methods for several textured models.
   Finally, to evaluate the quality of a given parameterization, we describe an image-based error measure that takes into account
stretch, seams, smoothness, packing efficiency, and surface visibility.
Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling—Geometric
algorithms, languages, and systems
General Terms: Algorithms
Additional Key Words and Phrases: Surface parameterization, segmentation, texture mapping, topology




1.   INTRODUCTION
Surface parameterization is a well-studied problem in computer graphics. In general, surface parame-
terization refers to segmenting a 3D surface into one or more patches and unfolding them onto a plane
without any overlap. Borrowing terminology from mathematics, this is often referred to as creating an
atlas of charts for a given surface. Surface parameterization is necessary for many graphics applica-
tions in which properties of a 3D surface (colors, normal) are sampled and stored in a texture map. The
quality of the parameterization greatly affects the quality of subsequent applications. One of the most


This work is supported by NSF grants ACI-0083836, DMS-0138420, and DMS-0107396.
Authors’ addresses: E. Zhang, School of Electrical Engineering and Computer Science, Oregon State University, 102 Dearborn
Hall, Corvallis, OR 97331-3202; email: Zhange@cc.gatech.edu; K. Mischaikow, Center for Dynamical Systems and Nonlinear
Studies, School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332; email: mischaik@math.gatech.edu; G. Turk,
College of Computing/GVU Center, Georgia Institute of Technology, Atlanta, GA 30332; email:turk@cc.gatech.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0001 $5.00

                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 1–27.
2     •      E. Zhang et al.




Fig. 1. The feature regions (left) and the unfolded patches (right, colors are used to encode surface normal) for the bunny surface
using our algorithm.


important quality measurements is stretch. When unfolding a surface onto a plane, stretching occurs if
the surface contains highly spherical or hyperbolic regions. High stretch in a parameterization results
in an uneven sampling rate across the surface.
   We observe that many objects can be decomposed into a set of “simple” shapes that roughly approx-
imate cylinders, cones, flat disks, and spheres. Cylinders, cones, and planes are developable surfaces,
which are Euclidean by nature. Unfolding them results in little stretch, without any overlap. In this
article, we make use of some distance-based surface functions to divide a manifold surface into feature
regions, each of which is similar to one of the simple shapes. In Figure 1 (left), the bunny surface is
decomposed into four feature regions (ears, head, and body) using our segmentation algorithm. These
regions are converted into patches and unfolded with little stretch (right, colors are used to encode
surface normal).
   Existing patch unfolding techniques are often carried out in two stages: an initial patch layout to
achieve some objective such as conformal mapping, followed by an interior vertex optimization based on
some stretch metric. We observe that an ideal surface parameterization between a patch and its textural
image is an isometry, that is, a bijective map that preserves distances. The Green-Lagrange deformation
tensor has the property that it measures anisotropic stretch faithfully and penalizes undersampling
more severely than oversampling. In addition, it can be seen as a balance between area-preserving
mappings and conformal mappings. We use this metric to guide the vertex optimization process for
patch unfolding. In addition, we use what we call scaffold triangles to convert the original boundary
vertices into “interior” vertices, which can then be freely moved around within the same optimization
framework. This is a new way of creating nonconvex patches that may even have holes.
   In this article, we present an automatic surface parameterization technique that consists of several
new ideas and improves upon existing techniques in the following aspects. For patch creation, instead
of relying on local curvature information for feature detection as in the case of most previous param-
eterization methods, we extract and segment large protrusions based on the topological analysis of
some distance-based surface functions that are global in nature. This results in a small number of
large patches that can be unfolded with relatively little stretch. For patch unfolding, we use the Green-
Lagrange tensor to measure stretch and to guide the stretch optimization process. In addition, we create
a “virtual boundary” to allow the patch boundaries to be optimized, without the need to check for global
self-intersections. Finally, we describe a novel image-based quality metric for surface parameterization
that implicitly takes into account stretch, seams, packing efficiency, smoothness, and surface visibility.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                          Feature-Based Surface Parameterization and Texture Mapping                •     3

   The remainder of the article is organized as follows. In Section 2, we review existing surface pa-
rameterization techniques. Then, we present our feature-based patch creation method in Section 3,
followed by our new balanced stretch metric in Section 4.1, boundary vertex optimization technique
in Section 4.2, and our packing algorithm in Section 5. In Section 6, we show the results of applying
our technique to various 3D models and describe our image-based quality metric for parameterization
techniques. Section 7 provides a summary of our contributions and a discussion of some possible future
work.

2.    PREVIOUS WORK
There has been a considerable amount of recent work in the graphics community on building a surface
parameterization by unfolding a polygonal surface into planar patches. Much of the motivation for this
is for texture mapping, the mapping of pixels from a rectangular domain (the texture map) onto a surface
that is described by a collection of polygons. The surface parameterization problem is to subdivide the
given surface into a (hopefully small) number of patches that are then flattened onto a plane and
arranged in a texture map. Uses for surface parameterization include surface painting [Hanrahan and
Haeberli 1990], fast rendering of procedural textures [Perlin 1985; Turk 2001; Wei and Levoy 2001; Carr
and Hart 2002], applying photographed color variations onto digitized surfaces [Cignoni et al. 1998],
and creating normal maps from detailed geometry [Sander et al. 2001]. These same parameterization
methods may also be used for remeshing, that is, for creating a new mesh from the original surface [Alliez
et al. 2002]. Remeshing can be used to improve the triangle shapes, to vary the triangle size according
to curvature details, and to induce semi-regular tessellations. Recently, octrees have been used to store
colors in 3D for surface texturing without any parameterization [Benson and Davis 2002; DeBry et al.
2002]. Although octree techniques are supported with programmable GPU’s, they are not yet directly
supported by graphics hardware.

2.1   Patch Creation
There are two common approaches to the patch creation problem. The first of these is to find a single
cut for the surface that makes the modified surface topologically equivalent to a disk [Piponi and
Borshukov 2000; Gu et al. 2002; Sheffer and Hart 2002; Erickson and Har-Peled 2002; Ni et al. 2004].
This approach has the virtue of creating as few seams as possible, but will often introduce large stretch
between the patch and the surface. Such stretching is undesirable because different portions of the
surface are represented using quite different amounts of color detail, as measured in pixel resolution
in the texture map.
  The other major approach is to divide the surface into a collection of patches that can be unfolded
                                                                                                e
with little stretch [Eck et al. 1995; Lee et al. 1998; Sander et al. 2001; Alliez et al. 2002; L´ vy et al. 2002;
Sorkine et al. 2002]. Though stretch is minimized, this approach creates seams between the patches.
These seams cause problems when creating textured images of the surface because the color variation
across the seams must be treated with extreme care or the seams will be noticeable. Some methods
create small disk-like patches [Eck et al. 1995; Lee et al. 1998; Sander et al. 2001; Alliez et al. 2002],
                                                                                                       e
while others attempt to create large patches that match the features contained in the object [L´ vy et al.
2002; Sorkine et al. 2002; Katz and Tal 2003]. Our own work takes this latter approach. We cut the
surface into multiple patches, but according to the large geometric features of the surface. For example,
we would like to recognize the head and limbs of an animal as important features and to create patches
                                              e
that respect these features. The work of L´ vy et al. [2002] and Katz and Tal [2003] have similar goals,
although their feature-based patch creation methods are quite different than our own.
  The definition of the term “geometric feature” varies in different contexts. For surface reconstruction
and mesh simplification, features are often defined in terms of local curvature. This is reasonable
                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
4     •      E. Zhang et al.

because high curvature regions are exactly what these applications are trying to preserve. On the other
hand, surface parameterization algorithms incur higher stretch on a smooth surface with long thin
protrusions than a noisy surface with small protrusions. In this work, we define geometric features as
large protrusions, and our algorithm segments a surface based on its features by performing topological
analysis of some distance-based surface functions.

2.2    Patch Unfolding
There have been many patch unfolding techniques. The classical approach treats the patch unfolding
problem as finding the minimum of some functional that measures the difference between a parameter-
ization with isometry [Eck et al. 1995; Floater 1997]. First, the boundary vertices are assigned initial
positions (usually on a circle or a square). Then the parameterization for the interior vertices is deter-
mined by solving a large linear system or through a nonlinear optimization process. Others have used
stretch measures such as the Green-Lagrange deformation tensor [Maillot et al. 1993] and a variant of
Dirichlet energy [Hormann and Greiner 1999]. Sander et al. [2001] define a geometric stretch metric
that is based on the average and maximal stretch in all directions of a triangle. Sorkine et al. [2002] and
Khodakovsky et al. [2003] have devised stretch metrics based on the maximum and minimum eigen-
values of the stretch tensor. Sander et al. [2001] also propose a post-processing vertex optimization step
that improves their geometric stretch. As we describe later, Sander’s patch optimization approach was
an inspiration for our own work. To allow the boundary vertices of a patch to be free from the arbitrary
                      e
initial assignment, L´ vy et al. [2002] use a least-squares conformal mapping, and Desbrun et al. [2002]
propose an equivalent formulation, Discrete, Natural Conformal Parameterization. Sander et al. [2002]
allow boundary vertices to move, while checking for global intersections. Lee et al. [2002] add layers of
“virtual boundaries” as part of the edge springs to allow the patch boundaries to have natural shapes.
Recently, Sheffer and deSturler [2001; 2002] propose to use an angle-based flattening approach for patch
unfolding. This approach measures stretch in term of the angles deficits between the triangles on the
surface and their textural images, and it removes the need to check for global self-intersections.

3.    FEATURE-BASED PATCH CREATION
Our feature-based patch creation method is carried out in three stages:
(1) genus reduction: a surface with handles (nonzero genus) is converted into a genus zero surface.
(2) feature identification: a genus zero surface is divided into a number of relatively simple shapes.
(3) patch creation: every simple shape is cut into one or two topological disks.
  For both genus reduction and feature identification, we build a surface-based Reeb graph based on the
average geodesic distance introduced by Hilaga et al. [2001]. This graph consists of vertices and edges
in the mesh surface, and we call it an embedded Reeb graph. When properly constructed, this graph
reveals the location of the handles and protrusions in the surface. Since our goal is to create patches
that are topological disks, we need to perform our operations in a topologically consistent manner. For
this purpose, we use surface region growing for all three stages: genus reduction, feature identification
and patch creation. Starting from an initial triangle, we grow a region by adding one triangle at a time
until the whole surface has been covered or until some other stopping criterion has been met. We will
now describe the average geodesic distance function and the embedded Reeb graph that it induces.

3.1    The Average Geodesic Distance Function
The average geodesic distance function was introduced by Hilaga et al. [2001] for the purpose of shape
matching. This is a function A(p) that takes on a scalar value at each point p on the surface S. Let
g (p, q) be the geodesic distance between two points p and q on S. Then the average geodesic distance
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                 Feature-Based Surface Parameterization and Texture Mapping                     •     5




Fig. 2. The average geodesic distance function (AGD) on the dinosaur model is color-coded in the left of this figure. The global
minimum is located underneath the belly, colored in red. Levelsets are painted in repeated patterns of red, green, and blue.
Notice that the tips of large protrusions (horns, legs, tail) are local maxima of AGD. The middle, figure shows the embedded Reeb
graph created by surface-growing based on AGD. Local maxima are indicated by red spheres, and saddle points are highlighted
by blue spheres. Successive critical points are connected by surface paths shown in solid yellow (visible) and dash green (hidden).
The final surface segmentation result based on our algorithm is shown on the right.




       Fig. 3. Comparison among three AGD functions for the dragon: AGD1 (left), AGD2 (middle), and AGD∞ (right).

of p is defined as follows:

                                                                q∈S   g (p, q)d q
                                                  A(p) =                            .                                               (1)
                                                                  Area(S)
A(p) is a member of the following set of functions:

                                                            n    q∈S   g n (p, q)d q
                                                An (p) =                                                                            (2)
                                                                      Area(S)
with n = 1. When n → ∞, A∞ (p) := limn→∞ An (p) = maxq∈S g (p, q), which measures the maximal
distance between p and any point on S. We define for n = 1, 2, . . . , ∞
                                                                     An (p)
                                                AG Dn (p) :=                    .                                                   (3)
                                                                  minq∈S An (q)
   For any n ≥ 1, AG Dn has several useful properties. First, its value measures how “isolated” a point
is from the rest of the surface. Second, its local maxima coincide with the tips of the geometric features
contained in the model. Third, it is scale-invariant and can be used to compare features from different
shapes. Figure 2 (left) shows a polygonal model of a dinosaur, color-coded according to AGD2 . The red
region on the dinosaur’s belly signifies that points in this region have low values of AGD2 . Higher values
adjacent to this middle region are colored in green and then in blue. The colors then cycle repeatedly
through red, green, and blue. Note that the tips of the large features of this object (legs, horns, tail) are
marked by local maxima of AGD2 . (In subsequent sections we will use the term local maxima of AGD
and tips interchangeably.) In practice, we use AGD2 since it seems to produce smooth results. Figure 3
                                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
6     •      E. Zhang et al.

compares the levelsets of the following three functions on the dragon: AGD1 (left), AGD2 (middle), and
AGD∞ (right). From now on, we will use the term AGD to mean AGD2 .
  For a genus zero surface, we use AGD to identify and measure its geometric features. The tip of
a protrusion is a local maximum. Larger values at local maxima signify larger protrusions. Notice
that a noisy surface often contains many small bumps that correspond to local maxima with relatively
small AGD values. Creating patches based on these bumps will increase the amount of seams, without
significantly reducing stretch. Therefore, we only consider local maxima whose AGD values are above
a threshold. In practice, we find choosing any number in [1.3, 1.5] as the threshold for minimal feature
size produces reasonable results, and we use 1.4 for all our test models.
  Computing AGD exactly would be quite costly. We closely follow the algorithm of Hilaga et al. [2001] to
quickly compute a satisfactory approximation of AGD. Briefly, the geodesic distances are not calculated
from all the surface points, but rather from a small number of evenly spaced points on the surface.
We find the geodesic distances from each of these points to all other points efficiently using the fast-
marching method for surfaces [Kimmel and Sethian 1998].

3.2    Building an Embedded Reeb Graph
To find handles and large protrusions in a model, we perform topological analysis of AGD and construct
an embedded Reeb graph that is induced by AGD. The leaf nodes of are situated at the tips of protru-
sions, and the loops in reveal the existence of handles. We construct by performing region-growing
in the increasing order of AGD and tracking the topological changes in the wavefront. This is based
upon ideas from Morse Theory [Milnor 1963] and Reeb graphs [Reeb 1946], which we will review here.
  Let f be a smooth function defined on a smooth surface S ⊂ R3 . For any point p0 ∈ S, let µ = (u, v)
be a parameterization of some neighborhood of p0 in S such that µ(0, 0) = p0 . The gradient ∇ f and
the Hessian H f are defined as follows:
                                                   2                    
                                 ∂ f /∂u              ∂ f /∂u2 ∂ 2 f /∂u∂v
                          ∇f =           , Hf =                         .                     (4)
                                 ∂ f /∂v             ∂ f /∂u∂v ∂ f /∂v
                                                      2           2      2


p0 is a critical point of f if ∇ f (0, 0) = 0. Otherwise, p0 is regular. A critical point p0 is said to be
nondegenerate if H f (0, 0) does not have any zero eigenvalues. In this case, p0 can be classified as a
mimimum/saddle/maximum if H f (0, 0) has zero/one/two negative eigenvalues. f is Morse over S if
 f possesses no degenerate critical points. Morse theory relates the critical points of Morse functions
to the topology of the underlying surface. For instance, when S is a closed orientable 2-manifold with
Euler characteristic χ(S) (twice the number of handles plus two), the following is true for any Morse
function f defined on S with α maxima, β saddles, and γ minima.
                                                         α − β + γ = χ (S).                            (5)
Banchoff extends Morse theory to triangular meshes [1970].
  A continuous function f : S → R induces a Reeb graph f , which can be used to reveal the topological
and skeletal structures of S [Hilaga et al. 2001]. Formally, we define an equivalence relationship ∼ f on
S as follows. Let p, q ∈ S be two points, then p ∼ f q if and only if f (p) = f (q) and p and q belong to
the same connected component of f −1 ( f (p)). Notice f does not have to be Morse. Figure 4 illustrates
an example Reeb graph that corresponds to the vertical height function, defined on a 3D surface. Many
applications make use of Reeb graphs, such as shape-matching [Hilaga et al. 2001] and topological
simplification [Wood et al. 2004]. AGD is, in general, not Morse. For instance, it is a constant function
on a sphere, in which case every point is a degenerate critical point. Axen and Edelsbrunner [1998]
show that a function can be perturbed into a Morse function with surface wave traversal, provided
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                 Feature-Based Surface Parameterization and Texture Mapping                   •     7




Fig. 4. An example of a Reeb graph (right) for a vertical height function, defined on a 3D surface of genus one (left). The critical
points are highlighted by colored spheres (red for maxima, green for minima, and blue for saddles). The number of loops in the
graph equals the number of handles in the surface.

that the mesh is properly subdivided. We use a similar strategy except that we record critical triangles
instead of critical vertices.
   Our algorithm for building an embedded Reeb graph starts with computing AGD for every vertex.
For a triangle T = {v1 , v2 , v3 }, we define AGD(T ) = min{AG D(v1 ), AG D(v2 ), AG D(v3 )}. Starting with a
triangle whose AGD value equals the global minimum, we add one triangle at a time in the increasing
order of the AGD until the surface is covered. The boundary of the visited region consists of a number
of loops. We label a triangle, when it is added, according to one of the following five criteria:
(1) Minimum: where one new boundary loop starts. For our application, there is only one such triangle,
    one of the global minima of the AGD.
(2) Maximum: where one boundary loop vanishes. This is the tip of a protrusion.
(3) Splitting saddle: where one boundary loop intersects itself and splits into two.
(4) Merging saddle: where two boundary loops intersect and merge into one. This signifies the formation
    of a handle.
(5) Regular: where the number of boundary loops does not change.
 A triangle that is not regular is a critical triangle. Let n be the genus of the surface, and let Nmax ,
Nmin , Nss , and Nms be the number of the triangles that are maxima, minima, splitting saddles, and
merging saddles. Then we have,
                                                                        Nms = n                                                   (6)
                                            Nmax − Nss + Nms + Nmin = 2.                                                          (7)
Equation 7 corresponds to the handle-body decomposition of a closed and orientable piecewise linear
2-manifold [Rourke and Sanderson 1972]. Interested readers may refer to Lopes et al. [2003] for more
details. For our application, Nmin = 1. Furthermore, we mark the center of a critical triangle as the
position of the corresponding critical point. The region on the surface swept out between a pair of critical
triangles (not including these critical triangles) is homeomorphic to a cylinder without caps. Let A and
B be a pair of critical triangles, and assume that A is visited earlier than B. We refer to A as the parent
critical triangle and B as the child critical triangle. For a genus zero surface, every child critical triangle
has a single parent. For surfaces with a genus greater than zero, a child critical triangle may have one
or two parents. Let RAB be the connecting region between A and B, which consists of a set of regular
triangles = {T1 , . . . , Tk } in the order of which they are visited. There is a shortest path that connects A
and B using the edges of the set of triangles {A} {B} RAB . We construct the embedded Reeb graph
by finding the shortest paths between every pair of parent/child critical triangles.
   As mentioned earlier, the embedded Reeb graph is much like a Reeb graph that corresponds to
AGD. It reveals the distribution of the geometric features over the surface. The middle of Figure 2
shows the embedded Reeb graph of the dinosaur. Local maxima are highlighted with red spheres, while
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
8     •      E. Zhang et al.




Fig. 5. Embedded Reeb graphs for the bunny surface with different filtering constants α: 1.01 (left), 1.1 (middle), and 1.5 (right).
We use 1.1 as the filtering constant for all the test models.

blue spheres indicate the location of splitting saddle points. The global minimum is marked with a light
blue sphere on the belly. Successive critical points are connected by paths on the surface, which are
drawn in solid yellow (visible) and dash green (hidden). Note that local maxima coincide with the tips
of the geometric features (horns, feet, and tail).
   Since complex surfaces often contain many small protrusions (bumps), the embedded Reeb graph
can contain an excessive number of local maxima and saddle points. This increases the subsequent
processing time since the number of features is much more than what we consider large (or “persistent”
as described in Edelsbrunner et al. [2003]). We use the following filtering scheme to weed out extra
local maxima and splitting saddle points. During surface-growing, we alter the order in which triangles
are added. To be more specific, let t be the unvisited triangle with the smallest AGD value. If adding t
causes a boundary to split, we look for other triangles that could be added without causing a boundary
split. If one of these triangles, t satisfies:
                                                        AGD(t ) < αAGD(t)                                                      (8)
where α is a global filtering constant, then we add t instead of t. When there are multiple choices,
we choose the triangle with the smallest AGD value. Our filtering process is related to the concept
of topological persistence and simplification [Edelsbrunner et al. 2003], but with a different scalar
function and a different measure for persistence. Also, the simplification process is implicit. We apply
the filtering scheme to the bunny surface (Figure 5) with three different α’s: 1.01 (left), 1.1 (middle), 1.5
(right). Notice the excessive saddle points and maxima appear in the head and the paws when α = 1.01
(left). When α = 1.1 (middle), the local maxima that reveal large geometric structures are kept (the
tips of ears and the center of tail). Excessive filtering may result in a trivial embedded Reeb graph,
for example, α = 1.5 (right). This becomes a classical tradeoff between denoising and overblurring. In
practice, we find α ∈ [1.1, 1.3] works well, and we use α = 1.1 for all the test models shown in the article.
   For a genus n > 0 surface, there are n loops in the embedded Reeb graph that are homologically
inequivalent and form the bases of all loops in . In Section 3.4, we describe how we use these loops for
genus reduction, that is, converting a genus n surface into a genus zero surface.
3.3    Feature Identification
Once the tip of a protrusion is located, we construct a closed curve γ on the surface that separates the
feature from the remaining body. Using the terminology from Erickson and Har-Peled [2002], γ is a
separating cycle. We compute γ in two steps. First, we find a separating region R corresponding to the
tip of the protrusion. Next, we construct γ from R.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                Feature-Based Surface Parameterization and Texture Mapping                    •     9




Fig. 6. In this figure, we find a separating region for the bunny’s ear. In the left we graph A(x), the area of the regions given by
evenly spaced distances from the ear’s tip (red), and the smoothed areas (blue). We then calculate A (x), the second derivative
(middle). The maximum of A (x) corresponds to the place where the ear joins the head (right, the green band).


   To find a separating region for the tip point p of a feature, we first calculate the function f p (q) =
g (p, q), the surface geodesic distance function with respect to p. f p is normalized to take on values
in [0, 1]. We consider the regions bounded by iso-value curves of this function. Specifically, we divide
the interval [0, 1] into k equal sections. Next, by performing region-growing from p, we partition the
surface into levelset bands based on the values of f p in these intervals:

                                                            i−1             i
                                          M i := q ∈ S|         ≤ f p (q) ≤                                                        (9)
                                                             k              k
                                                               Ai := Area(M i ).                                                  (10)

   The construction of levelsets in Equation 9 is inspired by Morse theory. The area of this sequence of
bands changes slowly along a protrusion, but it changes abruptly where the feature joins to the rest of
the surface. We find the separating region by analyzing {Ai }, which we treat as a continuous function
A(x). Along a perfect cylindrical feature, A(x) is constant. In the case of a cone, the function grows
linearly. At places where a protrusion joins the main body, A(x) will have a sudden increase, and this
will be the boundary of the feature. We find these increases by looking for the maxima in A (x), the
second derivative of A(x). To eliminate small undulations in A(x), we first low-pass filter A(x) using
a Gaussian function for N times. Both k and N affect the efficiency of separating region detection.
The larger k is, the more samples are used to discretize A(x), and the more likely small noise will be
considered as potential places for the separating region. Similarly, if N is too large, the location for the
separating region may be shifted or even lost. In practice, we use k = 100 and N = 30 for all our test
models. These choices seem to produce reasonable results. Figure 6 illustrates the process of feature
identification for the bunny’s ear.
   Let m be the location where A (m) takes on its maximum value. We define the separating region
R := {q ∈ S | m − ≤ f p (q) ≤ m + }, where we typically use = 0.02. By choosing a positive , we
intentionally make R large for two reasons. First, poor triangulations may cause R to be nonseparating,
that is, it does not contain a separating cycle. Second, we would like more flexibility to allow the
separating cycle (within this region) to be short and smooth.
   The topology of the separating region R can be rather complex if there are other features that join
the surface in nearby places. The only guarantee R provides is that it indeed separates the feature
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
10      •     E. Zhang et al.

from the rest of the surface. We produce a separating cycle γ from R as follows. First, we reduce R
into its skeleton, that is, a collection of edges in the surface that separate the feature from the rest
of the surface. Dangling edges are removed as well. Gu et al. [2002] perform a similar operation to
produce geometry images from meshes. Next, we find a separating cycle ρ from this skeleton. Finally,
we construct another separating cycle γ that is based on ρ, but that is in general shorter and smoother.
These operations are easy to implement on meshes and we describe them in detail next.
   (1) Reduce a separating region R into its skeleton and remove dangling edges. This is achieved
by treating R as a 2-complex (with boundary edges) and repeatedly performing “elementary col-
lapses” [Kaczynski et al. 2004], in which a triangle with at least one boundary edge is removed from the
complex along with one of the boundary edges. In the end, all 2-cells (triangles) are removed, and the
2-complex is reduced to a 1-complex. When there are multiple choices of boundary edges to collapse, we
select the edge with the largest AGD value, which tends to be closer to the feature tip p than the other
edges in the 2-complex. The resulting graph is a skeleton of R with dangling edges. We remove the
dangling edges through elementary collapses on the 1-complex. This results in a collection of loops, one
of which meets our requirement as the separating cycle. The others fall into two categories: separating
cycles for some geometric features inside the feature region, and separating cycles for some geometric
features outside the feature region.
   (2) Eliminate from the 1-complex separating cycles that are either inside or outside the feature region.
To remove the loops outside the feature region, we perform region-growing from the feature tip p with
the constraint that no triangles can be added that cross an edge in the loops computed from the steps in
(1). This makes the loops outside the feature region unreachable from p. For loops inside the feature re-
gion, the average AGD values of their vertices are, in general, greater than that on the separating cycle.
Therefore, these loops can be easily identified and discarded. This step produces a separating cycle ρ.
   (3) Shorten and smooth the separating cycle ρ. We choose two vertices t1 and t2 on ρ that are the
closest to the feature tip p. We find two paths that connect t1 and t2 to p, respectively. The two paths
divide the feature region into two disjoint regions. Within each region, there is a shortest path between
t1 and t2 . Together, they form a separating cycle, which tends to be shorter and smoother than ρ. By
repeating this process twice, we obtain a desired separating cycle γ .
   Figure 7 illustrates this process. For a feature point p and a separating region R (a, the shaded
region), we reduce R to its skeleton through elementary collapses (b). Next, loops that are either inside
or outside the feature region of p are eliminated (c). In the bottom row, the separating cycle ρ is shortened
and smoothed to produce γ , through step 3 (d-f).
   A separating cycle divides S into two surfaces with boundaries. We eliminate these boundaries by
“filling in” the holes with triangles. Basically, we compute c, the average position of the boundary
vertices, and make c a new vertex. We then triangulate the hole by connecting c to the vertices on
the boundary. The filler triangles are subdivided twice, followed by Laplacian smoothing on the newly
created vertices. Some filler triangles can be seen where the head has been separated from the neck
of the bunny in Figure 1. These filler triangles are flagged so that they have minimal effect on patch
unfolding. They become what we call scaffold triangles, to be described later.
   We repeat the feature identification process for the resulting surfaces until the original surface is
divided into a set of feature regions and there are no more feature regions to be found. Figure 1 shows
the result of this process on the bunny, in which four regions were created.
   Our feature identification algorithm assumes that a single loop always divides the surface into two
disjoint regions, which is not necessarily true for surfaces with handles. For these surfaces, the topology
of the separating region can be arbitrarily complex where many small handles are clustered together.
To avoid dealing with this situation, we perform genus reduction before feature identification. Genus
reduction converts a genus n > 0 surface into a genus zero surface.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Feature-Based Surface Parameterization and Texture Mapping                   •      11




Fig. 7. This figure illustrates our algorithm for producing a separating cycle from a separating region. In (a), a separating region
R (shaded) for p is bounded by the red-curves. Next in (b), R is reduced to its skeleton through elementary collapses. The skeleton
consists of three loops: ρ1 , ρ2 , and ρ3 . Notice that ρ2 is not reachable from p, and ρ3 has a higher average AGD value than ρ1 .
By eliminating them, we obtain a separating cycle ρ = ρ1 (c). To smooth ρ, we find two points t1 and t2 in ρ and the shortest
paths that connect t1 p and t2 p (green curves in (d)). Together the two curves divide the region bounded by ρ into two subregions.
Inside each subregion, we construct a shortest path between t1 and t2 (the red curves in (e) and (f)). The union of the two paths
forms a separating cycle γ that is, in general, shorter and smoother than ρ.

3.4   Genus Reduction
For a genus n surface (n > 0), a loop does not always divide the surface into two disjoint connected
components. Loops with this property are associated to the elements of the first homology group, which
form an Abelian group with 2n generators. Using the terminology from [Erickson and Har-Peled 2002],
these loops are nonseparating cycles. Conceptually, the easiest way to think of how these loops arise is
to imagine a hollow handle connected to the rest of the surface; one of the loops cuts across the handle
and the other follows the handle. Observe that for the first type of loops there are two “passages” back
to the rest of the surface. Our strategy for genus reduction is to identify an appropriate nonseparating
cycle for every handle and cut the surface open along the cycles. Each operation converts a handle
into one or two protrusions and reduces the genus of the surface by one. We repeat this process until
the surface contains no handles. Erickson and Har-Peled [2002] have proved that finding the minimal
length cuts needed to turn a surface into a disk is NP-hard, so heuristics are used in practice to find
cuts that are short in length. Genus reduction may be performed using a number of already existing
techniques, including Guskov and Wood [2001], Lazarus et al. [2001], Erickson and Har-Peled [2002],
Gu et al. [2002], Sheffer and Hart [2002], and Wood et al. [2004]. We choose to perform genus reduction
using the embedded Reeb graph and the same distance function that we use for feature identification.
Figure 8 shows our genus reduction algorithm on the dragon (genus one).
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
12      •     E. Zhang et al.




Fig. 8. This figure illustrates the process of genus reduction for the dragon. After the embedded Reeb graph is computed
(a, the graph colored in yellow), we find the independent nonseparating cycle contained in (b, yellow) and perform region-
growing from this loop in both directions until the wavefronts meet (b, the blue and green regions). The meeting point is shown
in (c). We find a path within the green and blue regions that connect the meeting point to the original nonseparating cycle. The
paths form a nonseparating cycle (c and d, the red loop), which can be used for turn a handle into one or two protrusions.

  First, we compute the embedded Reeb graph induced by AGD (a, the yellow graph) and locate all
the basis loops in (b, the yellow loop). Recall (Section 3.2) that a merging saddle point qi signals that
a handle has formed (a, the green sphere). The start of a handle is a splitting saddle point pi , which is,
in general, located near the ends of the passages that connect the handle to the rest of the surface. We
construct a basis loop ρ by computing two shortest paths in that connect qi and pi (b, yellow loop).
  Next, for each basis loop ρ, we create a nearby nonseparating cycle γ for one of the passages by
performing region-growing from ρ in the increasing order of the distance function from pi (c and d,
the red loop). To do so, we treat ρ as the two boundary loops of a region with no interiors. Denote this
region R. Since the surface has handles, region-growing from R causes the two boundary loops to meet
at a merging saddle point r. Figure 8 (b) shows the shapes of the two regions swept by the two loops
when they meet (c, blue and green regions). Within each region, there is a shortest path between r and
ρ. Together, the two paths form a nonseparating cycle γ (d, red loop) that is, in general, shorter and
smoother than ρ.
  Finally, the surface is cut open along γ and the holes are filled with scaffold triangles. This reduces
the genus of the surface by one. We repeat this process until the surface contains no handles, at which
point it is ready for feature identification (Section 3.3). Figure 9 shows the nonseparating cycles that
are generated using our genus reduction algorithm for three surfaces. Notice that these loops appear
in intuitive places and tend to be short, smooth, and nonwinding.
3.5    Patch Creation
Through genus reduction and feature identification, a surface is decomposed into a set of simple shapes
(features) that are topological spheres without large protrusions. Our next task is to create one or two
patches (topological disks) for every feature shape so that unfolding them results in little stretch. This
is carried out in two stages. First, we classify the feature shapes as belonging to one of the following
three profiles: a linear ellipsoid, a flat ellipsoid, and a sphere. Next, we create patches for every feature
shape based on its profile.
   The classification step requires computing the eigenvalues of a covariance matrix for every feature
shape F , which is a triangular mesh. We compute the covariance matrix M F in closed forms by following
the method of Gottschalk et al. [1996] that begins by thinking of F as having been sampled “infinitely
densely” by points over its surface. First, we calculate the mean µ of these points by integrating over all
the triangles. Similarly, we compute M F relative to µ by performing integrations. The three categories
of features are then distinguished based on the eigenvalues from M F :
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Feature-Based Surface Parameterization and Texture Mapping                   •      13




Fig. 9. This figure displays the nonseparating cycles that are used for genus reduction for the Buddha (a), the dragon (upper
right), and the feline (lower right). Each separating cycle consists of a sequence of edges in the surface that form a closed loop.
Notice they appear in intuitive places. Furthermore, they tend to be short, smooth, and nonwinding.

—Three nearly equal eigenvalues (a sphere).
—One eigenvalue much larger than the other two (a linear ellipsoid).
—Two nearly equal eigenvalues that are much larger than the third (a flat ellipsoid).
  Let α, β, γ be the three eigenvalues of M F after normalization such that α 2 + β 2 + γ 2 = 1. The set of
all valid configurations is:
                                     C := {(α, β, γ )|α 2 + β 2 + γ 2 = 1, α, β, γ ≥ 0}.                                        (11)
  C is a spherical triangle in the first octant. There are seven special configurations that correspond
to the three linear ellipsoids, the three flat ellipsoids, and the perfect sphere. By building the Voronoi
regions on C using spherical distances, we can classify every shape based on the position of its con-
figuration in C. Alternatively, one can use the classification measure proposed by Kindlmann and
Weinstein [1999], which produces similar classifications for our test models.
  In the case of a linear ellipsoid, we find a pair of points (p, q) on F such that they achieve the maximum
surface distance. This can be approximated by letting p be a global maximum of AGD, and letting q be
the point that is furthest away from p on the surface. We then find the shortest path γ between p and
q and cut the surface along γ by duplicating all of its vertices except p and q. This converts the surface
into a single patch (a topological disk).
  For the flat ellipsoid case, we first identify the eigenvector associated with the smallest covariance
eigenvalue. Then we find the two most distant surface points x1 and x2 along this vector in opposite
directions away from the surface’s center µ. Using region-growing, we find the Voronoi regions for x1
and x2 . Both regions are homeomorphic to a disk.
  In the case of a sphere, we could treat it as a flat ellipsoid and create two patches that are much like
hemispheres. However, unfolding these patches would cause high stretch. Instead, we use an approach
that is inspired by the two identical patches of a baseball (see Figure 10, upper left). We construct these
regions based on two C-shaped curves, each of which travels halfway around one of the two mutually
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
14      •     E. Zhang et al.




Fig. 10. The “baseball” decomposition of the Venus (lower left) and the corresponding normal maps from patch unfolding using
different stretch metrics: Sander’s metric [Sander et al. 2001] (middle) and the Green-Lagrange tensor (right, Section 4.1).

perpendicular great circles. To compute these curves, we find the three pairs of antipodal points on
the surface that passes through the surface center µ, along the three eigenvector directions. Call these
points x1 , x2 , y 1 , y 2 , z 1 , z 2 . One C-curve passes through x1 , y 1 , x2 , and the other connects z 1 , y 2 , z 2 .
Using region-growing, we compute the “baseball decomposition” of the surface by building the surface
Voronoi regions corresponding to the C-curves. The lower left of Figure 10 shows one of these curves
and their corresponding Voronoi regions (red and green) for the Venus. Also shown in the same figure
are the normal maps corresponding to the decomposition with two different patch unfolding methods:
Sander’s metric [Sander et al. 2001] (middle), and the Green-Lagrange tensor (right, Section 4.1). In
the next section, we will show that patch unfolding using the Green-Lagrange tensor results in less
overall stretch than using Sander’s metric.
   Sometimes a feature is a curved long cylinder, such as the feline’s tail, whose covariance analysis
is similar to that a flat ellipsoid or a sphere. In this case, the center µ is situated outside the volume
enclosed by the surface and not all three pairs of antipodal points can be found. When this happens, we
simply treat the surface as a linear ellipsoid.

4.    PATCH UNFOLDING
A class of traditional patch unfolding methods are based on discrete conformal mappings [Eck et al.
                       e
1995; Floater 1997; L´ vy et al. 2002]. By fixing the texture coordinates of the boundary vertices, the
texture coordinates of the interior vertices can be solved through a closed form system. These methods
                                                   e
are fast and stable, and the solution is unique [L´ vy et al. 2002]. However, conformal mappings do not
preserve areas. Regions can be stretched or compressed, causing uneven sampling rates. Sander et al.
[2001] have proposed a post-processing step in which the texture coordinates of the interior vertices
are optimized to reduce a form of geometric stretch (which we will refer to as Sander’s stretch metric),
and their work inspired our own stretch optimization. We seek a definition of stretch that provides a
balance between conformal mappings and area-preserving mappings.

4.1    The Green-Lagrange Tensor: a Balanced Stretch Metric
An isometry between two surfaces is a bijective mapping f that maintains distances between the two
metric spaces, that is, d ( f (x), f ( y)) = d (x, y) for all points x and y in the domain. An ideal surface
parameterization P would be an isometry between the surface S and its images in the texture map I ,
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                           Feature-Based Surface Parameterization and Texture Mapping                      •      15

which means an everywhere even sampling is possible based on P . For most patches no isometric pa-
rameterization exists, except in the case of developable surfaces. Classical results from Riemannian
geometry state that there exists a conformal (“angle-preserving”) mapping between S and I . Some
parameterization methods first compute a conformal parameterization for a patch, and then optimize
                                                                                    e
the interior vertices based on some stretch metric [Sander et al. 2001; L´ vy et al. 2002; Sheffer and
                                                                           e
de Sturler 2002]. Sander’s metric (used in Sander et al. [2001]; L´ vy et al. [2002]) helps balance the
sampling given by the parameterization. Unfortunately, it does not always distinguish between isome-
tries and anisotropic stretch. To illustrate this point and to introduce our new balanced stretch metric,
we review Sander’s metric and related background.
   For a triangle T = { p1 , p2 , p3 } in the surface S ⊂ R3 , and its corresponding texture coordinates
U = {u1 , u2 , u3 } in R2 = s, t , the parameterization P : U → T is the unique affine mapping that maps
ui to pi (1 ≤ i ≤ 3). To be more specific, let A(v1 , v2 , v3 ) be the area of the triangle formed by vertices v1 ,
v2 and v3 , then
                                     A(u, u2 , u3 ) p1 + A(u1 , u, u3 ) p2 + A(u1 , u2 , u) p3
                          P (u) =                                                              .                               (12)
                                                         A(u1 , u2 , u3 )
Let Ps and Pt denote the partial derivatives of P . The metric tensor induced by P is:
                                                 Ps · Ps Ps · Pt            a b
                                         G=                           =         .                                              (13)
                                                 Ps · Pt Pt · Pt            b c
The eigenvalues of G are

                                                       (a + c) ±       (a − c)2 + 4b2
                                    {γmax , γmin } =                                                                           (14)
                                                                        2
which represents the maximal and minimal stretch of a nonzero vector. Sander’s metric is defined as
the average stretch metric in all possible directions, that is,

                                    L2 (T ) =     (γmax + γmin )/2 =
                                                    2      2
                                                                             (a + c)/2.                                        (15)
The metric has a lower bound of 1, and isometries achieve this lower bound.
  Equation 12 assumes that the area of the triangle equals its textural image. When computing the
global stretch, we assume the total area of the surface equals the total area of the textural image. This
means that we need to add a global scale factor to each triangle. Let A(t) and A (t) be the surface area
and textural area of a triangle t, respectively. The global factor is
                                                                    A(t)
                                                  ρ=          t∈S
                                                                                                                               (16)
                                                             t∈S    A (t)
and we rewrite Equation 12 as:
                                     A(u, u2 , u3 ) p1 + A(u1 , u, u3 ) p2 + A(u1 , u2 , u) p3
                          P (u) =                                                              .                               (17)
                                                         ρ A(u1 , u2 , u3 )
   Notice that individual triangles, in general, have scale factors different from ρ. Unfortunately under
this scenario, there are anisotropic stretch for which Sander’s stretch metric also gives a value of one.
In particular, this metric cannot distinguish between isotropic and anisotropic stretch. For instance, all
of the following tensors
                                            1 0   0.5 0     1 0.5
                                                ,        ,                                                                     (18)
                                            0 1    0 1.5   0.5 1
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
16      •     E. Zhang et al.




Fig. 11. This figure compares two surface parameterizations for the bunny obtained by vertex optimization based on Sander’s
stretch metric [Sander et al. 2001] (middle) and the Green-Lagrange tensor (right). Optimization based on Sander’s metric causes
high anisotropic stretch, especially the two largest patches. Compare the tail and the two rear legs (the square bumps on each side).

result in the same stretch measured in Sander’s stretch metric, but the first one is clearly the most
desirable. For this reason, we use the Green-Lagrange tensor to measure stretch and to guide patch
optimization. Using the Green-Lagrange tensor as the stretch metric has been proposed before [Maillot
et al. 1993]. However, it has not been used for patch optimization.
   Using the above terminology, the Green-Lagrange tensor of G t is defined as G t − I , in which I
is the identity matrix. The square of the Frobenius norm of this tensor is T (G t ) = ( G t − I F )2 =
(a − 1)2 + 2b2 + (c − 1)2 . It is zero if and only if G t is an isometry. We therefore define the stretch as
  Et2 = 2T (G t ) = 2((a − 1)2 + 2b2 + (c − 1)2 ) = [(a − c)2 + 4b2 ] + [(a + c − 2)2 ] = Econformal + Earea . (19)
                                                                                           2            2


Notice that for the tensor to be conformal, we need a = c and b = 0. When these conditions are met,
the tensor becomes a scaling of magnitude a = c. It is an isometry if a = c = 1. This metric seeks
                                                           2
a balance between the angle-preserving energy term Econformal and the area-preserving energy term
Earea . A triangle’s mapping is an isometry if and only if Et = 0. This metric distinguishes between
  2

anisotropic stretch and isometries. In addition, it penalizes both undersampling and oversampling.
However, the penalty is more severe for undersampling. This is desirable for texture mapping when a
global isometry is not available. We note that Sorkine et al. [2002] devise a different stretch metric that
also distinguishes anisotropic stretches from an isometry. We choose not to use their metric because
it uses a max function, causing it to give equal stretch values to some cases that we feel should be
distinguished.
   The total balanced stretch of a patch S is therefore,
                                     E 2 (S) =          (at − ct )2 + 4bt + [(at + ct − 2)2 ] .
                                                                        2
                                                                                                                               (20)
                                                 t∈S

The ideal value E(S) for a patch S is zero, meaning all triangles in the patch are mapped isometrically.
  Figure 11 compares the unfolding of the bunny surface using Sander’s metric (middle) and the Green-
Lagrange tensor (right). Notice that on the two largest patches, unfolding with Sander’s metric produces
anisotropic stretch (the tail and the two rear legs). The Green-Lagrange tensor performs well on all of
these patches. Figure 10 shows the same comparison for the Venus. Again, optimization using Sander’s
metric causes anisotropic stretch. In Section 6.1, we will show the Green-Lagrange tensor also performs
better in terms of image fidelity, despite sometimes having lower packing efficiencies.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                       Feature-Based Surface Parameterization and Texture Mapping              •      17

4.2   Boundary Optimization with Scaffold Triangles
The process of patch optimization refers to moving vertices in the plane to minimize a given stretch
metric. Most patch optimization methods handle the boundary vertices of a patch differently from the
interior vertices. For an initial layout, boundary vertices are typically either mapped to the vertices
of a convex polygon, or placed through conformal mappings. Sander et al. [2001] perform a nonlinear
optimization on the interior vertices by moving one vertex at a time along some randomly chosen line to
improve stretch. This process is very effective in reducing stretch during unfolding. Later, they extend
the optimization framework to handle patch boundaries [Sander et al. 2002]. However, global foldovers
may occur when a boundary vertex accidentally “walks” inside another triangle that is spatially faraway
on the surface. To prevent this from happening, Sander et al. [2002] perform a global intersection
test when performing optimization on a boundary vertex. However, this process is computationally
expensive.
   We introduce a new optimization method that allows the boundary vertices to move freely without the
need to check for global foldovers. First, we compute an initial harmonic parameterization as described
in Floater [1997]. Next, we construct a “virtual boundary” (a square) in the parameterization plane
that encloses the patch. The 3D coordinates of the square are assigned to be mutually different and
outside the convex hull of the patch in the 3D space. As we will see next, the exact coordinates of
the virtual boundary are insignificant, provided that they do not coincide with each other or with the
patch. Scaffold triangles are used to triangulate the region between the original patch boundary and
the virtual boundary. Finally, we perform patch optimization [Sander et al. 2001] on the “enlarged”
patch. There are two issues regarding scaffold triangles that need attention.
(1) How should we define stretch for scaffold triangles?
(2) How can we define and maintain their connectivity?
   The first issue is handled as follows: the stretch of a scaffold triangle is defined as infinity if there is
a foldover, otherwise it is defined as zero. This allows a boundary vertex to move within its immediate
incident triangles to obtain better stretch without the need to check for global foldovers. Furthermore,
the exact 3D coordinates of the virtual boundary are insignificant.
   The second issue appears when the initial connectivity of scaffold triangles unnecessarily constrains
the movements of boundary vertices. This is because scaffold triangles are designed to prevent global
foldovers, that is, one patch vertex “walks” onto a patch triangle other than its immediate neighbor-
ing triangles, which unfortunately include the scaffold triangles. To remedy this overly conservative
approach, we allow scaffold regions to be retriangulated at the end of each optimization step in which
all the vertices have been moved. For any edge between two scaffold triangles, we perform an edge flip
operation if it improves the triangles’ aspect ratios.
   Figure 12 (right) illustrates the effect of using scaffold triangles on an example patch on a cube
(b: without scaffold triangles; d: with scaffold triangles). Notice that scaffold triangles allow the opti-
mization to achieve a zero stretch in this case.
   The shape of the virtual boundary and the connectivity of the scaffold triangles are insignificant
since they merely serve as a placeholder to allow the boundary vertices of a patch to move freely
without causing global foldovers. This is different from the work of Lee et al. [2002], in which virtual
boundaries are constructed as parts of edge springs to obtain an initial parameterization. In their
work, the shape and the connectivity of the virtual boundaries directly affect the stretch of the resulting
parameterization. Indeed, several layers of virtual boundaries are often required to produce reasonable
results using their method. In our work, only one layer is required.
   Scaffold triangles also arise from hole-filling operations that occurred during genus reduction and
feature identification. They are treated similarly as the scaffold triangles from the virtual boundary,
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
18      •     E. Zhang et al.




Fig. 12. This figure demonstrates the effect of scaffold triangles on patch unfolding. A patch that consists of two sides of a cube
(a, colored in yellow) is unfolded using three methods: optimization without scaffold triangles (b), with scaffold triangles but
without optimization (c), and with scaffold triangles and optimization (d). Scaffold triangles are colored in gray. Notice when both
optimization and scaffold triangles are used, the patch is unfolded with a zero stretch.


that is, they do not contribute to the stretch metric unless there are foldovers, and their connectivity
can be changed through retriangulation. Several of the patches in the texture maps shown in Figure 13
(right column, the dinosaur) and Figure 15 (bottom row, the dragon) have holes that make use of scaffold
triangles.

5.   PACKING
The final step of surface parameterization is patch packing, which refers to arranging unfolded patches
inside a rectangular region (texture map) without any overlaps. The ratio between the total space oc-
cupied by the patches and the area of the rectangle is the packing efficiency. Higher packing efficiency
indicates less wasted space in the final texture map. The problem of finding an optimal packing is a
special instance of an NP-hard problem: containment and minimum enclosure. The problem has been
studied extensively in the textile industry and the computational geometry community [Milenkovic
1998]. Several packing algorithms have been proposed as parts of some surface parameterization tech-
                              e
niques [Sander et al. 2001; L´ vy et al. 2002]. These methods are very effective when all the patches are
nearly circular and have similar sizes. Since our patch creation technique tends to produce a small num-
ber of large and often-elongated patches, we developed a packing technique that takes into account the
orientations of the patches and, in general, achieves better packing efficiencies. Later, we discovered that
our method is very similar to the packing technique developed independently by Sander et al. [2003].
  Our packing algorithm is based on the following two observations. First, our patch creation method
tends to produce a small number of patches. Second, several of these patches are large and have
elongated shapes. The first observation enables us to perform an optimal searching that would have
been impractical for patch creation methods that produce hundreds of patches. The second observation
indicates that the orientations of the large and elongated patches can help create gaps, into which
smaller patches can be placed.
  Our algorithm consists of two stages: initialization and placement. During the initialization stage,
we create a “canvas”, that is, an N × N grid structure at the textural resolution. Every cell in the
canvas is marked as “unoccupied”. Under the same resolution, we discretize the bounding box of every
patch into a 2D grid and mark any cell intersecting the patch as “occupied”. We obtain eight variations
of grids for each patch by the combination of reflections with respect to the patch’s vertical axis, the
horizontal axis, and the diagonal.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Feature-Based Surface Parameterization and Texture Mapping                  •      19




Fig. 13. This figure compares the packing results using the algorithm of Levy et al. [2002] (top row) and our algorithm (bottom
row) for three models: the feline (left), the Buddha (middle), and the dinosaur (right). Notice the space under the “horizon” is
used to pack smaller patches, and some patches are reflected diagonally to achieve a tighter packing.

  During the placement stage, we insert the patches, one-by-one, into the canvas in the decreasing
order of patch size (area). The first patch is placed at the lower left corner of the canvas. After a patch is
inserted, we update the status of cells in the canvas that have been covered by the newly placed patch.
Before inserting the next patch Pi , we examine its eight variations to find the one that minimizes the
wasted space in the canvas. To be precise, let α, a m × n grid cells, be a variation for Pi . We wish to
place the lower-left corner of α in the (a, b) grid cell in the canvas such that the following conditions are
met:

(1) For any occupied grid cell ( p, q) in α, the corresponding grid cell (a + p, b + q) in the canvas is
    unoccupied.
(2) α minimizes max(a + m, b + n).

   In other words, we wish to place the patch as close to the lower left corner of the canvas as possible.
Once the best variation is chosen, we translate and scale the patch to reflect its position and orientation
in the canvas.
   After all patches have been inserted, usually only M ×M grid cells in the canvas have become occupied.
For all our test models, M is between one-third and one-half of the size of the canvas. We perform scaling
to all patches with the same factor so that the M × M grid cells are mapped to [0, 1] × [0, 1].
   Figure 13 shows the improvement of our packing algorithm (bottom row) over the algorithm by L´ vy   e
et al. [2002] (top row) with three test models: the feline (left), the Buddha (middle), and the dinosaur
(right). Notice the space under the “horizon” is reused to pack small patches, and some patches are
rotated/reflected to achieve a tighter packing.
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
20      •     E. Zhang et al.




Fig. 14. This figure shows the result of our feature segmentation method on various test models. The cow, the horse and the
rabbit are genus zero surfaces. The genus of the dragon, the Buddha, and the feline are one, six, and two, respectively.


6.   RESULTS
We have applied our feature-based surface parameterization method to a number of test models. The
results for the bunny and the dinosaur are shown in Figure 1 and 2, respectively. In Figure 14, we
show the results of some other 3D models, including three surfaces with nonzero genus (the Buddha,
the dragon, and the feline). Notice, in general, the feature regions are intuitive. For example, the horns
and legs of animals are segmented from the bodies, and the Buddha’s stand is identified as a single
feature (a flat ellipsoid). (We wish to emphasize that no real animals were harmed during our research.)
Figure 10 (right) and Figure 13 (lower middle) show the normal maps of the Venus and the Buddha,
respectively. In a normal map, colors (R, G, B) are used to encode unit surface normals (x, y, z) [Sander
et al. 2001]. Because of the many sharp creases on the Venus and the Buddha, patch creation methods
based on surface curvature would have split the surfaces into many tiny patches, and such an example
                   e
can be found in L´ vy [2003]. Our method, however, was able to create large patches with little stretch.
  Figure 15 shows textured models (left column) and the corresponding texture maps (right column) of
the Buddha (top), the feline (middle), and the dragon (bottom). Table I gives the average stretch for the
patches of the test models and the times for patch creation and unfolding using our method. The texture
used for the Buddha is a wood texture from Perlin’s noise [1985]. The textures used for the feline and
the dragon were created by performing example-based texture synthesis directly on the surfaces [Turk
2001; Wei and Levoy 2001].
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                             Feature-Based Surface Parameterization and Texture Mapping                 •      21




Fig. 15. This figure shows the parameterization of three models using our feature-based algorithm: textured models (left) and
texture layouts (right, 512 × 512).



                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
22      •     E. Zhang et al.

       Table I. Average stretch (measured in Green-Lagrange) and timing results (minutes:seconds) for our
                  patch creation and unfolding algorithm. Times are measured on a 2.4 GHz PC
        model                       #              #                 stretch                 patch creation   patch unfolding
        name                    polygons        patches         (Green-Lagrange)                 time              time
        Buddha                   20,000           28                  1.56                        6:32             27:29
        Buddha (large)          100,000           16                  1.27                       39:25            168:43
        bunny                    10,000             6                 0.23                        1:07               8:28
        cow                      10,524           29                  0.28                        2:15               4:01
        dinosaur                 10,636           14                  0.25                        1:37               5:53
        dragon                   20,000           24                  0.83                        3:00             16:23
        dragon (large)          100,000           39                  0.49                       38:00            124:32
        feline                   10,000           41                  0.22                        2:31               2:32
        feline (large)          100,000           46                  0.29                       32:57            109:27
        horse                    10,000           27                  0.22                        1:30               3:21
        Venus                    10,000             2                 0.17                        0:11             11:38
        rabbit                   10,000             8                 0.24                        0:53               4:50


6.1    Measuring Quality
Measuring the quality of a surface parameterization is an important yet complicated issue. It has
several components.

(1) Stretch affects the sampling rate across the surface.
(2) Seams cause discontinuities across patch boundaries.
(3) Smoothness measures the amount of sharp changes in the sampling rates across interior edges of
    a patch.
(4) Packing efficiency determines the efficient use of the texture map.
   When evaluating a surface parameterization method, it is not clear how these components should be
combined measure the quality of the resulting map. On the other hand, for texture mapping applications,
the quality of a surface parameterization should reflect “image fidelity”, that is, the faithfulness of
the images produced using the texture maps to the images for which the surface signals are directly
computed. Next, we present an image-based metric, which draws inspiration from the work on image-
driven mesh simplification [Lindstrom and Turk 2000].
   Given a surface parameterization P , we first compute a continuous and smooth surface signal and
store the result in a texture map based on P . Then, we render the surface from many viewpoints using
the texture map, and compare the image differences with respect to the true surface signals. In practice,
we choose 20 orthographic viewpoints that are the vertices of a surrounding dodecahedron. Let M 0 be
the surface with the signal directly computed, and M i be the textured surface with the texture size of
2i × 2i . The RMS “image” error between the images is calculated as:

                                                                                20
                                                   RMS ( M i , M 0 ) =               Din .                                      (21)
                                                                               n=1

Here, Din is the squared sum of pixel-wise intensity difference between the n-th image of M i and M 0 .
Equation 21 can be seen as the discretization of the following functional:

                                                              p∈S   D 2 (M 0 (p), M (p))V (p)d p
                                      E(M , M 0 ) =                                                  .                          (22)
                                                                         p∈S   V (p)d p

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Feature-Based Surface Parameterization and Texture Mapping                    •      23

Here M 0 and M refer to the original and the reconstructed surface signals (in colors), respectively.
D(M 0 (p), M (p)) is a perceptual metric between colors. For our application, we use

                  D((r1 , g 1 , b1 ), (r2 , g 2 , b2 )) =   ((r2 − r1 )2 + ( g 2 − g 1 )2 + (b2 − b1 )2 )/3.                    (23)

V (p) is the view-independent surface visibility as defined in Zhang and Turk [2002], which measures
the visibility of p with respect to viewpoints on a surrounding sphere of an infinite radius. Therefore,
E(M , M 0 ) takes into account surface visibility in addition to color errors caused by stretch, seams,
packing efficiencies, and smoothness of the parameterization. As demonstrated in Lindstrom and Turk
[2000] and Zhang and Turk [2002], the error metric can be sampled with a small number of viewpoints
that are evenly spaced in the view space.
  One possible ideal surface signal can be obtained by first spreading a set of evenly spaced points on the
surface and building a smooth function that uses these points as the bases. However, computing such
a function over a surface with complex geometry is time-consuming. In contrast, a 3D checkerboard
pattern is relatively easy to compute, and it has the nice property that the largest differential in
frequencies in all directions is bounded. Although not perfect, it is nonetheless a good starting point.
To make the signal continuous, we replace each “box” section with a “hat”. The frequency in each main
axial direction is the same. In practice, we use 1/16 of the maximum side of the bounding box of the
surface as the frequency.
  Table II compares two unfolding methods, optimization with Sander’s metric and the Green-Lagrange
tensor for nine test models. Notice optimization with our metric produces lower stretch for all the test
models. Furthermore, despite sometimes having lower packing efficiencies, optimization with our metric
produces lower image errors for all the test cases. Figure 16 provides a visual comparison between the
ideal signal (bottom-middle), the textured model using optimization with Sander’s metric (bottom-left)
and the Green-Lagrange tensor (bottom-right) for the Buddha model with the texture map of size
128 × 128. Notice the different level of blurring in the left image (front body and base) due to a highly
uneven sampling rate. This phenomenon is less noticeable in the right image that uses our approach.
Compare their corresponding texture maps: left row (Sander’s metric) and right row (our technique).

7.   CONCLUSION AND FUTURE WORK
In this article, we present an automatic surface parameterization method in which manifold surfaces
are divided into feature regions for patch creation. By performing topological analysis of the average
geodesic distance function, we are able to divide the surface into a small number of large patches that
can be unfolded with little stretch. For patch unfolding, we use the Green-Lagrange tensor to guide the
vertex optimization process. Also, we use scaffold triangles to allow the boundary vertices of a patch
to be optimized. Although they were developed with texture mapping in mind, we think our surface
segmentation and genus reduction methods might also be useful for other applications. Finally, we
describe an image-based quality measure for surface parameterization techniques.
   There are several areas for improvement. First, the nonlinear optimization stage of our patch un-
folding algorithm is rather slow. Sander et al. [2002] propose a hierarchical optimization framework,
which we would like to adapt to our system with scaffold triangles. Second, our feature identification
technique sometimes creates more feature regions than necessary. For instance, the horse’s legs are
divided into two or three regions (Figure 14). While this does not seem to dramatically affect the quality
of final texture maps, we are looking for alternative methods that can determine separating regions
more robustly. For instance, we would like to see whether AGD could be used to decide the boundaries
of separating regions. Since AGD is less noisy and is intrinsic to the surface, we expect that it contain
more useful information.
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
24      •       E. Zhang et al.

    Table II. This table compares two stretch metrics for guiding Sander-style vertex optimization. With the
  exception of the columns labeled “Image Error”, the top row in each data cell are the results using Sander’s
metric, and the bottom rows are the results using the Green-Lagrange tensor (Section 4.1). For a comprehensive
comparison, three measurements are provided: average stretch (the first two columns), packing efficiency (third
 column), and image-based error metric (Section 6.1, the last two columns). The numbers in the “Image Error”
 columns are the percentage of error difference of the image error caused by Sander’s metric to the image error
caused by our technique. For all nine test models, optimization with the Green-Lagrange tensor results in lower
   average stretch (either measured using Sander’s metric or the Green-Lagrange tensor). Despite sometimes
       having lower packing efficiencies, our technique results in less image errors for all the test models
                                Comparison for patch unfolding with different optimization metrics
                                       (top row using Sander’s metric [Sander et al. 2001],
                                            bottom row using Green-Lagrange tensor)
                      Stretch measured in        Stretch measured in         Packing        Image Error   Image Error
                        Sander’s metric            Green-Lagrange             Ratio          128 × 128     256 × 256
     Buddha                   1.27                      26.80                  0.67            8.28%        10.77%
                              1.18                       1.56                  0.68
     bunny                    1.13                       3.92                  0.60           14.46%        10.93%
                              1.02                       0.23                  0.65
     cow                      1.11                       3.07                  0.73            1.89%         1.62%
                              1.03                       0.28                  0.65
     dinosaur                 1.07                       1.55                  0.59           13.22%         5.16%
                              1.03                       0.25                  0.66
     dragon                   1.26                      13.78                  0.67           11.14%        12.84%
                              1.13                       0.83                  0.67
     feline                   1.10                       1.73                  0.66            7.25%         3.46%
                              1.02                       0.22                  0.64
     horse                    1.09                       1.65                  0.67            5.57%         0.93%
                              1.03                       0.22                  0.66
     Venus                    1.10                       2.99                  0.59           16.29%        15.26%
                              1.02                       0.17                  0.66
     rabbit                   1.12                       3.00                  0.68            8.73%         4.54%
                              1.03                       0.24                  0.65




  Many additional topics in this area are interesting to us. First, our algorithm does not directly
minimize the amount of seams caused by the segmentation. It would be desirable to have control over
seams.
  Second, although our image-based quality measure implicitly takes into account stretch, seams,
smoothness, packing efficiency and visibility, it may be desirable to understand and control their impact
on the quality of a surface parameterization. For instance, while the Green-Lagrange tensor is a good
indicator of stretch, how it relates to the image-based quality measure deserves further investigation.
  Third, we are still looking for other functions for surface segmentation. Our metric has been chiefly
based on surface distance, which is intrinsic to the surface. Are there other functions that combine both
intrinsic and extrinsic properties of the surfaces that might result in an even better segmentation? For
instance, is there a surface function that helps find a separating cycle for the feline’s wing exactly where
human would place it?
  Finally, surface parameterization is important for many applications, one of which is surface visu-
alization. A complex surface often contains interiors and concavities that are difficult to see from the
outside viewpoints. We would like to investigate the use of parameterization for surface exploration,
giving the user the ability to navigate, orient, and focus.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Feature-Based Surface Parameterization and Texture Mapping                   •      25




Fig. 16. Comparisons between patch unfolding with Sander’s metric [Sander et al. 2001] (left column) and with Green-Lagrange
stretch tensor (right column) for the Buddha using the 3D texture described in Section 6.1 (original signal is shown in the middle
of the bottom row). In each column, from top-to-bottom, are the normal map, the map of the 3D texture, and the textured model.
Notice in the left column (Sander’s metric), the patches created from the base are assigned more space than those from the
Buddha’s torso. This causes a loss of signal in the textured model (face, body, and feet). On the other hand, optimization using
the Green-Lagrange tensor (right column) produces a more even sampling rate and the reconstruction error is less noticeable.


                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
26      •     E. Zhang et al.

ACKNOWLEDGMENTS
                                                                                          e
We would like to thank the following people and groups for the 3D models they provided: Zo¨ Wood and
          o
Peter Schr¨ der, Mark Levoy and the Stanford Graphics Group, Andrzej Szymczak, and Cyberware. We
also appreciate the discussions with Jarek Rossignac and Andrzej Szymczak. Finally, we wish to thank
our anonymous reviewers for their valuable comments and suggestions.

REFERENCES
ALLIEZ, P., MEYER, M., AND DESBRUN, M. 2002. Interactive geometry remeshing. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3
  (July), 347–354.
AXEN, U. AND EDELSBRUNNER, H. 1998. Auditory morse analysis of triangulated manifolds. Mathematical Visualization, H. C.
  Hege and K. Polthier, Eds. Springer-Verlag, Heidelberg, Germany, 223–236.
BANCHOFF, T. F. 1970. Critical points and curvature for embedded polyhedral surfaces. American Mathematical Monthly 77,
  475–485.
BENSON, D. AND DAVIS, J. 2002. Octree textures. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 785–790.
CARR, N. A. AND HART, J. C. 2002. Meshed atlases for real-time procedural solid texturing. ACM Trans. Graph. 21, 2, 106–131.
CIGNONI, P., MONTANI, C., ROCCHINI, C., AND SCOPIGNO, R. 1998. A general method for recovering attribute values on simplified
  meshes. IEEE Visualization Proceeding, 59–66.
DEBRY, D., GIBBS, J., PETTY, D. D., AND ROBINS, N. 2002. Painting and rendering textures on unparameterized models. ACM
  Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 763–768.
DESBRUN, M., MEYER, M., AND ALLIEZ, P. 2002. Intrinsic parameterizations of surface meshes. Proceeding of Eurographics,
  209–218.
ECK, M., DEROSE, T., DUCHAMP, T., HOPPE, H., LOUNSBERY, M., AND STUETZLE, W. 1995. Multiresolution analysis of arbitrary
  meshes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1995), 173–182.
EDELSBRUNNER, H., HARER, J., AND ZOMORODIAN, A. 2003. Hierarchical Morse-Smale complexes for piecewise linear 2-manifolds.
  Discrete Comput. Geom. 30, 87–107.
ERICKSON, J. AND HAR-PELED, S. 2002. Optimally cutting a surface into a disk. Symposium on Computational Geometry, 244–253.
FLOATER, M. S. 1997. Parameterization and smooth approximation of surface triangulations. Comput. Aid. Geomet. Design 14, 3
  (July), 231–250.
GOTTSCHALK, S., LIN, M. C., AND MANOCHA, D. 1996.        OBB-Tree: A hierarchical structure for rapid interference detection.
  Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1996), 171–180.
GU, X., GORTLER, S. J., AND HOPPE, H. 2002. Geometry images. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 355–361.
GUSKOV, I. AND WOOD, Z. 2001. Topological noise removal. Graphics Interface, 19–26.
HANRAHAN, P. AND HAEBERLI, P. E. 1990. Direct WYSIWYG painting and texturing on 3d shapes. Computer Graphics Proceedings,
  Annual Conference Series (SIGGRAPH 1990), 215–223.
HILAGA, M., SHINAGAWA, Y., KOHMURA, T., AND KUNII, T. L. 2001. Topology matching for fully automatic similarity estimation of
  3d shapes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001), 203–212.
HORMANN, K. AND GREINER, G. 1999. MIPS: An efficient global parameterization method. Curve and Surface Design: Saint-Malo
                            e
  1999, Laurent, Sablonni` re and Schumaker, Eds. Vanderbilt University Press. 153–162.
KACZYNSKI, T., MISCHAIKOW, K., AND MROZEK, M. 2004. Computational Homology. Applied Mathematical Sciences 157, Springer-
  Verlag.
KATZ, S. AND TAL, A. 2003. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM Trans. Graph. (SIGGRAPH
  2003) 22, 3 (July), 954–961.
                                     ¨
KHODAKOVSKY, A., LITKE, N., AND SCHRODER, P. 2003. Globally smooth parameterizations with low distortion. ACM Trans. Graph.
  (SIGGRAPH 2003) 22, 3 (July), 350–357.
KIMMEL, R. AND SETHIAN, J. A. 1998. Computing geodesic paths on manifolds. Proceedings of National Academy of Sciences 95, 15
  (July), 8431–8435.
KINDLMANN, G. AND WEINSTEIN, D. 1999. Hue-balls and lit-tensors for direct volume rendering of diffusion tensor fields. IEEE
  Visualization Proceeding, 183–189.
LAZARUS, F., POCCHIOLA, M., VEGTER, G., AND VEROUST, A. 2001.     Computing a canonical polygonal schema of an orientable
  triangulated surface. 17th ACM Symposium on Computational Geometry, 80–89.
                              ¨
LEE, A., SWELDENS, W., SCHRODER, P., COSWAR, L., AND DOBKIN, D. 1998. MAPS: Multiresolution adaptive parameterization of
  surfaces. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1998), 95–104.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Feature-Based Surface Parameterization and Texture Mapping                  •      27

LEE, Y., KIM, H. S., AND LEE, S. 2002. Mesh parameterization with a virtual boundary. Computers & Graphics (Special Issue
  of the 3rd Israel-Korea Binational Conference on Geometric Modeling and Computer Graphics) 26, 5, 677–686.
L´ VY, B. 2003. Least squares conformal maps gallery. http://www.loria.fr/levy/Galleries/LSCM/index.html.
 E
L´ VY, B., PETITJEAN, S., RAY, N., AND MAILLOT, J. 2002. Least squares conformal maps for automatic texture atlas generation.
 E
  ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 362–371.
LINDSTROM, P. AND TURK, G. 2000. Image-driven simplification. ACM Trans. Graph. 19, 3, 204–241.
LOPES, H., ROSSIGNAC, J., SAFONOVA, A., SZYMCZAK, A., AND TAVARES, G. 2003. Edgebreaker: A simple compression algorithm for
  surfaces with handles. Comput. Graph. Int. J. 27, 4, 553–567.
MAILLOT, J., YAMIA, H., AND VERROUST, A. 1993. Interactive texture mapping. Computer Graphics Proceedings, Annual Confer-
  ence Series (SIGGRAPH 1993), 27–34.
MILENKOVIC, V. J. 1998. Rotational polygon containment and minimum enclosure. Proceedings of the 14th Annual Symposium
  on Computational Geometry, ACM.
MILNOR, J. 1963. Morse Theory. Annals of Mathematical Studies. Princeton University Press, Princeton, NJ.
NI, X., GARLAND, M., AND HART, J. C. 2004. Fair morse functions for extracting the topological structure of a surface mesh. ACM
  Trans. Graph. (SIGGRAPH 2004) 23, 3 (Aug.), 613–622.
PERLIN, K. 1985. An image synthesizer. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1985), 287–
  296.
PIPONI, D. AND BORSHUKOV, G. 2000. Seamless texture mapping of subdivision surfaces by model pelting and texture blending.
  Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2000), 471–478.
REEB, G. 1946. Sur les points singuliers díune forme de pfaff completement integrable ou díune fonction numerique [on the
  singular points of a completely integrable pfaff form or of a numerical function]. Comptes Randus Acad. Sciences Paris 222,
  847–849.
ROURKE, C. AND SANDERSON, B. 1972. Introduction to Piecewise-Linear Topology. Springer Verlag.
SANDER, P. V., GORTLER, S. J., SNYDER, J., AND HOPPE, H. 2002. Signal-specialized parameterization. Proceedings of the 13th
  Eurographics Workshop on Rendering, 87–100.
SANDER, P. V., SNYDER, J., GORTLER, S. J., AND HOPPE, H. 2001.        Texture mapping progressive meshes. Computer Graphics
  Proceedings, Annual Conference Series (SIGGRAPH 2001), 409–416.
SANDER, P. V., WOOD, Z. J., GORTLER, S. J., SNYDER, J., AND HOPPE, H. 2003. Multi-chart geometry images. Proceedings of the 1st
  Symposium on Geometry Processing.
SHEFFER, A. AND DE STURLER, E. 2001. Parameterization of faceted surfaces for meshing using angle based flattening. Engineer.
  with Comput. 17, 3, 326–337.
SHEFFER, A. AND DE STURLER, E. 2002. Smoothing an overlay grid to minimize linear distortion in texture mapping. ACM Trans.
  Graph. 21, 4, 874–890.
SHEFFER, A. AND HART, J. C. 2002. Seamster: Inconspicuous low-distortion texture seam layout. IEEE Visualization Proceeding,
  291–298.
SORKINE, O., COHEN-OR, D., GOLDENTHAL, R., AND LISCHINSKI, D. 2002. Bounded-distortion piecewise mesh parameterization.
  IEEE Visualization Proceeding, 355–362.
TURK, G. 2001. Texture synthesis on surfaces. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001),
  347–354.
WEI, L. Y. AND LEVOY, M. 2001. Texture synthesis over arbitrary manifold surfaces. Computer Graphics Proceedings, Annual
  Conference Series (SIGGRAPH 2001), 355–360.
                                             ¨
WOOD, Z., HOPPE, H., DESBRUN, M., AND SCHRODER, P. 2004. Removing excess topology from isosurfaces. ACM Trans. Graph. 23, 2,
  190–208.
ZHANG, E. AND TURK, G. 2002. Visibility-guided simplification. IEEE Visualization Proceeding, 267–274.


Received June 2003; revised April 2004; accepted August 2004




                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
On C 2 Triangle/Quad Subdivision
SCOTT SCHAEFER and JOE WARREN
Rice University


In this article, we present a subdivision scheme for mixed triangle/quad meshes that is C 2 everywhere except for isolated,
extraordinary points. The rules that we describe are the same as Stam and Loop’s scheme [2003] except that we perform
an unzipping pass prior to subdivision. This simple modification improves the smoothness along the ordinary triangle/quad
boundary from C 1 to C 2 , and creates a scheme capable of subdividing arbitrary meshes. Finally, we end with a proof based on
Levin and Levin’s [2003] joint spectral radius calculation to show our scheme is indeed C 2 along the triangle/quad boundary.
Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling
General Terms: Algorithms
Additional Key Words and Phrases: Subdivision, triangle/quad, joint spectral radius




1.    INTRODUCTION
Subdivision has become a staple of the geometric modeling community allowing coarse, polygonal shapes
to represent highly refined, smooth shapes with guaranteed continuity properties. Previously, there has
been a dichotomy between polygonal primitives that subdivision schemes operate on. Two of the most
popular subdivision schemes, Loop [1987] and Catmull and Clark [1978], operate on triangle and quad
meshes, respectively.

1.1    Stam and Loop’s Scheme
Recently, Stam and Loop [2003] introduced a generalization of Loop and Catmull-Clark subdivision
that unifies these schemes together and operates on mixed triangle/quad surfaces. The subdivision
scheme that they present reproduces Loop subdivision on the triangular portions of the mesh and
Catmull-Clark subdivision on the quadrilateral polygons. Furthermore, the authors derive subdivision
rules for extraordinary vertices composed of both quads and triangles where the subdivision scheme
is C 1 .
   Stam and Loop [2003] created their generalization of triangle and quad subdivision by utilizing
the fact that both Loop and Catmull-Clark subdivision can be written as linear subdivision, followed
                               o
by averaging [Zorin and Schr¨ der 2001; Stam 2001; Warren and Weimer 2001]. For triangle/quad
meshes, linear subdivision splits triangles into four new triangles and quads into four new quads. This
process introduces what Stam and Loop called an ordinary edge along the triangle/quad boundary
where vertices are contained by two adjacent quads and three adjacent triangles (see Figure 1).

Authors’ address: Department of Computer Science, Rice University, 6100 South Main St., Houston, TX 77251-1892; email:
sschaefe@rice.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0028 $5.00

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 28–36.
                                                                               On C 2 Triangle/Quad Subdivision            •      29




Fig. 1. Linear subdivision for triangle/quad meshes. An ordinary triangle/quad configuration is introduced all along the boundary
edge.




                 Fig. 2. Averaging masks for Catmull-Clark (left), Loop (middle) and Triangle/Quad (right).




         Fig. 3. Levin/Levin’s rules for the central edge (left and middle). Unzipping rule for triangular side (right).

   Once linear subdivision is complete, an averaging pass is applied to the mesh. Figure 2 shows the
averaging mask for the ordinary case of Catmull-Clark and Loop subdivision. Stam and Loop [2003]
noticed that the averaging masks for triangle and quad subdivision looked remarkably similar and
hypothesized that the averaging mask for mixed triangle/quad surfaces at the ordinary boundary would
simply be the mask shown on the right of Figure 2. The authors then used this observation to generalize
the averaging mask to arbitrary configurations of quads and triangles around a vertex. Finally, Stam
and Loop show that their scheme is C 2 everywhere except for extraordinary points and the ordinary,
triangle/quad boundary, where their scheme is C 1 .

1.2   Levin and Levin’s Scheme
To remedy this smoothness problem along ordinary triangle/quad edges, Levin and Levin [2003] intro-
duced a set of modified rules along the triangle/quad boundary, shown in Figure 3. The authors also
present the concept of an “unzipping” mask, shown in Figure 3 (right). Prior to subdivision, points along
the regular triangle/quad boundary are replicated; one set of vertices for the quadrilateral polygons and
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
30      •     S. Schaefer and J. Warren




Fig. 4. Subdivision for triangle/quad meshes as centroid averaging. Centroids of each type of polygon weighted by the angle
spanned in the ordinary configuration (left). Averaging rule at ordinary boundary formed from centroids (right).

another for the triangular polygons. This replication essentially “unzips” the mesh into disjoint pieces
consisting of only triangles or only quads. The replicated vertices for the quadrilateral polygons retain
their original positions; however, the vertices along the boundary for the triangular polygons have the
mask in Figure 3 (right) applied to them. When subdivision is performed, the new vertices on the quadri-
lateral portions of the mesh use only the quadrilateral vertices, while new vertices on the triangular
portions of the mesh use only triangular vertices. The vertices actually on the triangle/quad boundary
use only the original vertices of the mesh.
   Levin and Levin [2003] then prove that these modified rules generate a surface that is C 2 across the
triangle/quad boundary. As part of their proof, the authors present a sufficient test for C 2 smoothness
based on a joint spectral radius calculation between two subdivision matrices, and show that their
modifications generate a C 2 subdivision scheme. However, this subdivision scheme can be difficult to
apply in practice due to the special rules introduced along the triangle/quad boundary, which also have
larger support than the C 1 rules and do not readily fit into the averaging subdivision framework.

2.   THE UNIFIED SUBDIVISION SCHEME
Our implementation of triangle/quad subdivision uses the centroid averaging approach described by
Warren and Schaefer [2003]. This method proceeds in two stages: linear subdivision and centroid
averaging where linear subdivision is identical to Stam and Loop’s [2003] linear subdivision pass.
   The output of linear subdivision is a refined mesh taken as input to the averaging pass. In this
pass, the authors provided a different extension from Stam and Loop’s [2003] scheme to extraordinary
vertices contained by both quads and triangles using a centroid averaging approach. For each vertex in
the input mesh, averaging repositions that vertex to its final location after one round of subdivision. To
reposition a vertex, averaging finds all polygons containing that vertex and computes the set of centroids
shown in Figure 4 for those polygons. For quads, the centroid is simply 1 of each of its vertices, summed
                                                                         4
together. For triangles, the centroid is calculated using 1 of the vertex being repositioned and 3 of the
                                                          4                                         8
edge adjacent vertices. Next, a weighted average of these centroids is computed to reposition the central
                                                                                                π
vertex by weighting each centroid by the angle spanned by that polygon in the ordinary case ( 2 for quads
and π for triangles). This centroid averaging rule reproduces Stam and Loop’s [2003] averaging rule
     3
along the ordinary triangle/quad edge, but provides a different generalization to other configurations
of polygons.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                              On C 2 Triangle/Quad Subdivision             •      31




Fig. 5. Unzipping mask for the vertices part of the triangle/quad boundary in the ordinary case (left) and the arbitrary case
(right). n is the number of edges incident to the vertex that are part of the network of triangle/quad boundary edges.




Fig. 6. Initial shape (upper left). Unzipped shape (upper right). Linear subdivision (bottom left). Averaging pass zips mesh back
together (bottom right).

   Like Levin and Levin [2003], we utilize an unzipping mask during subdivision and augment the
triangle/quad subdivision scheme of Warren and Schaefer [2003] with an unzipping pass. However, our
unzipping mask differs from Levin and Levin’s choice and is shown in Figure 5. We have also extended
our unzipping masks to arbitrary configurations of edges that are part of the triangle/quad boundary,
which allows us to subdivide a greater variety of surfaces.
   Prior to subdivision, we identify edges on the surface contained by both triangles and quads. These
edges define a network of curves on the surface. Then we apply the unzipping masks (Ut , Uq ) to this
curve network to generate separate triangle and quad vertices along the triangle/quad boundary (we
also designate vertices contained completely by triangles or completely by quads to be triangle and
quad vertices, respectively). Next, we apply linear subdivision and averaging to the resulting points.
Our only modification that we make to Warren and Schaefer’s [2003] scheme is that we require that each
centroid is calculated using vertices only of the same type as the polygon; that is, triangle centroids
are calculated using only triangle vertices and, similarly, quad centroids are calculated using only
quad vertices. This small modification generates surfaces that are C 2 across the ordinary triangle/quad
boundary, which we prove in Section 3. Furthermore, these changes also extend the subdivision scheme
to arbitrary surfaces such as nonmanifold surfaces.
   The entire subdivision process is depicted in Figure 6. Starting with an initial shape, we first unzip
the surface into disjoint pieces, consisting of entirely triangles or entirely quads, by applying the masks
in Figure 5. Next, we perform linear subdivision on the separate pieces. Finally, we close the surface
back together by performing averaging, which completes one round of subdivision.
   Figure 7 illustrates a more complicated example of a mouse brain subdivided using this method. This
model is actually composed of a network of surfaces that partition the brain into different anatomical
regions. Therefore, edges where multiple anatomical regions meet are nonmanifold. The surface itself is
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
32      •     S. Schaefer and J. Warren




Fig. 7. Initial tri/quad surface of a mouse brain and the subdivided model (top). In the bottom, the cerrebellum was extracted
and subdivided with crease edges highlighted on the surface.




Fig. 8. Coefficients of the Box spline reproducing x 2 (left), xy (middle) and y 2 (right). The boundary vector for the quadrilateral
and triangular side are highlighted.


composed of triangles and quads subdivided using our method. On the bottom of the figure, we extracted
the cerrebellum and subdivided the model with the triangle/quad structure explicitly shown.

3.    ANALYSIS
3.1    Necessary Conditions
Given an ordinary triangle/quad boundary (shown in Figure 8), we define S to be the subdivision matrix
for Stam and Loop’s [2003] scheme, formed by centroid averaging. For the subdivision scheme to be C 2
in the functional sense, S must satisfy

                                                              Sz i = λi z i                                                     (1)

where λi = 1, 1 , 1 , 1 , 1 , 1 and z i are the corresponding eigenvectors producing the polynomials 1, x, y,
                 2 2 4 4 4
x 2 , xy, y 2 [Warren and Weimer 2001]. While S satisfies equation 1 for i = 1 . . . 5, S fails that condition
for z 6 (corresponding to y 2 ). Figure 8 shows the coefficients of the vertices that reproduce the quadratic
polynomials x 2 , xy, and y 2 over the triangular and quadrilateral portions of the mesh. Notice that the
coefficients for y 2 do not agree at the boundary, so S cannot possibly be C 2 at the boundary.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                 On C 2 Triangle/Quad Subdivision             •      33

                                                         ˆ            ˆ
  Our goal is to construct a new subdivision scheme S such that S satisfies equation 1 for i = 1 . . . 5,
           ˆ
and that Sv = 1 v where v is a new eigenvector corresponding to y 2 . To analyze the case of y 2 further,
                4
let vt be the coefficients that reproduce y 2 on the triangle vertices and the boundary, but zero on the
quadrilateral vertices. Similarly, let vq be the coefficients reproducing y 2 on the quadrilateral vertices
and the boundary, but zero on the triangle vertices. We define our new eigenvector v to be of the form
                                     
                                     
                                             vt         triangle vertices
                                v = αvt + (1 − α)vq boundary                                           (2)
                                     
                                             vq         quad vertices

We now construct unzipping matrices Ut and Uq such that

                                        vt triangle vertices and boundary
                             Ut v =     0 quad vertices

                                        0 triangle vertices
                             Uq v =
                                        vq boundary and quad vertices

Using Equation 2, we solve for the unzipping matrices as
                                    
                                    
                                              1         triangle vertices
                                         1−α 11+α 1−α
                             Ut =      (    , 12 , 24 ) boundary
                                     24
                                              0         quad vertices
                                    
                                    
                                             0        triangle vertices
                                         −α 12+α −α
                             Uq =      ( ,       , ) boundary
                                     24 12 24
                                             1        quad vertices

Notice that Levin and Levin’s [2003] choice for the unzipping mask corresponds to α = 0. Levin and
Levin’s special boundary rules were then chosen to satisfy equation 1 for that particular choice of α.
  With these definitions we now partition S into the form S = St + Sq where St and Sq are formed by
centroid averaging on the triangle and quad portions of the mesh, respectively. St and Sq then satisfy
                                           1
                                            4 vt
                                                   triangle vertices
                                             1
                                   St vt =     vt   boundary
                                           8
                                            0      quad vertices
                                           
                                            0
                                                   triangle vertices
                                   Sq vq =   1
                                               vq   boundary
                                           8
                                           1
                                               v
                                             4 q
                                                    quad vertices

Notice that, on the boundary, St and Sq produce 1 vt and 1 vq , respectively, because the subdivision
                                                  8       8
matrices contain halfmasks (shown in Figure 9), formed from centroid averaging.
                          ˆ
 Our subdivision scheme S is then represented as

                                           ˆ
                                           S = St Ut + Sq Uq .
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
34      •     S. Schaefer and J. Warren




Fig. 9. Mask calculated through centroid averaging on each side of the triangle/quad boundary. The mask is exactly half of the
regular mask for triangular or quadrilateral surfaces.

         ˆ
Applying S to v yields
                                            ˆ
                                            Sv = St Ut v + Sq Uq v
                                               = t vt + Sq vq
                                                 S
                                                         1
                                                          v
                                                         4 t
                                                                triangle vertices
                                               =    1
                                                      vt + 1 vq boundary
                                                 8 1 8
                                                        4 q
                                                           v    quad vertices
The final piecewise definition corresponds to exactly 1 v. Since Ut and Uq do not modify the boundary
                                                             4
                                                         ˆ
for the eigenvectors z i for i = 1 . . . 5 and Sv = 1 v, S satisfies the necessary conditions for C 2 continuity
                                               ˆ
                                                    4
at the boundary.
  Finally, our scheme has a simple interpretation in terms of averaging; however, any choice of α > 0
yields an unzipping mask with negative coefficients. Therefore, our choice of subdivision mask does not
yield surfaces that necessarily lie within the convex hull of the control points. Levin and Levin [2003]
chose their unzipping mask and subsequent subdivision rules on the boundary such that their surfaces
satisfy this convex hull property.
3.2    Sufficient Conditions
To analyze the smoothness of the subdivision scheme that we present, we use a sufficient test described
by Levin and Levin [2003]. This smoothness test requires that the subdivision scheme is C 2 away from
the boundary edge, and that the subdivision matrix for a point on the boundary satisfies the necessary
conditions from Section 3.1. Furthermore, the subdivision scheme along the edge must satisfy a joint
spectral radius condition.
  To perform the joint spectral radius test, we require two subdivision matrices (A and B) that map an
edge L on the boundary to two smaller edges (L1 and L2 ) after one round of subdivision. The matrices
A and B should contain all of the vertices that influence the surface over the edges L1 and L2 . Next,
we find a diagonalizing matrix W such that
                                                                              C0
                                                        W −1 AW =
                                                                            0 Y0
                                                                                                                          (3)
                                                              −1            θ C1
                                                        W          BW =
                                                                            0 Y1
where is a diagonal matrix with the entries 1, 1 , 1 , 1 , 1 , 1 , and θ is an upper-triangular matrix with
                                                2 2 4 4 4
the same diagonal entries as . Finally, we use Y 0 and Y 1 to compute
                                                                                        1
                               ρ [k] (Y 0 , Y 1 ) = (Max Y k Y      k−1
                                                                          ...Y   1   ∞) k   where   i   ∈ {0, 1}.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                            On C 2 Triangle/Quad Subdivision            •      35




Fig. 10. Curvature plots of the finely subdivided shape from Figure 6 for Stam and Loop’s [2003] scheme (bottom left) and our
modified scheme (bottom right).




                  Fig. 11. Characteristic maps for several different configurations of quads and triangles.


According to Levin and Levin [2003], if there exists a k such that ρ [k] < 1 , then the subdivision scheme
                                                                            4
is C 2 at the boundary.
   The obvious choice for constructing the matrix W is to simply use all of the eigenvectors of A. However,
this approach can be numerically unstable if the matrix has small eigenvalues. Levin and Levin [2003]
suggest that W be formed from the right eigenvectors associated with the eigenvalues from and a
basis of the null space from the corresponding left eigenvectors. Since symbolic math packages such as
Mathematica can generate the eigenvectors corresponding to exactly, this method yields a numerically
stable method for creating W .
   When applying the spectral radius technique to our subdivision scheme, we calculated ρ [8] = 0.229725.
Since ρ [8] < 1 and our scheme satisfies the necessary conditions for polynomial generation, we conclude
              4
that our subdivision scheme is C 2 at triangle/quad boundaries. Figure 10 shows a curvature plot of a
highly subdivided model from Figure 6. Notice the color discontinuity at the triangle/quad boundary
in Stam and Loop’s [2003] scheme, where as our modification generates continuous curvature at the
boundary.

3.3   Extraordinary Vertices
Notice that this unzipping pass changes the subdivision rules at extraordinary vertices contained by
both triangles and quads from the original triangle/quad subdivision scheme. Therefore, the smooth-
ness results of the original scheme do not apply at these vertices and must be established separately.
We provide no formal proof of smoothness here. However, we have enumerated through all possible
configurations of quads and triangles for vertices of valence 3 through 10 and checked the eigenvalues
as well as the characteristic map for these vertices. The eigenvalues all satisfy the necessary condition
from Zorin [2000] for the scheme to be C 1 . The characteristic maps computed at these vertices also
appear to be regular and injective. Several of these maps are shown in Figure 11, but we offer no proof
at arbitrary valences. We have also inspected surfaces in these configurations and they appear visually
smooth as well.
                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
36      •     S. Schaefer and J. Warren

4.   CONCLUSION
We have presented a subdivision scheme for mixed triangle/quad surfaces that is C 2 everywhere except
for isolated, extraordinary vertices. The subdivision scheme itself is the same as Stam and Loop’s
[2003] triangle/quad scheme except that we perform an unzipping pass before subdivision. Our choice
of the unzipping mask does not yield special rules in the implementation and lends itself to real-world
applications as the method is easy to code since it is a simple extension of ordinary triangle/quad
subdivision. Furthermore, our subdivision scheme provides rules for handling arbitrary triangle/quad
surfaces including nonmanifold surfaces. Finally, we applied Levin and Levin’s [2003] sufficiency test
for C 2 smoothness to prove our modification to the subdivision produces C 2 surfaces at the triangle/quad
boundary.

REFERENCES
CATMULL, E. AND CLARK, J. 1978. Recursively generated b-spline surfaces on arbitrary topological meshes. Comput. Aided
  Design 10, 350–355.
LEVIN, A. AND LEVIN, D. 2003. Analysis of quasi uniform subdivision. Applied Computat. Harmon. Analy. 15, 1, 18–32.
LOOP, C. 1987. Smooth subdivision surfaces based on triangles. University of Utah, Department of Mathematics, Masters
  Thesis.
STAM, J. 2001. On subdivision schemes generalizing uniform b-spline surfaces of arbitrary degree. Comput. Aided Geomet.
  Design 18, 383–396.
STAM, J. AND LOOP, C. 2003. Quad/triangle subdivision. Comput. Graph. For. 22, 1, 1–7.
WARREN, J. AND SCHAEFER, S. 2003. A factored approach to subdivision surfaces. Submitted to Comput. Graph. Applicat.
WARREN, J. AND WEIMER, H. 2001. Subdivision Methods for Geometric Design. Morgan Kaufmann.
ZORIN, D. 2000. Smoothness of subdivision on irregular meshes. Construt. Approx. 16, 3, 359–397.
                    ¨
ZORIN, D. AND SCHRODER, P. 2001. A unified framework for primal/dual quadrilateral subdivision schemes. Comput. Aided
  Geomet. Design 18, 5, 429–454.


Received September 2003; revised May 2004; accepted August 2004




ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
Simulating the Dynamics of Auroral Phenomena
GLADIMIR V. G. BARANOSKI and JUSTIN WAN
University of Waterloo
JON G. ROKNE
University of Calgary
and
IAN BELL
University of Waterloo


Simulating natural phenomena has always been a focal point for computer graphics research. Its importance goes beyond the
production of appealing presentations, since research in this area can contribute to the scientific understanding of complex
natural processes. The natural phenomena, known as the Aurora Borealis and Aurora Australis, are geomagnetic phenomena
of impressive visual characteristics and remarkable scientific interest. Aurorae present a complex behavior that arises from
interactions between plasma (hot, ionized gases composed of ions, electrons, and neutral atoms) and Earth’s electromagnetic
fields. Previous work on the visual simulation of auroral phenomena have focused on static physical models of their shape, modeled
from primitives, like sine waves. In this article, we focus on the dynamic behavior of the aurora, and we present a physically-based
model to perform 3D visual simulations. The model takes into account the physical parameters and processes directly associated
with plasma flow, and can be extended to simulate the dynamics of other plasma phenomena as well as astrophysical phenomena.
The partial differential equations associated with these processes are solved using a complete multigrid implementation of the
electromagnetic interactions, leading to a simulation of the shape and motion of the auroral displays. In order to illustrate the
applicability of our model, we provide simulation sequences rendered using a distributed forward mapping approach.
Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimension Graphics and Realism
General Terms: Algorithms
Additional Key Words and Phrases: Atmospheric effects, natural phenomena, plasma phenomena, rendering




1.   INTRODUCTION
The Aurora Borealis, or “Northern Lights”, and Aurora Australis, its southern counterpart, are light
emissions caused by the collision of an ionized stream of charged particles with high altitude atmo-
spheric atoms and molecules [Eather 1980]. These emissions are considered to be among the most
fascinating and mysterious of Nature’s spectacles by those who have been fortunate enough to see
them. Taylor in National Geographic calls them “Earth’s Grand Show of Lights” [Taylor 2001]. An
inherent characteristic of auroral displays is that they move and can change shape both slowly and

Funding is gratefully acknowledged from both the National Sciences and Engineering Council of Canada (NSERC Grants 213281,
239162, and 691266) and the Canada Foundation for Innovation (CFI Project 6218).
Author’s address: G. V. G. Baranoski, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo,
200 University Avenue West, Waterloo, ON N2L 3G1, Canada.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0037 $5.00
                                                                ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 37–59.
38      •     G. V. G. Baranoski et al.

rapidly. These complex and stochastic motions are directly related to the plasma processes. In this
article, a novel physically-based model to perform 3D visual simulations of the dynamics of the auroral
phenomena is presented. Furthermore, auroral surges (Figure 14), which have not been previously
simulated, are also rendered. This model, therefore, enables the simulation of shape and motion of the
aurorae in a realistic manner.
   The aurorae are frequently visible at high latitudes. They can also be seen less frequently at lower
latitudes, especially around solar sunspot maxima (a period of great solar turbulence that occurs ap-
proximately every 11 years [Odenwald 2000]). For instance, the last solar cycle of the 20th century
reached its maximum in the year 2000, and it produced amazingly bright aurorae for the next couple
of years. Aurorae are also a prominent visual phenomena in space as seen from spacecrafts orbiting
the Earth. Further afar, they occur on planets with strong magnetic fields such as Jupiter and Saturn.
Hence, the simulation of auroral displays is also of interest for the production of night sky visualiza-
tion sequences used in planetarium shows. Moreover, the algorithms used in our plasma-model may be
applied to the simulation of other plasma-related phenomena, such as gas discharges encountered in
neon and fluorescent lights, welding arcs, and lightning.
   On the scientific side, the importance of the visual simulation of auroral dynamics is related to the
links to plasma physics. As much as 99% of the matter in the universe is thought to exist in a plasma
state [Bittencourt 1986; Chen 1984; NRC 1995]. The auroral displays provide a natural laboratory in
which the complex workings of the plasma universe can be studied. Therefore, our simulations can be
used in the evaluation of auroral theories as well as in the visualization of other plasma phenomena.
Furthermore, plasma kinetic theory can be used to study the development of galaxies, since, although
stars in a galaxy are not charged, they behave like particles in a plasma [Bittencourt 1986; Chen 1984;
NRC 1995]. Another area of application might be the study of the effects of nuclear explosions in the
upper atmosphere. These explosions were found to create aurora-like displays [Dupont 2004].
   Yet the main motivation for this research is the spellbinding majesty of a moving auroral display
[Taylor 2001].

1.1    Related Work
The spectacular auroral motions and shape changes involve to some extent shear, meaning differential
motions within different parts of a fluid or gaseous material. These motions are common in nature.
   Shear eddies occur, for instance, on the surface of water in streams and in smoke curling up from a
fire, and they are responsible for clear-air turbulence and gusty winds [Davis 1992]. The shear found
in the aurorae is associated with plasma processes, and the most fascinating consequences are the
formation of rotational distortions having different spatio-temporal scales.
   Blinn and Wolf [Upson et al. 1987] produced a film in 1983 entitled “Jupiter’s Magnetosphere: The
Movie”. In this film, representations of spacecraft observations as well as analytic models of the plasma
and magnetic fields in the Jovian system were employed to visualize the morphology and dynamical
structure of Jupiter’s magnetosphere. To the best of our knowledge, this film represents the first use
of a plasma model by the graphics community. Recent works involving plasma models in the graphics
literature focus on the visualization of plasma phenomena occurring in laboratory experiments. For ex-
ample, Parker et al. [1995] developed a gyrokinetic particle simulation to visualize a plasma turbulence
in a Tokomak (an apparatus for producing controlled fusion reaction in hot plasma), and Ljung et al.
[2000] presented a system for the animation of the evolution of an instability that gives rise to a plasma
surfatron, a mechanism for particle acceleration of importance in particle accelerators, in the context of
electron acceleration in astrophysical shocks and in the solar corona. It is relevant to note the work of
Schussman et al. [2000] who presented techniques for the visualization of magnetic field data from the
DIII-D Tokomak, a fusion facility that conducts both experimental and computational plasma research.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                   Simulating the Dynamics of Auroral Phenomena               •      39

   Recently, Baranoski et al. [2000, 2003] presented a static physically-based algorithm for the render-
ing of auroral displays which accounts for important auroral visual features such as their characteristic
spectral and intensity variation. Their work, however, does not take into account the electromagnetic in-
stabilities responsible for the time varying behavior of the aurorae. Nonetheless, it provides an accurate
framework for the rendering of auroral images. In this article, the dynamical behavior of the aurorae
is modeled by taking into account physical parameters and processes directly associated with plasma
flow. The set of partial differential equations (PDEs) associated with these processes is solved using a
complete multigrid implementation of the electromagnetic interactions between charged particles in
the ionosphere, enabling us to produce a visual display of the “dancing lights”.
   The simulation of natural phenomena involving shear and fluid flow is an active area of research
in computer graphics [Foster and Metaxas 1997; Foster and Fedkiw 2001; Kajiya and Herzen 1984;
Myazaki et al. 2001; Stam and Fiume 1993; Stam 1999]. A comprehensive review of computer graph-
ics simulations of fluid flow is beyond the scope of this article; however, we should remark that the
plasma instability responsible for the stochastic and complex nature of the aurorae is similar to the
Kelvin-Helmholtz instability (KHI) for sheared fluids [Tajima 1989]. In spite of that, we believe that a
more natural way to model plasma phenomena is to simulate the equations of plasma flow directly. The
use of a plasma model allows the control of the simulations by auroral physically meaningful parame-
ters. Moreover, the use of a plasma model enables the accurate reproduction of the rapid intermediate
scale motions often seen in the aurorae such as the motion of rays and folds along auroral forms.
   Recent works on night sky phenomena have focused on obtaining images that are not only realistic,
but are also predictive. This objective is achieved by adhering to physical principles and incorporating
measured data to the simulations. After all, it is far easier to control a simulation using familiar phys-
ical concepts than through arcane parameters as appropriately stated by Arvo [1995]. As illustrative
examples of this trend, besides the work by Baranoski et al. [2000, 2003] mentioned above, we should
note the works by Nadeau et al. [2001] and Jensen et al. [2001].
   Nadeau et al. [2001] presented star and nebula visualization techniques for creating a 3D volu-
metric representation of the Orion Nebula, which was used in a planetarium daily show. Jensen
et al. [2001] developed a physically-based model of the night sky which also has educational ap-
plications. Although the Jensen et al. model does not include auroral displays, their night sky im-
ages can be combined with our auroral simulations. They also highlighted the role of naturally il-
luminated night scenes in the history of image-making and their importance in the area of realistic
rendering.
   Simulating plasma instabilities as well as fluid instabilities involves the solution of systems of PDEs.
In these situations, the production of simulation sequences used to depict time dependencies or to
analyze motion requires that such systems be solved repeatedly. Usually standard numerical methods,
such as the Conjugate Gradient (CG) and Fast Fourier Transform (FFT)[Golub and Loan 1989], are
applied. Stam [1995, 1999] has highlighted the theoretical advantages of multigrid methods [Hackbusch
1985] for solving such systems. In order to solve the PDEs arising from our plasma simulations, we
present a practical and fast multigrid implementation. We note that our multigrid implementation
is matrix-free, that is, it does not store the coarse grid matrices. Previous multigrid implementations
found in the computer graphics literature, such as the one by Witkin and Kass [1991] in the their texture
synthesis algorithm, used coarse grids to speed up the computations of explicit steady state solutions.
Here, we use multigrid in the context of time dependent problems, and in each implicit time step, the
multigrid is applied to solve a linear system. It also worth noting that Weimer and Warren [1999] used
a full multigrid approach for fluid flow which is appropriate for solving nonlinear equations. In our
implementation, we used a V-cycle multigrid which is more appropriate for solving linear equations
such as those arising in our simulations.
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
40      •     G. V. G. Baranoski et al.




Fig. 1. Satellite snapshot of an auroral oval around the Earth’s north magnetic pole. a) Viewed from the satellite position and
b) viewed from a position normal to the Earth’s north pole. (Courtesy of NASA Polar Ultraviolet Imager.)

1.2    Organization
The next section provides an overview of relevant auroral physics concepts and describes the main
motions and shape changes of the aurorae. In Section 3, the modeling framework used in this research
is outlined. Section 4 presents the algorithms used to simulate the dynamics of the auroral phenomena.
The rendering issues are addressed in Section 5. Section 6 presents the results of our simulations and
discusses performance issues. Section 7 outlines applications and directions for future work. Finally,
Section 8 closes the article.

2.    AURORAL PHENOMENA
In this section, we outline the main physical aspects directly related to the simulation of auroral dynam-
ics. A reader interested in a detailed description of auroral physics is referred to more comprehensive
texts on these phenomena [Brekke and Egeland 1994; Davis 1992; Eather 1980; Jones 1974; Omholt
1971]. The diagrams of auroral distortions presented in this section use as reference an auroral local co-
ordinate system represented by xyz, where the z-axis corresponds to the direction parallel to the Earth’s
magnetic field vector B, in the upwards direction, but not necessarily perpendicular to the ground.

2.1    Overview
The particles responsible for the colorful auroral displays are electrons that originate from solar flares
and coronal mass ejections and become the “solar wind” [Burtnyk 2000]. After interacting with the
Earth’s magnetosphere, these particles migrate along the magnetic field lines, and “dive” towards oval
regions around the Earth’s magnetic poles under certain conditions. These regions are called the auroral
ovals. Figure 1 shows a satellite snapshot of an aurora oval around the Earth’s north magnetic pole.
  The shape of an aurora is determined by the energy and density of the electrons entering the at-
mosphere as well as the local variations in the Earth’s magnetic and electric fields. The most common
aurorae can be thought of as “curtains” of light emissions from “sheets” of falling electrons, which move
along the Earth’s magnetic field.
  As the electrons travel down along the Earth’s magnetic field lines, they suffer many random de-
flections which are caused by collisions with atoms of atmospheric gases such as oxygen and nitrogen.
These deflections spread the electrons out horizontally. When the electrons collide with atmospheric
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                 Simulating the Dynamics of Auroral Phenomena                 •      41




Fig. 2. Line drawings of cross sections of auroral arcs (not to scale) illustrating the three major shear-type rotational distortions:
a) curls, b) folds, and c) spirals. The symbol denotes a vector normal to the plane of the paper.

constituents, the atoms are excited, and after a period of time, they may emit a photon. Statistically,
several collisions must occur before a photon is emitted. As a result of these photon emissions, the
curtains may be colored, bright yellow-green at the bottom, perhaps red at the top, and a yellow-orange
transition may also be present. Often they form arcs that can extend over 2000km whose width (thick-
ness) may vary from several hundred meters to several kilometers. These arcs can bend and fold, almost
like real curtains, thereby generating bright vertical streaks.
2.2   Auroral Shear-Type Distortions
The sheets of precipitating electrons form an auroral stream that is subject to instabilities responsible
for the shear-type rotational distortions seen in auroral arcs. Hallinan and Davis [1970] identified the
three distinct types of distortions: curls, folds and spirals (Figure 2). Curls are small scale and have a
diameter of 1km or less. Folds are intermediate scale distortions and have a diameter of approximately
20km or less. Spirals, or surges, are the largest auroral distortions and diameters are typically around
20–1300km [Partamies et al. 2001]. In addition to the different spatial scales, these phenomena have
different temporal scales. Curls have a lifetime of a fraction of a second, while folds can exist for more
than a second, and spirals may last minutes.
   Curls and folds are responsible for distinct visual features present in auroral displays. The light
emanating from convoluted folds in auroral displays often creates the impression of vertical, or near-
vertical, “stripes” [Bryant 1999]. Despite their highly transient nature, curls are largely responsible for
another important auroral feature, namely, electron beams evolving into thin field-aligned filaments or
“rays”. An auroral ray typically has a vertical dimension up to several hundred kilometers. Curls are
usually 2–10km apart, while folds are approximately 20km apart. Spirals are also relatively common
auroral structures, visible in more than 30% of the Defense Meteorological Satellite Program observa-
tions1 [Davis and Hallinan 1976]. They can form a street of two or more spirals approximately 100km
apart. The spirals in the street usually have similar configuration and size, but they may also grow
monotonically along the auroral display [Davis and Hallinan 1976].
2.3   Auroral Motions
As pointed out by Davis [1992], auroral motions and shape changes seem bewildering, yet they present
some order when examined in detail. Many of the motions and shape changes associated with the
auroral distortions are caused by warping of the incoming particle streams by the attendant magnetic

1 http://www.ngdc.noaa.gov/dmsp/dmsp.html.

                                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
42      •     G. V. G. Baranoski et al.




 Fig. 3. Line drawings illustrating an auroral arc break-up. The symbol          denotes a vector normal to the plane of the paper.




Fig. 4. Sketch showing different stages of an auroral warping process: a) initial arc, b) fold, c) further distorted fold, and d) spiral
array. The symbol denotes a vector normal to the plane of the paper.


and electric fields. In this section, we outline the most important of these changes according to their
spatial scale.

  2.3.1 Small Scale. When an electron particle stream responsible for producing a small scale struc-
ture becomes slightly more dense in a region, it is likely to become contorted and develop a curl [Davis
1992]. The curl formation process is similar to the spiral formation process and leads to essentially the
same final configuration. However, the entire curl formation can occur so rapidly (0.25–0.75s [Trondsen
1998]) that an observer on the ground has difficulty following the process.

   2.3.2 Intermediate Scale. Once a curl vortex develops, it tends to cause the formation of another
curl a short distance along the arc. The net result is the development of an array of evenly spaced
curls along the arc, manifesting as rays. If another arc is nearby, the rays come under the influence
of its electrical field and, therefore, move along the arc [Davis 1992]. If the process that leads to curl
formation continues a bit longer, then small, uniformly spaced folds occur, and sometimes the arc may
split into two separate arcs as the folds break apart (Figure 3).
   Folds have apparent horizontal velocities in the range of 0–5km/s [Hallinan and Davis 1970]. The
apparent horizontal velocities of curls lie in the range of 0–90km/s, with preferred speeds in the range
of 0–8km/s [Trondsen 1998]. To have a better idea of the magnitude of these speeds, recall that an
auroral arc may extend over the entire field-of-view of an observer on the ground. For this observer
the auroral rays seem to “travel” back and forth between two extreme points in the horizon in-few
seconds.

   2.3.3 Large Scale. When an incoming stream of electrons that produces an auroral arc develops a
slight irregularity, the irregularity can grow [Hallinan 1976, 1981; Levy and Hockney 1968; Partamies
et al. 2001]. As it does, the stream becomes distorted, carrying incoming particles to new locations
in the auroral domain [Davis 1992]. The growth of this irregularity depends on the thickness of the
electron stream. The larger the stream length-to-thickness ratio, the faster the irregularity grows with
the distance [Hallinan 1976; Levy and Hockney 1968]. Figure 4 shows an auroral arc being affected by
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                   Simulating the Dynamics of Auroral Phenomena               •      43

the warping process, first developing a fold, and then progressing into a complex spiral shape. These
large auroral vortices have an average speed of 4km/s [Partamies et al. 2001].

3.   MODELING FRAMEWORK OVERVIEW
Computer models of plasma can be broadly divided in three kinds: Eulerian, Lagrangian and hybrids
of these [Hockney 1966]. The Eulerian model assumes that plasma behaves like a fluid, or that it obeys
macroscopic differential equations for average quantities. The Lagrangian model follows the motion
of a large number of particles as they move in time under their mutual electromagnetic interactions.
Although this model acts on the microscopic particle level, it allows the particle ensemble to “make
its own mind” about the macroscopic and collective behavior. This aspect is relevant in the simulation
of phenomena with noticeable stochastic traits and different spatio-temporal scales such as aurorae.
We use a hybrid Eulerian-Lagrangian approach in our modeling of auroral phenomena. Such a hybrid
approach has been successfully used in computational physics simulations for many years [Hockney
1966; Levy and Hockney 1968; Hockney and Eastwood 1988].
   In many scientific fields, the behavior of natural phenomena is described using PDEs. Recently,
PDE-based modeling and numerical techniques from computational fluid dynamics (CFD) have been
applied to computer graphics, especially for animations involving fluids. For instance, in Fedkiw et al.
[2001], smoke was modeled by inviscid Euler equations. Level set and semi-Lagrangian techniques were
applied in the numerical computation. In Rasmussen et al. [2003], large scale smoke phenomena were
modeled by 2D incompressible Euler equations, together with the Kolmogorov spectrum. In Kunimatsu
et al. [2001], water was modeled by Navier–Stokes equations. The Eulerian approach was adopted and
the volume-of-fluid method was used to track free water surfaces. Another approach is to use semi-
Lagrangian and level set method with inertialess particles [Foster and Fedkiw 2001]. The particle
level set technique was further modified in Enright et al. [2002] to obtain more accurate results. In
Nguyen et al. [2002] and Feldman et al. [2003], fire and explosion were modeled by incompressible Euler
equations. The latter used nonzero velocity divergence to capture fluid expansion during explosion.
Cloud simulation was modeled by Boussinesq’s equations in Miyazaki et al. [2002].
   At first glance, the similarities between neutral fluids and plasma [Tang 2002] might suggest that the
CFD formulation would be sufficient to model the behavior of both media. However, in order to perform
predictive simulations of the dynamics of plasma phenomena, one must account for their distinctive
features. For example, turbulence dynamics in plasmas involve electromagnetic fields and a much larger
number of relevant variables than for neutral fluids [Yoshizawa et al. 2001]. Furthermore, while the
effects of global boundary conditions, such as wall boundaries, often play an important role in the growth
of neutral fluid disturbances [Drazin and Reid 1981; Hinze 1975; Yoshizawa 1998], plasma turbulence
is greatly affected by spatial inhomogeneities and plasma configurations. These inhomogeneities are
coupled together to drive or suppress turbulent fluctuations, which often have a very long correlation
length along the magnetic field line and are quasi-two-dimensional [Yoshizawa et al. 2001]. Hence, to
predictively simulate structure formation and transitions in plasma phenomena, such as the aurorae,
one needs to account for the mutual interactions between plasma inhomogeneities and the electric-field
structure and fluctuations.
   Although the physical laws that affect the behavior of plasma phenomena, such as the Maxwell’s
equations, are well known, progress in this area was only made possible by advances in computational
methods [Gombosi et al. 2004; Tang 2002]. Generally, an analytical, closed-form solution for these
equations is difficult to obtain. One can only obtain a unique solution by imposing various restrictions
[Volakis and Kempel 1995], and they are normally solved using numerical methods such as finite
difference (FD) [Taflove 2000], finite volume (FV) [Anderson 2001], and finite element (FE) [Silvester
and Ferrari 1990] methods and hybrids of these [Volakis and Kempel 1995]. Among these methods,
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
44      •     G. V. G. Baranoski et al.

FD’s are the most commonly applied to the PDE formulation of Maxwell’s equations in the time domain
[Anderson 2001]. This method is based on the straight discretization of the differential equations, and
                                                ¨    ¨
it is suitable for modeling complex media [Karkkainen 2002]. This motivated its use in the simulation
algorithms described in this article. In order to accelerate the convergence of the iterative solutions of the
PDEs, we used a multigrid method. Recently, Aruliah and Asher [2002] also used a multigrid scheme to
speed-up the simulation of three-dimensional electromagnetic problems in geophysical regimes. While
in our simulations multigrid is used to solve the Maxwell’s potential equation directly, Aruliah and
Asher [2002] used a multigrid scheme to approximately invert a block diagonal matrix, which, in turn,
was used as a preconditioner for the discretization matrix.
   The “electrons” in a plasma simulation correspond to millions of physical electrons. Provided that the
physical phenomena have wavelengths that are long, compared with the average electrons spacings,
and time scales that are short, compared with the time for the graininess to have significant effect, this
description of “electrons” gives an accurate representation and leads to the concept of computational
“superparticle” [Hockney and Eastwood 1988]. The graininess introduced by the paucity of superpar-
ticles is minimized by smoothing the short-range forces. One way of regarding the superparticles is as
finite-sized clouds of electrons, where the position of a superparticle is the center of mass of the clouds
and their velocities the mean velocities of the clouds [Hockney and Eastwood 1988].
   We used a similar representation in our simulations in order to minimize computational overhead.
Instead of tracking the around 1012 individual plasma particles that cause auroral phenomena, we
follow the path of superparticles which we regard as beams of electrons. As such, we use normalized
values for the physical quantities involved in our simulations, namely, charge, mass, and permissivity.
The Lagrangian treatment is used to account for the interactions among beams and between beams
and atmospheric constituents. The Eulerian treatment is applied when spatial scales much greater
than the relatively small amplitude motions of the beams are considered.
   The usual approach in computer experiments is to devise a model which is sufficiently detailed to
reproduce the important physical aspects faithfully and yet not so detailed that the calculations become
impractical. This approach becomes even more pertinent when the main purpose is to produce visual
simulations whose purpose is to highlight the qualitative features of a given phenomenon. Selecting
the “best” modeling approach depends on the relevant physical length and time scales.
   The motion of the electron beams has two components: parallel and perpendicular to the electron
stream. The parallel component is associated with the kinetic energy of the incoming particles. The
perpendicular component is due to the warping of the electron stream. The differences of the magnitudes
of the time scales of these two components [Hallinan 1981; Brekke and Egeland 1994] means that they
can be decoupled without incurring a loss of realism in their visual three dimensional representation.
   The key stage of our modeling framework is the simulation of the warping process, which changes the
perpendicular velocity, v⊥ , and the position of electron beams, P , at the top of the electron stream, at
each instance of time. The mathematics of this process can be described by an electro-magnetic version
of the Kelvin-Helmholtz instability (KHI) [Hallinan 1976, 1981; Levy and Hockney 1968; Partamies
et al. 2001]. Section 4 describes the warping algorithm used in our simulations in detail.
   The position of each beam at the top of the electron stream is considered as the starting point of
its descending trajectory. The position update resulting from the warping process is associated with
a steady state of incoming particles. A precipitating electron beam is tracked downwards until it be-
comes indistinguishable from the ambient electron population. During their descending trajectories,
the electron beams may be deflected several times due to collisions with atmospheric constituents.
Light emissions may occur at these deflection points, causing the auroral displays. These emissions are
mapped to the screen plane of a virtual camera using a splatting approach [Westover 1991]. Rendering
issues are further examined in Section 5.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                         Simulating the Dynamics of Auroral Phenomena                  •      45

 The auroral dynamics simulation uses the auroral local coordinate system defined in Section 2. The
mapping of auroral emissions for rendering purposes uses a global coordinate system represented by
XYZ, where the Y-axis corresponds to the direction normal to the ground.

4.    DYNAMICS SIMULATION
The electron beams interact with one another through the electric field, E, generated by their mutual
electrostatic repulsion, and the Earth’s magnetic field, B. Thus, the Lorentz force on an electron beam
ek with charge qk , moving with a perpendicular velocity v⊥ is
                                                          k

                                          F k = qk ( E + v⊥ × B),
                                                          k                                                                  (1)
and the motion of an electron beam of mass mk is determined by Newton’s law
                                                   d v⊥
                                              mk      k
                                                        = F k.                                                               (2)
                                                    dt
  Thus, to update the positions of the electron beams, we need to compute E and B. In the auroral
warping simulation, the Earth’s magnetic field is modeled as a uniform magnetic field of strength | B|.
At any instance of time, the electron beams and, hence E, are located in the plane perpendicular to B.
By Gauss’ law, E satisfies
                                                             ρ
                                                ∇·E=              ,                                                          (3)
                                                             0

where   0   is the permissivity of free space, and ρ is the charge density function
                                                   N
                                         ρ(P ) =         qk δ(P − Pk ).                                                      (4)
                                                   k=1

Here we model the small mass, low density electron beams with charge qk , at the position Pk , as point
particles via the Dirac delta function δ(·). From the Maxwell equations, which describe the dynamics of
charged particles interacting with an electromagnetic field, we obtain

                                                ∇ × E = 0.                                                                   (5)
The electrostatic potential φ is introduced such that

                                                E = −∇φ,                                                                     (6)
and the divergence of the potential gradient results in the Poisson Equation given by
                                                             ρ
                                               −∇ 2 φ =           ,                                                          (7)
                                                              0

where ∇ 2 φ is the Laplacian of the potential. By solving Equations (2) and (7) over time, we can model
the perpendicular motion of the electron beams, which is the crucial aspect of the auroral dynamics.

4.1   Warping Algorithm
The potential Equation (7) and the electron beam velocity Equations (2) form a coupled continuous-
particle system, which is simulated using the particle-mesh method exploiting the force-at-a-point
formulation and the field equation for the potential. For the velocity equation, we use a Lagrangian
grid approach, whereas for the potential equation, we use an Eulerian approach as suggested by Levy
                                                                      ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
46      •     G. V. G. Baranoski et al.




Fig. 5. A cross-section of an auroral stream, modeled by a strip of electron beams with periodic and Neuman boundary conditions,
along the x-axis and y-axis, respectively.

and Hockney [1968]. The perpendicular velocities and positions of the electron beams are advanced in
time as follows:
(1)   from the positions of the electron beams, a charge distribution (ρ) is derived;
(2)   from the charge distribution, using Poisson Equation (7), the electrostatic potential is calculated;
(3)   from the potential, the electric field ( E) acting on each electron beam is derived using Equation (6);
(4)   Newton’s law of motion (Equation (2)), using the local electron field and the Earth’s magnetic field
      ( B) is then integrated resulting in a new position and velocity for each electron beam.
The cycle is then repeated at step (1). The aurora dynamics is initiated from incoming sheets of particles
at the auroral oval (Figure 1) forming an auroral stream, which is modeled by a strip of incoming electron
beams with periodic and Neuman boundary conditions along the x-axis and y-axis respectively. The strip
in Figure 5 is a small cross-section of part of an auroral oval (Figure 1). As an initial condition, the
electron beams are distributed randomly in the interior of this strip of thickness γ and length λ, and
have zero perpendicular velocity. Faster initial growth of auroral distortions will occur for larger values
of the λ/γ ratio.
  4.1.1 Charge Assignment. In the hybrid Eulerian-Lagrangian formulation, the field and particle
quantities are defined on different grids. To obtain the charge density at mesh points from the distri-
bution of particles, we allocate the particle charges based on the CIC (cloud-in-cell) [Birdsall and Fuss
1969] charge assignment scheme: the charge density ρi, j , at the grid point G (i, j ) , is given by the sum of
weighted charges, ωk qk , of the electron beams ek in the cells surrounding the grid point G (i, j ) , divided
by the cell area:
                                              1
                                  ρi, j =                                       ωk qk ,
                                          cell area e ∈cells with vertex(i, j )
                                                                   k

as indicated in Figure 6(a). However, instead of finding which electron beams contribute to the charge
at the grid point G (i, j ) , we sweep through each electron beam and distribute its charge with weights
ωi∗ , j ∗ , i ∗ ∈ {i, i + 1}, j ∗ ∈ { j, j + 1}, to the vertices of the grid cell it belongs to, as indicated in Figure 6(b).
 k

To be consistent with the definition of the continuous charge density (4), the weight ωi, j must be given   k

by
                                                       ωi, j = S (G (i, j ) − Pk ),
                                                        k

where S (·) is a modified Dirac delta function with compact support parameterized by , which has the
property that
                                                              lim S (·) = δ(·),
                                                              →0
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                               Simulating the Dynamics of Auroral Phenomena                   •    47




      Fig. 6. a) Charge accumulation at grid point G (i, j ) . b) Charge distribution of ek to the neighboring grid points.

in the sense of distributions. It can be interpreted physically by ascribing a finite width ( ) to the
distribution of charge in each particle. In the CIC scheme, S is given by the hat-shape function [Birdsall
and Fuss 1969].
  4.1.2 Poisson Equation. The gradient of the potential function satisfies Equation (7). On an Eulerian
grid with mesh size h, the partial differential equation can be discretized as
                                   4φi, j − φi+1, j − φi−1, j − φi, j −1 − φi, j +1   ρi, j
                                                         2
                                                                                    =       ,
                                                        h                               0

which leads to a set of linear equations, denoted by A = ρ, where is a vector of unknown potentials
and ρ is a vector of (scaled) charge densities. The matrix A is large, sparse, symmetric, and positive
definite. The complexity of most matrix solvers depends on the size of A. For the simulation of auroral
dynamics, a fine grid is required to resolve the layer structure, especially when the formation of surges
is simulated. Thus, an efficient linear solver is indispensable.
   Direct methods, such as banded Gaussian elimination (GE), can be used to exploit the banded struc-
ture of A. For matrices of size n × n, the complexity of banded GE is O(n2 ), which is still computation-
ally impractical. Iterative methods, such as preconditioned conjugate gradient (PCG) with incomplete
Cholesky factorization as preconditioner, have been successfully used in computer graphics for fluid
simulations [Fedkiw et al. 2001; Foster and Fedkiw 2001]. While the theoretical complexity is reduced
to O(n1.5 ), it is still expensive for the long time simulations presented in this article, which require fine
grids as stated above. The FFT method [Stam 1999] can improve the complexity further to O(n log2 n),
but still it is not optimal in the sense given below.
  4.1.3 A Practical Multigrid Implementation. The multigrid method is an optimal scheme in which
the convergence rate is independent of the problem size [Trottenber et al. 2001]. Thus, as opposed to
the methods previously discussed, it has a linear complexity of O(n), where the complexity constant is
known to be small for Poisson equations, typically smaller than 10. It exploits the properties of the linear
equations at the PDE level, rather than the matrix structure. It was first proposed by Fedorenko [1961]
in the 1960’s, and made popular by Brandt [1977] and others for scientific applications. We refer the
interested reader to the survey paper by Chan and Wan [2000] on recent developments of linear elliptic
multigrid methods. Early attempts in using multigrid methods to solve a multiple scattering problem
[Stam 1995] raised the issue of memory due to storage of coarse grid quantities. Here, we introduce
a matrix-free implementation which is extremely memory-efficient. We remark that similar multigrid
schemes can also be used in other plasma and astrophysical simulations as mentioned in Section 7.
  The idea of the multigrid method is to accelerate the convergence of the iterations by solving the PDEs
on a sequence of coarse grids. The basic principle is to eliminate high frequency errors by relaxation
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
48      •     G. V. G. Baranoski et al.




        Fig. 7. A multigrid V-cycle: presmoothing, restriction, coarse grid correction, interpolation and post-smoothing.

smoothing, and low frequency errors by coarse grid correction. Let h be the current approximate
solution on the fine grid with mesh size h, and ε h = − h be the error. By the dissipative nature of
the Poisson equation, the error in the numerical solution, ˜ h , obtained from a few relaxation steps, will
become smooth. Thus, one can easily compute an accurate coarse grid approximation by projecting the
residual error, r h = ρ − A ˜ h , onto the coarse grid with mesh size H and solve the coarse grid error
equation
                                                              A H ε H = Rh r h ,
                                                                         H

with less computational effort, since there are fewer grid points on the coarse grid. Here A H is the
coarse grid matrix and Rh the full weighting restriction operator. Finally, the coarse grid error is then
                         H

interpolated back to the fine grid to update the fine grid solution
                                                          new
                                                                 = ˜ h + PH ε H ,
                                                                          h


where P H is the linear interpolation operator. This process completes one multigrid V-cycle (Figure 7).
        h

  We note that only the actions of the coarse grid, interpolation, and restriction operators are needed.
Thus, we never form nor store these matrices. Consequently, only 3.5 fine grid vectors are needed in
our implementation, which requires even less memory than CG. For the interested reader, an iteration
count comparison between our multigrid method and PCG can be found in the original technical report
written by the authors [Baranoski et al. 2002].
   4.1.4 Velocity and Position Update. The electric field Ei, j is defined on the Eulerian grid whose value
is given by numerical differentiation of φi, j . However, electron beams are located on the Lagrangian
grid, which, in general, does not overlap with the Eulerian. Thus, the electric field on an electron beam
ek , located at Pk , is obtained by interpolating the electric fields at the nearest four corners, that is, the
reverse process of charge distribution as in Figure 6(b). We then update the perpendicular velocities
and positions of the electron beams by integrating Equation (2).
4.2    Trajectories of Electron Beams
The electron beam’s velocity vector, v, is defined as the overall direction of progression during its de-
scending motion. As the beams travel down, they may suffer several random deflections. Their trajec-
tories are simulated incrementally using the stochastic algorithm described by Baranoski et al. [2003].
This algorithm takes into account the spatial inhomogeneity of auroral electrons and their initial en-
ergy to compute the deflection points and the displacements of an electron beam along the magnetic
field lines.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                            Simulating the Dynamics of Auroral Phenomena                •      49




Fig. 8. Sketch showing that the apparent surface brightness of an auroral display is proportional to the number of emissions
along the line of sight represented by the vector s.




                     Fig. 9. Diagram showing angular displacements used to distributed emission rays.


     Each path is simulated incrementally, using a parametric displacement u ∈ [0..1] such that

                                                 unew = uold + (d u ξ1 ),

where ξ1 is an uniformly distributed random number in the interval [0..1], and d u corresponds to a
parametric interval (dimensionless) between collisions which is adjusted according to the initial energy
of the incoming particles. For instance, an electron with 10 keV (60000km/s) can collide 300 times
before becoming indistinguishable from the ambient electrons at an altitude of about 100km above the
ground [Brekke and Egeland 1994]. In this case, we could set d u = 300 .
                                                                     1



5.    RENDERING ISSUES
Aurorae are view-dependent phenomena, that is, the apparent surface brightness of an aurora is pro-
portional to the integrated emission-per-unit volume along the line of sight (Figure 8). Hence, a forward
mapping, or splatting, approach can be used to map auroral emissions to the screen plane [Baranoski
et al. 2003; Westover 1991].
   Recall that after being hit by an electron, an atmospheric atom becomes excited, and, after a period
of time, it may emit a photon. Statistically, the intensity contribution spreads radially around the
propagation direction, and follows a Gaussian distribution along that dimension [Borovsky et al. 1991;
Borovsky and Suszcynsky 1993]. In order to account for this phenomenon, we implemented an algorithm
to perform a distributed forward mapping of auroral emissions. The world coordinates for each deflection
point are used to compute the line of sight vector s. This vector is randomly perturbed through angular
displacements α and β (Figure 9) to generate a light emission ray r. The angle α represents the azimuthal
angle around s. The angle β represents the polar angle with respect to s, and it corresponds to the
exponential glow decay observed in auroral arcs [Borovsky and Suszcynsky 1993]. These angles are
                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
50       •    G. V. G. Baranoski et al.

given by
                                                                        γ ξ2
                                                     (α, β) = (1 − e− 240 , 2π ξ3 ),
where ξ2 and ξ3 are uniformly distributed random numbers ∈ [0, 1], and γ represents the initial thick-
ness of the auroral stream.
  The accuracy and performance of the simulation is dependent on the number, N , of emission rays.
Experiments show, however, asymptotic convergence of the results for larger values of N .
  The emissions mapped to the screen plane are scaled according to auroral spectral emission and
intensity profiles [Baranoski et al. 2003], which correlate with the height of the emission point [Brekke
and Egeland 1994]. The conversion of auroral spectral emissions to a color space follows a straight-
forward CIE XYZ to RGB mapping procedure. We include the strongest auroral spectral emissions
in our simulations, namely, 630.0nm (red line), 557.7nm (green line) and 427.8nm (blue line). More
wavelengths can be considered at the expense of higher computational time.
  The different lifetimes of transition states in auroral emissions—110s for the red line, 0.7s for the
green line, and 0.001s for the blue line—cause the photon emissions to occur in distinct volumes around
the principal direction of emission. In order to simulate this distribution of auroral emissions, we
convolve the image with a color-dependent Gaussian low-pass filter [Baranoski et al. 2003; Westover
1991].
  An auroral display also exhibits global temporal variations, captured in photographs as blurred forms,
due to finite exposure times. For simulating this global blurring effect we perform a convolution using a
temporal low-pass filter [Baranoski et al. 2000; Castleman 1996]. The longer the sampled window is in
time, the blurrier the results, similar to the effects captured in real photographs with longer exposure
times.
  The final stage of our rendering pipeline consists of blending the auroral display with a night sky
background and a foreground. We use a composite approach in which we initialize the color buffer with
the background scene, superimpose the auroral display, and, finally, filter out regions of the image that
would be covered by foreground objects.

6.      RESULTS
In this section, we present examples of simulations of auroral motions and shape changes at different
spatio-temporal scales.2 The stochastic nature of auroral displays reduces the viability of a quantitative
analysis of their visual simulations. An alternative available is to qualitatively analyze the simulations
in comparison with photographs of real auroral displays. However, as mentioned earlier, photographs of
an aurora are usually blurred due to low light conditions and significant exposure times, and sometimes
present background artifacts caused by grain when high speed film is used (Figures 10 and 13). For
this reason we accounted for photographic blur in our simulations.
  Despite the blur, features readily recognizable by viewers in real aurorae were preserved, for example,
presence of rayed structures and vertical distribution of intensity (Figure 11), as well as their charac-
teristic spatio-temporal variations. This aspect was verified through comparisons with observations of
the real phenomena, including the positive feedback from scientists working in the auroral imaging
field [Hallinan 2001; Trondsen 2001].
  Figures 11, 12 and 14 present still frames from simulation sequences produced using our model. For
these sequences, we used a 1024 × 1024 mesh for the starting points, and varying values for the vertical
displacement of the electron beams. The length and thickness of the initial auroral stream used in these

2 The   simulation sequences (quicktime format) described in this article are available at http://www.curumin.uwaterloo.
ca/ ˜ gvgbaran/dynamics.html.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                   Simulating the Dynamics of Auroral Phenomena                •      51




                 Fig. 10. Photograph of an auroral spiral. (Courtesy of Norbert Rosing.)




         Fig. 11. Simulation of an auroral spiral showing a rayed structure and an arc break up.




Fig. 12. Frames of a simulation sequence showing the motions and shape changes of an auroral rayed band.


                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
52      •     G. V. G. Baranoski et al.




Fig. 13. Black and white all-sky photographs showing different stages of an auroral surge formation. (Courtesy of Kirsti
Kauristie.)

simulations were 2000km, and 9km, respectively. The computations were performed on a SGI550, with
dual 850Mhz Pentium3 processors.
  Figure 12 presents six frames from a simulation sequence showing the motions and shape changes
of an auroral rayed band. In this sequence, one can observe the rapid movement of auroral rays,
the formation and motion of folds, and a rotational distortion of the auroral band. In this simulation,
the number of electron beams was 4 × 104 , with a parametric collision interval (Section 4.2) of 0.05.
The time spent to compute the starting points and the descending trajectories for each time step was
2.6s and 10.8s, respectively. The frames were rendered at a resolution of 320 × 240 pixels, and using
10 rays per-emission-point.
  Figure 13 presents two all-sky photographs taken during an auroral surge formation in the northern
hemisphere. Figure 14 presents three frames from a simulation sequence, illustrating a simulation of
such natural phenomena at twice normal speed. In this simulation, the number of electrons beams was
104 , with a parametric collision interval of 0.1. The time spent to compute the starting points and the
descending trajectories for each time step was 1.7s and 6s, respectively. The frames were rendered at a
resolution of 480 × 360 pixels, and using 5 rays per-emission-point.
  Recently, scientists, using NASA’s polar spacecraft, have captured images of the aurorae on film as
they were dancing simultaneously around both magnetic poles3 . This film shows the Aurora Borealis
and Aurora Australis expanding and brightening, in parallel, at opposite ends of the Earth, confirming
the three-century-old theory that aurorae in the northern and southern hemisphere are nearly mirror
images of each other. In the case of auroral spirals, the distinct feature is their opposite sense of
rotation. Usually they are counterclockwise in the southern hemisphere when viewed from the ground,
and clockwise in the northern hemisphere.
  Figure 15 presents six frames from a simulation sequence, illustrating an auroral surge formation in
the southern hemisphere. In this sequence, besides the warping process leading to an auroral spiral, we
can also notice the formation and motion of folds. In this simulation, there were 104 electrons beams,
with a parametric collision interval of 0.2. The time spent to compute the starting points and the
descending trajectories for each time step was 1.7s and 10.27s, respectively. The frames were rendered
at a resolution of 320 × 240 pixels, and using 20 rays per-emission-point.
  Using the multigrid algorithm, we were able to reduce substantially the time required in the warp-
ing computations. For instance, on the 1024 × 1024 auroral mesh used in our simulations, multi-
grid is 96 times faster than CG. Due to this reduction, the computation of the descending trajectories

3 http://eiger.physics.uiowa.edu/ ˜ vis/conjugate   aurora/.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                           Simulating the Dynamics of Auroral Phenomena                •      53




 Fig. 14. Frames from a simulation sequence illustrating an auroral surge (spiral) formation in the northern hemisphere.

became the bottleneck of our simulation pipeline, with its cost growing linearly with respect to the
number of electron beams. A divide-and-conquer strategy can be applied, however, to decrease the
total running time of these computations. In other words, one can process n trajectories separately
in several processors. For instance, for 105 electron beams, Baranoski and Rokne [2002] obtained a
                                                                      ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
54      •     G. V. G. Baranoski et al.




        Fig. 15. Frames of a simulation sequence illustrating an auroral surge formation in the southern hemisphere.


speedup of 4.6 using five Compaq Alpha 500Mhz PWS processors. Nonetheless, we believe that there
is still room for improvement on the parallelization of the algorithms. We intend to continue to work
on the parallelization, and to exploit graphics hardware for the real time simulation of moving auroral
displays.


7.   DIRECTIONS FOR APPLICATIONS AND FUTURE RESEARCH
The physical simulation and visual representation of natural phenomena is an evolving area of re-
search within computer graphics where significant progress has been made. However, the important
class of natural processes collectively known as plasma phenomena present significant computational
challenges and remain relatively unexplored [Gombosi et al. 2004]. The aurorae are among the most
spectacular and most important from a scientific point of view due to the link they provide between
solar activities and the Earth. In fact, the aurorae can be thought of as “footprints” of events and en-
ergetic process occurring in the Earth’s magnetosphere. For this reason, even after 200 years of study,
an astonishing amount of basic research is still being done on the aurora and its electromagnetic and
electrodynamical properties [Paxton and Meng 1999].
   The complexities of numerical simulations in the physical sciences often require visual representa-
tions. Complex situations, for example, those found in plasma studies, may be dissected into simpler
components to help the understanding of the underlying phenomena and obtain a clear theoretical
picture [Tang 2002]. Moreover, animated sequences derived from computer simulations give a picture
of a simulated phenomenon through time, illustrating qualitative features of any instabilities that may
not be evident from a theoretical point of view. In fact, visualization plays a critical role in going from
the raw nonlinear solution of the complex equations governing these phenomena to a simplified model
explaining the essential underlying plasma physics as Parker et al. [1995] appropriately stated. This
contribution of computer graphics to scientific applications was highlighted by D. Greenberg in the
Steven Coons Award Lecture in 1987 [Greenberg 1988]:

       If computer graphics is to have a role in improving the future of our civilization, the real
       value will be in its application to science, engineering and design.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                   Simulating the Dynamics of Auroral Phenomena              •      55

  Viewed in this context, we foresee three main directions for applications and future research associ-
ated with the work presented in this article:
—visualization of plasma phenomena,
—visualization of Earth’s electro-magnetic environment in space, and
—visualization of astrophysical phenomena.
   In order to understand plasma phenomena, a combination of extensive data analysis, theory, model-
ing, laboratory experiments, and in situ (space born) observation is required. Plasma science impacts
daily life in many significant ways. In 1995, a panel of scientists at the National Research Council [NRC
1995] recommended that NASA and NSF fund a vigorous observational program related to plasma sci-
ence, both on Earth and in space. The program should include both in situ and ground-based facilities,
properly balanced with programs in theory, modeling, and laboratory experiments (see also NRC 2003).
   In our everyday lives, we encounter plasma in many situations, such as the flash of a lightning bolt,
the conducting of gas inside a fluorescent tube or neon sign, the ionization in a rocket exhaust, gas
lasers, and lightning discharges as some of the more familiar examples [Chen 1984; Smirnov 2001].
In the neighborhood of Earth, one encounters plasma in the Van Allen radiation belts and the solar
winds as soon as one leaves the Earth’s atmosphere. Further afar, stellar interiors and atmospheres, as
well as gaseous nebulae, are composed of plasma. For example, the Crab Nebula contains a magnetic
field and it is a rich source of plasma phenomena. It also contains a visual pulsar. Theories of pulsars
picture them as rapidly rotating neutron stars with plasmas emitting synchrotron radiation from the
surface [Bittencourt 1986; Chen 1984]. Occasionally, one also encounters an entirely new physical
situation which is unlike anything previously observed either in space or in laboratory, and this opens
new avenues of research. One example is the dusty plasma of comets and planetary rings that are
dominated by the dynamics of charged macroparticles for which gravitational and electromagnetic
effects are of comparable importance [NRC 1995].
   In the type of particle model used in this article, the computer particle retains much of the identity
of the atomic constituents of the material being simulated, yet does not correspond one-to-one to the
simulated particles as do atoms to particles in molecular dynamics simulations. The “electron beams” or
“superparticles” in these simulations correspond to millions of physical electrons. The plasma phenom-
ena mentioned above can also be simulated using this approach. Furthermore, all of the mathematical
models to which particle simulation methods are applied can be formulated as a set of particles inter-
acting through fields. The most important field equation is the Poisson’s equation, which pertains to all
plasma examples listed above, and it is also found in the simulation of clustering of galaxies [Hockney
and Eastwood 1988].
   Electromagnetism is a fundamental topic [Belcher and Bessette 2001]. It is, however, one of the
most difficult subjects for students to master. Belcher and Bessette [2001] addressed this difficulty
by creating 3D field line animations used to aid student intuition about electromagnetic phenomena
occurring in the laboratory. The auroral displays are similarly caused by electromagnetic phenomena,
but at a planetary scale, that is, the solar wind impinges on the Earth’s magnetosphere, which shields
us from this radiation, distorting the field lines of the magnetosphere in the process. Visualization of
such phenomena is important not only from a educational point of view [Nylund and Holland 1999], but
also from a research point of view since it can be used to support improved predictions of space weather
and its effects on systems operating in space as well as on Earth’s surface [Williams et al. 1999]. We
also note that the magnetic storms responsible for auroral displays can damage satellites, power grids,
pipelines, and spacecrafts [Paxton and Meng 1999; Odenwald 2000].
   Plasma simulations are relevant to almost every area of astrophysics, ranging from stellar and inter-
stellar plasmas, to star clusters and galaxies [NRC 1995]. The significant mutual interactions among
                                                            ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
56      •     G. V. G. Baranoski et al.

stars and galaxies are provided by Newtonian gravity. The latter is an inverse-square force like the
electrostatic force, so it is natural that there should be many similarities between the dynamics of these
systems and plasma physics [Binney 1993]. For example, both stellar dynamics and plasma dynamics
account for shear (the simplest model of a stellar disc, which dominates most of galaxies, is the so-called
shearing sheet [Binney 1993]) and can be simulated using “superparticles” models. Superparticles in a
galaxy model represent millions of real stars. Length and time scales are determined by the dimensions
and motion of the spiral arm structures. Collisional effects are made small by smoothing the force of
interaction at short range, and by having a large number of superparticles within the range of smooth-
ing [Hockney and Eastwood 1988]. One of the principal applications of such a model is in the study of
spiral patterns arising in galaxies [Hockney and Eastwood 1988]. Such patterns are visually similar to
the spiral structures arising from our auroral dynamics simulation. This is not a surprise considering
that several processes familiar from plasma physics play important roles in structuring stellar systems.
Obviously, gravity, unlike the electrostatic interaction, is always attractive, and this difference between
the two systems invariably leaves its imprint [Binney 1993]. Consequently, there are also differences be-
tween the way a given process works in the plasma and stellar-dynamics. We believe, however, that such
cross-fertilization is beneficial for both areas of research, and our intent is to fully explore these parallels.

8.   CONCLUSION
In this article, we have presented a novel physically-based model for the 3D visual simulation of auroral
dynamics. The model employs a complete multigrid PDE simulation of the electromagnetic interactions
between auroral electrons. It allows an accurate and efficient visualization of auroral phenomena at
different spatio-temporal scales. As far as we have been able to ascertain, this article describes the first
physically-based visual simulations of these phenomena in the scientific literature, leading to realistic
visualization of the shape and the motion of these fascinating phenomena. Previous auroral simulations
have been restricted to the static and KHI free case [Baranoski et al. 2000].
   One of the most important research problems facing the computer graphics field since its beginning,
is the correct simulation of physical phenomena. As a result, the simulation of natural phenomena such
as water, fire, clouds, smoke and so forth. has attracted significant research efforts. These efforts have
been concentrated in the less than 1% of the universe where plasmas are not readily apparent, leaving
the remaining 99% filled with plasma phenomena to be investigated by future researchers. Although
scientific and commercial applications have different constraints, we believe that computer graphics
researchers working in either area can benefit from efficient and predictive algorithms for the realistic
simulation of plasma phenomena such as the majestic Northern Lights.

ACKNOWLEDGMENTS
Thanks are due to Trond Trondsen and Thomas Hallinan for their valuable scientific feedback on the
simulation of auroral phenomena; Kirsti Kauristie and Norbert Rosing for allowing us to use their
                                                                                      ¨
auroral photographs; Adriane Baranoski for helping us with the diagrams; Tamer Ozsu for his encour-
agement; NASA Polar Ultraviolet Imager for granting us access to a satellite snapshot of an auroral
oval; Gail Chopiak for her logistical assistance; Jessica Hodgins and John Hart for their helpful editorial
suggestions; and the anonymous reviewers for their useful insights.

REFERENCES
ANDERSON, U. 2001. Time-domain methods for the Maxwell equations. Ph.D. thesis, Department of Numerical Analysis and
  Computer Science, Royal Institute of Technology, Sweden.
ARULIAH, D. AND ASCHER, U. 2002. Multigrid preconditioning for Krylov methods for time-harmonic Maxwell’s equations in
  three dimensions. SIAM J. Scientif. Comput. 24, 2, 702–718.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                             Simulating the Dynamics of Auroral Phenomena                 •      57

ARVO, J. 1995. Analytic methods for simulated light transport. Ph.D. thesis, Yale University.
BARANOSKI, G. AND ROKNE, J. 2002. Using a HPC system for the simulation of the trajectories of solar wind particles in the
  ionosphere. In High Performance Computing Systems and Applications, N. Dimopoulos and K. Li, Eds. Kluwer Academic
  Publishers, Norwell, MA. (Chapter 2), 317–329.
BARANOSKI, G., ROKNE, J., SHIRLEY, P., TRONDSEN, T., AND BASTOS, R. 2000. Simulating the aurora borealis. In the 8th Pacific
  Conference on Computer Graphics and Applications. IEEE Computer Society, Los Alamitos, CA. 2–14.
BARANOSKI, G., ROKNE, J., SHIRLEY, P., TRONDSEN, T., AND BASTOS, R. 2003. Simulating the aurora. J. Visualiza. Comput. Ani-
  mat. 14, 1 (Feb.), 43–59.
BARANOSKI, G., WAN, J., ROKNE, J. G., AND BELL, I. 2002. Simulating the dynamics of the dancing lights. Tech. Rep. CS-2002-16,
  (April). School of Computer Science, University of Waterloo.
BELCHER, J. AND BESSETTE, R. 2001. Using 3D animation in teaching introductory electromagnetism. Comput. Graph. 35, 1
  (Feb.), 18–21.
BINNEY, J. 1993. Gravitational plasmas. In Plasma Physics An Introductory Course, R. Dendy, Ed. Cambridge University
  Press, Cambridge, UK. 291–318.
BIRDSALL, C. AND FUSS, D. 1969. Clouds-in-clouds, clouds-in-cells physics for many-body plasma simulation. J. Comput. Phys. 3,
  494–511.
BITTENCOURT, J. 1986. Fundamentals of Plasma Physics. Pergamon Press, Oxford, UK.
BOROVSKY, J. AND SUSZCYNSKY, D. 1993. Optical measurements of the fine structure of auroral arcs. In Auroral Plasma Dynamics,
  R. Lysak, Ed. American Geophysical Union, Washington, D.C., 25–30. (Vol. 80 of Geophys. Monogr. Series.)
BOROVSKY, J., SUSZCYNSKY, D., BUCHWALD, M., AND DEHAVEN, H. 1991. Measuring the thickness of auroral curtains. Arctic 44, 3,
  231–238.
BRANDT, A. 1977. Multi-level adaptive solutions to boundary-value problems. Math. Comp. 31, 333–390.
                                                                                         o
BREKKE, A. AND EGELAND, A. 1994. The Northern Lights, Their Heritage and Science. Gr¨ ndahl og Dreyers Forlag, AS, Oslo.
BRYANT, D. A. 1999. Electron Acceleration in the Aurora and Beyond. Institute of Physics Publishing, Bristol, UK.
BURTNYK, K. 2000. Anatomy of an aurora. Sky and Telescope 99, 3 (March), 35–40.
CASTLEMAN, K. 1996. Digital Image Processing. Prentice-Hall, New York.
CHAN, T. AND WAN, W. 2000. Robust multigrid methods for elliptic linear systems. J. Comput. Appl. Math. 123, 323–352.
CHEN, F. 1984. Introduction to Plasma Physics and Controlled Fusion, 2nd Ed. Plenun Press, New York.
DAVIS, N. 1992. The Aurora Watchers Handbook. University of Alaska Press, Fairbanks.
DAVIS, T. AND HALLINAN, T. 1976. Auroral spirals. 1. observations. J. Geophys. Research. 81, 22 (Aug.), 3953–3958.
DRAZIN, P. AND REID, W. 1981. Hydrodynamic Stability. Cambridge University Press, Cambridge.
DUPONT, D. 2004. Nuclear explosions in orbit. Scientific American, 100–107.
EATHER, R. 1980. Majestic Lights. American Geophysical Union, Washington.
ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. SIGGRAPH Proceedings,
  Annual Conference Series, 736–744.
FEDKIW, R., STAM, J., AND JENSEN, H. 2001. Visual simulation of smoke. SIGGRAPH Proceedings, Annual Conference Series,
  15–22.
FEDORENKO, R. 1961. A relaxation method for solving elliptic difference equations. USSR Comput. Math. Phys. 1, 1092–1096.
FELDMAN, B., O’BRIEN, J., AND ARIKAN, O. 2003. Animating suspended particle explosions. SIGGRAPH Proceedings, Annual
  Conference Series, 708–715.
FOSTER, N. AND FEDKIW, R. 2001. Practical animation of fluids. SIGGRAPH Proceedings, Annual Conference Series, 23–30.
FOSTER, N. AND METAXAS, D. 1997. Modeling the motion of a hot, turbulent gas. SIGGRAPH Proceedings, Annual Conference
  Series, 181–188.
GOLUB, G. AND LOAN, C. V. 1989. Matrix Computations, Second ed. John Hopkins University Press, Baltimore.
GOMBOSI, T., POWELL, K., ZEEEUW, D., CLAUER, C., HANSEN, K., MANCHESTER, W., RIDLEY, A., ROUSSSEV, I., ABD Q. F., STOUT, I. S.,
  AND TOTH, G. 2004.      Solution-adaptive magnetohydrodynamics for space plasmas: Sun-to-Earth simulations. Computing in
  Science and Engineering, 14–35.
GREENBERG, D. 1988. 1987 Steven A. Coons Award Lecture. Comput. Graph. 22, 1 (Feb.), 7–14.
HACKBUSCH, W. 1985. Multi-grid Methods and Applications. Springer Verlag, Berlin.
HALLINAN, T. 1976. Auroral spirals. 2. theory. J. Geophys. Resear. 81, 22 (Aug.), 3959–3965.
HALLINAN, T. 1981. The distribution of vorticity in auroral arcs. In Physics of Auroral Arc Formation, S. Akasofu and J. Kan,
  Eds. American Geophysicsl Union, Washington, D.C.
HALLINAN, T. 2001. Personal communication. Geophysical Institute, University of Alaska.
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
58      •     G. V. G. Baranoski et al.

HALLINAN, T. AND DAVIS, T. 1970. Small-scale auroral distortions. Planet. Space Sci. 18, 1735–1744.
HINZE, J. 1975. Turbulence. McGraw-Hill, New York.
HOCKNEY, R. 1966. Computer experiment of anomalus diffusion. Phys. Fluids 9, 9 (Sept.), 1826–1835.
HOCKNEY, R. AND EASTWOOD, J. 1988. Computer Simulation Using Particles. Institute of Physics, Bristol.
JENSEN, H., DURAND, F., STARK, M., PREMOZE, S., DORSEY, J., AND SHIRLEY, P. 2001. A physically-based night sky model. SIGGRAPH
  Proceedings, Annual Conference Series, 399–408.
JONES, A. 1974. Aurora. D. Reidel Publishing Company, Dordrecht, Holland.
KAJIYA, J. AND HERZEN, B. V. 1984. Ray tracing volume densities. Comput. Graph. 18, 3 (July), 165–174.
       ¨
K¨ RKKAINEN, K. 2002. On the finite-difference modelling of electromagnetic problems in structured lattices. Ph.D. thesis,
  A
  Electromagnetics Laboratory, Department of Electrical and Communications Engineering, Helsinki University of Technology,
  Finland.
KUNIMATSU, A., WATANABE, Y., FUJII, H., SAITO, T., HIWADA, K., TAKAHASHI, T., AND UEKI, H. 2001. Fast simulation of and rendering
  techniques for fluid objects. Comput. Graph. For. 20, 3, 57–66.
LEVY, R. AND HOCKNEY, R. 1968. Computer experiments on low-density crossed-field electron beams. Phys. Fluids 11, 4 (April),
  766–771.
LJUNG, P., DIECKMANN, M., ANDERSSON, N., AND YNNERMAN, A. 2000. Interactive visualization of particle-in-cell simulations. In
  IEEE Visualization 2000. 469–472.
MIYAZAKI, R., DOBASHI, Y., AND NISHITA, T. 2002. Simulation of cumuliform clouds based on computational fluid dynamics. In
  Short Presentations of Eurographics 2002, I. Navazo and P. Slusallek, Eds. The Eurographics Association. 405–410.
MYAZAKI, R., OSHIDA, S., DOBASHI, Y., AND NISHITA, T. 2001. A method for modeling clouds based on atmospheric fluid dynamics.
  In the 9th Pacific Conference on Computer Graphics and Applications. IEEE Computer Society, Los Alamitos, CA. 363–372.
NADEAU, D., GENETTI, J., NAPEAR, S., PAILTHORPE, B., EMMART, C., WESSELAK, E., AND DAVIDSON, D. 2001. Visualizing stars and
  emission nebulas. Comput. Graphics Forum 20, 1 (March), 27–33.
NGUYEN, D. Q., FEDKIW, R., AND JENSEN, H. W. 2002. Physically based modeling and animation of fire. SIGGRAPH Proceedings,
  Annual Conference Series, 721–728.
NRC. 1995. Plasma Science from Fundamental Research to Technological Applications. National Academy Press, Washington,
  D.C.
NRC. 2003. The Sun to the Earth and Beyond. National Academy Press, Washington, D.C.
NYLUND, S. AND HOLLAND, D. 1999. Themes and trends in space science data processing and visualization. John Hopkins APL
  Tech. Digest 20, 4, 533–543.
ODENWALD, S. 2000. Solar storms: The silent menace. Sky and Telescope 99, 3 (March), 41–56.
OMHOLT, A. 1971. The Optical Aurora. Springer-Verlag, New York.
PARKER, S., CUMMINGS, J., AND SAMTANEY, R. 1995. Visualization of plasma turbulence. IEEE Comput. Graph. Appl. 15, 2, 7–10.
PARTAMIES, N., FREEMAN, M., AND KAURISTIE, K. 2001. On the winding of auroral spirals: Interhemispheric observations and
  Hallinan’s theory revisited. J. Geophys. Resear. 106 (A12), 28913–28924.
PARTAMIES, N., KAURISTIE, K., PULKKINEN, T., AND BRITTNACHER, M. 2001.            Statistical study of auroral spirals. J. Geophys.
  Resear. 106, 15415–15428.
PAXTON, L. AND MENG, C. 1999. Auroral imaging and space-based optical remote sensing. John Hopkins APL Tech. Digest 20, 4,
  544–555.
RASMUSSEN, N., NGUYEN, D. Q., GEIGER, W., AND FEDKIW, R. 2003. Smoke simulation for large scale phenomena. SIGGRAPH
  Proceedings, Annual Conference Series, 703–707.
SCHUSSMAN, G., MA, K., SCHISSEL, D., AND EVANS, T. 2000. Visualizing DIII-D Tokamak magnetic field lines. In IEEE Visualization
  2000. 501–504.
SILVESTER, P. AND FERRARI, R. 1990. Finite Elements for Electrical Engineers, 2nd Ed. Cambridge University Press, Cambridge.
SMIRNOV, B. 2001. Physics of Ionized Gases. John Wiley & Sons, New York.
STAM, J. 1995. Multiple scattering as a diffusion process. In Rendering Techniques’95 (Proceedings of the Sixth Eurographics
  Rendering Workshop), P. M. Hanrahan and W. Purgathofer, Eds. SIGGRAPH Proceedings, Annual Conference Series, 41–50.
STAM, J. 1999. Stable fluids. SIGGRAPH Proceedings, Annual Conference Series, 121–128.
STAM, J. AND FIUME, E. 1993. Turbulent wind fields for gaseous phenomena. SIGGRAPH Proceedings, Annual Conference
  Series, 369–376.
TAFLOVE, A. 2000. Computational Electrodynamics: The finite-difference time-domain method, 2nd Ed. Artech House, Boston,
  MA.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                             Simulating the Dynamics of Auroral Phenomena                •      59

TAJIMA, T. 1989. Computational Plasma Physics: With Applications to Fusion and Astrophysics. Addison-Wesley, Redwood
  City, CA.
TANG, W. 2002. Advanced computations in plasma physics. Phys. Plasmas 9, 5, 1856–1872.
TAYLOR, K. 2001. Auroras earth’s grand show of lights. National Geographic, 48–63.
TRONDSEN, T. 1998. High spatial and temporal resolution auroral imaging. Ph.D. thesis, Department of Physics, Faculty of
  Science, University of Tromsø, Norway.
TRONDSEN, T. 2001. Personal communication. Institute for Space Research, The University of Calgary.
                                     ¨                                     ¨
TROTTENBER, U., OOSTERLEE, C., SCHULLER, A., BRANDT, A., OSWALD, P., AND STUBEN, K. 2001. Multigrid. Academic Press, New
  York.
UPSON, C., BARR, A., REEVES, B., WOLFF, R., AND WOLFRAM, S. 1987. The physical simulation and visual representation of natural
  phenomena. Comput. Graph. 21, 4, 355–356.
VOLAKIS, J. AND KEMPEL, L. 1995. Electromagnetics: Computational methods and considerations. IEEE Computat. Sci. Engi-
  neer. 2, 1, 42–57.
WEIMER, H. AND WARREN, J. 1999. Subdivision schemes for fluid flow. SIGGRAPH Proceedings, Annual Conference Series,
  111–120.
WESTOVER, L. 1991. Splatting: A parallel, feed-forward volume rendering algorithm. Ph.D. thesis, Department of Computer
  Science, University of North Carolina at Chapel Hill.
WILLIAMS, D., MAUK, B., MITCHELL, D., ROELOF, E., AND ZANETTI, L. 1999. Radiation belts and beyond. John Hopkins APL Tech.
  Digest 20, 4, 544–555.
WITKIN, A. AND KASS, M. 1991. Reaction-diffusion textures. Comput. Graph. (SIGGRAPH Proceedings) 25, 4, 299–308.
YOSHIZAWA, A. 1998. Hydrodynamic and Magnetohydrodynamic Turbulent Flows: Modelling and Statistical Theory. Kluver,
  Dordrecht.
YOSHIZAWA, A., ITOH, S., ITOH, K., AND YOKOI, N. 2001. Turbulence theories and modelling of fluids and plasma. Plasma Phys.
  and Controlled Fusion 43, 1–144.


Received May 2002; accepted October 2004




                                                                        ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
On the Optimality of Spectral Compression of Mesh Data
MIRELA BEN-CHEN and CRAIG GOTSMAN
Technion—Israel Institute of Technology


Spectral compression of the geometry of triangle meshes achieves good results in practice, but there has been little or no theoretical
support for the optimality of this compression. We show that, for certain classes of geometric mesh models, spectral decomposition
using the eigenvectors of the symmetric Laplacian of the connectivity graph is equivalent to principal component analysis on
that class, when equipped with a natural probability distribution. Our proof treats connected one-and two-dimensional meshes
with fixed convex boundaries, and is based on an asymptotic approximation of the probability distribution in the two-dimensional
case. The key component of the proof is that the Laplacian is identical, up to a constant factor, to the inverse covariance matrix
of the distribution of valid mesh geometries. Hence, spectral compression is optimal, in the mean square error sense, for these
classes of meshes under some natural assumptions on their distribution.
Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling; E.4 [Coding
and Information Theory]
General Terms: Theory
Additional Key Words and Phrases: Triangle mesh, spectral decomposition, Laplacian



1.   INTRODUCTION
Triangle meshes are a popular way of representing 3D shape models. As the size and detail of the
models grow, compression of the models becomes more and more important. The size of the mesh data
files can be reduced by compressing either the geometry or the connectivity of the mesh, or both.
   There has been much research into both geometry and connectivity coding. Most connectivity coding
schemes, for example the Edgebreaker [Rossignac 1999] and the TG methods [Touma and Gotsman
1998], are based on traversing the mesh and generating code symbols representing new vertices or faces
as they are traversed. The quality of the compression results from the entropy of the symbol sequence.
   Typical geometry coding schemes are based on the fact that the coordinates of a mesh are not in-
dependent, and specifically, the coordinates of neighboring vertices are highly correlated, especially
in smooth meshes. This correlation can be exploited by using “prediction rules”—the coordinates of a
vertex are predicted from the coordinates of neighboring vertices, and only the prediction error vector
is coded [Taubin and Rossignac 1998; Touma and Gotsman 1998]. The better the prediction rule is, the
smaller the errors are, and the smaller the entropy of the code will be.
   Spectral compression of mesh geometry [Karni and Gotsman 2000] also exploits the correlation among
neighboring vertices, and implicitly applies a prediction rule that every vertex is the simple average

This work was partially supported by the Israel Ministry of Science Grant 01-01-01509, the German-Israel Fund (GIF)
Grant I-627-45.6/1999, European FP5 RTN Grant HPRN-CT-1999-00117 (MINGLE), and European FP6 NoE Grant 506766
(AIM@SHAPE).
Authors’ addresses: Department of Computer Science, Technion-Israel Institute of Technology, Haifa 32000 Israel; email:
{mirela,gotsman}@cs.technion.ac.il.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0060 $5.00
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 60–80.
                                                   The Optimality of Spectral Compression of Mesh Data               •      61




                             Fig. 1. A simple irregular mesh and its symmetric Laplacian.

of all its immediate neighbors. Inspired by traditional signal coding, spectral decomposition has been
proposed for lossy transform coding of the geometry of a mesh with irregular connectivity. Although
the method yields good results in practice, there is little theoretical support for the optimality of such
compression.
   Motivated by the optimality of the Discrete Cosine Transform (DCT) in signal processing [Rao and
Yip 1990], we wish to prove a similar result for spectral compression, namely, that it is optimal for
certain classes of irregular meshes. Our proof covers connected meshes in one and two dimensions
with fixed convex boundaries. The proof for the two-dimensional case is based on an asymptotic normal
approximation.

1.1   Previous Work
Let G be a graph, G = (V , E), where V is the vertex set, and E the edge set. A k-dimensional mesh
M is M = (G, R), R = (X (1) , X (2) , . . . , X (k) ), where X (i) is a real vector of the i-th dimension coordinate
values of the mesh vertices. We sometimes refer to E as the connectivity and to R as the geometry of
the mesh M .
  Given a mesh M = (G, R), G = (V , E), the symmetric Laplacian of M is the matrix L:
                                                       
                                                        di i = j
                                                       
                                              Li, j = 0               /
                                                              (i, j ) ∈ E
                                                       
                                                       
                                                         −1 (i, j ) ∈ E
where d i is the number of neighbors (valence) of the i-th vertex. A mesh with constant valences is called
a regular mesh; otherwise, it is called an irregular mesh. See Figure 1 for an example of an irregular
mesh, and its corresponding Laplacian.
  Spectral decomposition was first introduced by Karni and Gotsman [2000] as a tool for mesh geometry
compression. In their work, the eigenvectors of the Laplacian matrix are used as the basis for the
decomposition of all meshes having the same connectivity. Karni and Gotsman argued that since these
vectors were a generalization of the Fourier basis vectors obtained for regular (grid-type) connectivities,
they should work just as well for irregular connectivities. Indeed, they showed that the coefficients of
such decomposition decay rapidly to zero (see Figure 2). Hence, a small number of coefficients, combined
with aggressive quantization, can be used to code the mesh geometry. The eigenvalues corresponding
to the eigenvectors are analogous to the concept of frequency in harmonic analysis.
  Eigenanalysis of the graph Laplacian is a standard tool in spectral graph theory [Chung 1997]. Several
variants of the Laplacian have been studied. Eigenvectors of Laplacians are also used for graph drawing
[Hall 1970; Koren 2003], graph partitioning [Alpert and Yao 1995], and parametrization [Gotsman et al.
2003]. For a more extensive survey of the uses of Laplacians in digital mesh processing, see Gotsman
[2003]. The inverse of the Laplacian is closely related to discrete Green’s functions [Chung and Yau
2000], and, as we shall see, plays a central role in this article.
                                                                    ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
62      •     M. Ben-Chen and C. Gotsman




Fig. 2. Spectral decomposition. (a) The horse model and (b, c, d) the spectral coefficients of its decomposition in the X , Y , and
Z dimensions.

   The rest of the article is organized as follows. Section 2 defines the terminology used throughout the
article. Section 3 reviews the concept of principal component analysis, a key tool in our proofs, which
is the motivation for studying the eigenvectors of the covariance matrix. Sections 4 and 5 prove the
optimality result for the 1D and 2D cases, respectively. To conclude, Section 6 discusses our model and
explores future research directions.

2.   DEFINITIONS
The decomposition of a vector X nx1 , by the orthonormal basis of R n : Unxn = {U1 , . . . , Un }, is:
                                                                    n
                                                              X =         ci Ui ,                                             (1)
                                                                    i=1

where C ={ci } are the coefficients of the decomposition.
 The reconstruction of a vector X from its decomposition using the first k coefficients is:
                                                                          k
                                                          X (U,k) =           ci Ui .                                         (2)
                                                                        i=1

  Note that the order of the basis vectors is important, as only a prefix of them is used in the k-th
reconstruction.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                 The Optimality of Spectral Compression of Mesh Data              •      63

   Because of the orthonormality of U , the Parseval identity maintains that c = X . A decomposition
is useful when a small number of the coefficients contain a large portion of the norm (energy) of the
vector. If this is the case, those coefficients alone may be used in the reconstruction, and the Euclidean
distance between the reconstructed vector and the original X will be small. This is a useful feature for
compression applications.
   For a specific vector X , the best basis will always be {X , 0, 0, . . . , 0}, since then all the energy is con-
tained in the first coefficient. Hence, there is a meaningful answer to the question “what is the optimal
basis for decomposition” only if we consider an entire family of vectors (finite or infinite). In our context,
the family will be all the geometries which are valid for a given mesh connectivity E. These geometries
will be specified by imposing an appropriate probability distribution D, derived from E, on R n .
   If X is a random variable, we denote by Exp(X ) the expectation of X , by Var(X ) the variance of X
for scalar X , and by Cov(X ) the covariance matrix of X for vector X .
   Given a probability distribution D on R n , we say that a vector basis U is an optimal basis for D, if
for any other basis W , and for every 1 ≤ k ≤ n:
                                                     2                         2
                                  Exp X − X (U,k)        ≤ Exp X − X (W,k)                                              (3)
where the expectation is taken over D. This is called optimality in the Mean Square Error (MSE) sense.

3.   PRINCIPAL COMPONENT ANALYSIS
Approximation of random signals has been studied extensively in signal processing. A well-known
optimality result, which we will rely on heavily, is related to the so-called Principal Component Analysis
(PCA) procedure.
  Assume a random vector X ∈ R n , sampled from distribution D, having zero mean vector and co-
variance matrix C. Denote by { i |i = 1, . . , n} the eigenvectors of C, with corresponding eigenvalues
{λi |i = 1, . . . , n}, ordered such that λ1 ≥ λ2 ≥ · · · ≥ λn . The matrix is called the Principal Component
Analysis of X , sometimes known as the Karhunen-Loeve or Hotelling transform of X .
  It is well known [Jolliffe 1986] that the PCA is optimal in the MSE sense, as defined in (3). Note that
   does not depend on k—the number of vectors used for the reconstruction—so the optimal basis for
reconstruction from k + 1 basis vectors contains the optimal basis for reconstruction from k vectors.
  When the class of meshes is finite and given, containing T meshes, for example, an animation se-
quence of a mesh with a fixed connectivity, the natural distribution is a uniform distribution on this
finite set. In this case, the PCA of that class may be computed using the numerical Singular Value
Decomposition (SVD) procedure [Press et al. 1987] on a matrix of size 3n × T , consisting of T columns,
where the i-th column contains the coordinate values of the i-th mesh in the sequence. This was proposed
by Alexa and Muller [2000] for compressing animated meshes.
  To compute the PCA of an infinite continuum of meshes with a given connectivity, where none of
them is explicitly given, we must first make some assumptions about the distribution D of this family,
and then compute the covariance matrix C of this distribution. We will do this, and then show that
C is essentially identical to the inverse of the mesh Laplacian matrix, up to a constant factor. Due to
this special relationship, both matrices have identical eigenvectors (in opposite order), from which our
optimality theorem will follow.

4.   1D MESHES
Before we proceed, a note about the connection between the proof for one dimension, and the proof for
two dimensions is in order. Basically, for both cases we prove the same theorem—that L = αC −1 , for
some constant α. While for the one-dimensional case the theorem can be proved relatively easily, in two
dimensions we resort to asymptotic approximations.
                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
64      •     M. Ben-Chen and C. Gotsman




                                Fig. 3. The “chain” connectivity” graph for n = 5 in one dimension.

4.1    The Model
In one dimension, a mesh is simply an ordered set of points on the real line. There is only one “universal”
(regular) connectivity—each vertex is connected to exactly two neighbors, except the two boundary
vertices. For example, for n = 5, the connectivity graph is as shown in Figure 3.
  We say that a vector X = {X 1 , X 2 , . . . , X n } is a valid geometry in 1D if and only if b0 ≤ X 1 ≤
X 2 ≤ · · · ≤ X n ≤ b1 , where b0 and b1 are fixed boundary points. This is equivalent to saying that the
“chain” connectivity does not “fold” on itself.
4.2    The Geometry Distribution
Let U1 , U2 , . . . , Un be independent random variables, distributed uniformly on (a, b), and let a ≤ U(1) ≤
U(2) ≤ · · · ≤ U(n) ≤ b be these values after sorting them in increasing order. U(1) , U(2) , . . . , U(n) are called
uniform order statistics [David 1981]. We assume the valid geometries are distributed like the uniform
order statistics on (b0 , b1 ). Such a distribution will be uniform over all the valid geometries, because
every valid geometry, that is every order statistics vector, can be generated by n! permutations of the
original variables, which are uniformly distributed. The mean geometry for this distribution is the grid
geometry, where the distances between the vertices are equal.
4.3    The Optimal Basis
We now proceed to the main theorem in its one-dimensional version.
  THEOREM 1 The optimal basis for the decomposition of one dimensional meshes are the eigenvectors
{ i |i = 1 . . n} of the symmetric Laplacian, ordered such that the corresponding eigenvalues are µ1 ≤
µ2 ≤ · · · ≤ µn .
  As stated in Section 3, the optimal basis is the PCA, which is the matrix of the eigenvectors of the
covariance matrix of the mesh geometry distribution. Let X = {X 1 , X 2 , . . . , X n } be a random valid
geometry with boundaries b0 and b1 , and C its covariance matrix. Assume j ≤ i , since Cij = Cji . Now
by definition
                                            Cij = Exp(X i X j ) − Exp(X i )Exp(X j ).
  The first step in finding the optimal basis is computing the covariance matrix C. In order to do that,
we will need some more definitions:
  Define X 0 = b0 , X n+1 = b1 , and Y i = X i − X i−1 for i = 1 . . n + 1. Y i are called uniform spacings. It is
well-known [Pyke 1965] that Y i are interchangeable random variables, namely, that they are identically
distributed, and all pairs of variables have the same joint distribution:
                                 Var(Y i ) = Var(Y 1 ) = v(n)             ∀i ∈ {1, . . . , n + 1}
                                                                                                                 (4)
                           Cov(Y i , Y j ) = Cov(Y 1 , Y 2 ) = c(n)    ∀i, j ∈ {1, . . . , n + 1}, i = j
for some functions v(n) and c(n) which depend only on n and the boundaries b0 and b1 (but not on i and
 j ).
  LEMMA 1.1.        If X is a random valid 1D geometry, C its covariance matrix, and j ≤ i, then:
                                                     Cij = jv(n) + (ij − j )c(n).

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                               The Optimality of Spectral Compression of Mesh Data                                      •   65

  PROOF.    From the definition of Y i we have that:
                                                                                      i
                                                              X i = b0 +                  Yk.                                                               (5)
                                                                                  k=1
  Then, for j ≤ i, by substituting (5) in the covariance definition, and since the Y i are interchangeable
as defined in (4), we have:
                                                                                  i         j
             Cij = Exp(X i X j ) − Exp(X i ) Exp(X j ) =                                         [Exp(Y k Y r ) − Exp(Y k ) Exp(Y r )]
                                                                                k=1 r=1
                         j                                          i       j
                   =          Exp Y r2 − Exp(Y r )2 +                             [Exp(Y k Y r ) − Exp(Y k ) Exp(Y r )]                                     (6)
                       r=1                                         k=1     r=1
                                                                           r=k

                         j                  i     j
                   =         var(Y 1 ) +               cov(Y 1 , Y 2 ) = jv(n) + (ij − j )c(n).
                       r=1                 k=1   r=1
                                                 r=k


  We will now relate the variance and covariance of the uniform spacings, which will allow us to simplify
the expression in (6).
  LEMMA 1.2. If v(n) and c(n) are the variance and covariance, respectively, of uniform spacings as
defined in (4), then:
                                                  1
                                          c(n) = − v(n).
                                                  n
  PROOF.    Consider the following expression:
                                                                  2
                                                                                                              2
                                                       n+1                                       n+1
                                           Exp               Yi        − Exp                          Yi         .
                                                        i=1                                       i=1

  On the one hand, developing the expression, by expanding the sums and using the linearity of the
expectation, yields:
                     2
                                          2
                   n+1                           n+1                       n+1 n+1                                     n+1 n+1
           Exp          Yi      − Exp                Yi          =                       Exp(Y i Y j ) −                        Exp(Y i ) Exp(Y j )
                   i=1                           i=1                       i=1 j =1                                    i=1 j =1

                                                                           n+1 n+1
                                                                   =                       [Exp(Y i Y j ) − Exp(Y i ) Exp(Y j )]
                                                                           i=1 j =1
                                                                           n+1
                                                                   =                  Exp Y j − Exp(Y j )2
                                                                                            2

                                                                           j =1
                                                                                n+1 n+1
                                                                           +                     [Exp(Y i Y j ) − Exp(Y i ) Exp(Y j )]                      (7)
                                                                                i=1       j =1
                                                                                          i= j

                                                                           n+1                          n+1 n+1
                                                                   =              var(Y 1 ) +                          cov(Y 1 , Y 2 )
                                                                           j =1                         i=1   j =1
                                                                                                              i= j

                                                                   = (n + 1)v(n) + n(n + 1)c(n).
                                                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
66      •     M. Ben-Chen and C. Gotsman

  On the other hand, since the sum of the Y i is the constant (b1 − b0 ), we have:
                                2
                                                      2
                                   n+1                          n+1
                         Exp             Yi      − Exp               Yi     = (b1 − b0 )2 − (b1 − b0 )2 = 0.    (8)
                                    i=1                          i=1

  Comparing (7) and (8), we have:
                                                       (n + 1)v(n) + n(n + 1)c(n) = 0                             (9)
                                                               1
                                                       c(n) = − v(n)
                                                               n
  Note that the covariance of two spacings is always negative. Intuitively, this is because one spacing
may grow only at the expense of the other spacings, since their sum is constant.
  By using the two previous lemmas, we can now simplify the expression of the covariance, by substi-
tuting (9) in (6), to get:
                                                                v(n)
                                      Cij = j (n − i + 1)            for every 1 ≤ j ≤ i ≤ n.
                                                                 n
So, the covariance matrix C is
                                                         j (n − i + 1) v(n)    1≤ j ≤i≤n
                                               Cij =                    n
                                                                                                                 (10)
                                                        Cji                    1≤i< j ≤n
  Now let us examine the matrix product L · C. The valence d i of all the (interior) vertices is 2, since
every vertex has exactly two neighbors. Hence, the Laplacian is the n × n matrix:
                                                              
                                         2 −1      0 ···    0
                                                            .
                                                     ..     .
                                     −1     2 −1        .   .
                                                              
                                                ..            
                                 L= 0              .       0
                                                              
                                     .                        
                                     . ..                     
                                     .       . −1      2 −1 
                                         0 ···     0 −1     2 (n×n)
where n is the number of interior vertices.
  Note that the Laplacian has entries only for the interior vertices, so the first and last rows do not sum
to zero, since they belong to vertices neighboring on the boundaries. This property makes the Laplacian
invertible.
  By substituting (10) in the product L·C, it is easy to see that:
                                                                         k(n) i = j
                                                       (L · C)i, j =
                                                                         0    i = j,
where k(n) depends only on n, and is
                                                                       v(n)(n + 1)
                                                           k(n) =                  ,
                                                                            n
which implies that
                                                               L · C = k(n)I
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                          The Optimality of Spectral Compression of Mesh Data                     •      67




Fig. 4. 2D geometries with common connectivity and boundary: (a) An invalid 2D geometry. The highlighted triangle is folded
over. (b, c) Valid 2D geometries.

or
                                                          L = k(n)C −1 .
  From the optimality of PCA, we know that the eigenvectors { i |i = 1 . . n} of C, ordered such that
λ1 ≥ λ2 ≥ · · · ≥ λn are the optimal basis. Since the eigenvectors of C −1 are the same as the eigenvectors
of C, but the eigenvalues are reciprocated (µi = 1/λi ), the optimal basis for the decomposition of 1D
meshes are the eigenvectors of the symmetric Laplacian, ordered so the corresponding eigenvalues are
µ1 ≤ µ2 ≤ · · · ≤ µn .
  Note that the exact covariance matrix depends on k(n), which depends on n and the boundaries b0
and b1 .

5.    2D MESHES
5.1   The Model
In two dimensions, a geometry R = (X , Y ) is valid for the triangle mesh M = (G, R), G = (V , E) if the
triangulation of (X , Y ) using E does not contain intersecting edges, namely, the triangles do not fold
over onto each other, or, in other words, all have the same orientation. Assume M has a fixed boundary,
whose coordinates are
                                     (X b1 , X b2 , . . . , X bk ) = (xb1 , xb2 , . . . , xbk )
                                      (Y b1 , Y b2 , . . . , Y bk ) = ( y b1 , y b2 , . . . , y bk ),
forming a convex shape. A few examples of valid and invalid geometries for the same connectivity and
convex boundaries are shown in Figure 4.

5.2   The Geometry Distribution
To prove an optimality theorem such as Theorem 1 in the two-dimensional case, we have to impose
some natural probability distribution on the class of valid geometries. This is not easy, in general. For
the time being, we will assume that the following three properties hold for each coordinate of the valid
geometries’ distribution. Later we will justify these assumptions.
(1) X i |X j =i = x j is normally distributed for every i, meaning that the conditional distribution of the
    coordinates of one vertex given the coordinates of all the others is Gaussian.
(2) Exp(X i |X j =i = x j ) = di
                              1
                                 j ∈N (i) x j , meaning that the expectation of the Gaussian distribution
    mentioned in 1 is the average coordinates of just the neighbors of i.
                                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
68      •     M. Ben-Chen and C. Gotsman

(3) Cov(X i , X j | X k = xk , k = i, j, (i, j ) ∈ E) = 0, meaning that the covariance of the coordinates of every
    two vertices i, j which are not neighbors, conditioned on the coordinates of all other vertices, is zero.
  From now on, we will only refer to the X dimension, but all the theorems hold for the Y dimension
too.

5.3    The Optimal Basis
The 2D version of our optimality theorem is:
  THEOREM 2. If the distribution of valid geometries of 2D meshes has the properties defined in
Section 5.2, then the optimal basis for the decomposition of 2D meshes with connectivity graph G =
(V , E) are the eigenvectors { i |i = 1 . . n} of the symmetric Laplacian of G, ordered such that the corre-
sponding eigenvalues are µ1 ≤ µ2 ≤ · · · ≤ µn .
  We first show that if the distribution of valid geometries has the first two properties described in
Section 5.2, then it is multivariate normal. Note that just normal conditionals do not necessarily imply
multivariate normality. However, the following Lemma characterizes the multivariate normal distri-
bution via its conditionals:
  LEMMA 2.1. (ARNOLD ET AL. 1999). Let X = {X 1 , X 2 , . . . , X n } be a random vector. If:
(1) X i |X j =i = x j is normally distributed for every i, and
(2) Exp(X i |X j = x j ) is linear in x j and not constant, for every i,
then X has a multivariate normal distribution.
   It is easy to see that the conditions of the lemma are satisfied if we assume the distribution of valid
geometries has the first two properties described in Section 5.2. The first condition of the lemma is
identical to the first property described in Section 5.2. The second condition is implied by the second
property—if the conditional expectation of a vertex is the average of its neighbors, then it is linear in x j .
In addition, since the mesh is connected, and there are no isolated vertices, the conditional expectation
cannot be a constant with respect to the x j . Thus, the second condition of Lemma 2.1 also holds. Since
both the conditions of Lemma 2.1 hold, we conclude that the distribution of valid geometries (assuming
Section 5.2) is multivariate normal.
   Now that we have characterized the distribution of the geometries, we proceed to compute the covari-
ance matrix, which is the key for proving optimality. The next two lemmas characterize the structure
of K —the inverse of the covariance matrix. Combined, they show that K is essentially identical to the
mesh Laplacian.
   LEMMA 2.2. Let C be the covariance matrix of the X component of a random valid geometry R =
(X , Y ) of a mesh M = (G, R), G = (V , E). Let K = C −1 . Then for every (i, j ), such that i = j , and
(i, j ) ∈ E, K i, j = 0.
        /
  PROOF. We need the following lemma which describes a few known properties of the inverse covari-
ance matrix K of a multinormal distribution:
   LEMMA 2.3 (LAURITZEN 1996). Let X be a multivariate normal random variable X ∼ N (µ,                    ), and
let K = −1 . Then:
(1) K ij = −Cov(X i , X j |X k=i, j = xk )(K ii K jj − K ij ).
                                                         2

                                                                                           K
(2) Exp(X i |X j =i = x j ) = Exp(X i ) +           j =i   βij (x j − Exp(X j )), βij = − K ij .
                                                                                               ii
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                  The Optimality of Spectral Compression of Mesh Data                    •      69

Part 1 of Lemma 2.3, and the vanishing conditional covariance described in the third property of
Section 5.2 imply that if i and j are not neighbors, then K ij = 0.
   For the entries of K corresponding to the vertices and edges of the connectivity graph, we need the
following Lemma.
   LEMMA 2.4. Let C be the covariance matrix of the X component of a random valid geometry R = (X , Y )
of a mesh M = (G, R), G = (V , E). Let K = C −1 . Then, there exists a constant α, such that:
(1) For every (i, j ) ∈ E, K ij = −α
(2) For every i ∈ V , K ii = αd i

  PROOF.    From part 2 of Lemma 2.3, we know that
                          Exp(X i |X j =i = x j ) = Exp(X i ) +           βij (x j − Exp(X j )).
                                                                   j =i

On the other hand, we assumed in the second property of Section 5.2 that
                                                                   1
                                       Exp(X i |X j =i = x j ) =                     xj.
                                                                   di     j ∈N (i)

Since the linear coefficients of x j must be equal in both expressions, we have
                                                          K ij  1
                                                βij = −        = .                                                           (11)
                                                          K ii  di
C is a covariance matrix, so both C and K are symmetric, hence
                                           K ii                   K jj
                                                = −K ij = −K ji =                                                            (12)
                                           di                     dj
for every (i, j ) ∈ E.
  Consider the diagonal of K . It is easy to see that if the mesh is connected, all the values K ii /d i must
equal a constant α that does not depend on i: Define K 11 /d 1 = α. From (12) we have that for all the
neighbors j ∈ N (1), K jj /d j = K 11 /d 1 = α. The same can be done inductively for the neighbors of j and
so on. Finally, since the mesh is connected, every vertex i has a path to the first vertex, so it must hold
that:
                                                     K ii
                                                          =α                                              (13)
                                                     di
for every i ∈ V . Substituting (13) in (12) implies
                                                     K ij = −α                                                               (14)
for every (i, j ) ∈ E
 Combining Lemmas 2.2 and 2.4, we conclude:
 Let C be the covariance matrix of the X component of a random valid geometry R = (X , Y ) of a mesh
M = (G, R), G = (V , E). Let K = C −1 . Then K is:
                                              
                                               −α (i, j ) ∈ E
                                              
                                        K ij = αd i i = j
                                              
                                              
                                                0   otherwise
                                                                        ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
70      •     M. Ben-Chen and C. Gotsman

  Returning to the definition of the symmetric Laplacian in Section 1.1, we see that:
                                                              C −1 = K = αL,
where α is a constant that depends on n alone.
 As in the one-dimensional case, this concludes the proof of Theorem 2.
Note that the exact values of the covariance matrix depend on α, which depends on the fixed boundary
values.
  Our theorem makes some powerful assumptions on the distribution of valid 2D geometries. We will
now show why it is reasonable to make such assumptions, by describing a natural model for generating
valid 2D geometries, which turns out to have a distribution with the required properties.
  Following Tutte [1963], Floater [1997] proved that a 2D geometry with a convex boundary is valid if
and only if each vertex is a convex combination of its neighbors. This implies that a geometry (X , Y )
with a convex boundary (Bx , B y ) is valid, if and only if there exists a matrix W such that
                                                              X = WX + Bx ,
                                                              Y = WY + BY                                           (15)
where Bx and B y are
                                          xi   i ∈ {b1 , . . . , bk }                yi    i ∈ {b1 , . . . , bk }
                               Bxi =                                     Byi =
                                          0    otherwise                             0     otherwise
and W is
                                                     wij                     /
                                                              (i, j ) ∈ E, i ∈ {b1 , . . . , bk }
                                          Wij =                                                      .
                                                     0        otherwise
The weights wij are positive and normalized:
                                                    n
                                                                   /
                                                         Wij = 1 i ∈ {b1 , . . . , bk }.
                                                   i=1

We will call W a barycentric coordinates matrix.
  This characterization of valid 2D meshes yields a convenient way to define a probability distribution
on them. Instead of specifying the distribution of the valid geometries X (which seems to be quite diffi-
cult), we specify the distribution of the barycentric coordinates matrices. We assume that the barycentric
coordinates matrices are distributed as follows:
  For each interior vertex i, with valence d i , let
                                                     wij = D ij = U(ij +1) − U(ij ) ,                               (16)
where U(ij ) are d i − 1 order statistics over (0, 1), with U(0) = 0, and Ud (i) = 1. D ij are known as uniform
                                                             i             i

spacings [Pyke 1965]. This guarantees that the nonzero Wij are indeed positive and all the internal
vertices’ rows sum to one.
  Note that such a distribution is not guaranteed to generate a uniform distribution of valid geometries.
Since barycentric coordinates are not unique, that is, more than one set of barycentric coordinates can
generate the same geometry, the use of barycentric coordinates may introduce a bias which will prefer
certain valid geometries over others.
  We now address the following question: If the barycentric coordinates matrices W are distributed as
in (16), how are the geometries distributed? Given the matrix W and the boundary B, the geometry X
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                The Optimality of Spectral Compression of Mesh Data                                        •    71

can be expressed as
                                                           X = (I − W )−1 B,
where I is the identity matrix.
  LEMMA 2.5. Let X be the x coordinate of a random valid geometry, whose barycentric coordinates are
distributed as in (16). Then the distribution of X has the following properties:
(1) The limit distribution of X i |X j =i = x j as d i → ∞ is normal for every i,
(2) Exp(X i |X j =i = x j ) =   1
                                di   j ∈N (i)   xj,
(3) Cov(X i , X j |X k = xk , k = i, j ) = 0 for every two vertices i, j which are not neighbors.

  PROOF. From the definition of X and W in (15) and (16), respectively, it is easy to see that the
conditioned variables (X i |X j = x j ) are

                                                  (X i |X j = x j ) =                       D ij x j ,                                                         (17)
                                                                                 j ∈N (i)

where D ij are uniform spacings, N (i) is the set of neighbors of the i-th vertex, and the x j are constants.
The central limit theorem for functions of uniform spacings [Pyke 1965] implies that for vertices with
large valence:
                                                (X i |X j = x j ) ∼ Normal µi , σi2                                                                            (18)
where Normal is the Gaussian distribution. This proves the first part of the Lemma.
Since uniform spacings are interchangeable random variables which sum to unity:
                                                            di
                                                                   D ij = 1             ∀i
                                                           j =1

it follows that
                                                                                        1
                                            Exp D ij = Exp D1 =
                                                            i
                                                                                                       ∀ j.                                                    (19)
                                                                                        di
Substituting the expectation of the spacings (19), in (17), we obtain

                                                                                                                         1
               µi = Exp(X i |X j = x j ) = Exp                             D ij x j    =               Exp D ij x j =                    xj.                   (20)
                                                                j ∈N (i)                    j ∈N (i)
                                                                                                                         di   j ∈N (i)

Note that the x j are constants, since they are conditioned upon, so Exp(D ij x j ) = Exp(D ij )x j .
This proves the second part of the Lemma.
Let i, j be two nonadjacent vertices. Consider the covariance of X i and X j conditioned on the rest of
the vertices:
Cov X i , X j |X k = xk , k = i, j = Exp(X i X j |X k = xk ) − Exp(X i |X k = xk )Exp(X j |X k = xk )

                                     = Exp                       i
                                                                Dr xr                  Dm xm − Exp
                                                                                        j                                 i
                                                                                                                         Dr xr Exp                          j
                                                                                                                                                           Dm xm
                                                      r∈N (i)              m∈N ( j )                           r∈N (i)                         m∈N ( j )

                                     =                          Exp         i     j
                                                                           Dr xr Dm xm         −                       Exp    i
                                                                                                                             Dr xr   Exp Dm xm .
                                                                                                                                          j

                                         r∈N (i) m∈N ( j )                                         r∈N (i) m∈N ( j )
                                                                                            ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
72      •     M. Ben-Chen and C. Gotsman

                                  i                           j
Since i and j are not neighbors, Dr and Dm are disjoint sets of independent uniform spacings, which
implies that:
                                         Exp Dr xr Dm xm = Exp Dr xr Exp Dm xm
                                              i     j           i         j
                                                                                                                                 (21)
and thus,
                                              Cov(X i , X j |X k = xk , k = i, j ) = 0                                           (22)
for every nonadjacent i and j .
This proves the third part of the Lemma and concludes its proof.
   We now have a model for generating valid 2D meshes, which yields a distribution that has the prop-
erties required in Section 5.2, with just one problem—the first property of Section 5.2 requires a normal
conditional distribution, and all we have is a normal limit distribution as d → ∞. Central limit theo-
rems with asymptotic parameter d → ∞ give very good approximations already for modest values of d .
Here the asymptotic parameter is the valence, which is 6, on the average. The following experimental
results show that it seems to be large enough for the normal approximation to be reasonable.
   The Gaussian distribution of the coordinates can be seen in Figure 5, where some values of the
coordinates of a vertex are sampled from their distribution and visualized as point clouds, as normal
QQ-Plots [Rice 1988], and as a histogram.
   Figure 6 visualizes another experimental test of the conditional distribution of the vertices. Here,
for each vertex in the mesh, the conditional distribution for the vertex was calculated by fixing all its
neighbors, and randomizing the vertex’ values. Then a Bera-Jarque test for normality [Bera and Jarque
1981] was performed for each vertex. The symbol used to draw each vertex indicates what p value the
vertex received in the normality test.
   Since our result is an asymptotic result, an immediate question arises as to how good this approxima-
tion is in practice. Figures 7 and 8 show some experimental results, which give evidence to support the
quality of the approximation. Both figures compare the results obtained with the symmetric Laplacian
to the results obtained by an empirical covariance matrix. The empirical covariance matrix is computed
by generating a random geometries matrix X = {X 1 , . . . , X m }, where each X i is a geometry sampled
from the distribution described above. Then we compute COV(X ) = Exp(XXT ) − Exp(X )Exp(X T ),
where Exp(X ) is the mean vector of X . Figure 7 shows a color-coded map of the difference matrix
|L−1 − αC|/|αC|, where the division is performed element-by-element. The darker the color, the closer
this difference is to zero. It is evident that the two matrices are practically identical. Figure 8 shows
the normalized energy contained in the first k coefficients for the spectral decomposition of a geometry
vector, and the decomposition using the (eigenvectors of the) empirical covariance matrix. The higher
this energy is, the fewer coefficients we need to use to express more features of the geometry. It can be
seen that the spectral and the PCA bases give very similar results.
   A fundamental question is whether the distribution we have described above also applies to 3D
meshes. It will be difficult to answer this question. Instead, we investigated the similarity between
the population of 2D meshes governed by our distribution, and 3D meshes found in practice. For this,
we appeal to the following simple observation: the effectiveness of the spectral compression method is
                                                                                                                                   −→

Fig. 5. The distribution of a single vertex coordinate in a random 2D mesh, using the distribution of (16). Only one vertex is
sampled, while its neighbors and the rest of the mesh are fixed. (a)-(c) represent the distribution of one vertex, and (d)-(f) represent
the distribution of another vertex. (a), (d): Each point in the gray cloud represents the coordinates of this vertex in a random valid
geometry. The mesh connectivities are drawn in black lines using the mean valid geometry for the given connectivities (zoomed
in from the full mesh). (b), (e): The distribution of the x coordinate is visualized by normal QQ-plots. In such a plot, a normal
probability distribution will appear linear. (c), (f): The distribution of the x coordinate is visualized by histograms.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
The Optimality of Spectral Compression of Mesh Data             •      73




               ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
74      •     M. Ben-Chen and C. Gotsman




Fig. 6. For each vertex in the mesh, the conditional distribution of that vertex was measured by fixing all its neighbors, and
randomizing the vertex’ values. The Bera-Jarque test for normality [Bera and Jarque 1981] was performed. The vertices are
marked according to the resulting p-values. Unmarked vertices have p < 0.001.

based on the fact that, in practice, a vertex will be close to the average of its neighbours. For example,
a mesh geometry in which each vertex is located exactly at the average of its neighbours, contains
no information beyond the locations of the boundary vertices. Indeed, this is also the nature of the
mean of our 2D distribution. Hence, the distribution of the quantity “the normalized distance between
a vertex and the average of its neighbours” across a mesh is a key measure of the efficiency of spectral
compression on this mesh. We have computed and compared the distribution of this quantity for a
number of well-known 3D meshes, and also for 2D meshes sampled from our distribution. Figure 9
shows that these distributions are all very similar. While this does not prove anything about the more
subtle properties of the distributions, these results lend some support to our hope that our distribution
also reflects well the important properties of the 3D mesh distribution.
   It is worth mentioning that defining multivariate normal distributions by specifying the coefficients of
the conditional means is a well-known model [Besag and Kooperberg 1995], usually referred to as condi-
tional autoregressive, or auto-normal formulation. Such models frequently arise in Bayesian networks,
where they describe interactions between random variables that have neighborhood relationships.

6.   CONCLUSION, DISCUSSION, AND FUTURE WORK
We have shown that under a few natural assumptions on the distributions of geometries, given 1D
and 2D connectivities, the covariance matrix of the mesh geometries is the inverse of the symmetric
Laplacian matrix of that connectivity, and hence has the same eigenvectors (in reversed order). This
implies that spectral decomposition of meshes, which uses the eigenvectors of the symmetric Laplacian
as the basis, is optimal.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                         The Optimality of Spectral Compression of Mesh Data                •      75




Fig. 7. The difference matrix |L−1 − αC|/|αC|, color-coded. The darker the color, the closer to zero the value in the difference
matrix. The maximal (white) entry is approximately 3.4.




Fig. 8. The “energy” contained in the first k coefficients, for a mesh with n = 1000 vertices. The graph shows i=1…k ci / i=1..n ci ,
                                                                                                                     2          2

where ci are the coefficients of the decomposition by the PCA basis (computed using the empirical covariance matrix), and the
spectral basis. Note that this figure shows the coefficients’ energy for a single mesh, whereas our theory relates to the similarity
of the coefficients on the average over many possible valid meshes.

  On the practical side, to achieve an optimal decomposition, the geometries should be centered to
have a zero mean, before the decomposition. This is crucial since, as described in Section 3, PCA is
optimal only for a random vector X , which has a zero mean. The geometries as we described them do
not have such a property, therefore they should be centered before decomposition by subtracting away
their mean, which can be shown to be Exp(X ) = L−1 BX —the “Tutte configuration” with the given
boundary, namely, where each interior vertex is at the centroid of its neighbors..
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
76      •     M. Ben-Chen and C. Gotsman




ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                          The Optimality of Spectral Compression of Mesh Data                           •     77

  There are a few more directions that are worth exploring. These are described in the next sections.

6.1   Other Distributions
Our proof is based heavily on our assumptions about the distribution of meshes in one and two dimen-
sions. The distributions that we impose are obviously not the only ones possible. In one dimension,
one may use the same scheme as in two dimensions, and generate the geometries using a random
barycentric coordinates matrix. Another possibility is to define X i to be ij =1 Y j / n=1 Y j , where Y i
                                                                                             j
are uniformly distributed random variables. This is guaranteed to generate a monotonically increasing
sequence. The optimality proof in the one-dimensional case hinges on the key fact that the random
variables X i − X i−1 are interchangeable, hence the proof holds for any distribution that satisfies that
condition. Specifically, this is the case for both of the models just described, even though these do
not generate uniform distributions on the class of valid geometries. It is encouraging to see that the
optimality result is not too sensitive to the geometry distribution.
   In two dimensions, there are two main variations on our model. One possibility is not to use barycen-
tric coordinates at all: For example, one can generate random X and Y coordinates, and keep only the
(X , Y ) vector pairs that form a valid geometry. A geometry will be valid if all the triangles have the
same orientation. Obviously, this is not an efficient method to sample this distribution, since for large
n, the probability that a random (X , Y ) pair will form a valid geometry is very small. The advantage of
this process, however, is that it will generate geometries distributed uniformly over the class of valid
geometries, but it is not clear whether our proof extends to this distribution.
   Another possibility is to generate random 2D geometries by modifying the distribution of the barycen-
tric coordinates matrix. For example, instead of being uniform spacings, the barycentric coordinates can
be wij = Y i / i=1 . . d Y i , where Y i are independent, identically distributed, uniform random variables.
However, here too it is not clear whether the optimality result will still hold.
   Although our 2D optimality result was derived using properties of the normal distribution, which is
just an approximation of the true geometry distribution, we believe that the result actually holds for
the true distribution of the geometries, as in the 1D case, without resorting to normal approximations.
This, unfortunately, will probably require a completely different proof.
   For both 1D and 2D meshes, our proof is based on the fact that the inverse covariance matrix has
the same eigenvectors as the Laplacian, and there is no constraint on the eigenvalues as long as the
reverse ordering is preserved. This fact implies another way to apply our proof to other distributions:
one can prove that the inverse covariance equals to some integer power of the Laplacian, or any other
function of the Laplacian that doesn’t change the eigenvectors and the order of the eigenvalues of the
resulting matrix.

6.2   The Decay Rate of the Spectral Coefficients
For compression applications, it is important to know how fast the spectral coefficients decay to zero.
This property relates directly to the rate/distortion relation of the compression, and is also known as
“Energy Packing Efficiency”. Based on our proof that the Laplacian equals the inverse covariance up to a
constant, we can show that the spectral coefficients for the meshes treated here decrease on the average.
This, in itself, is an important fact, since a priori there is no reason they should even decrease. Let X
←−
                                                                                                              (xi −Avg (x j ))2
Fig. 9. The empirical distribution of the relative distance of a vertex from the average of its neighbours:   Avg ((xi −x j )2 )
                                                                                                                                 |j   ∈ Neigh(i),
where xi is the value of the X coordinate of the i-th vertex. The graphs show the histogram of this value, for (a) the “Cow” model,
(b) the “Beethoven” model, (c) the “Triceratops” model, and (d) 2D geometries sampled from the distribution described in this
article. All four distributions (a-d) are on the same plot, using linear (e) and log (f) scale.

                                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
78      •     M. Ben-Chen and C. Gotsman




Fig. 10. 3D geometries with common connectivity and boundaries. (a) The barycentric coordinates matrix is (I − L ), where L
is the Laplacian L normalized to unit row sums, and the displacement has constant variance for all vertices. (b) The barycentric
coordinates matrix is (I − L ), and the displacement is smooth. It was generated by randomizing displacement values on the
boundaries, and computing the displacement at each vertex by N = L−1 BN . (c, d) The displacement is computed as in (b),
but random barycentric coordinates matrices are used (e, f) Random barycentric coordinates matrix and random Gaussian
displacement with variance 1/n2 , where n is the number of vertices in the mesh.


be a random geometry (column) vector, and C = T X the corresponding spectral coefficient vector.
By definition,     is the orthogonal matrix of the column eigenvectors of L—the mesh Laplacian—in
increasing order of the eigenvalues: L =               , where    = diag(µ1 , . . , µn ) is the diagonal matrix
of L’s eigenvalues, in that order. We have proven that the geometry covariance matrix is Cov(X ) =
αL−1 . Hence Cov(C) = Cov( T X ) = T Cov(X ) = α T L−1 . Now, since L−1 =                       −1 T
                                                                                                     , we obtain
                   −1 T              −1
Cov(C) = α    T
                               =α       . This means that the spectral coefficients are pairwise uncorrelated
and decrease on the average. Since, by definition, Exp(X ) = 0, this implies that also Exp(C) = 0, so the
variance of C is (α/µ1 , . . , α/µn ), which obviously decreases.
  For the 1D, case, the exact eigenvalues of L are known, so we can find the decay rate of these
coefficients. In 1D, the Laplacian is a symmetric tridiagonal Toeplitz matrix, whose eigenvalues are
                                                           πi
known [Hartfiel and Mayer 1998] to be µi = 4 sin2 ( 2(n+1) ). For large n, the argument of the sin function
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                      The Optimality of Spectral Compression of Mesh Data               •      79

is very small, so the sin function may be approximated by its argument. This means that the inverse
eigenvalues, and hence the spectral coefficients, decay like θ (1/i 2 ).
6.3   Extension to 3D Meshes
In three dimensions, matters are more complicated. Just applying the barycentric method (15) to a
(nonplanar) 3D boundary results in a smooth, featureless surface interpolating the boundary vertices.
This is obviously not rich enough to capture interesting 3D shapes.
   A possible natural extension of the 2D model, which allows for 3D features to emerge, is the following
linear system: X = WX + N , where N is a random displacement vector, independent of X , and
the system has fixed boundary vertices. This means that each vertex is a convex combination of its
neighbors up to a displacement. The displacement can be any reasonable random variable, as long as
it is independent of X . The variance of the displacement values will influence the smoothness of the
mesh. A smooth mesh can be created by randomizing displacement values on the boundaries, and then
computing the displacement values on the internal vertices by computing N = L−1 BN , where L is the
Laplacian, and BN are the displacement values on the boundaries. The barycentric coordinates matrix
W can be generated by uniform spacings, as in the two-dimensional case. See Figure 10 for examples of
3D meshes with different displacements and different barycentric coordinates matrices based on this
model.
   The optimality proof for the 2D case is based on the following two key properties of the multivariate
random variable X : the normal distribution of the conditionals, and the linear conditional expectation.
Both those properties carry over to 3D, when using the model we have proposed, due to the displacements
N being independent of X .
   Obviously, such a 3D model is not really satisfying, since it cannot generate “real-life” meshes—no
cows will be born from random barycentric coordinates and displacements. An interesting experiment
may be to generate a family of geometries from a connectivity, for example by using the “Connectivity
Shapes” method [Isenburg et al. 2001]. Analyzing the distribution of geometries generated this way
may provide an insight for finding the optimal basis for “real-life” 3D meshes. Unfortunately, such an
analysis may be mathematically complex.

REFERENCES
                  ¨
ALEXA, M. AND MULLER, W. 2000. Representing animations by principal components. Comput. Graph. For. 19, 3 (Aug.), 411–418.
ALPERT, C. J. AND YAO, S. Z. 1995. Spectral partitioning: The more eigenvectors, the better. 32th ACM/IEEE Design Automation
  Conference, San Francisco, CA. (June) 195–200.
ARNOLD, B. C., CASTILLO, E., AND SARABIA, J. 1999. Conditional Specification of Statistical Models. Springer-Verlag.
BESAG, J. AND KOOPERBERG, C. 1995. On conditional and intrinsic autoregressions. Biometrika, 82, 4, 733–746.
BERA, A. K. AND JARQUE, C. M. 1981. An efficient large-sample test for normality of observations and regression residuals.
  Working Paper in Economics and Econometrics 40, Australian National University.
CHUNG, F. R. K. 1997. Spectral graph theory. Conference Board of Mathematical Science 92. American Mathematical Society.
CHUNG, F. R. K. AND YAU, S. T. 2000. Discrete Green’s functions. J. Combinator. Theo. A 91 191–214.
DAVID, H. A. 1981. Order Statistics. 2nd Ed. Wiley, New York.
FLOATER, M. S. 1997. Parameterization and smooth approximation of surface triangulations. Comput. Aided Geometr. Design,
  14, 231–250.
GOTSMAN, C., GU, X., AND SHEFFER, A. 2003. Fundamentals of spherical parameterization for 3D meshes. In Proceedings of ACM
  SIGGRAPH 2003.
GOTSMAN, C. 2003. On graph partitioning, spectral analysis, and digital mesh processing. In Proceedings of Solid Modeling
  International.
HALL, K. M. 1970. An r-dimensional quadratic placement algorithm. Manage. Science 17, 219–229.
HARTFIEL, D. J. AND MEYER, C. D. 1998. On the structure of stochastic matrices with a subdominant eigenvalue near 1. Linear
  Algebra and Its Appl. 272, 193–203.
                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
80      •     M. Ben-Chen and C. Gotsman

ISENBURG, M., GUMHOLD, S., AND GOTSMAN, C. 2001. Connectivity shapes. In Proceedings of Visualization 2001.
JOLLIFFE, I. T. 1986. Principal Component Analysis. Springer-Verlag, New-York.
LAURITZEN, S. 1996. Graphical Models. Oxford University Press.
KARNI, Z. AND GOTSMAN, C. 2000. Spectral compression of mesh geometry. In Proceedings of ACM SIGGRAPH 2000, 279–286.
KOREN, Y. 2003. On spectral graph drawing. In Proceedings of International Computing and Combinatorics Conference
  (COCOON’03). Lecture Notes in Computer Science, Vol. 2697, Springer Verlag.
PRESS, W. H., TEUKOLSKY, S. A., VETTERLING, W. T., AND FLANNERY, B. P. 1987. Numerical Recipes. Cambridge University Press.
PYKE, R. 1965. Spacings. J. Royal Statist. Soc. B, 395–439.
RAO, K. R. AND YIP, P. 1990. Discrete Cosine Transform. Academic Press.
RICE, J. A. 1988. Mathematical Statistics and Data Analysis. Brooks Cole Publishing.
ROSSIGNAC, J. 1999. EdgeBreaker: Connectivity compression for triangle meshes. IEEE Trans. Visual. Comput. Graph., 47–61.
TAUBIN, G. 1995. A signal processing approach to fair surface design. Proceedings of ACM SIGGRAPH ’1995, 351–358.
TAUBIN, G. AND ROSSIGNAC, J. 1998. Geometric compression through topological surgery. ACM Trans. Graph. 17, 2, 84–115.
TOUMA, C. AND GOTSMAN, C. 1998. Triangle mesh compression. In Proceedings of Graphics Interface, 26–34.
TUTTE, W. T. 1963. How to draw a graph. In Proceedings of the London Mathematical Society 13, 3, 743–768.


Received July 2003; revised February 2004, June 2004; accepted September 2004




ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
Stable But Nondissipative Water
OH-YOUNG SONG, HYUNCHEOL SHIN, and HYEONG-SEOK KO
Seoul National University


This article presents a physically-based technique for simulating water. This work is motivated by the “stable fluids” method,
developed by Stam [1999], to handle gaseous fluids. We extend this technique to water, which calls for the development of
methods for modeling multiphase fluids and suppressing dissipation. We construct a multiphase fluid formulation by combining
the Navier–Stokes equations with the level set method. By adopting constrained interpolation profile (CIP)-based advection,
we reduce the numerical dissipation and diffusion significantly. We further reduce the dissipation by converting potentially
dissipative cells into droplets or bubbles that undergo Lagrangian motion. Due to the multiphase formulation, the proposed
method properly simulates the interaction of water with surrounding air, instead of simulating water in a void space. Moreover,
the introduction of the nondissipative technique means that, in contrast to previous methods, the simulated water does not
unnecessarily lose mass, and its motion is not damped to an unphysical extent. Experiments showed that the proposed method
is stable and runs fast. It is demonstrated that two-dimensional simulation runs in real-time.
Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.6.8
[Simulation and Modeling]: Types of Simulation—Animation
General Terms: Algorithm
Additional Key Words and Phrases: CIP method, multiphase fluid, natural phenomena, Navier–Stokes equation, physically based
animation, semi-Lagrangian method, stable fluids, water




1.   INTRODUCTION
Water, which covers two thirds of the earth, undergoes myriad types of motion in its constant interac-
tions with air, solids, and living creatures. Water has featured prominently in several recent feature
animations, including Finding Nemo and The Perfect Storm. The success of these movies depended
greatly on visual effects in the animation of water. Physically-based approaches have been shown to
effectively reproduce water movement, with quite impressive results [Foster and Fedkiw 2001; Enright
et al. 2002].
   However, several open challenges remain in this field. One key issue is speeding up the simulation
of water. In the case of gaseous phenomena, interactive simulation methods already have been in-
troduced by Stam [1999]. The method is called “stable fluids”, which allows a large simulation time
step to be used without causing instabilities. Unfortunately, this method is known to suffer from large
amounts of numerical dissipation, which results in loss of mass. This is not important when simu-
lating dissipative media such as fog or smoke, but it is not tolerable when animating intrinsically

This research was supported by the Korea Ministry of Information and Communication. This research was also partially supported
by the Automation and Systems Research Institute at Seoul National University and the Brain Korea 21 Project.
Authors’ address: Graphics and Media Lab, School of Electrical Engineering #001, Seoul National University, San 56-1,
Shillim-dong, Kwanak-ku, Seoul, 151–741, Korea; email: song@graphics.snu.ac.kr.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0081 $5.00
                                                                ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 81–97.
82      •     O.-Y. Song et al.

nondissipative substances like water. Another undesirable property of the stable fluids method that
must be noted is numerical diffusion, which dampens the fluid motion. Although damping is an inher-
ent property of all fluids, the damping caused by numerical diffusion in the stable fluids method is too
severe. Therefore, if we wish to simulate water using an approach based on the stable fluids method,
we must modify that method to prevent the numerical dissipation and reduce the numerical diffusion.
   This article presents a new physically-based method for simulating water. The proposed method,
which is based on the semi-Lagrangian methodology, retains the speed and stability of the stable fluids
technique, while including mechanisms to fix the problems of numerical dissipation and diffusion. To
obtain nondissipative water, we adopt the constrained interpolation profile (CIP) method, which has
been shown to remarkably reduce dissipation due to the use of coarse grids. To prevent dissipation due
to the use of a large time step, we propose a novel particle-based approach, which we show to be quite
effective at preventing dissipation of small-scale features. This particle-based approach is also used to
simulate droplets and bubbles, which contributes to the overall visual realism. In addition, compared to
existing methods, the proposed method simulates water-air interactions more accurately by employing
the multiphase dynamic equations that account for the presence of air.
   The rest of the article is organized as follows: Section 2 reviews previous work; Section 3 formulates
the multiphase fluid; Section 4 describes the CIP-based fluid solver; Sections 5 and 6 present our
particle-based technique for preventing dissipation; Section 7 reports our experimental results; and
finally, Section 8 concludes the article.

2.   PREVIOUS WORK
Early work on physically-based simulation of water for graphics applications concentrated on animating
the height-field representation of the water surface. To obtain interactive performance, researchers
used the two-dimensional (2D) approximation of the Navier–Stokes equations. Kass and Miller [1990]
generated the height fields using an approximate version of the 2D shallow water equations. To simulate
water–object interactions, Chen and Lobo [1995] solved the 2D Navier–Stokes equation that includes
pressure. O’Brien and Hodgins [1995] proposed a method for simulating splashing liquids by integrating
a particle system into a 2D height-field model.
  Height fields cannot be used to represent water that is undergoing a highly dynamic motion such
as pouring. To handle such motions, researchers turned to the 3D Navier–Stokes equations. Foster
and Metaxas [1996, 1997a] animated 3D liquids by modifying the Marker and Cell method proposed
by Harlow and Welch [1965]. In addition, Foster and Metaxas [1997b] simulated gases by using an
explicit finite difference approximation of the Navier–Stokes equations. Stam [1999] introduced the
unconditionally stable fluid model, which utilizes the semi-Lagrangian method in combination with an
implicit solver. This model gave significantly improved simulation speeds, but suffered from numerical
dissipation. To reduce the dissipation in simulations of gaseous fluids, Fedkiw et al. [2001] proposed the
use of vorticity confinement and cubic interpolation. Based on the stable semi-Lagrangian frame-work,
Rasmussen et al. [2003] proposed an efficient method for depicting large-scale gaseous phenomena, and
Feldman et al. [2003] proposed an explosion model that incorporated a particle-based combustion model
into the semi-Lagrangian framework. Treuille et al. [2003] proposed a constrained optimization tech-
nique for keyframe control of fluid simulations and McNamara et al. [2004] improved the optimization
speed drastically by adapting the adjoint method.
  In order to handle 3D liquids, the semi-Lagrangian scheme must be augmented with a robust and ac-
curate method for tracking the liquid surface. To address this issue, Foster and Fedkiw [2001] proposed
a novel method for representing a dynamically evolving liquid surface, which was based on combining
the level set method with massless marker particles. Enright et al. [2002] improved this hybrid scheme
by introducing the “particle level set method”, which could capture water surface with a remarkable
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                    Stable But Nondissipative Water            •      83

accuracy. The particle level set method was employed in recent studies of fluids by Carlson et al. [2004],
Goktekin et al. [2004], and Losasso et al. [2004].
   Takahashi et al. [2003] simulated multiphase fluids by employing the CIP method, coupled with the
volume of fluid scheme; their method simulated the water-air interaction properly, instead of simulating
water in a void space. When we are to animate water at an interactive rate, as demonstrated by Stam
[1999] in the case of gas, then the use of large time steps should be allowed. But it can cause dissipation
of mass. In Foster and Fedkiw [2001] and Enright et al. [2002], the time step size had to be restricted
to prevent loss of mass. Although the CIP scheme used by Takahashi et al. [2003] lessened the degree
of the dissipation, loss of mass was still noticeable when large time steps were used.
   Several particle-based methods have been proposed as alternatives to the above grid-based ap-
proaches. Miller and Pearce [1989] simulated fluid behavior using particles connected with viscous
springs. Terzopoulos et al. [1989] adopted a molecular dynamics model to simulate particles in the liq-
uid phase. Stam and Fiume [1995] introduced “smoothed particle hydrodynamics” (SPH) to depict fire
and gaseous phenomena. In SPH, the fluid is modeled as a collection of particles with a smoothed poten-
                 z
tial field. Premoˇ e et al. [2003] introduced the use of the moving particle semi-implicit method (MPS)
for simulating incompressible multiphase fluids. One drawback of particle-based methods is that, if
insufficient particles are used, they tend to produce grainy surfaces. To prevent this, a sufficiently large
number of particles must be used, which increases the computational cost.

3.    FORMULATION AND OVERVIEW
In order to produce realistic movement of water in the presence of air, we base our method on the mul-
tiphase Navier–Stokes equations in combination with the level set method. The multiphase Navier–
Stokes equations can simultaneously represent both water and air. The level set method, which can
represent the water-air interface as an implicit surface, has been shown to be a robust method for
capturing topological changes of water surfaces. Furthermore, the surface curvature can be accurately
calculated from the level set values, and hence the surface tension, which is proportional to the curva-
ture, can be easily incorporated into the dynamic simulation.
   We start by introducing the incompressible Navier–Stokes equations for a multiphase fluid. Let
u = (u, v, w) denote the velocity field of the fluid. Then, the flow of fluid is described by

                                                 ∇ · u = 0,                                                          (1)

and
                                    ∂u            f ν        ∇p
                                       = −u · ∇u + + ∇ 2 u −    ,                                                    (2)
                                    ∂t            ρ ρ        ρ
where p is the pressure, ρ is the density, ν is the kinematic viscosity, and f represents the external forces
per volume. Equations (1) and (2) state that mass and momentum, respectively, should be conserved. To
treat the immiscible multiphase fluid consisting of water and air within a single formulation, we employ
the level set function φ [Osher and Sethian 1988; Sussman et al. 1994]. φ is an implicit signed distance
function defined to be positive for water, and negative for air. Thus, the sign of φ also determines the
density and viscosity of the medium.
  The water dynamically evolves in space and time according to the underlying fluid velocity field u.
The updates in the level set values due to u are expressed by the level set equation:
                                             ∂φ
                                                + u · ∇φ = 0.                                                        (3)
                                             ∂t
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
84      •     O.-Y. Song et al.

The surface of water, which will be a focal point throughout this work, can be obtained by tracking the
locations for which φ = 0.
  To solve the above equations numerically, we divide the space into a finite number of cells.We evaluate
the pressure and level set values at the cell center, but we evaluate the velocity at the center of each
cell face. This approach is the classical staggered grid discretization [Harlow and Welch 1965], which
naturally enforces boundary condition and avoids the checkerboard pattern of the pressure [Trottenberg
et al. 2001]. At each time step, our simulator performs the following three steps:

(1) Advect the level set: The level set φ is advected according to Equation (3), which causes the density
    and viscosity fields appearing in Equation (2) to be updated.
(2) Update the velocity: Equation (2) is solved for u using the following procedure [Stam 1999]: (1)
    calculate the advection component u · ∇u using the semi-Lagrangian method; (2) apply the forces
    f/ρ; (3) add the effect of the viscous term ν/ρ∇ 2 u by employing implicit central differencing; and
    (4) project the velocity field so that the condition ∇ · u = 0 is met.
(3) Simulate droplets/bubbles: Droplets/bubbles are identified and simulated using the particle dy-
    namics until they merge into the body of water/air.
Execution of the above procedure produces the φ and u of the next time step. The method for implement-
ing Steps 1 and 2 is presented in the following section, and the implementation of Step 3 is described
in Section 5.

4.    CIP-BASED FLUID SIMULATOR
The framework of our fluid simulator is based on the semi-Lagrangian scheme, which was briefly
introduced in Section 3. A detailed description of the semi-Lagrangian scheme can be found in Stam
                            oe
[1999] and Staniforth and Cˆ t` [1991]. In the method proposed here, we make several modifications to
the previous semi-Lagrangian scheme to reduce the numerical dissipation and diffusion. This section
describes those modifications.

4.1    CIP Advection
In the semi-Lagrangian scheme, advection is implemented by referring to the function value at x−u t.1
Since the physical values of φ and u are defined only at discrete points, the function values for x − u t
can be obtained by linearly interpolating the neighboring grid points. This approach is computationally
efficient and unconditionally stable [Stam 1999]. However, it may smooth out the subcell features. This
problem, referred to as nonphysical numerical diffusion, causes the movement of fluid to be excessively
damped, which hampers the generation of turbulent effects such as the formation of water droplets or
air bubbles in violently interacting multiphase fluids. It also causes dissipation of the mass.
   Fortunately, an anti-diffusive technique called the Constrained Interpolation Profile (CIP) method
is proposed by Yabe and Aoki [1991] and Yabe et al. [2001]. We adopt the CIP method when solving
Equation (3), resulting in a reduction of mass dissipation, and also when solving the advection term
u · ∇u in Equation (2), leading to a reduction in the degree of damping. Adoption of the CIP method
allows us to simulate phenomena such as turbulent flows or swirls with reasonable visual quality.
   The key idea of the CIP method is to use not only the function values at the grid points, but also
the spatial derivatives at those points for constructing the profile inside the grid cell. For example, in
the one-dimensional case, the profile corresponding to [xi , xi+1 ] now has four conditions φi , φi , φi+1 , and

1 In fact, more sophisticated backtracking methods can be used. In this work, we used the second order Runge-Kutta backtracking.

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                               Stable But Nondissipative Water            •      85

φi+1 , and can be represented by the third order polynomial

                                            (X ) = [(a X + b)X + φi )]X + φi ,                                                  (4)
where X = x − xi for x ∈ [xi , xi+1 ]. The coefficients a and b can be expressed in terms of the four
conditions:
                                          a = (φi + φi+1 )/ x 2 − 2 φ/ x 3 ,
                                           b = 3 φ/ x 2 − (2φi + φi+1 )/ x,
where x = xi+1 − xi and φ = φi+1 − φi . The spatial derivatives used in the CIP method are directly
determined from the differentiated version of the original advection equation.
  To advect the level set values, we differentiate Equation (3) with respect to ξ , which gives
                                               ∂φξ
                                                   + u · ∇φξ = −uξ · ∇φ,                                                        (5)
                                                ∂t
where φξ = ∂φ/∂ξ , uξ = ∂u/∂ξ , and ξ is one of the spatial variables (x, y, z). The task of solving the above
equation for φξ can be performed in two steps: first, solving the nonadvective part ∂φξ /∂t = −uξ · ∇φ
using finite differencing; then advecting the result according to
                                                    ∂φξ
                                                        + u · ∇φξ = 0.                                                          (6)
                                                     ∂t
Noting that the advections of Equations (3) and (6) are driven by the same velocity field u, we can
advect both φ and (φx , φ y , φz ) by referring to the same point of the profile. For the one-dimensional case,
suppose that the grid point x j is backtracked to xr ∈ [xi , xi+1 ]. Then, the advected results φ j and φ j can
be obtained by evaluating Equation (4) and its differentiated form at X r = xr − xi . Namely, φ j = (X r )
and φ j = (X r ).
   A problem can arise if we use the profile of Equation (4) as it stands. In the one-dimensional case, for
example, the value (X r ) may lie outside the range [φi , φi+1 ], which can cause instabilities. One solution
to this problem is the rational CIP method proposed by Xiao et al. [1996], which suppresses the above
oscillation problem by using a rational function instead of a polynomial function. In the present work,
we developed an alternative CIP scheme that explicitly modifies the derivatives, with this modification
being applied only to the cells in which oscillations are present. This scheme, which is described in
detail in the Appendix, guarantees a monotonic profile; hence, we call it monotonic CIP. Because it uses
polynomials, monotonic CIP runs faster than rational CIP.
   Higher (i.e., two- or three-) dimensional CIPs have been proposed by Yabe et al. [2001]. In those
methods, however, the derivative constraints are not applied at every grid point, which can result in
a nonmonotonic profile. Here we implement higher dimensional CIPs based on the one-dimensional
monotonic CIP solver. Our implementation of higher dimensional CIPs, which is described below, is
always stable. Consider the two-dimensional case shown in Figure 1(a), where r is the backtracked
point. For this system, we must determine [φ, φx , φ y , φxy ]r from the values at the four corners: (i, j ),
(i + 1, j ), (i, j + 1), and (i + 1, j + 1). Here, [·]x denotes the function and derivative values at point
x. The one-dimensional solver can determine [φ, φx ]a from [φ, φx ](i, j ) and [φ, φx ](i+1, j ) , and [φ y , φxy ]a
from [φ y , φxy ](i, j ) and [φ y , φxy ](i+1, j ) . Similarly, it can determine [φ, φx , φ y , φxy ]b . Then, it can determine
[φ, φ y ]r from [φ, φ y ]a and [φ, φ y ]b , and [φx , φxy ]r from [φx , φxy ]a and [φx , φxy ]b .
   In the three-dimensional case, the CIP implementation additionally calls for the values of φxy , φ y z ,
φxz , and φx y z . Obtaining those values by analytic differentiation of the original formula involves a large
amount of computation. However, our experiments indicated that approximating the second order and
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
86      •      O.-Y. Song et al.




Fig. 1. (a) Two-dimensional CIP interpolation; the x-axis is along the horizontal direction and the y-axis is along the vertical
direction. (b) Schematic of a situation where φi−1 , φi , and φi+1 are aligned.

higher derivatives by finite differencing of the first order derivatives caused no visually noticeable
difference in the result, but significantly reduced the amount of computation.
   We close this section by noting the attractive aspects of CIP in comparison with other interpola-
tion methods. In cases where φi−1 , φi , and φi+1 are aligned, such as that shown in Figure 1(b), spline
techniques that do not utilize the derivative information interpret φ as being straight. In contrast,
because CIP utilizes the spatial derivatives φi−1 , φi , and φi+1 of the original equation, it results in more
accurate modeling of the real situation. Therefore, the CIP method allows us to use a fairly coarse
grid. Another advantageous feature of the CIP method is that, whereas high-order interpolation meth-
ods for the semi-Lagrangian scheme such as the cubic spline, quintic Lagrange [Staniforth and Cˆ t`        oe
1991], and monotonic cubic polynomial [Fedkiw et al. 2001] methods require three stencils (or four grid
points) to construct the profile in the one-dimensional case, the profile used in the CIP method can
be constructed with information from a single cell. This feature is particularly useful when treating
boundaries.

4.2    Force
We now apply the external forces, f, to the result of the advection. The external forces consist of gravity
and surface tension. In the interactive system we develop, mouse or keyboard input can also be used
to generate additional forces. The gravitational force is expressed as ρg, where g is the gravitational
acceleration. The surface tension is given by
                                                      fst = −ρσ κ(φ)δε (φ)∇φ,                                               (7)
where σ is a constant coefficient, κ is the local curvature given by
                                                                             ∇φ
                                                          κ(φ) = ∇ ·             .                                          (8)
                                                                            |∇φ|
δε is a smeared delta function which, in the present work, has the form
                                                                           πφ
                                                       1
                                                            +   1
                                                                     cos    ε
                                                                                :   |φ| ≤ ε,
                                         δε (φ) =      2ε       2ε                                                          (9)
                                                      0                         :   otherwise,

where we use ε = 1.5 x for the smearing width. Equation (7) is the continuum surface force (CSF)
model proposed by Brackbill et al. [1992]. It allows us to treat surface tension as a body force, and does
not require explicit information on the geometry and position of the water surface.
  In order to show surface tension effects accurately, a strict time step restriction of O( x 1.5 ) is required
[Brackbill et al. 1992; Sussman et al. 1994]. Since we do not restrict time steps in this work, the effects

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                                               Stable But Nondissipative Water          •      87

may be somewhat suppressed. In large scale liquid phenomena such as the ones experimented in this
article (Section 7), however, the surface tension effects are visually not significant.

4.3   Projection
The result obtained by applying the external forces then goes through the diffusion step, ν/ρ∇ 2 u, for
which we use the implicit solver following the approach described in Stam [1999]. To process the last
term of Equation (2), −∇ p/ρ, we impose the fluid incompressibility condition represented in Equa-
tion (1), which produces the Poisson equation:
                                                                ∇p             ∇ ·u ˜
                                                       ∇·                =            ,                                                     (10)
                                                                ρ                 t
      ˜
where u is the intermediate velocity field obtained by processing Equation (2) up to the diffusion term.
Equation (10) can be discretized as
                                                 −1      −1         −1           −1
                                                ρn+ 1 + ρn− 1 pn − ρn+ 1 pn+1 − ρn− 1 pn−1
                                                   2            2                  2                 2
                                   n={i, j,k}
                                                                                                                                            (11)
                                                 =−    1
                                                        t
                                                                         un+ 1 − un− 1 ,
                                                                         ˜       ˜
                                                                               2           2
                                                            n={i, j,k}

where pn±1 are the pressure values taken from the centers of the neighboring cells, and un± 1 and ρn± 1
                                                                                            ˜
                                                                                                2        2
are the velocity and density values taken from the cell faces. We can assemble the collection of equations
of the form of Equation (11), covering the domain space into a large linear system
                                                                 Ap = b,                                                                    (12)
where p is the vector of unknown pressures required to make the velocity field divergence free. A
is a positive-definite, symmetric matrix in which density is a variable in space. This system can be
efficiently solved using the incomplete-Cholesky preconditioned conjugate gradient method (ICCG)
[Golub and Loan 1996].2 The solution of Equation (12) is then used to calculate ∇ p at the cell faces.
Finally, we obtain the divergence free velocity field by
                                                                               ∇p
                                                            u=u−
                                                              ˜            t      .                                                         (13)
                                                                               ρ

5.    PREVENTION OF DISSIPATION IN SMALL-SCALE FEATURES
Although the CIP-based simulator described in the previous section can represent the content of each
cell up to the third order, it still suffers from the following two problems: (1) when a large time step is
used, it produces non-negligible mass errors; and (2) it cannot represent subcell-level features such as
water droplets and air bubbles. In this section, we develop a novel technique to solve these problems.
The proposed technique uses particles3 to complement the grid-based framework, thereby overcoming
some of the limitations of that framework.

2 Because  we use the conjugate gradient method for solving the linear system which is the most time consuming part, the overall
time complexity order in our method is approximately O(n3/2 ) in the two-dimensional case and O(n4/3 ) in the three-dimensional
case, where n is the total number of grid points.
3 Note that our use of particles should not be confused with that proposed by Enright et al. [2002]. They use massless marker

particles to obtain a more accurate water surface, whereas we use physical particles that have mass as well as volume to represent
water droplets and bubbles. Compared to their method, our method requires far fewer particles and, as a consequence, has a
negligible computational overhead.


                                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
88        •    O.-Y. Song et al.




Fig. 2. Identification of droplets: (a) Apparent droplet, (b) Unapparent droplet. In (b), δ is a small positive value such that δ < ε.

5.1       Identification of Dissipative Cases
If the level set value of a particular cell center is positive, whereas those of adjacent cell centers are
negative, we can interpret the isolated region to be a droplet as shown in Figure 2(a). A potential
problem arising from having such a small isolated region is that it may dissipate or even be lost in
subsequent simulation. In such cases, therefore, we transform the region into a droplet that undergoes
Lagrangian motion from that moment onwards. We determine the volume4 of the droplet based on the
level set values. The approximation we use is

                                      Vf =           H(φ(x))d x ≈       Hε (φ(xc )) x y z,                                     (14)
                                                 c                  c

where the set c represents the droplet (the circular region of Figure 2(a)), c is the index ranging over
the cells that cover c (the shaded cells of Figure 2(a)), and Hε is the smeared Heaviside function5
                                      
                                      0
                                                              : φ < −ε
                                              φ            πφ
                              Hε (φ) = 2 + 2ε + 2π sin ε
                                         1         1
                                                               : |φ| ≤ ε                            (15)
                                      
                                      1                       : φ > ε.
As in Equation (9), we use ε = 1.5 x. Since the fluid content of the isolated region is already transformed
into a droplet, we reduce the level set values of the region to values that are small enough (e.g., −ε)
that the region will not be mistakenly identified as containing water during subsequent processing.
Generation of bubbles can be thought of as the dual of that of droplets; identification of isolated regions
and approximation of bubble volume can be done with the same procedure and same equations, except
that, in this case, we use the negated version of original level set values.
  In addition to the cases in which the isolated region can be identified by simply looking at the signs of
the level set values, referred to here as “apparent” cases, there also exist “unapparent” cases in which
the isolated region cannot be identified from the sign of the level set values. For example, consider the
case in which a 2 × 2 grid with small negative level set values is surrounded by the cells with large
negative level set values, as shown in Figure 2(b). Since the level set function we use represents signed
distance, small negative values imply that water is nearby. Therefore, the situation considered here can
be reconciled only by introducing a droplet somewhere around ?-marked location of Figure 2(b). The
volume of the droplet is again computed by Equation (14). During dynamic simulation of a multiphase

4 Since  we assume incompressibility of both media, mass-preservation is equivalent to volume-preservation.
5 We   also smear the discontinuities in density/viscosity across the air-water interface using this function.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                            Stable But Nondissipative Water         •      89

fluid, the above situation can occur when a body of water is splitting (or two bodies of air are merging).
Cases of unapparent bubbles are treated similarly.
   There is another source of errors of misinterpreting the phase, which is related to the backtracking in
semi-Lagrangian advection. Specifically, when a large time step is used, backtracking may leave some
cells never referred. In such cases, the masses of those cells are lost in the subsequent time steps. We
prevent this type of error by converting the nonreferred cells into droplets/bubbles. Again, the volumes
of the cells are determined by evaluating Equation (14) for the nonreferred cells. The droplets/bubbles
are then advected by the underlying velocities. Droplets/bubbles that are advected into the water/air
medium are ignored.
   In pure grid-based methods, the above cases are beyond the resolution limit and thus no droplet/bubble
is formed. By introducing a small amount of extra computation, however, the procedures described above
can generate droplets/bubbles. This not only reduces the mass dissipation but also enhances the visual
details of the dynamically evolving water.
   The volume approximation of Equation (14), which uses the smeared Heaviside function given in
Equation (15), theoretically has first-order accuracy. Our experiments showed that it is computationally
efficient and gives sufficient visual realism for our purposes. If greater accuracy is needed, contouring
methods such as the marching cube algorithm [Lorensen and Cline 1987] can be used.

5.2   Dynamic Simulation of Droplets/Bubbles
As a droplet/bubble advances with the initial velocity, it experiences gravitational and drag forces, as
well as pressure from the surrounding fluid, which cause the fragment to accelerate or decelerate. The
forces acting on the fragment can be summarized as

                                 f = m f g + αd r 2 (u − x) u − x − V f ∇ p,
                                                         ˙      ˙                                                       (16)
where m f is the mass, V f is the volume, αd is the drag coefficient, r is the radius, x is the current velocity
                                                                                      ˙
of the fragment, and u is the interpolated velocity of the grid-based fluid measured at the center of the
fragment. The third term, which represents the force due to the pressure difference, produces buoyancy.
In Equation (16), the second and third terms model the interaction with the neighboring fluid. Therefore,
the action force given in Equation (16) must be coupled with the reaction force −αd r 2 (u − x) u − x +
                                                                                                   ˙      ˙
V f ∇ p, acting on the grid-based fluid model. The movement of fragments over time is obtained by
standard numerical integration. If two or more fragments overlap during the dynamic simulation, they
are merged into a single larger fragment.

5.3   Restitution of Droplets/Bubbles
When either of the following two conditions are met, we restitute the droplets/bubbles to the grid-based
fluid model: (1) when the volume of a fragment becomes larger than twice the cell size, or (2) when a
fragment hits the surface or moves into the same phase fluid.
  In the first case, the fragment has become too big to be represented as a nondeforming particle,
and thus its behavior is better modeled within the grid-based framework. Therefore, in such cases, we
remove the fragment and restitute its mass to the grid-based fluid model. This restitution is executed
by updating the level set values and setting the grid point velocities to the fragment velocity. We update
the level set values by
                                         φ(xi ) = sp (r p − |xi − x p |),                                               (17)
where sp = +1 for the case of a water droplet, and −1 for the case of an air bubble, r p is the radius of
the fragment, x p is the center of the fragment, and xi is the grid point being updated.
                                                                   ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
90      •     O.-Y. Song et al.




      Fig. 3. Instantaneous hollows and ripples on the water surface created by restitution of Lagrangian droplets/bubbles.


  The second case corresponds to the situation in which a droplet/bubble returns to the body of water/air,
and, therefore, we remove the fragment. In the case where the fragment hits a surface cell, the cell
velocity is updated by taking the average of the previous cell velocity and the fragment velocity. As
for the level set values, we determine the new values for the cells covering the fragment by taking the
inverse functions of Equations (14) and (15). In the case where the fragment moves into the same phase
fluid, we perform the same procedure as described above, pretending that it hit a surface cell.
  The above procedure we devised for updating the level set values and cell velocities, interestingly,
contributes to creating visual details at water surface. The procedure in fact forms small ripples: a
droplet falling into water contributes a downward velocity to the cell, which generates an instantaneous
hollow. But soon, the region is pushed back and forms a small bump (See Figure 3.).

5.4    Variations for Visual Realism
In the above procedures, several parameters can be controlled to adjust the visual aspects of the
simulation. (1) In order to produce splashes and bubbles of smaller scales, we can generate multi-
ple droplets/bubbles of smaller size, instead of generating a single fragment of about the cell size. In
practice, we let the number of generated fragments be proportional to the velocity of the fluid portion
being converted. (2) We can model the geometrical shape of a fragment as an ellipsoid instead of a
sphere, in accordance with the velocity. (3) In order to represent foam at the water surface, instead
of removing bubbles immediately after they appear on the surface, we assign them a life time that is
inversely proportional to the bubble size.


6.    ADDITIONAL CONSIDERATIONS
6.1    Reinitialization of Level Set Values
As the level set is advected and makes interactions with the droplets/bubbles as described in the previous
section, the signed-distances can be corrupted. This may lead to the development of noisy features that
can be problematic when approximations such as finite-differencing are used. For example, such noisy
features can lead to large errors in the curvature calculation. For this reason, we need to introduce a
procedure to rectify the level set to ensure that it maintains the signed distance property. This procedure
is called reinitialization or redistaincing [Sussman et al. 1994, 1998; Peng et al. 1999; Sethian 1996;
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                        Stable But Nondissipative Water            •      91

Tsai et al. 2003]. We used Sussman’s method [1998]. The method is performed using
                                        ∂φ
                                           + sgn(φ)(|∇φ| − 1) = 0,                                                     (18)
                                        ∂τ
where τ is a fictitious time and sgn(φ) is a smeared signum function given by
                                                             φ
                                       sgn(φ) =                           .                                            (19)
                                                     φ2   + |∇φ|2 ( x)2
In most cases, the corruption is not severe and the procedure recovers the correct signed distance values
within several fictitious time steps. Hence, the reinitialization step is computationally inexpensive.
We can speed up the procedure by reinitializing only the portion within a band of the water surface
[Sussman et al. 1998]. Details on the discretization of Equation (18) can be found in Sussman et al.
[1998] and Osher and Fedkiw [2002].

6.2   Interaction with Rigid Objects
This section presents a simple technique for simulating the influence of water on rigid objects, as well
as that of rigid objects on water. We employ the volume of solid technique proposed by Takahashi et al.
[2003] with modifications inspired by the work of Foster and Fedkiw [2001] and Enright et al. [2002].
The proposed technique may not produce valid results in an engineering sense, but it does reproduce
the physical aspects of object-water interactions with acceptable visual realism.
   When a rigid object is immersed in water, we mark the fluid cells whose centers are contained within
the object. A self-evident constraint in this case is that the fluid should not flow into the rigid object;
this can be checked for each marked cell s by calculating us · ns , where us is the fluid velocity, and ns
is the normal of the object surface. If us · ns < 0, then the fluid is flowing into the solid and, to stop
this, we remove the normal component of us , while leaving the tangential component unchanged. The
above procedure is for calculating the change in fluid velocity caused by the presence of an object. The
movement of an object caused by the surrounding fluid is simulated by considering the fluid pressure
acting on the surface of the object. Given that the gradient of the pressure generates forces, the total
external force F and moment T acting on the center of mass rc of the rigid object can be expressed as


                                       F = Mg +          (−∇ ps · ns )ns S,                                            (20)
                                                     s

                                  T=        (rs − rc ) × (−∇ ps · ns )ns S,                                            (21)
                                        s

where M is the mass of the object, s is the index ranging over the marked cells, ps is the fluid pressure
of the cell, rs is the position of the cell, and S is the area of the object surface subsumed in the cell.
When simulating water that is interacting with a quickly moving rigid object, a reasonably small time
step size must be used to calculate the movement of rigid object in a moderate level of accuracy.

6.3   Discussion on Mass Errors
The procedures described in Sections 4 and 5 are quite effective at blocking mass dissipation in the
numerical simulation of multiphase fluids. However, the errors resulting from the use of finite-sized
grids and finite time steps are inevitable, and in rare cases, these errors can accumulate to some
extent. In situations where the amount of water needs to be kept constant, we optionally perform a
global compensation procedure based on modification of the level set values at every time step.
                                                                  ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
92      •     O.-Y. Song et al.

  If the initial volume of water is V0 , and the volume at each time step is Vi , we should compensate
the water volume by Vi = V0 − Vi at the end of each simulation step. The volume Vi and the area of
water surface Si can be calculated using

                                    Vi =       H(φ(x))d x ≈        Hε (φ(x)) x y z + V p ,                                 (22)


                                Si =       δ(φ(x))|∇φ(x)|d x ≈        δε (φ(x))|∇φ(x)| x y z,                              (23)

where V p is the total volume of the droplets. Because the level set is a signed distance function, we can
use the approximation
                                                              Vi ≈ Si φ,                                                   (24)
which allows us to calculate the scalar value φ. Now, adding φ to the previous level set values of the
entire domain (for both water and air) results in raising the water level by φ, while not changing the
geometrical shape of the water surface.
   The above compensation procedure can have a first-order error, since we use the smeared Heaviside
function. However, the compensation procedure performed at every time step is done with respect to
V0 , and the error is distributed over all of the cells in the system. Therefore, even though the procedure
is not physically-based, it effectively prevents the error accumulation without producing any noticeable
artifacts.
   The contribution of this procedure in the prevention of mass dissipation is small. For example, in the
2D water drop experiments on a grid of resolution 64 × 48, the procedure compensated only 0.7% of
errors. Furthermore, contribution from the global compensation decreases as we increase the resolution
of the grid. In fact, in obtaining the 3D demos, we did not run the compensation procedure.6

7.   EXPERIMENTAL RESULTS
The technique presented in this article is implemented in two dimensional and three dimensional
versions, on a PC with an Intel Pentium 4 3.2 GHz processor, 1 GB memory. The innovations introduced
in the present work make possible the real-time simulation of the movement of 2D water. Moreover, the
fast simulation speed of the proposed method enabled highly interactive control of the water motion
through several types of action. For example, by clicking or dragging the mouse, we could add a volume
of water, create/remove solid walls, or introduce force fields, all with immediate effect. This section
summarizes our experimental results.
  The 2D simulator based on a grid of resolution 64 × 48 ran at 30 ∼ 60 fps, which is sufficiently fast to
enable real-time simulation of water that is subjected to interventions such as applying forces or adding
water volumes. Clip 1 contains a real-time video capture taken during the simulation (see Figure 4 for
a snapshot). Although the CFL number (i.e., u t/ x) was 25 or higher, the simulation ran stably. The
mass error is not evident, and the movement of water is clearly nondissipative. We were able to let
ink mix over the water medium, as shown in Stam [1999], which provided a perceptual cue on the
movement of water. The clip shows (1) the movement of water at different gravitational conditions, (2)
the generation of droplets/bubbles in turbulent water, (3) the behavior of water when interacting with
solids, (4) changes in buoyancy due to changes in the density of the fluid, (5) the result of replacing the
CIP advection with linear advection, which noticeably increases dissipation and damping, and (6) the
interactions water makes with nonvoid air due to our multiphase implementation of the fluid.

                                                                                                   t/ x > 50), the procedure can
6 In very coarse grids (e.g., 15×15), or when very large time steps are used (e.g., the CFL number u

produce a large amount of modification in the position or curvature. In such cases, the effect of compensation can be noticeable.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                        Stable But Nondissipative Water            •      93




                                       Fig. 4. Real-time 2D simulation.




                                      Fig. 5. A football thrown into water.

  We ran the 3D version of our simulator to experiment the cases shown in Figures 5, 6, and 7, and
produced the animations shown in Clips 2, 3, and 4, respectively. In the case shown in Figure 5, a football
was thrown into water with a large initial velocity, which realistically produced a violent splash of water
and a trail of air bubbles behind the ball. The density ratio of the ball relative to water was 0.75. In the
case shown in Figure 6, a cubic container containing water and an object was rotated. The collision the
water made with the wall created a highly dynamic water surface and plenty of droplets and bubbles. In
the case shown in Figure 7, an empty cup was drowned into water, which caused the air to be released
and to produce a dynamic response with water. If only water-phase was simulated, water would fill
the cup without experiencing any resistance, thus the cup would submerge smoothly, which is not
physically correct. We constrain all fluid fragments to be within the fluid simulation domain. So, the
                                                                  ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
94      •      O.-Y. Song et al.




                                      Fig. 6. Simulation of water in a rotating cubic container.




                                                  Fig. 7. A cup drowned into water.



              Table I. 3D Simulation Information (Density Ratio is Air:Water:Object. All experiments are
              performed on the same condition except object densities; the viscosity coefficient of water is
            1.0 × 10−2 , the viscosity coefficient of air is 2.0 × 10−4 , and the surface tension coefficient is 0.5.)
        Experiment        Density Ratio     Total Frame       Simulation Time (sec/frame)   Rendering Time (sec/frame)
        Football           1:800:600            180                      51.7                         162.1
        Container          1:800:500            260                      40.3                         282.6
        Cup                1:800:750            126                      34.0                         159.1




droplets getting out of the vertical boundaries (the walls and the front transparent glass) are stopped,
and then slide down along the surface of the wall. The simulations were performed on a 80 × 80 × 80
grid. Simulation of one time step took about 30-40 seconds. A fixed time step of t = 30 second was
                                                                                          1

used in all the above 2D and 3D simulations, except for the case of the football. As for the football
example, in order to obtain a reasonable visual quality, we adjusted t so that the CFL number does
not exceed 5.0. Extraction of water surface was done using the marching cube algorithm [Lorensen and
Cline 1987], and rendering was done by Mental Ray. The simulation information for each experiment
along with the time required to simulate the motion and render the images appears in Table I.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                              Stable But Nondissipative Water            •      95




            Fig. 8. Plot of (φi , φi+1 ): the shaded region guarantees monotonic profiles in the case of    φ ≥ 0.

8.   CONCLUSION
The introduction to the graphics community of the semi-Lagrangian methodology by Stam in 1999
opened the way for the interactive manipulation/control of gaseous fluids. Since then, it has been an
open problem to extend the technique to water. In this article, we have proposed a solution to this
problem.
   The problem mainly consisted of two technical challenges: (1) modeling a multiphase fluid, and (2)
finding a dissipation-free technique. In this article, the problem of modeling a multiphase fluid was
solved by combining the Navier–Stokes equations with the level set method. The problem of preventing
dissipation was solved by two means: (1) adoption of the CIP method, which could model the subcell-level
details with third-order accuracy; (2) development of a particle-based method to prevent dissipation in
small-scale features such as droplets/bubbles.
   Instead of simulating water in a void space, the multiphase fluid formulation proposed here properly
simulates the dynamic movement of water, while it engages in complex interactions with surrounding
air. Due to the measures taken to prevent dissipation in the proposed method, the simulated water does
not unnecessarily lose volume or its motion is not damped to an unphysical extent.
   Although “stable but nondissipative water” this article proposes makes a significant improvements
in modeling nondissipative behavior of water, the focus has not been put on producing an accurate
shape of water surface. When an application requires more accurate modeling of water surface, then
Enright et al. [2002] can be adopted to the framework we propose, but with an increased amount
of computation. As for modeling droplets/bubbles, to our knowledge, the present work is the first that
proposed a mechanism for two-way exchange of masses between the grid-based framework and particles
for the conservation of mass and momentum. More accurate modeling of the geometrical shape and size
of droplets/bubbles needs further study.

APPENDIX

Monotonic CIP
In this appendix, we develop a method for modifying φi and φi+1 , such that the profile becomes monotonic.
If, for simplicity, the grid size (xi+1 − xi ) is 1, differentiation of Equation (4) produces
                     (X ) = (3X 2 − 4X + 1)φi + (3X 2 − 2X )φi+1 − (6 φ)X 2 + (6 φ)X .                                       (25)
                                                                        ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
96      •     O.-Y. Song et al.

When φ ≥ 0, the necessary and sufficient condition for the profile to be monotonically increasing inside
the cell is (X ) ≥ 0. By manipulating Equation (25), we found that this condition can be reduced to
                                            [φi ≥ 0,     φi+1 ≥ 0,    φi + φi+1 ≤ 3 φ] OR
                                  [9 φ − 6 φ(φi + φi+1 ) + (φi + φi+1 )2 − φi φi+1 ≤ 0],
which corresponds to the shaded region in Figure 8. Similarly, when φ < 0, the necessary and sufficient
condition for the profile to be monotonically decreasing is
                                          [φi ≤ 0,     φi+1 ≤ 0,     φi + φi+1 ≥ −3 φ] OR
                                  [9 φ + 6 φ(φi + φi+1 ) + (φi + φi+1 )2 − φi φi+1 ≤ 0].
  Therefore, the monotonic CIP technique works in the following way: when (φi , φi+1 ) does not belong to
the shaded region, we modify the values so that the tuple goes into the region. Although the modification
may create a more diffusive profile than the original one, the result is oscillation-free and still has a
third-order accuracy in space.

ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their comments and suggestions.
REFERENCES
BRACKBILL, J. U., KOTHE, D. B., AND ZEMACH, C. 1992. A continuum method for modeling surface tension. J. Comp. Phys. 100,
  335–354.
CARLSON, M., MUCHA, R. J., AND TURK, G. 2004. Rigid fluid: Animating the interplay between rigid bodies and fluid. ACM Trans.
  Graph. (Proceedings of ACM SIGGRAPH 2004) 23, 3, 377–384.
CHEN, J. X. AND LOBO, N. D. V. 1995. Toward interactive-rate simulation of fluids with moving obstacles using Navier–Stokes
  equations. Graph. Models Image Process. 57, 2, 107–116.
ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. ACM Trans. Graph.
  (Proceedings of ACM SIGGRAPH 2002) 21, 3, 736–744.
FEDKIW, R., STAM, J., AND JENSEN, H. W. 2001. Visual simulation of smoke. Comput. Graph. (Proceedings of ACM SIGGRAPH
  2001) 35, 15–22.
FELDMAN, B. E., O’BRIEN, J. F., AND ARIKAN, O. 2003. Animating suspended particle explosions. ACM Trans. Graph. (Proceedings
  of ACM SIGGRAPH 2003) 22, 3, 708–715.
FOSTER, N. AND FEDKIW, R. 2001. Practical animation of liquids. Comput. Graph. (Proceedings of ACM SIGGRAPH 2001) 35,
  23–30.
FOSTER, N. AND METAXAS, D. 1996. Realistic animation of liquids. Graph. Models Image Process. 58, 5, 471–483.
FOSTER, N. AND METAXAS, D. 1997a. Controlling fluid animation. In Comput. Graph. Inter. 97, 178–188.
FOSTER, N. AND METAXAS, D. 1997b.          Modeling the motion of a hot, turbulent gas. Comput. Graph. (Proceedings of ACM
  SIGGRAPH ’97) 31, Annual Conference Series, 181–188.
GOLUB, G. H. AND LOAN, C. F. V. 1996. Matrix Computations. The John Hopkins Univserity Press.
HARLOW, F. H. AND WELCH, J. E. 1965. Numerical calculation of time-dependent viscous incompressible flow of fluid with free
  surface. Phys. Fluids 8, 12, 2182–2189.
KASS, M. AND MILLER, G. 1990. Rapid, stable fluid dynamics for computer graphics. Comput. Graph. (Proceedings of ACM
  SIGGRAPH ’90) 24, 4, 49–57.
LORENSEN, W. E. AND CLINE, H. E. 1987. Marching cubes: A high resolution 3D surface construction algorithm. Comput. Graph.
  (Proceedings of ACM SIGGRAPH ’87) 21, 4, 163–169.
LOSASSO, F., GIBOU, F., AND FEDKIW, R. 2004. Simulating water and smoke with an octree data structure. ACM Trans. Graph.
  (Proceedings of ACM SIGGRAPH 2004) 23, 3, 457–462.
                                       ´
MCNAMARA, A., TREUILLE, A., POPOVIC, Z., AND STAM, J. 2004. Fluid control using the adjoint method. ACM Trans. Graph.
  (Proceedings of ACM SIGGRAPH 2004) 23, 3, 449–456.
MILLER, G. AND PEARCE, A. 1989.          Globular dynamics: A connected particle system for animating viscous fluids. Com-
  put.Graph. 13, 3, 305–309.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                                Stable But Nondissipative Water            •      97

O’BRIEN, J. AND HODGINS, J. 1995. Dynamic simulation of splashing fluids. In Proceedings of Computer Animation 95, 198–205.
O’BRIEN, T. G. G. A. W. B. J. F. 2004. A method for animating viscoelastic fluids. ACM Trans. Graph. (Proceedings of ACM
  SIGGRAPH 2004) 23, 3, 463–468.
OSHER, S. AND FEDKIW, R. 2002. The Level Set Method and Dynamic Implicit Surfaces. Springer-Verlag, New York.
OSHER, S. AND SETHIAN, J. A. 1988. Fronts propagating with curvature dependent speed: Algorithms based in hamilton-jacobi
  formulations. J. Comp. Phys. 79, 12–49.
PENG, D., MERRIMAN, B., OSHER, S., ZHAO, H., AND KANG, M. 1999. A pde-based fast local level set method. J. Comp. Phys. 155,
  410–438.
       ˇ
PREMOZE, S., TASDIZEN, T., BIGLER, J., LEFOHN, A., AND WHITAKER, R. T. 2003. Particle-based simulation of fluids. In Eurographics
  2003 Proceedings. Blackwell Publishers, 401–410.
RASMUSSEN, N., NGUYEN, D. Q., GEIGER, W., AND FEDKIW, R. 2003. Smoke simulation for large scale phenomena. ACM Trans.
  Graph. (Proceedings of ACM SIGGRAPH 2003) 22, 3, 703–707.
SETHIAN, J. A. 1996.         Fast marching level set methods for three dimensional photolithography development. SPIE 2726,
  261–272.
STAM, J. 1999. Stable fluids. Comput. Graph. (Proceedings of ACM SIGGRAPH ’99) 33, Annual Conference Series, 121–128.
STAM, J. AND FIUME, E. 1995. Depicting fire and other gaseous phenomena using diffusion processes. Comput. Graph. (Pro-
  ceedings of ACM SIGGRAPH ’95) 29, Annual Conference Series, 129–136.
                         ˆ `
STANIFORTH, A. AND COTE, J. 1991.          Semi-lagrangian integration scheme for atmospheric model—a review. Mon. Weather
  Rev. 119, 12, 2206–2223.
SUSSMAN, M., FATEMI, E., SMEREKA, P., AND OSHER, S. 1998. An improved level set method for incompressible two-phase flows.
  Comput. Fluids 27, 663–680.
SUSSMAN, M., SMEREKA, P., AND OSHER, S. 1994. A level set approach for computing solutions to incompressible two-phase flow.
  J. Comp. Phys. 114, 146–159.
TAKAHASHI, T., FUJII, H., KUNIMATSU, A., HIWADA, K., SAITO, T., TANAKA, K., AND UEKI, H. 2003. Realistic animation of fluid with
  splash and foam. In Eurographics 2003 Proceedings. Blackwell Publishers, 391–400.
TERZOPOULOS, D., PLATT, J., AND FLEISCHER, K. 1989. Heating and melting deformable models (from goop to glop). In Proceedings
  of Graphics Interface ’89. 219–226.
                                        ´
TREUILLE, A., MCNAMARA, A., POPOVIC, Z., AND STAM, J. 2003. Keyframe control of smoke simulations. ACM Trans. Graph.
  (Proceedings of ACM SIGGRAPH 2003) 22, 3, 716–723.
                                            ˝
TROTTENBERG, U., OOSTERLEE, C., AND SCHULLER, A. 2001. Multigrid. Academic Press.
TSAI, Y.-H. R., CHENG, L.-T., OSHER, S., AND ZHAO, H.-K. 2003. Fast sweeping algorithms for a class of hamilton–jacobi equations.
  SIAM J. Numer. Anal. 41, 673–694.
XIAO, F., YABE, T., AND ITO, T. 1996. Constructing oscillation preventing scheme for advection equation by rational function.
  Comp. Phys. Comm. 93, 1–12.
YABE, T. AND AOKI, T. 1991. A universal solver for hyperbolic equations by cubic-polynomial interpolation i. one-dimensional
  solver. Comp. Phys. Comm. 66, 219–232.
YABE, T., XIAO, F., AND UTSUMI, T. 2001. The constrained interpolation profile method for multiphase analysis. J. Comp. Phys. 169,
  556–593.


Received July 2004; revised October 2004; accepted October 2004




                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
A Physically-Based Motion Retargeting Filter
SEYOON TAK
Samsung Advanced Institute of Technology
and
HYEONG-SEOK KO
Seoul National University


This article presents a novel constraint-based motion editing technique. On the basis of animator-specified kinematic and dynamic
constraints, the method converts a given captured or animated motion to a physically plausible motion. In contrast to previous
methods using spacetime optimization, we cast the motion editing problem as a constrained state estimation problem, based
on the per-frame Kalman filter framework. The method works as a filter that sequentially scans the input motion to produce a
stream of output motion frames at a stable interactive rate. Animators can tune several filter parameters to adjust to different
motions, turn the constraints on or off based on their contributions to the final result, or provide a rough sketch (kinematic hint)
as an effective way of producing the desired motion. Experiments on various systems show that the technique processes the
motions of a human with 54 degrees of freedom, at about 150 fps when only kinematic constraints are applied, and at about
10 fps when both kinematic and dynamic constraints are applied. Experiments on various types of motion show that the proposed
method produces remarkably realistic animations.
Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.2.9
[Artificial Intelligence]: Robotics—Kinematics and Dynamics
General Terms: Algorithms, Experimentation

Additional Key Words and Phrases: Motion retargeting, animation w/constraints, physically based animation



1.   INTRODUCTION
Motion editing is an active research problem in computer animation. Its function is to convert the motion
of a source subject or character into a new motion of a target character while satisfying a given set of
kinematic and dynamic constraints, as shown schematically in Figure 1. This type of motion editing,
in which the animator specifies what they want in the form of constraints, is called constraint-based
motion editing, and has been studied by numerous researchers [Gleicher 1998; Lee and Shin 1999; Choi
                     c
and Ko 2000; Popovi´ and Witkin 1999; Shin et al. 2001].
   Motion editing must compensate for both body differences and motion differences. When the an-
thropometric scale of the target character differs from that of the source character, the original motion

This work was supported by the Korea Ministry of Information and Communication and the Overhead Research Fund of Seoul
National University.
This work was also supported in part by the Automation and Systems Research Institute at Seoul National University, and the
Brain Korea 21 Project.
Authors’ addresses: S. Tak, Samsung Advanced Institute of Technology, San 14-1, Nongseo-Ri. Giheung-eup, Yongin-si, 449-712,
Korea; email: tak@sait.samsung.co.kr, tak@graphics.snu.ac.kr; H.-S. Ko, Graphics and Media Lab, Seoul National University,
San 56-1, Shillim-dong, Kwanak-ku, Seoul 151-741, Korea; email: ko@graphics.snu.ac.kr.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0098 $5.00
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 98–117.
                                                         A Physically-Based Motion Retargeting Filter             •      99




                              Fig. 1. The constraint-based motion editing problem.


should be kinematically retargeted to the new character. Characteristics that affect body dynamics, such
as segment weights and joint strengths, should be accounted for if we are to generate a dynamically
plausible motion of the target character. For example, the kicking motion of a professional soccer player
cannot be reproduced by an unskilled person of equivalent anthropometric characteristics. Therefore,
the motion editing algorithm should resolve both the kinematic and dynamic aspects of the source-to-
target body differences. In addition, motion editing should provide means to create variations from the
original motion. For example, starting from an original walking motion on a level surface, an animator
may need to create longer steps or uphill steps.
   This article proposes a novel constraint-based motion editing technique that differs significantly from
existing methods in that it is a per-frame algorithm. The traditionally employed spacetime optimization
methods can be used for interactive editing of short motion sequences and produce physically plausible
motions. However, the processing times of these methods increase proportional (or at a higher rate) to
the length of the motion sequence. In contrast, our algorithm functions as a filter of the original motion
that processes the sequence of frames in a pipeline fashion. Thus, the animator can view the processed
frames at a stable interactive rate as soon as the filter has started processing the motion, rather than
having to wait for all frames to be processed as is the case in spacetime optimization methods.
   The per-frame approach has previously been taken by several researchers for the kinematic motion
editing problem in which only kinematic constraints are imposed [Lee and Shin 1999; Choi and Ko
2000; Shin et al. 2001]. However, the problem of motion editing with both kinematic and dynamic con-
straints poses two significant challenges. (1) Dynamic constraints are highly nonlinear, compared to
kinematic constraints. Such nonlinearity prohibits the constraint solver from reaching a convergent so-
lution within a reasonable amount of time. (2) Dynamic constraints involve velocities and accelerations,
whereas kinematic constraints involve only positions. It is this significant distinction that makes the
per-frame approach inherently difficult for dynamic constraints; kinematic constraints can be indepen-
dently formulated for individual frames, whereas the velocity and acceleration terms in the dynamic
constraint equations call for knowledge of quantities from other frames. The interdependency between
those terms makes the process look like a chain reaction, where imposing dynamic constraints at a
single frame, calls for the participation of the positions and velocities of the entire motion sequence.
   We overcome the challenges outlined above by casting the motion editing problem as a constrained
state estimation problem, based on the Kalman filter framework. We make the method function as a
per-frame filter by incorporating the motion parameters and the desired constraints into a specialized
Kalman filter formulation. To handle the nonlinearity of complex constraints more accurately, we em-
ploy the unscented Kalman filter, which is reported to be superior in its accuracy to the other variants
of the Kalman filter or the Jacobian-based approximation [Wan and van der Merwe 2001].
   To apply Kalman filtering to the problem of motion editing, however, we must treat the position, ve-
locity, and acceleration as independent degrees of freedom (DOFs). Under this treatment, the resulting
motion parameter values may violate the relationship that exists between the position, velocity, and
acceleration values describing a particular motion. We resolve this problem by processing the Kalman
                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
100      •      S. Tak and H.-S. Ko

filter output with a least-squares curve fitting technique. We refer to this processing as the least-squares
filter. Unlike the Kalman filter that processes each frame independently, the least-squares filter requires
data over a certain range of frames for curve fitting.
  Therefore, the proposed motion editing filter is basically a concatenation of the Kalman filter and the
least-squares filter. It functions as an enhancement operator; the first application of the filter may not
produce a completely convergent solution, but repeated applications refine the result until a reasonable
solution is reached. Such incremental refinement can be valuable in practice, because most animators
prefer to see a rough outline of the motion interactively before carrying out the longer calculation
necessary to obtain the final motion. Furthermore, they can provide a rough sketch of the desired
motion before the filtering begins, which is an effective way of reflecting their intuitive ideas as well as
overcoming the locality nature of the proposed algorithm.
  Our motion editing technique is well suited for interactive applications; we can add or remove some
or all of the kinematic and dynamic constraints depending on whether they significantly affect the
type of motion being animated. When only kinematic constraints are imposed, one application of the
filter produces a convergent solution and the motion editing algorithm runs in real-time. As dynamic
constraints are added, the filter must be applied several times to obtain convergent results, but the
editing process still runs at an interactive speed.

2.    RELATED WORK
The establishment of motion capture as a commonplace technique has heightened interest in meth-
ods for modifying or retargeting a captured motion to different characters. Motion editing/synthesizing
methods can be classified into four groups: (1) methods that involve only kinematic constraints, (2) meth-
ods that involve both kinematic and dynamic constraints, (3) the spacetime constraints methods that
do not exploit the captured motion, and (4) motion generating techniques based on dynamic simulation.
  Gleicher [1997, 1998] formulated the kinematic version of the motion editing problem as a space-
time optimization over the entire motion. Lee and Shin [1999] decomposed the problem into per-frame
inverse kinematics, followed by curve fitting for motion smoothness. Choi and Ko [2000] developed
a retargeting algorithm that works online, which is based on the per-frame inverse rate control, but
avoids discontinuities by imposing motion similarity as a secondary task. Shin et al. [2001] proposed
a different online retargeting algorithm based on the dynamic importance of the end-effectors. A good
survey of the constraint-based motion editing methods is provided by Gleicher [2001]. The way our
motion editing technique works most resembles the approach of Lee and Shin [1999], in that both tech-
niques are per-frame methods with a post-filter operation. However, in our method, the post-filter is
applied only to recently processed frames and, as a consequence, the whole process works as a per-frame
filter.
  It is interesting to note that the methods based on kinematic constraints quite effectively generate
useful variations of the original motion. However, when the dynamic context is significantly different in
the source and target motions, the motion generated by kinematic editing is unacceptable. Pollard et al.
[2000] proposed a force-scaling technique for fast motion transformation. Tak et al. [2000] introduced
a spacetime optimization technique for correcting a given motion into a dynamically balanced one.
        c
Popovi´ and Witkin [1999] addressed the physically-based motion editing problem using spacetime
optimization. Because optimization, subject to dynamic constraints (i.e. Newton’s law), can take a
prohibitive amount of computation, they introduced a character simplification technique to make the
problem tractable. The most significant distinction between our method and spacetime optimization
methods is that, instead of looking at the entire duration of a motion, our technique works on a per-
frame basis. As a result, the outcome of each frame is available at a uniform interactive rate, since it
requires a deterministic amount of computation.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                      A Physically-Based Motion Retargeting Filter            •     101

   An interesting work that solves the retargeting problem in the robotics context is Yamane and
Nakamura [2000, 2003], which is similar to our approach in that it transforms a given motion into
a physically consistent one on a per-frame basis. Their dynamics filter first computes the desired ac-
celerations by feedback controllers referring to the reference motion, and then modifies the result by
projection to the null-space of the equation of motion to make it a dynamically consistent one. Our
approach differs from theirs in that we use an iterative algorithm that consists of two consecutive fil-
ters and we process position, velocity, and acceleration simultaneously rather than acceleration alone,
thus increasing the applicability. For the same reason, it is difficult to control kinematic constraints in
their method, since they deal with only accelerations in the filtering process and then integrate them to
obtain the final positional data. Also, as they pointed out in the paper, the sensitiveness to the reference
motion which causes filter divergence and the difficulty of tuning parameters (feedback gains, pseu-
doinverse weights) are unsolved problems. Another recent work similar to ours is Shin et al. [2003],
which improves physical plausibility of edited motions by enforcing ZMP constraints and momentum
constraints. While our method is an iterative filtering process over all DOFs, they sequentially adjusted
user-specified individual DOF using approximated closed-form dynamic equations for efficiency.
   Many of the kinematic and physically-based motion editing techniques mentioned above derive from
the spacetime constraints method proposed by Witkin and Kass [1988]. However, when this original
method is applied to a complex articulated figure, the dimensional explosion and severe nonlinearity
of the problem usually leads to impractical computational loads or lack of convergence. Several groups
[Cohen 1992; Liu et al. 1994; Rose et al. 1996] have attempted to improve the classical spacetime
constraints algorithm and its applicability. In a recent work that synthesizes a dynamic motion from
                                 c
a rough sketch, Liu and Popovi´ [2002] circumvented the problems by approximating the Newtonian
dynamics with linear and angular momentum patterns during the motion. Another optimization-based
motion synthesis algorithm was proposed by Fang and Pollard [2003], which showed a linear-time
performance.
   Our constraint solver is built on the Kalman filter framework. There have been several previous
attempts to treat constraints using the Kalman filter. Maybeck[1979] introduced the notion that the
Kalman filter can be used to solve linear constraints by regarding them as perfect measurements,
while other researchers [Geeter et al. 1997; Simon and Chia 2002] built constraint solvers based on the
extended Kalman filter to solve nonlinear constraints. However, as many researchers have pointed out
[Julier and Uhlmann 1997; Wan and van der Merwe 2000], the extended Kalman filter can produce
inaccurate results at nonlinearities. We used the unscented Kalman filter to better handle the severe
nonlinearities in the dynamic constraints. A good introduction to the Kalman filter can be found in
Welch and Bishop [2001].
   The preliminary version of this work was presented in Tak et al. [2002]. In the current article,
we give a significantly improved exposition of the technique as well as extend the technique. The
formulation is now more rigorous, and the method is compared with other methods so that its limitations
and strengths are highlighted. By addressing momentum conservation in the flight phases, and the
redundancy problem during the double support phases, we widen the applicability of the algorithm.
New experiments that show the extended features are reported.

3.   OVERVIEW
Figure 2 shows an outline of the overall structure of our motion editing process. Animators first provide
the input motion of the source character along with a set of kinematic and dynamic constraints. Then
a Kalman filter that is tailored to the motion editing problem produces the motion parameter values,
which are post-processed by the least-squares curve fitting module. We apply the Kalman filter and
least-squares filter repeatedly until it converges to an acceptable result.
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
102      •      S. Tak and H.-S. Ko




                                       Fig. 2. Overall structure of the motion editing process.

     Several important issues must be addressed in the implementation of the process outlined above:
—What kinds of constraints are needed to generate a desired motion? How should those constraints be
 formulated? These issues are addressed in Section 4.
—How is the Kalman filter applied to our motion editing problem? The details are presented in Section 5.
—The Kalman filter processes position, velocity, and acceleration as independent variables, which can
 corrupt the imperative relationship among those variables. How is this rectified by the least-squares
 filter? This is explained in Section 6.

4.    FORMULATING CONSTRAINTS
The collection of all the kinematic and dynamic constraints on the motion of a character with        DOFs
can be summarized into the form
                                                              H(q, q, q) = Z,
                                                                   ˙ ¨                                   (1)
where q = q(t) is the -dimensional vector that completely describes the kinematic configuration of
the character at time t. This vector contains a mixture of positional and orientational quantities, but
when it is clear from the context, we call the entire vector simply the position.
                                             ˜
   The vector valued function H : R3 → R that maps a 3 -dimensional vector to a ˜ = ˜ K + ˜ B +
 ˜ T + ˜ M dimensional vector can be written as
                                                                  
                                                             ˙ ¨
                                                     H K (q, q, q)
                                                    H (q, q, q) 
                                                             ˙ ¨ 
                                                    B
                                     H(q, q, q) = 
                                            ˙ ¨                    .                                (2)
                                                    HT (q, q, q) 
                                                             ˙ ¨
                                                                             ˙ ¨
                                                                     H M (q, q, q)
 ˜ K , ˜ B , ˜ T , and ˜ M are the dimensions of the kinematic, balance, torque limit, and momentum
constraints, respectively. Therefore we can view the function H as a block matrix of the component
constraint functions H K , H B , HT , and H M , as shown in the right-hand side of Equation (2). The
values of ˜ K , ˜ B , ˜ T , and ˜ M depend on how each type of constraint participates in the current
editing process. For example, when only one end-effector position constraint is imposed, ˜ K = 3. If an
additional orientational constraint is imposed, then ˜ K becomes 6. Z is a ˜ -dimensional vector that
does not contain any variables, and can be represented as the block matrix Z = [ZT ZT ZT ZT ]T .
                                                                                       K    B     T     M
The goals of this section are (1) to find the formulations for each of the component constraint functions
H K , H B , HT , and H M , and (2) to find the values for the component constraint goals Z K , Z B , ZT , and
Z M , which correspond to the constraints specified by the animators.
   The constraint solver this article proposes requires only the formulation of the component functions,
but does not require their derivatives or inverse functions. Constraints are resolved by the black box
composed of the Kalman filter and least-squares filter.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                          A Physically-Based Motion Retargeting Filter                                 •     103




                              Fig. 3. The zero moment point (ZMP) and its trajectory correction.

4.1      Kinematic Constraints
Kinematic constraints specify the end-effectors to be positioned at the desired locations e by
                                                              hfk (q) = e,                                                                                    (3)
where the function hfk is the forward kinematic equations for the end-effectors under consideration.
Therefore, H K is simply formulated as
                                                      H K (q, q, q) = hfk (q),
                                                              ˙ ¨                                                                                             (4)
and the constraint goal is given by Z K = e.
4.2      Balance Constraints
Because humans are two-legged creatures, balancing is an important facet of their motion that must be
adequately captured if an animation is to appear realistic. Dynamic balance is closely related to the zero
moment point (ZMP), that is, the point at which the net moment of the inertial forces and gravitational
                                                    c
forces of all the body segments is zero [Vukobratovi´ et al. 1990]. The ZMP, at a particular instant, is a
function of the character motion, and can be obtained by solving the following equation for Pzmp
                                                (ri − Pzmp ) × {mi (ri − g)} = 0,
                                                                    ¨                                                                                         (5)
                                            i

where mi and ri are the mass and center of mass of the ith segment (Figure 3(a)), respectively, and g
is the acceleration of gravity.
   In our work, we regard an animated motion to be dynamically balanced at time t if the projection P2D      zmp
of the ZMP is located inside the supporting area (the convex hull containing all the ground contacts).
Taking ri = (xi , y i , z i ), g = (0, − g (≈ 9.8), 0), and setting the y component of Pzmp to zero, Equation (5)
produces the following analytical solution for P2D :       zmp
                                                                    
                                                  i mi ( y i + g )xi − i mi xi y i
                                                         ¨                    ¨
                                                             i mi ( y i + g )
                                          =                                            = hzmp (q, q, q).
                                                                    ¨
                                   P2D
                                    zmp                                                             ˙ ¨                                                       (6)
                                                  i mi ( y i + g )z i − i mi z i y i
                                                         ¨                    ¨
                                                             i mi ( y i + g )
                                                                    ¨

             ¨                          ˙      ¨
Since ri and ri can be expressed by q, q, and q, we can view the above result as giving the formula for
the function hzmp (q, q, q).1 Some portions of P2D , obtained by evaluating the above formula, may lie
                      ˙ ¨                        zmp

1 Note                                                                                                     i   mi xi       i   mi z i
      that for a static posture, the ZMP in Equation (6) reduces to the center of gravity (                     mi
                                                                                                                       ,        mi
                                                                                                                                        ), the projection point of
                                                                                                               i               i
the center of mass of the whole body, on the ground.
                                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
104        •    S. Tak and H.-S. Ko

outside the supporting area as shown in Figure 3(c). In our work, balancing is achieved by modifying
                                                                                ˜
the motion (i.e. q, q, and q) such that hzmp (q, q, q) follows a new trajectory P2D given by
                    ˙      ¨                     ˙ ¨                              zmp

                                                      P2D (t)
                                                       zmp              :   if P2D (t) ∈ S
                                                                                zmp
                                      ˜
                                      P2D (t) =                                              ,                               (7)
                                        zmp
                                                      projS (P2D (t))
                                                              zmp       :   otherwise
where S is the supporting area and projS is the operator that projects the given point into the area S
as shown in Figure 3(c). Finally, the balance constraint is formulated as
                                                   H B (q, q, q) = hzmp (q, q, q),
                                                           ˙ ¨              ˙ ¨                                              (8)
            ˜
and Z B = P2D .
              zmp
   It should be noted that the notion of balance in this article is somewhat subtle, and different from the
usual meaning—not falling. A number of researchers in robotics and graphics have proposed balancing
techniques. One approach, based on the inverted pendulum model, ensures balanced motion by main-
taining the position and velocity of the center of gravity (COG) within a stable region [Faloutsos et al.
2001; Zordan and Hodgins 2002]. The same goal has also been achieved by tracking the ZMP trajectory
as an index of stability [Dasgupta and Nakamura 1999; Oshita and Makinouchi 2001; Sugihara et al.
2002]. In these previous studies, balancing was achieved by controlling the joint torques to prevent the
characters from falling.
   If our objective is to analyze the moment of a legged figure with respect to the ground, we can
equivalently represent the figure as an inverted pendulum (Figure 3(b)). In this conceptualization,
the location of the pendulum base corresponds to the ZMP. In every real-world motion with a ground
contact, even in falling motion, the ZMP always lies within the supporting area2 [Vukobratovi´ et al.c
1990]. Physically, therefore, the ZMP is not related to the act of balancing. Rather, the ZMP concept
is related to physical validness. Physical validness has no meaning in real-world motions, but has a
significant meaning in motion editing. We can judge that a computer-generated motion is out of balance
if the motion is physically invalid according to the ZMP criterion. Our balance constraint formulation,
based on the ZMP, provides an effective way to modify a given motion to achieve dynamic balance.
4.3    Torque Limit Constraints
The torque a human can exert at each joint is limited. However, computer-generated human motion can
violate this principle, potentially giving rise to motions that look physically unrealistic or uncomfortable
[Lee et al. 1990; Ko and Badler 1996; Komura et al. 1999]. To address this issue, we allow animators to
specify torque limit constraints. The motion editing algorithm must, therefore, modify the given motion
such that the joint torques of the new motion are within the animator-specified limits. We need to find
                                               ˙ ¨
the formulation of the function HT (q, q, q) and the goal ZT that achieves the modification.
  First, we must calculate the torque profile of the original motion to see if it contains any torque limit
violations. We let τ (t) = [τ1 (t) · · · τ −6 (t)]T be the torque vector at time t, which is the collection of the
scalar torques corresponding to the ( −6) joint DOFs. The inverse dynamics problem τ (t) = htrq (q, q, q)     ˙ ¨
has been extensively studied in the robotics literature [Craig 1989; Shabana 1994]. Here we use the O( )
                                                                                    ˙ ¨
Newton-Euler method, which does not give an explicit formula for htrq (q, q, q), but instead recursively
computes the torque values. For the closed-loop formed during the double support phases, we resort to
the approximation method proposed by Ko and Badler [1996].

2 During flight phases, however, the evaluation of P2D in Equation (6) results in dividing by zero, thus the ZMP is not defined.
                                                      zmp
This is consistent with our intuition that balance is a meaningful concept only when the body makes contact with the ground.
In flight phases, therefore, we deactivate the balance constraint. Detecting the initiation of the fight phase is done by examining
the positions of the feet. The simulation results are insensitive to the noise at the phase boundaries.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                 A Physically-Based Motion Retargeting Filter             •     105

  When the torque τ j (t), computed as described above, exceeds the specified limit τ max , we reduce it to
                                                                                     j
the given limit. Thus, the corrected torque profile τ j of joint j is given by
                                                   ˜
                                               τ j (t)       :    if τ j (t) ≤ τ max
                                                                                 j
                                   τ j (t) =
                                   ˜                                                   .                                         (9)
                                               τ max
                                                 j           :    otherwise
In our implementation, the torque limit τ max was given by animators experimentally, but can also be
                                          j
determined using joint strength data from the biomechanics [Winter 1990].
  Finally, the torque constraints are formulated as
                                        HT (q, q, q) = htrq (q, q, q),
                                               ˙ ¨              ˙ ¨                                                            (10)
with ZT = τ .
          ˜

4.4   Momentum Constraints
The momentum constraints are derived from Newton’s second law, which states that the rates of change
of the linear and angular momenta are equal to the sums of the resultant forces and moments acting
on the figure, respectively. In the supporting phases, the interaction between the feet and the ground
leads to quite complex patterns in the character’s momentum behavior. Therefore, we do not impose
momentum constraints in the supporting phases. In flight phases, however, gravity is the only external
force. Thus the linear momentum and the net angular momentum of the entire body must satisfy
 ˙                      ˙
P = mall c = mall g and L = i mi (ri − c) × (¨ i − c) = 0 (point mass model assumed), where mall is the
          ¨                                  r     ¨
                                                                          c
total mass and c is the center of mass of the entire figure [Liu and Popovi´ 2002]. Hence, we formulate
the momentum constraints during flight phases as
                                                                     c−g
                                                                     ¨
                              H M (q, q, q) =
                                      ˙ ¨                                                  ,                                   (11)
                                                         i   mi (ri − c) × (¨ i − c)
                                                                            r     ¨
with ZT = 0 in this case.

5.    KALMAN FILTER-BASED MOTION EDITING
Once the constraints are formulated as shown in Equation (1), the task of modifying the original motion
to meet the constraints is accomplished using Kalman filtering. The important decisions in our tailoring
of Kalman filtering to motion editing problem are (1) the choices for the process and measurement model,
and (2) using the unscented Kalman filter, rather than the extended Kalman filter. We begin this section
with a brief explanation of how Kalman filtering works. Then we show how the motion editing problem
is formulated in the framework of Kalman filtering.

5.1   How Kalman Filtering Works
Kalman filtering is the problem of sequentially estimating the states of a system from a set of mea-
surement data available online [Maybeck 1979; Welch and Bishop 2001]. The behavior of a Kalman
filter is largely determined by defining the process model xk+1 = f(xk , vk ) and the measurement model
zk = h(xk , nk ), where xk represents the state of the system at time tk , zk is the observed measurement,
and vk and nk are the process and measurement noise. For example, if we are to model the freefall of
a stone that is being recorded by a digital camera, xk is the random variable that represents the 3D
position of the stone, Pxk (which will appear in the subsequent descriptions) is the covariance of xk ,
and zk represents the 2D position of the stone recorded in the photograph. We define the process model
f so that it predicts the next state from the value of the current state (using knowledge of Newtonian
mechanics). The uncertainties due to factors such as air resistance and wind are modeled by vk , which
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
106      •      S. Tak and H.-S. Ko

is assumed to follow a Gaussian distribution. We define the measurement model h such that it describes
in principle the relationship between the state xk and the measurement zk . nk models the measurement
errors, and is also assumed to follow a Gaussian distribution. The Kalman filter recursively estimates
the mean and covariance of the state using the following predictor-corrector algorithm.
                            Predict (time update)             Correct (measurement update)
                               x− = f(xk−1 , 0)
                               ˆk      ˆ                       xk = x− + Kk (zk − h(x− , 0))
                                                                ˆ   ˆk              ˆk
The time update predicts the a priori estimate x− , and the measurement update corrects x− by referring
                                               ˆk                                       ˆk
                                                                 ˆ
to the new measurement zk to obtain the a posteriori estimate xk . The Kalman gain Kk is determined
from the process and measurement models according to the procedure described in Section 5.3.

5.2    Our Formulation of Motion Editing Problem
When formulating the constraint-based motion editing problem using a Kalman filter, the most sub-
stantial step is the determination of the process and measurement models. We define the process model
as x− = [qk qk qk ], where qk = q(tk ) is the value of q at the discrete time step tk . In this case,
    ˆk           ˙   ¨
                                                   ˆ
function f does not depend on the previous state xk−1 , but comes directly from the original motion. We
use Z in Equation (1) as the measurements, and denote the value of Z at time tk as Zk . We define the
                              ˙ ¨
measurement model as H(q, q, q) of Equation (1).
  The rationale behind the definition outlined above is that the original motion contains excellent
kinematic and dynamic motion quality, so by starting from this motion we intend to preserve its quality
in the final motion.

5.3    Motion Editing Algorithm Based on the UKF
Since the constraint functions in Equation (2) are highly nonlinear, the original version of the Kalman
filter, which was designed for linear systems, does not properly handle the motion editing problem
considered here. The extended Kalman filter (EKF) was developed to handle nonlinearity through a
Jacobian-based approximation, but recently the unscented Kalman filter (UKF) was proposed to better
handle severe nonlinearity.
   The UKF was first proposed by Julier et al. [1997], and further developed by others [Wan and van der
Merwe 2000; van der Merwe and Wan 2001]. The basic difference between the EKF and the UKF
lies in the way they handle nonlinear functions. The computational core of the Kalman filter consists
of the evaluation of the posterior mean and covariance when a distribution with the prior mean and
covariance goes through the nonlinear functions of the process and the measurement models.
   As shown in Figure 4(b), the EKF approximates the posterior mean by evaluating the nonlinear
function at the prior mean, and approximates the posterior covariance as the product of the Jacobian
and the prior covariance. However, it has been reported that this method can lead to inaccuracy and
occasional divergence of the filter [Julier and Uhlmann 1997; Wan and van der Merwe 2000].
   The UKF addresses this problem using a deterministic sampling approach that approximates the
posterior mean and covariance from the transformed results of a fixed number of samples as shown in
Figure 4(c). Given a nonlinear function h(x) = z defined for n-dimensional state vectors x, the UKF
first chooses 2n+ 1 sample points Xi that convey the prior state distribution (mean and covariance of x),
after which it evaluates the nonlinear function h at these points, producing the transformed sample
points Zi . The UKF then approximates the posterior mean and covariance by calculating the weighted
mean and covariance (Z− and Pzz in Step 4 of the procedure summarized below) of the transformed
                        ˆ
                          k
                                ˆ
sample points, which is accurate to the 2nd order for any nonlinearity as opposed to the 1st order in the
EKF. The superior performance of the UKF over the EKF is thoroughly discussed in Wan and van der
Merwe [2001], along with the quantitative analysis on a broad class of nonlinear estimation problems.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                   A Physically-Based Motion Retargeting Filter             •     107




Fig. 4. Comparison of mean and covariance approximations after the nonlinear transformation h is applied (excerpted from
Wan and van der Merwe [2001]): solid, dotted, dashed lines in the transformed results correspond to the ideal-, EKF-, and
UKF-transformed mean and covariance, respectively. In (b), J is the Jacobian matrix of h.

  Now, we summarize the steps involved in the proposed UKF-based constraint solver. The inputs fed
                                                          ˙ ¨
into the solver at each frame k are the source motion [qk qk qk ] and the constraint goal Zk .
  For each kth frame,

(1) Using the process model definition discussed in Section 5.2, the prediction step is straightforward3 :

                                                  x− = [qk qk qk ]
                                                  ˆk       ˙ ¨
                                                                                    
                                                         Vx.pos · · ·         0
                                                                                    ,                                          (12)
                                                  P− = 
                                                  ˆ
                                                     k          Vx.vel               
                                                               0      ···   Vx.acc

    where Vx.∗ are the process noise covariances. Since our process model is not defined in terms of the
    previous state, P− does not depend on Pk−1 . Therefore, we simply use the constant matrix shown
                    ˆ
                      k
                                            ˆ
    above for every frame.
(2) We construct (2n + 1) sample points from x− and P− by
                                              ˆk     ˆ
                                                       k

                              X 0 = x−
                                    ˆk                      W0 = κ/(n + κ)    i=0
                              Xi =   x−
                                     ˆk              ˆ − )i Wi = 1/{2(n + κ)} i = 1, . . . , n
                                          + ( (n + κ)Pk                                                                          (13)
                              Xi =   x−
                                     ˆk   − ( (n +   κ)P− )i
                                                       ˆ
                                                         k     Wi = 1/{2(n + κ)} i = n + 1, . . . , 2n,
                                     √
    where κ is a scaling parameter, ( ·)i signifies the ith row or column of the matrix square root, and
                                                                                2n
    Wi is the weight associated with the ith sample point, chosen such that i=0 Wi = 1. Our choice for
    κ is based on Wan and van der Merwe [2001].

3 The   state vector and covariance matrix contain only positional components when only kinematic constraints are involved.
                                                                            ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
108      •      S. Tak and H.-S. Ko

(3) We transform the sample points in Step 2 through the measurement model defined in Section 5.2
    to obtain

                                                      Zi = H(Xi )       i = 0, . . . , 2n.                                    (14)

(4) The predicted measurement Z− is given by the weighted sum of the transformed sample points,
                                 ˆ
                                   k
    and the innovation covariance and the cross-covariance are computed as

                                             Z− =
                                             ˆ
                                               k
                                                          2n
                                                          i=0   Wi Zi
                                             ˆ zz =
                                             P            2n
                                                          i=0   Wi (Zi − Z− )(Zi − Z− )T + Nz
                                                                         ˆ
                                                                           k
                                                                                   ˆ
                                                                                     k
                                                                                                                              (15)
                                             ˆ
                                             Pxz =        2n
                                                          i=0   Wi (Xi − x− )(Zi − Z− )T ,
                                                                         ˆk
                                                                                   ˆ
                                                                                      k


      where Nz is the measurement noise covariance.
(5) The Kalman gain and the final state update are given by

                                                       Kk = Pxz P−1
                                                            ˆ ˆ
                                                                 zz
                                                                                                                              (16)
                                                       xk = x− + Kk (Zk − Z− ).
                                                       ˆ    ˆk            ˆ
                                                                            k


The behavior of the filter can be controlled by adjusting the following parameters:

—The process noise covariance Vx.∗ , which is a diagonal matrix. Each diagonal element of this matrix
 represents the degree of uncertainty of the corresponding DOF. The values of these diagonal elements
 are related to the degree of displacement that occurs in the filtering process. A larger value of a
 particular element results in a bigger displacement of the corresponding DOF.
—The measurement noise covariance Nz , which is also a diagonal matrix. Each diagonal element is
 related to the rigidity of one of the constraints on the motion. Typically, these elements are set to
 zero, treating the constraints as perfect measurements. Using nonzero diagonal elements is useful
 when two constraints conflict with each other, because the constraint with the larger covariance (soft
 constraint) yields to the one with the smaller covariance (hard constraint).



6.    LEAST-SQUARES FILTER
                                                                    ˙      ¨
The Kalman filter described in the previous section handles q, q, and q as independent variables.
As a result, the filtered result may not satisfy the relationship between the position, velocity, and
acceleration. The role of the least-squares filter is to rectify any corruption of this relationship that
occurred during Kalman filtering.4 Because the least-squares filter is basically a curve fitting procedure,
it has a tendency to smooth out the jerkiness that may be introduced by the per-frame handling of the
motion data.


4 If only kinematic constraints are involved, the least-squares filter need not be applied. However, even in this case, the animator

may choose to apply the least-squares filter to eliminate potential artifacts arising from the use of a per-frame approach.


ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                             A Physically-Based Motion Retargeting Filter            •     109

                                                                ˘
  To find the control points (in 1D) that fit the complete profile q of a particular DOF, we formulate a
B-spline curve that conforms to
                                                       B c = q,
                                                             ˘
                                                                                     
                                 B1 (t1 )    · · · BM (t1 )                          q1
                                    .                 .                           . 
                                    .                 .                           . 
                                    .                 .                           . 
                                                                                 q 
                                B1 (tN )    · · · BM (tN )                        N
                                                                                   
                                 ˙
                                 B1 (t1 )    · · · B M (t1 ) 
                                                    ˙                  c1           q1 
                                                                                  ˙ ,                                 (17)
                                    .                 .             .              
                        B=          .                 .      , c =  .    , q =  . 
                                    .                 .             .     ˘  .  .
                                                                                     
                                ˙
                                 B1 (tN )    · · · BM (tN ) 
                                                   ˙                   cM           qN 
                                                                                 ˙ 
                                 ¨          ··· B  ¨ M (t1 )                      q1 
                                B1 (t1 )                                          ¨ 
                                                                                  . 
                                    .
                                     .                 .
                                                       .                           . 
                                    .                 .                           . 
                                 ¨ 1 (tN )
                                 B           ··· B ¨ M (tN )                             ¨
                                                                                         qN

where Bi (t), Bi (t), and Bi (t)(i = 1, . . . , M ) are the B-spline basis functions and their first and second
              ˙           ¨
derivatives, respectively, and the scalar values ci are the control points. For a 3 DOF joint, we need to
formulate four such equations corresponding to the four components of the quaternion representation,
and renormalize the resulting values to maintain the unity of the quaternions. The spacing of the control
points along the time axis can be determined by the user, possibly by considering the distribution of
frequency components in the motion. In our experiments, we place a control point at every fourth frame
in common cases, but when the smoothing artifact in high-frequency motions is visible, we place a
control point at every second frame.
  The above problem is that of an over-constrained linear system. Therefore, we find the control points
                                                ˘
c that best approximate the given data q by minimizing the weighted objective function
                                               (B c − q)T W(B c − q),
                                                      ˘           ˘                                                       (18)
where W is a 3N × 3N diagonal matrix that controls the relative influences of q, q, and q. The classical
                                                                                ˙      ¨
linear algebra solution to this problem is
                                             c = (BT W B)−1 W q = B# q,
                                                              ˘      ˘                                                    (19)
where B# is the weighted pseudo-inverse matrix of B, and is easily computed because BT W B is a
well-conditioned matrix. Finally, we obtain the least-squares filtered motion by evaluating the B-spline
curves at the discrete time steps.
   Note that B, and accordingly B# , are band-diagonal sparse matrices. This means that each control
                                                              ˘
point in Equation (19) is determined by the section in q that corresponds to the nonzero entries of
B# . This locality suggests that the control points should be computed from the motion data within the
window shown in Figure 5, which moves along the time axis as the Kalman filter advances. Suppose
c1 , c2 , . . . , ci−1 have already been computed. When the center of the window of width W is positioned
                                                                     ˘
over the control point ci , we determine ci using the portion of q that exists within the window. This
scheme is equivalent to the global fitting described previously, if the window size is large enough. The
plot of accuracy versus window size depends on the order and knot-interval of the B-spline curve. In
our experiments, W = 64 gave almost the same accuracy as the global fitting.

                                                                     ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
110      •      S. Tak and H.-S. Ko




Fig. 5. Sweeping of the Kalman filter (the thick vertical line) and the least-squares filter (the rectangle centered at the figure).




7.    DISCUSSION
In this section, we analyze our algorithm in comparison with previous approaches and determine its
limitations as well as its strengths. This will help us to better understand what type of motion this
technique is best suited for.
   Statistical inference vs. derivative-based optimization. One of the main differences between our algo-
rithm and the previous methods is the statistical approach we take in dealing with the nonlinearity of
constraints. While derivative-based optimization uses Jacobians or Hessians of the constraints and ob-
jective function, our statistical method approximates the nonlinear function by evaluating the function
values on a set of samples, which can be viewed in some sense as statistical derivatives. Even though
the derivative-based approach theoretically has the same expressive power, in practice, we found that
the statistical approach provides more flexibility to address stiff problems.
   Local (per-frame) vs. global method. Another key difference, which might be the most significant
reason our method works fast, is that the method splits the given motion editing problem into per-frame
pieces with post filtering, instead of casting it as one large optimization problem as in the previous
methods. A limitation of our per-frame approach is that it can not make any dynamic anticipation
on a large timescale. Therefore, the algorithm is not suited for editing a motion that requires global
modification on the whole duration of the motion. For example, to generate the motion of punching
an object with a dynamic constraint that the final speed of the fist should be considerably larger than
that of the original motion, a large pull-back action is expected long before the arm makes the hit.
However, our algorithm would attempt to modify the motion only in the neighborhood of the final
frame. In fact, other optimization techniques that use local derivatives may have the potential to suffer
this locality problem. In this work, the problem can be circumvented by providing a rough sketch of the
desired motion as a new source motion. We call it a kinematic hint. Kinematic hints can be an effective
means to produce the desired motion when our method is interactively used by an animator who has
an intuitive idea of the form of the final motion.
   Filter tuning. The filter parameters (Vx.∗ and Nz in the UKF, and the weight parameter W in the least-
squares filter) significantly affect the filtering performance. Therefore, unless the parameter values are
carefully chosen through consideration of the type of constraints and target motion, the filter may not
work properly. For example, too large Vx.∗ may lead to filter divergence, and too small Vx.∗ may result in
slow convergence. Choosing appropriate filter parameters required trial-and-error procedures at first,
but soon we found that the following guidelines work well in most examples: Vx.pos is typically chosen
from the range 10−9 ∼10−10 , and we use Vx.vel = αVx.pos and Vx.acc = α 2 Vx.pos , where α =10∼100. The values
of Nz are set to zero in most cases. One exception to this rule is when the current constraint goal is
too far, in which case we initially set Nz to a small, but nonzero value (e.g. 10−10 ), and then adaptively
decrease them to zero. Finally, we found that the weight parameter, W = diag[I, βI, β 2 I], best scales
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                      A Physically-Based Motion Retargeting Filter            •     111

the relative influences of q, q, and q when β ≈ 0.1. In fact, all the examples presented in this article
                               ˙        ¨
are made based on this guideline.
   Convergence problem. Because of the nonlinear iterative nature of the problem, convergence is an
important issue. It is virtually impossible to characterize the analytical condition under which the
repeated applications of the Kalman filter and least-squares filter converge. According to our experi-
ments, when only kinematic constraints exist, the technique produces a convergent solution with one
or, at most, two applications of the filter. On the other hand, when we edit a motion involving dynamic
constraints, the effects of the Kalman filter on position, velocity, and acceleration are, in part, cancelled
by the following least-squares filter. Especially when the target motion is highly dynamic (e.g. such mo-
                             c
tions as in Liu and Popovi´ [2002] and Fang and Pollard [2003] in which the velocity and acceleration
can have severely undulating patterns), the cancellation effect of the two filters may become dominant,
and our method may not able to find a convergent solution. However, it is worth noting that, in most
cases, we could find the filter parameters and/or kinematic hints such that 3-5 filter applications attain
a reasonable level of dynamic quality.
   Dynamic quality reduction in the final filter application. While the resulting motion from the re-
peated filter applications possesses desired dynamic quality, the application of the last least-squares
filter can ruin the kinematic constraints to a noticeable degree. In such a case, we run the Kalman
filter with the dynamic constraints turned off, which produced kinematically accurate motion, poten-
tially destroying the previously attained dynamic quality. According to the experiments, however, the
dynamic quality reduction was not significant.
   Per-frame constraints generation. The global method requires the animators to set only high-level
goals (such as a jumping height); the rest is generated by the algorithm. On the other hand, our per-
frame algorithm requires the animators to supply kinematic and dynamic constraints for each frame
(in the form of trajectory, etc.). For example, to make the character kick a certain object, the animator
has to construct the desired trajectory that passes the object by modifying the originally given foot
trajectory. It is an extra burden for the animator compared to the global method. On the other hand, it
can be viewed as a means to control the details of the (kicking) motion.


8.   RESULTS
Our motion editing system was implemented as a Maya plug-in on a PC with a Pentium-4 2.53 GHz
processor and a GeForce 4 graphics board. All the motion sequences used were captured at 30 Hz. The
human model had a total of 54 DOFs, including 6 DOFs for the root located at the pelvis. The root
orientation and joint angles were all represented by quaternions.
   Below, we refer to the filtering with only kinematic constraints as kinematic filtering and denote
i consecutive applications of the filter by i(K), and we refer to the filtering with both kinematic and
dynamic constraints as dynamic filtering and denote j consecutive applications of the filter by j (D).
In the following experiments, we used the full set of DOFs in the kinematic filtering, but omitted
several less influential joints (for example, wrists, elbow, ankles, and neck) in the dynamic filtering
(total 27-DOFs) to improve the performance.
   This section reports the results of five experiments. The filter applications used and the resulting
frame rates in these experiments are summarized in Table I. The decision on the number and types of
filter applications are up to the users. They can apply the filtering repeatedly until they find the result
satisfactory, and also they can start by imposing kinematic hint for better performance. The animations
of the results can be found at http://graphics.snu.ac.kr/∼tak/filter.htm.
   Dancing (on-line kinematic retargeting). Referring to Animation 1A (Figure 6), we retargeted the
dancing motion of the middle character to the characters on the left (shorter limbs and longer torso) and
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
112      •      S. Tak and H.-S. Ko

                                                                Table I.
                      motion          # frames     # frames/knot      # filtering operation     average frame rate
                      dancing            660                                  1(K)                    150.0
                      wide steps         260              4               2(D) + 1(K)                  15.4
                      golf swing         216              2            1(K) + 3(D) + 1(K)              15.2
                      limbo walk         260              4            1(K) + 2(D) + 1(K)              13.7
                      jump kick          152              2               3(D) + 1(K)                  11.3
                     Summary of the experimental conditions and resulting frame rates. Frame rates were estimated from
                     pure computation time excluding the visualization time




         Fig. 6. Dancing: A dancing motion of the character in the middle is retargeted to the other characters online.


right (longer limbs and shorter torso). The foot trajectories of the source character were used without
scaling as the kinematic constraints of the target characters. The height differences were accounted for
by raising or lowering the root of the source character by a constant amount. In this experiment, a single
application of the Kalman filter (without the least-squares filter) generated the target motions without
any artifacts. Animation 1B is the real-time screen capture during the production of Animation 1A. In
other examples, which are not included in the article, if the end-effector goal was too far or followed
a severely nonsmooth path, Kalman filtering alone could produce temporal artifacts. In such cases,
application of the least-squares filter solved the problem, although it created a delay of about one
second. In numerous experiments, the kinematic filtering worked stably and robustly.
   Wide Steps. In this experiment, we considered the problem of converting the normal walking steps
in Animation 2A into the sequence of wide steps taken to go over a long narrow puddle. The kinematic
filtering produced the physically implausible result shown in Animation 2B (Figure 7(a)). To make
the motion physically plausible, we added a balance constraint, and applied dynamic filtering twice,
followed by a final kinematic filtering to impose the foot constraints (2(D) + 1(K)); these calculations
produced Animation 2C (Figure 7(b)). In this animation, the character now sways his body to meet
the balance constraints. The previous two results are compared in Animation 2D. Animation 2E is the
real-time screen capture during the production of the above animations.
   Golf Swing. This experiment shows how our technique retargets the golf swing shown in Animation
3A (Figure 8(a)) when a heavy rock (8kg, which is 1/8 of the character’s total mass) is attached to the
head of the club. We imposed balance constraints, but did not impose torque constraints because simply
raising the club immediately causes torque limit violations at the shoulder and elbow. To facilitate
convergence, we provided a kinematic hint, which is actually one kinematic filtering 1(K), mainly
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                 A Physically-Based Motion Retargeting Filter             •     113




                                 Fig. 7. Wide Steps: (a) kinematic-filtered (b) dynamic-filtered.




                        Fig. 8. Golf Swing: (a) original motion (b) with a heavier club (dynamic-filtered).

consisting of the shift of the pelvis with both feet fixed (Animation 3B). 3(D) + 1(K) filter applications
onto the kinematic hint produced Animation 3C (Figure 8(b)). In the resulting motion, the upper-body
of the character makes a large movement to counterbalance the heavy club. The original and retargeted
motions are compared in Animation 3D.
   Limbo Walk. In this experiment, the walking motion shown in Animation 2A is retargeted to a
limbo walk (Animation 4A = Animation 2A). We placed a limbo bar at 4/5 of the height of the character.
Balance constraints, along with a (soft) kinematic constraint on the head height,5 produced Animation
4B (Figure 9(a)), which is obviously not a limbo motion. To fix the problem, we provided a kinematic
hint. A further 2(D) + 1(K) filter application produced the realistic limbo walk shown in Animation 4C

5 The   height was lowered a few steps ahead.
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
114      •      S. Tak and H.-S. Ko




                          Fig. 9. Limbo Walk: dynamic-filtered (a) without and (b) with kinematic hint.




                      Fig. 10. Jump Kick: (a) original motion (b) when a load is attached (dynamic-filtered).

(Figure 9(b)). In another experiment, we animated the limbo motion of a character whose torso was
twice as heavy as that of the original character, and we imposed the torque limit constraint. In this
setup, the character could not bend the torso to the same extent as in the original case. Thus the
kinematic constraint, which was the soft constraint, could not be satisfied (Animation 4D).
  Jump Kick. This experiment shows how our motion editing technique adjusts the jump kick motion
shown in Animation 5A (Figure 10(a)) to allow for the addition of a 10kg sandbag to the right calf of
the character. The motion consists of a supporting phase, a flight phase, and another supporting phase.
We imposed balance and torque constraints along with foot position constraints during the supporting
phases, and torque and momentum constraints during the flight phase. Since the position of the landing
foot at the end of the flight phase is affected by the attached weight, we adjusted the footprints to account
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                A Physically-Based Motion Retargeting Filter             •     115

for this. Without any kinematic hints, the algorithm produced a convergent result after 3(D)+1(K) filter
applications. In the resulting motion, shown in Animation 5B, it is evident that the character cannot
lift the weighted leg as high as the original character due to the torque constraints.6 The momentum
constraints make the upper body bend forward to compensate for the momentum change in the right
leg (Figure 10(b)).

9.     CONCLUSION
In this article, we have presented a novel interactive motion editing technique for obtaining a physically
plausible motion from a given captured or animated motion. To date, most methods for carrying out
such motion retargeting have been formulated as a spacetime constraints problem. In contrast to these
previous methods, our method is intrinsically a per-frame algorithm; once the kinematic and dynamic
constraint goals are specified, the proposed algorithm functions as a filter that sequentially scans the
input motion to produce a stream of output motion frames at a stable interactive rate.
   The proposed method requires interactive tuning of the filter parameters to adapt it to the differ-
ent motions. Several (consecutive) applications of the filter may be required to achieve the desired
convergence or quality, because the filter works as an enhancement operator. Experiments on a large
variety of motions revealed that, in most cases, 3-5 applications are sufficient to produce realistic
motions.
   The method works in a scalable fashion. It provides various ways to trade off run time and animator
effort against motion quality. (1) Animators can interactively control the type and amount of kinematic
and dynamic constraints to shape the desired motion. (2) Animators can control the number of times
the filter is applied according to the final quality that is required. (3) Animators can avoid the potential
problem of slow convergence by providing a kinematic hint.
   This work takes an exciting step forward in the constraint-based motion editing. Now, physically
plausible motions can be produced by filtering existing motions on a per-frame basis.

ACKNOWLEDGMENTS
We would like to thank Michael Gleicher and Norman Badler for their insightful comments. We are
greatly indebted to Oh-young Song. Without his keen insightful comments in the initial conception,
this work would not have been possible.

REFERENCES
CHOI, K. AND KO, H. 2000. On-line motion retargetting. J. Visualiza. Comput. Anim. 11, 5, 223–235.
COHEN, M. F. 1992. Interactive spacetime constraint for animation. In Comput. Graphics (Proceedings of ACM SIGGRAPH
  92) 26, 2, ACM, 293–302.
CRAIG, J. J. 1989. Introduction to Robotics. Addison-Wesley.
DASGUPTA, A. AND NAKAMURA, Y. 1999. Making feasible walking motion of humanoid robots from human motion capture data.
  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 2. 1044–1049.
FALOUTSOS, P., VAN DE PANNE, M., AND TERZOPOULOS, D. 2001. Composable controllers for physics-based character animation. In
  Proceedings of ACM SIGGRAPH 2001. Computer Graphics Proceedings. ACM Press. 251–260.
FANG, A. C. AND POLLARD, N. S. 2003. Efficient synthesis of physically valid human motion. ACM Trans. Graph. 22, 3, 417–426.
GEETER, J. D., BRUSSEL, H. V., AND SCHUTTER, J. D. 1997. A smoothly constrained kalman filter. IEEE Trans. Pattern Anal.
  Mach. Intell., 1171–1177.
GLEICHER, M. 1997. Motion editing with spacetime constraints. In Proceedings of the 1997 Symposium on Interactive 3D
  Graphics.

6 An   approach based on the muscle force model may generate a more plausible motion than clamping the joint torques.
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
116      •      S. Tak and H.-S. Ko

GLEICHER, M. 1998.          Retargetting motion to new characters. In Proceedings of ACM SIGGRAPH 98. Computer Graphics
  Proceedings. ACM Press. 33–42.
GLEICHER, M. 2001. Comparing constraint-based motion editing methods. Graphical Models 63, 2, 107–134.
JULIER, S. J. AND UHLMANN, J. K. 1997. A new extension of the kalman filter to nonlinear systems. In Proceedings of AeroSense:
  The 11th International Symposium on Aerospace/Defense Sensing, Simulation and Controls.
KO, H. AND BADLER, N. I. 1996. Animating human locomotion in real-time using inverse dynamics, balance and comfort control.
  IEEE Comput. Graph. Applic. 16, 2, 50–59.
KOMURA, T., SHINAGAWA, Y., AND KUNII, T. L. 1999. Calculation and visualization of the dynamic ability of the human body. J.
  Visualiz. Comput. Anim. 10, 57–78.
LEE, J. AND SHIN, S. Y. 1999. A hierarchical approach to interactive motion editing for human-like figures. In Proceedings of
  ACM SIGGRAPH 99. Computer Graphics Proceedings. ACM Press. 39–48.
LEE, P., WEI, S., ZHAO, J., AND BADLER, N. I. 1990. Stregth guided motion. In Comput. Graph. (Proceedings of ACM SIGGRAPH
  90). 24, 3. ACM, 253–262.
                          ´
LIU, C. K. AND POPOVIC, Z. 2002.         Synthesis of complex dynamic character motion from simple animations. ACM Trans.
  Graph. 21, 3, 408–416.
LIU, Z., GORTLER, S. J., AND COHEN, M. F. 1994. Hierarchical spacetime control. In Proceedings of ACM SIGGRAPH 94. Computer
  Graphics Proceedings. ACM Press. 35–42.
MAYBECK, P. S. 1979. Stochastic Models, Estimation, and Control. Vol. 1. Academic Press, Inc.
OSHITA, M. AND MAKINOUCHI, A. 2001. A dynamic motion control technique for human-like articulated figures. In Proceedings
  of Eurographics 2001.
POLLARD, N. S. AND BEHMARAM-MOSAVAT, F. 2000. Force-based motion editing for locomotion tasks. In Proceedings of the IEEE
  International Conference on Robotics and Automation (ICRA), Vol. 1. 663–669.
        ´
POPOVIC, Z. AND WITKIN, A. 1999. Physically based motion transformation. In Proceedings of ACM SIGGRAPH 99. Computer
  Graphics Proceedings. ACM Press. 11–20.
ROSE, C., GUENTER, B., BODENHEIMER, B., AND COHEN, M. F. 1996. Efficient generation of motion transitions using spacetime
  constraints. In Proceedings of ACM SIGGRAPH 96. Computer Graphics Proceedings. ACM Press. 147–154.
SHABANA, A. A. 1994. Computational Dynamics. John Wiley & Sons, Inc.
SHIN, H. J., KOVAR, L., AND GLEICHER, M. 2003.         Physical touch-up of human motions. In Proceedings of Pacific Graphics
  2003.
SHIN, H. J., LEE, J., SHIN, S. Y., AND GLEICHER, M. 2001. Computer puppetry: An importance-based approach. ACM Trans.
  Graph. 20, 2, 67–94.
SIMON, D. AND CHIA, T. 2002.         Kalman filtering with state equality constraints. IEEE Trans. Aeros. Electr. Syst. 39, 128–
  136.
SUGIHARA, T., NAKAMURA, Y., AND INOUE, H. 2002. Realtime humanoid motion generation through zmp manipulation based on
  inverted pendulum control. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 2.
  1404–1409.
TAK, S., SONG, O., AND KO, H. 2000. Motion balance filtering. Comput. Graph. For. (Eurographics 2000) 19, 3, 437–446.
TAK, S., SONG, O., AND KO, H. 2002. Spacetime sweeping: An interactive dynamic constraints solver. In Proceedings of Computer
  Animation 2002. 261–270.
VAN DER MERWE, R. AND WAN, E. A. 2001.          The squre-root unscented kalman filter for state and parameter-estimation. In
  Proceedings of International Conference on Acoustics, Speech, and Signal Processing.
VAN DE PANNE, M. 1996. Parameterized gait synthesis. IEEE Comput. Graph. Applica. 16, 2, 40–49.
              ´                                      ´
VUKOBRATOVIC, M., BOROVAC, B., SURLA, D., AND STOKIC, D. 1990. Biped Locomotion: Dynamics, Stability, Control and Application.
  Springer Verlag.
WAN, E. A. AND VAN DER MERWE, R. 2000. The unscented kalman filter for nonlinear estimation. In Proceedings of Symposium
  2000 on Adaptive Systems for Signal Processing, Communication and Control.
WAN, E. A. AND VAN DER MERWE, R. 2001. Kalman Filtering and Neural Networks (Chapter 7. The Unscented Kalman Filter).
  John Wiley & Sons.
WELCH, G. AND BISHOP, G. 2001. An introduction to the kalman filter. ACM SIGGRAPH 2001 Course Notes.
WINTER, D. A. 1990. Biomechanics and Motor Control of Human Movement. John-Wiley, New York.
WITKIN, A. AND KASS, M. 1988. Spacetime constraints. In Comput. Graph. (Proceedings of ACM SIGGRAPH 88). 22, 4. ACM,
  159–168.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                             A Physically-Based Motion Retargeting Filter            •     117

YAMANE, K. AND NAKAMURA, Y. 2000.       Dynamics filter: Concept and implementation of online motion generator for human
  figures. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 1. 688–694.
YAMANE, K. AND NAKAMURA, Y. 2003.       Dynamics filter—concept and implementation of online motion generator for human
  figures. IEEE Trans. Robot. Autom. 19, 3, 421–432.
ZORDAN, V. B. AND HODGINS, J. K. 2002. Motion capture-driven simulations that hit and react. In 2002 ACM SIGGRAPH
  Symposium on Computer Animation. 89–96.


Received May 2003 revised October 2003, July 2004; accepted October 2004




                                                                     ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
Perceptual Photometric Seamlessness
in Projection-Based Tiled Displays
ADITI MAJUMDER
University of California, Irvine
and
RICK STEVENS
University of Chicago, Argonne National Laboratory


Arguably, the most vexing problem remaining for multi-projector displays is that of photometric (brightness) seamlessness within
and across different projectors. Researchers have strived for strict photometric uniformity that achieves identical response at
every pixel of the display. However, this goal typically results in displays with severely compressed dynamic range and poor
image quality.
   In this article, we show that strict photometric uniformity is not a requirement for achieving photometric seamlessness. We
introduce a general goal for photometric seamlessness by defining it as an optimization problem, balancing perceptual uniformity
with display quality. Based on this goal, we present a new method to achieve perceptually seamless high quality displays. We first
derive a model that describes the photometric response of projection-based displays. Then we estimate the model parameters
and modify them using perception-driven criteria. Finally, we use the graphics hardware to reproject the image computed using
the modified model parameters by manipulating only the projector inputs at interactive rates.
   Our method has been successfully demonstrated on three different practical display systems at Argonne National Laboratory,
made of 2 × 2 array of four projectors, 2 × 3 array of six, projectors, and 3 × 5 array of fifteen projectors. Our approach is
efficient, automatic and scalable—requiring only a digital camera and a photometer. To the best of our knowledge, this is the
first approach and system that addresses the photometric variation problem from a perceptual stand point and generates truly
seamless displays with high dynamic range.
Categories and Subject Descriptors: I.3.3 [Computer Graphics]: Picture/Image Generation—Display algorithms; I.4.0 [Image
Processing and Computer Vision]: General—Image displays; I.4.8 [Image Processing and Computer Vision]: Scene
Analysis—Color; Photometry; H.1.2 [Models and Principles]: User/Machine Systems—Human factors
General Terms: Algorithms, Human Factors
Additional Key Words and Phrases: Projection-based displays, tiled displays, color calibration




1.   INTRODUCTION
Large-area multi-projector displays offer an inexpensive way to display high-resolution life-size im-
ages. These displays are used extensively for large scale scientific visualizations and for virtual reality
applications in defense, entertainment, simulation, and training. Very high resolution displays made

Authors’ addresses: A. Majumder, Department of Computer Science, 444 Computer Science Building, University of California,
Irvine, CA 92696; email: majumder@cs.uci.edu; R. Stevens, Mathematics and Computer Science Division, Argonne National
Laboratory, 9700 South Cass Avenue, Building 221, Argonne, IL 60439.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0118 $5.00
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 118–139.
                                      Photometric Seamlessness in Projection-Based Tiled Displays             •     119

of 40–50 projectors are being built at the National Center for Supercomputing Applications at the
University of Illinois, Urbana-Champaign and various U.S. national laboratories.
   In building large area displays, there exist several issues such as driving architecture and data
distribution [Samanta et al. 1999; Humphreys et al. 2000, 2001; Buck et al. 2000; Humphreys and
Hanrahan 1999; Humphreys et al. 2001], but geometric misalignment and color variation are the
most salient issues to be addressed to make multi-projector displays perceptually “seamless”. Several
algorithms achieve geometrically undistorted and aligned displays [Raskar et al. 1998, 1999; Yang et al.
2001; Hereld et al. 2002; Raskar 1999; Cruz-Neira et al. 1993; Chen et al. 2002]. But the color variation
is still a significant obstacle.
   Color is a three-dimensional quantity defined by one-dimensional luminance (defining brightness)
and two-dimensional chrominance (defining hue and saturation). Thus, the color variation problem
involves spatial variation in both luminance and chrominance. Majumder and Stevens [2004] and
Majumder [2002] show that most current tiled displays made of projectors of the same model show
large spatial variation in luminance while the chrominance is almost constant spatially. Also, humans
are at least an order of magnitude more sensitive to luminance variation than to chrominance variation
[Chorley and Laylock 1981; Goldstein 2001; Valois and Valois 1990]. Thus, perceptually, the subproblem
of photometric variation (luminance variation) is the most significant contributor to the color variation
problem.
   Majumder and Stevens [2004] classify the spatial color variation in multi-projector displays in three
different classes: the variation within a single projector (intra-projector variation), across different pro-
jectors (inter-projector variation) and in the overlap regions (overlap variation). Most existing methods
[Pailthorpe et al. 2001; Stone 2001a, 2001b; Cazes et al. 1999; Majumder et al. 2000] do not address
the spatial variation within a projector and assume that every pixel of a projector has identical color
response. This assumption simplifies the color response reconstruction since the color response of each
projector needs to be estimated at only one spatial location using a radiometer or a photometer. How-
ever, Majumder and Stevens [2002, 2004], Stone [2001a, 2001b] show this to be an over-simplifying
assumption. Even within a single projector, a photometric fall-off as large as 40–60% from the center of
the display to the fringes is observed. Thus, methods that ignore this spatial variation cannot achieve
seamless displays. Blending or feathering techniques address only the variation in the overlap regions
and try to smooth color transitions across these regions. This can be achieved in software [Raskar et al.
1998], or by physical masks mounted at the projector boundaries [Li et al. 2000], or by optical masks
inserted in the light path of the projector [Chen and Johnson 2001]. However, these methods do not
estimate the variation in the overlap region and assume linear response projectors. This inaccuracy
results in softening of the seams in the overlapping region, rather then removing them. Nayar et al.
[2003] addresses the problem of using a single projector on screens with imperfect reflectance (like a
poster or a brick wall), and hence addresses only intra-projector variations. Majumder and Stevens
[2002] and Majumder et al. [2003] present methods that attempt to estimate and correct intra- inter-
and overlap photometric variations. However, due to the goal of achieving strict photometric unifor-
mity, the photometric response of every pixel in the display is matched with the response of the “worst”
possible pixel. This leads to poor image quality and low dynamic range (Figure 7).
   In this article, we show that strict photometric uniformity is not required to achieve a photometrically
seamless display where we cannot tell the number of projectors making up the display. Analysis of the
spatial color variation in a stand-alone projector supports this realization. A stand-alone projector is
not photometrically uniform, yet it is seamless (Figure 2). Hence, in this article, we demonstrate that
a smoothly varying photometric response that exploits human perceptual limitations is sufficient to
achieve seamlessness in a multi-projector display. Further, this less restrictive goal provides extra
leverage to increase the overall dynamic range, producing a display of far superior quality.
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
120      •      A. Majumder and R. Stevens

     Based on the above concept, following are the main contributions of this work.
(1) We realize that all pixels need not have identical response to achieve a seamless display. Instead, a
    perceptual uniformity can be achieved by exploiting the limitation of human vision and this weaker
    constraint can be leveraged to increase the display quality.
(2) We formalize such a goal of photometric seamlessness as an optimization problem that maximizes
    display quality, while minimizing the noticeable photometric variations.
(3) We design and implement a new practical method to solve this optimization problem and generate
    photometrically seamless high quality display. Our method is automatic, scalable, and solves the
    intra-, inter- and overlap photometric variations in a unified manner. Further, this method is imple-
    mented using commodity graphics hardware to correct any imagery projected on a multi-projector
    display at interactive rates.
  In order to achieve the above results, we first derive a model to comprehensively describe the photo-
metric variation across multi-projector displays (Section 2). Next, we formalize the problem of achiev-
ing photometric seamlessness as an optimization problem (Section 3). Finally, we present a method to
achieve this photometric seamlessness (Section 4).

2.    THE PHOTOMETRIC VARIATION
In this section, we derive an equation to describe the photometric variation in Lambertian multi-
projector displays. Table I provides a list of all the symbols and parameters used in this article, their
nomenclature, and description.
2.1     Definitions
A planar multi-projector display is a display made of N projectors, projecting on a planar display screen.
Each projector is denoted by P j , 1 ≤ j ≤ N . Figure 1 shows a simple two-projector display wall.
  A unified display coordinate space is defined by u, v, and w axes, where u and v describe the display
plane and w is perpendicular to it. The projector coordinates (s j , t j ) can be related to the display
coordinates (u, v) by a geometric warp G j , such that
                                                         (u, v) = G j (s j , t j ).                                           (1)
Note that G j also depends on the internal and external geometric parameters of projector P j including
its focal length, position, and orientation in the display coordinate space. For all practical purposes,
these geometric parameters do not change since the projectors and screen do not move relative to each
other. Hence we define G j as a function of the projector coordinates only.
   A projector has three channels, {r, g , b}. A channel is denoted by l ∈ {r, g , b}, and the corresponding
input by il ∈ {ir , i g , ib}, 0.0 ≤ il ≤ 1.0.1
2.2     Describing the Variation
We define a function L(u, v, i) as the luminance projected at any display coordinate (u, v) for an input
i = (ir , i g , ib) to all the projectors projecting at (u, v). We derive L here from the extensive analysis of
projection based displays presented in Majumder and Stevens [2004] and Majumder [2002].

Single Projector: Ideally, if the luminance from different channels of a projector are independent of
each other, the luminance L(s, t, i) at projector coordinate (s, t) for input i is given by the addition of

1 This considers only three primary systems. The DLP projectors that use a clear filter for projecting the grays behave like a four

primary system.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                             Photometric Seamlessness in Projection-Based Tiled Displays                 •     121

      Table I. Legend of all symbols and parameters, in the order in which they appear in the article. The
      parameters can be associated with a single projector or a multi-projector display. The difference has
                                been emphasized by underlining the association
       Symbols             Nomenclature                                          Description
       l             Channel                      l ∈ {r, g , b} and denotes the red, green and blue channels.
       il            Channel input                Input for channel l .
       i             Input                        The three channel input, (ir , i g , ib ).
       (s, t)        Projector coordinates        2D coordinates of a projector.
       (u, v)        Display coordinates          2D coordinates of the multi-projector display.
       G(s, t)                                    The projector geometric warp relating (s, t) with (u, v).
       N p (u, v)                                 Number of projectors overlapping at (u, v).
       L(s, t, i)                                 Luminance projected at (s, t) of a projector for input i.
       hl (il )      Transfer function of         The fraction of the maximum luminance of channel l projected
                       channel l                    for input il in a projector.
       B(s, t)       Black luminance function     The black offset at (s, t) of a projector.
       Wl (s, t)     Maximum luminance            The maximum luminance projected from channel l at (s, t) of a
                       function of channel l        projector.
       L(u, v, i)                                 The luminance projected at (u, v) of a multi-projector display
                                                    when N p (u, v) projectors are projecting input i.
       Hl (il )      Common transfer function     The spatially invariant input transfer function of channel l in a
                       of channel l                 multi-projector display.
       Wl (u, v)     Maximum display              The maximum luminance projected from channel l at (u, v) in a
                       luminance function of        multi-projector display.
                       channel l
       B(u, v)       Black display luminance      The black offset at (u, v) of a multi-projector display.
                       function
       L (u, v, i)                                The luminance projected after modification at (u, v) in a
                                                    multi-projector display when all N p (u, v) projectors are
                                                    projecting input i.
       Wl (u, v)     Smooth maximum display       The maximum luminance projected from channel l of a
                      luminance                     multi-projector display at (u, v) after modification. function of
                                                    channel l
       B (u, v)      Smooth black display         The black offset at (u, v) of a multi-projector display after
                       luminance function           modification.
       il            Modified channel input        The reprojected input for channel l .
       Sl (u, v)     Display scaling map of       The attenuation at (u, v) for channel l in a multi-projector
                       channel l                    display to achieve the smooth display luminance functions.
       Ol (u, v)     Display offset map of        The offset at (u, v) for channel l in a multi-projector display to
                       channel l                    achieve the smooth display luminance functions
       (Sl , Ol )    Display smoothing maps of    The scaling and offset map of channel l in a multi-projector
                       channel l                    display.
       Sl (s, t)     Projector scaling map of     The attenuation at (s, t) for channel l of a single projector to
                       channel l                    achieve the smooth display luminance functions.
       Ol (s, t)     Projector offset map of      The offset at (u, v) for channel l of a single projector to achieve
                       channel l                    the smooth display luminance functions.
       (Sl , Ol )    Projector smoothing maps     The scaling and the offset maps of a single projector.
       λ             Smoothing parameter          The parameter used during optimization for generating the
                                                    smooth display luminance functions.



the luminances from the three channels. However, in practice, in addition to the light projected from
the three channels, some extra leakage light is also projected at all times, commonly called the black
offset. We represent this by a spatially varying function, B(s, t), called the black luminance function.
   The maximum luminance function, Wl (s, t), for channel l at projector coordinate (s, t), is defined as
the maximum of the luminances projected by all inputs il . Note that, due to the presence of the black
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
122      •      A. Majumder and R. Stevens




                                           Fig. 1. Projector and display coordinate space.

luminance function, B(s, t), the range of luminance that is actually contributed from the channel l is
given by Wl (s, t)− B(s, t). Thus, for any other input il , the luminance contribution of channel l , Dl (s, t, il ),
is a fraction of Wl (s, t) − B(s, t),
                                              Dl (s, t, il ) = hl (il )(Wl (s, t) − B(s, t))                     (2)
where 0.0 ≤ hl (il ) ≤ 1.0. Majumder [2002] shows that hl does not vary spatially. Hence, we express hl
as a function of il only, and and call it the transfer function for channel l . Note that hl is similar to the
gamma function in other displays. In projectors, this function cannot be expressed by a power function
and, therefore, we prefer to call it the transfer function.
  Thus, the luminance projected by a practical projector at projector coordinate (s, t) is given by the
summation of the contributions from its three channels and the black luminance function as
                               L(s, t, i) = Dr (s, t, ir ) + D g (s, t, i g ) + Db(s, t, i g ) + B(s, t).
Substituting Equation (2) in the above equation,

                                  L(s, t, i) =                   hl (il )(Wl (s, t) − B(s, t)) + B(s, t).        (3)
                                                  l ∈{r, g ,b}

The maximum channel luminance functions (Wl ) and the black luminance function (B) are together
called the projector luminance functions.

Multi-Projector Display: Let N P denote the set of projectors overlapping at a display coordinate
(u, v). The luminance, L(u, v, i) at (u, v) for input i in a multi-projector display is given by the addition
of luminance L j from every projector j in N P ,
                                                  L(u, v, i) =                 L j (s j , t j , i),              (4)
                                                                      j ∈N P

where (u, v) = G j (s j , t j ), by Equation (1).2 The variation in L with respect to (u, v) describes the
photometric variation in a multi-projector display.
2 When   p is not a constant, G, Wl , and B are also dependent on p.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Photometric Seamlessness in Projection-Based Tiled Displays                  •     123




Fig. 2. The left plot shows the maximum luminance function for the green channel (W g ) in green and the black luminance
function (B) in red for a single projector display. The right picture shows the image seen by the viewer corresponding to W g in
the left plot. Note that even though this image looks seamless to our eye, the corresponding luminance plot on the left is not flat
or uniform.

3.   PERCEPTUAL PHOTOMETRIC SEAMLESSNESS
The goal of photometric seamlessness is to generate a multi-projector display that looks like a single
projector display, such that we cannot tell the number of projectors making up the display. It has been
a common assumption that to achieve this, strict photometric uniformity, that is, identical brightness
response at every pixel of the display, has to be achieved. Mathematically, a display is photometrically
uniform if, for the same input i, the luminances at any two display coordinates (u1 , v1 ) and (u2 , v2 ), are
equal, that is,
                                               ∀i, L(u1 , v1 , i) = L(u2 , v2 , i).                                               (5)
Analyzing Equation (3), we can say that in single projector displays, if the transfer function (hl (il )) is
spatially invariant, the luminance functions (Wl (s, t) and B(s, t)) should be spatially constant to achieve
strict photometric uniformity.
   However, if we analyze the behavior of a single projector, we find that though the transfer function
is indeed spatially invariant [Majumder 2002], the luminance functions of Wl and B are hardly flat,
as shown in Figures 2 and 3. But the human eye perceives the image projected by a single projector
as uniform. Thus, we realize that strict photometric uniformity is not required for a seamless display;
rather a smoothly varying response ensuring perceptual uniformity is sufficient. Formally, a display is
perceptually uniform if, for the same input i, the luminances from any two display coordinates (u1 , v1 )
and (u2 , v2 ) differ within a certain threshold that cannot be detected by the human eye, that is,
                                            ∀i, L(u1 , v1 , i) − L(u2 , v2 , i) ≤     ,                                           (6)
where is a function that can depend on various parameters like distance between (u1 , v1 ) and (u2 , v2 ),
resolution of the display, distance of viewer, viewing angle, human perception limitations, and some-
times even the task to be accomplished by the user. Further, note that strict photometric uniformity is
a special case of this perceptual uniformity when = 0, ∀(u1 , v1 ), (u2 , v2 ).
  High dynamic range and brightness are essential for good image quality [Debevec and Malik 1997;
Larson 2001]. Similarly, high lumens rating (high brightness) and low black offsets (high contrast)
are essential for good display quality. However, the criterion of perceptual uniformity alone may not
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
124      •      A. Majumder and R. Stevens




                         Fig. 3. This plot shows the zoomed in view of the black luminance function (B).

ensure a good display quality. For example, strict photometric uniformity, which implies perceptual
uniformity, does not ensure good display quality since it forces the display quality to match the ‘worst’
pixel on the display, leading to compression in dynamic range, as illustrated in Figure 7. But the goal of
perceptual uniformity (Equation (6)), being less restrictive, can provide the extra leverage to increase
the overall display quality. So, we define the problem of achieving photometric seamlessness as an
optimization problem where the goal is to achieve perceptual uniformity while maximizing the display
quality. This optimization is a general concept and can have different formalizations based on two
factors: the parameters on which of Equation (6) depends, and the way display quality is defined. We
present one such formalization while presenting our algorithm in Section 5.

4.    OUR ALGORITHM
In this section, we present our algorithm to generate perceptually seamless high quality multi-projector
displays. Our algorithm has three steps:
(1) Reconstruction: First, the parameters of function L described in Section 2 are estimated using a
    photometer and a digital camera.
(2) Modification: Next, these estimated parameters are modified by solving the optimization problem
    while taking perception into account. The hypothetical display defined by the modified parameters
    is seamless and has high dynamic range.
(3) Reprojection: Finally, this hypothetical display is realized using the practical display by factoring
    all the changes in the model parameters into any input image at interactive rates. In other words,
    the actual image projected using the hypothetical seamless display is the same as the modified
    image projected by the practical display.

4.1    Reconstruction
There are three projector parameters on which L in Equation (4) depends on: the transfer functions hl ,
and the luminance functions Wl and B. In this step, we reconstruct these parameters for each projector.

Transfer Function (hl ): Since hl (il ) is spatially constant [Majumder 2002], we use a point measure-
ment device like a photometer to estimate it at one spatial coordinate for each projector. This slower
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Photometric Seamlessness in Projection-Based Tiled Displays                    •     125




        Fig. 4. The transfer function (left) and the inverse transfer function (left) for a single channel of a projector.

measurement process (1–20 seconds-per-measurement) is justified since hl changes little temporally
[Majumder 2002] and needs to be measured very infrequently. In our experiments, measuring hl once
in 9–12 months was sufficient. However, a method proposed recently in Raij et al. [2003] can be used
to estimate hl using a camera.
   Here, we describe how we estimate h g . Other transfer functions, hr and hb, are estimated in an
analogous manner. We define input iG = (0, i g , 0), 0.0 ≤ i g ≤ 1.0 as one where the green channel input
is i g and the red and blue channel inputs are 0. We first measure the luminance response, L(s, t, iG ),
for O(28 ) values of i g and sample the input space of channel g densely. Since this luminance response
is measured at one spatial location, we omit the spatial coordinates and denote it as L(iG ) only.
   The measured luminance, L(iG ), includes the black offset and is not monotonic with respect to iG .
So, we find the maximum input range iG m ≤ iG ≤ iG M within which L(iG ) is monotonic. iG m = (0, i g m , 0)
and iG M = (0, i g M , 0) denote respectively; the lower and the upper boundary of this range. h g is then
estimated from Equation (3) as
                                                               L(iG ) − L(iG m )
                                                h g (i g ) =                       .                                                (7)
                                                               L(iG M ) − L(iG m )
h g thus estimated is a monotonic function with min∀i g h g (i g ) = h g (i g m ) = 0 and max∀i g h g (i g ) =
h g (i g M ) = 1 (Figure 4).

Luminance Functions (Wl , B): Wl and B are spatially varying functions. Hence, a digital camera
is used to estimate them using techniques presented in Majumder and Stevens [2002]. First, for a
suitable position and the orientation of the camera looking at the entire display, we estimate the
geometric warping function that transforms every projector pixel to the appropriate camera pixel.
We use camera-based geometric calibration methods [Hereld et al. 2002; Chen et al. 2002; Yang et al.
2001; Raskar 1999; Raskar et al. 1999] to estimate this transformation. Such methods also finds G,
the geometric warp between the projector and display coordinate system. Then, for each luminance
function, an appropriate test image is projected by the projector, captured by the camera, and the
luminance functions are estimated. Note that the images to estimate these functions must be taken
from the same position and orientation as the images for geometric calibration.
   The test images are comprised of identical input at every projector coordinate. From Equation (3), we
find that the input that has to be projected to estimate B is (irm , i g m , ibm ), that is, the input that projects
minimum luminance for each channel. Similarly, to estimate the maximum luminance function for the
                                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
126      •      A. Majumder and R. Stevens




Fig. 5. To compute the maximum luminance function for the green channel of each projector, we need only four pictures. This
reduction in the number of images is achieved by turning on more than one nonoverlapping projectors while capturing each
image. Top: Pictures taken for a display made of a 2 × 2 array of 4 projectors. Bottom: The pictures taken for a display made of a
3 × 5 array of 15 projectors.

green channel, W g , we have to project (irm , i g M , ibm ), that is, the input that projects maximum luminance
from the green channel and minimum luminance from the other two channels. The test images for
estimating W g of all the projectors are shown in Figure 5. Figure 2 shows W g and B estimated in the
some manner for one of these projectors. The computation of test images for the estimation of Wr and
Wb are done analogously.
4.2    Modification
The goal of the modification step is to define a hypothetical display from the given display that produces
seamless image. To achieve our goal of making a multi-projector display look like a single projector
display, we define the parameters for a multi-projector display analogous to the parameters hl , Wl ,
and B for a single projector display. Perceptual factors are taken into account while defining these
parameters. Unlike a single projector display where hl is spatially constant, different parts of the
multi-projector display have different hl s since they are projected from different projectors. Further,
for the same reason, functions analogous to the luminance functions (Wl and B) of a single projector
cannot be defined for a multi-projector display. So, to achieve our goal, the following modifications are
performed.
(1) A perceptually appropriate common transfer function Hl is chosen that is spatially invariant
    throughout the multi-projector display.
(2) Luminance functions Wl and B for the entire multi-projector display are identified.
(3) Finally, these luminance functions are made perceptually uniform like that of a single projector
    display (Figure 2).
  4.2.1 Choosing a Common Transfer Function. We choose a common transfer function Hl for each
channel l , that satisfies the three conditions: Hl (0) = 0, Hl (1) = 1, and Hl (il ) is monotonic. The transfer
function hl of all the different projectors is then replaced by this common Hl to assure a spatially
invariant transfer function across the whole multi-projector display. So, Equation (4) becomes

                        L(u, v, i) =           Hl (il )       (Wl j (s j , t j ) − B j (s j , t j )) +       B j (s j , t j ),   (8)
                                          l               j                                              j

where Wl j and B j are the luminance functions for projector P j .
  We use Hl = il2 which is commonly used to approximate the logarithmic response of the human eye
to varying luminance. In previous methods [Majumder and Stevens 2002], Hl was chosen to be a linear
function, Hl (il ) = il , and resulted in washed out images.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                        Photometric Seamlessness in Projection-Based Tiled Displays                          •     127

  4.2.2   Identifying the Display Luminance Functions. In Equation (8), using
                                             Wl (u, v) =          Wl j (s j , t j )
and
                                              B(u, v) =           B j (s j , t j ),
with appropriate coordinate transformation between projector and display coordinate space, we get

                         L(u, v, i) =                  Hl (il )(Wl (u, v) − B(u, v)) + B(u, v).                                     (9)
                                        l ∈{r, g ,b}

Note that the above equation for the whole multi-projector display is exactly analogous to that of a
single projector (Equation (3)). Hence, we call Wl and B the maximum display luminance function for
channel l and the black display luminance function, respectively. Figure 6 shows W g for a four projector
display, and W g and B for a fifteen projector display.
   4.2.3 Modifying the Display Luminance Functions. The multi-projector display defined by
Equation (9) still does not look like a single projector display because, unlike the single projector lumi-
nance functions (Figure 2), the analogous multi-projector luminance functions (Figure 6) have sharp
discontinuities that result in perceivable seams in the display. However, perception studies show that
humans are sensitive to significant luminance discontinuities, but can tolerate smooth luminance vari-
ations [Chorley and Laylock 1981; Goldstein 2001; Valois and Valois 1990]. So, to make these functions
perceptually uniform, we smooth Wl (u, v) and B(u, v) to generate Wl (u, v) and B (u, v), and call them
the smooth display luminance functions.
   We formulate this smoothing as an optimization problem where we find a Wl (u, v) that minimizes the
deviation from the original Wl (u, v), assuring high dynamic range, and, at the same time, maximizes
its smoothness, assuring perceptual uniformity. The smoothing criteria is chosen based on quantitative
measurement of the limitation of the human eye in perceiving smooth luminance variations. We present
a constrained gradient-based smoothing method to find the optimal solution to this problem. This
smoothing is explained in detail in Section 5.
   Figure 6 shows W g (u, v) for a fifteen-projector display. The perceptually seamless luminance response
of the display L (u, v, i) generated is derived from Equation (9):

                        L (u, v, i) =                  Hl (il )(Wl (u, v) − B (u, v)) + B (u, v).                                 (10)
                                        l ∈{r, g ,b}

When both Wl (u, v) and B (u, v) are smooth, and the transfer function Hl is spatially invariant, L (u, v, i))
is also smooth and satisfies Equation (6).

4.3   Reprojection
In the previous section, we theoretically modified the display parameters to generate a hypothetical
display that would project a seamless imagery. Now we have to make the practical display behave
like the hypothetical display with these modified parameters. The projector hardware does not offer us
precision control to directly modify Wl (u, v), B(u, v), and hl to create the hypothetical display. So, we
achieve the effects of these modifications by changing only the input il at every projector coordinate.
This step is called reprojection. In this section, we explain how this modification is achieved for any one
projector of a multi-projector display, at its coordinate (s, t). Since the modification is pixel-based, we
have left out the (s, t) from all the equations in this section.
                                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
128      •      A. Majumder and R. Stevens




Fig. 6. Left Column: The estimated maximum display luminance function of green channel (W g ) for a 2 × 2 array of projectors
(top). The estimated maximum display luminance function for green channel (W g ) and the black display luminance function (B) of
a 3×5 array of projectors (bottom). The high luminance regions in both correspond to the overlap regions across different projectors.
Right Column: Smooth maximum display luminance function for green channel (W g ) achieved by applying the constrained
gradient-based smoothing algorithm on the corresponding maximum display luminance function on the left column. Note that
these are perceptually smooth even though they are not geometrically smooth.


  For a given il , the actual response of the display is given by,

                                                                   (hl (il )(Wl − B)) + B.
                                                    l ∈{r, g ,b}


The goal of reprojection is to get the response

                                                                  (Hl (il )(Wl − B )) + B
                                                   l ∈{r, g ,b}
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                Photometric Seamlessness in Projection-Based Tiled Displays                         •     129

to simulate a modified perceptually seamless display. So, we modify the input il to il such that
                                         (Hl (il )(Wl − B )) + B =                      (hl (il )(Wl − B)) + B.                          (11)
                          l ∈{r, g ,b}                                   l ∈{r, g ,b}

It can be shown that the following il solves Equation (11).
                                                          −1
                                                    il = hl (Hl (il )Sl + Ol ),                                                          (12)
        −1
where hl is the inverse transfer function of a projector and can be computed directly from hl as shown
Figure 4, and Sl (u, v) and Ol (u, v) are called the display scaling map and display offset map, respectively,
and are given by
                                          Wl (u, v) − B (u, v)                 B (u, v) − B(u, v)
                        Sl (u, v) =                            ; Ol (u, v) =                        .                                    (13)
                                          Wl (u, v) − B(u, v)                3(Wl (u, v) − B(u, v))
Intuitively, the scaling map represents the pixel-wise attenuation factor needed to achieve the smooth
maximum display luminance functions. The offset map represents the pixel-wise offset factor that is
needed to correct for the varying black offset across the display. Together, they comprise what we call the
display smoothing maps for each channel. From these display smoothing maps, the projector smoothing
maps, Sl j (s j , t j ) and Ol j (s j , t j ), for channel l of projector P j are cut out, using the geometric warp G j
as follows,
                            Sl j (s j , t j ) = Sl (G j (s j , t j )); Ol j (s j , t j ) = Ol (G j (s j , t j )).                        (14)
Figure 9 shows the scaling maps thus generated for the whole display and one projector.
  Therefore, any image projected by a projector can be corrected by applying the following three steps
in succession to every channel input, as illustrated in Figure 9.
(1) First, the common transfer function is applied to the input image.
(2) Then, the projector smoothing maps are applied. This involves pixel-wise multiplication of the
    attenuation map, and then, the addition of the the offset map.
(3) Finally, the inverse transfer function of the projector is applied to generate the corrected image.

5.   SMOOTHING DISPLAY LUMINANCE FUNCTIONS
In this section, we describe in detail the method to generate W and B from W and B, respectively. We
approximate B (u, v) as
                                                    B (u, v) = max B(u, v),
                                                                     ∀u,v

since the variation in B(u, v) is almost negligible when compared to Wl (u, v) (Figure 6). B (u, v) defines
the minimum luminance that can be achieved at all display coordinates.
   We use the constrained gradient-based smoothing method to generate Wl (u, v) from Wl (u, v). Wl (u, v)
is defined by the following optimization constraints.
   Capability Constraint. This constraint of Wl ≤ Wl ensures that Wl never goes beyond the maximum
luminance achievable by the display, Wl . In practice, with discrete sampling of these functions,
                                                W [u][v] < W[u][v],                      ∀u, v.
  Perceptual Uniformity Constraint. This constraint assures that Wl has a smooth variation, imper-
ceptible to humans.
                                                          ∂Wl  1
                                                              ≤ × Wl ,
                                                           ∂x  λ
                                                                                    ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
130      •      A. Majumder and R. Stevens

                                                        ∂W
where λ is the smoothing parameter and ∂ x l is the gradient of Wl , along any direction x. Compare this
inequality with Equation (6). In the discrete domain, when the gradient is expressed as a linear filter
involving the eight neighbors (u , v ) of a pixel (u, v), u ∈ {u − 1, u, u + 1} and v ∈ {v − 1, v, v + 1}, this
constraint is given by
                                  |W [u][v] − W [u ][v ]|         1
                                                              ≤     W [u][v],         ∀u, v, u , v .
                                      |u − u |2 + |v − v |2       λ
 Display Quality Objective Function. The above two constraints can yield many feasible Wl . To maxi-
mize dynamic range, the integration of Wl has to be maximized. In discrete domain, this is expressed as
                                                              X −1 Y −1
                                                   maximize               W [u][v],
                                                              u=0 v=0

where X and Y denote the height and width of the multi-projector display in number of pixels.
   We have designed a fast and efficient dynamic programming method that gives the optimal solution
for this optimization in linear time with respect to the number of pixels in the display, that is, O(XY).
The time taken to compute this solution on an Intel Pentium III 2.4GHz processor for displays with
9 million pixels is less than one second. The pseudo-code for the algorithm is given in Appendix A.
   The solution to the above optimization problem smooths the luminance variation across the display.
The general idea that smoothing the luminance response would achieve seamless results has been used
effectively in the image processing domain in the past [Gonzalez and Woods 1992; Land 1964; Land
and McCann 1971]. However, the luminance variation correction for multi-projector displays cannot be
achieved just by smoothing. For example, gradient-or curvature-based linear smoothing filters, which
are popular operators in image processing applications, smooth the hills and fill the valleys. However,
our constraints are such that while the hills can be smoothed, the troughs cannot be filled up since the
response so achieved will be beyond the display capability of the projectors. So, the desired smoothing
for this particular application was formalized as an optimization problem.
   Finally, note that this method not only adjusts the variations in the overlap region, but corrects
for intra and inter projector variations also, without treating any one of them as a special case. This
differentiates our method from any existing edge-blending method that addresses overlap regions only.

5.1    Smoothing Parameter
The smoothing parameter λ is derived from the human contrast sensitivity function (CSF) [Valois and
Valois 1990; Chorley and Laylock 1981]. Contrast threshold defines the minimum percentage change
in luminance that can be detected by a human being at a particular spatial frequency. The human CSF
is a plot of the contrast sensitivity (reciprocal of contrast threshold) with respect to spatial frequency
(Figure 8). The CSF is bow-shaped with maximum sensitivity at a frequency of 5 cycle-per degree of angle
subtended on the eye. In other words, a variation of less than 1% in luminance will be imperceptible
to the humans eye for a spatial grating of 5 cycles/degree. For other frequencies, a greater variation
can be tolerated. This fact is used to derive the relative luminance that a human being can tolerate at
every pixel of a display as follows.
   Let d be the perpendicular distance of the user from the display (can be estimated by the position
of the camera used for calibration), r is the resolution of the display in pixels-per-unit-distance, and τ
is the contrast threshold that humans can tolerate per-degree of visual angle (1% at peak sensitivity).
From this, the number of pixels subtended, per degree of the human eye, is given by d πr . Since the
                                                                                            180
peak sensitivity occurs at 5 cycles per degree of visual angle, the number of display pixels per cycle of
                          d πr
the grating is given by 180×5 . Within the above pixels, a luminance variation of τ will go undetected by
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                      Photometric Seamlessness in Projection-Based Tiled Displays             •     131

human beings. Thus,
                                                    d πr
                                                λ=        .                                          (15)
                                                    900τ
  For our fifteen-projector display, r is about 30 pixels-per-inch. For a user about 6 feet away from
the display, by substituting τ = 0.01 in Equation (15), we get a λ of about 800. Note that as the user
moves farther away from the display λ goes up, that is, the surface needs to be smoother and will have
lower dynamic range. This also explains the variations perceptible in some of our results in Figure 7,
10, 11, and 12, though they are imperceptible when seen in person on the large display. Since the
results in this article are highly scaled down images of the display, they simulate a situation where the
user is infinitely far away from the display and, therefore, infinite lambda is required to make all
the luminance variations imperceptible. The smoothing parameter chosen for a large display is much
smaller and hence is not suitable for small images on paper.

6.   IMPLEMENTATION
The algorithm pipeline is illustrated in Figure 9. Reconstruction, modification, and part of the repro-
jection are done offline. These comprise the calibration step. The outputs of this step are the projector
smoothing maps, the projector inverse transfer functions, and the common transfers function for each
channel. These are then used in the per projector image correction step to correct any image projected
on the display, using Equation (12).

Scalability: The limited resolution of the camera can affect the scalability of the reconstruction for very
large displays (made of 40–50 projectors). So, we used techniques presented in Majumder et al. [2003]
to design a scalable version of the reconstruction method. We rotate and zoom the camera to estimate
the luminance functions of different parts of the display from different views. These are then stitched
together to create the luminance function of the whole display. The results of this scalable version of
our algorithm are shown in Figure 10. The details of this method are available in Majumder et al. [2003].

Real Time Implementation: We achieve the per-pixel image correction interactively in commodity
graphics hardware using pixels shaders. In our real-time correction, the scene is first rendered to tex-
ture, and then the luminance corrections are applied. Multi-texturing is used for applying the common
transfer function and the smoothing maps. The inverse transfer function is applied using dependent 2D
texture look-ups. This is followed by an image warp to correct the geometric misalignments. Choosing
the common transfer function i 2 , as opposed to i 1.8 (as used in the Macintosh Operating System) and
i 2.2 (as used in Windows), helps us to implement this step in the pixel shader using multi-texturing as
a multiplication of the image by itself.

7.   RESULTS
We have applied our algorithm in practice to display walls of different sizes at Argonne National
Laboratory. Figures 7, 11 and 12 show the digital photographs of our results on a 2 × 3 array of six
projectors (4-feet wide and 3-feet high), 3 × 5 array of fifteen projectors(10-feet wide and 8-feet high),
and a 2 × 2 (2.5-feet wide and 1.5-feet high) array of four projectors.
   The most popular approach for color correction to date is to use software edge-blending in the overlap
regions, or hardware edge-blending with custom-designed metal masks and mounts on the optical path
of each projector. Both of these methods are featured on the display walls at Argonne National Labora-
tory. The results of these solutions, illustrated in Figure 13, still show the seams. Since these solutions
assume constant luminance function for each projector, their correction results in darker boundaries
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
132      •      A. Majumder and R. Stevens




Fig. 7. Digital photographs of a fifteens projector tiled display (8 × 10 in size) before any correction (top), after constraint
gradient-based smoothing with smoothing parameter of λ = 400, (second from top), λ = 800 (third from top), and after photometric
uniformity, that is, λ = ∞ (bottom). Note that the dynamic range of the display reduces as the smoothing parameter increases.
λ = ∞ is the special case of photometric uniformity where the display has a very low dynamic range.

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                             Photometric Seamlessness in Projection-Based Tiled Displays                  •     133




Fig. 8. The contrast sensitivity function (CSF) of the human eye. The top-most curve is for grating with brightness (mean) of
5-foot Lamberts. As the brightness of the grating decreases, the contrast sensitivity decreases as shown by the lower curves, for
0.5-, 0.05-, 0.005-, and 0.005-foot Lamberts (courtsey Valois and Valois [1990]).




Fig. 9. This figure illustrates the complete algorithm. For the sake of simplicity, we have not included the black luminance
functions and the offset maps in this figure.

around projectors. Further, these solutions are rigid and expensive to maintain. The photometric unifor-
mity method we developed to alleviate the problem, presented in Majumder and Stevens [2002, 2004],
results in severe compression in the dynamic range of the display. In addition, the transfer functions
are not addressed correctly and hence the method leads to washed out images. Figure 7 compares the
results of the new method presented in this article, with one that achieves photometric uniformity.
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
134      •      A. Majumder and R. Stevens




Fig. 10. Digital photographs of a display made of a 2 × 4 array of eight projectors (4 × 8 in size) . This display was corrected
using the scalable version of our algorithm. The luminance functions of the left and right half of this eight-projector display (each
made of 2 × 2 array of four projectors) were estimated from two different camera positions. They were then stitched together to
create the display luminance functions. Top: Before correction. Bottom: After constrained gradient-based luminance smoothing.


   Further, our luminance smoothing does not introduce additional chrominance seams. Majumder
[2003] shows that if the shape of the maximum luminance function across different channels are similar,
chrominance seams are not possible. To assure a similar shape of the modified luminance functions
across different channels, even after the application of our method, we apply the smoothing to the
normalized luminance functions for each channel. Also, since a camera cannot provide us with, an
absolute measurement of the luminance variations in the first place, this does not affect the accuracy
of the method in any way.
   Further, Equation (4) assumes view-independent or Lambertian displays. Since our screens are non-
Lambertian in practice, our correction is accurate from the position of the camera used for reconstruct-
ing the model parameters. However, our result looks seamless for a wide range of viewing angles and
distances from the wall. We use a Jenmar screen in a back projection system that has a gain of approx-
imately 2.0. For this display, we see no artifacts if the view direction makes an angle of about 20–90
degrees with the plane of the screen. Only for less than 20 degrees are the boundaries of the projectors
visible—but as smooth edges, to some extent like the result of an edge-blending method. Figure 14
shows the effect.
   Typically, the black offset, B, is less than 0.4% of the maximum display luminance function, Wl ,
and has negligible effect on the smoothing maps. We implemented an alternate version of our system
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                             Photometric Seamlessness in Projection-Based Tiled Displays                 •     135




Fig. 11. Digital photographs of a display made of a 3 × 5 array of fifteen projectors (10-feet wide and 8-feet high). Top: Before
correction. Bottom: Perceptual photometric seamlessness after applying constrained gradient-based luminance smoothing.

assuming B = 0. Except for a slight increase in the dynamic range, the results were very similar to
those produced by our implementation without this assumption.
  We have run interactive 3D applications using our real-time implementation. First, the 3D model is
rendered to a texture of the size of each projector image. This step is essential for any geometry, and
                                                                         ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
136      •      A. Majumder and R. Stevens




Fig. 12. Digital photographs of displays made of a 2 × 2 and a 2 × 3 array of four (1.5 × 2.5 in size) and six (3 × 4 in size)
projectors, respectively. Left: Before correction. Right: After constrained gradient-based luminance smoothing. Note that we are
able to achieve perceptual seamlessness even for flat colors (top), the most critical test image for our algorithm.




Fig. 13. The fifteen-projector tiled display before blending (left), after software blending (middle), and after optical blending
using a physical mask (right).


color corrections reduces the frame rate by about 25–40%. However, the luminance correction which
is applied to this texture is independent of the model size and reduces the frame rate by a constant,
about 10–20 frames-per-second. The details of the real-time implementation are available at Binns
et al. [2002].

8.    CONCLUSION
We have demonstrated that smoothing the photometric response of the display, based on a perceptual
criteria (far less restrictive than strict photometric uniformity), is effective to achieve perceptual unifor-
mity in multi-projector displays. We achieve such a smoothing by solving an optimization problem that
minimizes the perceptible photometric variation and maximizes the dynamic range of the display. We
have presented an efficient, automatic, and scalable algorithm that solves this optimization to generate
photometrically seamless (but not uniform) high-quality multi-projector displays.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                            Photometric Seamlessness in Projection-Based Tiled Displays                •     137




Fig. 14. The correction applied to a non-Lambertian screen and viewed from a view direction that makes an angle of less than
20 degrees with the screen.

   However, we believe that our work is just the first step towards solving the more general problem
of color seamlessness in multi-projector displays. Though we do not deal with chrominance explic-
itly, if all projectors have identical red, green, and blue chromaticity, setting the red, green, and blue
transfer functions to match across all projectors will balance the red, green, and blue mixture to give
a common white point and grayscale. This is the best that can be done without considering gamut
mapping techniques. However, we can envision devising a 5D optimization method that considers the
gamut of each projector while smoothing the 5D color response (one parameter for luminance, two pa-
rameters for chrominance, and two parameters for spatial coordinates). Such a method would address
chrominance issues in practical displays where no two projectors usually have identical red, green,
and blue chromaticity. Further, to enable different defense applications, self-calibrating systems that
can correct themselves in real-time from arbitrary images projected on the display, need to be devised.
Finally, we need to design a perceptual metric to evaluate the results of the color correction methods
quantitatively.

APPENDIX

A.   THE SMOOTHING ALGORITHM
∀(u, v),               W (u, v) ← W(u, v)
δ←1  λ

for u = 0 to X − 1
  for v = 0 to Y − 1             √
    W (u, v) ← min(W (u, v), (1 + 2δ)W (u − 1, v − 1),
                   (1 + δ)W (u − 1, v), (1 + δ)W (u, v − 1))

for u = X − 1 down to 0
  for v = 0 to Y − 1             √
    W (u, v) ← min(W (u, v), (1 + 2δ)W (u + 1, v − 1),
                   (1 + δ)W (u + 1, v), (1 + δ)W (u, v − 1))
                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
138      •      A. Majumder and R. Stevens


for u = 0 to X − 1
  for v = Y − 1 down to 0        √
    W (u, v) ← min(W (u, v), (1 + 2δ)W (u − 1, v + 1),
                   (1 + δ)W (u − 1, v), (1 + δ)W (u, v + 1))

for u = X − 1 down to 0
  for v = Y − 1 to 0             √
    W (u, v) ← min(W (u, v), (1 + 2δ)W (u + 1, v + 1),
                   (1 + δ)W (u + 1, v), (1 + δ)W (u, v + 1))

ACKNOWLEDGMENTS
We thank Sandy Irani of the Department of Computer Science at the University of California, Irvine,
for helping us to find an optimal dynamic programming solution to our optimization problem and
proving its optimality. We thank David Jones, Matthew McCrory, and Michael E. Papka of Argonne
National Laboratory for helping with the real-time implementation of our method on the multi-projector
displays. We thank Mark Hereld of the Argonne National Laboratory; Gopi Meenakshisundaram of the
Department of Computer Science at the University of California Irvine; Herman Towles, Henry Fuchs,
Greg Welch, Anselmo Lastra, and Gary Bishop of Department of Computer Science, University of North
Carolina at Chapel Hill, for several insightful discussions during the course of this work.

REFERENCES
BINNS, J., GILL, G., HERELD, M., JONES, D., JUDSON, I., LEGGETT, T., MAJUMDER, A., MCCROY, M., PAPKA, M. E., AND STEVENS, R. 2002.
  Applying geometry and color correction to tiled display walls (poster). IEEE Visualization.
BUCK, I., HUMPHREYS, G., AND HANRAHAN, P. 2000. Tracking graphics state for networked rendering. Proceedings of Eurograph-
  ics/SIGGRAPH Workshop on Graphics Hardware, 87–95.
CAZES, A., BRAUDAWAY, G., CHRISTENSEN, J., CORDES, M., DECAIN, D., LIEN, A., MINTZER, F., AND WRIGHT, S. L. 1999. On the color
  calibration of liquid crystal displays. SPIE Conference on Display Metrology, 154–161.
CHEN, C. J. AND JOHNSON, M. 2001. Fundamentals of scalable high resolution seamlessly tiled projection system. Proceedings
  of SPIE Projection Displays VII 4294, 67–74.
CHEN, H., SUKTHANKAR, R., WALLACE, G., AND LI, K. 2002. Scalable alignment of large-format multi-projector displays using
  camera homography trees. Proceedings of IEEE Visualization, 339–346.
CHORLEY, R. AND LAYLOCK, J. 1981. Human factor consideration for the interface between electro-optical display and the human
  visual system. In Displays, Vol. 4.
CRUZ-NEIRA, C., SANDIN, D. J., AND A. DEFANTI, T. 1993.          Surround-screen projection-based virtual reality: The design and
  implementation of the CAVE. In Proceedings of ACM SIGGRAPH, 135–142.
DEBEVEC, P. E. AND MALIK, J. 1997. Recovering high dynamic range radiance maps from photographs. Proceedings of ACM
  SIGGRAPH, 369–378.
GOLDSTEIN, E. B. 2001. Sensation and Perception. Wadsworth Publishing Company.
GONZALEZ, R. C. AND WOODS, R. E. 1992. Digital Image Processing. Addison Wesley.
HERELD, M., JUDSON, I. R., AND STEVENS, R. 2002. Dottytoto: A measurement engine for aligning multi-projector display systems.
  Argonne National Laboratory preprint ANL/MCS-P958-0502.
HUMPHREYS, G., BUCK, I., ELDRIDGE, M., AND HANRAHAN, P. 2000. Distributed rendering for scalable displays. Proceedings of
  IEEE Supercomputing.
HUMPHREYS, G., ELDRIDGE, M., BUCK, I., STOLL, G., EVERETT, M., AND HANRAHAN, P. 2001. Wiregl: A scalable graphics system for
  clusters. Proceedings of ACM SIGGRAPH, 129–140.
HUMPHREYS, G. AND HANRAHAN, P. 1999. A distributed graphics system for large tiled displays. In Proceedings of IEEE Visual-
  ization. 215–223.
LAND, E. 1964. The retinex. Amer. Scient. 52, 2, 247–264.
LAND, E. AND MCCANN, J. 1971. Lightness and retinex theory. J. Opti. Soc. Amer. 61, 1, 1–11.
LARSON, G. W. 2001. Overcoming gamut and dynamic range limitation in digital images. SIGGRAPH Course Notes.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                Photometric Seamlessness in Projection-Based Tiled Displays                    •     139

LI, K., CHEN, H., CHEN, Y., CLARK, D. W., COOK, P., DAMIANAKIS, S., ESSL, G., FINKELSTEIN, A., FUNKHOUSER, T., KLEIN, A., LIU, Z., PRAUN,
  E., SAMANTA, R., SHEDD, B., SINGH, J. P., TZANETAKIS, G., AND ZHENG, J. 2000. Early experiences and challenges in building and
  using a scalable display wall system. IEEE Comput. Graph. Applicat. 20, 4, 671–680.
MAJUMDER, A. 2002. Properties of color variation across multi-projector displays. Proceedings of SID Eurodisplay, 807–810.
MAJUMDER, A. 2003. A practical framework to achieve perceptually seamless multi-projector displays, phd thesis. Tech. rep.,
  University of North Carolina at Chapel Hill.
MAJUMDER, A., HE, Z., TOWLES, H., AND WELCH, G. 2000. Achieving color uniformity across multi-projector displays. Proceedings
  of IEEE Visualization, 117–124.
MAJUMDER, A., JONES, D., MCCRORY, M., PAPKA, M. E., AND STEVENS, R. 2003. Using a camera to capture and correct spatial
  photometric variation in multi-projector displays. IEEE International Workshop on Projector-Camera Systems.
MAJUMDER, A. AND STEVENS, R. 2002. LAM: Luminance attenuation map for photometric uniformity in projection based displays.
  Proceedings of ACM Virtual Reality and Software Technology, 147–154.
MAJUMDER, A. AND STEVENS, R. 2004. Color nonuniformity in projection-based displays: Analysis and solutions. IEEE Trans.
  Visualiz. Comput. Graph. 10, 2 (March/April), 177–188.
NAYAR, S. K., PERI, H., GROSSBERG, M. D., AND BELHUMEUR, P. N. 2003.             IEEE International Workshop on Projector-Camera
  Systems.
PAILTHORPE, B., BORDES, N., BLEHA, W., REINSCH, S., AND MORELAND, J. 2001. High-resolution display with uniform illumination.
  Proceedings Asia Display IDW, 1295–1298.
RAIJ, A., GILL, G., MAJUMDER, A., TOWLES, H., AND FUCHS, H. 2003. Pixelflex2: A comprehensive, automatic, casually-aligned
  multi-projector display. IEEE International Workshop on Projector-Camera Systems.
RASKAR, R. 1999. Immersive planar displays using roughly aligned projectors. In Proceedings of IEEE Virtual Reality 2000.
  109–116.
RASKAR, R., BROWN, M., YANG, R., CHEN, W., TOWLES, H., SEALES, B., AND FUCHS, H. 1999. Multi projector displays using camera
  based registration. Proceedings of IEEE Visualization, 161–168.
RASKAR, R., WELCH, G., CUTTS, M., LAKE, A., STESIN, L., AND FUCHS, H. 1998. The office of the future: A unified approach to image
  based modeling and spatially immersive display. In Proceedings of ACM SIGGRAPH, 168–176.
SAMANTA, R., ZHENG, J., FUNKHOUSE, T., LI, K., AND SINGH, J. P. 1999. Load balancing for multi-projector rendering systems. In
  SIGGRAPH/Eurographics Workshop on Graphics Hardware. 107–116.
STONE, M. C. 2001a. Color and brightness appearance issues in tiled displays. IEEE Comput. Graph. Applicat., 58–66.
STONE, M. C. 2001b. Color balancing experimental projection displays. 9th IS&T/SID Color Imaging Conference, 342–347.
VALOIS, R. L. D. AND VALOIS, K. K. D. 1990. Spatial Vision. Oxford University Press.
YANG, R., GOTZ, D., HENSLEY, J., TOWLES, H., AND BROWN, M. S. 2001. Pixelflex: A reconfigurable multi-projector display system.
  Proceedings of IEEE Visualization, 167–174.


Received May 2004; revised August 2004; accepted September 2004




                                                                               ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
Controllable Smoke Animation With Guiding Objects
LIN SHI and YIZHOU YU
University of Illinois at Urbana-Champaign


This article addresses the problem of controlling the density and dynamics of smoke (a gas phenomenon) so that the synthetic
appearance of the smoke (gas) resembles a still or moving object. Both the smoke region and the target object are represented
as implicit functions. As a part of the target implicit function, a shape transformation is generated between an initial smoke
region and the target object. In order to match the smoke surface with the target surface, we impose carefully designed velocity
constraints on the smoke boundary during a dynamic fluid simulation. The velocity constraints are derived from an iterative
functional minimization procedure for shape matching. The dynamics of the smoke is formulated using a novel compressible
fluid model which can effectively absorb the discontinuities in the velocity field caused by imposed velocity constraints while
reproducing realistic smoke appearances. As a result, a smoke region can evolve into a regular object and follow the motion of
the object, while maintaining its smoke appearance.
Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling—Physically
based modeling; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.6.8 [Simulation and
Modeling]: Types of Simulation—Animation
General Terms: Algorithms, Experimentation, Theory
Additional Key Words and Phrases: Constrained animation, fluid simulation, implicit functions, level sets, shape matching, shape
transformations, velocity constraints




1.   INTRODUCTION
Amorphous but elegantly moving matters, such as clouds, fog, and smoke, give people plenty of space
for imagination. We are excited when a cloud in the sunset sky assumes the approximate shape of an
animal or some other real object. It is indeed an exhilarating event because of its rareness. For the
same reason, ghosts and deities are usually described as manifesting themselves from smoke or clouds.
A famous fairy tale has the following scene:
       As Aladdin rubbed the lamp to try to get a better look, the lamp came to life; the lamp launched
       a long, blue stream upward; the blue smoke rose toward the ceiling, and finally became an
       enormous, blue genie!
  We would like to develop techniques for digitally reproducing similar effects. Such techniques have
many applications in the entertainment industry, especially in advertising and film making. In some of
the recent movies, there have been voxel water horses emerging from a flooding river [Kapler 2002], and

This work was supported by the National Science Foundation CCR-0132970 and start-up funds from the University of Illinois
at Urbana Champaign.
Authors’ address: Department of Computer Science, University of Illinois, 201 North Goodwin Avenue, Urbana, IL 61801; email:
{linshi,yyz}@cs.uiuc.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0140 $5.00
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 140–164.
                                              Controllable Smoke Animation With Guiding Objects              •     141

the mummy manifests itself from sand. Our goal in this article is to introduce methods that produce
physically plausible motion for a gas phenomenon, which at the same time, assumes a recognizable
static or dynamic shape. In the rest of the article, we choose smoke as a representative of such gas
phenomena. Nevertheless, the approach introduced here is not only limited to smoke.
  Our goal in this article has the following implications:
—the motion during two-way transitions between irregular smoke regions and regular object shapes
 should be natural and have realistic smoke appearances;
—the global shape of the smoke should be able to approximate a static or moving object for an arbitrarily
 long period of time while maintaining its characteristic local structure and motion;
—When smoke objects interact with each other or with the environment, the objects should exhibit the
 properties of smoke so that a strong wind or other regular objects can easily destroy the shape of
 such objects.
An example with a smoke horse is shown in Figure 1(a) to illustrate this goal.
   Smoke consists of a collection of light-scattering tiny particles floating in the air. Creating dramatic
effects is challenging since the smoke density in a fluid medium always tends to drift from a nonuniform
distribution to a uniform one. Solving the proposed problem requires the maximum level of control of
this process while maintaining a believable appearance of smoke. When there is a conflict between
controllability and physics rules, we choose to relax the physics rules since the desired effects can
largely be considered a supernatural phenomenon. Our goal in this article is consistent with one of
the general objectives of graphics research: the development of techniques that allow easy user-level
control of the modeling and animation processes.
   This article presents an effective solution to the proposed problem. Our solution involves implicit
functions (level sets) defined for both smoke-object transitions and object motion. Since they are func-
tions of both space and time, these level sets represent both the shape of the target objects as well as
their evolution over time. These implicit functions serve as the underlying “storyboard” guiding the
motion of the smoke. Thus, the problem becomes how to impose constraints on the motion of the smoke
so that the smoke density distribution approximately matches these evolving level sets while the real-
istic appearance of smoke is being maintained. This is actually a control problem that can be solved by
a dynamic feedback process. The basic idea of our solution lies in the use of artificial feedback forces on
the smoke so that subtle changes in the movement of the smoke reduces the shape discrepancy between
the smoke and the target object. Such feedback forces are actually realized by velocity adjustments and
constraints. They do not exist in the real world, and need to be carefully orchestrated to achieve the
desired effects.
   Major contributions of this article include an overall framework for solving the proposed problem,
an automatic scheme for target object matching based on velocity constraints imposed on the motion
of the smoke, and an empirical compressible fluid model for effectively integrating constraints into
the velocity field. These velocity constraints are derived from a shape matching functional. Simple but
effective methods for smoke objects to interact with each other or the environment are also developed.

2.   RELATED WORK
The modeling of smoke and other gaseous phenomena has received much attention from the computer
graphics community over the last two decades. Early models focused on a particular phenomenon and
animated the smoke density directly without modeling its velocity [Gardner 1985; Perlin 1985; Perlin
and Hoffert 1989; Ebert et al. 1998]. Additional details were added using solid textures whose parame-
ters were animated over time. A common feature shared by these models and the voxel tool presented
in Kapler [2002] is that they lack any dynamical feedback which is crucial to realistic animation.
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
142      •      L. Shi and Y. Yu




Fig. 1. (a) An initial smoke blob evolves into a smoke horse. A wind blows the head away. When the wind recedes, the head
grows back. (b) The underlying shape transformation for the smoke-horse transition.


ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                              Controllable Smoke Animation With Guiding Objects              •     143

   A more natural way to model the motion of smoke is to simulate the equations of fluid dynamics. In
recent work, relatively coarse grids were used in Foster and Metaxas [1997b] to produce nice smoke
motion in three-dimensions. Their simulations are only stable if the time step is sufficiently small.
To alleviate this problem and make the simulations more efficient, Stam [1999] introduced a new
model which is unconditionally stable and could be run at any speed in. This was achieved using a
combination of a semi-Lagrangian advection scheme [Staniforth and Cote 1991] and implicit solvers.
Vorticity confinement and a higher-order interpolation technique were introduced in Fedkiw et al.
[2001]. As a result, the simulations can keep finer details on relatively coarse grids. Recently, this fluid
simulation method has been generalized to surfaces of arbitrary topology, including Catmull-Clark
subdivision surfaces [Stam 2003] and triangle meshes [Shi and Yu 2004].
   The level set method [Osher and Sethian 1988; Sethian 1999; Osher and Fedkiw 2001] is a robust
computational method to track the surface of a volume of fluid. In Foster and Fedkiw [2001], Foster
and Fedkiw made important contributions to fluid simulation and control through the introduction of
a hybrid liquid volume model, combining implicit surfaces and massless marker particles. The work
was further improved in terms of accuracy in Enright et al. [2002] by using particles on both sides of
the interface between air and water. The level set method has also inspired interesting methods for
fast surface reconstruction from unorganized points [Zhao et al. 2001], and geometric object editing
[Museth et al. 2002]. What is important in these methods is the quality of the resulting static geometry
instead of the dynamic surface evolution itself. In comparison, we are more concerned with the quality
of the dynamics themselves for the realistic appearance of smoke.
   In terms of fluid control, Stam [1996] applied multiple types of vector fields to control the global
trajectories of fluids. Foster and Metaxas [1997a] introduced embedded controllers that allow animators
to specify and control a three-dimensional fluid animation. In Foster and Fedkiw [2001] and Lamorlette
and Foster [2002], animator-designed “fake” space curves and surfaces can be applied to control the
motion and structures of fluids. The tangents of the curves or the normals of the surfaces indicate the
directions of motion. However, none of these attempts allows the user to enforce higher-level objectives,
such as matching the shape of a still or moving object.
   A technique to make fluids match regular shapes was first reported in Yu and Shi [2002], which
involves carefully designed force fields and a modified diffusion equation for smoke gathering. A similar
approach that can produce results with improved visual quality was also taken by Fattal and Lischinski
[2004]. Although being able to control the smoke density, the smoke gathering term in these techniques
tends to generate “ghosting” effects, which means that the target shape looks like it is simply emerging
from an amorphous cloud of smoke. In comparison, the level set method, taken by this article, can better
control the smoke shape without introducing this type of artifact.
   A gradient-based iterative optimization approach for controlling fluid simulations through user-
specified objectives was introduced in Treuille et al. [2003] and McNamara et al. [2004]. The problem is
cast as matching dynamically-evolved fluid density with the specified density distributions at a set of
keyframes. The elegant part of this approach is dynamically simulating the derivatives of the velocity
field in the same framework for simulating the velocity field itself. However, since the derivatives of the
velocity field with respect to each control parameter need to be computed throughout a portion of an
animation sequence, this approach is computationally expensive. The adjoint method was adopted in
McNamara et al. [2004] to significantly improve the efficiency of these derivative evaluations for each
iteration. Since it is a gradient-based optimization, a number of iterations are still necessary before
convergence, and there is also the peril of local minima. In comparison, the method introduced in this
article is noniterative, and therefore, less expensive.
   In the aforementioned methods, once the fluid density converges to the target shape, the fluid either
becomes static without dynamic appearances, or starts to evolve freely without any control. However,
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
144      •      L. Shi and Y. Yu

the method in this article can maintain interesting dynamic fluid appearances around the target shapes
for an arbitrarily long time. In addition, since we adopt distance transforms, our method is less likely
to be trapped in local minima.
  Warping and morphing techniques [Wolberg 1990; Gomes et al. 1997] are very useful tools for trans-
forming images and objects. Warping does not have a clear target image or object, while morphing
usually produces transition between two objects. Sims [1992] introduced a technique for successively
warping images using vector fields in. However, the results from the operation are not well controlled.
3D volume morphing methods [Hughes 1992; Lerios et al. 1995; Turk and O’Brien 1999; Alexa et al.
2000] can achieve well-behaved shape interpolation by considering both boundary and interior points.
Actively moving implicit surfaces [Desbrun and Cani-Gascuel 1998] can be used to generate metamor-
phosis between shapes without a correspondence. Most morphing techniques focus on the smoothness of
the planned transition instead of its physical plausibility. In many experiments, we adopt the method in
Turk and O’Brien [1999] to generate an underlying shape transformation guiding smoke-object shape
transitions.

3.    OVERVIEW
In our method, we define an implicit function for the target object and try to drive an irregularly shaped
smoke region so that a specific isosurface of the density distribution of the smoke closely matches the
boundary (zero) level set of the target object. The implicit function for the target object is named
the guiding implicit function and is denoted as D(x, t), where x represents a point in a 2D or 3D
space, and the extra variable t means the target shape may move or deform over time. The density
distribution of the simulated smoke is denoted as ρ(x, t). We use dynamic force feedback to actively
influence the smoke isosurface so that the resulting dynamic surface, represented by ρ = {x|ρ(x, t)−τ =
0}, approximately matches the zero level set of the target object, D = {x|D(x, t) = 0}, where τ is a
threshold used for defining a boundary isosurface for the smoke region which is assumed to have
internal densities higher than τ . Note that ρ(x, t) − τ = 0 is actually an implicit function for the smoke
region. At every time step, given the shape discrepancy between the two isosurfaces, feedback forces
are applied to the smoke boundary to reduce the amount of discrepancy. Since forces and accelerations
are connected through Newton’s law, in practice, such feedback forces are actually realized by velocity
adjustments.
  There are two essential stages in an animation where smoke evolves into an object. The first stage
involves a shape transition between the smoke region and the target object. In the second stage, the
transformed smoke region needs to keep track of the object’s own nonrigid deformation or rigid-body
motion. These two stages can be treated in a unified framework. The initial smoke region obtained by
thresholding can be considered as an irregular object. 3D shape morphing techniques can be applied to
generate a morph sequence between the initial smoke shape and the target object shape (Figure 1(b)).
As a result, we can obtain an intermediate shape at any instance during the whole transition period.
Thus, the shape transition between the smoke and the target object can be viewed as a nonrigid shape
deformation, and any method designed for the second stage can be applied as well to this stage, where
the morph sequence is used to generate the guiding implicit function which the smoke tries to match.
Therefore, we only need to develop an approach for the smoke to track object motion. A static object is
a special case for this problem.
  Nevertheless, tracking object motion using the smoke is by no means trivial. When there is tem-
poral coherence and the frame-to-frame motion of the target object is small, it is possible to obtain
velocity constraints on the boundary isosurface of the smoke by exploiting the local gradient flow
of the boundary. If a physical simulation of the smoke satisfies these boundary velocity constraints,
the difference between the two boundary isosurfaces will be decreased. However, when there is fast
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Controllable Smoke Animation With Guiding Objects              •     145

frame-to-frame motion, the target object from two consecutive frames might have a huge gap in posi-
tion or orientation. Local gradient structure of the implicit functions cannot guarantee efficient tracking
results any more. When this happens, we use the target object to transport the smoke inside from its
location in the previous frame, to its location in the current frame as if the target object is a con-
tainer with a hard boundary. The hard boundary disappears once the smoke is in position again.
Note that this step may not be physically realistic, and is specially designed to achieve the desired
phenomenon.
   The overall solution to the proposed problem has the following components:

—The animator needs to pick a target object and choose a smoke region from a simulation. A 3D morph
 sequence is created between the initial smoke shape and the target object. The target object for each
 frame is voxelized into a discrete implicit function.

—For each frame, the system determines whether large motion has occurred by checking the amount of
 overlap between the two underlying object shapes at the previous and current frames. The strategy
 for smoke transportation is executed if the amount of overlap falls below a predefined threshold.
 Small motion tracking is always performed, whether large motion occurred or not.

—During each iteration, the system also simulates potential interactions among multiple smoke objects
 and interactions between a smoke object and its environment. The shape of a smoke region may
 be destroyed during such interactions, and a new morph sequence can be generated between the
 remaining smoke and the target object.


4.   GUIDING OBJECTS
The input to our system includes the target objects and their motion. When the objects are not static, the
configuration of the objects needs to be specified at every frame. The type of motion may include simple
rigid-body motion, more complicated articulated body motion, or even nonrigid deformations. All target
objects are internally represented as implicit functions. Our representation for the guiding implicit
function D(x, t) is the signed distance function, which is zero on the object boundary and positive at
the interior. The signed distance function has the advantage of conveniently providing the shortest
distance between any point on the smoke boundary and the boundary of the guiding implicit function.
Therefore, input objects need to be converted to this representation even if they are already given as
implicit functions.
   Since smoke simulation will be performed on a volume grid, the smoke implicit function ρ(x, t) − τ
is always directly represented on this discrete grid in the form of a density value at each voxel. The
boundary isosurface of this function is obtained by labeling the voxels containing a density value close to
the threshold τ . The signed distance function of a guiding object at a specific frame is also represented on
the same volume grid by first discretizing the object’s original representation, followed by a conversion
to signed distance values.
   As discussed in Section 3, a part of the animation may involve a shape transition between the smoke
region and the target object. An intermediate shape should be generated at each frame during this
transition. This intermediate shape serves as the guiding object for that frame, and it needs to be
represented as an implicit function as well. In practice, we apply 3D shape morphing and masking
techniques (see Section 7.1) to generate a shape transition sequence which can produce such an in-
termediate shape at any intermediate frame. These intermediate shapes should be represented as or
converted to signed distance functions whenever they are needed.
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
146      •      L. Shi and Y. Yu

5.    SHAPE MATCHING
5.1    Velocity Constraints for Small Motion
Suppose at a certain time ti during simulation, the guiding implicit function is D(x, ti ). We would like
to evolve the smoke density at the previous step, ρ(x, ti−1 ), so that the zero isosurface of the updated
function, ρ(x, ti ) − τ , approximately matches the boundary of the guiding object.
  Since we are concerned with matching two isosurfaces, let us first look at a criterion for measuring
shape discrepancy. If we represent an object as a point set, two objects A and B exactly match each
other if and only if both sets A − B and B − A are empty, which is also equivalent to that the volumes of
both A − B and B − A are zero. The summed volumes of these two sets indicate the level of discrepancy
between two shapes. Mathematically, we need to use characteristic functions of the shapes and integrals
to represent these two volumes. Let us define χ D (x, ti ) = 1 if D(x, ti ) ≥ 0, and χ D (x, ti ) = 0, otherwise;
and define χρ (x, ti−1 ) = 1 if ρ(x, ti−1 ) − τ ≥ 0, and χρ (x, ti−1 ) = 0, otherwise. The level of discrepancy
between the two zero isosurfaces can be measured by the following integral,

                         ev =      χρ (x, ti−1 )(1 − χ D (x, ti ))d x +       χ D (x, ti )(1 − χρ (x, ti−1 ))d x,   (1)

where the global minimum is zero and can be reached when the two characteristic functions coincide.
Equation (1) can be simplified to χ D d x + χρ (1 − 2χ D )d x, where χρ is the shape variable, and χ D
is the fixed guiding shape for a specific frame since we would like ρ(x, ti−1 ) to approximate D(x, ti ).
Therefore, reducing the shape discrepancy between the two is equivalent to minimizing the following
functional,

                                                               χρ (1 − 2χ D )d x.                                   (2)

  According to calculus of variations [Gelfand and Fomin 1963], one can show that the first variation
of the integral in Equation (2), with respect to the smoke boundary surface, is simply based on the
second part of its integrand and the normal directions of the smoke boundary. Therefore, the negative
variational gradient minimizing the functional in Equation (2) with respect to the smoke boundary is
as follows.
                                               δev                            ∇ρ
                                                              = (1 − 2χ D )                 ,                       (3)
                                               δ ρ   x=xb                     ∇ρ     x=xb

                                                   ∇ρ
where ρ represents the smoke boundary, and ∇ρ represents the unit inward normal at the smoke
boundary. Detailed discussion on the derivation of this variational gradient is presented in Appendix A.
   Consider a specific point xb on the boundary of the smoke, the sign of D(xb , ti ) indicates its location
with respect to the guiding object. Note that the gradient of ρ(x, ti−1 ) at xb points to the interior of
the smoke and is perpendicular to the boundary. Under the small motion assumption, the smoke and
guiding object should have overlap. If xb is inside the guiding object, we need to move it slightly along
the negative gradient direction; otherwise, move it along the positive gradient direction. Equation (3)
means iteratively perturbing all the smoke boundary points simultaneously in this way would gradually
decrease the costs in Equation (2) and Equation (1). Since a new smoke boundary is formed by the
relocated points after each iteration, the gradient of the new boundary should be used for moving points
in the subsequent iteration. This scheme for functional minimization resembles gradient descent for
regular function minimization. As gradient descent, this scheme cannot always converge to the global
minimum if the initial shape of the unknown is not sufficiently close to the target shape. For example,
if the smoke region does not have overlap with the guiding object, this scheme would gradually shrink
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                     Controllable Smoke Animation With Guiding Objects                 •     147

the smoke region until it disappears and, at the same time, decrease the cost in Equation (1) to the
volume of the guiding object which is actually a correct local minimum.
    In the current context, we can modify the above iterative minimization scheme to avoid the local
minimum for two separate shapes. When D(xb , ti ) is negative, indicating xb is outside the guiding
object, it should be moved in a direction in which the directional derivative of D(x, ti ) is positive to
bring it closer to the boundary of the guiding object. Since this condition must be satisfied either by
the positive or by the negative gradient direction of ρ(x, ti−1 ), we only need to choose the right one of
these two instead of always following the positive gradient. We still keep the original scheme when xb
is inside the guiding object.
    Since we would like to realistically evolve the boundary of ρ(x, ti−1 ) − τ into the target shape, the
dynamic evolution should follow the above iterative procedure to minimize the integral in Equation (1),
as well as follow the physics rules in smoke simulation as closely as possible. To achieve these goals, the
smoke simulation should satisfy velocity constraints derived from the minimization procedure. Since
Equation (3) shows a first-order scheme, we should adopt its direction field, but adjust its magnitude
to improve stability.
    A velocity u on the smoke boundary can be decomposed into a normal component un and a tangential
component ut . Based on Equation (3), the normal component un at a smoke boundary point xb , at time
ti , is defined to be
                                                                           ∇ρ
                       un (xb , ti ) = Cn · min(d max , |D(xb , ti )|) ·      · msgn(xb , ti ),                               (4)
                                                                           ∇ρ
where Cn is a constant scaling parameter, |D(xb , ti )| is the magnitude of the signed distance function,
indicating how far away xb is from the current boundary of the guiding object, d max is a clamping
upper bound for the distance, and msgn(xb , ti ) adjusts the direction of the velocity vector according to
our modified scheme. Specifically, msgn(xb , ti ) = −1, if D(xb , ti ) > 0; msgn(xb , ti ) = sgn(∇ D · ∇ρ), if
D(xb , ti ) ≤ 0. The incorporation of the term min(d max , |D(xb , ti )|) can alleviate overshooting when xb is
already close to the boundary of the guiding object. However, this term should only be present in the
normal component when the smoke region already has some overlap with the guiding object. Otherwise,
it would reduce the velocity of the smoke to zero when it touches the boundary of the guiding object, and
keep the smoke from entering the object. Various situations for setting up normal velocity constraints
are summarized in Figure 2. Note that when the volume of the smoke differs from the object, these
velocity constraints do not preserve mass. This is a tradeoff we need to make between physics and shape
matching.
   Constraints on the tangential component are also crucial. Although the tangential component does
not directly affect the shape of the level sets, it does affect the surrounding velocity field. While a zero
tangential component would make the smoke surface less alive, the numerical stability of the shape
matching procedure may be compromised if it becomes overly large. The maximum allowable tangential
component is actually dependent on the local geometry of the smoke surface. If the surface is flat,
the tangential displacement can be large without destroying the original shape while a highly curved
surface is certainly more vulnerable. Therefore, the magnitude of the tangential component should have
an upper bound related to the surface curvature. Since we would like to follow physical simulation as
faithfully as possible, we can simply clamp the physically-generated tangential components against the
upper bound when their magnitude becomes too large. Thus, the constrained tangential component is
simply defined to be
                                                                     Ct     u∗
                                      ut (xb , ti ) = min     u∗ ,           t
                                                                               ,                                              (5)
                                                               t
                                                                     K      u∗
                                                                             t

                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
148      •      L. Shi and Y. Yu




Fig. 2. Normal velocity constraints on the smoke boundary for various situations. They are based on our revised minimization
scheme for Equation (2). (a) The smoke region is completely outside the guiding object; (c) the smoke region partially overlaps
with the guiding object; (b) & (d) a portion of the smoke boundary touches the boundary of the guiding object. The smoke velocity
should not be affected in (b), while it should be reduced to zero in (d); (e) the smoke region is completely inside the guiding object;
(f) the smoke region encloses the guiding object.


where u∗ is the tangential component generated from a simulation, Ct is a constant parameter, and K is
        t
the surface curvature. In practice, we use mean curvature. A robust implementation of mean curvature
on a volume grid can be found in Museth et al. [2002].
  Note that the normal and tangential velocity constraints are derived velocities for matching two
shapes. They are different from those constraints defined in Fedkiw et al. [2001] and Foster and Fedkiw
[2001] for fluid-object interaction and fluid control. Our guiding objects are invisible “ghost” objects,
and they do not directly interact with the smoke.

5.2    Velocity Constraints for Large Motion
When the frame-to-frame motion of the guiding object becomes excessively large, the two instances of
the guiding object at two consecutive frames may have little overlap, or no overlap at all. Although the
matching scheme developed in the previous section can eventually converge, many iterations would be
needed to actually reach convergence. For efficiency considerations, we directly transport the smoke
from the location of the first object instance to the second. During this direct transportation, the guiding
object is assumed to have hard boundaries, and the part of the smoke that is already inside becomes
trapped and moves together with the guiding object. Meanwhile, velocity constraints should still be
imposed at the boundary and interior of the guiding object so that these constraints can bring along
the surrounding region and generate turbulent flows to create a fluid appearance and “evidences” for the
large motion. These constraints are not for transporting the smoke inside the guiding object. Suppose
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                 Controllable Smoke Animation With Guiding Objects                   •     149

a point on the guiding object moved from xi−1 to xi during two consecutive frames at time ti−1 and ti , a
velocity constraint
                                                        xi − xi−1
                                                uL =
                                                        ti − ti−1
should be imposed at xi−1 at time ti−1 . There is such a constraint for every voxel on and inside the
guiding object at ti−1 . Note that the smoke transportation scheme in this section only accounts for
large rigid-body motion and leaves nonrigid deformation to the velocity constraints in the previous
section.
  The velocity constraints we have come up with need to be applied either at the boundary or interior
of the smoke when the shape difference between the guiding object and the smoke region becomes
sufficiently large. To make this statement more concrete, we are going to discuss in the following
sections how to measure the shape difference and how to detect and evolve the smoke boundary.

5.3   Error Metrics
We use two different error metrics for measuring the shape difference. The velocity constraints are
only applied when the specific metric being used exceeds some threshold. The first one is the volume
discrepancy ev , defined in Equation (1). The second metric is a generalization of the L p norm to object
boundaries,
                                                                      1/ p

                                       eL p =            p
                                                       D (x, t)d x           ,                                              (6)
                                                   ρ


where p is positive, ρ is the boundary of the smoke, and D(x, t) is the distance function for the guiding
object at time t. The latter two quantities have been defined in Section 3. A special case of the L p norm
is the L∞ norm, which is equivalent to
                                            e L∞ = max D(x, t).                                                             (7)
                                                       x∈    ρ


    Although the velocity constraints were derived using the first metric, both metrics have the same
global minimum and can effectively reduce the shape difference. However, the dynamic behavior of
the smoke under their respective control can be quite different, especially when p becomes large
in the second metric. For instance, ev allows small portions of the smoke boundary to be far away
from the guiding object as long as the integral remains small, while e L∞ keeps all the boundary points
of the smoke within a certain distance from the guiding object. As a result, ev or e L p , with low p values,
give rise to more realistic smoke appearances on the boundary and less clear object structures, while
e L p , with high p values, becomes more appropriate when clear object structures are desirable.

5.4   Smoke Evolution and Boundary Detection
The smoke implicit function φ = ρ − τ is evolved passively over time by the wind velocity field {u},
which may be partially constrained. It can be easily shown [Osher and Sethian 1988] that the equation
to update φ is as follows.
                                                φt + u · ∇φ = 0.                                                            (8)
Except for the density threshold τ for boundary detection, this equation coincides with the advection
equation for the smoke density in Fedkiw et al. [2001]. This equation can be solved either by the semi-
Lagrangian method, or by the upwind scheme [Sethian 1999]. Note that both operate on the whole
voxel grid instead of boundary voxels only. We have implemented both methods. Both of them can
                                                                     ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
150      •      L. Shi and Y. Yu

produce visually realistic results. Since a smoke boundary tends to evolve relatively slowly, the choice
of a numerical method is not very critical.
   Our normal and tangential velocity constraints are positioned at the boundary of the smoke region.
Enforcing these constraints requires the detection of the smoke boundary at every time step. This can
be easily achieved since this smoke boundary is just the zero isosurface of the smoke implicit function.

6.    SMOKE SIMULATION FOR CONSTRAINED VELOCITIES
Since we exploit velocity constraints to achieve shape matching, a smoke simulation framework that
can effectively incorporate hard-wired velocity constraints without producing visual discontinuities is
desirable. Before we introduce our revised model for smoke simulation, let us first have a look at the
already adopted mathematical formulations for fluid simulation in the graphics literature [Foster and
Metaxas 1997b; Stam 1999; Yngve et al. 2000; Fedkiw et al. 2001].
6.1    Previous Formulations
The dynamics of a compressible fluid can be modeled using the following Navier-Stokes equations:
                             ∂ρ f
                                   = −∇ · (ρu),                                                           (9)
                              ∂t
                              ∂u                             ν
                           ρf      = −ρ f (u · ∇)u + ν∇ 2 u + H(u) + ρ f f − ∇ P,                        (10)
                              ∂t                             3
where ρ f is the density of the fluid which is different from the smoke density, u represents the velocity
field, P is the pressure term, f is an external force field, and ν is the kinematic viscosity of the fluid. The
x component of H(u) is ∇ · ( ∂u ), and the y and z components are defined similarly. The first equation
                              ∂x
arises from the conservation of mass, but the volume of the fluid is compressible with increasing density
for a decreasing volume. The second equation is for the conservation of momentum; the first term on the
right hand side is the convective term; the second and third terms model accelerations due to viscous
forces; the last two terms model accelerations due to external forces and forces arising from the pressure
gradient.
  These equations can effectively model high-speed velocity fields such as shock waves generated by
explosions [Yngve et al. 2000]. However, a strict time step condition is necessary for stable numerical
solutions. For liquids and low-speed gaseous phenomena, the compressibility effects are negligible, and
conservation of mass becomes equivalent to conservation of volume. The assumption of incompressibility
leads to the following equations which make more efficient numerical methods possible.
                                          ∇ · u = 0,                                                    (11)
                                           ∂u                  1
                                               , = −(u · ∇)u −    ∇ P + ν∇ 2 u + f,                     (12)
                                           ∂t                  ρf
where the first equation means the fluid is volume-preserving, and the second one is similar to the
corresponding equation in the aforementioned model for compressible fluids. In Stam [1999], Stam de-
veloped an unconditionally stable numerical method for these equations by adopting semi-Lagrangian
tracing and the Helmholtz-Hodge Decomposition.
  If we focus on gases, the effects of viscosity are also negligible. Therefore, the diffusion terms in the
above models can be left out. Simulating gaseous phenomena thus reduces to solving the following
incompressible and inviscid Euler equations [Fedkiw et al. 2001]:
                                                  ∇ · u = 0,                                            (13)
                                                   ∂u
                                                        = −(u · ∇)u − ∇ P + f,                          (14)
                                                    ∂t
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Controllable Smoke Animation With Guiding Objects                 •     151


where the factor ρ1f in Equation (12) has been integrated into the “pressure” P which is used to guarantee
zero divergence, but does not represent actual pressure anymore.

6.2   An Empirical Equation for Compressible Gases
The basic framework in this article is a dynamic feedback system where the velocity constraints defined
in Section 5 are dynamically updated every time step, according to the shape discrepancy between the
smoke region and the underlying guiding object. The purpose of such velocity constraints is to reduce
the amount of discrepancy. These velocity constraints have poor spatial and temporal coherence because
constraints at spatially or temporally adjacent voxels may be quite different. In addition, the constraints
are only imposed at the boundary of the smoke region which is essetially a thin layer in the simulation
volume. Since these artifical velocity constraints do not exist in the real world, we argue that existing
fluid simulation methods cannot incorporate them without introducing artifacts.
   First, the incompressible scheme in Equations (13) and (14) does not work well. If we impose veloc-
ity constraints which generate discontinuities in the velocity field, concentrated high pressure tends to
appear rapidly in regions nearby to guarantee zero divergence. Those sudden high pressures further in-
fluence surrounding velocities and generate temporal discontinuities and visual artifacts. Such artifacts
have been observed in our experiments. Second, existing compressible schemes would not fit our pur-
pose very well, either. Simulating true compressibility is very expensive and requires a strict time step
condition. On the other hand, an unphysical scheme called “artificial compressibility” was introduced
in Chorin [1967]. It allows a certain degree of compressibility during a transition period and converges
to incompressible fluid simulation when a steady solution has been reached. However, this convergence
is achieved over time, instead of within each time step, and is only guaranteed when external forces are
absent. In our simulation, incoherent external forces are exerted constantly at every time step since
we need to update the velocity constraints. Thus, the scheme in Chorin [1967] may not converge. In
addition, the sound speed used in this scheme needs to be unusually small to prevent temporal discon-
tinuities. An unreasonable sound speed further compromises the realism in the simulation results.
   The nature of our control scheme demands a new fluid simulation technique which does not have
to be physically accurate, but needs to produce results that are visually appealing. What we need
is a simulation technique that approximates the behavior of incompressible fluids while absorbing
discontinuities where velocity constraints are present. We would also like the technique to be stable
and efficient. To achieve our goals, we propose an empirical scheme that does not strictly enforce
incompressibility. Recall that the pressure P in Equation (14) can be estimated numerically using the
Poisson equation [Stam 1999; Fedkiw et al. 2001]

                                                     1
                                            ∇2 P =      ∇ · u,                                                        (15)
                                                      t
where t is the size of the simulation time step. It has been shown in Stam [1999] that imposing
a feedback force field −∇ P to the velocity field strictly enforces zero divergence. Our technique first
decomposes the pressure field into two components, −µ t∇ 2 P and P = P + µ t∇ 2 P , where µ is a
constant factor with units of area divided by time, and the negative sign in front of the first component
is due to the fact that a local maximum of P is typically a local minimum of ∇ 2 P . Note that the first
component is the negative Laplacian of the pressure field, and the second one is a blurred version of
the original pressure. This decomposition of the pressure field, in turn, splits the feedback force field
into two components, µ∇( t∇ 2 P ) and −∇ P − µ∇( t∇ 2 P ). To allow a certain degree of compressibility,
especially in the regions with velocity constraints, our scheme only applies the first force component to
the velocity field immediately, while buffering the second one for later time steps. Thus, the discretized
                                                                 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
152      •      L. Shi and Y. Yu

version of Equation (14) changes to
                                      u(t +     t) − u(t)
                                                          = −(u · ∇)u + µ∇( t∇ 2 P ) + f.             (16)
                                                 t
Because of Equation (15), t∇ 2 P in Equation (16) can be replaced with ∇ · u. If we further replace
the left-hand side of Equation (16) with a continuous partial derivative of the velocity field, we arrive
at the following single partial differential equation that is capable of simulating compressible gaseous
phenomena:
                                    ∂u
                                        = −(u · ∇)u + µ∇(∇ · u) + f,                                (17)
                                     ∂t
where 0 ≤ µ < ∞, with a typical value between 0 and 1. It is used to adjust the magnitude of the
feedback force from the pressure field to the velocity field.
  Since our scheme only uses a filtered version of the original pressure every time step, the feedback
force from the pressure to the velocity field is weakened. Nevertheless, this does not mean a portion
of the pressure simply disappears, but mean that the release of the energy preserved in the rest of
the pressure is delayed. In Appendix B, we show that, for a bounded workspace, without sources and
sinks, ∇(∇ · u) = 0 everywhere is equivalent to ∇ · u = 0 everywhere. This result indicates that our
new formulation of the feedback force from the pressure to the velocity field can eventually reduce
the divergence of the velocity to zero when there are no external forces. In addition, our formulation
allows large time steps and does not involve a sound speed. While other decompositions are possible, the
Laplacian operator in our pressure decomposition makes it particularly convenient to robustly obtain
the components through a diffusion process, which will be discussed in Section 7.2.
  In practice, we have found that our formulation can effectively reproduce realistic fluid motion as
well as incorporate frequently inserted velocity constraints, without generating obvious visual discon-
tinuities (see Section 8.1).

7.    IMPLEMENTATION
7.1    Shape Transition
   7.1.1 Variational Shape Morphing and Interpolation. As our first option, we apply the shape trans-
formation method introduced in Turk and O’Brien [1999] to generate a morph sequence between the
shape of a smoke region and a target object. This method represents the whole morph sequence as a
variational implicit function defined in a space of n + 1 dimensions. This implicit function, which is
based on radial basis functions, interpolates the source and target shapes defined in an n-dimensional
space and handles topological shape changes automatically. The extra dimension is aligned with the
temporal axis so that the source and target shapes are n-dimensional slices of the implicit function at
time zero and one, respectively. In our situation, we apply this method to obtain an interpolating 4D
function (r, s) such that (r, 0) and (r, 1) reproduce the starting smoke shape and the target object,
respectively. By fixing s to a value between 0 and 1, (r, s) represents the analytic from of an inter-
mediate shape. In practice, we found that the rate of shape transformation usually was not uniform
and the target shape started to loom only when the time became very close to one, if the target shape
was complicated and the parameter s was scheduled linearly with respect to the actual time scale.
Therefore, we decided to warp the temporal axis and schedule s as a piecewise linear function of the
actual time t used for an animation. The local slope of the piecewise linear function is used to adjust the
transformation rate and make it perceptually more uniform. Better morph sequences can be obtained
when the source and target objects are well aligned in terms of position and orientation. The amount
of relative translation and rotation should then be uniformly distributed back across the frames of the
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                               Controllable Smoke Animation With Guiding Objects               •     153

resulting sequence so that there is simultaneous deformation, translation, and rotation from frame-to-
frame. It is desirable for the intermediate shapes to have an approximately constant volume since a
gas usually does not significantly change volume even when it is compressible. Therefore, we verify the
volume at each intermediate frame, and apply morphological operations on the voxel grid to shrink or
expand the shape.
  The same variational implicit functions for shape transformation can be used for surface interpolation
[Carr et al. 2001; Turk and O’Brien 2002]. Therefore, they can be used for generating an analytic
implicit function to approximate a polygonal mesh model by interpolating the vertices of the mesh. The
interpolating implicit function and the polygonal mesh have the same dimensionality. Guiding objects
originally represented as polygonal meshes are converted into implicit functions in this way.

   7.1.2 Shape Masking. Shape masking is an easier-to-implement alternative for generating shape
transitions. In this scheme, the target object can be simply a fixed shape undergoing rigid-body motion.
We also require an additional masking shape, M (x, t), whose scale, position, and orientation can all
vary with time. The guiding shape at each time step is defined to be the intersection between the target
and masking objects. For example, the masking object can be a moving sphere with changing radius. At
the beginning of a transition, the sphere has a tiny radius and does not have any overlap with the target
object. The sphere then moves closer to the target object while increasing its radius so that the amount
of overlap becomes larger. At the end, the sphere encompasses the target object and the transition is
complete. One caveat with this scheme is that the volume of the guiding shape changes with time,
which requires an increasing amount of smoke during the transition.

7.2   Numerical Smoke Simulation
We have implemented numerical solutions for two formulations, our new compressible fluid formula-
tion and the incompressible and inviscid formulation in Fedkiw et al. [2001]. We also implemented
enhancements that allow the integration of our velocity constraints. The details of these enhancements
will be introduced in the next section.
   The implementation of the incompressible scheme consists of three basic steps [Fedkiw et al. 2001]:
compute an intermediate fluid velocity field, {u∗ }, from Equation (14), ignoring the pressure term by
first adding external force times the time step, and then solving the advection part, (u · ∇)u, by using
semi-Lagrangian tracing [Staniforth and Cote 1991]; obtain the pressure P by solving the Poisson
equation,

                                                     1
                                            ∇2 P =      ∇ · u∗ ;                                                    (18)
                                                      t
where    t is the size of the time step; finally, subtract the gradient of P from the intermediate velocity,

                                             u = u∗ −    t∇ P.                                                      (19)

  To simulate our compressible fluid formulation, we still keep the first step unchanged, but revise the
other two. Although our formulation does not involve pressure, directly solving Equation (17) may not
be stable. Introducing pressure during the numerical process improves stability. Therefore, we actually
solve Equations (15) and (16) instead. In the above steps, after obtaining the pressure by solving the
Poisson equation, we apply a diffusion process,

                                              ∂P
                                                 = µ∇ 2 P,                                                          (20)
                                              ∂t
                                                               ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
154      •      L. Shi and Y. Yu

and solve for the new pressure, P , using a stable implicit method,
                                                          P −P
                                                               = µ∇ 2 P ,                               (21)
                                                            t
where the Laplacian of the new pressure ∇ 2 P , instead of ∇ 2 P , appears on the right-hand side. The new
pressure can be obtained by solving the sparse linear system arising from discretizing this equation.
Finally, the intermediate velocity is updated as follows,
                                                      u = u∗ + µ t∇(∇ 2 P ).                            (22)
Note that solving the diffusion equation to obtain the new pressure is an extra step we performed
during smoke simulation.
  We discretize the workspace into a finite volume grid. Keep all vector components on the faces of the
voxels, and retain all the scalar fields at the center of each voxel. The velocity at any point inside a voxel
can be obtained by linearly interpolating each component of the velocity vector separately. Additional
implementation details about discretization and estimation of finite differences can be found in Fedkiw
et al. [2001] and Stam [1999].
  Our overall numerical solution involves multiple (around three) time steps between two consecutive
frames. Small and large object motion tracking are carried out in separate time steps. There is at
most one optional time step dealing with large motion between two frames. And there may be one or
more time steps for small motion tracking since multiple iterations may be necessary to achieve good
boundary matching. In the following, we introduce the details of these time steps in the context of the
numerical solution for the incompressible formulation in Fedkiw et al. [2001]. It is straightforward to
modify these details for our compressible fluid formulation.
  A time step for small motion tracking has the following substeps:
—Compute the intermediate fluid velocity field {u∗ }, detect the boundary of the smoke region using a
 density threshold, and set up velocity constraints at the boundary by modifying {u∗ }.
—Solve the final fluid velocity field using Equations (18) and (19), while satisfying the constraints.
—Use the obtained velocity field to evolve the smoke density distribution applying the semi-Lagrangian
 method.
The optional time step for large motion between two frames is executed before the other time steps. It
has the following substeps:
—Obtain the intermediate fluid velocity field and set up velocity constraints for large motion.
—Solve the final fluid velocity field while satisfying the constraints.
—Use the obtained velocity field to evolve the smoke outside the guiding object. Use the guiding object
 to transport the smoke inside, and the transported smoke density overwrites existing density at each
 destination voxel.
If we choose to apply our compressible fluid formulation, the second substeps in both types of time steps
should solve Equations (18), (21), and (22) instead. It is straightforward to apply additional vorticity
confinement [Fedkiw et al. 2001] in the second substeps as well since these substeps are basically
the major part of a single forward simulation step. In our experiments, we always apply vorticity
confinement.
7.3    Enforcing Velocity Constraints
Most of the time, we only set up and enforce velocity constraints at a selected subset of the voxels on
the smoke boundary for small motion tracking. This process is initiated once the adopted error metric
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                 Controllable Smoke Animation With Guiding Objects              •     155

(see Section 5.3) exceeds a prescribed threshold. A larger threshold produces more lively smoke motion,
while a smaller threshold shows the object boundary more clearly. The voxels on the smoke boundary
are sorted according to their distance to the guiding object. Voxels with larger distance are assigned
higher priority. Velocity constraints are assigned to the boundary voxels with highest priority first.
If the shape discrepancy does not drop significantly, additional voxels with the next level of priority
become constrained. This process continues until the error drops below the threshold, or most of the
boundary voxels are constrained. Once the error becomes sufficiently small, constrained voxels get
released gradually in the reversed order.
   The desired velocity constraints are imposed after the computation of the intermediate velocity field
{u∗ }. Solving the Poisson equation may alter the velocities in these constraints and make it difficult
for the smoke to converge to the target shape if these constraints are not explicitly enforced. Enforcing
these constraints means the constrained velocities should not be affected during this step. Equation (19)
indicates that ∇ P should be zero at the constrained voxels. Therefore, the pressure p needs to satisfy
such gradient constraints in addition to the Poisson equation.
   Using finite differences to discretize both left- and right-hand sides of Equation (18), we obtain a
sparse linear system of equations, AP = b, where P is the vector of unknown pressures, and A is the
coefficient matrix with a sparse structure. A discrete gradient estimation at a voxel involves three or
six of its direct neighbors, depending on whether central differences are being used. For symmetry
consideration, when ∇ P = 0 at voxel (i, j, k), we impose that P (i, j, k) should equal the pressure at
its six direct neighbors. For every equation involving P (i, j, k) in the linear system, we need to replace
P (i, j, k) with the pressure at one of its direct neighbors. Eventually, we can eliminate P (i, j, k) from all
the equations. Changes like this result in a linear system with a reduced number of unknown variables.
Preconditioned conjugate gradient (PCG) with Incomplete Cholesky Factorization (ICF) [Golub and
Loan 1996] is a natural choice to solve this sparse linear system.
   Since the smoke boundary surface is typically a closed surface, we avoid strictly enforcing boundary
conditions everywhere on the surface because that would result in a singular matrix A if the volume
inside the smoke region is not preserved. When it is necessary to cover the whole smoke boundary
with constraints, we reduce the density of the constraints by setting a small distance threshold d t .
Any voxels within that distance from a constrained voxel are not allowed to be constrained. We typically
choose d t to be between 1 and 3 voxels. Alternatively, Dirichlet boundary conditions can be adopted by
directly specifying pressure values on the smoke boundary. For example, one can force voxel (i, j, k)
and all its direct neighbors to have zero pressure.

7.4   Smoke Transportation for Large Motion
We assume that the frame-to-frame motion of the guiding object can be decomposed into a rigid-body
transformation and a nonrigid deformation. Smoke transportation only handles large rigid transforma-
tion and leaves small rigid transformation, as well as additional nonrigid deformation, to small motion
tracking. To move the smoke to a new position and orientation specified by the rigid transformation, we
apply backward mapping with a modified trilinear interpolation scheme. A voxel inside the transformed
object takes the smoke density at a corresponding position in the original object. The density at the
corresponding position should be interpolated only from the densities at those surrounding voxels that
are also inside the original object to preserve the object boundary.

7.5   Thin Parts
It is hard for the smoke surface to reach some of the thin parts of the guiding object. That is because
the gradient and curvature estimations are inaccurate at these locations due to insufficient sampling.
We explicitly label those voxels that should have smoke but actually do not after a long time. If there is
                                                                ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
156      •      L. Shi and Y. Yu




Fig. 3. A comparison between two smoke formulations. The first row is generated by a model-enforcing incompressibility. The
second row is generated by our model with compressibility. They are visually similar, which indicates our model can also produce
realistic smoke appearances. Grid size 200 × 200 × 200.

a smoke-filled voxel which is adjacent to one of those tagged voxels, we set a velocity constraint at the
first voxel and the velocity points to the tagged voxel. Thus, smoke can be propagated gradually into
the thin parts.

7.6    Time Complexity
Since we enforce shape matching by embedding velocity constraints into a conventional smoke simu-
lation, the time complexity of our approach is slightly higher than the latter. The extra work at each
time step includes obtaining the signed distance function of the guiding object and solving the diffusion
equation for the pressure, among others. The complexity of the distance transform is O(n log m), where
n is the number of voxels in the grid and m is the maximum number of voxels on the propagating front
[Sethian 1999]. In the worst case, m = O(n), which usually does not occur unless the object boundary
becomes close to a fractal surface. In all our experiments, the actual running time of the distance trans-
form is less than an original smoke simulation step without constraints. The complexity of solving the
sparse linear system arising from the diffusion equation is on the same order as that of solving the
Poisson equation which is part of the original simulation. Solving these two equations is the most time-
consuming step that occupies approximately 60% of the total running time. The complexity of all the
other extra work is linearly proportional to the size of the voxel grid. Therefore, in practice, our complete
algorithm maintains the same order of magnitude of the original complexity of smoke simulation.

8.    EXPERIMENTAL RESULTS
We have successfully tested our complete algorithm in a variety of examples. Some of the images from
these examples are shown in Figure 1 and Figures 4–8. The target objects in these examples include
both free-form objects and letters. Most of the target objects are converted to implicit functions from
triangular meshes. If there are multiple objects in the same example, smoke simulation was performed
on all of them simultaneously. Most of the simulations have been finished on an AMD 2100+ processor.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                        Controllable Smoke Animation With Guiding Objects                 •     157




                Fig. 4. Two heart-shaped smoke objects collide and merge into one. Grid size 128 × 128 × 128.




Fig. 5. Large motion tracking for a synthetic character. Constraints for large motion induce a velocity field for the environment.
There are a few sources on the floor to produce smoke for the environment. Grid size 128 × 128 × 128.



A voxel grid of size between 64 × 64 × 64 and 200 × 200 × 200 has been adopted for the simulations.
The examples took 15 seconds/frame on the lower end of the grid resolution and 60 seconds/frame on
the higher end. The final images were rendered by ray-tracing the smoke volume density distribution
[Kajiya and von Herzen 1984].
   The density threshold τ for smoke boundary detection is interactively determined at the beginning
of the smoke-object shape transition and fixed throughout a whole simulation. We have found that
typically this threshold falls between 0.1 and 0.5. In the following, we specify other parameters used
during experiments for a normalized grid with one unit length for each of its dimensions. In Equation (4),
we typically set d max between 0.05 and 0.1, and set Cn around 25. In Equation (5), we set Ct around 2.
These parameters were chosen with the assumption that the maximum velocity of the smoke is around
1 unit-length per-second. Obviously, by adjusting the maximum velocity, we can change the overall pace
of the smoke.
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
158      •      L. Shi and Y. Yu




Fig. 6. (a)-(c) A smoke cow generated by our method; (d)-(f) a smoke cow generated by the hard boundary scheme. Grid size
128 × 128 × 128.




Fig. 7. The smoke switches back and forth between the shapes of a “check” and an “X”. Only one cycle is shown here. Grid size
128 × 128 × 128.

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                       Controllable Smoke Animation With Guiding Objects                •     159




Fig. 8. (a)-(d) Smoke rises to form the shape of four letters. The shapes start to disappear once the velocity constraints are
released in (d). Grid size 128 × 128 × 128.

8.1   Validation of the Compressible Fluid Model
We have validated our empirical fluid model by visually comparing the results from our model with
results from the model in Fedkiw et al. [2001]. We have performed two different types of comparisons.
The first set of comparisons are between smoke simulations without velocity constraints. They are
meant to verify whether our model can be used for a general smoke simulation as opposed to the special
application in this article. As a result, the visual quality of the smoke sequences generated by our model
is comparable to those in Fedkiw et al. [2001]. It can produce realistic smoke appearances with rolling
effects. Figure 3 shows one comparison in this category.
   The second type of comparisons are between simulations with our derived velocity constraints.
We have confirmed from these comparisons that our new model can integrate these velocity con-
straints better with fewer visual artifacts. Introducing our velocity constraints into a model enforc-
ing incompressibility would generate obvious artifacts since the nonzero divergence caused by the
constraints influence surrounding velocities immediately. The reason our model can perform better is
that it can absorb velocity discontinuities into pressure and release the energy in the pressure gradually.

8.2   Interaction with the Environment
The smoke objects should interact with the environment like smoke. When there is a sufficiently strong
wind, the animator can choose to release the velocity constraints in the influence region of the wind
and let the smoke move freely. When there is an intervening real object, the animator can release the
original velocity constraints in the contact region and let the smoke be controlled by the boundary
conditions on the real object surface as in Fedkiw et al. [2001]. In the example shown in Figure 1(a),
a strong wind blows away the head of a smoke horse. However, the head grows back when the wind
recedes. This is because we keep the underlying target object unchanged, and the target object induces
                                                                        ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
160      •      L. Shi and Y. Yu

velocity constraints pointing towards the head region. In such examples, the animator needs to specify
the spatial region where constraints should be released as well as the starting time and duration. In
Figure 1(a), the user simply places a partitioning plane at the bottom of the horse’s neck. Constraints
on the same side of the plane as the head are released for a short period of time. To preserve temporal
coherence, the number of constraints can be decreased or increased gradually to produce a smooth
transition.

8.3    Interaction between Smoke Objects
When two smoke objects collide, their density distributions overlap and they should naturally merge into
a larger smoke region since they do not have hard boundaries. Other scalar fields, such as temperature
and color, can also be advected by semi-Lagrangian tracing and blended together in the same way as
the density. The underlying guiding objects should either be merged or replaced with a new one which
may require a new shape transition. Velocity constraints should only be imposed on the new boundary
afterwards. In the example shown in Figure 4, two heart-shaped smoke objects merge into a larger one
and their original colors get advected and blended together. The heart-shaped target object is defined
by a closed-form implicit function: (2x 2 + y 2 − z 2 − 1)3 − 0.1x 2 z 3 − y 2 z 3 = 0.

8.4    Large Motion
To demonstrate large motion tracking, we use a synthetic character equipped with motion-captured
data as the underlying guiding object. The limbs of this articulated character have large frame-to-frame
motion so that a small amount of smoke escapes. We also put a few smoke sources on the floor. The results
from this example are shown in Figure 5. In general, smoke does not move very fast. Its appearance
would look less natural if we force the smoke to follow fast motion. Nevertheless, it can be clearly seen
that the velocity constraints induced by the fast motion of the limbs creates an interesting velocity field
for the rest of the environment and the smoke around the character follows this velocity field.

8.5    Tradeoff between Control and Smoke Appearance
The number of velocity constraints on the smoke boundary is the most important factor affecting the
realism of the smoke simulation because our velocity constraints are kinematic constraints that do
not involve dynamics. The shape of the smoke region is free to evolve without any constraints, but is
guaranteed to match the guiding object with dense constraints. Thus, we can achieve various levels of
tradeoff between control and smoke appearance by changing the number of constraints. However, the
user does not have direct control over the number of constraints which is automatically adjusted at
each time step according to the error metrics, the error thresholds, and other parameters, as discussed
in Section 5.3 and 7.3. There are no velocity constraints at all if the shape discrepancy is below the
error threshold(s); otherwise, the number of constraints is increasing, and realism is partially given
up for control. Therefore, in general, larger error thresholds allow more realistic smoke appearance,
but less object structures. The parameter µ in Equation (17) can also affect the liveliness of the smoke.
Larger µ’s provide stronger force feedback from the pressure, and typically lead to more lively motion.
Figure 9 gives a comparison of the visual quality generated by different combinations of error metrics,
thresholds, and µ. In this experiment, we chose to use both ev and e L∞ simultaneously. For ev , the
threshold is set between 0% and 5% of the volume of the target object. For e L∞ , the error threshold is
set between 0 and 4 voxels in a 128 × 128 × 128 grid. µ is set between 0.1 and 0.3.

8.6    Comparisons with other Control Schemes
A simple solution to our problem is to perform smoke simulation in the volume enclosed by the boundary
surface of the guiding implicit function. Obviously, the smoke is forced to follow the object when it moves.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                         Controllable Smoke Animation With Guiding Objects                 •     161




Fig. 9. A comparison of the visual quality of the smoke objects by different combinations of error metrics, thresholds, and the
pressure diffusion parameter. (a) The error thresholds ev = 0%, e L∞ = 0 voxels, and the diffusion parameter µ = 0.1; (b) ev = 3%,
e L∞ = 2.5 voxels, and µ = 0.1; (c) ev = 5%, e L∞ = 4 voxels, and µ = 0.3. Grid size 128 × 128 × 128.

This solution is similar to our scheme for large motion, but the hard boundary used for large motion
disappears once the smoke has been moved to the target location. There are a few limitations with
this solution using a hard boundary. Even though the boundary surface is not visualized during final
rendering, it will still be obvious since the smoke stops or reflects at the boundary. A smooth, hard
boundary for the smoke is not a realistic phenomenon. The second problem is that the smoke tends
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
162      •      L. Shi and Y. Yu

to be stationary, without interesting motion, inside a closed volume with a limited size. A side-by-side
comparison is made between this simple scheme and our method in Figure 6. In the results generated by
our method, the smoke close to the boundary of the target object still has natural motion and advection.
   Our control scheme also has advantages and differences if compared to the method in Treuille et al.
[2003] and McNamara et al. [2004]. First, our scheme is more efficient by introducing velocity con-
straints instead of evaluating derivatives of the velocity field with respect to all control parameters.
The time complexity of our method is on the same order of magnitude as a single smoke simulation
without shape matching. For example, Figure 7 shows an example where the smoke switches back and
forth twice between the shapes of a “check” and an “X” on a 128 × 128 × 128 grid. It took only 4 hours
15 minutes to generate the 1250-frame sequence. Second, our scheme can maintain dynamic smoke
appearances around a shape for an arbitrarily long time. Nevertheless, if we release the velocity con-
straints, the smoke becomes free to evolve (Figure 8(d)). As shown in Figure 1(a), we can also release
some of the constraints, and then regain control as we wish. A notable difference is that our algorithm
is targeted at object shapes, while the method in Treuille et al. [2003] is designed for arbitrary density
distributions at keyframes.
   A video including the aforementioned examples may be found at:
   http://www-sal.cs.uiuc.edu/~yyz/movie/smoke_obj.avi

9.    CONCLUSIONS AND FUTURE WORK
We have presented a novel technique to control the density and dynamics of smoke (a gas phenomenon)
so that the synthetic appearance of the smoke (gas) resembles a still or moving object. The main focus
has been on controllability and appearance. In order to match the smoke surface with the target sur-
face, we represent both smoke and objects as implicit functions and impose carefully designed velocity
constraints derived from a shape matching functional. An empirical compressible fluid model has been
introduced for effectively integrating constraints into the velocity field while maintaining realistic fluid
appearances. The overall framework represents a significant advance over previous methods for con-
trolling fluids. The implementation of our system is built upon recent advances in smoke simulation
and shape transformation.
   We would like to extend this work to 2D images and video objects. The difference between a 3D object
and a 2D image is that an object is a binary function, while an image has multiple levels of intensities
or colors. Matching the smoke boundary with the object boundary should be extended to matching the
level sets of smoke density with the level sets of image intensities or colors. Nevertheless, we expect the
approach introduced here can be generalized to handle that circumstance by matching the boundaries
of these level sets. Unlike 3D animated objects addressed in this article, the motion of video objects is
unknown, and needs to be solved using computer vision techniques such as optical flow [Beauchemin
and Barron 1995; Black and Anandan 1996]. Once the motion of these objects has been estimated, the
approach in this article can also be adapted to produce fluid appearances for them.

APPENDIX

A.    VARIATIONAL GRADIENT
In this section, we provide an intuitive derivation of Equation (3) from Equation (2) in a 2D space. The
derivation for a 3D space can be obtained similarly. The smoke boundary is first discretized into a finite
set of points. Consider one of the points xi shown in Figure 10. We need to perturb the position of xi
to reduce the integral in Equation (2). Obviously, xi should move further into the interior of the target
object. If xi moves along the local normal at xi by an infinitesimal distance, Equation (2) is reduced
by an amount equal to the area enclosed by the polygon xi−1 xi xi+1 xiN , where xiN represents the new
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                        Controllable Smoke Animation With Guiding Objects                 •     163




Fig. 10. Moving points on the smoke boundary along their respective normal directions can reduce the shape discrepancy
between the smoke region and the target object most quickly.

location of xi . If xi moves along some other arbitrary direction to xi by the same infinitesimal distance,
Equation (2) is reduced by an amount equal to the area enclosed by the polygon xi−1 xi xi+1 xi . Since xi xiN
is perpendicular to the local boundary, when the lengths of xi−1 xi and xi xi+1 become sufficiently small,
the area of xi−1 xi xi+1 xiN is guaranteed to be larger than the area of xi−1 xi xi+1 xi . That is, Equation (2)
can be decreased most quickly by moving xi along the local normal of the smoke boundary. This also
holds for all other points on the same boundary. By increasing the number of points in the boundary
discretization, the distance between two adjacent points can be arbitrarily close to zero. By definition
[Gelfand and Fomin 1963], the variational derivative of the functional in Equation (2), with respect to the
geometry of the smoke boundary, exists. The variational derivative, with respect to a specific boundary
point, is the unit normal vector at that point, multiplied by an appropriate sign given in Equation (3).

B.   AN EQUIVALENT CONDITION FOR ∇ · U = 0
We show that, for a bounded workspace , without sources and sinks, ∇(∇ · u) = 0 everywhere is
equivalent to ∇ · u = 0 everywhere. Without loss of generality, suppose there exists a point x0 ∈ such
that ∇ · u(x0 ) > 0. Since is bounded,

                                                         (∇ · u)d x = 0.

Therefore, there must be another point x1 ∈          such that ∇ · u(x1 ) < 0. Consider an arbitrary path
between x0 and x1 . There must exist a point xm on that path such that ∇(∇ · u(xm )) = 0. Otherwise,
∇ · u(x0 ) = ∇ · u(x1 ). Thus, ∇ · u must be zero everywhere.
REFERENCES
ALEXA, M., COHEN-OR, D., AND LEVIN, D. 2000.            As-rigid-as-possible shape interpolation. In SIGGRAPH 2000 Conference
  Proceedings. 157–164.
BEAUCHEMIN, S. AND BARRON, J. 1995. The computation of optical flow. ACM Comput. Surv. 27, 3, 433–467.
BLACK, M. AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields.
  Comput. Vision Image Understand. 63, 1, 75–104.
CARR, J., MITCHELL, T., BEATSON, R., CHERRIE, J., FRIGHT, W., MCCALLUM, B., AND EVANS, T. 2001. Reconstruction and representation
  of 3d objects with radial basis functions. In SIGGRAPH 2001 Conference Proceedings. 67–76.
CHORIN, A. 1967. A numerical method for solving incompressible viscous flow problems. Computat. Phys. 2, 12–26.
DESBRUN, M. AND CANI-GASCUEL, M.-P. 1998. Active implicit surface for animation. In Proceedings of Graphics Interface.
EBERT, D., MUSGRAVE, K., PEACHY, D., PERLIN, K., AND WORLEY, S. 1998. Texturing and Modeling: A Procedural Approach, 2nd
  Ed. AP Professional.
                                                                          ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
164      •      L. Shi and Y. Yu

ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. ACM Trans. Graph. 21, 3,
  736–744.
FATTAL, R. AND LISCHINSKI, D. 2004. Target-driven smoke animation. ACM Trans. Graph. 23, 3, 439–446.
FEDKIW, R., STAM, J., AND JENSEN, H. 2001. Visual simulation of smoke. In SIGGRAPH 01 Conference Proceedings. 15–22.
FOSTER, N. AND FEDKIW, R. 2001. Practical animation of liquids. In SIGGRAPH 2001 Conference Proceedings. 23–30.
FOSTER, N. AND METAXAS, D. 1997a. Controlling fluid animation. In Proceedings of Computer Graphics International. 178–188.
FOSTER, N. AND METAXAS, D. 1997b. Modeling the motion of a hot, turbulent gas. In SIGGRAPH 97 Conference Proceedings.
  181–188.
GARDNER, G. 1985. Visual simulation of clouds. In Proc. of SIGGRAPH’85. 297–304.
GELFAND, I. AND FOMIN, S. 1963. Calculus of Variations. Prentice-Hall.
GOLUB, G. AND LOAN, C. V. 1996. Matrix Computations. The Johns Hopkins Unversity Press.
GOMES, J., BEIER, T., COSTA, B., DARSA, L., AND VELHO, L. 1997. Warping and morphing of graphical objects. SIGGRAPH 97
  Course Notes.
HUGHES, J. F. 1992. Scheduled fourier volume morphing. In SIGGRAPH 92 Conference Proceedings. 43–46.
KAJIYA, J. AND VON HERZEN, B. 1984. Ray tracing volume densities. Comput. Graph. (SIGGRAPH 84 Proceedings) 18, 3.
KAPLER, A. 2002.       Evolution of a vfx voxel tool. In SIGGRAPH 2002 Sketches & Applications, Conference Abstracts and
  Applications. 179.
LAMORLETTE, A. AND FOSTER, N. 2002. Structural modeling of flames for a production environment. In SIGGRAPH 02 Conference
  Proceedings. 729–735.
LERIOS, A., GARFINKLE, C., AND LEVOY, M. 1995. Feature-based volume metamorphosis. In SIGGRAPH 95 Conference Proceed-
  ings. 449–456.
MCNAMARA, A., TREUILLE, A., POPOVIC, Z., AND STAM, J. 2004. Fluid control using the adjoint method. ACM Trans. Graph. 23, 3,
  447–454.
MUSETH, K., BREEN, D., WHITAKER, R., AND BARR, A. 2002. Level set surface editing operators. ACM Trans. Graph. 21, 3, 330–338.
OSHER, S. AND FEDKIW, R. 2001. Level set methods: an overview and some recent results. Computat. Phys. 169, 2.
OSHER, S. AND SETHIAN, J. 1988. Fronts propagating with curvature dependent speed: Algorithms based on hamilton-jacobi
  formulations. Comput. Phys. 79, 12–49.
PERLIN, K. 1985. An image synthesizer. In Proceedings of SIGGRAPH’85. 287–296.
PERLIN, K. AND HOFFERT, E. 1989. Hypertexture. In Proceedings of SIGGRAPH’89. 253–262.
SETHIAN, J. 1999. Level Set Methods and Fast Marching Methods. Cambridge University Press.
SHI, L. AND YU, Y. 2004. Inviscid and incompressible fluid simulation on triangle meshes. Comput. Animat. Virt. Worlds 15, 3-4,
  173–181.
SIMS, K. 1992. Choreographed image flow. J. Visualiz. Comput. Animat. 3, 31–43.
STAM, J. 1995. Multi-scale stochastic modeling of complex natural phenomena. Ph.D. Thesis, University of Toronto.
STAM, J. 1999. Stable fluids. In SIGGRAPH 99 Conference Proceedings. 121–128.
STAM, J. 2003. Flows on surfaces of arbitrary topology. ACM Trans. Graph. 22, 3, 724–731.
STANIFORTH, A. AND COTE, J. 1991. Semi-lagrangian integration schemes for atmospheric models: A review. Monthly Weather
  Review 119, 2206–2223.
TREUILLE, A., MCNAMARA, A., POPOVIC, Z., AND STAM, J. 2003. Keyframe control of smoke simulations. ACM Trans. Graph. 22, 3,
  716–723.
TURK, G. AND O’BRIEN, J. 1999.       Shape transformation using variational implicit functions. In SIGGRAPH 99 Conference
  Proceedings. 335–342.
TURK, G. AND O’BRIEN, J. 2002. Modelling with implicit surfaces that interpolate. ACM Trans. Graph. 21, 4, 855–873.
WOLBERG, G. 1990. Digital Image Warping. IEEE Computer Society Press.
YNGVE, G., O’BRIEN, J., AND HODGINS, J. 2000. Animating explosions. In SIGGRAPH 2000 Conference Proceedings. 29–36.
YU, Y. AND SHI, L. 2002.        Object modeling and animation with smoke. Tech. Rep. No. UIUCDCS-R-2002-2262 (Engr.
  UILU-ENG-2002-1705), Computer Science, University of Illinois at Urbana-Champaign, http://www.cs.uiuc.edu/Dienst/UI/2.0/
  Describe/ncstrl.uiuc cs/UIUCDCS-R-2002-2262 (Jan.).
ZHAO, H.-K., OSHER, S., AND FEDKIW, R. 2001. Fast surface reconstruction using the level set method. In 1st IEEE Workshop on
  Variational and Level Set Methods.


Received July 2003; revised May 2004, August 2004; accepted October 2004

ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
Adaptation of Performed Ballistic Motion
                       ˇ ´
ADNAN SULEJMANPASIC and JOVAN POPOVIC ´
Massachusetts Institute of Technology


Adaptation of ballistic motion demands a technique that can make required adjustments in anticipation of flight periods when
only some physically consistent changes are possible. This article describes a numerical procedure that adjusts a physically
consistent motion to fulfill new adaptation requirements expressed in kinematic and dynamic constraints. This iterative procedure
refines the original motion with a sequence of minimal adjustments, implicitly favoring motions that are similar to the original
performance, and transforming any input motion, including those that are difficult to characterize with an objective function. In
total, over twenty adaptations were generated from two recorded performances, a run and a jump, by varying foot placement,
restricting muscle use, adding new environment constraints, and changing the length and mass of specific limbs.
Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; G.1.6
[Numerical Analysis]: Optimization—Constrained Optimization; G.1.7 [Numerical Analysis]: Ordinary Differential Equa-
tions—Boundary value problems
General Terms: Algorithms, Design
Additional Key Words and Phrases: Animation with constraints, physically based animation, motion capture




1.   INTRODUCTION
Ballistic motions such as jumps, runs, and other acrobatic maneuvers consist of flight periods during
which performers are propelled by the force of gravity and momentum alone. In anticipation of such
periods, performers execute specific actions to accomplish the desired outcome. For example, before a
twisting jump, a performer bends down, bursts upward to propel himself, and simultaneously spins his
body to generate the angular momentum that twists his body in the air. An adaptation technique must
emulate this anticipation when adding a twist to a recorded jump performance.
  We describe a numerical method that creates such anticipation in ballistic motions of a full human
figure with many (in our case, 42) degrees of freedom. This iterative algorithm modifies a physically
consistent input motion with a sequence of minimal modifications until all adaptation goals are met.
Each adjustment modifies the entire motion to anticipate the lack of control in flight with changes
at other times, when the appropriate control is available. As a result, lengthening the jump with
kinematic constraints produces a motion with a deeper bend in the knees before the figure leaps from
the ground.1

1 This approach is not required for motions without a ballistic period, such as walking and reaching; simpler kinematic techniques

                                                                                 c
are effective in these cases [Bruderlin and Williams 1995; Witkin and Popovi´ 1995].


This research was sponsored by the National Science Foundation and the NTT Corporation.
Authors’ address: MIT CSAIL, The Stata Center, 32 Vassar Street, Cambridge, MA 02139; email: {adnans, jovan}@csail.mit.edu.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists,
or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested
from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org.
 c 2005 ACM 0730-0301/05/0100-0165 $5.00
                                                              ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 165–179.
166      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´

  Existing approaches in the literature employ optimization methods and caution of convergence diffi-
culties, particularly for physically based problems with many degrees of freedom. Our method does not
explicitly optimize any objective function. Instead, it searches for a motion that meets the constraints
with a bias for motions that are similar to the original performance. The absence of the objective function
allows it to transform any physically consistent motion, including biological motions that are typically
difficult to characterize with an objective function.
  We describe a simple scaling strategy for drastically improving the convergence of our iterative
method. Unlike the previous methods, which simplified the state space or the governing dynamics
equations, the scaling permits adaptations with restricted muscle use and with general interaction
between the character and the environment. The same scaling strategy also improves the optimal
trajectory formulations such as the method of spacetime constraints [Witkin and Kass 1988], which
allows us to compare such optimizations with our method.

2.    BACKGROUND
Motion adaptation has emerged as an important computer graphics problem with the maturation
of acquisition technology for archiving high-quality human motion. The first methods broadened the
applicability of acquired motion with the introduction of tools that displace (warp) motion to satisfy
                                                                               c
new kinematic constraints [Bruderlin and Williams 1995; Witkin and Popovi´ 1995]. An animator can
displace the foot in a few keyframes and the editing tool propagates the changes to the remaining frames
with the interpolated displacements. Although this fast and simple approach is sufficient in many cases,
it only satisfies kinematic constraints at the displaced keyframes. If animators want to ensure that the
feet do not go below the ground at any point in the animation, they have to introduce displacements at
every frame, which defeats the purpose of using the automatic techniques in the first place.
   Currently, these problems can be resolved with a technique that alternates between inverse kinemat-
ics, which enforces specified constraints, and filtering, which smooths the changes [Lee and Shin 1999].
A more general approach, and the one that is closer to our technique, is an optimal trajectory method
that minimizes the difference from the original motion, while enforcing kinematic constraints through-
out the motion [Gleicher 1997, 1998]. The motion retargeting techniques described in these papers do
not generate physically consistent motions, but suggest applying a sequence of minimal modifications
to the input motion. This is also an approach we use to adapt motions in a physically consistent manner.
   Optimal trajectory methods were introduced to computer graphics by Witkin and Kass [1988], who
developed the spacetime constraints technique for motion synthesis. Dynamics is an integral part of
that formulation, which showed that by optimizing power consumption, a numerical procedure could
select a natural motion among the many physically valid alternatives. Extending these observations to
generate natural motions for a full human figure proved difficult [Cohen 1992; Liu et al. 1994] even as
complex simulations of the human musculoskeletal system were verified by experimental data [Pandy
and Anderson 2000]. The one exception was a technique for generating short motion transitions [Rose
et al. 1996].
   It seemed necessary to simplify the state space by reducing the number of degrees of freedom, or to
approximate the governing dynamics equations. The first technique for adaptation of ballistic motion
projected the original motion onto a simple character, where it made the adjustments using the simpli-
                                                                                      c
fied dynamics, before projecting the changes back onto the full human figure [Popovi´ and Witkin 1999].
However, the final projection step does not produce a physically consistent final motion. Our technique
produces physically consistent motions without any simplification.
   The governing physical laws could also be simplified with momentum constraints, which enforce
                                             c
typical momentum profiles [Liu and Popovi´ 2002; Abe et al. 2004], or with aggregate-force constraints,
which eliminate internal torques and constrain only the total (aggregate) force [Fang and Pollard 2003].
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                      Adaptation of Performed Ballistic Motion             •     167




Fig. 1. The adaptation of a human broad jump (left figure) generates a new physically consistent jump (right figure) with a
staggered takeoff and landing. An animator specifies only the new foot placement to evoke this change.

These simplifications yield efficient adaptation methods at the cost of ignoring the internal muscle use.
This implies that an animator is not allowed to restrict muscle usage and not allowed to specify an
efficiency criterion that depends on such muscle activation.
   Another option is to reparameterize the degrees of freedom with principal components through sta-
tistical analysis of variations observed in a specific behavior [Safonova et al. 2004]. The inferred low-
dimensional representation facilitates convergence of optimal trajectory methods without simplifying
dynamics. This approach complements our technique, but must be used carefully to construct low-
dimensional subspaces that are capable of encoding motions that satisfy arbitrary kinematic constraints
                                    sc
[Safonova et al. 2004; Sulejmanpaˇ i´ 2004].
   Optimal trajectory methods are ideally suited for the adaptation of ballistic motions that require an-
ticipatory planning. In scenarios in which anticipation plays a less prominent role, dynamic properties
such as balancing or physically valid interaction with the environment might be more easily accom-
plished with kinematic and dynamic filtering [Zordan and Hodgins 1999; Yamane and Nakamura 2000;
Tak et al. 2002; Shin et al. 2003]. These approaches are effective if the adjustments are small, and the
required look-ahead is localized. When this is not the case, the simultaneous approach of optimal tra-
jectory methods and our trajectory adaptation is a better choice.

3.    ADAPTATION
Adaptation reuses a recorded human motion by conforming it to make the required changes. The entire
process resembles the standard keyframing techniques, which allow the animator to specify positions of
hands, feet, or other end-effectors, with the added capability to enforce laws of physics, limit the use of
specific muscles, and otherwise restrict the dynamics with constraints. Figure 1 shows an adaptation
of a broad jump: the input motion, in which the performer jumps and lands with parallel footing, is
transformed into a physically valid jump with staggered foot placement.

3.1   Formulation
Adaptation computes the motion q, the internal muscle use f, and the external environment-reaction
forces determined by the Lagrange multipliers λ. The solution satisfies the adaptation goals expressed
in kinematic, K, and dynamic, D, constraints2 :

                                                              K(q) = 0
                                                                                                                                  (1)
                                                        D(q, f, λ) = 0.

2 Inequality constraints, which are excluded from Equation (1) for simplicity, are also supported with an active-set method, which

replaces the active inequality constraints with equalities.
                                                                           ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
168      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´

Here, the dynamic constraints contain both the governing physical equations and other restrictions on
dynamic quantities such as ground-reaction forces and internal muscle usage.
  The constraints alone do not uniquely determine the desired trajectories. Instead, the transformation
seeks to identify natural motions within the set of feasible trajectories that satisfy the constraints.
Optimal trajectory methods accomplish this goal by maximizing the smoothness of muscle actuation
         c
[Popovi´ and Witkin 1999] or the proximity to the input motion [Gleicher 1997]. Our method does not
explicitly optimize any such criteria: instead, it modifies input trajectories with a sequence of alterations
until it constructs trajectories that meet the constraints. This choice allows us to transform any input
motion, including those that are difficult to characterize with an objective function, as is often the case
with natural motions. Our method applies minimal alterations throughout the transformation process
to keep the resulting motion close to the original input, thereby preserving its natural characteristics.
Section 4.1 describes a gradient method for computing these alterations efficiently.

3.2    Dynamic Constraints
Dynamic constraints ensure physical consistency and provide additional controls for animators. For
example, animators can restrict the use of a specific muscle with inequality constraints on joint torques.
The nonlinear equality constraints that enforce the laws of physics are hardest to fulfill and their
formulation has a profound effect on the efficiency of the numerical solution. Their proper scaling
improves convergence and enables adaptation of motions with many degrees of freedom. When internal
torques are not restricted, this performance can be further improved with a formulation that eliminates
the internal torques to reduce the order of constraints [Fang and Pollard 2003].
  3.2.1 Standard Formulation. The differential algebraic equations that express Newton’s laws are
derived with the Euler-Lagrange equations from the Lagrangian L of the human figure:
                                          d ∂L           ∂L      ∂P
                                                                     
                                             dt ∂ qr˙
                                                      − ∂qr − λT ∂qr
                                                                    
                          DT (q, f, λ) ≡  d ∂ L − ∂ L − λT ∂P − f i  = 0,
                                          dt ∂ q˙i                                           (2)
                                                      ∂qi     ∂qi
                                                                P(q)
where the index r enumerates the degrees of freedom in the unactuated root joint (global position and
global orientation) and the index i enumerates the remaining degrees of freedom in joints actuated
by muscle forces f i . For a human figure, the complexity of Euler-Lagrange equations and associated
derivatives necessitates systematic evaluation [Liu 1996] and elimination of redundant computations
with caching or a recursive formulation [Hollerbach 1980].
  The algebraic equations P and the matching Lagrange multipliers λ define the dynamic interaction
between the character and the environment.3 For example, during ground contact these constraints
maintain the contact between the feet and the ground. Such holonomic constraints produce algebraic
equations that depend only on the kinematic quantities, but they are still a component of the dynamics
constraints because of the associated multiplier terms λT ∂P , which describe the exchange of forces
                                                            ∂q
between the character and the environment. In contrast, pure kinematic constraints are separated
from the dynamic constraints because they do not have the associated multiplier terms.
   3.2.2 Low-Order Formulation. When internal torques f i are not bound by the objective function
(to select a natural motion) or by the constraints (to restrict the use of a muscle), the torques are free

3 Another option is to select an independent set of reduced coordinates for which the environment constraints are satisfied by

construction. Section 3.2.3 discusses the two design choices in more detail.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                Adaptation of Performed Ballistic Motion              •     169

variables, and can be eliminated from the optimization problem:
                                                    ∂L         ∂L         ∂P
                                               d
                                                    ∂ qr
                                                           −   ∂qr
                                                                     − λT ∂qr
                                DG (q, λ) ≡    dt      ˙
                                                                                 = 0.                                        (3)
                                                           P(q)
                                                                                       ∂P
On the ground, the total force on the body is given by the ground reaction force λT ∂qr . These reaction
forces are a function of the root position and the angle values of all other joints between the root and
the contact point. In flight, there are no environment constraints and the total force is zero.
   Eliminating optimization variables and lowering the order of dynamic constraints improves the ef-
ficiency of numerical solutions—often by an order of magnitude—but might have an adverse effect on
the smoothness of the resulting motion as freely varying torques may cause the body limbs to jerk un-
desirably. In many cases, minimizing the total change made to joint angles and multipliers corrects this
problem. Additionally, the torque constraints from Equation (2) can be reinserted and their smoothness
controlled by the number of control points in the spline parameterization. Even when the visual quality
of the final motion requires optimizing every joint-angle, torque, and multiplier value, the efficiency
of adaptation with the low-order constraints of Equation (3) enables rapid prototyping before the final
refinements and smoothing are made.
  3.2.3 Discussion. The low-order dynamics constraints are equivalent to the aggregate-force con-
straints introduced by Fang and Pollard [2003]. Fang and Pollard use reduced coordinates to enforce
environment constraints explicitly. In contrast, our formulation adds Lagrange multipliers to determine
the forces needed to enforce the environment constraints. Baraff [1996] summarizes the differences be-
tween the reduced and multiplier formulations for general applications in computer graphics. For the
adaptation problem, we prefer the multiplier formulation for its explicit modeling of reaction forces
and for its systematic treatment of all environment constraints P, especially in the presence of cyclic
dependencies (e.g., both feet constrained to be on the ground simultaneously).
  The aggregate-force constraints are similar to the momentum constraints [Liu and Popovi´ 2002].    c
In flight, they are identical and equivalently state that the total angular momentum remains constant
while the center of mass follows a parabolic trajectory. On the ground, or whenever the human figure
interacts with the environment, the momentum constraints employ characteristic momentum patterns,
without modeling the reaction forces. These momentum patterns emerge naturally from our aggregate-
force formulation. In addition to this generality, our formulation exposes the reaction forces and enables
their use in objective functions (e.g., to match the impact forces in the original motion or to reduce them
for a “softer” run) and constraints (e.g., to keep the impact forces within a friction cone [Fang and Pollard
2003]).
  The two expressions for dynamics, with torques DT and without torques DG , define the two extremes
in a range of possibilities. Intermediate formulations, which include some but not all torques, can exploit
the benefits of the reduced formulation, which generates results rapidly but has limited applicability,
and the benefits of the full formulation, which generates the best results and applies to all adaptation
problems. For example, if the adaptation requires a jump with an injured ankle, the values of the ankle
torques can be restricted and added to the optimization along with the Euler-Lagrange equations for
the corresponding degrees of freedom. Or, if some of the limbs jerk undesirably, their torques can be
included and their change minimized along with the modifications to joint angles and multipliers.

4.   NUMERICAL SOLUTION
Our numerical method approximates trajectories with the cubic B-spline expansion [Cohen 1992] and
computes the expansion coefficients with collocation by fulfilling kinematic and dynamic constraints at
                                                                      ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
170       •                  sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´

prescribed time points. Because of the large number of unknowns, we employ a gradient method that
computes the adjustments by solving a sequence of quadratic subproblems with linearized constraints.
Proper scaling of constraint equations ensures efficient convergence of this method.
4.1     Gradient Method
Our adaptation technique selects a natural motion with a sequence of minimal modifications that
iteratively modify the original motion until all adaptation goals are met. The gradient method alternates
between computing the direction vector di of smallest magnitude, and computing the step size αi ∈ (0, 1]
for the modification in this direction. The small size of the individual direction vectors ensures that
the cumulative change is small, even though the proximity to the original motion is never enforced
explicitly.
   At each iteration, the direction for the next modification di is a minimum-norm solution of an un-
derdetermined linear system produced by a linearization of the kinematic and dynamic constraints
C = (K, D):
                                             di = arg min              d   2
                                                              d
                                                          ∂C                                           (4)
                                                 subject to   (xi ) d = 0.
                                                                      C(xi ) +
                                                           ∂x
The current iterate xi and the adjustment di contain the coefficients for joint angles, torques, and
multipliers. In effect, the solution to this quadratic problem computes the smallest change in joint
angles, reaction forces, and muscle actuations.
   The dimensions of the Jacobian matrix ∂C/∂x restricts the choice of numerical solutions to tech-
niques that can exploit its sparsity. The Conjugate-Gradient algorithm converges slowly because the
corresponding normal equations are poorly conditioned and do not improve with diagonal precondi-
tioning. A faster solution can be derived by explicit construction of the null-space basis with Q-less QR
factorization of the sparse Jacobian matrix [Demmel and Higham 1993]. In our implementation, we
chose the quadratic programming technique SQOPT [Gill et al. 1997] to support inequality constraints
with sparse LU factorization of the Jacobian matrix.
   The minimum-norm solution defines the direction for the line search, which computes the step size αi
for the modification that minimizes the distance of the next iterate xi+1 from the adaptation constraints4 :
                                                  αi = arg min                 C(xi + αdi ) 2 ,
                                                                  α                               .                   (5)
                                               xi+1 = xi + αi di .
The iteration stops once the constraints are satisfied with the desired accuracy. Although the magnitude
of the total adjustment could be improved by additional steps along the constraints surface, these
improvements have negligible effect on the visual quality of the resulting motion. A similar effect was
also observed in adaptation problems without dynamics [Gleicher 2004].
4.2     Scaling
Under restrictive theoretical assumptions, some gradient methods can be shown to produce the same
sequence of iterates regardless of the scaling. In practice, this scale invariance cannot be achieved and
proper scaling is essential to resolve the difficulties in the conditioning of difficult nonlinear problems
[Gill et al. 1989]. In our experiments, proper scaling of joint angles, torques, and multipliers improved
efficiency and prevented divergence.

4 The   same approach would accommodate a weighted norm, but our scaling made such norm unnecessary for our experiments.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                     Adaptation of Performed Ballistic Motion               •     171

   A simple physical pendulum, weighing 70 kg (a typical human weight) and 1.70 m long (a typical
human height), demonstrates the effect of scaling on the computation of physically valid trajectories
with collocation and gradient methods. Without scaling, the computation of the pendulum trajectory
requires 129 iterations and 11.6 seconds of computation time. With the simple scaling procedure de-
scribed in this section, the computation of identical trajectories requires 24 iterations and 1.96 seconds.
In both instances, the parameters of the gradient procedure are identical: the pendulum starts in a
configuration perpendicular to the direction of gravity and swings for 0.6 s, ending just before the back
swing. On adaptation problems, the effect is even more drastic: improper scaling prevents convergence
without excessively small steps, which can extend the computation of several seconds to as long as
several hours.
   The problem can be traced back to the discrepancy in the range of the state variables, the torques,
and the Lagrange multipliers, which in turn affects the scaling of the Jacobian matrices. The simplest
solution is to scale the mass density of each limb by a uniform constant factor s, and to solve for the new
joint angles q , torques f , and Lagrange multipliers λ . This scaling changes the Lagrangian L = sL
and the expression of the dynamics in terms of the unscaled Lagrangian L:

                                                            ∂L         ∂L       ∂P
                                                                                               
                                                    d
                                                  s dt       ∂ qr
                                                                ˙   − s ∂q − λ T ∂q
                                                                            r             r
                                                                                               
                            DT (q , f , λ ) ≡ s d
                                                     ∂L
                                                                −     ∂L
                                                                    s ∂q    −λT     ∂P         
                                                                                          − fi  .                                 (6)
                                                 dt      ˙
                                                      ∂ qi              i           ∂qi
                                                                     P(q )

As division by the factor s reveals, this scales the values of torques and Lagrange multipliers without
changing the joint-angle trajectories:

                                                             f λ
                                      DT (q , f , λ ) = DT q, ,  .                                                                 (7)
                                                             s s

Loss of accuracy, which could prevent accurate specification of kinematic constraints, is one possible
drawback of this simple scaling transformation. We avoid this problem in our experiments by empirically
choosing the scale factor s = 0.001. We found that larger values of 0.1 and 0.01 degraded convergence,
and smaller values of 0.0001 and less interfered with our internal constant, which determines the
feasibility of each constraint. The same scaling factor worked in all of our experiments, but should the
loss of accuracy become problematic with this simple strategy, proper scaling could also be established
by determining the range of torque and multiplier forces precisely [Gill et al. 1989].
  Scaling is not as critical for alternative formulations with momentum constraints [Liu and Popovi´   c
2002] and constrained aggregate forces [Fang and Pollard 2003]. Momentum constraints do not compute
Lagrange multipliers as they do not model ground-reaction forces. The aggregate-force constraints use
reduced coordinates, which do not employ Lagrange multipliers. This leaves only joint-angle trajectories
for unknowns, as internal torques are also not computed in either formulation. Because joint angles have
identical units, the problem is scaled automatically, but this convenience prevents these alternatives
from solving adaptation problems that must restrict muscle forces.

4.3   Initialization
                                                                                     ¯
The initial values q, f, and λ are estimated by fitting the joint-angle trajectories q in the original
performance. This initialization solves an optimal trajectory problem with the least squares objective
                                                                            ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
172      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´

function:
                                               arg      min               q(t) − q(t) 2 dt
                                                                                 ¯
                                                     q(t),f(t),λ(t)
                                                                                                      (8)
                                                   subject to         K(q) = 0
                                                                      D(q, f, λ) = 0.
A solution to a similar optimization problem was previously used to estimate the parameters of a
                                                             c
simplified character from the original performance [Popovi´ and Witkin 1999]. The key difference is
that our estimation computes joint angles, torques, and multipliers for the entire human figure using
the full dynamics of Equation (2), without any simplification.
   For this optimization, the torques and multipliers are initially set to zero, and the joint angles are
computed with cubic B-spline interpolation of joint angles in the original performance. Initialization is
performed only once for each recorded performance. Subsequently, the estimated trajectories are stored
with the motion and used for every adaptation problem.
   The initialization assumes that the original performance is a physically valid motion. When it is
not, the initialization might not successfully compute the torques and the multipliers required by our
adaptation technique. Although not explored in this article, a different initialization procedure might
                                                                                                 c
eliminate this requirement by consulting a stored database of transition poses [Liu and Popovi´ 2002],
or by restricting the ground-reaction forces [Fang and Pollard 2003].

5.    RESULTS
In our adaptation experiments, we varied foot placement, restricted the use of specific muscles, intro-
duced new environment constraints, added additional weights, and changed the skeleton dimensions to
generate over twenty adaptations from two performances: a run and a jump. The distinctive attributes
of the original motion were visibly preserved in all examples.
   In these experiments, we needed only a few constraints to indicate, the desired adjustments. In
contrast, kinematic techniques would require many manually specified constraints to ensure realism
of resulting motions. We relied primarily on point constraints to indicate placement of feet and hands.
For aesthetic reasons, cone constraints were used to enforce natural joint-angle limits and to prevent
interpenetration of limbs. Full pose constraints were used sparingly because they were not necessary to
generate high-quality adaptations, and because they are, in general, difficult to specify. On occasion, it
was easier to prevent limb intersections with a full pose constraint than with a cone constraint for each
joint. The figures include a visual illustration of almost all kinematic constraints required to generate
the motion shown. The only exceptions are inequality constraints (feet above the ground, feet clearing
the hurdle, or hand above the rim) and orientation constraints, which specified starting and final body
orientation for the twist jumps.
5.1    Discretization
Our implementation uses a collocation method to enforce the constraints at designated discrete time
points, which are also called collocation points. During takeoff, landing, and other periods with contact
between the character and the environment, the environment constraints maintain the contact and
the Lagrange multipliers determine the associated reaction forces. These multipliers and environment
constraints are not part of the dynamic constraints evaluated on collocation points in periods of free
flight.
  The unknowns, which include the joint-angles, torques, and multipliers, are also discretized. The
Lagrange multipliers λ are sampled at the collocation points, and the joint-angle q and torque trajec-
tories f are parameterized with cubic B-spline curves. Because the number of multiplier samples and
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                             Adaptation of Performed Ballistic Motion             •     173

                                                                 Table I.
                                                                           unknowns           dynamic constraints
                        Motion      q c. p.    f c. p.   λ samples      reduced   full        reduced      full
                        Jump          24         30          20           1128   2388           288       1296
                        Run           30         38          20           1320   2916           276       1572
                       Our implementation approximates each joint-angle q and torque trajectory f with a cubic B-spline
                       curve. The Lagrange multipliers λ are sampled at prescribed points that enforce the environment
                       constraints and are not approximated by a cubic B-spline curve. Our adaptation can use the full
                       dynamics of Equation (2), or the reduced formulation of Equation (3). If the internal torques are
                       not smoothed or restricted, then the reduced formulation enables faster convergence.




Fig. 2. Motion adaptation adjusts motions of high-dimensional human characters. A skeleton configuration with 42 degrees of
freedom was used in our adaptation experiments. The mass of each limb was assigned as a percentage of the total body mass.


B-spline control points determines the total number of unknowns,5 the appropriate setting is chosen
through trial-and-error to minimize the total number of unknowns. In general, we begin by matching
the control points, and collocation points, and then reduce the number of control points until we reach
the minimum necessary to avoid an overconstrained formulation. Table I lists the number of samples
and control points used in our experiments.

5.2   Initialization
The original performances were captured in a motion-capture studio. Standard commercial tools [Vicon
2003] were used to compute the limb lengths for the 42 degree-of-freedom character shown in Figure 2.
The mass of each limb is computed as a percentage of the total weight with the ratios shown in Figure 2.
These ratios were manually adjusted for the limbs in our skeleton from the weight ratios reported in a
standard biomechanical reference [Winter 1990].
  Given the inferred physical parameters and the original motion, the initialization (Section 4.3) es-
timates the joint-angle trajectories that are physically consistent with internal muscle use and the
environmental reaction forces. For this process, the environmental constraints and the duration of
each ground-contact and free-flight stage were identified manually by selecting the appropriate frames
in the original performance.

5 In a jump, for example, the unknowns for the full dynamics of Equation (2) include 24 × 42 = 1008 joint-angle control points

(24 for each of 42 degrees of freedom), 30 × 42 = 1260 torque control points, and 20 × 6 = 120, lagrange multiplier samples
(6 multipliers for location of both feet in each of 20 time samples). The reduced dynamics of Equation (3) have fewer unknowns
because they do not include torques.
                                                                                  ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
174      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´




Fig. 3. All motions were generated by adapting two performances: a human jump (a) and a run (j). The figures visualize the
constraints required for each adaptation. Red spheres represent point constraints, red cones the joint-limit constraints visualized
by the limits drawn around the corresponding limb, and the transparent gray boxes identify pose constraints, which constrain
the values of all joints in the skeleton. The shin is colored purple when the knee is constrained to a fixed value. A line extending
from a knee shows the maximum allowed range for a knee joint. Some inequality and orientation constraints are not shown.


5.3    Jumps
The original jump, after initialization, is shown in Figure 3(a). Our adaptation process transforms this
jump into new motions that meet the desired constraints. In Figure 1, for example, the constraints
specify new foot placement, and the adaptation produces a jump with staggered takeoff and landing.
In Figure 3(b), the constraints maintain the foot placement in the original jump and add a briefcase
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                          Adaptation of Performed Ballistic Motion           •     175

to increase the weight of the right arm. The adaptation lowers the motion of the heavy right arm and
compensates by raising the lighter left arm higher. This motion is then further refined in Figure 3(c),
with a constraint that limits the torque applied around the right ankle joint. This upper bound simulates
the effect of an injured right ankle. The healthy left leg carries more weight throughout the takeoff
and trails the injured right leg to generate momentum needed for the liftoff. In the air, the same leg
hurries forward to absorb most of the landing impact. In Figure 3(d), a similar upper bound constrains
the torque applied at the right shoulder joint. As a result, the limp right arm dangles throughout the
motion and the upper body compensates to swing it forward. As with a briefcase jump, the healthy arm
is lifted higher to increase the vertical momentum.
   In Figures 3(e) and 3(f), the constraints enforce the pelvis orientation, along with the foot placement.
This produces twisting jumps: first in 3(e) with the original takeoff and modified landing, and then in
3(f) with the modified takeoff and the original landing. In the second example, we needed to adjust the
original jump to rotate the takeoff stage. This manual adjustment is a limitation that’s revisited in
Section 5.6. Still, such adjustments are simple to perform because our adaptation process resolves any
discontinuities such as the gap between the takeoff stage and the remaining motion.
   In Figure 3(g), a new environment constraint replaces the ground with a high bar. In the new “landing”
stage, the feet are unconstrained and the constraints maintain the contact between the hands and the
high bar. As a result, the character jumps, reaches for the bar, and swings while holding onto it. To
initialize the optimization, we only use the first half of the original jump, from the start to the flight
apex. The last frame of this sequence was extended, as a still frame, for the duration of the swinging
stage. The adaptation adjusts the entire motion to produce a physically consistent motion that satisfies
new environment constraints.
   Both one-legged hops shown in Figure 3(h) and 3(i) were produced with adaptations that maintain
a fixed angle in the right knee. The first adaptation mimics the broad jump to create a character
that absorbs most of the impact with torques applied at the hip and knee joints. The second adaptation
produces a more typical one-legged landing by enforcing additional constraints that restrict the torques
applied at these two joints. As a result, the character avoids excessive strain by landing with an extended
leg.

5.4   Runs
The original run is shown in Figure 3(j). Our adaptation produces a hurdle jump shown in Figure 3(k) by
meeting the specified constraints: increased distance between the ground contacts and height required
to clear the hurdle. New distances are enforced by displacing all footprints. This generates bouncier
runs with longer strides. The hurdle constraint specifies a lower bound on the height attained by the
left and right toe. The dunk motion in Figure 3(l) is similar to the high-bar example. A new environment
constraint enforces the contact with the rim. Just as in the high-bar example, the last frame of the run
cycle was extended, as a still frame, for the duration of the swing stage. We’ve also modified the skeleton
by shortening its legs. The adaptation process compensates by producing a more explosive motion. Cone
constraints were added to prevent interpenetration of limbs. In Figure 3(m), the adaptation produces
a cross-step run with the proper step-to-step lean by satisfying the displaced footprints.

5.5   Comparison
We compared the efficiency of our adaptation method with the optimal trajectory method that employed
sequential quadratic programming (SQP) [Gill et al. 2002]. SQP is an iterative descent technique that
computes the optimal trajectory by minimizing the merit function along search directions given by
solutions to quadratic programming subproblems. The merit function balances the competing goals of
improving the objective function and remaining on the nonlinear constraint surface, while the quadratic
                                                             ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
176      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´




Fig. 4. The timing chart compares the efficiency of our adaptation method—with and without reduced dynamics—against the
best performing optimal trajectory method that maximized the smoothness of muscle actuation. The corpus of problems includes
all adaptations on which all three formulations were applicable. The adaptations were timed on a single-processor 1.7 GHz
Pentium III workstation with 1 GB of main memory. In general, the visual quality of each adaptation was similar, but on
occasion, the adaptation with reduced dynamics produced jerky motions.


subproblems linearize constraints and approximate the merit function with a quadratic expansion
around the current iterate. Just as with our gradient method, proper scaling and initialization ensured
that the SQP method solves adaptation problems for a full human figure.
  The SQP method uses an objective function to identify a unique motion that satisfies specified con-
straints. Prior literature proposed several objective functions with the aim of selecting motions that
                                                                                         ˙
appear most natural. The synthesis applications optimized power consumption ( f· θ 2 dt) [Witkin and
                                  2
Kass 1988], torque output ( f dt) [Liu et al. 1994], torque smoothness ( f     ¨ 2 dt) [Popovi´ and Witkin
                                                                                              c
                                        ¨
1999], and kinematic smoothness ( θ 2 dt) [Fang and Pollard 2003], while the adaptation applications
                                                                                              c
minimized either joint-angle displacement [Gleicher 1997], or mass displacement [Popovi´ and Witkin
1999] from the original motion.
  In our experiments with motions from Figures 4 and 3, optimization of torque smoothness consistently
generated better results than any other objective function. The power-consumption objective, which
selects the most efficient motion, produced natural motions on some adaptation problems, but performed
poorly in many cases, especially during adaptation of running motions. The most efficient motions
frequently lead to interpenetration of limbs (between the arms and the torso or the two legs), which
could not be resolved with joint-limit constraints. Optimization of torque output demonstrated identical
problems and also produced jerky motions unless used in combination with the kinematic-smoothness
objective. Similarly, the mass displacement objective, a kinematic analogue of the power consumption,
had to be applied in conjunction with kinematic smoothness to prevent the human figure from jumping
with a still body, and using only its ankles. We opted not to combine multiple objective functions
because the quality of the resulting motion became highly dependent on weights assigned to each
component in the objective function. Optimization of kinematic smoothness alone produced motions
that are smooth, but not natural. For example, during the jump takeoff, the body curled up into an
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                         Adaptation of Performed Ballistic Motion           •     177

unnatural position from which it would smoothly, but unnaturally, unwind to gain the appropriate
momentum.
   Figure 4 compares our adaptation technique (full and reduced dynamics) with the best performing
optimal formulation that maximizes the smoothness of internal torques. Note that proper scaling of
optimization variables permits efficient convergence of the optimal formulation even for the skeleton
with many degrees of freedom. With reduced dynamics, our adaptation technique converges faster
than the optimal formulation, by an order of magnitude on almost all problems. This improvement
is smaller with the full dynamics, which suggests that adaptation with reduced and full dynamics is
complementary. Once the desired constraints are met in rapid prototyping with the reduced dynamics,
the full dynamics can be used to generate smoother motions.

5.6   Limitations
The extent of any adaptation has its limits. If the difference between the initial and the final motion is
very large, motion adaptation will not be able to converge to a natural motion. For example, although
we could produce a jump with a half-twist directly from the original motion, the jump with a full-
twist required that we manually rotate the takeoff and the landing stages. Our technique, however,
eliminates the most tedious work. It removes the discontinuities between the flight and the transformed
ground stages, and adjusts them to add the required anticipation. The success of this simple approach
suggests that a combination of sequencing methods (e.g., Kovar et al. [2002]) and motion adaptation
could further extend the value of the original motion.
   Optimization is sensitive to discretization of joint-angle and torque trajectories. If too few control
points are chosen for the cubic B-spline approximation, the optimization might not converge because it
cannot satisfy the dynamic constraints. On the other hand, with too many control points, the solution
will satisfy the dynamics at prescribed time points, but will not look physically correct in between.
Often, trial-and-error is the best way to determine the settings. Table I shows the settings used for the
adaptation of jumps and runs.


6.    CONCLUSION
Motion adaptation with dynamics simplifies the generation of very complex human motions. An ani-
mator need specify only a sparse set of keyframes to define the goals for the motion, and the adaptation
will generate a new natural motion automatically. This method, like previously published optimal tra-
jectory techniques, benefits from restrictions imposed by the dynamics of ballistic motion—the same
restrictions that make manual adjustments particularly tedious. Although adaptation in the previously
published work relies on the simplification of dynamics constraints, this article demonstrates that such
simplifications are not necessary. This generalization allows for the specification of torque-based ob-
jective functions and torque-based constraints and broadens the set of motions that can be generated
with an adaptation approach.
  The adaptation problem provides a common ground for evaluation and comparison of optimal trajec-
tory methods. In this article, we define a numerical solution that adapts a motion with a biased search
for trajectories that satisfy the adaptation goals. The search favors the motions that are close to the
original performance to preserve the distinctive attributes of the original performance. Our method is
compared to an explicit optimization of muscle actuations on an identical set of adaptation problems.
The results show that our method efficiently produces visually indistinguishable solutions. In the fu-
ture, we would like to improve the robustness with automated selection of collocation points, and to
explore techniques that use sequencing methods (e.g., Kovar et al. [2002]) to construct a better initial
guess for each adaptation problem.
                                                            ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
178      •                   sc                c
                A. Sulejmanpaˇ i´ and J. Popovi´

REFERENCES
                               ´
ABE, Y., LIU, C. K., AND POPOVIC, Z. 2004. Momentum-based parameterization of dynamic character motion. In Proceedings of
  the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. At press.
BARAFF, D. 1996. Linear-time dynamics using lagrange multipliers. In Proceedings of SIGGRAPH 96. Computer Graphics
  Proceedings, Annual Conference Series. 137–146.
BRUDERLIN, A. AND WILLIAMS, L. 1995. Motion signal processing. In Proceedings of SIGGRAPH 95. Computer Graphics Pro-
  ceedings, Annual Conference Series. 97–104.
COHEN, M. F. 1992. Interactive spacetime control for animation. In Comput. Graph. (Proceedings of SIGGRAPH 92). Annual
  Conference Series. ACM, 293–302.
DEMMEL, J. W. AND HIGHAM, N. J. 1993. Improved error bounds for underdetermined system solvers. SIAM J. Matrix Anal.
  Applicat. 14, 1–14.
FANG, A. C. AND POLLARD, N. S. 2003. Efficient synthesis of physically valid human motion. ACM Trans. Graph. 22, 3 (July),
  417–426.
GILL, P. E., MURRAY, W., AND SAUNDERS, M. A. 1997. User’s guide for SQOPT 5.3: A fortran package for large-scale linear and
  quadratic programming. Tech. Rep. NA 97-4, University of California, San Diego.
GILL, P. E., MURRAY, W., AND SAUNDERS, M. A. 2002. SNOPT: An SQP algorithm for large-scale constrained optimization. SIAM
  J. Optimiz. 12, 4, 979–1006.
GILL, P. E., MURRAY, W., AND WRIGHT, M. H. 1989. Practical Optimization. Academic Press, London.
GLEICHER, M. 1997. Motion editing with spacetime constraints. In Symposium on Interactive 3D Graphics. 139–148.
GLEICHER, M. 1998. Retargeting motion to new characters. In Proceedings of SIGGRAPH 98. Computer Graphics Proceedings,
  Annual Conference Series. 33–42.
GLEICHER, M. 2004. Personal communication.
HOLLERBACH, J. M. 1980. A recursive lagrangian formulation of manipulator dynamics and a comparative study of dynamics
  formulation complexity. IEEE Trans. Syst., Man, Cybernet. 10, 11 (Nov.), 730–736.
KOVAR, L., GLEICHER, M., AND PIGHIN, F. 2002. Motion graphs. In Comput. Graph. (Proceedings of SIGGRAPH 2002). Annual
  Conference Series. ACM SIGGRAPH, 473–482.
LEE, J. AND SHIN, S. Y. 1999. A hierarchical approach to interactive motion editing for human-like figures. In Proceedings of
  SIGGRAPH 99. Computer Graphics Proceedings, Annual Conference Series. 39–48.
                        ´
LIU, C. K. AND POPOVIC, Z. 2002.       Synthesis of complex dynamic character motion from simple animations. ACM Trans.
  Graph. 21, 3 (July), 408–416.
LIU, Z. 1996.       Efficient animation techniques balancing both user control and physical realism. Ph.D. thesis, Princeton
  University.
LIU, Z., GORTLER, S. J., AND COHEN, M. F. 1994. Hierarchical spacetime control. In Proceedings of SIGGRAPH, 94. Computer
  Graphics Proceedings, Annual Conference Series. 35–42.
PANDY, M. G. AND ANDERSON, F. C. 2000. Dynamic simulation of human movement using large-scale models of the body. In
  Proceedings of the IEEE International Conference on Robotics and Automation. 676–681.
        ´
POPOVIC, Z. AND WITKIN, A. P. 1999. Physically based motion transformation. In Proceedings of SIGGRAPH 99. Computer
  Graphics Proceedings, Annual Conference Series. 11–20.
ROSE, C. F., GUENTER, B., BODENHEIMER, B., AND COHEN, M. F. 1996. Efficient generation of motion transitions using spacetime
  constraints. In Proceedings of SIGGRAPH 96, Computer Graphics. Annual Conference Series. 147–154.
SAFONOVA, A., HODGINS, J., AND POLLARD, N. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior-
  specific spaces. ACM Trans. Graph. 23, 3 (Aug.), 514–521.
SHIN, H. J., KOVAR, L., AND GLEICHER, M. 2003. Physical touchup of human motions. In Proceedings 11th Pacific Conference on
  Computer Graphics and Applications. 194–203.
             ˇ ´
SULEJMANPASIC, A. 2004. Adaptation of performed ballistic motion. M.S. thesis, Massachusetts Institution of Technology.
TAK, S., YOUNG SONG, O., AND KO, H.-S. 2002. Spacetime sweeping: An interactive dynamic constraints solver. In Proceedings
  of the Computer Animation (CA 2002). IEEE Computer Society, 261–270.
VICON. 2003. Vicon iQ Reference Manual. Vicon Motion Systems Inc., Lake Forest, CA.
WINTER, D. A. 1990. Biomechanics and Motor Control of Human Movement, 2nd Ed. John Wiley and Sons, Inc., New York.
WITKIN, A. AND KASS, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), Vol. 22. 159–168.
                       ´
WITKIN, A. AND POPOVIC, Z. 1995. Motion warping. In Proceedings of SIGGRAPH 95. Computer Graphics Proceedings, Annual
  Conference Series. 105–108.
ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.
                                                                Adaptation of Performed Ballistic Motion               •     179

YAMANE, K. AND NAKAMURA, Y. 2000. Dynamics filter—concept and implementation of on-line motion generator for human
  figures. In Proceedings of the IEEE International Conference on Robotics and Automation. 688–694.
ZORDAN, V. B. AND HODGINS, J. K. 1999. Tracking and modifying upper-body human motion data with dynamic simulation. In
  Computer Animation and Simulation ’99.


Recieved March 2004; revised September 2004; accepted September 2004




                                                                       ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.

				
DOCUMENT INFO
Shared By:
Stats:
views:18
posted:8/23/2011
language:English
pages:179
Description: Feature-Based Surface Parameterization and Texture Mapping