Document Sample

Feature-Based Surface Parameterization and Texture Mapping EUGENE ZHANG, KONSTANTIN MISCHAIKOW, and GREG TURK Georgia Institute of Technology Surface parameterization is necessary for many graphics tasks: texture-preserving simpliﬁcation, remeshing, surface painting, and precomputation of solid textures. The stretch caused by a given parameterization determines the sampling rate on the surface. In this article, we present an automatic parameterization method for segmenting a surface into patches that are then ﬂattened with little stretch. Many objects consist of regions of relatively simple shapes, each of which has a natural parameterization. Based on this observation, we describe a three-stage feature-based patch creation method for manifold surfaces. The ﬁrst two stages, genus reduction and feature identiﬁcation, are performed with the help of distance-based surface functions. In the last stage, we create one or two patches for each feature region based on a covariance matrix of the feature’s surface points. To reduce stretch during patch unfolding, we notice that stretch is a 2 × 2 tensor, which in ideal situations is the identity. Therefore, we use the Green-Lagrange tensor to measure and to guide the optimization process. Furthermore, we allow the boundary vertices of a patch to be optimized by adding scaffold triangles. We demonstrate our feature-based patch creation and patch unfolding methods for several textured models. Finally, to evaluate the quality of a given parameterization, we describe an image-based error measure that takes into account stretch, seams, smoothness, packing efﬁciency, and surface visibility. Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling—Geometric algorithms, languages, and systems General Terms: Algorithms Additional Key Words and Phrases: Surface parameterization, segmentation, texture mapping, topology 1. INTRODUCTION Surface parameterization is a well-studied problem in computer graphics. In general, surface parame- terization refers to segmenting a 3D surface into one or more patches and unfolding them onto a plane without any overlap. Borrowing terminology from mathematics, this is often referred to as creating an atlas of charts for a given surface. Surface parameterization is necessary for many graphics applica- tions in which properties of a 3D surface (colors, normal) are sampled and stored in a texture map. The quality of the parameterization greatly affects the quality of subsequent applications. One of the most This work is supported by NSF grants ACI-0083836, DMS-0138420, and DMS-0107396. Authors’ addresses: E. Zhang, School of Electrical Engineering and Computer Science, Oregon State University, 102 Dearborn Hall, Corvallis, OR 97331-3202; email: Zhange@cc.gatech.edu; K. Mischaikow, Center for Dynamical Systems and Nonlinear Studies, School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332; email: mischaik@math.gatech.edu; G. Turk, College of Computing/GVU Center, Georgia Institute of Technology, Atlanta, GA 30332; email:turk@cc.gatech.edu. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0001 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 1–27. 2 • E. Zhang et al. Fig. 1. The feature regions (left) and the unfolded patches (right, colors are used to encode surface normal) for the bunny surface using our algorithm. important quality measurements is stretch. When unfolding a surface onto a plane, stretching occurs if the surface contains highly spherical or hyperbolic regions. High stretch in a parameterization results in an uneven sampling rate across the surface. We observe that many objects can be decomposed into a set of “simple” shapes that roughly approx- imate cylinders, cones, ﬂat disks, and spheres. Cylinders, cones, and planes are developable surfaces, which are Euclidean by nature. Unfolding them results in little stretch, without any overlap. In this article, we make use of some distance-based surface functions to divide a manifold surface into feature regions, each of which is similar to one of the simple shapes. In Figure 1 (left), the bunny surface is decomposed into four feature regions (ears, head, and body) using our segmentation algorithm. These regions are converted into patches and unfolded with little stretch (right, colors are used to encode surface normal). Existing patch unfolding techniques are often carried out in two stages: an initial patch layout to achieve some objective such as conformal mapping, followed by an interior vertex optimization based on some stretch metric. We observe that an ideal surface parameterization between a patch and its textural image is an isometry, that is, a bijective map that preserves distances. The Green-Lagrange deformation tensor has the property that it measures anisotropic stretch faithfully and penalizes undersampling more severely than oversampling. In addition, it can be seen as a balance between area-preserving mappings and conformal mappings. We use this metric to guide the vertex optimization process for patch unfolding. In addition, we use what we call scaffold triangles to convert the original boundary vertices into “interior” vertices, which can then be freely moved around within the same optimization framework. This is a new way of creating nonconvex patches that may even have holes. In this article, we present an automatic surface parameterization technique that consists of several new ideas and improves upon existing techniques in the following aspects. For patch creation, instead of relying on local curvature information for feature detection as in the case of most previous param- eterization methods, we extract and segment large protrusions based on the topological analysis of some distance-based surface functions that are global in nature. This results in a small number of large patches that can be unfolded with relatively little stretch. For patch unfolding, we use the Green- Lagrange tensor to measure stretch and to guide the stretch optimization process. In addition, we create a “virtual boundary” to allow the patch boundaries to be optimized, without the need to check for global self-intersections. Finally, we describe a novel image-based quality metric for surface parameterization that implicitly takes into account stretch, seams, packing efﬁciency, smoothness, and surface visibility. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 3 The remainder of the article is organized as follows. In Section 2, we review existing surface pa- rameterization techniques. Then, we present our feature-based patch creation method in Section 3, followed by our new balanced stretch metric in Section 4.1, boundary vertex optimization technique in Section 4.2, and our packing algorithm in Section 5. In Section 6, we show the results of applying our technique to various 3D models and describe our image-based quality metric for parameterization techniques. Section 7 provides a summary of our contributions and a discussion of some possible future work. 2. PREVIOUS WORK There has been a considerable amount of recent work in the graphics community on building a surface parameterization by unfolding a polygonal surface into planar patches. Much of the motivation for this is for texture mapping, the mapping of pixels from a rectangular domain (the texture map) onto a surface that is described by a collection of polygons. The surface parameterization problem is to subdivide the given surface into a (hopefully small) number of patches that are then ﬂattened onto a plane and arranged in a texture map. Uses for surface parameterization include surface painting [Hanrahan and Haeberli 1990], fast rendering of procedural textures [Perlin 1985; Turk 2001; Wei and Levoy 2001; Carr and Hart 2002], applying photographed color variations onto digitized surfaces [Cignoni et al. 1998], and creating normal maps from detailed geometry [Sander et al. 2001]. These same parameterization methods may also be used for remeshing, that is, for creating a new mesh from the original surface [Alliez et al. 2002]. Remeshing can be used to improve the triangle shapes, to vary the triangle size according to curvature details, and to induce semi-regular tessellations. Recently, octrees have been used to store colors in 3D for surface texturing without any parameterization [Benson and Davis 2002; DeBry et al. 2002]. Although octree techniques are supported with programmable GPU’s, they are not yet directly supported by graphics hardware. 2.1 Patch Creation There are two common approaches to the patch creation problem. The ﬁrst of these is to ﬁnd a single cut for the surface that makes the modiﬁed surface topologically equivalent to a disk [Piponi and Borshukov 2000; Gu et al. 2002; Sheffer and Hart 2002; Erickson and Har-Peled 2002; Ni et al. 2004]. This approach has the virtue of creating as few seams as possible, but will often introduce large stretch between the patch and the surface. Such stretching is undesirable because different portions of the surface are represented using quite different amounts of color detail, as measured in pixel resolution in the texture map. The other major approach is to divide the surface into a collection of patches that can be unfolded e with little stretch [Eck et al. 1995; Lee et al. 1998; Sander et al. 2001; Alliez et al. 2002; L´ vy et al. 2002; Sorkine et al. 2002]. Though stretch is minimized, this approach creates seams between the patches. These seams cause problems when creating textured images of the surface because the color variation across the seams must be treated with extreme care or the seams will be noticeable. Some methods create small disk-like patches [Eck et al. 1995; Lee et al. 1998; Sander et al. 2001; Alliez et al. 2002], e while others attempt to create large patches that match the features contained in the object [L´ vy et al. 2002; Sorkine et al. 2002; Katz and Tal 2003]. Our own work takes this latter approach. We cut the surface into multiple patches, but according to the large geometric features of the surface. For example, we would like to recognize the head and limbs of an animal as important features and to create patches e that respect these features. The work of L´ vy et al. [2002] and Katz and Tal [2003] have similar goals, although their feature-based patch creation methods are quite different than our own. The deﬁnition of the term “geometric feature” varies in different contexts. For surface reconstruction and mesh simpliﬁcation, features are often deﬁned in terms of local curvature. This is reasonable ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 4 • E. Zhang et al. because high curvature regions are exactly what these applications are trying to preserve. On the other hand, surface parameterization algorithms incur higher stretch on a smooth surface with long thin protrusions than a noisy surface with small protrusions. In this work, we deﬁne geometric features as large protrusions, and our algorithm segments a surface based on its features by performing topological analysis of some distance-based surface functions. 2.2 Patch Unfolding There have been many patch unfolding techniques. The classical approach treats the patch unfolding problem as ﬁnding the minimum of some functional that measures the difference between a parameter- ization with isometry [Eck et al. 1995; Floater 1997]. First, the boundary vertices are assigned initial positions (usually on a circle or a square). Then the parameterization for the interior vertices is deter- mined by solving a large linear system or through a nonlinear optimization process. Others have used stretch measures such as the Green-Lagrange deformation tensor [Maillot et al. 1993] and a variant of Dirichlet energy [Hormann and Greiner 1999]. Sander et al. [2001] deﬁne a geometric stretch metric that is based on the average and maximal stretch in all directions of a triangle. Sorkine et al. [2002] and Khodakovsky et al. [2003] have devised stretch metrics based on the maximum and minimum eigen- values of the stretch tensor. Sander et al. [2001] also propose a post-processing vertex optimization step that improves their geometric stretch. As we describe later, Sander’s patch optimization approach was an inspiration for our own work. To allow the boundary vertices of a patch to be free from the arbitrary e initial assignment, L´ vy et al. [2002] use a least-squares conformal mapping, and Desbrun et al. [2002] propose an equivalent formulation, Discrete, Natural Conformal Parameterization. Sander et al. [2002] allow boundary vertices to move, while checking for global intersections. Lee et al. [2002] add layers of “virtual boundaries” as part of the edge springs to allow the patch boundaries to have natural shapes. Recently, Sheffer and deSturler [2001; 2002] propose to use an angle-based ﬂattening approach for patch unfolding. This approach measures stretch in term of the angles deﬁcits between the triangles on the surface and their textural images, and it removes the need to check for global self-intersections. 3. FEATURE-BASED PATCH CREATION Our feature-based patch creation method is carried out in three stages: (1) genus reduction: a surface with handles (nonzero genus) is converted into a genus zero surface. (2) feature identiﬁcation: a genus zero surface is divided into a number of relatively simple shapes. (3) patch creation: every simple shape is cut into one or two topological disks. For both genus reduction and feature identiﬁcation, we build a surface-based Reeb graph based on the average geodesic distance introduced by Hilaga et al. [2001]. This graph consists of vertices and edges in the mesh surface, and we call it an embedded Reeb graph. When properly constructed, this graph reveals the location of the handles and protrusions in the surface. Since our goal is to create patches that are topological disks, we need to perform our operations in a topologically consistent manner. For this purpose, we use surface region growing for all three stages: genus reduction, feature identiﬁcation and patch creation. Starting from an initial triangle, we grow a region by adding one triangle at a time until the whole surface has been covered or until some other stopping criterion has been met. We will now describe the average geodesic distance function and the embedded Reeb graph that it induces. 3.1 The Average Geodesic Distance Function The average geodesic distance function was introduced by Hilaga et al. [2001] for the purpose of shape matching. This is a function A(p) that takes on a scalar value at each point p on the surface S. Let g (p, q) be the geodesic distance between two points p and q on S. Then the average geodesic distance ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 5 Fig. 2. The average geodesic distance function (AGD) on the dinosaur model is color-coded in the left of this ﬁgure. The global minimum is located underneath the belly, colored in red. Levelsets are painted in repeated patterns of red, green, and blue. Notice that the tips of large protrusions (horns, legs, tail) are local maxima of AGD. The middle, ﬁgure shows the embedded Reeb graph created by surface-growing based on AGD. Local maxima are indicated by red spheres, and saddle points are highlighted by blue spheres. Successive critical points are connected by surface paths shown in solid yellow (visible) and dash green (hidden). The ﬁnal surface segmentation result based on our algorithm is shown on the right. Fig. 3. Comparison among three AGD functions for the dragon: AGD1 (left), AGD2 (middle), and AGD∞ (right). of p is deﬁned as follows: q∈S g (p, q)d q A(p) = . (1) Area(S) A(p) is a member of the following set of functions: n q∈S g n (p, q)d q An (p) = (2) Area(S) with n = 1. When n → ∞, A∞ (p) := limn→∞ An (p) = maxq∈S g (p, q), which measures the maximal distance between p and any point on S. We deﬁne for n = 1, 2, . . . , ∞ An (p) AG Dn (p) := . (3) minq∈S An (q) For any n ≥ 1, AG Dn has several useful properties. First, its value measures how “isolated” a point is from the rest of the surface. Second, its local maxima coincide with the tips of the geometric features contained in the model. Third, it is scale-invariant and can be used to compare features from different shapes. Figure 2 (left) shows a polygonal model of a dinosaur, color-coded according to AGD2 . The red region on the dinosaur’s belly signiﬁes that points in this region have low values of AGD2 . Higher values adjacent to this middle region are colored in green and then in blue. The colors then cycle repeatedly through red, green, and blue. Note that the tips of the large features of this object (legs, horns, tail) are marked by local maxima of AGD2 . (In subsequent sections we will use the term local maxima of AGD and tips interchangeably.) In practice, we use AGD2 since it seems to produce smooth results. Figure 3 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 6 • E. Zhang et al. compares the levelsets of the following three functions on the dragon: AGD1 (left), AGD2 (middle), and AGD∞ (right). From now on, we will use the term AGD to mean AGD2 . For a genus zero surface, we use AGD to identify and measure its geometric features. The tip of a protrusion is a local maximum. Larger values at local maxima signify larger protrusions. Notice that a noisy surface often contains many small bumps that correspond to local maxima with relatively small AGD values. Creating patches based on these bumps will increase the amount of seams, without signiﬁcantly reducing stretch. Therefore, we only consider local maxima whose AGD values are above a threshold. In practice, we ﬁnd choosing any number in [1.3, 1.5] as the threshold for minimal feature size produces reasonable results, and we use 1.4 for all our test models. Computing AGD exactly would be quite costly. We closely follow the algorithm of Hilaga et al. [2001] to quickly compute a satisfactory approximation of AGD. Brieﬂy, the geodesic distances are not calculated from all the surface points, but rather from a small number of evenly spaced points on the surface. We ﬁnd the geodesic distances from each of these points to all other points efﬁciently using the fast- marching method for surfaces [Kimmel and Sethian 1998]. 3.2 Building an Embedded Reeb Graph To ﬁnd handles and large protrusions in a model, we perform topological analysis of AGD and construct an embedded Reeb graph that is induced by AGD. The leaf nodes of are situated at the tips of protru- sions, and the loops in reveal the existence of handles. We construct by performing region-growing in the increasing order of AGD and tracking the topological changes in the wavefront. This is based upon ideas from Morse Theory [Milnor 1963] and Reeb graphs [Reeb 1946], which we will review here. Let f be a smooth function deﬁned on a smooth surface S ⊂ R3 . For any point p0 ∈ S, let µ = (u, v) be a parameterization of some neighborhood of p0 in S such that µ(0, 0) = p0 . The gradient ∇ f and the Hessian H f are deﬁned as follows: 2 ∂ f /∂u ∂ f /∂u2 ∂ 2 f /∂u∂v ∇f = , Hf = . (4) ∂ f /∂v ∂ f /∂u∂v ∂ f /∂v 2 2 2 p0 is a critical point of f if ∇ f (0, 0) = 0. Otherwise, p0 is regular. A critical point p0 is said to be nondegenerate if H f (0, 0) does not have any zero eigenvalues. In this case, p0 can be classiﬁed as a mimimum/saddle/maximum if H f (0, 0) has zero/one/two negative eigenvalues. f is Morse over S if f possesses no degenerate critical points. Morse theory relates the critical points of Morse functions to the topology of the underlying surface. For instance, when S is a closed orientable 2-manifold with Euler characteristic χ(S) (twice the number of handles plus two), the following is true for any Morse function f deﬁned on S with α maxima, β saddles, and γ minima. α − β + γ = χ (S). (5) Banchoff extends Morse theory to triangular meshes [1970]. A continuous function f : S → R induces a Reeb graph f , which can be used to reveal the topological and skeletal structures of S [Hilaga et al. 2001]. Formally, we deﬁne an equivalence relationship ∼ f on S as follows. Let p, q ∈ S be two points, then p ∼ f q if and only if f (p) = f (q) and p and q belong to the same connected component of f −1 ( f (p)). Notice f does not have to be Morse. Figure 4 illustrates an example Reeb graph that corresponds to the vertical height function, deﬁned on a 3D surface. Many applications make use of Reeb graphs, such as shape-matching [Hilaga et al. 2001] and topological simpliﬁcation [Wood et al. 2004]. AGD is, in general, not Morse. For instance, it is a constant function on a sphere, in which case every point is a degenerate critical point. Axen and Edelsbrunner [1998] show that a function can be perturbed into a Morse function with surface wave traversal, provided ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 7 Fig. 4. An example of a Reeb graph (right) for a vertical height function, deﬁned on a 3D surface of genus one (left). The critical points are highlighted by colored spheres (red for maxima, green for minima, and blue for saddles). The number of loops in the graph equals the number of handles in the surface. that the mesh is properly subdivided. We use a similar strategy except that we record critical triangles instead of critical vertices. Our algorithm for building an embedded Reeb graph starts with computing AGD for every vertex. For a triangle T = {v1 , v2 , v3 }, we deﬁne AGD(T ) = min{AG D(v1 ), AG D(v2 ), AG D(v3 )}. Starting with a triangle whose AGD value equals the global minimum, we add one triangle at a time in the increasing order of the AGD until the surface is covered. The boundary of the visited region consists of a number of loops. We label a triangle, when it is added, according to one of the following ﬁve criteria: (1) Minimum: where one new boundary loop starts. For our application, there is only one such triangle, one of the global minima of the AGD. (2) Maximum: where one boundary loop vanishes. This is the tip of a protrusion. (3) Splitting saddle: where one boundary loop intersects itself and splits into two. (4) Merging saddle: where two boundary loops intersect and merge into one. This signiﬁes the formation of a handle. (5) Regular: where the number of boundary loops does not change. A triangle that is not regular is a critical triangle. Let n be the genus of the surface, and let Nmax , Nmin , Nss , and Nms be the number of the triangles that are maxima, minima, splitting saddles, and merging saddles. Then we have, Nms = n (6) Nmax − Nss + Nms + Nmin = 2. (7) Equation 7 corresponds to the handle-body decomposition of a closed and orientable piecewise linear 2-manifold [Rourke and Sanderson 1972]. Interested readers may refer to Lopes et al. [2003] for more details. For our application, Nmin = 1. Furthermore, we mark the center of a critical triangle as the position of the corresponding critical point. The region on the surface swept out between a pair of critical triangles (not including these critical triangles) is homeomorphic to a cylinder without caps. Let A and B be a pair of critical triangles, and assume that A is visited earlier than B. We refer to A as the parent critical triangle and B as the child critical triangle. For a genus zero surface, every child critical triangle has a single parent. For surfaces with a genus greater than zero, a child critical triangle may have one or two parents. Let RAB be the connecting region between A and B, which consists of a set of regular triangles = {T1 , . . . , Tk } in the order of which they are visited. There is a shortest path that connects A and B using the edges of the set of triangles {A} {B} RAB . We construct the embedded Reeb graph by ﬁnding the shortest paths between every pair of parent/child critical triangles. As mentioned earlier, the embedded Reeb graph is much like a Reeb graph that corresponds to AGD. It reveals the distribution of the geometric features over the surface. The middle of Figure 2 shows the embedded Reeb graph of the dinosaur. Local maxima are highlighted with red spheres, while ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 8 • E. Zhang et al. Fig. 5. Embedded Reeb graphs for the bunny surface with different ﬁltering constants α: 1.01 (left), 1.1 (middle), and 1.5 (right). We use 1.1 as the ﬁltering constant for all the test models. blue spheres indicate the location of splitting saddle points. The global minimum is marked with a light blue sphere on the belly. Successive critical points are connected by paths on the surface, which are drawn in solid yellow (visible) and dash green (hidden). Note that local maxima coincide with the tips of the geometric features (horns, feet, and tail). Since complex surfaces often contain many small protrusions (bumps), the embedded Reeb graph can contain an excessive number of local maxima and saddle points. This increases the subsequent processing time since the number of features is much more than what we consider large (or “persistent” as described in Edelsbrunner et al. [2003]). We use the following ﬁltering scheme to weed out extra local maxima and splitting saddle points. During surface-growing, we alter the order in which triangles are added. To be more speciﬁc, let t be the unvisited triangle with the smallest AGD value. If adding t causes a boundary to split, we look for other triangles that could be added without causing a boundary split. If one of these triangles, t satisﬁes: AGD(t ) < αAGD(t) (8) where α is a global ﬁltering constant, then we add t instead of t. When there are multiple choices, we choose the triangle with the smallest AGD value. Our ﬁltering process is related to the concept of topological persistence and simpliﬁcation [Edelsbrunner et al. 2003], but with a different scalar function and a different measure for persistence. Also, the simpliﬁcation process is implicit. We apply the ﬁltering scheme to the bunny surface (Figure 5) with three different α’s: 1.01 (left), 1.1 (middle), 1.5 (right). Notice the excessive saddle points and maxima appear in the head and the paws when α = 1.01 (left). When α = 1.1 (middle), the local maxima that reveal large geometric structures are kept (the tips of ears and the center of tail). Excessive ﬁltering may result in a trivial embedded Reeb graph, for example, α = 1.5 (right). This becomes a classical tradeoff between denoising and overblurring. In practice, we ﬁnd α ∈ [1.1, 1.3] works well, and we use α = 1.1 for all the test models shown in the article. For a genus n > 0 surface, there are n loops in the embedded Reeb graph that are homologically inequivalent and form the bases of all loops in . In Section 3.4, we describe how we use these loops for genus reduction, that is, converting a genus n surface into a genus zero surface. 3.3 Feature Identiﬁcation Once the tip of a protrusion is located, we construct a closed curve γ on the surface that separates the feature from the remaining body. Using the terminology from Erickson and Har-Peled [2002], γ is a separating cycle. We compute γ in two steps. First, we ﬁnd a separating region R corresponding to the tip of the protrusion. Next, we construct γ from R. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 9 Fig. 6. In this ﬁgure, we ﬁnd a separating region for the bunny’s ear. In the left we graph A(x), the area of the regions given by evenly spaced distances from the ear’s tip (red), and the smoothed areas (blue). We then calculate A (x), the second derivative (middle). The maximum of A (x) corresponds to the place where the ear joins the head (right, the green band). To ﬁnd a separating region for the tip point p of a feature, we ﬁrst calculate the function f p (q) = g (p, q), the surface geodesic distance function with respect to p. f p is normalized to take on values in [0, 1]. We consider the regions bounded by iso-value curves of this function. Speciﬁcally, we divide the interval [0, 1] into k equal sections. Next, by performing region-growing from p, we partition the surface into levelset bands based on the values of f p in these intervals: i−1 i M i := q ∈ S| ≤ f p (q) ≤ (9) k k Ai := Area(M i ). (10) The construction of levelsets in Equation 9 is inspired by Morse theory. The area of this sequence of bands changes slowly along a protrusion, but it changes abruptly where the feature joins to the rest of the surface. We ﬁnd the separating region by analyzing {Ai }, which we treat as a continuous function A(x). Along a perfect cylindrical feature, A(x) is constant. In the case of a cone, the function grows linearly. At places where a protrusion joins the main body, A(x) will have a sudden increase, and this will be the boundary of the feature. We ﬁnd these increases by looking for the maxima in A (x), the second derivative of A(x). To eliminate small undulations in A(x), we ﬁrst low-pass ﬁlter A(x) using a Gaussian function for N times. Both k and N affect the efﬁciency of separating region detection. The larger k is, the more samples are used to discretize A(x), and the more likely small noise will be considered as potential places for the separating region. Similarly, if N is too large, the location for the separating region may be shifted or even lost. In practice, we use k = 100 and N = 30 for all our test models. These choices seem to produce reasonable results. Figure 6 illustrates the process of feature identiﬁcation for the bunny’s ear. Let m be the location where A (m) takes on its maximum value. We deﬁne the separating region R := {q ∈ S | m − ≤ f p (q) ≤ m + }, where we typically use = 0.02. By choosing a positive , we intentionally make R large for two reasons. First, poor triangulations may cause R to be nonseparating, that is, it does not contain a separating cycle. Second, we would like more ﬂexibility to allow the separating cycle (within this region) to be short and smooth. The topology of the separating region R can be rather complex if there are other features that join the surface in nearby places. The only guarantee R provides is that it indeed separates the feature ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 10 • E. Zhang et al. from the rest of the surface. We produce a separating cycle γ from R as follows. First, we reduce R into its skeleton, that is, a collection of edges in the surface that separate the feature from the rest of the surface. Dangling edges are removed as well. Gu et al. [2002] perform a similar operation to produce geometry images from meshes. Next, we ﬁnd a separating cycle ρ from this skeleton. Finally, we construct another separating cycle γ that is based on ρ, but that is in general shorter and smoother. These operations are easy to implement on meshes and we describe them in detail next. (1) Reduce a separating region R into its skeleton and remove dangling edges. This is achieved by treating R as a 2-complex (with boundary edges) and repeatedly performing “elementary col- lapses” [Kaczynski et al. 2004], in which a triangle with at least one boundary edge is removed from the complex along with one of the boundary edges. In the end, all 2-cells (triangles) are removed, and the 2-complex is reduced to a 1-complex. When there are multiple choices of boundary edges to collapse, we select the edge with the largest AGD value, which tends to be closer to the feature tip p than the other edges in the 2-complex. The resulting graph is a skeleton of R with dangling edges. We remove the dangling edges through elementary collapses on the 1-complex. This results in a collection of loops, one of which meets our requirement as the separating cycle. The others fall into two categories: separating cycles for some geometric features inside the feature region, and separating cycles for some geometric features outside the feature region. (2) Eliminate from the 1-complex separating cycles that are either inside or outside the feature region. To remove the loops outside the feature region, we perform region-growing from the feature tip p with the constraint that no triangles can be added that cross an edge in the loops computed from the steps in (1). This makes the loops outside the feature region unreachable from p. For loops inside the feature re- gion, the average AGD values of their vertices are, in general, greater than that on the separating cycle. Therefore, these loops can be easily identiﬁed and discarded. This step produces a separating cycle ρ. (3) Shorten and smooth the separating cycle ρ. We choose two vertices t1 and t2 on ρ that are the closest to the feature tip p. We ﬁnd two paths that connect t1 and t2 to p, respectively. The two paths divide the feature region into two disjoint regions. Within each region, there is a shortest path between t1 and t2 . Together, they form a separating cycle, which tends to be shorter and smoother than ρ. By repeating this process twice, we obtain a desired separating cycle γ . Figure 7 illustrates this process. For a feature point p and a separating region R (a, the shaded region), we reduce R to its skeleton through elementary collapses (b). Next, loops that are either inside or outside the feature region of p are eliminated (c). In the bottom row, the separating cycle ρ is shortened and smoothed to produce γ , through step 3 (d-f). A separating cycle divides S into two surfaces with boundaries. We eliminate these boundaries by “ﬁlling in” the holes with triangles. Basically, we compute c, the average position of the boundary vertices, and make c a new vertex. We then triangulate the hole by connecting c to the vertices on the boundary. The ﬁller triangles are subdivided twice, followed by Laplacian smoothing on the newly created vertices. Some ﬁller triangles can be seen where the head has been separated from the neck of the bunny in Figure 1. These ﬁller triangles are ﬂagged so that they have minimal effect on patch unfolding. They become what we call scaffold triangles, to be described later. We repeat the feature identiﬁcation process for the resulting surfaces until the original surface is divided into a set of feature regions and there are no more feature regions to be found. Figure 1 shows the result of this process on the bunny, in which four regions were created. Our feature identiﬁcation algorithm assumes that a single loop always divides the surface into two disjoint regions, which is not necessarily true for surfaces with handles. For these surfaces, the topology of the separating region can be arbitrarily complex where many small handles are clustered together. To avoid dealing with this situation, we perform genus reduction before feature identiﬁcation. Genus reduction converts a genus n > 0 surface into a genus zero surface. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 11 Fig. 7. This ﬁgure illustrates our algorithm for producing a separating cycle from a separating region. In (a), a separating region R (shaded) for p is bounded by the red-curves. Next in (b), R is reduced to its skeleton through elementary collapses. The skeleton consists of three loops: ρ1 , ρ2 , and ρ3 . Notice that ρ2 is not reachable from p, and ρ3 has a higher average AGD value than ρ1 . By eliminating them, we obtain a separating cycle ρ = ρ1 (c). To smooth ρ, we ﬁnd two points t1 and t2 in ρ and the shortest paths that connect t1 p and t2 p (green curves in (d)). Together the two curves divide the region bounded by ρ into two subregions. Inside each subregion, we construct a shortest path between t1 and t2 (the red curves in (e) and (f)). The union of the two paths forms a separating cycle γ that is, in general, shorter and smoother than ρ. 3.4 Genus Reduction For a genus n surface (n > 0), a loop does not always divide the surface into two disjoint connected components. Loops with this property are associated to the elements of the ﬁrst homology group, which form an Abelian group with 2n generators. Using the terminology from [Erickson and Har-Peled 2002], these loops are nonseparating cycles. Conceptually, the easiest way to think of how these loops arise is to imagine a hollow handle connected to the rest of the surface; one of the loops cuts across the handle and the other follows the handle. Observe that for the ﬁrst type of loops there are two “passages” back to the rest of the surface. Our strategy for genus reduction is to identify an appropriate nonseparating cycle for every handle and cut the surface open along the cycles. Each operation converts a handle into one or two protrusions and reduces the genus of the surface by one. We repeat this process until the surface contains no handles. Erickson and Har-Peled [2002] have proved that ﬁnding the minimal length cuts needed to turn a surface into a disk is NP-hard, so heuristics are used in practice to ﬁnd cuts that are short in length. Genus reduction may be performed using a number of already existing techniques, including Guskov and Wood [2001], Lazarus et al. [2001], Erickson and Har-Peled [2002], Gu et al. [2002], Sheffer and Hart [2002], and Wood et al. [2004]. We choose to perform genus reduction using the embedded Reeb graph and the same distance function that we use for feature identiﬁcation. Figure 8 shows our genus reduction algorithm on the dragon (genus one). ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 12 • E. Zhang et al. Fig. 8. This ﬁgure illustrates the process of genus reduction for the dragon. After the embedded Reeb graph is computed (a, the graph colored in yellow), we ﬁnd the independent nonseparating cycle contained in (b, yellow) and perform region- growing from this loop in both directions until the wavefronts meet (b, the blue and green regions). The meeting point is shown in (c). We ﬁnd a path within the green and blue regions that connect the meeting point to the original nonseparating cycle. The paths form a nonseparating cycle (c and d, the red loop), which can be used for turn a handle into one or two protrusions. First, we compute the embedded Reeb graph induced by AGD (a, the yellow graph) and locate all the basis loops in (b, the yellow loop). Recall (Section 3.2) that a merging saddle point qi signals that a handle has formed (a, the green sphere). The start of a handle is a splitting saddle point pi , which is, in general, located near the ends of the passages that connect the handle to the rest of the surface. We construct a basis loop ρ by computing two shortest paths in that connect qi and pi (b, yellow loop). Next, for each basis loop ρ, we create a nearby nonseparating cycle γ for one of the passages by performing region-growing from ρ in the increasing order of the distance function from pi (c and d, the red loop). To do so, we treat ρ as the two boundary loops of a region with no interiors. Denote this region R. Since the surface has handles, region-growing from R causes the two boundary loops to meet at a merging saddle point r. Figure 8 (b) shows the shapes of the two regions swept by the two loops when they meet (c, blue and green regions). Within each region, there is a shortest path between r and ρ. Together, the two paths form a nonseparating cycle γ (d, red loop) that is, in general, shorter and smoother than ρ. Finally, the surface is cut open along γ and the holes are ﬁlled with scaffold triangles. This reduces the genus of the surface by one. We repeat this process until the surface contains no handles, at which point it is ready for feature identiﬁcation (Section 3.3). Figure 9 shows the nonseparating cycles that are generated using our genus reduction algorithm for three surfaces. Notice that these loops appear in intuitive places and tend to be short, smooth, and nonwinding. 3.5 Patch Creation Through genus reduction and feature identiﬁcation, a surface is decomposed into a set of simple shapes (features) that are topological spheres without large protrusions. Our next task is to create one or two patches (topological disks) for every feature shape so that unfolding them results in little stretch. This is carried out in two stages. First, we classify the feature shapes as belonging to one of the following three proﬁles: a linear ellipsoid, a ﬂat ellipsoid, and a sphere. Next, we create patches for every feature shape based on its proﬁle. The classiﬁcation step requires computing the eigenvalues of a covariance matrix for every feature shape F , which is a triangular mesh. We compute the covariance matrix M F in closed forms by following the method of Gottschalk et al. [1996] that begins by thinking of F as having been sampled “inﬁnitely densely” by points over its surface. First, we calculate the mean µ of these points by integrating over all the triangles. Similarly, we compute M F relative to µ by performing integrations. The three categories of features are then distinguished based on the eigenvalues from M F : ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 13 Fig. 9. This ﬁgure displays the nonseparating cycles that are used for genus reduction for the Buddha (a), the dragon (upper right), and the feline (lower right). Each separating cycle consists of a sequence of edges in the surface that form a closed loop. Notice they appear in intuitive places. Furthermore, they tend to be short, smooth, and nonwinding. —Three nearly equal eigenvalues (a sphere). —One eigenvalue much larger than the other two (a linear ellipsoid). —Two nearly equal eigenvalues that are much larger than the third (a ﬂat ellipsoid). Let α, β, γ be the three eigenvalues of M F after normalization such that α 2 + β 2 + γ 2 = 1. The set of all valid conﬁgurations is: C := {(α, β, γ )|α 2 + β 2 + γ 2 = 1, α, β, γ ≥ 0}. (11) C is a spherical triangle in the ﬁrst octant. There are seven special conﬁgurations that correspond to the three linear ellipsoids, the three ﬂat ellipsoids, and the perfect sphere. By building the Voronoi regions on C using spherical distances, we can classify every shape based on the position of its con- ﬁguration in C. Alternatively, one can use the classiﬁcation measure proposed by Kindlmann and Weinstein [1999], which produces similar classiﬁcations for our test models. In the case of a linear ellipsoid, we ﬁnd a pair of points (p, q) on F such that they achieve the maximum surface distance. This can be approximated by letting p be a global maximum of AGD, and letting q be the point that is furthest away from p on the surface. We then ﬁnd the shortest path γ between p and q and cut the surface along γ by duplicating all of its vertices except p and q. This converts the surface into a single patch (a topological disk). For the ﬂat ellipsoid case, we ﬁrst identify the eigenvector associated with the smallest covariance eigenvalue. Then we ﬁnd the two most distant surface points x1 and x2 along this vector in opposite directions away from the surface’s center µ. Using region-growing, we ﬁnd the Voronoi regions for x1 and x2 . Both regions are homeomorphic to a disk. In the case of a sphere, we could treat it as a ﬂat ellipsoid and create two patches that are much like hemispheres. However, unfolding these patches would cause high stretch. Instead, we use an approach that is inspired by the two identical patches of a baseball (see Figure 10, upper left). We construct these regions based on two C-shaped curves, each of which travels halfway around one of the two mutually ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 14 • E. Zhang et al. Fig. 10. The “baseball” decomposition of the Venus (lower left) and the corresponding normal maps from patch unfolding using different stretch metrics: Sander’s metric [Sander et al. 2001] (middle) and the Green-Lagrange tensor (right, Section 4.1). perpendicular great circles. To compute these curves, we ﬁnd the three pairs of antipodal points on the surface that passes through the surface center µ, along the three eigenvector directions. Call these points x1 , x2 , y 1 , y 2 , z 1 , z 2 . One C-curve passes through x1 , y 1 , x2 , and the other connects z 1 , y 2 , z 2 . Using region-growing, we compute the “baseball decomposition” of the surface by building the surface Voronoi regions corresponding to the C-curves. The lower left of Figure 10 shows one of these curves and their corresponding Voronoi regions (red and green) for the Venus. Also shown in the same ﬁgure are the normal maps corresponding to the decomposition with two different patch unfolding methods: Sander’s metric [Sander et al. 2001] (middle), and the Green-Lagrange tensor (right, Section 4.1). In the next section, we will show that patch unfolding using the Green-Lagrange tensor results in less overall stretch than using Sander’s metric. Sometimes a feature is a curved long cylinder, such as the feline’s tail, whose covariance analysis is similar to that a ﬂat ellipsoid or a sphere. In this case, the center µ is situated outside the volume enclosed by the surface and not all three pairs of antipodal points can be found. When this happens, we simply treat the surface as a linear ellipsoid. 4. PATCH UNFOLDING A class of traditional patch unfolding methods are based on discrete conformal mappings [Eck et al. e 1995; Floater 1997; L´ vy et al. 2002]. By ﬁxing the texture coordinates of the boundary vertices, the texture coordinates of the interior vertices can be solved through a closed form system. These methods e are fast and stable, and the solution is unique [L´ vy et al. 2002]. However, conformal mappings do not preserve areas. Regions can be stretched or compressed, causing uneven sampling rates. Sander et al. [2001] have proposed a post-processing step in which the texture coordinates of the interior vertices are optimized to reduce a form of geometric stretch (which we will refer to as Sander’s stretch metric), and their work inspired our own stretch optimization. We seek a deﬁnition of stretch that provides a balance between conformal mappings and area-preserving mappings. 4.1 The Green-Lagrange Tensor: a Balanced Stretch Metric An isometry between two surfaces is a bijective mapping f that maintains distances between the two metric spaces, that is, d ( f (x), f ( y)) = d (x, y) for all points x and y in the domain. An ideal surface parameterization P would be an isometry between the surface S and its images in the texture map I , ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 15 which means an everywhere even sampling is possible based on P . For most patches no isometric pa- rameterization exists, except in the case of developable surfaces. Classical results from Riemannian geometry state that there exists a conformal (“angle-preserving”) mapping between S and I . Some parameterization methods ﬁrst compute a conformal parameterization for a patch, and then optimize e the interior vertices based on some stretch metric [Sander et al. 2001; L´ vy et al. 2002; Sheffer and e de Sturler 2002]. Sander’s metric (used in Sander et al. [2001]; L´ vy et al. [2002]) helps balance the sampling given by the parameterization. Unfortunately, it does not always distinguish between isome- tries and anisotropic stretch. To illustrate this point and to introduce our new balanced stretch metric, we review Sander’s metric and related background. For a triangle T = { p1 , p2 , p3 } in the surface S ⊂ R3 , and its corresponding texture coordinates U = {u1 , u2 , u3 } in R2 = s, t , the parameterization P : U → T is the unique afﬁne mapping that maps ui to pi (1 ≤ i ≤ 3). To be more speciﬁc, let A(v1 , v2 , v3 ) be the area of the triangle formed by vertices v1 , v2 and v3 , then A(u, u2 , u3 ) p1 + A(u1 , u, u3 ) p2 + A(u1 , u2 , u) p3 P (u) = . (12) A(u1 , u2 , u3 ) Let Ps and Pt denote the partial derivatives of P . The metric tensor induced by P is: Ps · Ps Ps · Pt a b G= = . (13) Ps · Pt Pt · Pt b c The eigenvalues of G are (a + c) ± (a − c)2 + 4b2 {γmax , γmin } = (14) 2 which represents the maximal and minimal stretch of a nonzero vector. Sander’s metric is deﬁned as the average stretch metric in all possible directions, that is, L2 (T ) = (γmax + γmin )/2 = 2 2 (a + c)/2. (15) The metric has a lower bound of 1, and isometries achieve this lower bound. Equation 12 assumes that the area of the triangle equals its textural image. When computing the global stretch, we assume the total area of the surface equals the total area of the textural image. This means that we need to add a global scale factor to each triangle. Let A(t) and A (t) be the surface area and textural area of a triangle t, respectively. The global factor is A(t) ρ= t∈S (16) t∈S A (t) and we rewrite Equation 12 as: A(u, u2 , u3 ) p1 + A(u1 , u, u3 ) p2 + A(u1 , u2 , u) p3 P (u) = . (17) ρ A(u1 , u2 , u3 ) Notice that individual triangles, in general, have scale factors different from ρ. Unfortunately under this scenario, there are anisotropic stretch for which Sander’s stretch metric also gives a value of one. In particular, this metric cannot distinguish between isotropic and anisotropic stretch. For instance, all of the following tensors 1 0 0.5 0 1 0.5 , , (18) 0 1 0 1.5 0.5 1 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 16 • E. Zhang et al. Fig. 11. This ﬁgure compares two surface parameterizations for the bunny obtained by vertex optimization based on Sander’s stretch metric [Sander et al. 2001] (middle) and the Green-Lagrange tensor (right). Optimization based on Sander’s metric causes high anisotropic stretch, especially the two largest patches. Compare the tail and the two rear legs (the square bumps on each side). result in the same stretch measured in Sander’s stretch metric, but the ﬁrst one is clearly the most desirable. For this reason, we use the Green-Lagrange tensor to measure stretch and to guide patch optimization. Using the Green-Lagrange tensor as the stretch metric has been proposed before [Maillot et al. 1993]. However, it has not been used for patch optimization. Using the above terminology, the Green-Lagrange tensor of G t is deﬁned as G t − I , in which I is the identity matrix. The square of the Frobenius norm of this tensor is T (G t ) = ( G t − I F )2 = (a − 1)2 + 2b2 + (c − 1)2 . It is zero if and only if G t is an isometry. We therefore deﬁne the stretch as Et2 = 2T (G t ) = 2((a − 1)2 + 2b2 + (c − 1)2 ) = [(a − c)2 + 4b2 ] + [(a + c − 2)2 ] = Econformal + Earea . (19) 2 2 Notice that for the tensor to be conformal, we need a = c and b = 0. When these conditions are met, the tensor becomes a scaling of magnitude a = c. It is an isometry if a = c = 1. This metric seeks 2 a balance between the angle-preserving energy term Econformal and the area-preserving energy term Earea . A triangle’s mapping is an isometry if and only if Et = 0. This metric distinguishes between 2 anisotropic stretch and isometries. In addition, it penalizes both undersampling and oversampling. However, the penalty is more severe for undersampling. This is desirable for texture mapping when a global isometry is not available. We note that Sorkine et al. [2002] devise a different stretch metric that also distinguishes anisotropic stretches from an isometry. We choose not to use their metric because it uses a max function, causing it to give equal stretch values to some cases that we feel should be distinguished. The total balanced stretch of a patch S is therefore, E 2 (S) = (at − ct )2 + 4bt + [(at + ct − 2)2 ] . 2 (20) t∈S The ideal value E(S) for a patch S is zero, meaning all triangles in the patch are mapped isometrically. Figure 11 compares the unfolding of the bunny surface using Sander’s metric (middle) and the Green- Lagrange tensor (right). Notice that on the two largest patches, unfolding with Sander’s metric produces anisotropic stretch (the tail and the two rear legs). The Green-Lagrange tensor performs well on all of these patches. Figure 10 shows the same comparison for the Venus. Again, optimization using Sander’s metric causes anisotropic stretch. In Section 6.1, we will show the Green-Lagrange tensor also performs better in terms of image ﬁdelity, despite sometimes having lower packing efﬁciencies. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 17 4.2 Boundary Optimization with Scaffold Triangles The process of patch optimization refers to moving vertices in the plane to minimize a given stretch metric. Most patch optimization methods handle the boundary vertices of a patch differently from the interior vertices. For an initial layout, boundary vertices are typically either mapped to the vertices of a convex polygon, or placed through conformal mappings. Sander et al. [2001] perform a nonlinear optimization on the interior vertices by moving one vertex at a time along some randomly chosen line to improve stretch. This process is very effective in reducing stretch during unfolding. Later, they extend the optimization framework to handle patch boundaries [Sander et al. 2002]. However, global foldovers may occur when a boundary vertex accidentally “walks” inside another triangle that is spatially faraway on the surface. To prevent this from happening, Sander et al. [2002] perform a global intersection test when performing optimization on a boundary vertex. However, this process is computationally expensive. We introduce a new optimization method that allows the boundary vertices to move freely without the need to check for global foldovers. First, we compute an initial harmonic parameterization as described in Floater [1997]. Next, we construct a “virtual boundary” (a square) in the parameterization plane that encloses the patch. The 3D coordinates of the square are assigned to be mutually different and outside the convex hull of the patch in the 3D space. As we will see next, the exact coordinates of the virtual boundary are insigniﬁcant, provided that they do not coincide with each other or with the patch. Scaffold triangles are used to triangulate the region between the original patch boundary and the virtual boundary. Finally, we perform patch optimization [Sander et al. 2001] on the “enlarged” patch. There are two issues regarding scaffold triangles that need attention. (1) How should we deﬁne stretch for scaffold triangles? (2) How can we deﬁne and maintain their connectivity? The ﬁrst issue is handled as follows: the stretch of a scaffold triangle is deﬁned as inﬁnity if there is a foldover, otherwise it is deﬁned as zero. This allows a boundary vertex to move within its immediate incident triangles to obtain better stretch without the need to check for global foldovers. Furthermore, the exact 3D coordinates of the virtual boundary are insigniﬁcant. The second issue appears when the initial connectivity of scaffold triangles unnecessarily constrains the movements of boundary vertices. This is because scaffold triangles are designed to prevent global foldovers, that is, one patch vertex “walks” onto a patch triangle other than its immediate neighbor- ing triangles, which unfortunately include the scaffold triangles. To remedy this overly conservative approach, we allow scaffold regions to be retriangulated at the end of each optimization step in which all the vertices have been moved. For any edge between two scaffold triangles, we perform an edge ﬂip operation if it improves the triangles’ aspect ratios. Figure 12 (right) illustrates the effect of using scaffold triangles on an example patch on a cube (b: without scaffold triangles; d: with scaffold triangles). Notice that scaffold triangles allow the opti- mization to achieve a zero stretch in this case. The shape of the virtual boundary and the connectivity of the scaffold triangles are insigniﬁcant since they merely serve as a placeholder to allow the boundary vertices of a patch to move freely without causing global foldovers. This is different from the work of Lee et al. [2002], in which virtual boundaries are constructed as parts of edge springs to obtain an initial parameterization. In their work, the shape and the connectivity of the virtual boundaries directly affect the stretch of the resulting parameterization. Indeed, several layers of virtual boundaries are often required to produce reasonable results using their method. In our work, only one layer is required. Scaffold triangles also arise from hole-ﬁlling operations that occurred during genus reduction and feature identiﬁcation. They are treated similarly as the scaffold triangles from the virtual boundary, ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 18 • E. Zhang et al. Fig. 12. This ﬁgure demonstrates the effect of scaffold triangles on patch unfolding. A patch that consists of two sides of a cube (a, colored in yellow) is unfolded using three methods: optimization without scaffold triangles (b), with scaffold triangles but without optimization (c), and with scaffold triangles and optimization (d). Scaffold triangles are colored in gray. Notice when both optimization and scaffold triangles are used, the patch is unfolded with a zero stretch. that is, they do not contribute to the stretch metric unless there are foldovers, and their connectivity can be changed through retriangulation. Several of the patches in the texture maps shown in Figure 13 (right column, the dinosaur) and Figure 15 (bottom row, the dragon) have holes that make use of scaffold triangles. 5. PACKING The ﬁnal step of surface parameterization is patch packing, which refers to arranging unfolded patches inside a rectangular region (texture map) without any overlaps. The ratio between the total space oc- cupied by the patches and the area of the rectangle is the packing efﬁciency. Higher packing efﬁciency indicates less wasted space in the ﬁnal texture map. The problem of ﬁnding an optimal packing is a special instance of an NP-hard problem: containment and minimum enclosure. The problem has been studied extensively in the textile industry and the computational geometry community [Milenkovic 1998]. Several packing algorithms have been proposed as parts of some surface parameterization tech- e niques [Sander et al. 2001; L´ vy et al. 2002]. These methods are very effective when all the patches are nearly circular and have similar sizes. Since our patch creation technique tends to produce a small num- ber of large and often-elongated patches, we developed a packing technique that takes into account the orientations of the patches and, in general, achieves better packing efﬁciencies. Later, we discovered that our method is very similar to the packing technique developed independently by Sander et al. [2003]. Our packing algorithm is based on the following two observations. First, our patch creation method tends to produce a small number of patches. Second, several of these patches are large and have elongated shapes. The ﬁrst observation enables us to perform an optimal searching that would have been impractical for patch creation methods that produce hundreds of patches. The second observation indicates that the orientations of the large and elongated patches can help create gaps, into which smaller patches can be placed. Our algorithm consists of two stages: initialization and placement. During the initialization stage, we create a “canvas”, that is, an N × N grid structure at the textural resolution. Every cell in the canvas is marked as “unoccupied”. Under the same resolution, we discretize the bounding box of every patch into a 2D grid and mark any cell intersecting the patch as “occupied”. We obtain eight variations of grids for each patch by the combination of reﬂections with respect to the patch’s vertical axis, the horizontal axis, and the diagonal. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 19 Fig. 13. This ﬁgure compares the packing results using the algorithm of Levy et al. [2002] (top row) and our algorithm (bottom row) for three models: the feline (left), the Buddha (middle), and the dinosaur (right). Notice the space under the “horizon” is used to pack smaller patches, and some patches are reﬂected diagonally to achieve a tighter packing. During the placement stage, we insert the patches, one-by-one, into the canvas in the decreasing order of patch size (area). The ﬁrst patch is placed at the lower left corner of the canvas. After a patch is inserted, we update the status of cells in the canvas that have been covered by the newly placed patch. Before inserting the next patch Pi , we examine its eight variations to ﬁnd the one that minimizes the wasted space in the canvas. To be precise, let α, a m × n grid cells, be a variation for Pi . We wish to place the lower-left corner of α in the (a, b) grid cell in the canvas such that the following conditions are met: (1) For any occupied grid cell ( p, q) in α, the corresponding grid cell (a + p, b + q) in the canvas is unoccupied. (2) α minimizes max(a + m, b + n). In other words, we wish to place the patch as close to the lower left corner of the canvas as possible. Once the best variation is chosen, we translate and scale the patch to reﬂect its position and orientation in the canvas. After all patches have been inserted, usually only M ×M grid cells in the canvas have become occupied. For all our test models, M is between one-third and one-half of the size of the canvas. We perform scaling to all patches with the same factor so that the M × M grid cells are mapped to [0, 1] × [0, 1]. Figure 13 shows the improvement of our packing algorithm (bottom row) over the algorithm by L´ vy e et al. [2002] (top row) with three test models: the feline (left), the Buddha (middle), and the dinosaur (right). Notice the space under the “horizon” is reused to pack small patches, and some patches are rotated/reﬂected to achieve a tighter packing. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 20 • E. Zhang et al. Fig. 14. This ﬁgure shows the result of our feature segmentation method on various test models. The cow, the horse and the rabbit are genus zero surfaces. The genus of the dragon, the Buddha, and the feline are one, six, and two, respectively. 6. RESULTS We have applied our feature-based surface parameterization method to a number of test models. The results for the bunny and the dinosaur are shown in Figure 1 and 2, respectively. In Figure 14, we show the results of some other 3D models, including three surfaces with nonzero genus (the Buddha, the dragon, and the feline). Notice, in general, the feature regions are intuitive. For example, the horns and legs of animals are segmented from the bodies, and the Buddha’s stand is identiﬁed as a single feature (a ﬂat ellipsoid). (We wish to emphasize that no real animals were harmed during our research.) Figure 10 (right) and Figure 13 (lower middle) show the normal maps of the Venus and the Buddha, respectively. In a normal map, colors (R, G, B) are used to encode unit surface normals (x, y, z) [Sander et al. 2001]. Because of the many sharp creases on the Venus and the Buddha, patch creation methods based on surface curvature would have split the surfaces into many tiny patches, and such an example e can be found in L´ vy [2003]. Our method, however, was able to create large patches with little stretch. Figure 15 shows textured models (left column) and the corresponding texture maps (right column) of the Buddha (top), the feline (middle), and the dragon (bottom). Table I gives the average stretch for the patches of the test models and the times for patch creation and unfolding using our method. The texture used for the Buddha is a wood texture from Perlin’s noise [1985]. The textures used for the feline and the dragon were created by performing example-based texture synthesis directly on the surfaces [Turk 2001; Wei and Levoy 2001]. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 21 Fig. 15. This ﬁgure shows the parameterization of three models using our feature-based algorithm: textured models (left) and texture layouts (right, 512 × 512). ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 22 • E. Zhang et al. Table I. Average stretch (measured in Green-Lagrange) and timing results (minutes:seconds) for our patch creation and unfolding algorithm. Times are measured on a 2.4 GHz PC model # # stretch patch creation patch unfolding name polygons patches (Green-Lagrange) time time Buddha 20,000 28 1.56 6:32 27:29 Buddha (large) 100,000 16 1.27 39:25 168:43 bunny 10,000 6 0.23 1:07 8:28 cow 10,524 29 0.28 2:15 4:01 dinosaur 10,636 14 0.25 1:37 5:53 dragon 20,000 24 0.83 3:00 16:23 dragon (large) 100,000 39 0.49 38:00 124:32 feline 10,000 41 0.22 2:31 2:32 feline (large) 100,000 46 0.29 32:57 109:27 horse 10,000 27 0.22 1:30 3:21 Venus 10,000 2 0.17 0:11 11:38 rabbit 10,000 8 0.24 0:53 4:50 6.1 Measuring Quality Measuring the quality of a surface parameterization is an important yet complicated issue. It has several components. (1) Stretch affects the sampling rate across the surface. (2) Seams cause discontinuities across patch boundaries. (3) Smoothness measures the amount of sharp changes in the sampling rates across interior edges of a patch. (4) Packing efﬁciency determines the efﬁcient use of the texture map. When evaluating a surface parameterization method, it is not clear how these components should be combined measure the quality of the resulting map. On the other hand, for texture mapping applications, the quality of a surface parameterization should reﬂect “image ﬁdelity”, that is, the faithfulness of the images produced using the texture maps to the images for which the surface signals are directly computed. Next, we present an image-based metric, which draws inspiration from the work on image- driven mesh simpliﬁcation [Lindstrom and Turk 2000]. Given a surface parameterization P , we ﬁrst compute a continuous and smooth surface signal and store the result in a texture map based on P . Then, we render the surface from many viewpoints using the texture map, and compare the image differences with respect to the true surface signals. In practice, we choose 20 orthographic viewpoints that are the vertices of a surrounding dodecahedron. Let M 0 be the surface with the signal directly computed, and M i be the textured surface with the texture size of 2i × 2i . The RMS “image” error between the images is calculated as: 20 RMS ( M i , M 0 ) = Din . (21) n=1 Here, Din is the squared sum of pixel-wise intensity difference between the n-th image of M i and M 0 . Equation 21 can be seen as the discretization of the following functional: p∈S D 2 (M 0 (p), M (p))V (p)d p E(M , M 0 ) = . (22) p∈S V (p)d p ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 23 Here M 0 and M refer to the original and the reconstructed surface signals (in colors), respectively. D(M 0 (p), M (p)) is a perceptual metric between colors. For our application, we use D((r1 , g 1 , b1 ), (r2 , g 2 , b2 )) = ((r2 − r1 )2 + ( g 2 − g 1 )2 + (b2 − b1 )2 )/3. (23) V (p) is the view-independent surface visibility as deﬁned in Zhang and Turk [2002], which measures the visibility of p with respect to viewpoints on a surrounding sphere of an inﬁnite radius. Therefore, E(M , M 0 ) takes into account surface visibility in addition to color errors caused by stretch, seams, packing efﬁciencies, and smoothness of the parameterization. As demonstrated in Lindstrom and Turk [2000] and Zhang and Turk [2002], the error metric can be sampled with a small number of viewpoints that are evenly spaced in the view space. One possible ideal surface signal can be obtained by ﬁrst spreading a set of evenly spaced points on the surface and building a smooth function that uses these points as the bases. However, computing such a function over a surface with complex geometry is time-consuming. In contrast, a 3D checkerboard pattern is relatively easy to compute, and it has the nice property that the largest differential in frequencies in all directions is bounded. Although not perfect, it is nonetheless a good starting point. To make the signal continuous, we replace each “box” section with a “hat”. The frequency in each main axial direction is the same. In practice, we use 1/16 of the maximum side of the bounding box of the surface as the frequency. Table II compares two unfolding methods, optimization with Sander’s metric and the Green-Lagrange tensor for nine test models. Notice optimization with our metric produces lower stretch for all the test models. Furthermore, despite sometimes having lower packing efﬁciencies, optimization with our metric produces lower image errors for all the test cases. Figure 16 provides a visual comparison between the ideal signal (bottom-middle), the textured model using optimization with Sander’s metric (bottom-left) and the Green-Lagrange tensor (bottom-right) for the Buddha model with the texture map of size 128 × 128. Notice the different level of blurring in the left image (front body and base) due to a highly uneven sampling rate. This phenomenon is less noticeable in the right image that uses our approach. Compare their corresponding texture maps: left row (Sander’s metric) and right row (our technique). 7. CONCLUSION AND FUTURE WORK In this article, we present an automatic surface parameterization method in which manifold surfaces are divided into feature regions for patch creation. By performing topological analysis of the average geodesic distance function, we are able to divide the surface into a small number of large patches that can be unfolded with little stretch. For patch unfolding, we use the Green-Lagrange tensor to guide the vertex optimization process. Also, we use scaffold triangles to allow the boundary vertices of a patch to be optimized. Although they were developed with texture mapping in mind, we think our surface segmentation and genus reduction methods might also be useful for other applications. Finally, we describe an image-based quality measure for surface parameterization techniques. There are several areas for improvement. First, the nonlinear optimization stage of our patch un- folding algorithm is rather slow. Sander et al. [2002] propose a hierarchical optimization framework, which we would like to adapt to our system with scaffold triangles. Second, our feature identiﬁcation technique sometimes creates more feature regions than necessary. For instance, the horse’s legs are divided into two or three regions (Figure 14). While this does not seem to dramatically affect the quality of ﬁnal texture maps, we are looking for alternative methods that can determine separating regions more robustly. For instance, we would like to see whether AGD could be used to decide the boundaries of separating regions. Since AGD is less noisy and is intrinsic to the surface, we expect that it contain more useful information. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 24 • E. Zhang et al. Table II. This table compares two stretch metrics for guiding Sander-style vertex optimization. With the exception of the columns labeled “Image Error”, the top row in each data cell are the results using Sander’s metric, and the bottom rows are the results using the Green-Lagrange tensor (Section 4.1). For a comprehensive comparison, three measurements are provided: average stretch (the ﬁrst two columns), packing efﬁciency (third column), and image-based error metric (Section 6.1, the last two columns). The numbers in the “Image Error” columns are the percentage of error difference of the image error caused by Sander’s metric to the image error caused by our technique. For all nine test models, optimization with the Green-Lagrange tensor results in lower average stretch (either measured using Sander’s metric or the Green-Lagrange tensor). Despite sometimes having lower packing efﬁciencies, our technique results in less image errors for all the test models Comparison for patch unfolding with different optimization metrics (top row using Sander’s metric [Sander et al. 2001], bottom row using Green-Lagrange tensor) Stretch measured in Stretch measured in Packing Image Error Image Error Sander’s metric Green-Lagrange Ratio 128 × 128 256 × 256 Buddha 1.27 26.80 0.67 8.28% 10.77% 1.18 1.56 0.68 bunny 1.13 3.92 0.60 14.46% 10.93% 1.02 0.23 0.65 cow 1.11 3.07 0.73 1.89% 1.62% 1.03 0.28 0.65 dinosaur 1.07 1.55 0.59 13.22% 5.16% 1.03 0.25 0.66 dragon 1.26 13.78 0.67 11.14% 12.84% 1.13 0.83 0.67 feline 1.10 1.73 0.66 7.25% 3.46% 1.02 0.22 0.64 horse 1.09 1.65 0.67 5.57% 0.93% 1.03 0.22 0.66 Venus 1.10 2.99 0.59 16.29% 15.26% 1.02 0.17 0.66 rabbit 1.12 3.00 0.68 8.73% 4.54% 1.03 0.24 0.65 Many additional topics in this area are interesting to us. First, our algorithm does not directly minimize the amount of seams caused by the segmentation. It would be desirable to have control over seams. Second, although our image-based quality measure implicitly takes into account stretch, seams, smoothness, packing efﬁciency and visibility, it may be desirable to understand and control their impact on the quality of a surface parameterization. For instance, while the Green-Lagrange tensor is a good indicator of stretch, how it relates to the image-based quality measure deserves further investigation. Third, we are still looking for other functions for surface segmentation. Our metric has been chieﬂy based on surface distance, which is intrinsic to the surface. Are there other functions that combine both intrinsic and extrinsic properties of the surfaces that might result in an even better segmentation? For instance, is there a surface function that helps ﬁnd a separating cycle for the feline’s wing exactly where human would place it? Finally, surface parameterization is important for many applications, one of which is surface visu- alization. A complex surface often contains interiors and concavities that are difﬁcult to see from the outside viewpoints. We would like to investigate the use of parameterization for surface exploration, giving the user the ability to navigate, orient, and focus. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 25 Fig. 16. Comparisons between patch unfolding with Sander’s metric [Sander et al. 2001] (left column) and with Green-Lagrange stretch tensor (right column) for the Buddha using the 3D texture described in Section 6.1 (original signal is shown in the middle of the bottom row). In each column, from top-to-bottom, are the normal map, the map of the 3D texture, and the textured model. Notice in the left column (Sander’s metric), the patches created from the base are assigned more space than those from the Buddha’s torso. This causes a loss of signal in the textured model (face, body, and feet). On the other hand, optimization using the Green-Lagrange tensor (right column) produces a more even sampling rate and the reconstruction error is less noticeable. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 26 • E. Zhang et al. ACKNOWLEDGMENTS e We would like to thank the following people and groups for the 3D models they provided: Zo¨ Wood and o Peter Schr¨ der, Mark Levoy and the Stanford Graphics Group, Andrzej Szymczak, and Cyberware. We also appreciate the discussions with Jarek Rossignac and Andrzej Szymczak. Finally, we wish to thank our anonymous reviewers for their valuable comments and suggestions. REFERENCES ALLIEZ, P., MEYER, M., AND DESBRUN, M. 2002. Interactive geometry remeshing. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 347–354. AXEN, U. AND EDELSBRUNNER, H. 1998. Auditory morse analysis of triangulated manifolds. Mathematical Visualization, H. C. Hege and K. Polthier, Eds. Springer-Verlag, Heidelberg, Germany, 223–236. BANCHOFF, T. F. 1970. Critical points and curvature for embedded polyhedral surfaces. American Mathematical Monthly 77, 475–485. BENSON, D. AND DAVIS, J. 2002. Octree textures. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 785–790. CARR, N. A. AND HART, J. C. 2002. Meshed atlases for real-time procedural solid texturing. ACM Trans. Graph. 21, 2, 106–131. CIGNONI, P., MONTANI, C., ROCCHINI, C., AND SCOPIGNO, R. 1998. A general method for recovering attribute values on simpliﬁed meshes. IEEE Visualization Proceeding, 59–66. DEBRY, D., GIBBS, J., PETTY, D. D., AND ROBINS, N. 2002. Painting and rendering textures on unparameterized models. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 763–768. DESBRUN, M., MEYER, M., AND ALLIEZ, P. 2002. Intrinsic parameterizations of surface meshes. Proceeding of Eurographics, 209–218. ECK, M., DEROSE, T., DUCHAMP, T., HOPPE, H., LOUNSBERY, M., AND STUETZLE, W. 1995. Multiresolution analysis of arbitrary meshes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1995), 173–182. EDELSBRUNNER, H., HARER, J., AND ZOMORODIAN, A. 2003. Hierarchical Morse-Smale complexes for piecewise linear 2-manifolds. Discrete Comput. Geom. 30, 87–107. ERICKSON, J. AND HAR-PELED, S. 2002. Optimally cutting a surface into a disk. Symposium on Computational Geometry, 244–253. FLOATER, M. S. 1997. Parameterization and smooth approximation of surface triangulations. Comput. Aid. Geomet. Design 14, 3 (July), 231–250. GOTTSCHALK, S., LIN, M. C., AND MANOCHA, D. 1996. OBB-Tree: A hierarchical structure for rapid interference detection. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1996), 171–180. GU, X., GORTLER, S. J., AND HOPPE, H. 2002. Geometry images. ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 355–361. GUSKOV, I. AND WOOD, Z. 2001. Topological noise removal. Graphics Interface, 19–26. HANRAHAN, P. AND HAEBERLI, P. E. 1990. Direct WYSIWYG painting and texturing on 3d shapes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1990), 215–223. HILAGA, M., SHINAGAWA, Y., KOHMURA, T., AND KUNII, T. L. 2001. Topology matching for fully automatic similarity estimation of 3d shapes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001), 203–212. HORMANN, K. AND GREINER, G. 1999. MIPS: An efﬁcient global parameterization method. Curve and Surface Design: Saint-Malo e 1999, Laurent, Sablonni` re and Schumaker, Eds. Vanderbilt University Press. 153–162. KACZYNSKI, T., MISCHAIKOW, K., AND MROZEK, M. 2004. Computational Homology. Applied Mathematical Sciences 157, Springer- Verlag. KATZ, S. AND TAL, A. 2003. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM Trans. Graph. (SIGGRAPH 2003) 22, 3 (July), 954–961. ¨ KHODAKOVSKY, A., LITKE, N., AND SCHRODER, P. 2003. Globally smooth parameterizations with low distortion. ACM Trans. Graph. (SIGGRAPH 2003) 22, 3 (July), 350–357. KIMMEL, R. AND SETHIAN, J. A. 1998. Computing geodesic paths on manifolds. Proceedings of National Academy of Sciences 95, 15 (July), 8431–8435. KINDLMANN, G. AND WEINSTEIN, D. 1999. Hue-balls and lit-tensors for direct volume rendering of diffusion tensor ﬁelds. IEEE Visualization Proceeding, 183–189. LAZARUS, F., POCCHIOLA, M., VEGTER, G., AND VEROUST, A. 2001. Computing a canonical polygonal schema of an orientable triangulated surface. 17th ACM Symposium on Computational Geometry, 80–89. ¨ LEE, A., SWELDENS, W., SCHRODER, P., COSWAR, L., AND DOBKIN, D. 1998. MAPS: Multiresolution adaptive parameterization of surfaces. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1998), 95–104. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Feature-Based Surface Parameterization and Texture Mapping • 27 LEE, Y., KIM, H. S., AND LEE, S. 2002. Mesh parameterization with a virtual boundary. Computers & Graphics (Special Issue of the 3rd Israel-Korea Binational Conference on Geometric Modeling and Computer Graphics) 26, 5, 677–686. L´ VY, B. 2003. Least squares conformal maps gallery. http://www.loria.fr/levy/Galleries/LSCM/index.html. E L´ VY, B., PETITJEAN, S., RAY, N., AND MAILLOT, J. 2002. Least squares conformal maps for automatic texture atlas generation. E ACM Trans. Graph. (SIGGRAPH 2002) 21, 3 (July), 362–371. LINDSTROM, P. AND TURK, G. 2000. Image-driven simpliﬁcation. ACM Trans. Graph. 19, 3, 204–241. LOPES, H., ROSSIGNAC, J., SAFONOVA, A., SZYMCZAK, A., AND TAVARES, G. 2003. Edgebreaker: A simple compression algorithm for surfaces with handles. Comput. Graph. Int. J. 27, 4, 553–567. MAILLOT, J., YAMIA, H., AND VERROUST, A. 1993. Interactive texture mapping. Computer Graphics Proceedings, Annual Confer- ence Series (SIGGRAPH 1993), 27–34. MILENKOVIC, V. J. 1998. Rotational polygon containment and minimum enclosure. Proceedings of the 14th Annual Symposium on Computational Geometry, ACM. MILNOR, J. 1963. Morse Theory. Annals of Mathematical Studies. Princeton University Press, Princeton, NJ. NI, X., GARLAND, M., AND HART, J. C. 2004. Fair morse functions for extracting the topological structure of a surface mesh. ACM Trans. Graph. (SIGGRAPH 2004) 23, 3 (Aug.), 613–622. PERLIN, K. 1985. An image synthesizer. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 1985), 287– 296. PIPONI, D. AND BORSHUKOV, G. 2000. Seamless texture mapping of subdivision surfaces by model pelting and texture blending. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2000), 471–478. REEB, G. 1946. Sur les points singuliers díune forme de pfaff completement integrable ou díune fonction numerique [on the singular points of a completely integrable pfaff form or of a numerical function]. Comptes Randus Acad. Sciences Paris 222, 847–849. ROURKE, C. AND SANDERSON, B. 1972. Introduction to Piecewise-Linear Topology. Springer Verlag. SANDER, P. V., GORTLER, S. J., SNYDER, J., AND HOPPE, H. 2002. Signal-specialized parameterization. Proceedings of the 13th Eurographics Workshop on Rendering, 87–100. SANDER, P. V., SNYDER, J., GORTLER, S. J., AND HOPPE, H. 2001. Texture mapping progressive meshes. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001), 409–416. SANDER, P. V., WOOD, Z. J., GORTLER, S. J., SNYDER, J., AND HOPPE, H. 2003. Multi-chart geometry images. Proceedings of the 1st Symposium on Geometry Processing. SHEFFER, A. AND DE STURLER, E. 2001. Parameterization of faceted surfaces for meshing using angle based ﬂattening. Engineer. with Comput. 17, 3, 326–337. SHEFFER, A. AND DE STURLER, E. 2002. Smoothing an overlay grid to minimize linear distortion in texture mapping. ACM Trans. Graph. 21, 4, 874–890. SHEFFER, A. AND HART, J. C. 2002. Seamster: Inconspicuous low-distortion texture seam layout. IEEE Visualization Proceeding, 291–298. SORKINE, O., COHEN-OR, D., GOLDENTHAL, R., AND LISCHINSKI, D. 2002. Bounded-distortion piecewise mesh parameterization. IEEE Visualization Proceeding, 355–362. TURK, G. 2001. Texture synthesis on surfaces. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001), 347–354. WEI, L. Y. AND LEVOY, M. 2001. Texture synthesis over arbitrary manifold surfaces. Computer Graphics Proceedings, Annual Conference Series (SIGGRAPH 2001), 355–360. ¨ WOOD, Z., HOPPE, H., DESBRUN, M., AND SCHRODER, P. 2004. Removing excess topology from isosurfaces. ACM Trans. Graph. 23, 2, 190–208. ZHANG, E. AND TURK, G. 2002. Visibility-guided simpliﬁcation. IEEE Visualization Proceeding, 267–274. Received June 2003; revised April 2004; accepted August 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. On C 2 Triangle/Quad Subdivision SCOTT SCHAEFER and JOE WARREN Rice University In this article, we present a subdivision scheme for mixed triangle/quad meshes that is C 2 everywhere except for isolated, extraordinary points. The rules that we describe are the same as Stam and Loop’s scheme [2003] except that we perform an unzipping pass prior to subdivision. This simple modiﬁcation improves the smoothness along the ordinary triangle/quad boundary from C 1 to C 2 , and creates a scheme capable of subdividing arbitrary meshes. Finally, we end with a proof based on Levin and Levin’s [2003] joint spectral radius calculation to show our scheme is indeed C 2 along the triangle/quad boundary. Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling General Terms: Algorithms Additional Key Words and Phrases: Subdivision, triangle/quad, joint spectral radius 1. INTRODUCTION Subdivision has become a staple of the geometric modeling community allowing coarse, polygonal shapes to represent highly reﬁned, smooth shapes with guaranteed continuity properties. Previously, there has been a dichotomy between polygonal primitives that subdivision schemes operate on. Two of the most popular subdivision schemes, Loop [1987] and Catmull and Clark [1978], operate on triangle and quad meshes, respectively. 1.1 Stam and Loop’s Scheme Recently, Stam and Loop [2003] introduced a generalization of Loop and Catmull-Clark subdivision that uniﬁes these schemes together and operates on mixed triangle/quad surfaces. The subdivision scheme that they present reproduces Loop subdivision on the triangular portions of the mesh and Catmull-Clark subdivision on the quadrilateral polygons. Furthermore, the authors derive subdivision rules for extraordinary vertices composed of both quads and triangles where the subdivision scheme is C 1 . Stam and Loop [2003] created their generalization of triangle and quad subdivision by utilizing the fact that both Loop and Catmull-Clark subdivision can be written as linear subdivision, followed o by averaging [Zorin and Schr¨ der 2001; Stam 2001; Warren and Weimer 2001]. For triangle/quad meshes, linear subdivision splits triangles into four new triangles and quads into four new quads. This process introduces what Stam and Loop called an ordinary edge along the triangle/quad boundary where vertices are contained by two adjacent quads and three adjacent triangles (see Figure 1). Authors’ address: Department of Computer Science, Rice University, 6100 South Main St., Houston, TX 77251-1892; email: sschaefe@rice.edu. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0028 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 28–36. On C 2 Triangle/Quad Subdivision • 29 Fig. 1. Linear subdivision for triangle/quad meshes. An ordinary triangle/quad conﬁguration is introduced all along the boundary edge. Fig. 2. Averaging masks for Catmull-Clark (left), Loop (middle) and Triangle/Quad (right). Fig. 3. Levin/Levin’s rules for the central edge (left and middle). Unzipping rule for triangular side (right). Once linear subdivision is complete, an averaging pass is applied to the mesh. Figure 2 shows the averaging mask for the ordinary case of Catmull-Clark and Loop subdivision. Stam and Loop [2003] noticed that the averaging masks for triangle and quad subdivision looked remarkably similar and hypothesized that the averaging mask for mixed triangle/quad surfaces at the ordinary boundary would simply be the mask shown on the right of Figure 2. The authors then used this observation to generalize the averaging mask to arbitrary conﬁgurations of quads and triangles around a vertex. Finally, Stam and Loop show that their scheme is C 2 everywhere except for extraordinary points and the ordinary, triangle/quad boundary, where their scheme is C 1 . 1.2 Levin and Levin’s Scheme To remedy this smoothness problem along ordinary triangle/quad edges, Levin and Levin [2003] intro- duced a set of modiﬁed rules along the triangle/quad boundary, shown in Figure 3. The authors also present the concept of an “unzipping” mask, shown in Figure 3 (right). Prior to subdivision, points along the regular triangle/quad boundary are replicated; one set of vertices for the quadrilateral polygons and ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 30 • S. Schaefer and J. Warren Fig. 4. Subdivision for triangle/quad meshes as centroid averaging. Centroids of each type of polygon weighted by the angle spanned in the ordinary conﬁguration (left). Averaging rule at ordinary boundary formed from centroids (right). another for the triangular polygons. This replication essentially “unzips” the mesh into disjoint pieces consisting of only triangles or only quads. The replicated vertices for the quadrilateral polygons retain their original positions; however, the vertices along the boundary for the triangular polygons have the mask in Figure 3 (right) applied to them. When subdivision is performed, the new vertices on the quadri- lateral portions of the mesh use only the quadrilateral vertices, while new vertices on the triangular portions of the mesh use only triangular vertices. The vertices actually on the triangle/quad boundary use only the original vertices of the mesh. Levin and Levin [2003] then prove that these modiﬁed rules generate a surface that is C 2 across the triangle/quad boundary. As part of their proof, the authors present a sufﬁcient test for C 2 smoothness based on a joint spectral radius calculation between two subdivision matrices, and show that their modiﬁcations generate a C 2 subdivision scheme. However, this subdivision scheme can be difﬁcult to apply in practice due to the special rules introduced along the triangle/quad boundary, which also have larger support than the C 1 rules and do not readily ﬁt into the averaging subdivision framework. 2. THE UNIFIED SUBDIVISION SCHEME Our implementation of triangle/quad subdivision uses the centroid averaging approach described by Warren and Schaefer [2003]. This method proceeds in two stages: linear subdivision and centroid averaging where linear subdivision is identical to Stam and Loop’s [2003] linear subdivision pass. The output of linear subdivision is a reﬁned mesh taken as input to the averaging pass. In this pass, the authors provided a different extension from Stam and Loop’s [2003] scheme to extraordinary vertices contained by both quads and triangles using a centroid averaging approach. For each vertex in the input mesh, averaging repositions that vertex to its ﬁnal location after one round of subdivision. To reposition a vertex, averaging ﬁnds all polygons containing that vertex and computes the set of centroids shown in Figure 4 for those polygons. For quads, the centroid is simply 1 of each of its vertices, summed 4 together. For triangles, the centroid is calculated using 1 of the vertex being repositioned and 3 of the 4 8 edge adjacent vertices. Next, a weighted average of these centroids is computed to reposition the central π vertex by weighting each centroid by the angle spanned by that polygon in the ordinary case ( 2 for quads and π for triangles). This centroid averaging rule reproduces Stam and Loop’s [2003] averaging rule 3 along the ordinary triangle/quad edge, but provides a different generalization to other conﬁgurations of polygons. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. On C 2 Triangle/Quad Subdivision • 31 Fig. 5. Unzipping mask for the vertices part of the triangle/quad boundary in the ordinary case (left) and the arbitrary case (right). n is the number of edges incident to the vertex that are part of the network of triangle/quad boundary edges. Fig. 6. Initial shape (upper left). Unzipped shape (upper right). Linear subdivision (bottom left). Averaging pass zips mesh back together (bottom right). Like Levin and Levin [2003], we utilize an unzipping mask during subdivision and augment the triangle/quad subdivision scheme of Warren and Schaefer [2003] with an unzipping pass. However, our unzipping mask differs from Levin and Levin’s choice and is shown in Figure 5. We have also extended our unzipping masks to arbitrary conﬁgurations of edges that are part of the triangle/quad boundary, which allows us to subdivide a greater variety of surfaces. Prior to subdivision, we identify edges on the surface contained by both triangles and quads. These edges deﬁne a network of curves on the surface. Then we apply the unzipping masks (Ut , Uq ) to this curve network to generate separate triangle and quad vertices along the triangle/quad boundary (we also designate vertices contained completely by triangles or completely by quads to be triangle and quad vertices, respectively). Next, we apply linear subdivision and averaging to the resulting points. Our only modiﬁcation that we make to Warren and Schaefer’s [2003] scheme is that we require that each centroid is calculated using vertices only of the same type as the polygon; that is, triangle centroids are calculated using only triangle vertices and, similarly, quad centroids are calculated using only quad vertices. This small modiﬁcation generates surfaces that are C 2 across the ordinary triangle/quad boundary, which we prove in Section 3. Furthermore, these changes also extend the subdivision scheme to arbitrary surfaces such as nonmanifold surfaces. The entire subdivision process is depicted in Figure 6. Starting with an initial shape, we ﬁrst unzip the surface into disjoint pieces, consisting of entirely triangles or entirely quads, by applying the masks in Figure 5. Next, we perform linear subdivision on the separate pieces. Finally, we close the surface back together by performing averaging, which completes one round of subdivision. Figure 7 illustrates a more complicated example of a mouse brain subdivided using this method. This model is actually composed of a network of surfaces that partition the brain into different anatomical regions. Therefore, edges where multiple anatomical regions meet are nonmanifold. The surface itself is ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 32 • S. Schaefer and J. Warren Fig. 7. Initial tri/quad surface of a mouse brain and the subdivided model (top). In the bottom, the cerrebellum was extracted and subdivided with crease edges highlighted on the surface. Fig. 8. Coefﬁcients of the Box spline reproducing x 2 (left), xy (middle) and y 2 (right). The boundary vector for the quadrilateral and triangular side are highlighted. composed of triangles and quads subdivided using our method. On the bottom of the ﬁgure, we extracted the cerrebellum and subdivided the model with the triangle/quad structure explicitly shown. 3. ANALYSIS 3.1 Necessary Conditions Given an ordinary triangle/quad boundary (shown in Figure 8), we deﬁne S to be the subdivision matrix for Stam and Loop’s [2003] scheme, formed by centroid averaging. For the subdivision scheme to be C 2 in the functional sense, S must satisfy Sz i = λi z i (1) where λi = 1, 1 , 1 , 1 , 1 , 1 and z i are the corresponding eigenvectors producing the polynomials 1, x, y, 2 2 4 4 4 x 2 , xy, y 2 [Warren and Weimer 2001]. While S satisﬁes equation 1 for i = 1 . . . 5, S fails that condition for z 6 (corresponding to y 2 ). Figure 8 shows the coefﬁcients of the vertices that reproduce the quadratic polynomials x 2 , xy, and y 2 over the triangular and quadrilateral portions of the mesh. Notice that the coefﬁcients for y 2 do not agree at the boundary, so S cannot possibly be C 2 at the boundary. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. On C 2 Triangle/Quad Subdivision • 33 ˆ ˆ Our goal is to construct a new subdivision scheme S such that S satisﬁes equation 1 for i = 1 . . . 5, ˆ and that Sv = 1 v where v is a new eigenvector corresponding to y 2 . To analyze the case of y 2 further, 4 let vt be the coefﬁcients that reproduce y 2 on the triangle vertices and the boundary, but zero on the quadrilateral vertices. Similarly, let vq be the coefﬁcients reproducing y 2 on the quadrilateral vertices and the boundary, but zero on the triangle vertices. We deﬁne our new eigenvector v to be of the form vt triangle vertices v = αvt + (1 − α)vq boundary (2) vq quad vertices We now construct unzipping matrices Ut and Uq such that vt triangle vertices and boundary Ut v = 0 quad vertices 0 triangle vertices Uq v = vq boundary and quad vertices Using Equation 2, we solve for the unzipping matrices as 1 triangle vertices 1−α 11+α 1−α Ut = ( , 12 , 24 ) boundary 24 0 quad vertices 0 triangle vertices −α 12+α −α Uq = ( , , ) boundary 24 12 24 1 quad vertices Notice that Levin and Levin’s [2003] choice for the unzipping mask corresponds to α = 0. Levin and Levin’s special boundary rules were then chosen to satisfy equation 1 for that particular choice of α. With these deﬁnitions we now partition S into the form S = St + Sq where St and Sq are formed by centroid averaging on the triangle and quad portions of the mesh, respectively. St and Sq then satisfy 1 4 vt triangle vertices 1 St vt = vt boundary 8 0 quad vertices 0 triangle vertices Sq vq = 1 vq boundary 8 1 v 4 q quad vertices Notice that, on the boundary, St and Sq produce 1 vt and 1 vq , respectively, because the subdivision 8 8 matrices contain halfmasks (shown in Figure 9), formed from centroid averaging. ˆ Our subdivision scheme S is then represented as ˆ S = St Ut + Sq Uq . ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 34 • S. Schaefer and J. Warren Fig. 9. Mask calculated through centroid averaging on each side of the triangle/quad boundary. The mask is exactly half of the regular mask for triangular or quadrilateral surfaces. ˆ Applying S to v yields ˆ Sv = St Ut v + Sq Uq v = t vt + Sq vq S 1 v 4 t triangle vertices = 1 vt + 1 vq boundary 8 1 8 4 q v quad vertices The ﬁnal piecewise deﬁnition corresponds to exactly 1 v. Since Ut and Uq do not modify the boundary 4 ˆ for the eigenvectors z i for i = 1 . . . 5 and Sv = 1 v, S satisﬁes the necessary conditions for C 2 continuity ˆ 4 at the boundary. Finally, our scheme has a simple interpretation in terms of averaging; however, any choice of α > 0 yields an unzipping mask with negative coefﬁcients. Therefore, our choice of subdivision mask does not yield surfaces that necessarily lie within the convex hull of the control points. Levin and Levin [2003] chose their unzipping mask and subsequent subdivision rules on the boundary such that their surfaces satisfy this convex hull property. 3.2 Sufﬁcient Conditions To analyze the smoothness of the subdivision scheme that we present, we use a sufﬁcient test described by Levin and Levin [2003]. This smoothness test requires that the subdivision scheme is C 2 away from the boundary edge, and that the subdivision matrix for a point on the boundary satisﬁes the necessary conditions from Section 3.1. Furthermore, the subdivision scheme along the edge must satisfy a joint spectral radius condition. To perform the joint spectral radius test, we require two subdivision matrices (A and B) that map an edge L on the boundary to two smaller edges (L1 and L2 ) after one round of subdivision. The matrices A and B should contain all of the vertices that inﬂuence the surface over the edges L1 and L2 . Next, we ﬁnd a diagonalizing matrix W such that C0 W −1 AW = 0 Y0 (3) −1 θ C1 W BW = 0 Y1 where is a diagonal matrix with the entries 1, 1 , 1 , 1 , 1 , 1 , and θ is an upper-triangular matrix with 2 2 4 4 4 the same diagonal entries as . Finally, we use Y 0 and Y 1 to compute 1 ρ [k] (Y 0 , Y 1 ) = (Max Y k Y k−1 ...Y 1 ∞) k where i ∈ {0, 1}. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. On C 2 Triangle/Quad Subdivision • 35 Fig. 10. Curvature plots of the ﬁnely subdivided shape from Figure 6 for Stam and Loop’s [2003] scheme (bottom left) and our modiﬁed scheme (bottom right). Fig. 11. Characteristic maps for several different conﬁgurations of quads and triangles. According to Levin and Levin [2003], if there exists a k such that ρ [k] < 1 , then the subdivision scheme 4 is C 2 at the boundary. The obvious choice for constructing the matrix W is to simply use all of the eigenvectors of A. However, this approach can be numerically unstable if the matrix has small eigenvalues. Levin and Levin [2003] suggest that W be formed from the right eigenvectors associated with the eigenvalues from and a basis of the null space from the corresponding left eigenvectors. Since symbolic math packages such as Mathematica can generate the eigenvectors corresponding to exactly, this method yields a numerically stable method for creating W . When applying the spectral radius technique to our subdivision scheme, we calculated ρ [8] = 0.229725. Since ρ [8] < 1 and our scheme satisﬁes the necessary conditions for polynomial generation, we conclude 4 that our subdivision scheme is C 2 at triangle/quad boundaries. Figure 10 shows a curvature plot of a highly subdivided model from Figure 6. Notice the color discontinuity at the triangle/quad boundary in Stam and Loop’s [2003] scheme, where as our modiﬁcation generates continuous curvature at the boundary. 3.3 Extraordinary Vertices Notice that this unzipping pass changes the subdivision rules at extraordinary vertices contained by both triangles and quads from the original triangle/quad subdivision scheme. Therefore, the smooth- ness results of the original scheme do not apply at these vertices and must be established separately. We provide no formal proof of smoothness here. However, we have enumerated through all possible conﬁgurations of quads and triangles for vertices of valence 3 through 10 and checked the eigenvalues as well as the characteristic map for these vertices. The eigenvalues all satisfy the necessary condition from Zorin [2000] for the scheme to be C 1 . The characteristic maps computed at these vertices also appear to be regular and injective. Several of these maps are shown in Figure 11, but we offer no proof at arbitrary valences. We have also inspected surfaces in these conﬁgurations and they appear visually smooth as well. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 36 • S. Schaefer and J. Warren 4. CONCLUSION We have presented a subdivision scheme for mixed triangle/quad surfaces that is C 2 everywhere except for isolated, extraordinary vertices. The subdivision scheme itself is the same as Stam and Loop’s [2003] triangle/quad scheme except that we perform an unzipping pass before subdivision. Our choice of the unzipping mask does not yield special rules in the implementation and lends itself to real-world applications as the method is easy to code since it is a simple extension of ordinary triangle/quad subdivision. Furthermore, our subdivision scheme provides rules for handling arbitrary triangle/quad surfaces including nonmanifold surfaces. Finally, we applied Levin and Levin’s [2003] sufﬁciency test for C 2 smoothness to prove our modiﬁcation to the subdivision produces C 2 surfaces at the triangle/quad boundary. REFERENCES CATMULL, E. AND CLARK, J. 1978. Recursively generated b-spline surfaces on arbitrary topological meshes. Comput. Aided Design 10, 350–355. LEVIN, A. AND LEVIN, D. 2003. Analysis of quasi uniform subdivision. Applied Computat. Harmon. Analy. 15, 1, 18–32. LOOP, C. 1987. Smooth subdivision surfaces based on triangles. University of Utah, Department of Mathematics, Masters Thesis. STAM, J. 2001. On subdivision schemes generalizing uniform b-spline surfaces of arbitrary degree. Comput. Aided Geomet. Design 18, 383–396. STAM, J. AND LOOP, C. 2003. Quad/triangle subdivision. Comput. Graph. For. 22, 1, 1–7. WARREN, J. AND SCHAEFER, S. 2003. A factored approach to subdivision surfaces. Submitted to Comput. Graph. Applicat. WARREN, J. AND WEIMER, H. 2001. Subdivision Methods for Geometric Design. Morgan Kaufmann. ZORIN, D. 2000. Smoothness of subdivision on irregular meshes. Construt. Approx. 16, 3, 359–397. ¨ ZORIN, D. AND SCHRODER, P. 2001. A uniﬁed framework for primal/dual quadrilateral subdivision schemes. Comput. Aided Geomet. Design 18, 5, 429–454. Received September 2003; revised May 2004; accepted August 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena GLADIMIR V. G. BARANOSKI and JUSTIN WAN University of Waterloo JON G. ROKNE University of Calgary and IAN BELL University of Waterloo Simulating natural phenomena has always been a focal point for computer graphics research. Its importance goes beyond the production of appealing presentations, since research in this area can contribute to the scientiﬁc understanding of complex natural processes. The natural phenomena, known as the Aurora Borealis and Aurora Australis, are geomagnetic phenomena of impressive visual characteristics and remarkable scientiﬁc interest. Aurorae present a complex behavior that arises from interactions between plasma (hot, ionized gases composed of ions, electrons, and neutral atoms) and Earth’s electromagnetic ﬁelds. Previous work on the visual simulation of auroral phenomena have focused on static physical models of their shape, modeled from primitives, like sine waves. In this article, we focus on the dynamic behavior of the aurora, and we present a physically-based model to perform 3D visual simulations. The model takes into account the physical parameters and processes directly associated with plasma ﬂow, and can be extended to simulate the dynamics of other plasma phenomena as well as astrophysical phenomena. The partial differential equations associated with these processes are solved using a complete multigrid implementation of the electromagnetic interactions, leading to a simulation of the shape and motion of the auroral displays. In order to illustrate the applicability of our model, we provide simulation sequences rendered using a distributed forward mapping approach. Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimension Graphics and Realism General Terms: Algorithms Additional Key Words and Phrases: Atmospheric effects, natural phenomena, plasma phenomena, rendering 1. INTRODUCTION The Aurora Borealis, or “Northern Lights”, and Aurora Australis, its southern counterpart, are light emissions caused by the collision of an ionized stream of charged particles with high altitude atmo- spheric atoms and molecules [Eather 1980]. These emissions are considered to be among the most fascinating and mysterious of Nature’s spectacles by those who have been fortunate enough to see them. Taylor in National Geographic calls them “Earth’s Grand Show of Lights” [Taylor 2001]. An inherent characteristic of auroral displays is that they move and can change shape both slowly and Funding is gratefully acknowledged from both the National Sciences and Engineering Council of Canada (NSERC Grants 213281, 239162, and 691266) and the Canada Foundation for Innovation (CFI Project 6218). Author’s address: G. V. G. Baranoski, Natural Phenomena Simulation Group, School of Computer Science, University of Waterloo, 200 University Avenue West, Waterloo, ON N2L 3G1, Canada. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0037 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 37–59. 38 • G. V. G. Baranoski et al. rapidly. These complex and stochastic motions are directly related to the plasma processes. In this article, a novel physically-based model to perform 3D visual simulations of the dynamics of the auroral phenomena is presented. Furthermore, auroral surges (Figure 14), which have not been previously simulated, are also rendered. This model, therefore, enables the simulation of shape and motion of the aurorae in a realistic manner. The aurorae are frequently visible at high latitudes. They can also be seen less frequently at lower latitudes, especially around solar sunspot maxima (a period of great solar turbulence that occurs ap- proximately every 11 years [Odenwald 2000]). For instance, the last solar cycle of the 20th century reached its maximum in the year 2000, and it produced amazingly bright aurorae for the next couple of years. Aurorae are also a prominent visual phenomena in space as seen from spacecrafts orbiting the Earth. Further afar, they occur on planets with strong magnetic ﬁelds such as Jupiter and Saturn. Hence, the simulation of auroral displays is also of interest for the production of night sky visualiza- tion sequences used in planetarium shows. Moreover, the algorithms used in our plasma-model may be applied to the simulation of other plasma-related phenomena, such as gas discharges encountered in neon and ﬂuorescent lights, welding arcs, and lightning. On the scientiﬁc side, the importance of the visual simulation of auroral dynamics is related to the links to plasma physics. As much as 99% of the matter in the universe is thought to exist in a plasma state [Bittencourt 1986; Chen 1984; NRC 1995]. The auroral displays provide a natural laboratory in which the complex workings of the plasma universe can be studied. Therefore, our simulations can be used in the evaluation of auroral theories as well as in the visualization of other plasma phenomena. Furthermore, plasma kinetic theory can be used to study the development of galaxies, since, although stars in a galaxy are not charged, they behave like particles in a plasma [Bittencourt 1986; Chen 1984; NRC 1995]. Another area of application might be the study of the effects of nuclear explosions in the upper atmosphere. These explosions were found to create aurora-like displays [Dupont 2004]. Yet the main motivation for this research is the spellbinding majesty of a moving auroral display [Taylor 2001]. 1.1 Related Work The spectacular auroral motions and shape changes involve to some extent shear, meaning differential motions within different parts of a ﬂuid or gaseous material. These motions are common in nature. Shear eddies occur, for instance, on the surface of water in streams and in smoke curling up from a ﬁre, and they are responsible for clear-air turbulence and gusty winds [Davis 1992]. The shear found in the aurorae is associated with plasma processes, and the most fascinating consequences are the formation of rotational distortions having different spatio-temporal scales. Blinn and Wolf [Upson et al. 1987] produced a ﬁlm in 1983 entitled “Jupiter’s Magnetosphere: The Movie”. In this ﬁlm, representations of spacecraft observations as well as analytic models of the plasma and magnetic ﬁelds in the Jovian system were employed to visualize the morphology and dynamical structure of Jupiter’s magnetosphere. To the best of our knowledge, this ﬁlm represents the ﬁrst use of a plasma model by the graphics community. Recent works involving plasma models in the graphics literature focus on the visualization of plasma phenomena occurring in laboratory experiments. For ex- ample, Parker et al. [1995] developed a gyrokinetic particle simulation to visualize a plasma turbulence in a Tokomak (an apparatus for producing controlled fusion reaction in hot plasma), and Ljung et al. [2000] presented a system for the animation of the evolution of an instability that gives rise to a plasma surfatron, a mechanism for particle acceleration of importance in particle accelerators, in the context of electron acceleration in astrophysical shocks and in the solar corona. It is relevant to note the work of Schussman et al. [2000] who presented techniques for the visualization of magnetic ﬁeld data from the DIII-D Tokomak, a fusion facility that conducts both experimental and computational plasma research. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 39 Recently, Baranoski et al. [2000, 2003] presented a static physically-based algorithm for the render- ing of auroral displays which accounts for important auroral visual features such as their characteristic spectral and intensity variation. Their work, however, does not take into account the electromagnetic in- stabilities responsible for the time varying behavior of the aurorae. Nonetheless, it provides an accurate framework for the rendering of auroral images. In this article, the dynamical behavior of the aurorae is modeled by taking into account physical parameters and processes directly associated with plasma ﬂow. The set of partial differential equations (PDEs) associated with these processes is solved using a complete multigrid implementation of the electromagnetic interactions between charged particles in the ionosphere, enabling us to produce a visual display of the “dancing lights”. The simulation of natural phenomena involving shear and ﬂuid ﬂow is an active area of research in computer graphics [Foster and Metaxas 1997; Foster and Fedkiw 2001; Kajiya and Herzen 1984; Myazaki et al. 2001; Stam and Fiume 1993; Stam 1999]. A comprehensive review of computer graph- ics simulations of ﬂuid ﬂow is beyond the scope of this article; however, we should remark that the plasma instability responsible for the stochastic and complex nature of the aurorae is similar to the Kelvin-Helmholtz instability (KHI) for sheared ﬂuids [Tajima 1989]. In spite of that, we believe that a more natural way to model plasma phenomena is to simulate the equations of plasma ﬂow directly. The use of a plasma model allows the control of the simulations by auroral physically meaningful parame- ters. Moreover, the use of a plasma model enables the accurate reproduction of the rapid intermediate scale motions often seen in the aurorae such as the motion of rays and folds along auroral forms. Recent works on night sky phenomena have focused on obtaining images that are not only realistic, but are also predictive. This objective is achieved by adhering to physical principles and incorporating measured data to the simulations. After all, it is far easier to control a simulation using familiar phys- ical concepts than through arcane parameters as appropriately stated by Arvo [1995]. As illustrative examples of this trend, besides the work by Baranoski et al. [2000, 2003] mentioned above, we should note the works by Nadeau et al. [2001] and Jensen et al. [2001]. Nadeau et al. [2001] presented star and nebula visualization techniques for creating a 3D volu- metric representation of the Orion Nebula, which was used in a planetarium daily show. Jensen et al. [2001] developed a physically-based model of the night sky which also has educational ap- plications. Although the Jensen et al. model does not include auroral displays, their night sky im- ages can be combined with our auroral simulations. They also highlighted the role of naturally il- luminated night scenes in the history of image-making and their importance in the area of realistic rendering. Simulating plasma instabilities as well as ﬂuid instabilities involves the solution of systems of PDEs. In these situations, the production of simulation sequences used to depict time dependencies or to analyze motion requires that such systems be solved repeatedly. Usually standard numerical methods, such as the Conjugate Gradient (CG) and Fast Fourier Transform (FFT)[Golub and Loan 1989], are applied. Stam [1995, 1999] has highlighted the theoretical advantages of multigrid methods [Hackbusch 1985] for solving such systems. In order to solve the PDEs arising from our plasma simulations, we present a practical and fast multigrid implementation. We note that our multigrid implementation is matrix-free, that is, it does not store the coarse grid matrices. Previous multigrid implementations found in the computer graphics literature, such as the one by Witkin and Kass [1991] in the their texture synthesis algorithm, used coarse grids to speed up the computations of explicit steady state solutions. Here, we use multigrid in the context of time dependent problems, and in each implicit time step, the multigrid is applied to solve a linear system. It also worth noting that Weimer and Warren [1999] used a full multigrid approach for ﬂuid ﬂow which is appropriate for solving nonlinear equations. In our implementation, we used a V-cycle multigrid which is more appropriate for solving linear equations such as those arising in our simulations. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 40 • G. V. G. Baranoski et al. Fig. 1. Satellite snapshot of an auroral oval around the Earth’s north magnetic pole. a) Viewed from the satellite position and b) viewed from a position normal to the Earth’s north pole. (Courtesy of NASA Polar Ultraviolet Imager.) 1.2 Organization The next section provides an overview of relevant auroral physics concepts and describes the main motions and shape changes of the aurorae. In Section 3, the modeling framework used in this research is outlined. Section 4 presents the algorithms used to simulate the dynamics of the auroral phenomena. The rendering issues are addressed in Section 5. Section 6 presents the results of our simulations and discusses performance issues. Section 7 outlines applications and directions for future work. Finally, Section 8 closes the article. 2. AURORAL PHENOMENA In this section, we outline the main physical aspects directly related to the simulation of auroral dynam- ics. A reader interested in a detailed description of auroral physics is referred to more comprehensive texts on these phenomena [Brekke and Egeland 1994; Davis 1992; Eather 1980; Jones 1974; Omholt 1971]. The diagrams of auroral distortions presented in this section use as reference an auroral local co- ordinate system represented by xyz, where the z-axis corresponds to the direction parallel to the Earth’s magnetic ﬁeld vector B, in the upwards direction, but not necessarily perpendicular to the ground. 2.1 Overview The particles responsible for the colorful auroral displays are electrons that originate from solar ﬂares and coronal mass ejections and become the “solar wind” [Burtnyk 2000]. After interacting with the Earth’s magnetosphere, these particles migrate along the magnetic ﬁeld lines, and “dive” towards oval regions around the Earth’s magnetic poles under certain conditions. These regions are called the auroral ovals. Figure 1 shows a satellite snapshot of an aurora oval around the Earth’s north magnetic pole. The shape of an aurora is determined by the energy and density of the electrons entering the at- mosphere as well as the local variations in the Earth’s magnetic and electric ﬁelds. The most common aurorae can be thought of as “curtains” of light emissions from “sheets” of falling electrons, which move along the Earth’s magnetic ﬁeld. As the electrons travel down along the Earth’s magnetic ﬁeld lines, they suffer many random de- ﬂections which are caused by collisions with atoms of atmospheric gases such as oxygen and nitrogen. These deﬂections spread the electrons out horizontally. When the electrons collide with atmospheric ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 41 Fig. 2. Line drawings of cross sections of auroral arcs (not to scale) illustrating the three major shear-type rotational distortions: a) curls, b) folds, and c) spirals. The symbol denotes a vector normal to the plane of the paper. constituents, the atoms are excited, and after a period of time, they may emit a photon. Statistically, several collisions must occur before a photon is emitted. As a result of these photon emissions, the curtains may be colored, bright yellow-green at the bottom, perhaps red at the top, and a yellow-orange transition may also be present. Often they form arcs that can extend over 2000km whose width (thick- ness) may vary from several hundred meters to several kilometers. These arcs can bend and fold, almost like real curtains, thereby generating bright vertical streaks. 2.2 Auroral Shear-Type Distortions The sheets of precipitating electrons form an auroral stream that is subject to instabilities responsible for the shear-type rotational distortions seen in auroral arcs. Hallinan and Davis [1970] identiﬁed the three distinct types of distortions: curls, folds and spirals (Figure 2). Curls are small scale and have a diameter of 1km or less. Folds are intermediate scale distortions and have a diameter of approximately 20km or less. Spirals, or surges, are the largest auroral distortions and diameters are typically around 20–1300km [Partamies et al. 2001]. In addition to the different spatial scales, these phenomena have different temporal scales. Curls have a lifetime of a fraction of a second, while folds can exist for more than a second, and spirals may last minutes. Curls and folds are responsible for distinct visual features present in auroral displays. The light emanating from convoluted folds in auroral displays often creates the impression of vertical, or near- vertical, “stripes” [Bryant 1999]. Despite their highly transient nature, curls are largely responsible for another important auroral feature, namely, electron beams evolving into thin ﬁeld-aligned ﬁlaments or “rays”. An auroral ray typically has a vertical dimension up to several hundred kilometers. Curls are usually 2–10km apart, while folds are approximately 20km apart. Spirals are also relatively common auroral structures, visible in more than 30% of the Defense Meteorological Satellite Program observa- tions1 [Davis and Hallinan 1976]. They can form a street of two or more spirals approximately 100km apart. The spirals in the street usually have similar conﬁguration and size, but they may also grow monotonically along the auroral display [Davis and Hallinan 1976]. 2.3 Auroral Motions As pointed out by Davis [1992], auroral motions and shape changes seem bewildering, yet they present some order when examined in detail. Many of the motions and shape changes associated with the auroral distortions are caused by warping of the incoming particle streams by the attendant magnetic 1 http://www.ngdc.noaa.gov/dmsp/dmsp.html. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 42 • G. V. G. Baranoski et al. Fig. 3. Line drawings illustrating an auroral arc break-up. The symbol denotes a vector normal to the plane of the paper. Fig. 4. Sketch showing different stages of an auroral warping process: a) initial arc, b) fold, c) further distorted fold, and d) spiral array. The symbol denotes a vector normal to the plane of the paper. and electric ﬁelds. In this section, we outline the most important of these changes according to their spatial scale. 2.3.1 Small Scale. When an electron particle stream responsible for producing a small scale struc- ture becomes slightly more dense in a region, it is likely to become contorted and develop a curl [Davis 1992]. The curl formation process is similar to the spiral formation process and leads to essentially the same ﬁnal conﬁguration. However, the entire curl formation can occur so rapidly (0.25–0.75s [Trondsen 1998]) that an observer on the ground has difﬁculty following the process. 2.3.2 Intermediate Scale. Once a curl vortex develops, it tends to cause the formation of another curl a short distance along the arc. The net result is the development of an array of evenly spaced curls along the arc, manifesting as rays. If another arc is nearby, the rays come under the inﬂuence of its electrical ﬁeld and, therefore, move along the arc [Davis 1992]. If the process that leads to curl formation continues a bit longer, then small, uniformly spaced folds occur, and sometimes the arc may split into two separate arcs as the folds break apart (Figure 3). Folds have apparent horizontal velocities in the range of 0–5km/s [Hallinan and Davis 1970]. The apparent horizontal velocities of curls lie in the range of 0–90km/s, with preferred speeds in the range of 0–8km/s [Trondsen 1998]. To have a better idea of the magnitude of these speeds, recall that an auroral arc may extend over the entire ﬁeld-of-view of an observer on the ground. For this observer the auroral rays seem to “travel” back and forth between two extreme points in the horizon in-few seconds. 2.3.3 Large Scale. When an incoming stream of electrons that produces an auroral arc develops a slight irregularity, the irregularity can grow [Hallinan 1976, 1981; Levy and Hockney 1968; Partamies et al. 2001]. As it does, the stream becomes distorted, carrying incoming particles to new locations in the auroral domain [Davis 1992]. The growth of this irregularity depends on the thickness of the electron stream. The larger the stream length-to-thickness ratio, the faster the irregularity grows with the distance [Hallinan 1976; Levy and Hockney 1968]. Figure 4 shows an auroral arc being affected by ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 43 the warping process, ﬁrst developing a fold, and then progressing into a complex spiral shape. These large auroral vortices have an average speed of 4km/s [Partamies et al. 2001]. 3. MODELING FRAMEWORK OVERVIEW Computer models of plasma can be broadly divided in three kinds: Eulerian, Lagrangian and hybrids of these [Hockney 1966]. The Eulerian model assumes that plasma behaves like a ﬂuid, or that it obeys macroscopic differential equations for average quantities. The Lagrangian model follows the motion of a large number of particles as they move in time under their mutual electromagnetic interactions. Although this model acts on the microscopic particle level, it allows the particle ensemble to “make its own mind” about the macroscopic and collective behavior. This aspect is relevant in the simulation of phenomena with noticeable stochastic traits and different spatio-temporal scales such as aurorae. We use a hybrid Eulerian-Lagrangian approach in our modeling of auroral phenomena. Such a hybrid approach has been successfully used in computational physics simulations for many years [Hockney 1966; Levy and Hockney 1968; Hockney and Eastwood 1988]. In many scientiﬁc ﬁelds, the behavior of natural phenomena is described using PDEs. Recently, PDE-based modeling and numerical techniques from computational ﬂuid dynamics (CFD) have been applied to computer graphics, especially for animations involving ﬂuids. For instance, in Fedkiw et al. [2001], smoke was modeled by inviscid Euler equations. Level set and semi-Lagrangian techniques were applied in the numerical computation. In Rasmussen et al. [2003], large scale smoke phenomena were modeled by 2D incompressible Euler equations, together with the Kolmogorov spectrum. In Kunimatsu et al. [2001], water was modeled by Navier–Stokes equations. The Eulerian approach was adopted and the volume-of-ﬂuid method was used to track free water surfaces. Another approach is to use semi- Lagrangian and level set method with inertialess particles [Foster and Fedkiw 2001]. The particle level set technique was further modiﬁed in Enright et al. [2002] to obtain more accurate results. In Nguyen et al. [2002] and Feldman et al. [2003], ﬁre and explosion were modeled by incompressible Euler equations. The latter used nonzero velocity divergence to capture ﬂuid expansion during explosion. Cloud simulation was modeled by Boussinesq’s equations in Miyazaki et al. [2002]. At ﬁrst glance, the similarities between neutral ﬂuids and plasma [Tang 2002] might suggest that the CFD formulation would be sufﬁcient to model the behavior of both media. However, in order to perform predictive simulations of the dynamics of plasma phenomena, one must account for their distinctive features. For example, turbulence dynamics in plasmas involve electromagnetic ﬁelds and a much larger number of relevant variables than for neutral ﬂuids [Yoshizawa et al. 2001]. Furthermore, while the effects of global boundary conditions, such as wall boundaries, often play an important role in the growth of neutral ﬂuid disturbances [Drazin and Reid 1981; Hinze 1975; Yoshizawa 1998], plasma turbulence is greatly affected by spatial inhomogeneities and plasma conﬁgurations. These inhomogeneities are coupled together to drive or suppress turbulent ﬂuctuations, which often have a very long correlation length along the magnetic ﬁeld line and are quasi-two-dimensional [Yoshizawa et al. 2001]. Hence, to predictively simulate structure formation and transitions in plasma phenomena, such as the aurorae, one needs to account for the mutual interactions between plasma inhomogeneities and the electric-ﬁeld structure and ﬂuctuations. Although the physical laws that affect the behavior of plasma phenomena, such as the Maxwell’s equations, are well known, progress in this area was only made possible by advances in computational methods [Gombosi et al. 2004; Tang 2002]. Generally, an analytical, closed-form solution for these equations is difﬁcult to obtain. One can only obtain a unique solution by imposing various restrictions [Volakis and Kempel 1995], and they are normally solved using numerical methods such as ﬁnite difference (FD) [Taﬂove 2000], ﬁnite volume (FV) [Anderson 2001], and ﬁnite element (FE) [Silvester and Ferrari 1990] methods and hybrids of these [Volakis and Kempel 1995]. Among these methods, ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 44 • G. V. G. Baranoski et al. FD’s are the most commonly applied to the PDE formulation of Maxwell’s equations in the time domain [Anderson 2001]. This method is based on the straight discretization of the differential equations, and ¨ ¨ it is suitable for modeling complex media [Karkkainen 2002]. This motivated its use in the simulation algorithms described in this article. In order to accelerate the convergence of the iterative solutions of the PDEs, we used a multigrid method. Recently, Aruliah and Asher [2002] also used a multigrid scheme to speed-up the simulation of three-dimensional electromagnetic problems in geophysical regimes. While in our simulations multigrid is used to solve the Maxwell’s potential equation directly, Aruliah and Asher [2002] used a multigrid scheme to approximately invert a block diagonal matrix, which, in turn, was used as a preconditioner for the discretization matrix. The “electrons” in a plasma simulation correspond to millions of physical electrons. Provided that the physical phenomena have wavelengths that are long, compared with the average electrons spacings, and time scales that are short, compared with the time for the graininess to have signiﬁcant effect, this description of “electrons” gives an accurate representation and leads to the concept of computational “superparticle” [Hockney and Eastwood 1988]. The graininess introduced by the paucity of superpar- ticles is minimized by smoothing the short-range forces. One way of regarding the superparticles is as ﬁnite-sized clouds of electrons, where the position of a superparticle is the center of mass of the clouds and their velocities the mean velocities of the clouds [Hockney and Eastwood 1988]. We used a similar representation in our simulations in order to minimize computational overhead. Instead of tracking the around 1012 individual plasma particles that cause auroral phenomena, we follow the path of superparticles which we regard as beams of electrons. As such, we use normalized values for the physical quantities involved in our simulations, namely, charge, mass, and permissivity. The Lagrangian treatment is used to account for the interactions among beams and between beams and atmospheric constituents. The Eulerian treatment is applied when spatial scales much greater than the relatively small amplitude motions of the beams are considered. The usual approach in computer experiments is to devise a model which is sufﬁciently detailed to reproduce the important physical aspects faithfully and yet not so detailed that the calculations become impractical. This approach becomes even more pertinent when the main purpose is to produce visual simulations whose purpose is to highlight the qualitative features of a given phenomenon. Selecting the “best” modeling approach depends on the relevant physical length and time scales. The motion of the electron beams has two components: parallel and perpendicular to the electron stream. The parallel component is associated with the kinetic energy of the incoming particles. The perpendicular component is due to the warping of the electron stream. The differences of the magnitudes of the time scales of these two components [Hallinan 1981; Brekke and Egeland 1994] means that they can be decoupled without incurring a loss of realism in their visual three dimensional representation. The key stage of our modeling framework is the simulation of the warping process, which changes the perpendicular velocity, v⊥ , and the position of electron beams, P , at the top of the electron stream, at each instance of time. The mathematics of this process can be described by an electro-magnetic version of the Kelvin-Helmholtz instability (KHI) [Hallinan 1976, 1981; Levy and Hockney 1968; Partamies et al. 2001]. Section 4 describes the warping algorithm used in our simulations in detail. The position of each beam at the top of the electron stream is considered as the starting point of its descending trajectory. The position update resulting from the warping process is associated with a steady state of incoming particles. A precipitating electron beam is tracked downwards until it be- comes indistinguishable from the ambient electron population. During their descending trajectories, the electron beams may be deﬂected several times due to collisions with atmospheric constituents. Light emissions may occur at these deﬂection points, causing the auroral displays. These emissions are mapped to the screen plane of a virtual camera using a splatting approach [Westover 1991]. Rendering issues are further examined in Section 5. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 45 The auroral dynamics simulation uses the auroral local coordinate system deﬁned in Section 2. The mapping of auroral emissions for rendering purposes uses a global coordinate system represented by XYZ, where the Y-axis corresponds to the direction normal to the ground. 4. DYNAMICS SIMULATION The electron beams interact with one another through the electric ﬁeld, E, generated by their mutual electrostatic repulsion, and the Earth’s magnetic ﬁeld, B. Thus, the Lorentz force on an electron beam ek with charge qk , moving with a perpendicular velocity v⊥ is k F k = qk ( E + v⊥ × B), k (1) and the motion of an electron beam of mass mk is determined by Newton’s law d v⊥ mk k = F k. (2) dt Thus, to update the positions of the electron beams, we need to compute E and B. In the auroral warping simulation, the Earth’s magnetic ﬁeld is modeled as a uniform magnetic ﬁeld of strength | B|. At any instance of time, the electron beams and, hence E, are located in the plane perpendicular to B. By Gauss’ law, E satisﬁes ρ ∇·E= , (3) 0 where 0 is the permissivity of free space, and ρ is the charge density function N ρ(P ) = qk δ(P − Pk ). (4) k=1 Here we model the small mass, low density electron beams with charge qk , at the position Pk , as point particles via the Dirac delta function δ(·). From the Maxwell equations, which describe the dynamics of charged particles interacting with an electromagnetic ﬁeld, we obtain ∇ × E = 0. (5) The electrostatic potential φ is introduced such that E = −∇φ, (6) and the divergence of the potential gradient results in the Poisson Equation given by ρ −∇ 2 φ = , (7) 0 where ∇ 2 φ is the Laplacian of the potential. By solving Equations (2) and (7) over time, we can model the perpendicular motion of the electron beams, which is the crucial aspect of the auroral dynamics. 4.1 Warping Algorithm The potential Equation (7) and the electron beam velocity Equations (2) form a coupled continuous- particle system, which is simulated using the particle-mesh method exploiting the force-at-a-point formulation and the ﬁeld equation for the potential. For the velocity equation, we use a Lagrangian grid approach, whereas for the potential equation, we use an Eulerian approach as suggested by Levy ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 46 • G. V. G. Baranoski et al. Fig. 5. A cross-section of an auroral stream, modeled by a strip of electron beams with periodic and Neuman boundary conditions, along the x-axis and y-axis, respectively. and Hockney [1968]. The perpendicular velocities and positions of the electron beams are advanced in time as follows: (1) from the positions of the electron beams, a charge distribution (ρ) is derived; (2) from the charge distribution, using Poisson Equation (7), the electrostatic potential is calculated; (3) from the potential, the electric ﬁeld ( E) acting on each electron beam is derived using Equation (6); (4) Newton’s law of motion (Equation (2)), using the local electron ﬁeld and the Earth’s magnetic ﬁeld ( B) is then integrated resulting in a new position and velocity for each electron beam. The cycle is then repeated at step (1). The aurora dynamics is initiated from incoming sheets of particles at the auroral oval (Figure 1) forming an auroral stream, which is modeled by a strip of incoming electron beams with periodic and Neuman boundary conditions along the x-axis and y-axis respectively. The strip in Figure 5 is a small cross-section of part of an auroral oval (Figure 1). As an initial condition, the electron beams are distributed randomly in the interior of this strip of thickness γ and length λ, and have zero perpendicular velocity. Faster initial growth of auroral distortions will occur for larger values of the λ/γ ratio. 4.1.1 Charge Assignment. In the hybrid Eulerian-Lagrangian formulation, the ﬁeld and particle quantities are deﬁned on different grids. To obtain the charge density at mesh points from the distri- bution of particles, we allocate the particle charges based on the CIC (cloud-in-cell) [Birdsall and Fuss 1969] charge assignment scheme: the charge density ρi, j , at the grid point G (i, j ) , is given by the sum of weighted charges, ωk qk , of the electron beams ek in the cells surrounding the grid point G (i, j ) , divided by the cell area: 1 ρi, j = ωk qk , cell area e ∈cells with vertex(i, j ) k as indicated in Figure 6(a). However, instead of ﬁnding which electron beams contribute to the charge at the grid point G (i, j ) , we sweep through each electron beam and distribute its charge with weights ωi∗ , j ∗ , i ∗ ∈ {i, i + 1}, j ∗ ∈ { j, j + 1}, to the vertices of the grid cell it belongs to, as indicated in Figure 6(b). k To be consistent with the deﬁnition of the continuous charge density (4), the weight ωi, j must be given k by ωi, j = S (G (i, j ) − Pk ), k where S (·) is a modiﬁed Dirac delta function with compact support parameterized by , which has the property that lim S (·) = δ(·), →0 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 47 Fig. 6. a) Charge accumulation at grid point G (i, j ) . b) Charge distribution of ek to the neighboring grid points. in the sense of distributions. It can be interpreted physically by ascribing a ﬁnite width ( ) to the distribution of charge in each particle. In the CIC scheme, S is given by the hat-shape function [Birdsall and Fuss 1969]. 4.1.2 Poisson Equation. The gradient of the potential function satisﬁes Equation (7). On an Eulerian grid with mesh size h, the partial differential equation can be discretized as 4φi, j − φi+1, j − φi−1, j − φi, j −1 − φi, j +1 ρi, j 2 = , h 0 which leads to a set of linear equations, denoted by A = ρ, where is a vector of unknown potentials and ρ is a vector of (scaled) charge densities. The matrix A is large, sparse, symmetric, and positive deﬁnite. The complexity of most matrix solvers depends on the size of A. For the simulation of auroral dynamics, a ﬁne grid is required to resolve the layer structure, especially when the formation of surges is simulated. Thus, an efﬁcient linear solver is indispensable. Direct methods, such as banded Gaussian elimination (GE), can be used to exploit the banded struc- ture of A. For matrices of size n × n, the complexity of banded GE is O(n2 ), which is still computation- ally impractical. Iterative methods, such as preconditioned conjugate gradient (PCG) with incomplete Cholesky factorization as preconditioner, have been successfully used in computer graphics for ﬂuid simulations [Fedkiw et al. 2001; Foster and Fedkiw 2001]. While the theoretical complexity is reduced to O(n1.5 ), it is still expensive for the long time simulations presented in this article, which require ﬁne grids as stated above. The FFT method [Stam 1999] can improve the complexity further to O(n log2 n), but still it is not optimal in the sense given below. 4.1.3 A Practical Multigrid Implementation. The multigrid method is an optimal scheme in which the convergence rate is independent of the problem size [Trottenber et al. 2001]. Thus, as opposed to the methods previously discussed, it has a linear complexity of O(n), where the complexity constant is known to be small for Poisson equations, typically smaller than 10. It exploits the properties of the linear equations at the PDE level, rather than the matrix structure. It was ﬁrst proposed by Fedorenko [1961] in the 1960’s, and made popular by Brandt [1977] and others for scientiﬁc applications. We refer the interested reader to the survey paper by Chan and Wan [2000] on recent developments of linear elliptic multigrid methods. Early attempts in using multigrid methods to solve a multiple scattering problem [Stam 1995] raised the issue of memory due to storage of coarse grid quantities. Here, we introduce a matrix-free implementation which is extremely memory-efﬁcient. We remark that similar multigrid schemes can also be used in other plasma and astrophysical simulations as mentioned in Section 7. The idea of the multigrid method is to accelerate the convergence of the iterations by solving the PDEs on a sequence of coarse grids. The basic principle is to eliminate high frequency errors by relaxation ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 48 • G. V. G. Baranoski et al. Fig. 7. A multigrid V-cycle: presmoothing, restriction, coarse grid correction, interpolation and post-smoothing. smoothing, and low frequency errors by coarse grid correction. Let h be the current approximate solution on the ﬁne grid with mesh size h, and ε h = − h be the error. By the dissipative nature of the Poisson equation, the error in the numerical solution, ˜ h , obtained from a few relaxation steps, will become smooth. Thus, one can easily compute an accurate coarse grid approximation by projecting the residual error, r h = ρ − A ˜ h , onto the coarse grid with mesh size H and solve the coarse grid error equation A H ε H = Rh r h , H with less computational effort, since there are fewer grid points on the coarse grid. Here A H is the coarse grid matrix and Rh the full weighting restriction operator. Finally, the coarse grid error is then H interpolated back to the ﬁne grid to update the ﬁne grid solution new = ˜ h + PH ε H , h where P H is the linear interpolation operator. This process completes one multigrid V-cycle (Figure 7). h We note that only the actions of the coarse grid, interpolation, and restriction operators are needed. Thus, we never form nor store these matrices. Consequently, only 3.5 ﬁne grid vectors are needed in our implementation, which requires even less memory than CG. For the interested reader, an iteration count comparison between our multigrid method and PCG can be found in the original technical report written by the authors [Baranoski et al. 2002]. 4.1.4 Velocity and Position Update. The electric ﬁeld Ei, j is deﬁned on the Eulerian grid whose value is given by numerical differentiation of φi, j . However, electron beams are located on the Lagrangian grid, which, in general, does not overlap with the Eulerian. Thus, the electric ﬁeld on an electron beam ek , located at Pk , is obtained by interpolating the electric ﬁelds at the nearest four corners, that is, the reverse process of charge distribution as in Figure 6(b). We then update the perpendicular velocities and positions of the electron beams by integrating Equation (2). 4.2 Trajectories of Electron Beams The electron beam’s velocity vector, v, is deﬁned as the overall direction of progression during its de- scending motion. As the beams travel down, they may suffer several random deﬂections. Their trajec- tories are simulated incrementally using the stochastic algorithm described by Baranoski et al. [2003]. This algorithm takes into account the spatial inhomogeneity of auroral electrons and their initial en- ergy to compute the deﬂection points and the displacements of an electron beam along the magnetic ﬁeld lines. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 49 Fig. 8. Sketch showing that the apparent surface brightness of an auroral display is proportional to the number of emissions along the line of sight represented by the vector s. Fig. 9. Diagram showing angular displacements used to distributed emission rays. Each path is simulated incrementally, using a parametric displacement u ∈ [0..1] such that unew = uold + (d u ξ1 ), where ξ1 is an uniformly distributed random number in the interval [0..1], and d u corresponds to a parametric interval (dimensionless) between collisions which is adjusted according to the initial energy of the incoming particles. For instance, an electron with 10 keV (60000km/s) can collide 300 times before becoming indistinguishable from the ambient electrons at an altitude of about 100km above the ground [Brekke and Egeland 1994]. In this case, we could set d u = 300 . 1 5. RENDERING ISSUES Aurorae are view-dependent phenomena, that is, the apparent surface brightness of an aurora is pro- portional to the integrated emission-per-unit volume along the line of sight (Figure 8). Hence, a forward mapping, or splatting, approach can be used to map auroral emissions to the screen plane [Baranoski et al. 2003; Westover 1991]. Recall that after being hit by an electron, an atmospheric atom becomes excited, and, after a period of time, it may emit a photon. Statistically, the intensity contribution spreads radially around the propagation direction, and follows a Gaussian distribution along that dimension [Borovsky et al. 1991; Borovsky and Suszcynsky 1993]. In order to account for this phenomenon, we implemented an algorithm to perform a distributed forward mapping of auroral emissions. The world coordinates for each deﬂection point are used to compute the line of sight vector s. This vector is randomly perturbed through angular displacements α and β (Figure 9) to generate a light emission ray r. The angle α represents the azimuthal angle around s. The angle β represents the polar angle with respect to s, and it corresponds to the exponential glow decay observed in auroral arcs [Borovsky and Suszcynsky 1993]. These angles are ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 50 • G. V. G. Baranoski et al. given by γ ξ2 (α, β) = (1 − e− 240 , 2π ξ3 ), where ξ2 and ξ3 are uniformly distributed random numbers ∈ [0, 1], and γ represents the initial thick- ness of the auroral stream. The accuracy and performance of the simulation is dependent on the number, N , of emission rays. Experiments show, however, asymptotic convergence of the results for larger values of N . The emissions mapped to the screen plane are scaled according to auroral spectral emission and intensity proﬁles [Baranoski et al. 2003], which correlate with the height of the emission point [Brekke and Egeland 1994]. The conversion of auroral spectral emissions to a color space follows a straight- forward CIE XYZ to RGB mapping procedure. We include the strongest auroral spectral emissions in our simulations, namely, 630.0nm (red line), 557.7nm (green line) and 427.8nm (blue line). More wavelengths can be considered at the expense of higher computational time. The different lifetimes of transition states in auroral emissions—110s for the red line, 0.7s for the green line, and 0.001s for the blue line—cause the photon emissions to occur in distinct volumes around the principal direction of emission. In order to simulate this distribution of auroral emissions, we convolve the image with a color-dependent Gaussian low-pass ﬁlter [Baranoski et al. 2003; Westover 1991]. An auroral display also exhibits global temporal variations, captured in photographs as blurred forms, due to ﬁnite exposure times. For simulating this global blurring effect we perform a convolution using a temporal low-pass ﬁlter [Baranoski et al. 2000; Castleman 1996]. The longer the sampled window is in time, the blurrier the results, similar to the effects captured in real photographs with longer exposure times. The ﬁnal stage of our rendering pipeline consists of blending the auroral display with a night sky background and a foreground. We use a composite approach in which we initialize the color buffer with the background scene, superimpose the auroral display, and, ﬁnally, ﬁlter out regions of the image that would be covered by foreground objects. 6. RESULTS In this section, we present examples of simulations of auroral motions and shape changes at different spatio-temporal scales.2 The stochastic nature of auroral displays reduces the viability of a quantitative analysis of their visual simulations. An alternative available is to qualitatively analyze the simulations in comparison with photographs of real auroral displays. However, as mentioned earlier, photographs of an aurora are usually blurred due to low light conditions and signiﬁcant exposure times, and sometimes present background artifacts caused by grain when high speed ﬁlm is used (Figures 10 and 13). For this reason we accounted for photographic blur in our simulations. Despite the blur, features readily recognizable by viewers in real aurorae were preserved, for example, presence of rayed structures and vertical distribution of intensity (Figure 11), as well as their charac- teristic spatio-temporal variations. This aspect was veriﬁed through comparisons with observations of the real phenomena, including the positive feedback from scientists working in the auroral imaging ﬁeld [Hallinan 2001; Trondsen 2001]. Figures 11, 12 and 14 present still frames from simulation sequences produced using our model. For these sequences, we used a 1024 × 1024 mesh for the starting points, and varying values for the vertical displacement of the electron beams. The length and thickness of the initial auroral stream used in these 2 The simulation sequences (quicktime format) described in this article are available at http://www.curumin.uwaterloo. ca/ ˜ gvgbaran/dynamics.html. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 51 Fig. 10. Photograph of an auroral spiral. (Courtesy of Norbert Rosing.) Fig. 11. Simulation of an auroral spiral showing a rayed structure and an arc break up. Fig. 12. Frames of a simulation sequence showing the motions and shape changes of an auroral rayed band. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 52 • G. V. G. Baranoski et al. Fig. 13. Black and white all-sky photographs showing different stages of an auroral surge formation. (Courtesy of Kirsti Kauristie.) simulations were 2000km, and 9km, respectively. The computations were performed on a SGI550, with dual 850Mhz Pentium3 processors. Figure 12 presents six frames from a simulation sequence showing the motions and shape changes of an auroral rayed band. In this sequence, one can observe the rapid movement of auroral rays, the formation and motion of folds, and a rotational distortion of the auroral band. In this simulation, the number of electron beams was 4 × 104 , with a parametric collision interval (Section 4.2) of 0.05. The time spent to compute the starting points and the descending trajectories for each time step was 2.6s and 10.8s, respectively. The frames were rendered at a resolution of 320 × 240 pixels, and using 10 rays per-emission-point. Figure 13 presents two all-sky photographs taken during an auroral surge formation in the northern hemisphere. Figure 14 presents three frames from a simulation sequence, illustrating a simulation of such natural phenomena at twice normal speed. In this simulation, the number of electrons beams was 104 , with a parametric collision interval of 0.1. The time spent to compute the starting points and the descending trajectories for each time step was 1.7s and 6s, respectively. The frames were rendered at a resolution of 480 × 360 pixels, and using 5 rays per-emission-point. Recently, scientists, using NASA’s polar spacecraft, have captured images of the aurorae on ﬁlm as they were dancing simultaneously around both magnetic poles3 . This ﬁlm shows the Aurora Borealis and Aurora Australis expanding and brightening, in parallel, at opposite ends of the Earth, conﬁrming the three-century-old theory that aurorae in the northern and southern hemisphere are nearly mirror images of each other. In the case of auroral spirals, the distinct feature is their opposite sense of rotation. Usually they are counterclockwise in the southern hemisphere when viewed from the ground, and clockwise in the northern hemisphere. Figure 15 presents six frames from a simulation sequence, illustrating an auroral surge formation in the southern hemisphere. In this sequence, besides the warping process leading to an auroral spiral, we can also notice the formation and motion of folds. In this simulation, there were 104 electrons beams, with a parametric collision interval of 0.2. The time spent to compute the starting points and the descending trajectories for each time step was 1.7s and 10.27s, respectively. The frames were rendered at a resolution of 320 × 240 pixels, and using 20 rays per-emission-point. Using the multigrid algorithm, we were able to reduce substantially the time required in the warp- ing computations. For instance, on the 1024 × 1024 auroral mesh used in our simulations, multi- grid is 96 times faster than CG. Due to this reduction, the computation of the descending trajectories 3 http://eiger.physics.uiowa.edu/ ˜ vis/conjugate aurora/. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 53 Fig. 14. Frames from a simulation sequence illustrating an auroral surge (spiral) formation in the northern hemisphere. became the bottleneck of our simulation pipeline, with its cost growing linearly with respect to the number of electron beams. A divide-and-conquer strategy can be applied, however, to decrease the total running time of these computations. In other words, one can process n trajectories separately in several processors. For instance, for 105 electron beams, Baranoski and Rokne [2002] obtained a ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 54 • G. V. G. Baranoski et al. Fig. 15. Frames of a simulation sequence illustrating an auroral surge formation in the southern hemisphere. speedup of 4.6 using ﬁve Compaq Alpha 500Mhz PWS processors. Nonetheless, we believe that there is still room for improvement on the parallelization of the algorithms. We intend to continue to work on the parallelization, and to exploit graphics hardware for the real time simulation of moving auroral displays. 7. DIRECTIONS FOR APPLICATIONS AND FUTURE RESEARCH The physical simulation and visual representation of natural phenomena is an evolving area of re- search within computer graphics where signiﬁcant progress has been made. However, the important class of natural processes collectively known as plasma phenomena present signiﬁcant computational challenges and remain relatively unexplored [Gombosi et al. 2004]. The aurorae are among the most spectacular and most important from a scientiﬁc point of view due to the link they provide between solar activities and the Earth. In fact, the aurorae can be thought of as “footprints” of events and en- ergetic process occurring in the Earth’s magnetosphere. For this reason, even after 200 years of study, an astonishing amount of basic research is still being done on the aurora and its electromagnetic and electrodynamical properties [Paxton and Meng 1999]. The complexities of numerical simulations in the physical sciences often require visual representa- tions. Complex situations, for example, those found in plasma studies, may be dissected into simpler components to help the understanding of the underlying phenomena and obtain a clear theoretical picture [Tang 2002]. Moreover, animated sequences derived from computer simulations give a picture of a simulated phenomenon through time, illustrating qualitative features of any instabilities that may not be evident from a theoretical point of view. In fact, visualization plays a critical role in going from the raw nonlinear solution of the complex equations governing these phenomena to a simpliﬁed model explaining the essential underlying plasma physics as Parker et al. [1995] appropriately stated. This contribution of computer graphics to scientiﬁc applications was highlighted by D. Greenberg in the Steven Coons Award Lecture in 1987 [Greenberg 1988]: If computer graphics is to have a role in improving the future of our civilization, the real value will be in its application to science, engineering and design. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 55 Viewed in this context, we foresee three main directions for applications and future research associ- ated with the work presented in this article: —visualization of plasma phenomena, —visualization of Earth’s electro-magnetic environment in space, and —visualization of astrophysical phenomena. In order to understand plasma phenomena, a combination of extensive data analysis, theory, model- ing, laboratory experiments, and in situ (space born) observation is required. Plasma science impacts daily life in many signiﬁcant ways. In 1995, a panel of scientists at the National Research Council [NRC 1995] recommended that NASA and NSF fund a vigorous observational program related to plasma sci- ence, both on Earth and in space. The program should include both in situ and ground-based facilities, properly balanced with programs in theory, modeling, and laboratory experiments (see also NRC 2003). In our everyday lives, we encounter plasma in many situations, such as the ﬂash of a lightning bolt, the conducting of gas inside a ﬂuorescent tube or neon sign, the ionization in a rocket exhaust, gas lasers, and lightning discharges as some of the more familiar examples [Chen 1984; Smirnov 2001]. In the neighborhood of Earth, one encounters plasma in the Van Allen radiation belts and the solar winds as soon as one leaves the Earth’s atmosphere. Further afar, stellar interiors and atmospheres, as well as gaseous nebulae, are composed of plasma. For example, the Crab Nebula contains a magnetic ﬁeld and it is a rich source of plasma phenomena. It also contains a visual pulsar. Theories of pulsars picture them as rapidly rotating neutron stars with plasmas emitting synchrotron radiation from the surface [Bittencourt 1986; Chen 1984]. Occasionally, one also encounters an entirely new physical situation which is unlike anything previously observed either in space or in laboratory, and this opens new avenues of research. One example is the dusty plasma of comets and planetary rings that are dominated by the dynamics of charged macroparticles for which gravitational and electromagnetic effects are of comparable importance [NRC 1995]. In the type of particle model used in this article, the computer particle retains much of the identity of the atomic constituents of the material being simulated, yet does not correspond one-to-one to the simulated particles as do atoms to particles in molecular dynamics simulations. The “electron beams” or “superparticles” in these simulations correspond to millions of physical electrons. The plasma phenom- ena mentioned above can also be simulated using this approach. Furthermore, all of the mathematical models to which particle simulation methods are applied can be formulated as a set of particles inter- acting through ﬁelds. The most important ﬁeld equation is the Poisson’s equation, which pertains to all plasma examples listed above, and it is also found in the simulation of clustering of galaxies [Hockney and Eastwood 1988]. Electromagnetism is a fundamental topic [Belcher and Bessette 2001]. It is, however, one of the most difﬁcult subjects for students to master. Belcher and Bessette [2001] addressed this difﬁculty by creating 3D ﬁeld line animations used to aid student intuition about electromagnetic phenomena occurring in the laboratory. The auroral displays are similarly caused by electromagnetic phenomena, but at a planetary scale, that is, the solar wind impinges on the Earth’s magnetosphere, which shields us from this radiation, distorting the ﬁeld lines of the magnetosphere in the process. Visualization of such phenomena is important not only from a educational point of view [Nylund and Holland 1999], but also from a research point of view since it can be used to support improved predictions of space weather and its effects on systems operating in space as well as on Earth’s surface [Williams et al. 1999]. We also note that the magnetic storms responsible for auroral displays can damage satellites, power grids, pipelines, and spacecrafts [Paxton and Meng 1999; Odenwald 2000]. Plasma simulations are relevant to almost every area of astrophysics, ranging from stellar and inter- stellar plasmas, to star clusters and galaxies [NRC 1995]. The signiﬁcant mutual interactions among ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 56 • G. V. G. Baranoski et al. stars and galaxies are provided by Newtonian gravity. The latter is an inverse-square force like the electrostatic force, so it is natural that there should be many similarities between the dynamics of these systems and plasma physics [Binney 1993]. For example, both stellar dynamics and plasma dynamics account for shear (the simplest model of a stellar disc, which dominates most of galaxies, is the so-called shearing sheet [Binney 1993]) and can be simulated using “superparticles” models. Superparticles in a galaxy model represent millions of real stars. Length and time scales are determined by the dimensions and motion of the spiral arm structures. Collisional effects are made small by smoothing the force of interaction at short range, and by having a large number of superparticles within the range of smooth- ing [Hockney and Eastwood 1988]. One of the principal applications of such a model is in the study of spiral patterns arising in galaxies [Hockney and Eastwood 1988]. Such patterns are visually similar to the spiral structures arising from our auroral dynamics simulation. This is not a surprise considering that several processes familiar from plasma physics play important roles in structuring stellar systems. Obviously, gravity, unlike the electrostatic interaction, is always attractive, and this difference between the two systems invariably leaves its imprint [Binney 1993]. Consequently, there are also differences be- tween the way a given process works in the plasma and stellar-dynamics. We believe, however, that such cross-fertilization is beneﬁcial for both areas of research, and our intent is to fully explore these parallels. 8. CONCLUSION In this article, we have presented a novel physically-based model for the 3D visual simulation of auroral dynamics. The model employs a complete multigrid PDE simulation of the electromagnetic interactions between auroral electrons. It allows an accurate and efﬁcient visualization of auroral phenomena at different spatio-temporal scales. As far as we have been able to ascertain, this article describes the ﬁrst physically-based visual simulations of these phenomena in the scientiﬁc literature, leading to realistic visualization of the shape and the motion of these fascinating phenomena. Previous auroral simulations have been restricted to the static and KHI free case [Baranoski et al. 2000]. One of the most important research problems facing the computer graphics ﬁeld since its beginning, is the correct simulation of physical phenomena. As a result, the simulation of natural phenomena such as water, ﬁre, clouds, smoke and so forth. has attracted signiﬁcant research efforts. These efforts have been concentrated in the less than 1% of the universe where plasmas are not readily apparent, leaving the remaining 99% ﬁlled with plasma phenomena to be investigated by future researchers. Although scientiﬁc and commercial applications have different constraints, we believe that computer graphics researchers working in either area can beneﬁt from efﬁcient and predictive algorithms for the realistic simulation of plasma phenomena such as the majestic Northern Lights. ACKNOWLEDGMENTS Thanks are due to Trond Trondsen and Thomas Hallinan for their valuable scientiﬁc feedback on the simulation of auroral phenomena; Kirsti Kauristie and Norbert Rosing for allowing us to use their ¨ auroral photographs; Adriane Baranoski for helping us with the diagrams; Tamer Ozsu for his encour- agement; NASA Polar Ultraviolet Imager for granting us access to a satellite snapshot of an auroral oval; Gail Chopiak for her logistical assistance; Jessica Hodgins and John Hart for their helpful editorial suggestions; and the anonymous reviewers for their useful insights. REFERENCES ANDERSON, U. 2001. Time-domain methods for the Maxwell equations. Ph.D. thesis, Department of Numerical Analysis and Computer Science, Royal Institute of Technology, Sweden. ARULIAH, D. AND ASCHER, U. 2002. Multigrid preconditioning for Krylov methods for time-harmonic Maxwell’s equations in three dimensions. SIAM J. Scientif. Comput. 24, 2, 702–718. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 57 ARVO, J. 1995. Analytic methods for simulated light transport. Ph.D. thesis, Yale University. BARANOSKI, G. AND ROKNE, J. 2002. Using a HPC system for the simulation of the trajectories of solar wind particles in the ionosphere. In High Performance Computing Systems and Applications, N. Dimopoulos and K. Li, Eds. Kluwer Academic Publishers, Norwell, MA. (Chapter 2), 317–329. BARANOSKI, G., ROKNE, J., SHIRLEY, P., TRONDSEN, T., AND BASTOS, R. 2000. Simulating the aurora borealis. In the 8th Paciﬁc Conference on Computer Graphics and Applications. IEEE Computer Society, Los Alamitos, CA. 2–14. BARANOSKI, G., ROKNE, J., SHIRLEY, P., TRONDSEN, T., AND BASTOS, R. 2003. Simulating the aurora. J. Visualiza. Comput. Ani- mat. 14, 1 (Feb.), 43–59. BARANOSKI, G., WAN, J., ROKNE, J. G., AND BELL, I. 2002. Simulating the dynamics of the dancing lights. Tech. Rep. CS-2002-16, (April). School of Computer Science, University of Waterloo. BELCHER, J. AND BESSETTE, R. 2001. Using 3D animation in teaching introductory electromagnetism. Comput. Graph. 35, 1 (Feb.), 18–21. BINNEY, J. 1993. Gravitational plasmas. In Plasma Physics An Introductory Course, R. Dendy, Ed. Cambridge University Press, Cambridge, UK. 291–318. BIRDSALL, C. AND FUSS, D. 1969. Clouds-in-clouds, clouds-in-cells physics for many-body plasma simulation. J. Comput. Phys. 3, 494–511. BITTENCOURT, J. 1986. Fundamentals of Plasma Physics. Pergamon Press, Oxford, UK. BOROVSKY, J. AND SUSZCYNSKY, D. 1993. Optical measurements of the ﬁne structure of auroral arcs. In Auroral Plasma Dynamics, R. Lysak, Ed. American Geophysical Union, Washington, D.C., 25–30. (Vol. 80 of Geophys. Monogr. Series.) BOROVSKY, J., SUSZCYNSKY, D., BUCHWALD, M., AND DEHAVEN, H. 1991. Measuring the thickness of auroral curtains. Arctic 44, 3, 231–238. BRANDT, A. 1977. Multi-level adaptive solutions to boundary-value problems. Math. Comp. 31, 333–390. o BREKKE, A. AND EGELAND, A. 1994. The Northern Lights, Their Heritage and Science. Gr¨ ndahl og Dreyers Forlag, AS, Oslo. BRYANT, D. A. 1999. Electron Acceleration in the Aurora and Beyond. Institute of Physics Publishing, Bristol, UK. BURTNYK, K. 2000. Anatomy of an aurora. Sky and Telescope 99, 3 (March), 35–40. CASTLEMAN, K. 1996. Digital Image Processing. Prentice-Hall, New York. CHAN, T. AND WAN, W. 2000. Robust multigrid methods for elliptic linear systems. J. Comput. Appl. Math. 123, 323–352. CHEN, F. 1984. Introduction to Plasma Physics and Controlled Fusion, 2nd Ed. Plenun Press, New York. DAVIS, N. 1992. The Aurora Watchers Handbook. University of Alaska Press, Fairbanks. DAVIS, T. AND HALLINAN, T. 1976. Auroral spirals. 1. observations. J. Geophys. Research. 81, 22 (Aug.), 3953–3958. DRAZIN, P. AND REID, W. 1981. Hydrodynamic Stability. Cambridge University Press, Cambridge. DUPONT, D. 2004. Nuclear explosions in orbit. Scientiﬁc American, 100–107. EATHER, R. 1980. Majestic Lights. American Geophysical Union, Washington. ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. SIGGRAPH Proceedings, Annual Conference Series, 736–744. FEDKIW, R., STAM, J., AND JENSEN, H. 2001. Visual simulation of smoke. SIGGRAPH Proceedings, Annual Conference Series, 15–22. FEDORENKO, R. 1961. A relaxation method for solving elliptic difference equations. USSR Comput. Math. Phys. 1, 1092–1096. FELDMAN, B., O’BRIEN, J., AND ARIKAN, O. 2003. Animating suspended particle explosions. SIGGRAPH Proceedings, Annual Conference Series, 708–715. FOSTER, N. AND FEDKIW, R. 2001. Practical animation of ﬂuids. SIGGRAPH Proceedings, Annual Conference Series, 23–30. FOSTER, N. AND METAXAS, D. 1997. Modeling the motion of a hot, turbulent gas. SIGGRAPH Proceedings, Annual Conference Series, 181–188. GOLUB, G. AND LOAN, C. V. 1989. Matrix Computations, Second ed. John Hopkins University Press, Baltimore. GOMBOSI, T., POWELL, K., ZEEEUW, D., CLAUER, C., HANSEN, K., MANCHESTER, W., RIDLEY, A., ROUSSSEV, I., ABD Q. F., STOUT, I. S., AND TOTH, G. 2004. Solution-adaptive magnetohydrodynamics for space plasmas: Sun-to-Earth simulations. Computing in Science and Engineering, 14–35. GREENBERG, D. 1988. 1987 Steven A. Coons Award Lecture. Comput. Graph. 22, 1 (Feb.), 7–14. HACKBUSCH, W. 1985. Multi-grid Methods and Applications. Springer Verlag, Berlin. HALLINAN, T. 1976. Auroral spirals. 2. theory. J. Geophys. Resear. 81, 22 (Aug.), 3959–3965. HALLINAN, T. 1981. The distribution of vorticity in auroral arcs. In Physics of Auroral Arc Formation, S. Akasofu and J. Kan, Eds. American Geophysicsl Union, Washington, D.C. HALLINAN, T. 2001. Personal communication. Geophysical Institute, University of Alaska. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 58 • G. V. G. Baranoski et al. HALLINAN, T. AND DAVIS, T. 1970. Small-scale auroral distortions. Planet. Space Sci. 18, 1735–1744. HINZE, J. 1975. Turbulence. McGraw-Hill, New York. HOCKNEY, R. 1966. Computer experiment of anomalus diffusion. Phys. Fluids 9, 9 (Sept.), 1826–1835. HOCKNEY, R. AND EASTWOOD, J. 1988. Computer Simulation Using Particles. Institute of Physics, Bristol. JENSEN, H., DURAND, F., STARK, M., PREMOZE, S., DORSEY, J., AND SHIRLEY, P. 2001. A physically-based night sky model. SIGGRAPH Proceedings, Annual Conference Series, 399–408. JONES, A. 1974. Aurora. D. Reidel Publishing Company, Dordrecht, Holland. KAJIYA, J. AND HERZEN, B. V. 1984. Ray tracing volume densities. Comput. Graph. 18, 3 (July), 165–174. ¨ K¨ RKKAINEN, K. 2002. On the ﬁnite-difference modelling of electromagnetic problems in structured lattices. Ph.D. thesis, A Electromagnetics Laboratory, Department of Electrical and Communications Engineering, Helsinki University of Technology, Finland. KUNIMATSU, A., WATANABE, Y., FUJII, H., SAITO, T., HIWADA, K., TAKAHASHI, T., AND UEKI, H. 2001. Fast simulation of and rendering techniques for ﬂuid objects. Comput. Graph. For. 20, 3, 57–66. LEVY, R. AND HOCKNEY, R. 1968. Computer experiments on low-density crossed-ﬁeld electron beams. Phys. Fluids 11, 4 (April), 766–771. LJUNG, P., DIECKMANN, M., ANDERSSON, N., AND YNNERMAN, A. 2000. Interactive visualization of particle-in-cell simulations. In IEEE Visualization 2000. 469–472. MIYAZAKI, R., DOBASHI, Y., AND NISHITA, T. 2002. Simulation of cumuliform clouds based on computational ﬂuid dynamics. In Short Presentations of Eurographics 2002, I. Navazo and P. Slusallek, Eds. The Eurographics Association. 405–410. MYAZAKI, R., OSHIDA, S., DOBASHI, Y., AND NISHITA, T. 2001. A method for modeling clouds based on atmospheric ﬂuid dynamics. In the 9th Paciﬁc Conference on Computer Graphics and Applications. IEEE Computer Society, Los Alamitos, CA. 363–372. NADEAU, D., GENETTI, J., NAPEAR, S., PAILTHORPE, B., EMMART, C., WESSELAK, E., AND DAVIDSON, D. 2001. Visualizing stars and emission nebulas. Comput. Graphics Forum 20, 1 (March), 27–33. NGUYEN, D. Q., FEDKIW, R., AND JENSEN, H. W. 2002. Physically based modeling and animation of ﬁre. SIGGRAPH Proceedings, Annual Conference Series, 721–728. NRC. 1995. Plasma Science from Fundamental Research to Technological Applications. National Academy Press, Washington, D.C. NRC. 2003. The Sun to the Earth and Beyond. National Academy Press, Washington, D.C. NYLUND, S. AND HOLLAND, D. 1999. Themes and trends in space science data processing and visualization. John Hopkins APL Tech. Digest 20, 4, 533–543. ODENWALD, S. 2000. Solar storms: The silent menace. Sky and Telescope 99, 3 (March), 41–56. OMHOLT, A. 1971. The Optical Aurora. Springer-Verlag, New York. PARKER, S., CUMMINGS, J., AND SAMTANEY, R. 1995. Visualization of plasma turbulence. IEEE Comput. Graph. Appl. 15, 2, 7–10. PARTAMIES, N., FREEMAN, M., AND KAURISTIE, K. 2001. On the winding of auroral spirals: Interhemispheric observations and Hallinan’s theory revisited. J. Geophys. Resear. 106 (A12), 28913–28924. PARTAMIES, N., KAURISTIE, K., PULKKINEN, T., AND BRITTNACHER, M. 2001. Statistical study of auroral spirals. J. Geophys. Resear. 106, 15415–15428. PAXTON, L. AND MENG, C. 1999. Auroral imaging and space-based optical remote sensing. John Hopkins APL Tech. Digest 20, 4, 544–555. RASMUSSEN, N., NGUYEN, D. Q., GEIGER, W., AND FEDKIW, R. 2003. Smoke simulation for large scale phenomena. SIGGRAPH Proceedings, Annual Conference Series, 703–707. SCHUSSMAN, G., MA, K., SCHISSEL, D., AND EVANS, T. 2000. Visualizing DIII-D Tokamak magnetic ﬁeld lines. In IEEE Visualization 2000. 501–504. SILVESTER, P. AND FERRARI, R. 1990. Finite Elements for Electrical Engineers, 2nd Ed. Cambridge University Press, Cambridge. SMIRNOV, B. 2001. Physics of Ionized Gases. John Wiley & Sons, New York. STAM, J. 1995. Multiple scattering as a diffusion process. In Rendering Techniques’95 (Proceedings of the Sixth Eurographics Rendering Workshop), P. M. Hanrahan and W. Purgathofer, Eds. SIGGRAPH Proceedings, Annual Conference Series, 41–50. STAM, J. 1999. Stable ﬂuids. SIGGRAPH Proceedings, Annual Conference Series, 121–128. STAM, J. AND FIUME, E. 1993. Turbulent wind ﬁelds for gaseous phenomena. SIGGRAPH Proceedings, Annual Conference Series, 369–376. TAFLOVE, A. 2000. Computational Electrodynamics: The ﬁnite-difference time-domain method, 2nd Ed. Artech House, Boston, MA. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Simulating the Dynamics of Auroral Phenomena • 59 TAJIMA, T. 1989. Computational Plasma Physics: With Applications to Fusion and Astrophysics. Addison-Wesley, Redwood City, CA. TANG, W. 2002. Advanced computations in plasma physics. Phys. Plasmas 9, 5, 1856–1872. TAYLOR, K. 2001. Auroras earth’s grand show of lights. National Geographic, 48–63. TRONDSEN, T. 1998. High spatial and temporal resolution auroral imaging. Ph.D. thesis, Department of Physics, Faculty of Science, University of Tromsø, Norway. TRONDSEN, T. 2001. Personal communication. Institute for Space Research, The University of Calgary. ¨ ¨ TROTTENBER, U., OOSTERLEE, C., SCHULLER, A., BRANDT, A., OSWALD, P., AND STUBEN, K. 2001. Multigrid. Academic Press, New York. UPSON, C., BARR, A., REEVES, B., WOLFF, R., AND WOLFRAM, S. 1987. The physical simulation and visual representation of natural phenomena. Comput. Graph. 21, 4, 355–356. VOLAKIS, J. AND KEMPEL, L. 1995. Electromagnetics: Computational methods and considerations. IEEE Computat. Sci. Engi- neer. 2, 1, 42–57. WEIMER, H. AND WARREN, J. 1999. Subdivision schemes for ﬂuid ﬂow. SIGGRAPH Proceedings, Annual Conference Series, 111–120. WESTOVER, L. 1991. Splatting: A parallel, feed-forward volume rendering algorithm. Ph.D. thesis, Department of Computer Science, University of North Carolina at Chapel Hill. WILLIAMS, D., MAUK, B., MITCHELL, D., ROELOF, E., AND ZANETTI, L. 1999. Radiation belts and beyond. John Hopkins APL Tech. Digest 20, 4, 544–555. WITKIN, A. AND KASS, M. 1991. Reaction-diffusion textures. Comput. Graph. (SIGGRAPH Proceedings) 25, 4, 299–308. YOSHIZAWA, A. 1998. Hydrodynamic and Magnetohydrodynamic Turbulent Flows: Modelling and Statistical Theory. Kluver, Dordrecht. YOSHIZAWA, A., ITOH, S., ITOH, K., AND YOKOI, N. 2001. Turbulence theories and modelling of ﬂuids and plasma. Plasma Phys. and Controlled Fusion 43, 1–144. Received May 2002; accepted October 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. On the Optimality of Spectral Compression of Mesh Data MIRELA BEN-CHEN and CRAIG GOTSMAN Technion—Israel Institute of Technology Spectral compression of the geometry of triangle meshes achieves good results in practice, but there has been little or no theoretical support for the optimality of this compression. We show that, for certain classes of geometric mesh models, spectral decomposition using the eigenvectors of the symmetric Laplacian of the connectivity graph is equivalent to principal component analysis on that class, when equipped with a natural probability distribution. Our proof treats connected one-and two-dimensional meshes with ﬁxed convex boundaries, and is based on an asymptotic approximation of the probability distribution in the two-dimensional case. The key component of the proof is that the Laplacian is identical, up to a constant factor, to the inverse covariance matrix of the distribution of valid mesh geometries. Hence, spectral compression is optimal, in the mean square error sense, for these classes of meshes under some natural assumptions on their distribution. Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling; E.4 [Coding and Information Theory] General Terms: Theory Additional Key Words and Phrases: Triangle mesh, spectral decomposition, Laplacian 1. INTRODUCTION Triangle meshes are a popular way of representing 3D shape models. As the size and detail of the models grow, compression of the models becomes more and more important. The size of the mesh data ﬁles can be reduced by compressing either the geometry or the connectivity of the mesh, or both. There has been much research into both geometry and connectivity coding. Most connectivity coding schemes, for example the Edgebreaker [Rossignac 1999] and the TG methods [Touma and Gotsman 1998], are based on traversing the mesh and generating code symbols representing new vertices or faces as they are traversed. The quality of the compression results from the entropy of the symbol sequence. Typical geometry coding schemes are based on the fact that the coordinates of a mesh are not in- dependent, and speciﬁcally, the coordinates of neighboring vertices are highly correlated, especially in smooth meshes. This correlation can be exploited by using “prediction rules”—the coordinates of a vertex are predicted from the coordinates of neighboring vertices, and only the prediction error vector is coded [Taubin and Rossignac 1998; Touma and Gotsman 1998]. The better the prediction rule is, the smaller the errors are, and the smaller the entropy of the code will be. Spectral compression of mesh geometry [Karni and Gotsman 2000] also exploits the correlation among neighboring vertices, and implicitly applies a prediction rule that every vertex is the simple average This work was partially supported by the Israel Ministry of Science Grant 01-01-01509, the German-Israel Fund (GIF) Grant I-627-45.6/1999, European FP5 RTN Grant HPRN-CT-1999-00117 (MINGLE), and European FP6 NoE Grant 506766 (AIM@SHAPE). Authors’ addresses: Department of Computer Science, Technion-Israel Institute of Technology, Haifa 32000 Israel; email: {mirela,gotsman}@cs.technion.ac.il. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0060 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 60–80. The Optimality of Spectral Compression of Mesh Data • 61 Fig. 1. A simple irregular mesh and its symmetric Laplacian. of all its immediate neighbors. Inspired by traditional signal coding, spectral decomposition has been proposed for lossy transform coding of the geometry of a mesh with irregular connectivity. Although the method yields good results in practice, there is little theoretical support for the optimality of such compression. Motivated by the optimality of the Discrete Cosine Transform (DCT) in signal processing [Rao and Yip 1990], we wish to prove a similar result for spectral compression, namely, that it is optimal for certain classes of irregular meshes. Our proof covers connected meshes in one and two dimensions with ﬁxed convex boundaries. The proof for the two-dimensional case is based on an asymptotic normal approximation. 1.1 Previous Work Let G be a graph, G = (V , E), where V is the vertex set, and E the edge set. A k-dimensional mesh M is M = (G, R), R = (X (1) , X (2) , . . . , X (k) ), where X (i) is a real vector of the i-th dimension coordinate values of the mesh vertices. We sometimes refer to E as the connectivity and to R as the geometry of the mesh M . Given a mesh M = (G, R), G = (V , E), the symmetric Laplacian of M is the matrix L: di i = j Li, j = 0 / (i, j ) ∈ E −1 (i, j ) ∈ E where d i is the number of neighbors (valence) of the i-th vertex. A mesh with constant valences is called a regular mesh; otherwise, it is called an irregular mesh. See Figure 1 for an example of an irregular mesh, and its corresponding Laplacian. Spectral decomposition was ﬁrst introduced by Karni and Gotsman [2000] as a tool for mesh geometry compression. In their work, the eigenvectors of the Laplacian matrix are used as the basis for the decomposition of all meshes having the same connectivity. Karni and Gotsman argued that since these vectors were a generalization of the Fourier basis vectors obtained for regular (grid-type) connectivities, they should work just as well for irregular connectivities. Indeed, they showed that the coefﬁcients of such decomposition decay rapidly to zero (see Figure 2). Hence, a small number of coefﬁcients, combined with aggressive quantization, can be used to code the mesh geometry. The eigenvalues corresponding to the eigenvectors are analogous to the concept of frequency in harmonic analysis. Eigenanalysis of the graph Laplacian is a standard tool in spectral graph theory [Chung 1997]. Several variants of the Laplacian have been studied. Eigenvectors of Laplacians are also used for graph drawing [Hall 1970; Koren 2003], graph partitioning [Alpert and Yao 1995], and parametrization [Gotsman et al. 2003]. For a more extensive survey of the uses of Laplacians in digital mesh processing, see Gotsman [2003]. The inverse of the Laplacian is closely related to discrete Green’s functions [Chung and Yau 2000], and, as we shall see, plays a central role in this article. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 62 • M. Ben-Chen and C. Gotsman Fig. 2. Spectral decomposition. (a) The horse model and (b, c, d) the spectral coefﬁcients of its decomposition in the X , Y , and Z dimensions. The rest of the article is organized as follows. Section 2 deﬁnes the terminology used throughout the article. Section 3 reviews the concept of principal component analysis, a key tool in our proofs, which is the motivation for studying the eigenvectors of the covariance matrix. Sections 4 and 5 prove the optimality result for the 1D and 2D cases, respectively. To conclude, Section 6 discusses our model and explores future research directions. 2. DEFINITIONS The decomposition of a vector X nx1 , by the orthonormal basis of R n : Unxn = {U1 , . . . , Un }, is: n X = ci Ui , (1) i=1 where C ={ci } are the coefﬁcients of the decomposition. The reconstruction of a vector X from its decomposition using the ﬁrst k coefﬁcients is: k X (U,k) = ci Ui . (2) i=1 Note that the order of the basis vectors is important, as only a preﬁx of them is used in the k-th reconstruction. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 63 Because of the orthonormality of U , the Parseval identity maintains that c = X . A decomposition is useful when a small number of the coefﬁcients contain a large portion of the norm (energy) of the vector. If this is the case, those coefﬁcients alone may be used in the reconstruction, and the Euclidean distance between the reconstructed vector and the original X will be small. This is a useful feature for compression applications. For a speciﬁc vector X , the best basis will always be {X , 0, 0, . . . , 0}, since then all the energy is con- tained in the ﬁrst coefﬁcient. Hence, there is a meaningful answer to the question “what is the optimal basis for decomposition” only if we consider an entire family of vectors (ﬁnite or inﬁnite). In our context, the family will be all the geometries which are valid for a given mesh connectivity E. These geometries will be speciﬁed by imposing an appropriate probability distribution D, derived from E, on R n . If X is a random variable, we denote by Exp(X ) the expectation of X , by Var(X ) the variance of X for scalar X , and by Cov(X ) the covariance matrix of X for vector X . Given a probability distribution D on R n , we say that a vector basis U is an optimal basis for D, if for any other basis W , and for every 1 ≤ k ≤ n: 2 2 Exp X − X (U,k) ≤ Exp X − X (W,k) (3) where the expectation is taken over D. This is called optimality in the Mean Square Error (MSE) sense. 3. PRINCIPAL COMPONENT ANALYSIS Approximation of random signals has been studied extensively in signal processing. A well-known optimality result, which we will rely on heavily, is related to the so-called Principal Component Analysis (PCA) procedure. Assume a random vector X ∈ R n , sampled from distribution D, having zero mean vector and co- variance matrix C. Denote by { i |i = 1, . . , n} the eigenvectors of C, with corresponding eigenvalues {λi |i = 1, . . . , n}, ordered such that λ1 ≥ λ2 ≥ · · · ≥ λn . The matrix is called the Principal Component Analysis of X , sometimes known as the Karhunen-Loeve or Hotelling transform of X . It is well known [Jolliffe 1986] that the PCA is optimal in the MSE sense, as deﬁned in (3). Note that does not depend on k—the number of vectors used for the reconstruction—so the optimal basis for reconstruction from k + 1 basis vectors contains the optimal basis for reconstruction from k vectors. When the class of meshes is ﬁnite and given, containing T meshes, for example, an animation se- quence of a mesh with a ﬁxed connectivity, the natural distribution is a uniform distribution on this ﬁnite set. In this case, the PCA of that class may be computed using the numerical Singular Value Decomposition (SVD) procedure [Press et al. 1987] on a matrix of size 3n × T , consisting of T columns, where the i-th column contains the coordinate values of the i-th mesh in the sequence. This was proposed by Alexa and Muller [2000] for compressing animated meshes. To compute the PCA of an inﬁnite continuum of meshes with a given connectivity, where none of them is explicitly given, we must ﬁrst make some assumptions about the distribution D of this family, and then compute the covariance matrix C of this distribution. We will do this, and then show that C is essentially identical to the inverse of the mesh Laplacian matrix, up to a constant factor. Due to this special relationship, both matrices have identical eigenvectors (in opposite order), from which our optimality theorem will follow. 4. 1D MESHES Before we proceed, a note about the connection between the proof for one dimension, and the proof for two dimensions is in order. Basically, for both cases we prove the same theorem—that L = αC −1 , for some constant α. While for the one-dimensional case the theorem can be proved relatively easily, in two dimensions we resort to asymptotic approximations. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 64 • M. Ben-Chen and C. Gotsman Fig. 3. The “chain” connectivity” graph for n = 5 in one dimension. 4.1 The Model In one dimension, a mesh is simply an ordered set of points on the real line. There is only one “universal” (regular) connectivity—each vertex is connected to exactly two neighbors, except the two boundary vertices. For example, for n = 5, the connectivity graph is as shown in Figure 3. We say that a vector X = {X 1 , X 2 , . . . , X n } is a valid geometry in 1D if and only if b0 ≤ X 1 ≤ X 2 ≤ · · · ≤ X n ≤ b1 , where b0 and b1 are ﬁxed boundary points. This is equivalent to saying that the “chain” connectivity does not “fold” on itself. 4.2 The Geometry Distribution Let U1 , U2 , . . . , Un be independent random variables, distributed uniformly on (a, b), and let a ≤ U(1) ≤ U(2) ≤ · · · ≤ U(n) ≤ b be these values after sorting them in increasing order. U(1) , U(2) , . . . , U(n) are called uniform order statistics [David 1981]. We assume the valid geometries are distributed like the uniform order statistics on (b0 , b1 ). Such a distribution will be uniform over all the valid geometries, because every valid geometry, that is every order statistics vector, can be generated by n! permutations of the original variables, which are uniformly distributed. The mean geometry for this distribution is the grid geometry, where the distances between the vertices are equal. 4.3 The Optimal Basis We now proceed to the main theorem in its one-dimensional version. THEOREM 1 The optimal basis for the decomposition of one dimensional meshes are the eigenvectors { i |i = 1 . . n} of the symmetric Laplacian, ordered such that the corresponding eigenvalues are µ1 ≤ µ2 ≤ · · · ≤ µn . As stated in Section 3, the optimal basis is the PCA, which is the matrix of the eigenvectors of the covariance matrix of the mesh geometry distribution. Let X = {X 1 , X 2 , . . . , X n } be a random valid geometry with boundaries b0 and b1 , and C its covariance matrix. Assume j ≤ i , since Cij = Cji . Now by deﬁnition Cij = Exp(X i X j ) − Exp(X i )Exp(X j ). The ﬁrst step in ﬁnding the optimal basis is computing the covariance matrix C. In order to do that, we will need some more deﬁnitions: Deﬁne X 0 = b0 , X n+1 = b1 , and Y i = X i − X i−1 for i = 1 . . n + 1. Y i are called uniform spacings. It is well-known [Pyke 1965] that Y i are interchangeable random variables, namely, that they are identically distributed, and all pairs of variables have the same joint distribution: Var(Y i ) = Var(Y 1 ) = v(n) ∀i ∈ {1, . . . , n + 1} (4) Cov(Y i , Y j ) = Cov(Y 1 , Y 2 ) = c(n) ∀i, j ∈ {1, . . . , n + 1}, i = j for some functions v(n) and c(n) which depend only on n and the boundaries b0 and b1 (but not on i and j ). LEMMA 1.1. If X is a random valid 1D geometry, C its covariance matrix, and j ≤ i, then: Cij = jv(n) + (ij − j )c(n). ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 65 PROOF. From the deﬁnition of Y i we have that: i X i = b0 + Yk. (5) k=1 Then, for j ≤ i, by substituting (5) in the covariance deﬁnition, and since the Y i are interchangeable as deﬁned in (4), we have: i j Cij = Exp(X i X j ) − Exp(X i ) Exp(X j ) = [Exp(Y k Y r ) − Exp(Y k ) Exp(Y r )] k=1 r=1 j i j = Exp Y r2 − Exp(Y r )2 + [Exp(Y k Y r ) − Exp(Y k ) Exp(Y r )] (6) r=1 k=1 r=1 r=k j i j = var(Y 1 ) + cov(Y 1 , Y 2 ) = jv(n) + (ij − j )c(n). r=1 k=1 r=1 r=k We will now relate the variance and covariance of the uniform spacings, which will allow us to simplify the expression in (6). LEMMA 1.2. If v(n) and c(n) are the variance and covariance, respectively, of uniform spacings as deﬁned in (4), then: 1 c(n) = − v(n). n PROOF. Consider the following expression: 2 2 n+1 n+1 Exp Yi − Exp Yi . i=1 i=1 On the one hand, developing the expression, by expanding the sums and using the linearity of the expectation, yields: 2 2 n+1 n+1 n+1 n+1 n+1 n+1 Exp Yi − Exp Yi = Exp(Y i Y j ) − Exp(Y i ) Exp(Y j ) i=1 i=1 i=1 j =1 i=1 j =1 n+1 n+1 = [Exp(Y i Y j ) − Exp(Y i ) Exp(Y j )] i=1 j =1 n+1 = Exp Y j − Exp(Y j )2 2 j =1 n+1 n+1 + [Exp(Y i Y j ) − Exp(Y i ) Exp(Y j )] (7) i=1 j =1 i= j n+1 n+1 n+1 = var(Y 1 ) + cov(Y 1 , Y 2 ) j =1 i=1 j =1 i= j = (n + 1)v(n) + n(n + 1)c(n). ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 66 • M. Ben-Chen and C. Gotsman On the other hand, since the sum of the Y i is the constant (b1 − b0 ), we have: 2 2 n+1 n+1 Exp Yi − Exp Yi = (b1 − b0 )2 − (b1 − b0 )2 = 0. (8) i=1 i=1 Comparing (7) and (8), we have: (n + 1)v(n) + n(n + 1)c(n) = 0 (9) 1 c(n) = − v(n) n Note that the covariance of two spacings is always negative. Intuitively, this is because one spacing may grow only at the expense of the other spacings, since their sum is constant. By using the two previous lemmas, we can now simplify the expression of the covariance, by substi- tuting (9) in (6), to get: v(n) Cij = j (n − i + 1) for every 1 ≤ j ≤ i ≤ n. n So, the covariance matrix C is j (n − i + 1) v(n) 1≤ j ≤i≤n Cij = n (10) Cji 1≤i< j ≤n Now let us examine the matrix product L · C. The valence d i of all the (interior) vertices is 2, since every vertex has exactly two neighbors. Hence, the Laplacian is the n × n matrix: 2 −1 0 ··· 0 . .. . −1 2 −1 . . .. L= 0 . 0 . . .. . . −1 2 −1 0 ··· 0 −1 2 (n×n) where n is the number of interior vertices. Note that the Laplacian has entries only for the interior vertices, so the ﬁrst and last rows do not sum to zero, since they belong to vertices neighboring on the boundaries. This property makes the Laplacian invertible. By substituting (10) in the product L·C, it is easy to see that: k(n) i = j (L · C)i, j = 0 i = j, where k(n) depends only on n, and is v(n)(n + 1) k(n) = , n which implies that L · C = k(n)I ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 67 Fig. 4. 2D geometries with common connectivity and boundary: (a) An invalid 2D geometry. The highlighted triangle is folded over. (b, c) Valid 2D geometries. or L = k(n)C −1 . From the optimality of PCA, we know that the eigenvectors { i |i = 1 . . n} of C, ordered such that λ1 ≥ λ2 ≥ · · · ≥ λn are the optimal basis. Since the eigenvectors of C −1 are the same as the eigenvectors of C, but the eigenvalues are reciprocated (µi = 1/λi ), the optimal basis for the decomposition of 1D meshes are the eigenvectors of the symmetric Laplacian, ordered so the corresponding eigenvalues are µ1 ≤ µ2 ≤ · · · ≤ µn . Note that the exact covariance matrix depends on k(n), which depends on n and the boundaries b0 and b1 . 5. 2D MESHES 5.1 The Model In two dimensions, a geometry R = (X , Y ) is valid for the triangle mesh M = (G, R), G = (V , E) if the triangulation of (X , Y ) using E does not contain intersecting edges, namely, the triangles do not fold over onto each other, or, in other words, all have the same orientation. Assume M has a ﬁxed boundary, whose coordinates are (X b1 , X b2 , . . . , X bk ) = (xb1 , xb2 , . . . , xbk ) (Y b1 , Y b2 , . . . , Y bk ) = ( y b1 , y b2 , . . . , y bk ), forming a convex shape. A few examples of valid and invalid geometries for the same connectivity and convex boundaries are shown in Figure 4. 5.2 The Geometry Distribution To prove an optimality theorem such as Theorem 1 in the two-dimensional case, we have to impose some natural probability distribution on the class of valid geometries. This is not easy, in general. For the time being, we will assume that the following three properties hold for each coordinate of the valid geometries’ distribution. Later we will justify these assumptions. (1) X i |X j =i = x j is normally distributed for every i, meaning that the conditional distribution of the coordinates of one vertex given the coordinates of all the others is Gaussian. (2) Exp(X i |X j =i = x j ) = di 1 j ∈N (i) x j , meaning that the expectation of the Gaussian distribution mentioned in 1 is the average coordinates of just the neighbors of i. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 68 • M. Ben-Chen and C. Gotsman (3) Cov(X i , X j | X k = xk , k = i, j, (i, j ) ∈ E) = 0, meaning that the covariance of the coordinates of every two vertices i, j which are not neighbors, conditioned on the coordinates of all other vertices, is zero. From now on, we will only refer to the X dimension, but all the theorems hold for the Y dimension too. 5.3 The Optimal Basis The 2D version of our optimality theorem is: THEOREM 2. If the distribution of valid geometries of 2D meshes has the properties deﬁned in Section 5.2, then the optimal basis for the decomposition of 2D meshes with connectivity graph G = (V , E) are the eigenvectors { i |i = 1 . . n} of the symmetric Laplacian of G, ordered such that the corre- sponding eigenvalues are µ1 ≤ µ2 ≤ · · · ≤ µn . We ﬁrst show that if the distribution of valid geometries has the ﬁrst two properties described in Section 5.2, then it is multivariate normal. Note that just normal conditionals do not necessarily imply multivariate normality. However, the following Lemma characterizes the multivariate normal distri- bution via its conditionals: LEMMA 2.1. (ARNOLD ET AL. 1999). Let X = {X 1 , X 2 , . . . , X n } be a random vector. If: (1) X i |X j =i = x j is normally distributed for every i, and (2) Exp(X i |X j = x j ) is linear in x j and not constant, for every i, then X has a multivariate normal distribution. It is easy to see that the conditions of the lemma are satisﬁed if we assume the distribution of valid geometries has the ﬁrst two properties described in Section 5.2. The ﬁrst condition of the lemma is identical to the ﬁrst property described in Section 5.2. The second condition is implied by the second property—if the conditional expectation of a vertex is the average of its neighbors, then it is linear in x j . In addition, since the mesh is connected, and there are no isolated vertices, the conditional expectation cannot be a constant with respect to the x j . Thus, the second condition of Lemma 2.1 also holds. Since both the conditions of Lemma 2.1 hold, we conclude that the distribution of valid geometries (assuming Section 5.2) is multivariate normal. Now that we have characterized the distribution of the geometries, we proceed to compute the covari- ance matrix, which is the key for proving optimality. The next two lemmas characterize the structure of K —the inverse of the covariance matrix. Combined, they show that K is essentially identical to the mesh Laplacian. LEMMA 2.2. Let C be the covariance matrix of the X component of a random valid geometry R = (X , Y ) of a mesh M = (G, R), G = (V , E). Let K = C −1 . Then for every (i, j ), such that i = j , and (i, j ) ∈ E, K i, j = 0. / PROOF. We need the following lemma which describes a few known properties of the inverse covari- ance matrix K of a multinormal distribution: LEMMA 2.3 (LAURITZEN 1996). Let X be a multivariate normal random variable X ∼ N (µ, ), and let K = −1 . Then: (1) K ij = −Cov(X i , X j |X k=i, j = xk )(K ii K jj − K ij ). 2 K (2) Exp(X i |X j =i = x j ) = Exp(X i ) + j =i βij (x j − Exp(X j )), βij = − K ij . ii ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 69 Part 1 of Lemma 2.3, and the vanishing conditional covariance described in the third property of Section 5.2 imply that if i and j are not neighbors, then K ij = 0. For the entries of K corresponding to the vertices and edges of the connectivity graph, we need the following Lemma. LEMMA 2.4. Let C be the covariance matrix of the X component of a random valid geometry R = (X , Y ) of a mesh M = (G, R), G = (V , E). Let K = C −1 . Then, there exists a constant α, such that: (1) For every (i, j ) ∈ E, K ij = −α (2) For every i ∈ V , K ii = αd i PROOF. From part 2 of Lemma 2.3, we know that Exp(X i |X j =i = x j ) = Exp(X i ) + βij (x j − Exp(X j )). j =i On the other hand, we assumed in the second property of Section 5.2 that 1 Exp(X i |X j =i = x j ) = xj. di j ∈N (i) Since the linear coefﬁcients of x j must be equal in both expressions, we have K ij 1 βij = − = . (11) K ii di C is a covariance matrix, so both C and K are symmetric, hence K ii K jj = −K ij = −K ji = (12) di dj for every (i, j ) ∈ E. Consider the diagonal of K . It is easy to see that if the mesh is connected, all the values K ii /d i must equal a constant α that does not depend on i: Deﬁne K 11 /d 1 = α. From (12) we have that for all the neighbors j ∈ N (1), K jj /d j = K 11 /d 1 = α. The same can be done inductively for the neighbors of j and so on. Finally, since the mesh is connected, every vertex i has a path to the ﬁrst vertex, so it must hold that: K ii =α (13) di for every i ∈ V . Substituting (13) in (12) implies K ij = −α (14) for every (i, j ) ∈ E Combining Lemmas 2.2 and 2.4, we conclude: Let C be the covariance matrix of the X component of a random valid geometry R = (X , Y ) of a mesh M = (G, R), G = (V , E). Let K = C −1 . Then K is: −α (i, j ) ∈ E K ij = αd i i = j 0 otherwise ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 70 • M. Ben-Chen and C. Gotsman Returning to the deﬁnition of the symmetric Laplacian in Section 1.1, we see that: C −1 = K = αL, where α is a constant that depends on n alone. As in the one-dimensional case, this concludes the proof of Theorem 2. Note that the exact values of the covariance matrix depend on α, which depends on the ﬁxed boundary values. Our theorem makes some powerful assumptions on the distribution of valid 2D geometries. We will now show why it is reasonable to make such assumptions, by describing a natural model for generating valid 2D geometries, which turns out to have a distribution with the required properties. Following Tutte [1963], Floater [1997] proved that a 2D geometry with a convex boundary is valid if and only if each vertex is a convex combination of its neighbors. This implies that a geometry (X , Y ) with a convex boundary (Bx , B y ) is valid, if and only if there exists a matrix W such that X = WX + Bx , Y = WY + BY (15) where Bx and B y are xi i ∈ {b1 , . . . , bk } yi i ∈ {b1 , . . . , bk } Bxi = Byi = 0 otherwise 0 otherwise and W is wij / (i, j ) ∈ E, i ∈ {b1 , . . . , bk } Wij = . 0 otherwise The weights wij are positive and normalized: n / Wij = 1 i ∈ {b1 , . . . , bk }. i=1 We will call W a barycentric coordinates matrix. This characterization of valid 2D meshes yields a convenient way to deﬁne a probability distribution on them. Instead of specifying the distribution of the valid geometries X (which seems to be quite difﬁ- cult), we specify the distribution of the barycentric coordinates matrices. We assume that the barycentric coordinates matrices are distributed as follows: For each interior vertex i, with valence d i , let wij = D ij = U(ij +1) − U(ij ) , (16) where U(ij ) are d i − 1 order statistics over (0, 1), with U(0) = 0, and Ud (i) = 1. D ij are known as uniform i i spacings [Pyke 1965]. This guarantees that the nonzero Wij are indeed positive and all the internal vertices’ rows sum to one. Note that such a distribution is not guaranteed to generate a uniform distribution of valid geometries. Since barycentric coordinates are not unique, that is, more than one set of barycentric coordinates can generate the same geometry, the use of barycentric coordinates may introduce a bias which will prefer certain valid geometries over others. We now address the following question: If the barycentric coordinates matrices W are distributed as in (16), how are the geometries distributed? Given the matrix W and the boundary B, the geometry X ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 71 can be expressed as X = (I − W )−1 B, where I is the identity matrix. LEMMA 2.5. Let X be the x coordinate of a random valid geometry, whose barycentric coordinates are distributed as in (16). Then the distribution of X has the following properties: (1) The limit distribution of X i |X j =i = x j as d i → ∞ is normal for every i, (2) Exp(X i |X j =i = x j ) = 1 di j ∈N (i) xj, (3) Cov(X i , X j |X k = xk , k = i, j ) = 0 for every two vertices i, j which are not neighbors. PROOF. From the deﬁnition of X and W in (15) and (16), respectively, it is easy to see that the conditioned variables (X i |X j = x j ) are (X i |X j = x j ) = D ij x j , (17) j ∈N (i) where D ij are uniform spacings, N (i) is the set of neighbors of the i-th vertex, and the x j are constants. The central limit theorem for functions of uniform spacings [Pyke 1965] implies that for vertices with large valence: (X i |X j = x j ) ∼ Normal µi , σi2 (18) where Normal is the Gaussian distribution. This proves the ﬁrst part of the Lemma. Since uniform spacings are interchangeable random variables which sum to unity: di D ij = 1 ∀i j =1 it follows that 1 Exp D ij = Exp D1 = i ∀ j. (19) di Substituting the expectation of the spacings (19), in (17), we obtain 1 µi = Exp(X i |X j = x j ) = Exp D ij x j = Exp D ij x j = xj. (20) j ∈N (i) j ∈N (i) di j ∈N (i) Note that the x j are constants, since they are conditioned upon, so Exp(D ij x j ) = Exp(D ij )x j . This proves the second part of the Lemma. Let i, j be two nonadjacent vertices. Consider the covariance of X i and X j conditioned on the rest of the vertices: Cov X i , X j |X k = xk , k = i, j = Exp(X i X j |X k = xk ) − Exp(X i |X k = xk )Exp(X j |X k = xk ) = Exp i Dr xr Dm xm − Exp j i Dr xr Exp j Dm xm r∈N (i) m∈N ( j ) r∈N (i) m∈N ( j ) = Exp i j Dr xr Dm xm − Exp i Dr xr Exp Dm xm . j r∈N (i) m∈N ( j ) r∈N (i) m∈N ( j ) ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 72 • M. Ben-Chen and C. Gotsman i j Since i and j are not neighbors, Dr and Dm are disjoint sets of independent uniform spacings, which implies that: Exp Dr xr Dm xm = Exp Dr xr Exp Dm xm i j i j (21) and thus, Cov(X i , X j |X k = xk , k = i, j ) = 0 (22) for every nonadjacent i and j . This proves the third part of the Lemma and concludes its proof. We now have a model for generating valid 2D meshes, which yields a distribution that has the prop- erties required in Section 5.2, with just one problem—the ﬁrst property of Section 5.2 requires a normal conditional distribution, and all we have is a normal limit distribution as d → ∞. Central limit theo- rems with asymptotic parameter d → ∞ give very good approximations already for modest values of d . Here the asymptotic parameter is the valence, which is 6, on the average. The following experimental results show that it seems to be large enough for the normal approximation to be reasonable. The Gaussian distribution of the coordinates can be seen in Figure 5, where some values of the coordinates of a vertex are sampled from their distribution and visualized as point clouds, as normal QQ-Plots [Rice 1988], and as a histogram. Figure 6 visualizes another experimental test of the conditional distribution of the vertices. Here, for each vertex in the mesh, the conditional distribution for the vertex was calculated by ﬁxing all its neighbors, and randomizing the vertex’ values. Then a Bera-Jarque test for normality [Bera and Jarque 1981] was performed for each vertex. The symbol used to draw each vertex indicates what p value the vertex received in the normality test. Since our result is an asymptotic result, an immediate question arises as to how good this approxima- tion is in practice. Figures 7 and 8 show some experimental results, which give evidence to support the quality of the approximation. Both ﬁgures compare the results obtained with the symmetric Laplacian to the results obtained by an empirical covariance matrix. The empirical covariance matrix is computed by generating a random geometries matrix X = {X 1 , . . . , X m }, where each X i is a geometry sampled from the distribution described above. Then we compute COV(X ) = Exp(XXT ) − Exp(X )Exp(X T ), where Exp(X ) is the mean vector of X . Figure 7 shows a color-coded map of the difference matrix |L−1 − αC|/|αC|, where the division is performed element-by-element. The darker the color, the closer this difference is to zero. It is evident that the two matrices are practically identical. Figure 8 shows the normalized energy contained in the ﬁrst k coefﬁcients for the spectral decomposition of a geometry vector, and the decomposition using the (eigenvectors of the) empirical covariance matrix. The higher this energy is, the fewer coefﬁcients we need to use to express more features of the geometry. It can be seen that the spectral and the PCA bases give very similar results. A fundamental question is whether the distribution we have described above also applies to 3D meshes. It will be difﬁcult to answer this question. Instead, we investigated the similarity between the population of 2D meshes governed by our distribution, and 3D meshes found in practice. For this, we appeal to the following simple observation: the effectiveness of the spectral compression method is −→ Fig. 5. The distribution of a single vertex coordinate in a random 2D mesh, using the distribution of (16). Only one vertex is sampled, while its neighbors and the rest of the mesh are ﬁxed. (a)-(c) represent the distribution of one vertex, and (d)-(f) represent the distribution of another vertex. (a), (d): Each point in the gray cloud represents the coordinates of this vertex in a random valid geometry. The mesh connectivities are drawn in black lines using the mean valid geometry for the given connectivities (zoomed in from the full mesh). (b), (e): The distribution of the x coordinate is visualized by normal QQ-plots. In such a plot, a normal probability distribution will appear linear. (c), (f): The distribution of the x coordinate is visualized by histograms. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 73 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 74 • M. Ben-Chen and C. Gotsman Fig. 6. For each vertex in the mesh, the conditional distribution of that vertex was measured by ﬁxing all its neighbors, and randomizing the vertex’ values. The Bera-Jarque test for normality [Bera and Jarque 1981] was performed. The vertices are marked according to the resulting p-values. Unmarked vertices have p < 0.001. based on the fact that, in practice, a vertex will be close to the average of its neighbours. For example, a mesh geometry in which each vertex is located exactly at the average of its neighbours, contains no information beyond the locations of the boundary vertices. Indeed, this is also the nature of the mean of our 2D distribution. Hence, the distribution of the quantity “the normalized distance between a vertex and the average of its neighbours” across a mesh is a key measure of the efﬁciency of spectral compression on this mesh. We have computed and compared the distribution of this quantity for a number of well-known 3D meshes, and also for 2D meshes sampled from our distribution. Figure 9 shows that these distributions are all very similar. While this does not prove anything about the more subtle properties of the distributions, these results lend some support to our hope that our distribution also reﬂects well the important properties of the 3D mesh distribution. It is worth mentioning that deﬁning multivariate normal distributions by specifying the coefﬁcients of the conditional means is a well-known model [Besag and Kooperberg 1995], usually referred to as condi- tional autoregressive, or auto-normal formulation. Such models frequently arise in Bayesian networks, where they describe interactions between random variables that have neighborhood relationships. 6. CONCLUSION, DISCUSSION, AND FUTURE WORK We have shown that under a few natural assumptions on the distributions of geometries, given 1D and 2D connectivities, the covariance matrix of the mesh geometries is the inverse of the symmetric Laplacian matrix of that connectivity, and hence has the same eigenvectors (in reversed order). This implies that spectral decomposition of meshes, which uses the eigenvectors of the symmetric Laplacian as the basis, is optimal. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 75 Fig. 7. The difference matrix |L−1 − αC|/|αC|, color-coded. The darker the color, the closer to zero the value in the difference matrix. The maximal (white) entry is approximately 3.4. Fig. 8. The “energy” contained in the ﬁrst k coefﬁcients, for a mesh with n = 1000 vertices. The graph shows i=1…k ci / i=1..n ci , 2 2 where ci are the coefﬁcients of the decomposition by the PCA basis (computed using the empirical covariance matrix), and the spectral basis. Note that this ﬁgure shows the coefﬁcients’ energy for a single mesh, whereas our theory relates to the similarity of the coefﬁcients on the average over many possible valid meshes. On the practical side, to achieve an optimal decomposition, the geometries should be centered to have a zero mean, before the decomposition. This is crucial since, as described in Section 3, PCA is optimal only for a random vector X , which has a zero mean. The geometries as we described them do not have such a property, therefore they should be centered before decomposition by subtracting away their mean, which can be shown to be Exp(X ) = L−1 BX —the “Tutte conﬁguration” with the given boundary, namely, where each interior vertex is at the centroid of its neighbors.. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 76 • M. Ben-Chen and C. Gotsman ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 77 There are a few more directions that are worth exploring. These are described in the next sections. 6.1 Other Distributions Our proof is based heavily on our assumptions about the distribution of meshes in one and two dimen- sions. The distributions that we impose are obviously not the only ones possible. In one dimension, one may use the same scheme as in two dimensions, and generate the geometries using a random barycentric coordinates matrix. Another possibility is to deﬁne X i to be ij =1 Y j / n=1 Y j , where Y i j are uniformly distributed random variables. This is guaranteed to generate a monotonically increasing sequence. The optimality proof in the one-dimensional case hinges on the key fact that the random variables X i − X i−1 are interchangeable, hence the proof holds for any distribution that satisﬁes that condition. Speciﬁcally, this is the case for both of the models just described, even though these do not generate uniform distributions on the class of valid geometries. It is encouraging to see that the optimality result is not too sensitive to the geometry distribution. In two dimensions, there are two main variations on our model. One possibility is not to use barycen- tric coordinates at all: For example, one can generate random X and Y coordinates, and keep only the (X , Y ) vector pairs that form a valid geometry. A geometry will be valid if all the triangles have the same orientation. Obviously, this is not an efﬁcient method to sample this distribution, since for large n, the probability that a random (X , Y ) pair will form a valid geometry is very small. The advantage of this process, however, is that it will generate geometries distributed uniformly over the class of valid geometries, but it is not clear whether our proof extends to this distribution. Another possibility is to generate random 2D geometries by modifying the distribution of the barycen- tric coordinates matrix. For example, instead of being uniform spacings, the barycentric coordinates can be wij = Y i / i=1 . . d Y i , where Y i are independent, identically distributed, uniform random variables. However, here too it is not clear whether the optimality result will still hold. Although our 2D optimality result was derived using properties of the normal distribution, which is just an approximation of the true geometry distribution, we believe that the result actually holds for the true distribution of the geometries, as in the 1D case, without resorting to normal approximations. This, unfortunately, will probably require a completely different proof. For both 1D and 2D meshes, our proof is based on the fact that the inverse covariance matrix has the same eigenvectors as the Laplacian, and there is no constraint on the eigenvalues as long as the reverse ordering is preserved. This fact implies another way to apply our proof to other distributions: one can prove that the inverse covariance equals to some integer power of the Laplacian, or any other function of the Laplacian that doesn’t change the eigenvectors and the order of the eigenvalues of the resulting matrix. 6.2 The Decay Rate of the Spectral Coefﬁcients For compression applications, it is important to know how fast the spectral coefﬁcients decay to zero. This property relates directly to the rate/distortion relation of the compression, and is also known as “Energy Packing Efﬁciency”. Based on our proof that the Laplacian equals the inverse covariance up to a constant, we can show that the spectral coefﬁcients for the meshes treated here decrease on the average. This, in itself, is an important fact, since a priori there is no reason they should even decrease. Let X ←− (xi −Avg (x j ))2 Fig. 9. The empirical distribution of the relative distance of a vertex from the average of its neighbours: Avg ((xi −x j )2 ) |j ∈ Neigh(i), where xi is the value of the X coordinate of the i-th vertex. The graphs show the histogram of this value, for (a) the “Cow” model, (b) the “Beethoven” model, (c) the “Triceratops” model, and (d) 2D geometries sampled from the distribution described in this article. All four distributions (a-d) are on the same plot, using linear (e) and log (f) scale. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 78 • M. Ben-Chen and C. Gotsman Fig. 10. 3D geometries with common connectivity and boundaries. (a) The barycentric coordinates matrix is (I − L ), where L is the Laplacian L normalized to unit row sums, and the displacement has constant variance for all vertices. (b) The barycentric coordinates matrix is (I − L ), and the displacement is smooth. It was generated by randomizing displacement values on the boundaries, and computing the displacement at each vertex by N = L−1 BN . (c, d) The displacement is computed as in (b), but random barycentric coordinates matrices are used (e, f) Random barycentric coordinates matrix and random Gaussian displacement with variance 1/n2 , where n is the number of vertices in the mesh. be a random geometry (column) vector, and C = T X the corresponding spectral coefﬁcient vector. By deﬁnition, is the orthogonal matrix of the column eigenvectors of L—the mesh Laplacian—in increasing order of the eigenvalues: L = , where = diag(µ1 , . . , µn ) is the diagonal matrix of L’s eigenvalues, in that order. We have proven that the geometry covariance matrix is Cov(X ) = αL−1 . Hence Cov(C) = Cov( T X ) = T Cov(X ) = α T L−1 . Now, since L−1 = −1 T , we obtain −1 T −1 Cov(C) = α T =α . This means that the spectral coefﬁcients are pairwise uncorrelated and decrease on the average. Since, by deﬁnition, Exp(X ) = 0, this implies that also Exp(C) = 0, so the variance of C is (α/µ1 , . . , α/µn ), which obviously decreases. For the 1D, case, the exact eigenvalues of L are known, so we can ﬁnd the decay rate of these coefﬁcients. In 1D, the Laplacian is a symmetric tridiagonal Toeplitz matrix, whose eigenvalues are πi known [Hartﬁel and Mayer 1998] to be µi = 4 sin2 ( 2(n+1) ). For large n, the argument of the sin function ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. The Optimality of Spectral Compression of Mesh Data • 79 is very small, so the sin function may be approximated by its argument. This means that the inverse eigenvalues, and hence the spectral coefﬁcients, decay like θ (1/i 2 ). 6.3 Extension to 3D Meshes In three dimensions, matters are more complicated. Just applying the barycentric method (15) to a (nonplanar) 3D boundary results in a smooth, featureless surface interpolating the boundary vertices. This is obviously not rich enough to capture interesting 3D shapes. A possible natural extension of the 2D model, which allows for 3D features to emerge, is the following linear system: X = WX + N , where N is a random displacement vector, independent of X , and the system has ﬁxed boundary vertices. This means that each vertex is a convex combination of its neighbors up to a displacement. The displacement can be any reasonable random variable, as long as it is independent of X . The variance of the displacement values will inﬂuence the smoothness of the mesh. A smooth mesh can be created by randomizing displacement values on the boundaries, and then computing the displacement values on the internal vertices by computing N = L−1 BN , where L is the Laplacian, and BN are the displacement values on the boundaries. The barycentric coordinates matrix W can be generated by uniform spacings, as in the two-dimensional case. See Figure 10 for examples of 3D meshes with different displacements and different barycentric coordinates matrices based on this model. The optimality proof for the 2D case is based on the following two key properties of the multivariate random variable X : the normal distribution of the conditionals, and the linear conditional expectation. Both those properties carry over to 3D, when using the model we have proposed, due to the displacements N being independent of X . Obviously, such a 3D model is not really satisfying, since it cannot generate “real-life” meshes—no cows will be born from random barycentric coordinates and displacements. An interesting experiment may be to generate a family of geometries from a connectivity, for example by using the “Connectivity Shapes” method [Isenburg et al. 2001]. Analyzing the distribution of geometries generated this way may provide an insight for ﬁnding the optimal basis for “real-life” 3D meshes. Unfortunately, such an analysis may be mathematically complex. REFERENCES ¨ ALEXA, M. AND MULLER, W. 2000. Representing animations by principal components. Comput. Graph. For. 19, 3 (Aug.), 411–418. ALPERT, C. J. AND YAO, S. Z. 1995. Spectral partitioning: The more eigenvectors, the better. 32th ACM/IEEE Design Automation Conference, San Francisco, CA. (June) 195–200. ARNOLD, B. C., CASTILLO, E., AND SARABIA, J. 1999. Conditional Speciﬁcation of Statistical Models. Springer-Verlag. BESAG, J. AND KOOPERBERG, C. 1995. On conditional and intrinsic autoregressions. Biometrika, 82, 4, 733–746. BERA, A. K. AND JARQUE, C. M. 1981. An efﬁcient large-sample test for normality of observations and regression residuals. Working Paper in Economics and Econometrics 40, Australian National University. CHUNG, F. R. K. 1997. Spectral graph theory. Conference Board of Mathematical Science 92. American Mathematical Society. CHUNG, F. R. K. AND YAU, S. T. 2000. Discrete Green’s functions. J. Combinator. Theo. A 91 191–214. DAVID, H. A. 1981. Order Statistics. 2nd Ed. Wiley, New York. FLOATER, M. S. 1997. Parameterization and smooth approximation of surface triangulations. Comput. Aided Geometr. Design, 14, 231–250. GOTSMAN, C., GU, X., AND SHEFFER, A. 2003. Fundamentals of spherical parameterization for 3D meshes. In Proceedings of ACM SIGGRAPH 2003. GOTSMAN, C. 2003. On graph partitioning, spectral analysis, and digital mesh processing. In Proceedings of Solid Modeling International. HALL, K. M. 1970. An r-dimensional quadratic placement algorithm. Manage. Science 17, 219–229. HARTFIEL, D. J. AND MEYER, C. D. 1998. On the structure of stochastic matrices with a subdominant eigenvalue near 1. Linear Algebra and Its Appl. 272, 193–203. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 80 • M. Ben-Chen and C. Gotsman ISENBURG, M., GUMHOLD, S., AND GOTSMAN, C. 2001. Connectivity shapes. In Proceedings of Visualization 2001. JOLLIFFE, I. T. 1986. Principal Component Analysis. Springer-Verlag, New-York. LAURITZEN, S. 1996. Graphical Models. Oxford University Press. KARNI, Z. AND GOTSMAN, C. 2000. Spectral compression of mesh geometry. In Proceedings of ACM SIGGRAPH 2000, 279–286. KOREN, Y. 2003. On spectral graph drawing. In Proceedings of International Computing and Combinatorics Conference (COCOON’03). Lecture Notes in Computer Science, Vol. 2697, Springer Verlag. PRESS, W. H., TEUKOLSKY, S. A., VETTERLING, W. T., AND FLANNERY, B. P. 1987. Numerical Recipes. Cambridge University Press. PYKE, R. 1965. Spacings. J. Royal Statist. Soc. B, 395–439. RAO, K. R. AND YIP, P. 1990. Discrete Cosine Transform. Academic Press. RICE, J. A. 1988. Mathematical Statistics and Data Analysis. Brooks Cole Publishing. ROSSIGNAC, J. 1999. EdgeBreaker: Connectivity compression for triangle meshes. IEEE Trans. Visual. Comput. Graph., 47–61. TAUBIN, G. 1995. A signal processing approach to fair surface design. Proceedings of ACM SIGGRAPH ’1995, 351–358. TAUBIN, G. AND ROSSIGNAC, J. 1998. Geometric compression through topological surgery. ACM Trans. Graph. 17, 2, 84–115. TOUMA, C. AND GOTSMAN, C. 1998. Triangle mesh compression. In Proceedings of Graphics Interface, 26–34. TUTTE, W. T. 1963. How to draw a graph. In Proceedings of the London Mathematical Society 13, 3, 743–768. Received July 2003; revised February 2004, June 2004; accepted September 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water OH-YOUNG SONG, HYUNCHEOL SHIN, and HYEONG-SEOK KO Seoul National University This article presents a physically-based technique for simulating water. This work is motivated by the “stable ﬂuids” method, developed by Stam [1999], to handle gaseous ﬂuids. We extend this technique to water, which calls for the development of methods for modeling multiphase ﬂuids and suppressing dissipation. We construct a multiphase ﬂuid formulation by combining the Navier–Stokes equations with the level set method. By adopting constrained interpolation proﬁle (CIP)-based advection, we reduce the numerical dissipation and diffusion signiﬁcantly. We further reduce the dissipation by converting potentially dissipative cells into droplets or bubbles that undergo Lagrangian motion. Due to the multiphase formulation, the proposed method properly simulates the interaction of water with surrounding air, instead of simulating water in a void space. Moreover, the introduction of the nondissipative technique means that, in contrast to previous methods, the simulated water does not unnecessarily lose mass, and its motion is not damped to an unphysical extent. Experiments showed that the proposed method is stable and runs fast. It is demonstrated that two-dimensional simulation runs in real-time. Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.6.8 [Simulation and Modeling]: Types of Simulation—Animation General Terms: Algorithm Additional Key Words and Phrases: CIP method, multiphase ﬂuid, natural phenomena, Navier–Stokes equation, physically based animation, semi-Lagrangian method, stable ﬂuids, water 1. INTRODUCTION Water, which covers two thirds of the earth, undergoes myriad types of motion in its constant interac- tions with air, solids, and living creatures. Water has featured prominently in several recent feature animations, including Finding Nemo and The Perfect Storm. The success of these movies depended greatly on visual effects in the animation of water. Physically-based approaches have been shown to effectively reproduce water movement, with quite impressive results [Foster and Fedkiw 2001; Enright et al. 2002]. However, several open challenges remain in this ﬁeld. One key issue is speeding up the simulation of water. In the case of gaseous phenomena, interactive simulation methods already have been in- troduced by Stam [1999]. The method is called “stable ﬂuids”, which allows a large simulation time step to be used without causing instabilities. Unfortunately, this method is known to suffer from large amounts of numerical dissipation, which results in loss of mass. This is not important when simu- lating dissipative media such as fog or smoke, but it is not tolerable when animating intrinsically This research was supported by the Korea Ministry of Information and Communication. This research was also partially supported by the Automation and Systems Research Institute at Seoul National University and the Brain Korea 21 Project. Authors’ address: Graphics and Media Lab, School of Electrical Engineering #001, Seoul National University, San 56-1, Shillim-dong, Kwanak-ku, Seoul, 151–741, Korea; email: song@graphics.snu.ac.kr. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0081 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 81–97. 82 • O.-Y. Song et al. nondissipative substances like water. Another undesirable property of the stable ﬂuids method that must be noted is numerical diffusion, which dampens the ﬂuid motion. Although damping is an inher- ent property of all ﬂuids, the damping caused by numerical diffusion in the stable ﬂuids method is too severe. Therefore, if we wish to simulate water using an approach based on the stable ﬂuids method, we must modify that method to prevent the numerical dissipation and reduce the numerical diffusion. This article presents a new physically-based method for simulating water. The proposed method, which is based on the semi-Lagrangian methodology, retains the speed and stability of the stable ﬂuids technique, while including mechanisms to ﬁx the problems of numerical dissipation and diffusion. To obtain nondissipative water, we adopt the constrained interpolation proﬁle (CIP) method, which has been shown to remarkably reduce dissipation due to the use of coarse grids. To prevent dissipation due to the use of a large time step, we propose a novel particle-based approach, which we show to be quite effective at preventing dissipation of small-scale features. This particle-based approach is also used to simulate droplets and bubbles, which contributes to the overall visual realism. In addition, compared to existing methods, the proposed method simulates water-air interactions more accurately by employing the multiphase dynamic equations that account for the presence of air. The rest of the article is organized as follows: Section 2 reviews previous work; Section 3 formulates the multiphase ﬂuid; Section 4 describes the CIP-based ﬂuid solver; Sections 5 and 6 present our particle-based technique for preventing dissipation; Section 7 reports our experimental results; and ﬁnally, Section 8 concludes the article. 2. PREVIOUS WORK Early work on physically-based simulation of water for graphics applications concentrated on animating the height-ﬁeld representation of the water surface. To obtain interactive performance, researchers used the two-dimensional (2D) approximation of the Navier–Stokes equations. Kass and Miller [1990] generated the height ﬁelds using an approximate version of the 2D shallow water equations. To simulate water–object interactions, Chen and Lobo [1995] solved the 2D Navier–Stokes equation that includes pressure. O’Brien and Hodgins [1995] proposed a method for simulating splashing liquids by integrating a particle system into a 2D height-ﬁeld model. Height ﬁelds cannot be used to represent water that is undergoing a highly dynamic motion such as pouring. To handle such motions, researchers turned to the 3D Navier–Stokes equations. Foster and Metaxas [1996, 1997a] animated 3D liquids by modifying the Marker and Cell method proposed by Harlow and Welch [1965]. In addition, Foster and Metaxas [1997b] simulated gases by using an explicit ﬁnite difference approximation of the Navier–Stokes equations. Stam [1999] introduced the unconditionally stable ﬂuid model, which utilizes the semi-Lagrangian method in combination with an implicit solver. This model gave signiﬁcantly improved simulation speeds, but suffered from numerical dissipation. To reduce the dissipation in simulations of gaseous ﬂuids, Fedkiw et al. [2001] proposed the use of vorticity conﬁnement and cubic interpolation. Based on the stable semi-Lagrangian frame-work, Rasmussen et al. [2003] proposed an efﬁcient method for depicting large-scale gaseous phenomena, and Feldman et al. [2003] proposed an explosion model that incorporated a particle-based combustion model into the semi-Lagrangian framework. Treuille et al. [2003] proposed a constrained optimization tech- nique for keyframe control of ﬂuid simulations and McNamara et al. [2004] improved the optimization speed drastically by adapting the adjoint method. In order to handle 3D liquids, the semi-Lagrangian scheme must be augmented with a robust and ac- curate method for tracking the liquid surface. To address this issue, Foster and Fedkiw [2001] proposed a novel method for representing a dynamically evolving liquid surface, which was based on combining the level set method with massless marker particles. Enright et al. [2002] improved this hybrid scheme by introducing the “particle level set method”, which could capture water surface with a remarkable ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 83 accuracy. The particle level set method was employed in recent studies of ﬂuids by Carlson et al. [2004], Goktekin et al. [2004], and Losasso et al. [2004]. Takahashi et al. [2003] simulated multiphase ﬂuids by employing the CIP method, coupled with the volume of ﬂuid scheme; their method simulated the water-air interaction properly, instead of simulating water in a void space. When we are to animate water at an interactive rate, as demonstrated by Stam [1999] in the case of gas, then the use of large time steps should be allowed. But it can cause dissipation of mass. In Foster and Fedkiw [2001] and Enright et al. [2002], the time step size had to be restricted to prevent loss of mass. Although the CIP scheme used by Takahashi et al. [2003] lessened the degree of the dissipation, loss of mass was still noticeable when large time steps were used. Several particle-based methods have been proposed as alternatives to the above grid-based ap- proaches. Miller and Pearce [1989] simulated ﬂuid behavior using particles connected with viscous springs. Terzopoulos et al. [1989] adopted a molecular dynamics model to simulate particles in the liq- uid phase. Stam and Fiume [1995] introduced “smoothed particle hydrodynamics” (SPH) to depict ﬁre and gaseous phenomena. In SPH, the ﬂuid is modeled as a collection of particles with a smoothed poten- z tial ﬁeld. Premoˇ e et al. [2003] introduced the use of the moving particle semi-implicit method (MPS) for simulating incompressible multiphase ﬂuids. One drawback of particle-based methods is that, if insufﬁcient particles are used, they tend to produce grainy surfaces. To prevent this, a sufﬁciently large number of particles must be used, which increases the computational cost. 3. FORMULATION AND OVERVIEW In order to produce realistic movement of water in the presence of air, we base our method on the mul- tiphase Navier–Stokes equations in combination with the level set method. The multiphase Navier– Stokes equations can simultaneously represent both water and air. The level set method, which can represent the water-air interface as an implicit surface, has been shown to be a robust method for capturing topological changes of water surfaces. Furthermore, the surface curvature can be accurately calculated from the level set values, and hence the surface tension, which is proportional to the curva- ture, can be easily incorporated into the dynamic simulation. We start by introducing the incompressible Navier–Stokes equations for a multiphase ﬂuid. Let u = (u, v, w) denote the velocity ﬁeld of the ﬂuid. Then, the ﬂow of ﬂuid is described by ∇ · u = 0, (1) and ∂u f ν ∇p = −u · ∇u + + ∇ 2 u − , (2) ∂t ρ ρ ρ where p is the pressure, ρ is the density, ν is the kinematic viscosity, and f represents the external forces per volume. Equations (1) and (2) state that mass and momentum, respectively, should be conserved. To treat the immiscible multiphase ﬂuid consisting of water and air within a single formulation, we employ the level set function φ [Osher and Sethian 1988; Sussman et al. 1994]. φ is an implicit signed distance function deﬁned to be positive for water, and negative for air. Thus, the sign of φ also determines the density and viscosity of the medium. The water dynamically evolves in space and time according to the underlying ﬂuid velocity ﬁeld u. The updates in the level set values due to u are expressed by the level set equation: ∂φ + u · ∇φ = 0. (3) ∂t ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 84 • O.-Y. Song et al. The surface of water, which will be a focal point throughout this work, can be obtained by tracking the locations for which φ = 0. To solve the above equations numerically, we divide the space into a ﬁnite number of cells.We evaluate the pressure and level set values at the cell center, but we evaluate the velocity at the center of each cell face. This approach is the classical staggered grid discretization [Harlow and Welch 1965], which naturally enforces boundary condition and avoids the checkerboard pattern of the pressure [Trottenberg et al. 2001]. At each time step, our simulator performs the following three steps: (1) Advect the level set: The level set φ is advected according to Equation (3), which causes the density and viscosity ﬁelds appearing in Equation (2) to be updated. (2) Update the velocity: Equation (2) is solved for u using the following procedure [Stam 1999]: (1) calculate the advection component u · ∇u using the semi-Lagrangian method; (2) apply the forces f/ρ; (3) add the effect of the viscous term ν/ρ∇ 2 u by employing implicit central differencing; and (4) project the velocity ﬁeld so that the condition ∇ · u = 0 is met. (3) Simulate droplets/bubbles: Droplets/bubbles are identiﬁed and simulated using the particle dy- namics until they merge into the body of water/air. Execution of the above procedure produces the φ and u of the next time step. The method for implement- ing Steps 1 and 2 is presented in the following section, and the implementation of Step 3 is described in Section 5. 4. CIP-BASED FLUID SIMULATOR The framework of our ﬂuid simulator is based on the semi-Lagrangian scheme, which was brieﬂy introduced in Section 3. A detailed description of the semi-Lagrangian scheme can be found in Stam oe [1999] and Staniforth and Cˆ t` [1991]. In the method proposed here, we make several modiﬁcations to the previous semi-Lagrangian scheme to reduce the numerical dissipation and diffusion. This section describes those modiﬁcations. 4.1 CIP Advection In the semi-Lagrangian scheme, advection is implemented by referring to the function value at x−u t.1 Since the physical values of φ and u are deﬁned only at discrete points, the function values for x − u t can be obtained by linearly interpolating the neighboring grid points. This approach is computationally efﬁcient and unconditionally stable [Stam 1999]. However, it may smooth out the subcell features. This problem, referred to as nonphysical numerical diffusion, causes the movement of ﬂuid to be excessively damped, which hampers the generation of turbulent effects such as the formation of water droplets or air bubbles in violently interacting multiphase ﬂuids. It also causes dissipation of the mass. Fortunately, an anti-diffusive technique called the Constrained Interpolation Proﬁle (CIP) method is proposed by Yabe and Aoki [1991] and Yabe et al. [2001]. We adopt the CIP method when solving Equation (3), resulting in a reduction of mass dissipation, and also when solving the advection term u · ∇u in Equation (2), leading to a reduction in the degree of damping. Adoption of the CIP method allows us to simulate phenomena such as turbulent ﬂows or swirls with reasonable visual quality. The key idea of the CIP method is to use not only the function values at the grid points, but also the spatial derivatives at those points for constructing the proﬁle inside the grid cell. For example, in the one-dimensional case, the proﬁle corresponding to [xi , xi+1 ] now has four conditions φi , φi , φi+1 , and 1 In fact, more sophisticated backtracking methods can be used. In this work, we used the second order Runge-Kutta backtracking. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 85 φi+1 , and can be represented by the third order polynomial (X ) = [(a X + b)X + φi )]X + φi , (4) where X = x − xi for x ∈ [xi , xi+1 ]. The coefﬁcients a and b can be expressed in terms of the four conditions: a = (φi + φi+1 )/ x 2 − 2 φ/ x 3 , b = 3 φ/ x 2 − (2φi + φi+1 )/ x, where x = xi+1 − xi and φ = φi+1 − φi . The spatial derivatives used in the CIP method are directly determined from the differentiated version of the original advection equation. To advect the level set values, we differentiate Equation (3) with respect to ξ , which gives ∂φξ + u · ∇φξ = −uξ · ∇φ, (5) ∂t where φξ = ∂φ/∂ξ , uξ = ∂u/∂ξ , and ξ is one of the spatial variables (x, y, z). The task of solving the above equation for φξ can be performed in two steps: ﬁrst, solving the nonadvective part ∂φξ /∂t = −uξ · ∇φ using ﬁnite differencing; then advecting the result according to ∂φξ + u · ∇φξ = 0. (6) ∂t Noting that the advections of Equations (3) and (6) are driven by the same velocity ﬁeld u, we can advect both φ and (φx , φ y , φz ) by referring to the same point of the proﬁle. For the one-dimensional case, suppose that the grid point x j is backtracked to xr ∈ [xi , xi+1 ]. Then, the advected results φ j and φ j can be obtained by evaluating Equation (4) and its differentiated form at X r = xr − xi . Namely, φ j = (X r ) and φ j = (X r ). A problem can arise if we use the proﬁle of Equation (4) as it stands. In the one-dimensional case, for example, the value (X r ) may lie outside the range [φi , φi+1 ], which can cause instabilities. One solution to this problem is the rational CIP method proposed by Xiao et al. [1996], which suppresses the above oscillation problem by using a rational function instead of a polynomial function. In the present work, we developed an alternative CIP scheme that explicitly modiﬁes the derivatives, with this modiﬁcation being applied only to the cells in which oscillations are present. This scheme, which is described in detail in the Appendix, guarantees a monotonic proﬁle; hence, we call it monotonic CIP. Because it uses polynomials, monotonic CIP runs faster than rational CIP. Higher (i.e., two- or three-) dimensional CIPs have been proposed by Yabe et al. [2001]. In those methods, however, the derivative constraints are not applied at every grid point, which can result in a nonmonotonic proﬁle. Here we implement higher dimensional CIPs based on the one-dimensional monotonic CIP solver. Our implementation of higher dimensional CIPs, which is described below, is always stable. Consider the two-dimensional case shown in Figure 1(a), where r is the backtracked point. For this system, we must determine [φ, φx , φ y , φxy ]r from the values at the four corners: (i, j ), (i + 1, j ), (i, j + 1), and (i + 1, j + 1). Here, [·]x denotes the function and derivative values at point x. The one-dimensional solver can determine [φ, φx ]a from [φ, φx ](i, j ) and [φ, φx ](i+1, j ) , and [φ y , φxy ]a from [φ y , φxy ](i, j ) and [φ y , φxy ](i+1, j ) . Similarly, it can determine [φ, φx , φ y , φxy ]b . Then, it can determine [φ, φ y ]r from [φ, φ y ]a and [φ, φ y ]b , and [φx , φxy ]r from [φx , φxy ]a and [φx , φxy ]b . In the three-dimensional case, the CIP implementation additionally calls for the values of φxy , φ y z , φxz , and φx y z . Obtaining those values by analytic differentiation of the original formula involves a large amount of computation. However, our experiments indicated that approximating the second order and ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 86 • O.-Y. Song et al. Fig. 1. (a) Two-dimensional CIP interpolation; the x-axis is along the horizontal direction and the y-axis is along the vertical direction. (b) Schematic of a situation where φi−1 , φi , and φi+1 are aligned. higher derivatives by ﬁnite differencing of the ﬁrst order derivatives caused no visually noticeable difference in the result, but signiﬁcantly reduced the amount of computation. We close this section by noting the attractive aspects of CIP in comparison with other interpola- tion methods. In cases where φi−1 , φi , and φi+1 are aligned, such as that shown in Figure 1(b), spline techniques that do not utilize the derivative information interpret φ as being straight. In contrast, because CIP utilizes the spatial derivatives φi−1 , φi , and φi+1 of the original equation, it results in more accurate modeling of the real situation. Therefore, the CIP method allows us to use a fairly coarse grid. Another advantageous feature of the CIP method is that, whereas high-order interpolation meth- ods for the semi-Lagrangian scheme such as the cubic spline, quintic Lagrange [Staniforth and Cˆ t` oe 1991], and monotonic cubic polynomial [Fedkiw et al. 2001] methods require three stencils (or four grid points) to construct the proﬁle in the one-dimensional case, the proﬁle used in the CIP method can be constructed with information from a single cell. This feature is particularly useful when treating boundaries. 4.2 Force We now apply the external forces, f, to the result of the advection. The external forces consist of gravity and surface tension. In the interactive system we develop, mouse or keyboard input can also be used to generate additional forces. The gravitational force is expressed as ρg, where g is the gravitational acceleration. The surface tension is given by fst = −ρσ κ(φ)δε (φ)∇φ, (7) where σ is a constant coefﬁcient, κ is the local curvature given by ∇φ κ(φ) = ∇ · . (8) |∇φ| δε is a smeared delta function which, in the present work, has the form πφ 1 + 1 cos ε : |φ| ≤ ε, δε (φ) = 2ε 2ε (9) 0 : otherwise, where we use ε = 1.5 x for the smearing width. Equation (7) is the continuum surface force (CSF) model proposed by Brackbill et al. [1992]. It allows us to treat surface tension as a body force, and does not require explicit information on the geometry and position of the water surface. In order to show surface tension effects accurately, a strict time step restriction of O( x 1.5 ) is required [Brackbill et al. 1992; Sussman et al. 1994]. Since we do not restrict time steps in this work, the effects ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 87 may be somewhat suppressed. In large scale liquid phenomena such as the ones experimented in this article (Section 7), however, the surface tension effects are visually not signiﬁcant. 4.3 Projection The result obtained by applying the external forces then goes through the diffusion step, ν/ρ∇ 2 u, for which we use the implicit solver following the approach described in Stam [1999]. To process the last term of Equation (2), −∇ p/ρ, we impose the ﬂuid incompressibility condition represented in Equa- tion (1), which produces the Poisson equation: ∇p ∇ ·u ˜ ∇· = , (10) ρ t ˜ where u is the intermediate velocity ﬁeld obtained by processing Equation (2) up to the diffusion term. Equation (10) can be discretized as −1 −1 −1 −1 ρn+ 1 + ρn− 1 pn − ρn+ 1 pn+1 − ρn− 1 pn−1 2 2 2 2 n={i, j,k} (11) =− 1 t un+ 1 − un− 1 , ˜ ˜ 2 2 n={i, j,k} where pn±1 are the pressure values taken from the centers of the neighboring cells, and un± 1 and ρn± 1 ˜ 2 2 are the velocity and density values taken from the cell faces. We can assemble the collection of equations of the form of Equation (11), covering the domain space into a large linear system Ap = b, (12) where p is the vector of unknown pressures required to make the velocity ﬁeld divergence free. A is a positive-deﬁnite, symmetric matrix in which density is a variable in space. This system can be efﬁciently solved using the incomplete-Cholesky preconditioned conjugate gradient method (ICCG) [Golub and Loan 1996].2 The solution of Equation (12) is then used to calculate ∇ p at the cell faces. Finally, we obtain the divergence free velocity ﬁeld by ∇p u=u− ˜ t . (13) ρ 5. PREVENTION OF DISSIPATION IN SMALL-SCALE FEATURES Although the CIP-based simulator described in the previous section can represent the content of each cell up to the third order, it still suffers from the following two problems: (1) when a large time step is used, it produces non-negligible mass errors; and (2) it cannot represent subcell-level features such as water droplets and air bubbles. In this section, we develop a novel technique to solve these problems. The proposed technique uses particles3 to complement the grid-based framework, thereby overcoming some of the limitations of that framework. 2 Because we use the conjugate gradient method for solving the linear system which is the most time consuming part, the overall time complexity order in our method is approximately O(n3/2 ) in the two-dimensional case and O(n4/3 ) in the three-dimensional case, where n is the total number of grid points. 3 Note that our use of particles should not be confused with that proposed by Enright et al. [2002]. They use massless marker particles to obtain a more accurate water surface, whereas we use physical particles that have mass as well as volume to represent water droplets and bubbles. Compared to their method, our method requires far fewer particles and, as a consequence, has a negligible computational overhead. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 88 • O.-Y. Song et al. Fig. 2. Identiﬁcation of droplets: (a) Apparent droplet, (b) Unapparent droplet. In (b), δ is a small positive value such that δ < ε. 5.1 Identiﬁcation of Dissipative Cases If the level set value of a particular cell center is positive, whereas those of adjacent cell centers are negative, we can interpret the isolated region to be a droplet as shown in Figure 2(a). A potential problem arising from having such a small isolated region is that it may dissipate or even be lost in subsequent simulation. In such cases, therefore, we transform the region into a droplet that undergoes Lagrangian motion from that moment onwards. We determine the volume4 of the droplet based on the level set values. The approximation we use is Vf = H(φ(x))d x ≈ Hε (φ(xc )) x y z, (14) c c where the set c represents the droplet (the circular region of Figure 2(a)), c is the index ranging over the cells that cover c (the shaded cells of Figure 2(a)), and Hε is the smeared Heaviside function5 0 : φ < −ε φ πφ Hε (φ) = 2 + 2ε + 2π sin ε 1 1 : |φ| ≤ ε (15) 1 : φ > ε. As in Equation (9), we use ε = 1.5 x. Since the ﬂuid content of the isolated region is already transformed into a droplet, we reduce the level set values of the region to values that are small enough (e.g., −ε) that the region will not be mistakenly identiﬁed as containing water during subsequent processing. Generation of bubbles can be thought of as the dual of that of droplets; identiﬁcation of isolated regions and approximation of bubble volume can be done with the same procedure and same equations, except that, in this case, we use the negated version of original level set values. In addition to the cases in which the isolated region can be identiﬁed by simply looking at the signs of the level set values, referred to here as “apparent” cases, there also exist “unapparent” cases in which the isolated region cannot be identiﬁed from the sign of the level set values. For example, consider the case in which a 2 × 2 grid with small negative level set values is surrounded by the cells with large negative level set values, as shown in Figure 2(b). Since the level set function we use represents signed distance, small negative values imply that water is nearby. Therefore, the situation considered here can be reconciled only by introducing a droplet somewhere around ?-marked location of Figure 2(b). The volume of the droplet is again computed by Equation (14). During dynamic simulation of a multiphase 4 Since we assume incompressibility of both media, mass-preservation is equivalent to volume-preservation. 5 We also smear the discontinuities in density/viscosity across the air-water interface using this function. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 89 ﬂuid, the above situation can occur when a body of water is splitting (or two bodies of air are merging). Cases of unapparent bubbles are treated similarly. There is another source of errors of misinterpreting the phase, which is related to the backtracking in semi-Lagrangian advection. Speciﬁcally, when a large time step is used, backtracking may leave some cells never referred. In such cases, the masses of those cells are lost in the subsequent time steps. We prevent this type of error by converting the nonreferred cells into droplets/bubbles. Again, the volumes of the cells are determined by evaluating Equation (14) for the nonreferred cells. The droplets/bubbles are then advected by the underlying velocities. Droplets/bubbles that are advected into the water/air medium are ignored. In pure grid-based methods, the above cases are beyond the resolution limit and thus no droplet/bubble is formed. By introducing a small amount of extra computation, however, the procedures described above can generate droplets/bubbles. This not only reduces the mass dissipation but also enhances the visual details of the dynamically evolving water. The volume approximation of Equation (14), which uses the smeared Heaviside function given in Equation (15), theoretically has ﬁrst-order accuracy. Our experiments showed that it is computationally efﬁcient and gives sufﬁcient visual realism for our purposes. If greater accuracy is needed, contouring methods such as the marching cube algorithm [Lorensen and Cline 1987] can be used. 5.2 Dynamic Simulation of Droplets/Bubbles As a droplet/bubble advances with the initial velocity, it experiences gravitational and drag forces, as well as pressure from the surrounding ﬂuid, which cause the fragment to accelerate or decelerate. The forces acting on the fragment can be summarized as f = m f g + αd r 2 (u − x) u − x − V f ∇ p, ˙ ˙ (16) where m f is the mass, V f is the volume, αd is the drag coefﬁcient, r is the radius, x is the current velocity ˙ of the fragment, and u is the interpolated velocity of the grid-based ﬂuid measured at the center of the fragment. The third term, which represents the force due to the pressure difference, produces buoyancy. In Equation (16), the second and third terms model the interaction with the neighboring ﬂuid. Therefore, the action force given in Equation (16) must be coupled with the reaction force −αd r 2 (u − x) u − x + ˙ ˙ V f ∇ p, acting on the grid-based ﬂuid model. The movement of fragments over time is obtained by standard numerical integration. If two or more fragments overlap during the dynamic simulation, they are merged into a single larger fragment. 5.3 Restitution of Droplets/Bubbles When either of the following two conditions are met, we restitute the droplets/bubbles to the grid-based ﬂuid model: (1) when the volume of a fragment becomes larger than twice the cell size, or (2) when a fragment hits the surface or moves into the same phase ﬂuid. In the ﬁrst case, the fragment has become too big to be represented as a nondeforming particle, and thus its behavior is better modeled within the grid-based framework. Therefore, in such cases, we remove the fragment and restitute its mass to the grid-based ﬂuid model. This restitution is executed by updating the level set values and setting the grid point velocities to the fragment velocity. We update the level set values by φ(xi ) = sp (r p − |xi − x p |), (17) where sp = +1 for the case of a water droplet, and −1 for the case of an air bubble, r p is the radius of the fragment, x p is the center of the fragment, and xi is the grid point being updated. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 90 • O.-Y. Song et al. Fig. 3. Instantaneous hollows and ripples on the water surface created by restitution of Lagrangian droplets/bubbles. The second case corresponds to the situation in which a droplet/bubble returns to the body of water/air, and, therefore, we remove the fragment. In the case where the fragment hits a surface cell, the cell velocity is updated by taking the average of the previous cell velocity and the fragment velocity. As for the level set values, we determine the new values for the cells covering the fragment by taking the inverse functions of Equations (14) and (15). In the case where the fragment moves into the same phase ﬂuid, we perform the same procedure as described above, pretending that it hit a surface cell. The above procedure we devised for updating the level set values and cell velocities, interestingly, contributes to creating visual details at water surface. The procedure in fact forms small ripples: a droplet falling into water contributes a downward velocity to the cell, which generates an instantaneous hollow. But soon, the region is pushed back and forms a small bump (See Figure 3.). 5.4 Variations for Visual Realism In the above procedures, several parameters can be controlled to adjust the visual aspects of the simulation. (1) In order to produce splashes and bubbles of smaller scales, we can generate multi- ple droplets/bubbles of smaller size, instead of generating a single fragment of about the cell size. In practice, we let the number of generated fragments be proportional to the velocity of the ﬂuid portion being converted. (2) We can model the geometrical shape of a fragment as an ellipsoid instead of a sphere, in accordance with the velocity. (3) In order to represent foam at the water surface, instead of removing bubbles immediately after they appear on the surface, we assign them a life time that is inversely proportional to the bubble size. 6. ADDITIONAL CONSIDERATIONS 6.1 Reinitialization of Level Set Values As the level set is advected and makes interactions with the droplets/bubbles as described in the previous section, the signed-distances can be corrupted. This may lead to the development of noisy features that can be problematic when approximations such as ﬁnite-differencing are used. For example, such noisy features can lead to large errors in the curvature calculation. For this reason, we need to introduce a procedure to rectify the level set to ensure that it maintains the signed distance property. This procedure is called reinitialization or redistaincing [Sussman et al. 1994, 1998; Peng et al. 1999; Sethian 1996; ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 91 Tsai et al. 2003]. We used Sussman’s method [1998]. The method is performed using ∂φ + sgn(φ)(|∇φ| − 1) = 0, (18) ∂τ where τ is a ﬁctitious time and sgn(φ) is a smeared signum function given by φ sgn(φ) = . (19) φ2 + |∇φ|2 ( x)2 In most cases, the corruption is not severe and the procedure recovers the correct signed distance values within several ﬁctitious time steps. Hence, the reinitialization step is computationally inexpensive. We can speed up the procedure by reinitializing only the portion within a band of the water surface [Sussman et al. 1998]. Details on the discretization of Equation (18) can be found in Sussman et al. [1998] and Osher and Fedkiw [2002]. 6.2 Interaction with Rigid Objects This section presents a simple technique for simulating the inﬂuence of water on rigid objects, as well as that of rigid objects on water. We employ the volume of solid technique proposed by Takahashi et al. [2003] with modiﬁcations inspired by the work of Foster and Fedkiw [2001] and Enright et al. [2002]. The proposed technique may not produce valid results in an engineering sense, but it does reproduce the physical aspects of object-water interactions with acceptable visual realism. When a rigid object is immersed in water, we mark the ﬂuid cells whose centers are contained within the object. A self-evident constraint in this case is that the ﬂuid should not ﬂow into the rigid object; this can be checked for each marked cell s by calculating us · ns , where us is the ﬂuid velocity, and ns is the normal of the object surface. If us · ns < 0, then the ﬂuid is ﬂowing into the solid and, to stop this, we remove the normal component of us , while leaving the tangential component unchanged. The above procedure is for calculating the change in ﬂuid velocity caused by the presence of an object. The movement of an object caused by the surrounding ﬂuid is simulated by considering the ﬂuid pressure acting on the surface of the object. Given that the gradient of the pressure generates forces, the total external force F and moment T acting on the center of mass rc of the rigid object can be expressed as F = Mg + (−∇ ps · ns )ns S, (20) s T= (rs − rc ) × (−∇ ps · ns )ns S, (21) s where M is the mass of the object, s is the index ranging over the marked cells, ps is the ﬂuid pressure of the cell, rs is the position of the cell, and S is the area of the object surface subsumed in the cell. When simulating water that is interacting with a quickly moving rigid object, a reasonably small time step size must be used to calculate the movement of rigid object in a moderate level of accuracy. 6.3 Discussion on Mass Errors The procedures described in Sections 4 and 5 are quite effective at blocking mass dissipation in the numerical simulation of multiphase ﬂuids. However, the errors resulting from the use of ﬁnite-sized grids and ﬁnite time steps are inevitable, and in rare cases, these errors can accumulate to some extent. In situations where the amount of water needs to be kept constant, we optionally perform a global compensation procedure based on modiﬁcation of the level set values at every time step. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 92 • O.-Y. Song et al. If the initial volume of water is V0 , and the volume at each time step is Vi , we should compensate the water volume by Vi = V0 − Vi at the end of each simulation step. The volume Vi and the area of water surface Si can be calculated using Vi = H(φ(x))d x ≈ Hε (φ(x)) x y z + V p , (22) Si = δ(φ(x))|∇φ(x)|d x ≈ δε (φ(x))|∇φ(x)| x y z, (23) where V p is the total volume of the droplets. Because the level set is a signed distance function, we can use the approximation Vi ≈ Si φ, (24) which allows us to calculate the scalar value φ. Now, adding φ to the previous level set values of the entire domain (for both water and air) results in raising the water level by φ, while not changing the geometrical shape of the water surface. The above compensation procedure can have a ﬁrst-order error, since we use the smeared Heaviside function. However, the compensation procedure performed at every time step is done with respect to V0 , and the error is distributed over all of the cells in the system. Therefore, even though the procedure is not physically-based, it effectively prevents the error accumulation without producing any noticeable artifacts. The contribution of this procedure in the prevention of mass dissipation is small. For example, in the 2D water drop experiments on a grid of resolution 64 × 48, the procedure compensated only 0.7% of errors. Furthermore, contribution from the global compensation decreases as we increase the resolution of the grid. In fact, in obtaining the 3D demos, we did not run the compensation procedure.6 7. EXPERIMENTAL RESULTS The technique presented in this article is implemented in two dimensional and three dimensional versions, on a PC with an Intel Pentium 4 3.2 GHz processor, 1 GB memory. The innovations introduced in the present work make possible the real-time simulation of the movement of 2D water. Moreover, the fast simulation speed of the proposed method enabled highly interactive control of the water motion through several types of action. For example, by clicking or dragging the mouse, we could add a volume of water, create/remove solid walls, or introduce force ﬁelds, all with immediate effect. This section summarizes our experimental results. The 2D simulator based on a grid of resolution 64 × 48 ran at 30 ∼ 60 fps, which is sufﬁciently fast to enable real-time simulation of water that is subjected to interventions such as applying forces or adding water volumes. Clip 1 contains a real-time video capture taken during the simulation (see Figure 4 for a snapshot). Although the CFL number (i.e., u t/ x) was 25 or higher, the simulation ran stably. The mass error is not evident, and the movement of water is clearly nondissipative. We were able to let ink mix over the water medium, as shown in Stam [1999], which provided a perceptual cue on the movement of water. The clip shows (1) the movement of water at different gravitational conditions, (2) the generation of droplets/bubbles in turbulent water, (3) the behavior of water when interacting with solids, (4) changes in buoyancy due to changes in the density of the ﬂuid, (5) the result of replacing the CIP advection with linear advection, which noticeably increases dissipation and damping, and (6) the interactions water makes with nonvoid air due to our multiphase implementation of the ﬂuid. t/ x > 50), the procedure can 6 In very coarse grids (e.g., 15×15), or when very large time steps are used (e.g., the CFL number u produce a large amount of modiﬁcation in the position or curvature. In such cases, the effect of compensation can be noticeable. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 93 Fig. 4. Real-time 2D simulation. Fig. 5. A football thrown into water. We ran the 3D version of our simulator to experiment the cases shown in Figures 5, 6, and 7, and produced the animations shown in Clips 2, 3, and 4, respectively. In the case shown in Figure 5, a football was thrown into water with a large initial velocity, which realistically produced a violent splash of water and a trail of air bubbles behind the ball. The density ratio of the ball relative to water was 0.75. In the case shown in Figure 6, a cubic container containing water and an object was rotated. The collision the water made with the wall created a highly dynamic water surface and plenty of droplets and bubbles. In the case shown in Figure 7, an empty cup was drowned into water, which caused the air to be released and to produce a dynamic response with water. If only water-phase was simulated, water would ﬁll the cup without experiencing any resistance, thus the cup would submerge smoothly, which is not physically correct. We constrain all ﬂuid fragments to be within the ﬂuid simulation domain. So, the ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 94 • O.-Y. Song et al. Fig. 6. Simulation of water in a rotating cubic container. Fig. 7. A cup drowned into water. Table I. 3D Simulation Information (Density Ratio is Air:Water:Object. All experiments are performed on the same condition except object densities; the viscosity coefﬁcient of water is 1.0 × 10−2 , the viscosity coefﬁcient of air is 2.0 × 10−4 , and the surface tension coefﬁcient is 0.5.) Experiment Density Ratio Total Frame Simulation Time (sec/frame) Rendering Time (sec/frame) Football 1:800:600 180 51.7 162.1 Container 1:800:500 260 40.3 282.6 Cup 1:800:750 126 34.0 159.1 droplets getting out of the vertical boundaries (the walls and the front transparent glass) are stopped, and then slide down along the surface of the wall. The simulations were performed on a 80 × 80 × 80 grid. Simulation of one time step took about 30-40 seconds. A ﬁxed time step of t = 30 second was 1 used in all the above 2D and 3D simulations, except for the case of the football. As for the football example, in order to obtain a reasonable visual quality, we adjusted t so that the CFL number does not exceed 5.0. Extraction of water surface was done using the marching cube algorithm [Lorensen and Cline 1987], and rendering was done by Mental Ray. The simulation information for each experiment along with the time required to simulate the motion and render the images appears in Table I. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 95 Fig. 8. Plot of (φi , φi+1 ): the shaded region guarantees monotonic proﬁles in the case of φ ≥ 0. 8. CONCLUSION The introduction to the graphics community of the semi-Lagrangian methodology by Stam in 1999 opened the way for the interactive manipulation/control of gaseous ﬂuids. Since then, it has been an open problem to extend the technique to water. In this article, we have proposed a solution to this problem. The problem mainly consisted of two technical challenges: (1) modeling a multiphase ﬂuid, and (2) ﬁnding a dissipation-free technique. In this article, the problem of modeling a multiphase ﬂuid was solved by combining the Navier–Stokes equations with the level set method. The problem of preventing dissipation was solved by two means: (1) adoption of the CIP method, which could model the subcell-level details with third-order accuracy; (2) development of a particle-based method to prevent dissipation in small-scale features such as droplets/bubbles. Instead of simulating water in a void space, the multiphase ﬂuid formulation proposed here properly simulates the dynamic movement of water, while it engages in complex interactions with surrounding air. Due to the measures taken to prevent dissipation in the proposed method, the simulated water does not unnecessarily lose volume or its motion is not damped to an unphysical extent. Although “stable but nondissipative water” this article proposes makes a signiﬁcant improvements in modeling nondissipative behavior of water, the focus has not been put on producing an accurate shape of water surface. When an application requires more accurate modeling of water surface, then Enright et al. [2002] can be adopted to the framework we propose, but with an increased amount of computation. As for modeling droplets/bubbles, to our knowledge, the present work is the ﬁrst that proposed a mechanism for two-way exchange of masses between the grid-based framework and particles for the conservation of mass and momentum. More accurate modeling of the geometrical shape and size of droplets/bubbles needs further study. APPENDIX Monotonic CIP In this appendix, we develop a method for modifying φi and φi+1 , such that the proﬁle becomes monotonic. If, for simplicity, the grid size (xi+1 − xi ) is 1, differentiation of Equation (4) produces (X ) = (3X 2 − 4X + 1)φi + (3X 2 − 2X )φi+1 − (6 φ)X 2 + (6 φ)X . (25) ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 96 • O.-Y. Song et al. When φ ≥ 0, the necessary and sufﬁcient condition for the proﬁle to be monotonically increasing inside the cell is (X ) ≥ 0. By manipulating Equation (25), we found that this condition can be reduced to [φi ≥ 0, φi+1 ≥ 0, φi + φi+1 ≤ 3 φ] OR [9 φ − 6 φ(φi + φi+1 ) + (φi + φi+1 )2 − φi φi+1 ≤ 0], which corresponds to the shaded region in Figure 8. Similarly, when φ < 0, the necessary and sufﬁcient condition for the proﬁle to be monotonically decreasing is [φi ≤ 0, φi+1 ≤ 0, φi + φi+1 ≥ −3 φ] OR [9 φ + 6 φ(φi + φi+1 ) + (φi + φi+1 )2 − φi φi+1 ≤ 0]. Therefore, the monotonic CIP technique works in the following way: when (φi , φi+1 ) does not belong to the shaded region, we modify the values so that the tuple goes into the region. Although the modiﬁcation may create a more diffusive proﬁle than the original one, the result is oscillation-free and still has a third-order accuracy in space. ACKNOWLEDGMENTS We would like to thank the anonymous reviewers for their comments and suggestions. REFERENCES BRACKBILL, J. U., KOTHE, D. B., AND ZEMACH, C. 1992. A continuum method for modeling surface tension. J. Comp. Phys. 100, 335–354. CARLSON, M., MUCHA, R. J., AND TURK, G. 2004. Rigid ﬂuid: Animating the interplay between rigid bodies and ﬂuid. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2004) 23, 3, 377–384. CHEN, J. X. AND LOBO, N. D. V. 1995. Toward interactive-rate simulation of ﬂuids with moving obstacles using Navier–Stokes equations. Graph. Models Image Process. 57, 2, 107–116. ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2002) 21, 3, 736–744. FEDKIW, R., STAM, J., AND JENSEN, H. W. 2001. Visual simulation of smoke. Comput. Graph. (Proceedings of ACM SIGGRAPH 2001) 35, 15–22. FELDMAN, B. E., O’BRIEN, J. F., AND ARIKAN, O. 2003. Animating suspended particle explosions. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2003) 22, 3, 708–715. FOSTER, N. AND FEDKIW, R. 2001. Practical animation of liquids. Comput. Graph. (Proceedings of ACM SIGGRAPH 2001) 35, 23–30. FOSTER, N. AND METAXAS, D. 1996. Realistic animation of liquids. Graph. Models Image Process. 58, 5, 471–483. FOSTER, N. AND METAXAS, D. 1997a. Controlling ﬂuid animation. In Comput. Graph. Inter. 97, 178–188. FOSTER, N. AND METAXAS, D. 1997b. Modeling the motion of a hot, turbulent gas. Comput. Graph. (Proceedings of ACM SIGGRAPH ’97) 31, Annual Conference Series, 181–188. GOLUB, G. H. AND LOAN, C. F. V. 1996. Matrix Computations. The John Hopkins Univserity Press. HARLOW, F. H. AND WELCH, J. E. 1965. Numerical calculation of time-dependent viscous incompressible ﬂow of ﬂuid with free surface. Phys. Fluids 8, 12, 2182–2189. KASS, M. AND MILLER, G. 1990. Rapid, stable ﬂuid dynamics for computer graphics. Comput. Graph. (Proceedings of ACM SIGGRAPH ’90) 24, 4, 49–57. LORENSEN, W. E. AND CLINE, H. E. 1987. Marching cubes: A high resolution 3D surface construction algorithm. Comput. Graph. (Proceedings of ACM SIGGRAPH ’87) 21, 4, 163–169. LOSASSO, F., GIBOU, F., AND FEDKIW, R. 2004. Simulating water and smoke with an octree data structure. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2004) 23, 3, 457–462. ´ MCNAMARA, A., TREUILLE, A., POPOVIC, Z., AND STAM, J. 2004. Fluid control using the adjoint method. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2004) 23, 3, 449–456. MILLER, G. AND PEARCE, A. 1989. Globular dynamics: A connected particle system for animating viscous ﬂuids. Com- put.Graph. 13, 3, 305–309. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Stable But Nondissipative Water • 97 O’BRIEN, J. AND HODGINS, J. 1995. Dynamic simulation of splashing ﬂuids. In Proceedings of Computer Animation 95, 198–205. O’BRIEN, T. G. G. A. W. B. J. F. 2004. A method for animating viscoelastic ﬂuids. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2004) 23, 3, 463–468. OSHER, S. AND FEDKIW, R. 2002. The Level Set Method and Dynamic Implicit Surfaces. Springer-Verlag, New York. OSHER, S. AND SETHIAN, J. A. 1988. Fronts propagating with curvature dependent speed: Algorithms based in hamilton-jacobi formulations. J. Comp. Phys. 79, 12–49. PENG, D., MERRIMAN, B., OSHER, S., ZHAO, H., AND KANG, M. 1999. A pde-based fast local level set method. J. Comp. Phys. 155, 410–438. ˇ PREMOZE, S., TASDIZEN, T., BIGLER, J., LEFOHN, A., AND WHITAKER, R. T. 2003. Particle-based simulation of ﬂuids. In Eurographics 2003 Proceedings. Blackwell Publishers, 401–410. RASMUSSEN, N., NGUYEN, D. Q., GEIGER, W., AND FEDKIW, R. 2003. Smoke simulation for large scale phenomena. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2003) 22, 3, 703–707. SETHIAN, J. A. 1996. Fast marching level set methods for three dimensional photolithography development. SPIE 2726, 261–272. STAM, J. 1999. Stable ﬂuids. Comput. Graph. (Proceedings of ACM SIGGRAPH ’99) 33, Annual Conference Series, 121–128. STAM, J. AND FIUME, E. 1995. Depicting ﬁre and other gaseous phenomena using diffusion processes. Comput. Graph. (Pro- ceedings of ACM SIGGRAPH ’95) 29, Annual Conference Series, 129–136. ˆ ` STANIFORTH, A. AND COTE, J. 1991. Semi-lagrangian integration scheme for atmospheric model—a review. Mon. Weather Rev. 119, 12, 2206–2223. SUSSMAN, M., FATEMI, E., SMEREKA, P., AND OSHER, S. 1998. An improved level set method for incompressible two-phase ﬂows. Comput. Fluids 27, 663–680. SUSSMAN, M., SMEREKA, P., AND OSHER, S. 1994. A level set approach for computing solutions to incompressible two-phase ﬂow. J. Comp. Phys. 114, 146–159. TAKAHASHI, T., FUJII, H., KUNIMATSU, A., HIWADA, K., SAITO, T., TANAKA, K., AND UEKI, H. 2003. Realistic animation of ﬂuid with splash and foam. In Eurographics 2003 Proceedings. Blackwell Publishers, 391–400. TERZOPOULOS, D., PLATT, J., AND FLEISCHER, K. 1989. Heating and melting deformable models (from goop to glop). In Proceedings of Graphics Interface ’89. 219–226. ´ TREUILLE, A., MCNAMARA, A., POPOVIC, Z., AND STAM, J. 2003. Keyframe control of smoke simulations. ACM Trans. Graph. (Proceedings of ACM SIGGRAPH 2003) 22, 3, 716–723. ˝ TROTTENBERG, U., OOSTERLEE, C., AND SCHULLER, A. 2001. Multigrid. Academic Press. TSAI, Y.-H. R., CHENG, L.-T., OSHER, S., AND ZHAO, H.-K. 2003. Fast sweeping algorithms for a class of hamilton–jacobi equations. SIAM J. Numer. Anal. 41, 673–694. XIAO, F., YABE, T., AND ITO, T. 1996. Constructing oscillation preventing scheme for advection equation by rational function. Comp. Phys. Comm. 93, 1–12. YABE, T. AND AOKI, T. 1991. A universal solver for hyperbolic equations by cubic-polynomial interpolation i. one-dimensional solver. Comp. Phys. Comm. 66, 219–232. YABE, T., XIAO, F., AND UTSUMI, T. 2001. The constrained interpolation proﬁle method for multiphase analysis. J. Comp. Phys. 169, 556–593. Received July 2004; revised October 2004; accepted October 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter SEYOON TAK Samsung Advanced Institute of Technology and HYEONG-SEOK KO Seoul National University This article presents a novel constraint-based motion editing technique. On the basis of animator-speciﬁed kinematic and dynamic constraints, the method converts a given captured or animated motion to a physically plausible motion. In contrast to previous methods using spacetime optimization, we cast the motion editing problem as a constrained state estimation problem, based on the per-frame Kalman ﬁlter framework. The method works as a ﬁlter that sequentially scans the input motion to produce a stream of output motion frames at a stable interactive rate. Animators can tune several ﬁlter parameters to adjust to different motions, turn the constraints on or off based on their contributions to the ﬁnal result, or provide a rough sketch (kinematic hint) as an effective way of producing the desired motion. Experiments on various systems show that the technique processes the motions of a human with 54 degrees of freedom, at about 150 fps when only kinematic constraints are applied, and at about 10 fps when both kinematic and dynamic constraints are applied. Experiments on various types of motion show that the proposed method produces remarkably realistic animations. Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.2.9 [Artiﬁcial Intelligence]: Robotics—Kinematics and Dynamics General Terms: Algorithms, Experimentation Additional Key Words and Phrases: Motion retargeting, animation w/constraints, physically based animation 1. INTRODUCTION Motion editing is an active research problem in computer animation. Its function is to convert the motion of a source subject or character into a new motion of a target character while satisfying a given set of kinematic and dynamic constraints, as shown schematically in Figure 1. This type of motion editing, in which the animator speciﬁes what they want in the form of constraints, is called constraint-based motion editing, and has been studied by numerous researchers [Gleicher 1998; Lee and Shin 1999; Choi c and Ko 2000; Popovi´ and Witkin 1999; Shin et al. 2001]. Motion editing must compensate for both body differences and motion differences. When the an- thropometric scale of the target character differs from that of the source character, the original motion This work was supported by the Korea Ministry of Information and Communication and the Overhead Research Fund of Seoul National University. This work was also supported in part by the Automation and Systems Research Institute at Seoul National University, and the Brain Korea 21 Project. Authors’ addresses: S. Tak, Samsung Advanced Institute of Technology, San 14-1, Nongseo-Ri. Giheung-eup, Yongin-si, 449-712, Korea; email: tak@sait.samsung.co.kr, tak@graphics.snu.ac.kr; H.-S. Ko, Graphics and Media Lab, Seoul National University, San 56-1, Shillim-dong, Kwanak-ku, Seoul 151-741, Korea; email: ko@graphics.snu.ac.kr. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0098 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 98–117. A Physically-Based Motion Retargeting Filter • 99 Fig. 1. The constraint-based motion editing problem. should be kinematically retargeted to the new character. Characteristics that affect body dynamics, such as segment weights and joint strengths, should be accounted for if we are to generate a dynamically plausible motion of the target character. For example, the kicking motion of a professional soccer player cannot be reproduced by an unskilled person of equivalent anthropometric characteristics. Therefore, the motion editing algorithm should resolve both the kinematic and dynamic aspects of the source-to- target body differences. In addition, motion editing should provide means to create variations from the original motion. For example, starting from an original walking motion on a level surface, an animator may need to create longer steps or uphill steps. This article proposes a novel constraint-based motion editing technique that differs signiﬁcantly from existing methods in that it is a per-frame algorithm. The traditionally employed spacetime optimization methods can be used for interactive editing of short motion sequences and produce physically plausible motions. However, the processing times of these methods increase proportional (or at a higher rate) to the length of the motion sequence. In contrast, our algorithm functions as a ﬁlter of the original motion that processes the sequence of frames in a pipeline fashion. Thus, the animator can view the processed frames at a stable interactive rate as soon as the ﬁlter has started processing the motion, rather than having to wait for all frames to be processed as is the case in spacetime optimization methods. The per-frame approach has previously been taken by several researchers for the kinematic motion editing problem in which only kinematic constraints are imposed [Lee and Shin 1999; Choi and Ko 2000; Shin et al. 2001]. However, the problem of motion editing with both kinematic and dynamic con- straints poses two signiﬁcant challenges. (1) Dynamic constraints are highly nonlinear, compared to kinematic constraints. Such nonlinearity prohibits the constraint solver from reaching a convergent so- lution within a reasonable amount of time. (2) Dynamic constraints involve velocities and accelerations, whereas kinematic constraints involve only positions. It is this signiﬁcant distinction that makes the per-frame approach inherently difﬁcult for dynamic constraints; kinematic constraints can be indepen- dently formulated for individual frames, whereas the velocity and acceleration terms in the dynamic constraint equations call for knowledge of quantities from other frames. The interdependency between those terms makes the process look like a chain reaction, where imposing dynamic constraints at a single frame, calls for the participation of the positions and velocities of the entire motion sequence. We overcome the challenges outlined above by casting the motion editing problem as a constrained state estimation problem, based on the Kalman ﬁlter framework. We make the method function as a per-frame ﬁlter by incorporating the motion parameters and the desired constraints into a specialized Kalman ﬁlter formulation. To handle the nonlinearity of complex constraints more accurately, we em- ploy the unscented Kalman ﬁlter, which is reported to be superior in its accuracy to the other variants of the Kalman ﬁlter or the Jacobian-based approximation [Wan and van der Merwe 2001]. To apply Kalman ﬁltering to the problem of motion editing, however, we must treat the position, ve- locity, and acceleration as independent degrees of freedom (DOFs). Under this treatment, the resulting motion parameter values may violate the relationship that exists between the position, velocity, and acceleration values describing a particular motion. We resolve this problem by processing the Kalman ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 100 • S. Tak and H.-S. Ko ﬁlter output with a least-squares curve ﬁtting technique. We refer to this processing as the least-squares ﬁlter. Unlike the Kalman ﬁlter that processes each frame independently, the least-squares ﬁlter requires data over a certain range of frames for curve ﬁtting. Therefore, the proposed motion editing ﬁlter is basically a concatenation of the Kalman ﬁlter and the least-squares ﬁlter. It functions as an enhancement operator; the ﬁrst application of the ﬁlter may not produce a completely convergent solution, but repeated applications reﬁne the result until a reasonable solution is reached. Such incremental reﬁnement can be valuable in practice, because most animators prefer to see a rough outline of the motion interactively before carrying out the longer calculation necessary to obtain the ﬁnal motion. Furthermore, they can provide a rough sketch of the desired motion before the ﬁltering begins, which is an effective way of reﬂecting their intuitive ideas as well as overcoming the locality nature of the proposed algorithm. Our motion editing technique is well suited for interactive applications; we can add or remove some or all of the kinematic and dynamic constraints depending on whether they signiﬁcantly affect the type of motion being animated. When only kinematic constraints are imposed, one application of the ﬁlter produces a convergent solution and the motion editing algorithm runs in real-time. As dynamic constraints are added, the ﬁlter must be applied several times to obtain convergent results, but the editing process still runs at an interactive speed. 2. RELATED WORK The establishment of motion capture as a commonplace technique has heightened interest in meth- ods for modifying or retargeting a captured motion to different characters. Motion editing/synthesizing methods can be classiﬁed into four groups: (1) methods that involve only kinematic constraints, (2) meth- ods that involve both kinematic and dynamic constraints, (3) the spacetime constraints methods that do not exploit the captured motion, and (4) motion generating techniques based on dynamic simulation. Gleicher [1997, 1998] formulated the kinematic version of the motion editing problem as a space- time optimization over the entire motion. Lee and Shin [1999] decomposed the problem into per-frame inverse kinematics, followed by curve ﬁtting for motion smoothness. Choi and Ko [2000] developed a retargeting algorithm that works online, which is based on the per-frame inverse rate control, but avoids discontinuities by imposing motion similarity as a secondary task. Shin et al. [2001] proposed a different online retargeting algorithm based on the dynamic importance of the end-effectors. A good survey of the constraint-based motion editing methods is provided by Gleicher [2001]. The way our motion editing technique works most resembles the approach of Lee and Shin [1999], in that both tech- niques are per-frame methods with a post-ﬁlter operation. However, in our method, the post-ﬁlter is applied only to recently processed frames and, as a consequence, the whole process works as a per-frame ﬁlter. It is interesting to note that the methods based on kinematic constraints quite effectively generate useful variations of the original motion. However, when the dynamic context is signiﬁcantly different in the source and target motions, the motion generated by kinematic editing is unacceptable. Pollard et al. [2000] proposed a force-scaling technique for fast motion transformation. Tak et al. [2000] introduced a spacetime optimization technique for correcting a given motion into a dynamically balanced one. c Popovi´ and Witkin [1999] addressed the physically-based motion editing problem using spacetime optimization. Because optimization, subject to dynamic constraints (i.e. Newton’s law), can take a prohibitive amount of computation, they introduced a character simpliﬁcation technique to make the problem tractable. The most signiﬁcant distinction between our method and spacetime optimization methods is that, instead of looking at the entire duration of a motion, our technique works on a per- frame basis. As a result, the outcome of each frame is available at a uniform interactive rate, since it requires a deterministic amount of computation. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 101 An interesting work that solves the retargeting problem in the robotics context is Yamane and Nakamura [2000, 2003], which is similar to our approach in that it transforms a given motion into a physically consistent one on a per-frame basis. Their dynamics ﬁlter ﬁrst computes the desired ac- celerations by feedback controllers referring to the reference motion, and then modiﬁes the result by projection to the null-space of the equation of motion to make it a dynamically consistent one. Our approach differs from theirs in that we use an iterative algorithm that consists of two consecutive ﬁl- ters and we process position, velocity, and acceleration simultaneously rather than acceleration alone, thus increasing the applicability. For the same reason, it is difﬁcult to control kinematic constraints in their method, since they deal with only accelerations in the ﬁltering process and then integrate them to obtain the ﬁnal positional data. Also, as they pointed out in the paper, the sensitiveness to the reference motion which causes ﬁlter divergence and the difﬁculty of tuning parameters (feedback gains, pseu- doinverse weights) are unsolved problems. Another recent work similar to ours is Shin et al. [2003], which improves physical plausibility of edited motions by enforcing ZMP constraints and momentum constraints. While our method is an iterative ﬁltering process over all DOFs, they sequentially adjusted user-speciﬁed individual DOF using approximated closed-form dynamic equations for efﬁciency. Many of the kinematic and physically-based motion editing techniques mentioned above derive from the spacetime constraints method proposed by Witkin and Kass [1988]. However, when this original method is applied to a complex articulated ﬁgure, the dimensional explosion and severe nonlinearity of the problem usually leads to impractical computational loads or lack of convergence. Several groups [Cohen 1992; Liu et al. 1994; Rose et al. 1996] have attempted to improve the classical spacetime constraints algorithm and its applicability. In a recent work that synthesizes a dynamic motion from c a rough sketch, Liu and Popovi´ [2002] circumvented the problems by approximating the Newtonian dynamics with linear and angular momentum patterns during the motion. Another optimization-based motion synthesis algorithm was proposed by Fang and Pollard [2003], which showed a linear-time performance. Our constraint solver is built on the Kalman ﬁlter framework. There have been several previous attempts to treat constraints using the Kalman ﬁlter. Maybeck[1979] introduced the notion that the Kalman ﬁlter can be used to solve linear constraints by regarding them as perfect measurements, while other researchers [Geeter et al. 1997; Simon and Chia 2002] built constraint solvers based on the extended Kalman ﬁlter to solve nonlinear constraints. However, as many researchers have pointed out [Julier and Uhlmann 1997; Wan and van der Merwe 2000], the extended Kalman ﬁlter can produce inaccurate results at nonlinearities. We used the unscented Kalman ﬁlter to better handle the severe nonlinearities in the dynamic constraints. A good introduction to the Kalman ﬁlter can be found in Welch and Bishop [2001]. The preliminary version of this work was presented in Tak et al. [2002]. In the current article, we give a signiﬁcantly improved exposition of the technique as well as extend the technique. The formulation is now more rigorous, and the method is compared with other methods so that its limitations and strengths are highlighted. By addressing momentum conservation in the ﬂight phases, and the redundancy problem during the double support phases, we widen the applicability of the algorithm. New experiments that show the extended features are reported. 3. OVERVIEW Figure 2 shows an outline of the overall structure of our motion editing process. Animators ﬁrst provide the input motion of the source character along with a set of kinematic and dynamic constraints. Then a Kalman ﬁlter that is tailored to the motion editing problem produces the motion parameter values, which are post-processed by the least-squares curve ﬁtting module. We apply the Kalman ﬁlter and least-squares ﬁlter repeatedly until it converges to an acceptable result. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 102 • S. Tak and H.-S. Ko Fig. 2. Overall structure of the motion editing process. Several important issues must be addressed in the implementation of the process outlined above: —What kinds of constraints are needed to generate a desired motion? How should those constraints be formulated? These issues are addressed in Section 4. —How is the Kalman ﬁlter applied to our motion editing problem? The details are presented in Section 5. —The Kalman ﬁlter processes position, velocity, and acceleration as independent variables, which can corrupt the imperative relationship among those variables. How is this rectiﬁed by the least-squares ﬁlter? This is explained in Section 6. 4. FORMULATING CONSTRAINTS The collection of all the kinematic and dynamic constraints on the motion of a character with DOFs can be summarized into the form H(q, q, q) = Z, ˙ ¨ (1) where q = q(t) is the -dimensional vector that completely describes the kinematic conﬁguration of the character at time t. This vector contains a mixture of positional and orientational quantities, but when it is clear from the context, we call the entire vector simply the position. ˜ The vector valued function H : R3 → R that maps a 3 -dimensional vector to a ˜ = ˜ K + ˜ B + ˜ T + ˜ M dimensional vector can be written as ˙ ¨ H K (q, q, q) H (q, q, q) ˙ ¨ B H(q, q, q) = ˙ ¨ . (2) HT (q, q, q) ˙ ¨ ˙ ¨ H M (q, q, q) ˜ K , ˜ B , ˜ T , and ˜ M are the dimensions of the kinematic, balance, torque limit, and momentum constraints, respectively. Therefore we can view the function H as a block matrix of the component constraint functions H K , H B , HT , and H M , as shown in the right-hand side of Equation (2). The values of ˜ K , ˜ B , ˜ T , and ˜ M depend on how each type of constraint participates in the current editing process. For example, when only one end-effector position constraint is imposed, ˜ K = 3. If an additional orientational constraint is imposed, then ˜ K becomes 6. Z is a ˜ -dimensional vector that does not contain any variables, and can be represented as the block matrix Z = [ZT ZT ZT ZT ]T . K B T M The goals of this section are (1) to ﬁnd the formulations for each of the component constraint functions H K , H B , HT , and H M , and (2) to ﬁnd the values for the component constraint goals Z K , Z B , ZT , and Z M , which correspond to the constraints speciﬁed by the animators. The constraint solver this article proposes requires only the formulation of the component functions, but does not require their derivatives or inverse functions. Constraints are resolved by the black box composed of the Kalman ﬁlter and least-squares ﬁlter. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 103 Fig. 3. The zero moment point (ZMP) and its trajectory correction. 4.1 Kinematic Constraints Kinematic constraints specify the end-effectors to be positioned at the desired locations e by hfk (q) = e, (3) where the function hfk is the forward kinematic equations for the end-effectors under consideration. Therefore, H K is simply formulated as H K (q, q, q) = hfk (q), ˙ ¨ (4) and the constraint goal is given by Z K = e. 4.2 Balance Constraints Because humans are two-legged creatures, balancing is an important facet of their motion that must be adequately captured if an animation is to appear realistic. Dynamic balance is closely related to the zero moment point (ZMP), that is, the point at which the net moment of the inertial forces and gravitational c forces of all the body segments is zero [Vukobratovi´ et al. 1990]. The ZMP, at a particular instant, is a function of the character motion, and can be obtained by solving the following equation for Pzmp (ri − Pzmp ) × {mi (ri − g)} = 0, ¨ (5) i where mi and ri are the mass and center of mass of the ith segment (Figure 3(a)), respectively, and g is the acceleration of gravity. In our work, we regard an animated motion to be dynamically balanced at time t if the projection P2D zmp of the ZMP is located inside the supporting area (the convex hull containing all the ground contacts). Taking ri = (xi , y i , z i ), g = (0, − g (≈ 9.8), 0), and setting the y component of Pzmp to zero, Equation (5) produces the following analytical solution for P2D : zmp i mi ( y i + g )xi − i mi xi y i ¨ ¨ i mi ( y i + g ) = = hzmp (q, q, q). ¨ P2D zmp ˙ ¨ (6) i mi ( y i + g )z i − i mi z i y i ¨ ¨ i mi ( y i + g ) ¨ ¨ ˙ ¨ Since ri and ri can be expressed by q, q, and q, we can view the above result as giving the formula for the function hzmp (q, q, q).1 Some portions of P2D , obtained by evaluating the above formula, may lie ˙ ¨ zmp 1 Note i mi xi i mi z i that for a static posture, the ZMP in Equation (6) reduces to the center of gravity ( mi , mi ), the projection point of i i the center of mass of the whole body, on the ground. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 104 • S. Tak and H.-S. Ko outside the supporting area as shown in Figure 3(c). In our work, balancing is achieved by modifying ˜ the motion (i.e. q, q, and q) such that hzmp (q, q, q) follows a new trajectory P2D given by ˙ ¨ ˙ ¨ zmp P2D (t) zmp : if P2D (t) ∈ S zmp ˜ P2D (t) = , (7) zmp projS (P2D (t)) zmp : otherwise where S is the supporting area and projS is the operator that projects the given point into the area S as shown in Figure 3(c). Finally, the balance constraint is formulated as H B (q, q, q) = hzmp (q, q, q), ˙ ¨ ˙ ¨ (8) ˜ and Z B = P2D . zmp It should be noted that the notion of balance in this article is somewhat subtle, and different from the usual meaning—not falling. A number of researchers in robotics and graphics have proposed balancing techniques. One approach, based on the inverted pendulum model, ensures balanced motion by main- taining the position and velocity of the center of gravity (COG) within a stable region [Faloutsos et al. 2001; Zordan and Hodgins 2002]. The same goal has also been achieved by tracking the ZMP trajectory as an index of stability [Dasgupta and Nakamura 1999; Oshita and Makinouchi 2001; Sugihara et al. 2002]. In these previous studies, balancing was achieved by controlling the joint torques to prevent the characters from falling. If our objective is to analyze the moment of a legged ﬁgure with respect to the ground, we can equivalently represent the ﬁgure as an inverted pendulum (Figure 3(b)). In this conceptualization, the location of the pendulum base corresponds to the ZMP. In every real-world motion with a ground contact, even in falling motion, the ZMP always lies within the supporting area2 [Vukobratovi´ et al.c 1990]. Physically, therefore, the ZMP is not related to the act of balancing. Rather, the ZMP concept is related to physical validness. Physical validness has no meaning in real-world motions, but has a signiﬁcant meaning in motion editing. We can judge that a computer-generated motion is out of balance if the motion is physically invalid according to the ZMP criterion. Our balance constraint formulation, based on the ZMP, provides an effective way to modify a given motion to achieve dynamic balance. 4.3 Torque Limit Constraints The torque a human can exert at each joint is limited. However, computer-generated human motion can violate this principle, potentially giving rise to motions that look physically unrealistic or uncomfortable [Lee et al. 1990; Ko and Badler 1996; Komura et al. 1999]. To address this issue, we allow animators to specify torque limit constraints. The motion editing algorithm must, therefore, modify the given motion such that the joint torques of the new motion are within the animator-speciﬁed limits. We need to ﬁnd ˙ ¨ the formulation of the function HT (q, q, q) and the goal ZT that achieves the modiﬁcation. First, we must calculate the torque proﬁle of the original motion to see if it contains any torque limit violations. We let τ (t) = [τ1 (t) · · · τ −6 (t)]T be the torque vector at time t, which is the collection of the scalar torques corresponding to the ( −6) joint DOFs. The inverse dynamics problem τ (t) = htrq (q, q, q) ˙ ¨ has been extensively studied in the robotics literature [Craig 1989; Shabana 1994]. Here we use the O( ) ˙ ¨ Newton-Euler method, which does not give an explicit formula for htrq (q, q, q), but instead recursively computes the torque values. For the closed-loop formed during the double support phases, we resort to the approximation method proposed by Ko and Badler [1996]. 2 During ﬂight phases, however, the evaluation of P2D in Equation (6) results in dividing by zero, thus the ZMP is not deﬁned. zmp This is consistent with our intuition that balance is a meaningful concept only when the body makes contact with the ground. In ﬂight phases, therefore, we deactivate the balance constraint. Detecting the initiation of the ﬁght phase is done by examining the positions of the feet. The simulation results are insensitive to the noise at the phase boundaries. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 105 When the torque τ j (t), computed as described above, exceeds the speciﬁed limit τ max , we reduce it to j the given limit. Thus, the corrected torque proﬁle τ j of joint j is given by ˜ τ j (t) : if τ j (t) ≤ τ max j τ j (t) = ˜ . (9) τ max j : otherwise In our implementation, the torque limit τ max was given by animators experimentally, but can also be j determined using joint strength data from the biomechanics [Winter 1990]. Finally, the torque constraints are formulated as HT (q, q, q) = htrq (q, q, q), ˙ ¨ ˙ ¨ (10) with ZT = τ . ˜ 4.4 Momentum Constraints The momentum constraints are derived from Newton’s second law, which states that the rates of change of the linear and angular momenta are equal to the sums of the resultant forces and moments acting on the ﬁgure, respectively. In the supporting phases, the interaction between the feet and the ground leads to quite complex patterns in the character’s momentum behavior. Therefore, we do not impose momentum constraints in the supporting phases. In ﬂight phases, however, gravity is the only external force. Thus the linear momentum and the net angular momentum of the entire body must satisfy ˙ ˙ P = mall c = mall g and L = i mi (ri − c) × (¨ i − c) = 0 (point mass model assumed), where mall is the ¨ r ¨ c total mass and c is the center of mass of the entire ﬁgure [Liu and Popovi´ 2002]. Hence, we formulate the momentum constraints during ﬂight phases as c−g ¨ H M (q, q, q) = ˙ ¨ , (11) i mi (ri − c) × (¨ i − c) r ¨ with ZT = 0 in this case. 5. KALMAN FILTER-BASED MOTION EDITING Once the constraints are formulated as shown in Equation (1), the task of modifying the original motion to meet the constraints is accomplished using Kalman ﬁltering. The important decisions in our tailoring of Kalman ﬁltering to motion editing problem are (1) the choices for the process and measurement model, and (2) using the unscented Kalman ﬁlter, rather than the extended Kalman ﬁlter. We begin this section with a brief explanation of how Kalman ﬁltering works. Then we show how the motion editing problem is formulated in the framework of Kalman ﬁltering. 5.1 How Kalman Filtering Works Kalman ﬁltering is the problem of sequentially estimating the states of a system from a set of mea- surement data available online [Maybeck 1979; Welch and Bishop 2001]. The behavior of a Kalman ﬁlter is largely determined by deﬁning the process model xk+1 = f(xk , vk ) and the measurement model zk = h(xk , nk ), where xk represents the state of the system at time tk , zk is the observed measurement, and vk and nk are the process and measurement noise. For example, if we are to model the freefall of a stone that is being recorded by a digital camera, xk is the random variable that represents the 3D position of the stone, Pxk (which will appear in the subsequent descriptions) is the covariance of xk , and zk represents the 2D position of the stone recorded in the photograph. We deﬁne the process model f so that it predicts the next state from the value of the current state (using knowledge of Newtonian mechanics). The uncertainties due to factors such as air resistance and wind are modeled by vk , which ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 106 • S. Tak and H.-S. Ko is assumed to follow a Gaussian distribution. We deﬁne the measurement model h such that it describes in principle the relationship between the state xk and the measurement zk . nk models the measurement errors, and is also assumed to follow a Gaussian distribution. The Kalman ﬁlter recursively estimates the mean and covariance of the state using the following predictor-corrector algorithm. Predict (time update) Correct (measurement update) x− = f(xk−1 , 0) ˆk ˆ xk = x− + Kk (zk − h(x− , 0)) ˆ ˆk ˆk The time update predicts the a priori estimate x− , and the measurement update corrects x− by referring ˆk ˆk ˆ to the new measurement zk to obtain the a posteriori estimate xk . The Kalman gain Kk is determined from the process and measurement models according to the procedure described in Section 5.3. 5.2 Our Formulation of Motion Editing Problem When formulating the constraint-based motion editing problem using a Kalman ﬁlter, the most sub- stantial step is the determination of the process and measurement models. We deﬁne the process model as x− = [qk qk qk ], where qk = q(tk ) is the value of q at the discrete time step tk . In this case, ˆk ˙ ¨ ˆ function f does not depend on the previous state xk−1 , but comes directly from the original motion. We use Z in Equation (1) as the measurements, and denote the value of Z at time tk as Zk . We deﬁne the ˙ ¨ measurement model as H(q, q, q) of Equation (1). The rationale behind the deﬁnition outlined above is that the original motion contains excellent kinematic and dynamic motion quality, so by starting from this motion we intend to preserve its quality in the ﬁnal motion. 5.3 Motion Editing Algorithm Based on the UKF Since the constraint functions in Equation (2) are highly nonlinear, the original version of the Kalman ﬁlter, which was designed for linear systems, does not properly handle the motion editing problem considered here. The extended Kalman ﬁlter (EKF) was developed to handle nonlinearity through a Jacobian-based approximation, but recently the unscented Kalman ﬁlter (UKF) was proposed to better handle severe nonlinearity. The UKF was ﬁrst proposed by Julier et al. [1997], and further developed by others [Wan and van der Merwe 2000; van der Merwe and Wan 2001]. The basic difference between the EKF and the UKF lies in the way they handle nonlinear functions. The computational core of the Kalman ﬁlter consists of the evaluation of the posterior mean and covariance when a distribution with the prior mean and covariance goes through the nonlinear functions of the process and the measurement models. As shown in Figure 4(b), the EKF approximates the posterior mean by evaluating the nonlinear function at the prior mean, and approximates the posterior covariance as the product of the Jacobian and the prior covariance. However, it has been reported that this method can lead to inaccuracy and occasional divergence of the ﬁlter [Julier and Uhlmann 1997; Wan and van der Merwe 2000]. The UKF addresses this problem using a deterministic sampling approach that approximates the posterior mean and covariance from the transformed results of a ﬁxed number of samples as shown in Figure 4(c). Given a nonlinear function h(x) = z deﬁned for n-dimensional state vectors x, the UKF ﬁrst chooses 2n+ 1 sample points Xi that convey the prior state distribution (mean and covariance of x), after which it evaluates the nonlinear function h at these points, producing the transformed sample points Zi . The UKF then approximates the posterior mean and covariance by calculating the weighted mean and covariance (Z− and Pzz in Step 4 of the procedure summarized below) of the transformed ˆ k ˆ sample points, which is accurate to the 2nd order for any nonlinearity as opposed to the 1st order in the EKF. The superior performance of the UKF over the EKF is thoroughly discussed in Wan and van der Merwe [2001], along with the quantitative analysis on a broad class of nonlinear estimation problems. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 107 Fig. 4. Comparison of mean and covariance approximations after the nonlinear transformation h is applied (excerpted from Wan and van der Merwe [2001]): solid, dotted, dashed lines in the transformed results correspond to the ideal-, EKF-, and UKF-transformed mean and covariance, respectively. In (b), J is the Jacobian matrix of h. Now, we summarize the steps involved in the proposed UKF-based constraint solver. The inputs fed ˙ ¨ into the solver at each frame k are the source motion [qk qk qk ] and the constraint goal Zk . For each kth frame, (1) Using the process model deﬁnition discussed in Section 5.2, the prediction step is straightforward3 : x− = [qk qk qk ] ˆk ˙ ¨ Vx.pos · · · 0 , (12) P− = ˆ k Vx.vel 0 ··· Vx.acc where Vx.∗ are the process noise covariances. Since our process model is not deﬁned in terms of the previous state, P− does not depend on Pk−1 . Therefore, we simply use the constant matrix shown ˆ k ˆ above for every frame. (2) We construct (2n + 1) sample points from x− and P− by ˆk ˆ k X 0 = x− ˆk W0 = κ/(n + κ) i=0 Xi = x− ˆk ˆ − )i Wi = 1/{2(n + κ)} i = 1, . . . , n + ( (n + κ)Pk (13) Xi = x− ˆk − ( (n + κ)P− )i ˆ k Wi = 1/{2(n + κ)} i = n + 1, . . . , 2n, √ where κ is a scaling parameter, ( ·)i signiﬁes the ith row or column of the matrix square root, and 2n Wi is the weight associated with the ith sample point, chosen such that i=0 Wi = 1. Our choice for κ is based on Wan and van der Merwe [2001]. 3 The state vector and covariance matrix contain only positional components when only kinematic constraints are involved. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 108 • S. Tak and H.-S. Ko (3) We transform the sample points in Step 2 through the measurement model deﬁned in Section 5.2 to obtain Zi = H(Xi ) i = 0, . . . , 2n. (14) (4) The predicted measurement Z− is given by the weighted sum of the transformed sample points, ˆ k and the innovation covariance and the cross-covariance are computed as Z− = ˆ k 2n i=0 Wi Zi ˆ zz = P 2n i=0 Wi (Zi − Z− )(Zi − Z− )T + Nz ˆ k ˆ k (15) ˆ Pxz = 2n i=0 Wi (Xi − x− )(Zi − Z− )T , ˆk ˆ k where Nz is the measurement noise covariance. (5) The Kalman gain and the ﬁnal state update are given by Kk = Pxz P−1 ˆ ˆ zz (16) xk = x− + Kk (Zk − Z− ). ˆ ˆk ˆ k The behavior of the ﬁlter can be controlled by adjusting the following parameters: —The process noise covariance Vx.∗ , which is a diagonal matrix. Each diagonal element of this matrix represents the degree of uncertainty of the corresponding DOF. The values of these diagonal elements are related to the degree of displacement that occurs in the ﬁltering process. A larger value of a particular element results in a bigger displacement of the corresponding DOF. —The measurement noise covariance Nz , which is also a diagonal matrix. Each diagonal element is related to the rigidity of one of the constraints on the motion. Typically, these elements are set to zero, treating the constraints as perfect measurements. Using nonzero diagonal elements is useful when two constraints conﬂict with each other, because the constraint with the larger covariance (soft constraint) yields to the one with the smaller covariance (hard constraint). 6. LEAST-SQUARES FILTER ˙ ¨ The Kalman ﬁlter described in the previous section handles q, q, and q as independent variables. As a result, the ﬁltered result may not satisfy the relationship between the position, velocity, and acceleration. The role of the least-squares ﬁlter is to rectify any corruption of this relationship that occurred during Kalman ﬁltering.4 Because the least-squares ﬁlter is basically a curve ﬁtting procedure, it has a tendency to smooth out the jerkiness that may be introduced by the per-frame handling of the motion data. 4 If only kinematic constraints are involved, the least-squares ﬁlter need not be applied. However, even in this case, the animator may choose to apply the least-squares ﬁlter to eliminate potential artifacts arising from the use of a per-frame approach. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 109 ˘ To ﬁnd the control points (in 1D) that ﬁt the complete proﬁle q of a particular DOF, we formulate a B-spline curve that conforms to B c = q, ˘ B1 (t1 ) · · · BM (t1 ) q1 . . . . . . . . . q B1 (tN ) · · · BM (tN ) N ˙ B1 (t1 ) · · · B M (t1 ) ˙ c1 q1 ˙ , (17) . . . B= . . , c = . , q = . . . . ˘ . . ˙ B1 (tN ) · · · BM (tN ) ˙ cM qN ˙ ¨ ··· B ¨ M (t1 ) q1 B1 (t1 ) ¨ . . . . . . . . . ¨ 1 (tN ) B ··· B ¨ M (tN ) ¨ qN where Bi (t), Bi (t), and Bi (t)(i = 1, . . . , M ) are the B-spline basis functions and their ﬁrst and second ˙ ¨ derivatives, respectively, and the scalar values ci are the control points. For a 3 DOF joint, we need to formulate four such equations corresponding to the four components of the quaternion representation, and renormalize the resulting values to maintain the unity of the quaternions. The spacing of the control points along the time axis can be determined by the user, possibly by considering the distribution of frequency components in the motion. In our experiments, we place a control point at every fourth frame in common cases, but when the smoothing artifact in high-frequency motions is visible, we place a control point at every second frame. The above problem is that of an over-constrained linear system. Therefore, we ﬁnd the control points ˘ c that best approximate the given data q by minimizing the weighted objective function (B c − q)T W(B c − q), ˘ ˘ (18) where W is a 3N × 3N diagonal matrix that controls the relative inﬂuences of q, q, and q. The classical ˙ ¨ linear algebra solution to this problem is c = (BT W B)−1 W q = B# q, ˘ ˘ (19) where B# is the weighted pseudo-inverse matrix of B, and is easily computed because BT W B is a well-conditioned matrix. Finally, we obtain the least-squares ﬁltered motion by evaluating the B-spline curves at the discrete time steps. Note that B, and accordingly B# , are band-diagonal sparse matrices. This means that each control ˘ point in Equation (19) is determined by the section in q that corresponds to the nonzero entries of B# . This locality suggests that the control points should be computed from the motion data within the window shown in Figure 5, which moves along the time axis as the Kalman ﬁlter advances. Suppose c1 , c2 , . . . , ci−1 have already been computed. When the center of the window of width W is positioned ˘ over the control point ci , we determine ci using the portion of q that exists within the window. This scheme is equivalent to the global ﬁtting described previously, if the window size is large enough. The plot of accuracy versus window size depends on the order and knot-interval of the B-spline curve. In our experiments, W = 64 gave almost the same accuracy as the global ﬁtting. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 110 • S. Tak and H.-S. Ko Fig. 5. Sweeping of the Kalman ﬁlter (the thick vertical line) and the least-squares ﬁlter (the rectangle centered at the ﬁgure). 7. DISCUSSION In this section, we analyze our algorithm in comparison with previous approaches and determine its limitations as well as its strengths. This will help us to better understand what type of motion this technique is best suited for. Statistical inference vs. derivative-based optimization. One of the main differences between our algo- rithm and the previous methods is the statistical approach we take in dealing with the nonlinearity of constraints. While derivative-based optimization uses Jacobians or Hessians of the constraints and ob- jective function, our statistical method approximates the nonlinear function by evaluating the function values on a set of samples, which can be viewed in some sense as statistical derivatives. Even though the derivative-based approach theoretically has the same expressive power, in practice, we found that the statistical approach provides more ﬂexibility to address stiff problems. Local (per-frame) vs. global method. Another key difference, which might be the most signiﬁcant reason our method works fast, is that the method splits the given motion editing problem into per-frame pieces with post ﬁltering, instead of casting it as one large optimization problem as in the previous methods. A limitation of our per-frame approach is that it can not make any dynamic anticipation on a large timescale. Therefore, the algorithm is not suited for editing a motion that requires global modiﬁcation on the whole duration of the motion. For example, to generate the motion of punching an object with a dynamic constraint that the ﬁnal speed of the ﬁst should be considerably larger than that of the original motion, a large pull-back action is expected long before the arm makes the hit. However, our algorithm would attempt to modify the motion only in the neighborhood of the ﬁnal frame. In fact, other optimization techniques that use local derivatives may have the potential to suffer this locality problem. In this work, the problem can be circumvented by providing a rough sketch of the desired motion as a new source motion. We call it a kinematic hint. Kinematic hints can be an effective means to produce the desired motion when our method is interactively used by an animator who has an intuitive idea of the form of the ﬁnal motion. Filter tuning. The ﬁlter parameters (Vx.∗ and Nz in the UKF, and the weight parameter W in the least- squares ﬁlter) signiﬁcantly affect the ﬁltering performance. Therefore, unless the parameter values are carefully chosen through consideration of the type of constraints and target motion, the ﬁlter may not work properly. For example, too large Vx.∗ may lead to ﬁlter divergence, and too small Vx.∗ may result in slow convergence. Choosing appropriate ﬁlter parameters required trial-and-error procedures at ﬁrst, but soon we found that the following guidelines work well in most examples: Vx.pos is typically chosen from the range 10−9 ∼10−10 , and we use Vx.vel = αVx.pos and Vx.acc = α 2 Vx.pos , where α =10∼100. The values of Nz are set to zero in most cases. One exception to this rule is when the current constraint goal is too far, in which case we initially set Nz to a small, but nonzero value (e.g. 10−10 ), and then adaptively decrease them to zero. Finally, we found that the weight parameter, W = diag[I, βI, β 2 I], best scales ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 111 the relative inﬂuences of q, q, and q when β ≈ 0.1. In fact, all the examples presented in this article ˙ ¨ are made based on this guideline. Convergence problem. Because of the nonlinear iterative nature of the problem, convergence is an important issue. It is virtually impossible to characterize the analytical condition under which the repeated applications of the Kalman ﬁlter and least-squares ﬁlter converge. According to our experi- ments, when only kinematic constraints exist, the technique produces a convergent solution with one or, at most, two applications of the ﬁlter. On the other hand, when we edit a motion involving dynamic constraints, the effects of the Kalman ﬁlter on position, velocity, and acceleration are, in part, cancelled by the following least-squares ﬁlter. Especially when the target motion is highly dynamic (e.g. such mo- c tions as in Liu and Popovi´ [2002] and Fang and Pollard [2003] in which the velocity and acceleration can have severely undulating patterns), the cancellation effect of the two ﬁlters may become dominant, and our method may not able to ﬁnd a convergent solution. However, it is worth noting that, in most cases, we could ﬁnd the ﬁlter parameters and/or kinematic hints such that 3-5 ﬁlter applications attain a reasonable level of dynamic quality. Dynamic quality reduction in the ﬁnal ﬁlter application. While the resulting motion from the re- peated ﬁlter applications possesses desired dynamic quality, the application of the last least-squares ﬁlter can ruin the kinematic constraints to a noticeable degree. In such a case, we run the Kalman ﬁlter with the dynamic constraints turned off, which produced kinematically accurate motion, poten- tially destroying the previously attained dynamic quality. According to the experiments, however, the dynamic quality reduction was not signiﬁcant. Per-frame constraints generation. The global method requires the animators to set only high-level goals (such as a jumping height); the rest is generated by the algorithm. On the other hand, our per- frame algorithm requires the animators to supply kinematic and dynamic constraints for each frame (in the form of trajectory, etc.). For example, to make the character kick a certain object, the animator has to construct the desired trajectory that passes the object by modifying the originally given foot trajectory. It is an extra burden for the animator compared to the global method. On the other hand, it can be viewed as a means to control the details of the (kicking) motion. 8. RESULTS Our motion editing system was implemented as a Maya plug-in on a PC with a Pentium-4 2.53 GHz processor and a GeForce 4 graphics board. All the motion sequences used were captured at 30 Hz. The human model had a total of 54 DOFs, including 6 DOFs for the root located at the pelvis. The root orientation and joint angles were all represented by quaternions. Below, we refer to the ﬁltering with only kinematic constraints as kinematic ﬁltering and denote i consecutive applications of the ﬁlter by i(K), and we refer to the ﬁltering with both kinematic and dynamic constraints as dynamic ﬁltering and denote j consecutive applications of the ﬁlter by j (D). In the following experiments, we used the full set of DOFs in the kinematic ﬁltering, but omitted several less inﬂuential joints (for example, wrists, elbow, ankles, and neck) in the dynamic ﬁltering (total 27-DOFs) to improve the performance. This section reports the results of ﬁve experiments. The ﬁlter applications used and the resulting frame rates in these experiments are summarized in Table I. The decision on the number and types of ﬁlter applications are up to the users. They can apply the ﬁltering repeatedly until they ﬁnd the result satisfactory, and also they can start by imposing kinematic hint for better performance. The animations of the results can be found at http://graphics.snu.ac.kr/∼tak/filter.htm. Dancing (on-line kinematic retargeting). Referring to Animation 1A (Figure 6), we retargeted the dancing motion of the middle character to the characters on the left (shorter limbs and longer torso) and ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 112 • S. Tak and H.-S. Ko Table I. motion # frames # frames/knot # ﬁltering operation average frame rate dancing 660 1(K) 150.0 wide steps 260 4 2(D) + 1(K) 15.4 golf swing 216 2 1(K) + 3(D) + 1(K) 15.2 limbo walk 260 4 1(K) + 2(D) + 1(K) 13.7 jump kick 152 2 3(D) + 1(K) 11.3 Summary of the experimental conditions and resulting frame rates. Frame rates were estimated from pure computation time excluding the visualization time Fig. 6. Dancing: A dancing motion of the character in the middle is retargeted to the other characters online. right (longer limbs and shorter torso). The foot trajectories of the source character were used without scaling as the kinematic constraints of the target characters. The height differences were accounted for by raising or lowering the root of the source character by a constant amount. In this experiment, a single application of the Kalman ﬁlter (without the least-squares ﬁlter) generated the target motions without any artifacts. Animation 1B is the real-time screen capture during the production of Animation 1A. In other examples, which are not included in the article, if the end-effector goal was too far or followed a severely nonsmooth path, Kalman ﬁltering alone could produce temporal artifacts. In such cases, application of the least-squares ﬁlter solved the problem, although it created a delay of about one second. In numerous experiments, the kinematic ﬁltering worked stably and robustly. Wide Steps. In this experiment, we considered the problem of converting the normal walking steps in Animation 2A into the sequence of wide steps taken to go over a long narrow puddle. The kinematic ﬁltering produced the physically implausible result shown in Animation 2B (Figure 7(a)). To make the motion physically plausible, we added a balance constraint, and applied dynamic ﬁltering twice, followed by a ﬁnal kinematic ﬁltering to impose the foot constraints (2(D) + 1(K)); these calculations produced Animation 2C (Figure 7(b)). In this animation, the character now sways his body to meet the balance constraints. The previous two results are compared in Animation 2D. Animation 2E is the real-time screen capture during the production of the above animations. Golf Swing. This experiment shows how our technique retargets the golf swing shown in Animation 3A (Figure 8(a)) when a heavy rock (8kg, which is 1/8 of the character’s total mass) is attached to the head of the club. We imposed balance constraints, but did not impose torque constraints because simply raising the club immediately causes torque limit violations at the shoulder and elbow. To facilitate convergence, we provided a kinematic hint, which is actually one kinematic ﬁltering 1(K), mainly ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 113 Fig. 7. Wide Steps: (a) kinematic-ﬁltered (b) dynamic-ﬁltered. Fig. 8. Golf Swing: (a) original motion (b) with a heavier club (dynamic-ﬁltered). consisting of the shift of the pelvis with both feet ﬁxed (Animation 3B). 3(D) + 1(K) ﬁlter applications onto the kinematic hint produced Animation 3C (Figure 8(b)). In the resulting motion, the upper-body of the character makes a large movement to counterbalance the heavy club. The original and retargeted motions are compared in Animation 3D. Limbo Walk. In this experiment, the walking motion shown in Animation 2A is retargeted to a limbo walk (Animation 4A = Animation 2A). We placed a limbo bar at 4/5 of the height of the character. Balance constraints, along with a (soft) kinematic constraint on the head height,5 produced Animation 4B (Figure 9(a)), which is obviously not a limbo motion. To ﬁx the problem, we provided a kinematic hint. A further 2(D) + 1(K) ﬁlter application produced the realistic limbo walk shown in Animation 4C 5 The height was lowered a few steps ahead. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 114 • S. Tak and H.-S. Ko Fig. 9. Limbo Walk: dynamic-ﬁltered (a) without and (b) with kinematic hint. Fig. 10. Jump Kick: (a) original motion (b) when a load is attached (dynamic-ﬁltered). (Figure 9(b)). In another experiment, we animated the limbo motion of a character whose torso was twice as heavy as that of the original character, and we imposed the torque limit constraint. In this setup, the character could not bend the torso to the same extent as in the original case. Thus the kinematic constraint, which was the soft constraint, could not be satisﬁed (Animation 4D). Jump Kick. This experiment shows how our motion editing technique adjusts the jump kick motion shown in Animation 5A (Figure 10(a)) to allow for the addition of a 10kg sandbag to the right calf of the character. The motion consists of a supporting phase, a ﬂight phase, and another supporting phase. We imposed balance and torque constraints along with foot position constraints during the supporting phases, and torque and momentum constraints during the ﬂight phase. Since the position of the landing foot at the end of the ﬂight phase is affected by the attached weight, we adjusted the footprints to account ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 115 for this. Without any kinematic hints, the algorithm produced a convergent result after 3(D)+1(K) ﬁlter applications. In the resulting motion, shown in Animation 5B, it is evident that the character cannot lift the weighted leg as high as the original character due to the torque constraints.6 The momentum constraints make the upper body bend forward to compensate for the momentum change in the right leg (Figure 10(b)). 9. CONCLUSION In this article, we have presented a novel interactive motion editing technique for obtaining a physically plausible motion from a given captured or animated motion. To date, most methods for carrying out such motion retargeting have been formulated as a spacetime constraints problem. In contrast to these previous methods, our method is intrinsically a per-frame algorithm; once the kinematic and dynamic constraint goals are speciﬁed, the proposed algorithm functions as a ﬁlter that sequentially scans the input motion to produce a stream of output motion frames at a stable interactive rate. The proposed method requires interactive tuning of the ﬁlter parameters to adapt it to the differ- ent motions. Several (consecutive) applications of the ﬁlter may be required to achieve the desired convergence or quality, because the ﬁlter works as an enhancement operator. Experiments on a large variety of motions revealed that, in most cases, 3-5 applications are sufﬁcient to produce realistic motions. The method works in a scalable fashion. It provides various ways to trade off run time and animator effort against motion quality. (1) Animators can interactively control the type and amount of kinematic and dynamic constraints to shape the desired motion. (2) Animators can control the number of times the ﬁlter is applied according to the ﬁnal quality that is required. (3) Animators can avoid the potential problem of slow convergence by providing a kinematic hint. This work takes an exciting step forward in the constraint-based motion editing. Now, physically plausible motions can be produced by ﬁltering existing motions on a per-frame basis. ACKNOWLEDGMENTS We would like to thank Michael Gleicher and Norman Badler for their insightful comments. We are greatly indebted to Oh-young Song. Without his keen insightful comments in the initial conception, this work would not have been possible. REFERENCES CHOI, K. AND KO, H. 2000. On-line motion retargetting. J. Visualiza. Comput. Anim. 11, 5, 223–235. COHEN, M. F. 1992. Interactive spacetime constraint for animation. In Comput. Graphics (Proceedings of ACM SIGGRAPH 92) 26, 2, ACM, 293–302. CRAIG, J. J. 1989. Introduction to Robotics. Addison-Wesley. DASGUPTA, A. AND NAKAMURA, Y. 1999. Making feasible walking motion of humanoid robots from human motion capture data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 2. 1044–1049. FALOUTSOS, P., VAN DE PANNE, M., AND TERZOPOULOS, D. 2001. Composable controllers for physics-based character animation. In Proceedings of ACM SIGGRAPH 2001. Computer Graphics Proceedings. ACM Press. 251–260. FANG, A. C. AND POLLARD, N. S. 2003. Efﬁcient synthesis of physically valid human motion. ACM Trans. Graph. 22, 3, 417–426. GEETER, J. D., BRUSSEL, H. V., AND SCHUTTER, J. D. 1997. A smoothly constrained kalman ﬁlter. IEEE Trans. Pattern Anal. Mach. Intell., 1171–1177. GLEICHER, M. 1997. Motion editing with spacetime constraints. In Proceedings of the 1997 Symposium on Interactive 3D Graphics. 6 An approach based on the muscle force model may generate a more plausible motion than clamping the joint torques. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 116 • S. Tak and H.-S. Ko GLEICHER, M. 1998. Retargetting motion to new characters. In Proceedings of ACM SIGGRAPH 98. Computer Graphics Proceedings. ACM Press. 33–42. GLEICHER, M. 2001. Comparing constraint-based motion editing methods. Graphical Models 63, 2, 107–134. JULIER, S. J. AND UHLMANN, J. K. 1997. A new extension of the kalman ﬁlter to nonlinear systems. In Proceedings of AeroSense: The 11th International Symposium on Aerospace/Defense Sensing, Simulation and Controls. KO, H. AND BADLER, N. I. 1996. Animating human locomotion in real-time using inverse dynamics, balance and comfort control. IEEE Comput. Graph. Applic. 16, 2, 50–59. KOMURA, T., SHINAGAWA, Y., AND KUNII, T. L. 1999. Calculation and visualization of the dynamic ability of the human body. J. Visualiz. Comput. Anim. 10, 57–78. LEE, J. AND SHIN, S. Y. 1999. A hierarchical approach to interactive motion editing for human-like ﬁgures. In Proceedings of ACM SIGGRAPH 99. Computer Graphics Proceedings. ACM Press. 39–48. LEE, P., WEI, S., ZHAO, J., AND BADLER, N. I. 1990. Stregth guided motion. In Comput. Graph. (Proceedings of ACM SIGGRAPH 90). 24, 3. ACM, 253–262. ´ LIU, C. K. AND POPOVIC, Z. 2002. Synthesis of complex dynamic character motion from simple animations. ACM Trans. Graph. 21, 3, 408–416. LIU, Z., GORTLER, S. J., AND COHEN, M. F. 1994. Hierarchical spacetime control. In Proceedings of ACM SIGGRAPH 94. Computer Graphics Proceedings. ACM Press. 35–42. MAYBECK, P. S. 1979. Stochastic Models, Estimation, and Control. Vol. 1. Academic Press, Inc. OSHITA, M. AND MAKINOUCHI, A. 2001. A dynamic motion control technique for human-like articulated ﬁgures. In Proceedings of Eurographics 2001. POLLARD, N. S. AND BEHMARAM-MOSAVAT, F. 2000. Force-based motion editing for locomotion tasks. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 1. 663–669. ´ POPOVIC, Z. AND WITKIN, A. 1999. Physically based motion transformation. In Proceedings of ACM SIGGRAPH 99. Computer Graphics Proceedings. ACM Press. 11–20. ROSE, C., GUENTER, B., BODENHEIMER, B., AND COHEN, M. F. 1996. Efﬁcient generation of motion transitions using spacetime constraints. In Proceedings of ACM SIGGRAPH 96. Computer Graphics Proceedings. ACM Press. 147–154. SHABANA, A. A. 1994. Computational Dynamics. John Wiley & Sons, Inc. SHIN, H. J., KOVAR, L., AND GLEICHER, M. 2003. Physical touch-up of human motions. In Proceedings of Paciﬁc Graphics 2003. SHIN, H. J., LEE, J., SHIN, S. Y., AND GLEICHER, M. 2001. Computer puppetry: An importance-based approach. ACM Trans. Graph. 20, 2, 67–94. SIMON, D. AND CHIA, T. 2002. Kalman ﬁltering with state equality constraints. IEEE Trans. Aeros. Electr. Syst. 39, 128– 136. SUGIHARA, T., NAKAMURA, Y., AND INOUE, H. 2002. Realtime humanoid motion generation through zmp manipulation based on inverted pendulum control. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 2. 1404–1409. TAK, S., SONG, O., AND KO, H. 2000. Motion balance ﬁltering. Comput. Graph. For. (Eurographics 2000) 19, 3, 437–446. TAK, S., SONG, O., AND KO, H. 2002. Spacetime sweeping: An interactive dynamic constraints solver. In Proceedings of Computer Animation 2002. 261–270. VAN DER MERWE, R. AND WAN, E. A. 2001. The squre-root unscented kalman ﬁlter for state and parameter-estimation. In Proceedings of International Conference on Acoustics, Speech, and Signal Processing. VAN DE PANNE, M. 1996. Parameterized gait synthesis. IEEE Comput. Graph. Applica. 16, 2, 40–49. ´ ´ VUKOBRATOVIC, M., BOROVAC, B., SURLA, D., AND STOKIC, D. 1990. Biped Locomotion: Dynamics, Stability, Control and Application. Springer Verlag. WAN, E. A. AND VAN DER MERWE, R. 2000. The unscented kalman ﬁlter for nonlinear estimation. In Proceedings of Symposium 2000 on Adaptive Systems for Signal Processing, Communication and Control. WAN, E. A. AND VAN DER MERWE, R. 2001. Kalman Filtering and Neural Networks (Chapter 7. The Unscented Kalman Filter). John Wiley & Sons. WELCH, G. AND BISHOP, G. 2001. An introduction to the kalman ﬁlter. ACM SIGGRAPH 2001 Course Notes. WINTER, D. A. 1990. Biomechanics and Motor Control of Human Movement. John-Wiley, New York. WITKIN, A. AND KASS, M. 1988. Spacetime constraints. In Comput. Graph. (Proceedings of ACM SIGGRAPH 88). 22, 4. ACM, 159–168. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. A Physically-Based Motion Retargeting Filter • 117 YAMANE, K. AND NAKAMURA, Y. 2000. Dynamics ﬁlter: Concept and implementation of online motion generator for human ﬁgures. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vol. 1. 688–694. YAMANE, K. AND NAKAMURA, Y. 2003. Dynamics ﬁlter—concept and implementation of online motion generator for human ﬁgures. IEEE Trans. Robot. Autom. 19, 3, 421–432. ZORDAN, V. B. AND HODGINS, J. K. 2002. Motion capture-driven simulations that hit and react. In 2002 ACM SIGGRAPH Symposium on Computer Animation. 89–96. Received May 2003 revised October 2003, July 2004; accepted October 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Perceptual Photometric Seamlessness in Projection-Based Tiled Displays ADITI MAJUMDER University of California, Irvine and RICK STEVENS University of Chicago, Argonne National Laboratory Arguably, the most vexing problem remaining for multi-projector displays is that of photometric (brightness) seamlessness within and across different projectors. Researchers have strived for strict photometric uniformity that achieves identical response at every pixel of the display. However, this goal typically results in displays with severely compressed dynamic range and poor image quality. In this article, we show that strict photometric uniformity is not a requirement for achieving photometric seamlessness. We introduce a general goal for photometric seamlessness by deﬁning it as an optimization problem, balancing perceptual uniformity with display quality. Based on this goal, we present a new method to achieve perceptually seamless high quality displays. We ﬁrst derive a model that describes the photometric response of projection-based displays. Then we estimate the model parameters and modify them using perception-driven criteria. Finally, we use the graphics hardware to reproject the image computed using the modiﬁed model parameters by manipulating only the projector inputs at interactive rates. Our method has been successfully demonstrated on three different practical display systems at Argonne National Laboratory, made of 2 × 2 array of four projectors, 2 × 3 array of six, projectors, and 3 × 5 array of ﬁfteen projectors. Our approach is efﬁcient, automatic and scalable—requiring only a digital camera and a photometer. To the best of our knowledge, this is the ﬁrst approach and system that addresses the photometric variation problem from a perceptual stand point and generates truly seamless displays with high dynamic range. Categories and Subject Descriptors: I.3.3 [Computer Graphics]: Picture/Image Generation—Display algorithms; I.4.0 [Image Processing and Computer Vision]: General—Image displays; I.4.8 [Image Processing and Computer Vision]: Scene Analysis—Color; Photometry; H.1.2 [Models and Principles]: User/Machine Systems—Human factors General Terms: Algorithms, Human Factors Additional Key Words and Phrases: Projection-based displays, tiled displays, color calibration 1. INTRODUCTION Large-area multi-projector displays offer an inexpensive way to display high-resolution life-size im- ages. These displays are used extensively for large scale scientiﬁc visualizations and for virtual reality applications in defense, entertainment, simulation, and training. Very high resolution displays made Authors’ addresses: A. Majumder, Department of Computer Science, 444 Computer Science Building, University of California, Irvine, CA 92696; email: majumder@cs.uci.edu; R. Stevens, Mathematics and Computer Science Division, Argonne National Laboratory, 9700 South Cass Avenue, Building 221, Argonne, IL 60439. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0118 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 118–139. Photometric Seamlessness in Projection-Based Tiled Displays • 119 of 40–50 projectors are being built at the National Center for Supercomputing Applications at the University of Illinois, Urbana-Champaign and various U.S. national laboratories. In building large area displays, there exist several issues such as driving architecture and data distribution [Samanta et al. 1999; Humphreys et al. 2000, 2001; Buck et al. 2000; Humphreys and Hanrahan 1999; Humphreys et al. 2001], but geometric misalignment and color variation are the most salient issues to be addressed to make multi-projector displays perceptually “seamless”. Several algorithms achieve geometrically undistorted and aligned displays [Raskar et al. 1998, 1999; Yang et al. 2001; Hereld et al. 2002; Raskar 1999; Cruz-Neira et al. 1993; Chen et al. 2002]. But the color variation is still a signiﬁcant obstacle. Color is a three-dimensional quantity deﬁned by one-dimensional luminance (deﬁning brightness) and two-dimensional chrominance (deﬁning hue and saturation). Thus, the color variation problem involves spatial variation in both luminance and chrominance. Majumder and Stevens [2004] and Majumder [2002] show that most current tiled displays made of projectors of the same model show large spatial variation in luminance while the chrominance is almost constant spatially. Also, humans are at least an order of magnitude more sensitive to luminance variation than to chrominance variation [Chorley and Laylock 1981; Goldstein 2001; Valois and Valois 1990]. Thus, perceptually, the subproblem of photometric variation (luminance variation) is the most signiﬁcant contributor to the color variation problem. Majumder and Stevens [2004] classify the spatial color variation in multi-projector displays in three different classes: the variation within a single projector (intra-projector variation), across different pro- jectors (inter-projector variation) and in the overlap regions (overlap variation). Most existing methods [Pailthorpe et al. 2001; Stone 2001a, 2001b; Cazes et al. 1999; Majumder et al. 2000] do not address the spatial variation within a projector and assume that every pixel of a projector has identical color response. This assumption simpliﬁes the color response reconstruction since the color response of each projector needs to be estimated at only one spatial location using a radiometer or a photometer. How- ever, Majumder and Stevens [2002, 2004], Stone [2001a, 2001b] show this to be an over-simplifying assumption. Even within a single projector, a photometric fall-off as large as 40–60% from the center of the display to the fringes is observed. Thus, methods that ignore this spatial variation cannot achieve seamless displays. Blending or feathering techniques address only the variation in the overlap regions and try to smooth color transitions across these regions. This can be achieved in software [Raskar et al. 1998], or by physical masks mounted at the projector boundaries [Li et al. 2000], or by optical masks inserted in the light path of the projector [Chen and Johnson 2001]. However, these methods do not estimate the variation in the overlap region and assume linear response projectors. This inaccuracy results in softening of the seams in the overlapping region, rather then removing them. Nayar et al. [2003] addresses the problem of using a single projector on screens with imperfect reﬂectance (like a poster or a brick wall), and hence addresses only intra-projector variations. Majumder and Stevens [2002] and Majumder et al. [2003] present methods that attempt to estimate and correct intra- inter- and overlap photometric variations. However, due to the goal of achieving strict photometric unifor- mity, the photometric response of every pixel in the display is matched with the response of the “worst” possible pixel. This leads to poor image quality and low dynamic range (Figure 7). In this article, we show that strict photometric uniformity is not required to achieve a photometrically seamless display where we cannot tell the number of projectors making up the display. Analysis of the spatial color variation in a stand-alone projector supports this realization. A stand-alone projector is not photometrically uniform, yet it is seamless (Figure 2). Hence, in this article, we demonstrate that a smoothly varying photometric response that exploits human perceptual limitations is sufﬁcient to achieve seamlessness in a multi-projector display. Further, this less restrictive goal provides extra leverage to increase the overall dynamic range, producing a display of far superior quality. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 120 • A. Majumder and R. Stevens Based on the above concept, following are the main contributions of this work. (1) We realize that all pixels need not have identical response to achieve a seamless display. Instead, a perceptual uniformity can be achieved by exploiting the limitation of human vision and this weaker constraint can be leveraged to increase the display quality. (2) We formalize such a goal of photometric seamlessness as an optimization problem that maximizes display quality, while minimizing the noticeable photometric variations. (3) We design and implement a new practical method to solve this optimization problem and generate photometrically seamless high quality display. Our method is automatic, scalable, and solves the intra-, inter- and overlap photometric variations in a uniﬁed manner. Further, this method is imple- mented using commodity graphics hardware to correct any imagery projected on a multi-projector display at interactive rates. In order to achieve the above results, we ﬁrst derive a model to comprehensively describe the photo- metric variation across multi-projector displays (Section 2). Next, we formalize the problem of achiev- ing photometric seamlessness as an optimization problem (Section 3). Finally, we present a method to achieve this photometric seamlessness (Section 4). 2. THE PHOTOMETRIC VARIATION In this section, we derive an equation to describe the photometric variation in Lambertian multi- projector displays. Table I provides a list of all the symbols and parameters used in this article, their nomenclature, and description. 2.1 Deﬁnitions A planar multi-projector display is a display made of N projectors, projecting on a planar display screen. Each projector is denoted by P j , 1 ≤ j ≤ N . Figure 1 shows a simple two-projector display wall. A uniﬁed display coordinate space is deﬁned by u, v, and w axes, where u and v describe the display plane and w is perpendicular to it. The projector coordinates (s j , t j ) can be related to the display coordinates (u, v) by a geometric warp G j , such that (u, v) = G j (s j , t j ). (1) Note that G j also depends on the internal and external geometric parameters of projector P j including its focal length, position, and orientation in the display coordinate space. For all practical purposes, these geometric parameters do not change since the projectors and screen do not move relative to each other. Hence we deﬁne G j as a function of the projector coordinates only. A projector has three channels, {r, g , b}. A channel is denoted by l ∈ {r, g , b}, and the corresponding input by il ∈ {ir , i g , ib}, 0.0 ≤ il ≤ 1.0.1 2.2 Describing the Variation We deﬁne a function L(u, v, i) as the luminance projected at any display coordinate (u, v) for an input i = (ir , i g , ib) to all the projectors projecting at (u, v). We derive L here from the extensive analysis of projection based displays presented in Majumder and Stevens [2004] and Majumder [2002]. Single Projector: Ideally, if the luminance from different channels of a projector are independent of each other, the luminance L(s, t, i) at projector coordinate (s, t) for input i is given by the addition of 1 This considers only three primary systems. The DLP projectors that use a clear ﬁlter for projecting the grays behave like a four primary system. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 121 Table I. Legend of all symbols and parameters, in the order in which they appear in the article. The parameters can be associated with a single projector or a multi-projector display. The difference has been emphasized by underlining the association Symbols Nomenclature Description l Channel l ∈ {r, g , b} and denotes the red, green and blue channels. il Channel input Input for channel l . i Input The three channel input, (ir , i g , ib ). (s, t) Projector coordinates 2D coordinates of a projector. (u, v) Display coordinates 2D coordinates of the multi-projector display. G(s, t) The projector geometric warp relating (s, t) with (u, v). N p (u, v) Number of projectors overlapping at (u, v). L(s, t, i) Luminance projected at (s, t) of a projector for input i. hl (il ) Transfer function of The fraction of the maximum luminance of channel l projected channel l for input il in a projector. B(s, t) Black luminance function The black offset at (s, t) of a projector. Wl (s, t) Maximum luminance The maximum luminance projected from channel l at (s, t) of a function of channel l projector. L(u, v, i) The luminance projected at (u, v) of a multi-projector display when N p (u, v) projectors are projecting input i. Hl (il ) Common transfer function The spatially invariant input transfer function of channel l in a of channel l multi-projector display. Wl (u, v) Maximum display The maximum luminance projected from channel l at (u, v) in a luminance function of multi-projector display. channel l B(u, v) Black display luminance The black offset at (u, v) of a multi-projector display. function L (u, v, i) The luminance projected after modiﬁcation at (u, v) in a multi-projector display when all N p (u, v) projectors are projecting input i. Wl (u, v) Smooth maximum display The maximum luminance projected from channel l of a luminance multi-projector display at (u, v) after modiﬁcation. function of channel l B (u, v) Smooth black display The black offset at (u, v) of a multi-projector display after luminance function modiﬁcation. il Modiﬁed channel input The reprojected input for channel l . Sl (u, v) Display scaling map of The attenuation at (u, v) for channel l in a multi-projector channel l display to achieve the smooth display luminance functions. Ol (u, v) Display offset map of The offset at (u, v) for channel l in a multi-projector display to channel l achieve the smooth display luminance functions (Sl , Ol ) Display smoothing maps of The scaling and offset map of channel l in a multi-projector channel l display. Sl (s, t) Projector scaling map of The attenuation at (s, t) for channel l of a single projector to channel l achieve the smooth display luminance functions. Ol (s, t) Projector offset map of The offset at (u, v) for channel l of a single projector to achieve channel l the smooth display luminance functions. (Sl , Ol ) Projector smoothing maps The scaling and the offset maps of a single projector. λ Smoothing parameter The parameter used during optimization for generating the smooth display luminance functions. the luminances from the three channels. However, in practice, in addition to the light projected from the three channels, some extra leakage light is also projected at all times, commonly called the black offset. We represent this by a spatially varying function, B(s, t), called the black luminance function. The maximum luminance function, Wl (s, t), for channel l at projector coordinate (s, t), is deﬁned as the maximum of the luminances projected by all inputs il . Note that, due to the presence of the black ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 122 • A. Majumder and R. Stevens Fig. 1. Projector and display coordinate space. luminance function, B(s, t), the range of luminance that is actually contributed from the channel l is given by Wl (s, t)− B(s, t). Thus, for any other input il , the luminance contribution of channel l , Dl (s, t, il ), is a fraction of Wl (s, t) − B(s, t), Dl (s, t, il ) = hl (il )(Wl (s, t) − B(s, t)) (2) where 0.0 ≤ hl (il ) ≤ 1.0. Majumder [2002] shows that hl does not vary spatially. Hence, we express hl as a function of il only, and and call it the transfer function for channel l . Note that hl is similar to the gamma function in other displays. In projectors, this function cannot be expressed by a power function and, therefore, we prefer to call it the transfer function. Thus, the luminance projected by a practical projector at projector coordinate (s, t) is given by the summation of the contributions from its three channels and the black luminance function as L(s, t, i) = Dr (s, t, ir ) + D g (s, t, i g ) + Db(s, t, i g ) + B(s, t). Substituting Equation (2) in the above equation, L(s, t, i) = hl (il )(Wl (s, t) − B(s, t)) + B(s, t). (3) l ∈{r, g ,b} The maximum channel luminance functions (Wl ) and the black luminance function (B) are together called the projector luminance functions. Multi-Projector Display: Let N P denote the set of projectors overlapping at a display coordinate (u, v). The luminance, L(u, v, i) at (u, v) for input i in a multi-projector display is given by the addition of luminance L j from every projector j in N P , L(u, v, i) = L j (s j , t j , i), (4) j ∈N P where (u, v) = G j (s j , t j ), by Equation (1).2 The variation in L with respect to (u, v) describes the photometric variation in a multi-projector display. 2 When p is not a constant, G, Wl , and B are also dependent on p. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 123 Fig. 2. The left plot shows the maximum luminance function for the green channel (W g ) in green and the black luminance function (B) in red for a single projector display. The right picture shows the image seen by the viewer corresponding to W g in the left plot. Note that even though this image looks seamless to our eye, the corresponding luminance plot on the left is not ﬂat or uniform. 3. PERCEPTUAL PHOTOMETRIC SEAMLESSNESS The goal of photometric seamlessness is to generate a multi-projector display that looks like a single projector display, such that we cannot tell the number of projectors making up the display. It has been a common assumption that to achieve this, strict photometric uniformity, that is, identical brightness response at every pixel of the display, has to be achieved. Mathematically, a display is photometrically uniform if, for the same input i, the luminances at any two display coordinates (u1 , v1 ) and (u2 , v2 ), are equal, that is, ∀i, L(u1 , v1 , i) = L(u2 , v2 , i). (5) Analyzing Equation (3), we can say that in single projector displays, if the transfer function (hl (il )) is spatially invariant, the luminance functions (Wl (s, t) and B(s, t)) should be spatially constant to achieve strict photometric uniformity. However, if we analyze the behavior of a single projector, we ﬁnd that though the transfer function is indeed spatially invariant [Majumder 2002], the luminance functions of Wl and B are hardly ﬂat, as shown in Figures 2 and 3. But the human eye perceives the image projected by a single projector as uniform. Thus, we realize that strict photometric uniformity is not required for a seamless display; rather a smoothly varying response ensuring perceptual uniformity is sufﬁcient. Formally, a display is perceptually uniform if, for the same input i, the luminances from any two display coordinates (u1 , v1 ) and (u2 , v2 ) differ within a certain threshold that cannot be detected by the human eye, that is, ∀i, L(u1 , v1 , i) − L(u2 , v2 , i) ≤ , (6) where is a function that can depend on various parameters like distance between (u1 , v1 ) and (u2 , v2 ), resolution of the display, distance of viewer, viewing angle, human perception limitations, and some- times even the task to be accomplished by the user. Further, note that strict photometric uniformity is a special case of this perceptual uniformity when = 0, ∀(u1 , v1 ), (u2 , v2 ). High dynamic range and brightness are essential for good image quality [Debevec and Malik 1997; Larson 2001]. Similarly, high lumens rating (high brightness) and low black offsets (high contrast) are essential for good display quality. However, the criterion of perceptual uniformity alone may not ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 124 • A. Majumder and R. Stevens Fig. 3. This plot shows the zoomed in view of the black luminance function (B). ensure a good display quality. For example, strict photometric uniformity, which implies perceptual uniformity, does not ensure good display quality since it forces the display quality to match the ‘worst’ pixel on the display, leading to compression in dynamic range, as illustrated in Figure 7. But the goal of perceptual uniformity (Equation (6)), being less restrictive, can provide the extra leverage to increase the overall display quality. So, we deﬁne the problem of achieving photometric seamlessness as an optimization problem where the goal is to achieve perceptual uniformity while maximizing the display quality. This optimization is a general concept and can have different formalizations based on two factors: the parameters on which of Equation (6) depends, and the way display quality is deﬁned. We present one such formalization while presenting our algorithm in Section 5. 4. OUR ALGORITHM In this section, we present our algorithm to generate perceptually seamless high quality multi-projector displays. Our algorithm has three steps: (1) Reconstruction: First, the parameters of function L described in Section 2 are estimated using a photometer and a digital camera. (2) Modiﬁcation: Next, these estimated parameters are modiﬁed by solving the optimization problem while taking perception into account. The hypothetical display deﬁned by the modiﬁed parameters is seamless and has high dynamic range. (3) Reprojection: Finally, this hypothetical display is realized using the practical display by factoring all the changes in the model parameters into any input image at interactive rates. In other words, the actual image projected using the hypothetical seamless display is the same as the modiﬁed image projected by the practical display. 4.1 Reconstruction There are three projector parameters on which L in Equation (4) depends on: the transfer functions hl , and the luminance functions Wl and B. In this step, we reconstruct these parameters for each projector. Transfer Function (hl ): Since hl (il ) is spatially constant [Majumder 2002], we use a point measure- ment device like a photometer to estimate it at one spatial coordinate for each projector. This slower ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 125 Fig. 4. The transfer function (left) and the inverse transfer function (left) for a single channel of a projector. measurement process (1–20 seconds-per-measurement) is justiﬁed since hl changes little temporally [Majumder 2002] and needs to be measured very infrequently. In our experiments, measuring hl once in 9–12 months was sufﬁcient. However, a method proposed recently in Raij et al. [2003] can be used to estimate hl using a camera. Here, we describe how we estimate h g . Other transfer functions, hr and hb, are estimated in an analogous manner. We deﬁne input iG = (0, i g , 0), 0.0 ≤ i g ≤ 1.0 as one where the green channel input is i g and the red and blue channel inputs are 0. We ﬁrst measure the luminance response, L(s, t, iG ), for O(28 ) values of i g and sample the input space of channel g densely. Since this luminance response is measured at one spatial location, we omit the spatial coordinates and denote it as L(iG ) only. The measured luminance, L(iG ), includes the black offset and is not monotonic with respect to iG . So, we ﬁnd the maximum input range iG m ≤ iG ≤ iG M within which L(iG ) is monotonic. iG m = (0, i g m , 0) and iG M = (0, i g M , 0) denote respectively; the lower and the upper boundary of this range. h g is then estimated from Equation (3) as L(iG ) − L(iG m ) h g (i g ) = . (7) L(iG M ) − L(iG m ) h g thus estimated is a monotonic function with min∀i g h g (i g ) = h g (i g m ) = 0 and max∀i g h g (i g ) = h g (i g M ) = 1 (Figure 4). Luminance Functions (Wl , B): Wl and B are spatially varying functions. Hence, a digital camera is used to estimate them using techniques presented in Majumder and Stevens [2002]. First, for a suitable position and the orientation of the camera looking at the entire display, we estimate the geometric warping function that transforms every projector pixel to the appropriate camera pixel. We use camera-based geometric calibration methods [Hereld et al. 2002; Chen et al. 2002; Yang et al. 2001; Raskar 1999; Raskar et al. 1999] to estimate this transformation. Such methods also ﬁnds G, the geometric warp between the projector and display coordinate system. Then, for each luminance function, an appropriate test image is projected by the projector, captured by the camera, and the luminance functions are estimated. Note that the images to estimate these functions must be taken from the same position and orientation as the images for geometric calibration. The test images are comprised of identical input at every projector coordinate. From Equation (3), we ﬁnd that the input that has to be projected to estimate B is (irm , i g m , ibm ), that is, the input that projects minimum luminance for each channel. Similarly, to estimate the maximum luminance function for the ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 126 • A. Majumder and R. Stevens Fig. 5. To compute the maximum luminance function for the green channel of each projector, we need only four pictures. This reduction in the number of images is achieved by turning on more than one nonoverlapping projectors while capturing each image. Top: Pictures taken for a display made of a 2 × 2 array of 4 projectors. Bottom: The pictures taken for a display made of a 3 × 5 array of 15 projectors. green channel, W g , we have to project (irm , i g M , ibm ), that is, the input that projects maximum luminance from the green channel and minimum luminance from the other two channels. The test images for estimating W g of all the projectors are shown in Figure 5. Figure 2 shows W g and B estimated in the some manner for one of these projectors. The computation of test images for the estimation of Wr and Wb are done analogously. 4.2 Modiﬁcation The goal of the modiﬁcation step is to deﬁne a hypothetical display from the given display that produces seamless image. To achieve our goal of making a multi-projector display look like a single projector display, we deﬁne the parameters for a multi-projector display analogous to the parameters hl , Wl , and B for a single projector display. Perceptual factors are taken into account while deﬁning these parameters. Unlike a single projector display where hl is spatially constant, different parts of the multi-projector display have different hl s since they are projected from different projectors. Further, for the same reason, functions analogous to the luminance functions (Wl and B) of a single projector cannot be deﬁned for a multi-projector display. So, to achieve our goal, the following modiﬁcations are performed. (1) A perceptually appropriate common transfer function Hl is chosen that is spatially invariant throughout the multi-projector display. (2) Luminance functions Wl and B for the entire multi-projector display are identiﬁed. (3) Finally, these luminance functions are made perceptually uniform like that of a single projector display (Figure 2). 4.2.1 Choosing a Common Transfer Function. We choose a common transfer function Hl for each channel l , that satisﬁes the three conditions: Hl (0) = 0, Hl (1) = 1, and Hl (il ) is monotonic. The transfer function hl of all the different projectors is then replaced by this common Hl to assure a spatially invariant transfer function across the whole multi-projector display. So, Equation (4) becomes L(u, v, i) = Hl (il ) (Wl j (s j , t j ) − B j (s j , t j )) + B j (s j , t j ), (8) l j j where Wl j and B j are the luminance functions for projector P j . We use Hl = il2 which is commonly used to approximate the logarithmic response of the human eye to varying luminance. In previous methods [Majumder and Stevens 2002], Hl was chosen to be a linear function, Hl (il ) = il , and resulted in washed out images. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 127 4.2.2 Identifying the Display Luminance Functions. In Equation (8), using Wl (u, v) = Wl j (s j , t j ) and B(u, v) = B j (s j , t j ), with appropriate coordinate transformation between projector and display coordinate space, we get L(u, v, i) = Hl (il )(Wl (u, v) − B(u, v)) + B(u, v). (9) l ∈{r, g ,b} Note that the above equation for the whole multi-projector display is exactly analogous to that of a single projector (Equation (3)). Hence, we call Wl and B the maximum display luminance function for channel l and the black display luminance function, respectively. Figure 6 shows W g for a four projector display, and W g and B for a ﬁfteen projector display. 4.2.3 Modifying the Display Luminance Functions. The multi-projector display deﬁned by Equation (9) still does not look like a single projector display because, unlike the single projector lumi- nance functions (Figure 2), the analogous multi-projector luminance functions (Figure 6) have sharp discontinuities that result in perceivable seams in the display. However, perception studies show that humans are sensitive to signiﬁcant luminance discontinuities, but can tolerate smooth luminance vari- ations [Chorley and Laylock 1981; Goldstein 2001; Valois and Valois 1990]. So, to make these functions perceptually uniform, we smooth Wl (u, v) and B(u, v) to generate Wl (u, v) and B (u, v), and call them the smooth display luminance functions. We formulate this smoothing as an optimization problem where we ﬁnd a Wl (u, v) that minimizes the deviation from the original Wl (u, v), assuring high dynamic range, and, at the same time, maximizes its smoothness, assuring perceptual uniformity. The smoothing criteria is chosen based on quantitative measurement of the limitation of the human eye in perceiving smooth luminance variations. We present a constrained gradient-based smoothing method to ﬁnd the optimal solution to this problem. This smoothing is explained in detail in Section 5. Figure 6 shows W g (u, v) for a ﬁfteen-projector display. The perceptually seamless luminance response of the display L (u, v, i) generated is derived from Equation (9): L (u, v, i) = Hl (il )(Wl (u, v) − B (u, v)) + B (u, v). (10) l ∈{r, g ,b} When both Wl (u, v) and B (u, v) are smooth, and the transfer function Hl is spatially invariant, L (u, v, i)) is also smooth and satisﬁes Equation (6). 4.3 Reprojection In the previous section, we theoretically modiﬁed the display parameters to generate a hypothetical display that would project a seamless imagery. Now we have to make the practical display behave like the hypothetical display with these modiﬁed parameters. The projector hardware does not offer us precision control to directly modify Wl (u, v), B(u, v), and hl to create the hypothetical display. So, we achieve the effects of these modiﬁcations by changing only the input il at every projector coordinate. This step is called reprojection. In this section, we explain how this modiﬁcation is achieved for any one projector of a multi-projector display, at its coordinate (s, t). Since the modiﬁcation is pixel-based, we have left out the (s, t) from all the equations in this section. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 128 • A. Majumder and R. Stevens Fig. 6. Left Column: The estimated maximum display luminance function of green channel (W g ) for a 2 × 2 array of projectors (top). The estimated maximum display luminance function for green channel (W g ) and the black display luminance function (B) of a 3×5 array of projectors (bottom). The high luminance regions in both correspond to the overlap regions across different projectors. Right Column: Smooth maximum display luminance function for green channel (W g ) achieved by applying the constrained gradient-based smoothing algorithm on the corresponding maximum display luminance function on the left column. Note that these are perceptually smooth even though they are not geometrically smooth. For a given il , the actual response of the display is given by, (hl (il )(Wl − B)) + B. l ∈{r, g ,b} The goal of reprojection is to get the response (Hl (il )(Wl − B )) + B l ∈{r, g ,b} ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 129 to simulate a modiﬁed perceptually seamless display. So, we modify the input il to il such that (Hl (il )(Wl − B )) + B = (hl (il )(Wl − B)) + B. (11) l ∈{r, g ,b} l ∈{r, g ,b} It can be shown that the following il solves Equation (11). −1 il = hl (Hl (il )Sl + Ol ), (12) −1 where hl is the inverse transfer function of a projector and can be computed directly from hl as shown Figure 4, and Sl (u, v) and Ol (u, v) are called the display scaling map and display offset map, respectively, and are given by Wl (u, v) − B (u, v) B (u, v) − B(u, v) Sl (u, v) = ; Ol (u, v) = . (13) Wl (u, v) − B(u, v) 3(Wl (u, v) − B(u, v)) Intuitively, the scaling map represents the pixel-wise attenuation factor needed to achieve the smooth maximum display luminance functions. The offset map represents the pixel-wise offset factor that is needed to correct for the varying black offset across the display. Together, they comprise what we call the display smoothing maps for each channel. From these display smoothing maps, the projector smoothing maps, Sl j (s j , t j ) and Ol j (s j , t j ), for channel l of projector P j are cut out, using the geometric warp G j as follows, Sl j (s j , t j ) = Sl (G j (s j , t j )); Ol j (s j , t j ) = Ol (G j (s j , t j )). (14) Figure 9 shows the scaling maps thus generated for the whole display and one projector. Therefore, any image projected by a projector can be corrected by applying the following three steps in succession to every channel input, as illustrated in Figure 9. (1) First, the common transfer function is applied to the input image. (2) Then, the projector smoothing maps are applied. This involves pixel-wise multiplication of the attenuation map, and then, the addition of the the offset map. (3) Finally, the inverse transfer function of the projector is applied to generate the corrected image. 5. SMOOTHING DISPLAY LUMINANCE FUNCTIONS In this section, we describe in detail the method to generate W and B from W and B, respectively. We approximate B (u, v) as B (u, v) = max B(u, v), ∀u,v since the variation in B(u, v) is almost negligible when compared to Wl (u, v) (Figure 6). B (u, v) deﬁnes the minimum luminance that can be achieved at all display coordinates. We use the constrained gradient-based smoothing method to generate Wl (u, v) from Wl (u, v). Wl (u, v) is deﬁned by the following optimization constraints. Capability Constraint. This constraint of Wl ≤ Wl ensures that Wl never goes beyond the maximum luminance achievable by the display, Wl . In practice, with discrete sampling of these functions, W [u][v] < W[u][v], ∀u, v. Perceptual Uniformity Constraint. This constraint assures that Wl has a smooth variation, imper- ceptible to humans. ∂Wl 1 ≤ × Wl , ∂x λ ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 130 • A. Majumder and R. Stevens ∂W where λ is the smoothing parameter and ∂ x l is the gradient of Wl , along any direction x. Compare this inequality with Equation (6). In the discrete domain, when the gradient is expressed as a linear ﬁlter involving the eight neighbors (u , v ) of a pixel (u, v), u ∈ {u − 1, u, u + 1} and v ∈ {v − 1, v, v + 1}, this constraint is given by |W [u][v] − W [u ][v ]| 1 ≤ W [u][v], ∀u, v, u , v . |u − u |2 + |v − v |2 λ Display Quality Objective Function. The above two constraints can yield many feasible Wl . To maxi- mize dynamic range, the integration of Wl has to be maximized. In discrete domain, this is expressed as X −1 Y −1 maximize W [u][v], u=0 v=0 where X and Y denote the height and width of the multi-projector display in number of pixels. We have designed a fast and efﬁcient dynamic programming method that gives the optimal solution for this optimization in linear time with respect to the number of pixels in the display, that is, O(XY). The time taken to compute this solution on an Intel Pentium III 2.4GHz processor for displays with 9 million pixels is less than one second. The pseudo-code for the algorithm is given in Appendix A. The solution to the above optimization problem smooths the luminance variation across the display. The general idea that smoothing the luminance response would achieve seamless results has been used effectively in the image processing domain in the past [Gonzalez and Woods 1992; Land 1964; Land and McCann 1971]. However, the luminance variation correction for multi-projector displays cannot be achieved just by smoothing. For example, gradient-or curvature-based linear smoothing ﬁlters, which are popular operators in image processing applications, smooth the hills and ﬁll the valleys. However, our constraints are such that while the hills can be smoothed, the troughs cannot be ﬁlled up since the response so achieved will be beyond the display capability of the projectors. So, the desired smoothing for this particular application was formalized as an optimization problem. Finally, note that this method not only adjusts the variations in the overlap region, but corrects for intra and inter projector variations also, without treating any one of them as a special case. This differentiates our method from any existing edge-blending method that addresses overlap regions only. 5.1 Smoothing Parameter The smoothing parameter λ is derived from the human contrast sensitivity function (CSF) [Valois and Valois 1990; Chorley and Laylock 1981]. Contrast threshold deﬁnes the minimum percentage change in luminance that can be detected by a human being at a particular spatial frequency. The human CSF is a plot of the contrast sensitivity (reciprocal of contrast threshold) with respect to spatial frequency (Figure 8). The CSF is bow-shaped with maximum sensitivity at a frequency of 5 cycle-per degree of angle subtended on the eye. In other words, a variation of less than 1% in luminance will be imperceptible to the humans eye for a spatial grating of 5 cycles/degree. For other frequencies, a greater variation can be tolerated. This fact is used to derive the relative luminance that a human being can tolerate at every pixel of a display as follows. Let d be the perpendicular distance of the user from the display (can be estimated by the position of the camera used for calibration), r is the resolution of the display in pixels-per-unit-distance, and τ is the contrast threshold that humans can tolerate per-degree of visual angle (1% at peak sensitivity). From this, the number of pixels subtended, per degree of the human eye, is given by d πr . Since the 180 peak sensitivity occurs at 5 cycles per degree of visual angle, the number of display pixels per cycle of d πr the grating is given by 180×5 . Within the above pixels, a luminance variation of τ will go undetected by ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 131 human beings. Thus, d πr λ= . (15) 900τ For our ﬁfteen-projector display, r is about 30 pixels-per-inch. For a user about 6 feet away from the display, by substituting τ = 0.01 in Equation (15), we get a λ of about 800. Note that as the user moves farther away from the display λ goes up, that is, the surface needs to be smoother and will have lower dynamic range. This also explains the variations perceptible in some of our results in Figure 7, 10, 11, and 12, though they are imperceptible when seen in person on the large display. Since the results in this article are highly scaled down images of the display, they simulate a situation where the user is inﬁnitely far away from the display and, therefore, inﬁnite lambda is required to make all the luminance variations imperceptible. The smoothing parameter chosen for a large display is much smaller and hence is not suitable for small images on paper. 6. IMPLEMENTATION The algorithm pipeline is illustrated in Figure 9. Reconstruction, modiﬁcation, and part of the repro- jection are done ofﬂine. These comprise the calibration step. The outputs of this step are the projector smoothing maps, the projector inverse transfer functions, and the common transfers function for each channel. These are then used in the per projector image correction step to correct any image projected on the display, using Equation (12). Scalability: The limited resolution of the camera can affect the scalability of the reconstruction for very large displays (made of 40–50 projectors). So, we used techniques presented in Majumder et al. [2003] to design a scalable version of the reconstruction method. We rotate and zoom the camera to estimate the luminance functions of different parts of the display from different views. These are then stitched together to create the luminance function of the whole display. The results of this scalable version of our algorithm are shown in Figure 10. The details of this method are available in Majumder et al. [2003]. Real Time Implementation: We achieve the per-pixel image correction interactively in commodity graphics hardware using pixels shaders. In our real-time correction, the scene is ﬁrst rendered to tex- ture, and then the luminance corrections are applied. Multi-texturing is used for applying the common transfer function and the smoothing maps. The inverse transfer function is applied using dependent 2D texture look-ups. This is followed by an image warp to correct the geometric misalignments. Choosing the common transfer function i 2 , as opposed to i 1.8 (as used in the Macintosh Operating System) and i 2.2 (as used in Windows), helps us to implement this step in the pixel shader using multi-texturing as a multiplication of the image by itself. 7. RESULTS We have applied our algorithm in practice to display walls of different sizes at Argonne National Laboratory. Figures 7, 11 and 12 show the digital photographs of our results on a 2 × 3 array of six projectors (4-feet wide and 3-feet high), 3 × 5 array of ﬁfteen projectors(10-feet wide and 8-feet high), and a 2 × 2 (2.5-feet wide and 1.5-feet high) array of four projectors. The most popular approach for color correction to date is to use software edge-blending in the overlap regions, or hardware edge-blending with custom-designed metal masks and mounts on the optical path of each projector. Both of these methods are featured on the display walls at Argonne National Labora- tory. The results of these solutions, illustrated in Figure 13, still show the seams. Since these solutions assume constant luminance function for each projector, their correction results in darker boundaries ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 132 • A. Majumder and R. Stevens Fig. 7. Digital photographs of a ﬁfteens projector tiled display (8 × 10 in size) before any correction (top), after constraint gradient-based smoothing with smoothing parameter of λ = 400, (second from top), λ = 800 (third from top), and after photometric uniformity, that is, λ = ∞ (bottom). Note that the dynamic range of the display reduces as the smoothing parameter increases. λ = ∞ is the special case of photometric uniformity where the display has a very low dynamic range. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 133 Fig. 8. The contrast sensitivity function (CSF) of the human eye. The top-most curve is for grating with brightness (mean) of 5-foot Lamberts. As the brightness of the grating decreases, the contrast sensitivity decreases as shown by the lower curves, for 0.5-, 0.05-, 0.005-, and 0.005-foot Lamberts (courtsey Valois and Valois [1990]). Fig. 9. This ﬁgure illustrates the complete algorithm. For the sake of simplicity, we have not included the black luminance functions and the offset maps in this ﬁgure. around projectors. Further, these solutions are rigid and expensive to maintain. The photometric unifor- mity method we developed to alleviate the problem, presented in Majumder and Stevens [2002, 2004], results in severe compression in the dynamic range of the display. In addition, the transfer functions are not addressed correctly and hence the method leads to washed out images. Figure 7 compares the results of the new method presented in this article, with one that achieves photometric uniformity. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 134 • A. Majumder and R. Stevens Fig. 10. Digital photographs of a display made of a 2 × 4 array of eight projectors (4 × 8 in size) . This display was corrected using the scalable version of our algorithm. The luminance functions of the left and right half of this eight-projector display (each made of 2 × 2 array of four projectors) were estimated from two different camera positions. They were then stitched together to create the display luminance functions. Top: Before correction. Bottom: After constrained gradient-based luminance smoothing. Further, our luminance smoothing does not introduce additional chrominance seams. Majumder [2003] shows that if the shape of the maximum luminance function across different channels are similar, chrominance seams are not possible. To assure a similar shape of the modiﬁed luminance functions across different channels, even after the application of our method, we apply the smoothing to the normalized luminance functions for each channel. Also, since a camera cannot provide us with, an absolute measurement of the luminance variations in the ﬁrst place, this does not affect the accuracy of the method in any way. Further, Equation (4) assumes view-independent or Lambertian displays. Since our screens are non- Lambertian in practice, our correction is accurate from the position of the camera used for reconstruct- ing the model parameters. However, our result looks seamless for a wide range of viewing angles and distances from the wall. We use a Jenmar screen in a back projection system that has a gain of approx- imately 2.0. For this display, we see no artifacts if the view direction makes an angle of about 20–90 degrees with the plane of the screen. Only for less than 20 degrees are the boundaries of the projectors visible—but as smooth edges, to some extent like the result of an edge-blending method. Figure 14 shows the effect. Typically, the black offset, B, is less than 0.4% of the maximum display luminance function, Wl , and has negligible effect on the smoothing maps. We implemented an alternate version of our system ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 135 Fig. 11. Digital photographs of a display made of a 3 × 5 array of ﬁfteen projectors (10-feet wide and 8-feet high). Top: Before correction. Bottom: Perceptual photometric seamlessness after applying constrained gradient-based luminance smoothing. assuming B = 0. Except for a slight increase in the dynamic range, the results were very similar to those produced by our implementation without this assumption. We have run interactive 3D applications using our real-time implementation. First, the 3D model is rendered to a texture of the size of each projector image. This step is essential for any geometry, and ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 136 • A. Majumder and R. Stevens Fig. 12. Digital photographs of displays made of a 2 × 2 and a 2 × 3 array of four (1.5 × 2.5 in size) and six (3 × 4 in size) projectors, respectively. Left: Before correction. Right: After constrained gradient-based luminance smoothing. Note that we are able to achieve perceptual seamlessness even for ﬂat colors (top), the most critical test image for our algorithm. Fig. 13. The ﬁfteen-projector tiled display before blending (left), after software blending (middle), and after optical blending using a physical mask (right). color corrections reduces the frame rate by about 25–40%. However, the luminance correction which is applied to this texture is independent of the model size and reduces the frame rate by a constant, about 10–20 frames-per-second. The details of the real-time implementation are available at Binns et al. [2002]. 8. CONCLUSION We have demonstrated that smoothing the photometric response of the display, based on a perceptual criteria (far less restrictive than strict photometric uniformity), is effective to achieve perceptual unifor- mity in multi-projector displays. We achieve such a smoothing by solving an optimization problem that minimizes the perceptible photometric variation and maximizes the dynamic range of the display. We have presented an efﬁcient, automatic, and scalable algorithm that solves this optimization to generate photometrically seamless (but not uniform) high-quality multi-projector displays. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 137 Fig. 14. The correction applied to a non-Lambertian screen and viewed from a view direction that makes an angle of less than 20 degrees with the screen. However, we believe that our work is just the ﬁrst step towards solving the more general problem of color seamlessness in multi-projector displays. Though we do not deal with chrominance explic- itly, if all projectors have identical red, green, and blue chromaticity, setting the red, green, and blue transfer functions to match across all projectors will balance the red, green, and blue mixture to give a common white point and grayscale. This is the best that can be done without considering gamut mapping techniques. However, we can envision devising a 5D optimization method that considers the gamut of each projector while smoothing the 5D color response (one parameter for luminance, two pa- rameters for chrominance, and two parameters for spatial coordinates). Such a method would address chrominance issues in practical displays where no two projectors usually have identical red, green, and blue chromaticity. Further, to enable different defense applications, self-calibrating systems that can correct themselves in real-time from arbitrary images projected on the display, need to be devised. Finally, we need to design a perceptual metric to evaluate the results of the color correction methods quantitatively. APPENDIX A. THE SMOOTHING ALGORITHM ∀(u, v), W (u, v) ← W(u, v) δ←1 λ for u = 0 to X − 1 for v = 0 to Y − 1 √ W (u, v) ← min(W (u, v), (1 + 2δ)W (u − 1, v − 1), (1 + δ)W (u − 1, v), (1 + δ)W (u, v − 1)) for u = X − 1 down to 0 for v = 0 to Y − 1 √ W (u, v) ← min(W (u, v), (1 + 2δ)W (u + 1, v − 1), (1 + δ)W (u + 1, v), (1 + δ)W (u, v − 1)) ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 138 • A. Majumder and R. Stevens for u = 0 to X − 1 for v = Y − 1 down to 0 √ W (u, v) ← min(W (u, v), (1 + 2δ)W (u − 1, v + 1), (1 + δ)W (u − 1, v), (1 + δ)W (u, v + 1)) for u = X − 1 down to 0 for v = Y − 1 to 0 √ W (u, v) ← min(W (u, v), (1 + 2δ)W (u + 1, v + 1), (1 + δ)W (u + 1, v), (1 + δ)W (u, v + 1)) ACKNOWLEDGMENTS We thank Sandy Irani of the Department of Computer Science at the University of California, Irvine, for helping us to ﬁnd an optimal dynamic programming solution to our optimization problem and proving its optimality. We thank David Jones, Matthew McCrory, and Michael E. Papka of Argonne National Laboratory for helping with the real-time implementation of our method on the multi-projector displays. We thank Mark Hereld of the Argonne National Laboratory; Gopi Meenakshisundaram of the Department of Computer Science at the University of California Irvine; Herman Towles, Henry Fuchs, Greg Welch, Anselmo Lastra, and Gary Bishop of Department of Computer Science, University of North Carolina at Chapel Hill, for several insightful discussions during the course of this work. REFERENCES BINNS, J., GILL, G., HERELD, M., JONES, D., JUDSON, I., LEGGETT, T., MAJUMDER, A., MCCROY, M., PAPKA, M. E., AND STEVENS, R. 2002. Applying geometry and color correction to tiled display walls (poster). IEEE Visualization. BUCK, I., HUMPHREYS, G., AND HANRAHAN, P. 2000. Tracking graphics state for networked rendering. Proceedings of Eurograph- ics/SIGGRAPH Workshop on Graphics Hardware, 87–95. CAZES, A., BRAUDAWAY, G., CHRISTENSEN, J., CORDES, M., DECAIN, D., LIEN, A., MINTZER, F., AND WRIGHT, S. L. 1999. On the color calibration of liquid crystal displays. SPIE Conference on Display Metrology, 154–161. CHEN, C. J. AND JOHNSON, M. 2001. Fundamentals of scalable high resolution seamlessly tiled projection system. Proceedings of SPIE Projection Displays VII 4294, 67–74. CHEN, H., SUKTHANKAR, R., WALLACE, G., AND LI, K. 2002. Scalable alignment of large-format multi-projector displays using camera homography trees. Proceedings of IEEE Visualization, 339–346. CHORLEY, R. AND LAYLOCK, J. 1981. Human factor consideration for the interface between electro-optical display and the human visual system. In Displays, Vol. 4. CRUZ-NEIRA, C., SANDIN, D. J., AND A. DEFANTI, T. 1993. Surround-screen projection-based virtual reality: The design and implementation of the CAVE. In Proceedings of ACM SIGGRAPH, 135–142. DEBEVEC, P. E. AND MALIK, J. 1997. Recovering high dynamic range radiance maps from photographs. Proceedings of ACM SIGGRAPH, 369–378. GOLDSTEIN, E. B. 2001. Sensation and Perception. Wadsworth Publishing Company. GONZALEZ, R. C. AND WOODS, R. E. 1992. Digital Image Processing. Addison Wesley. HERELD, M., JUDSON, I. R., AND STEVENS, R. 2002. Dottytoto: A measurement engine for aligning multi-projector display systems. Argonne National Laboratory preprint ANL/MCS-P958-0502. HUMPHREYS, G., BUCK, I., ELDRIDGE, M., AND HANRAHAN, P. 2000. Distributed rendering for scalable displays. Proceedings of IEEE Supercomputing. HUMPHREYS, G., ELDRIDGE, M., BUCK, I., STOLL, G., EVERETT, M., AND HANRAHAN, P. 2001. Wiregl: A scalable graphics system for clusters. Proceedings of ACM SIGGRAPH, 129–140. HUMPHREYS, G. AND HANRAHAN, P. 1999. A distributed graphics system for large tiled displays. In Proceedings of IEEE Visual- ization. 215–223. LAND, E. 1964. The retinex. Amer. Scient. 52, 2, 247–264. LAND, E. AND MCCANN, J. 1971. Lightness and retinex theory. J. Opti. Soc. Amer. 61, 1, 1–11. LARSON, G. W. 2001. Overcoming gamut and dynamic range limitation in digital images. SIGGRAPH Course Notes. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Photometric Seamlessness in Projection-Based Tiled Displays • 139 LI, K., CHEN, H., CHEN, Y., CLARK, D. W., COOK, P., DAMIANAKIS, S., ESSL, G., FINKELSTEIN, A., FUNKHOUSER, T., KLEIN, A., LIU, Z., PRAUN, E., SAMANTA, R., SHEDD, B., SINGH, J. P., TZANETAKIS, G., AND ZHENG, J. 2000. Early experiences and challenges in building and using a scalable display wall system. IEEE Comput. Graph. Applicat. 20, 4, 671–680. MAJUMDER, A. 2002. Properties of color variation across multi-projector displays. Proceedings of SID Eurodisplay, 807–810. MAJUMDER, A. 2003. A practical framework to achieve perceptually seamless multi-projector displays, phd thesis. Tech. rep., University of North Carolina at Chapel Hill. MAJUMDER, A., HE, Z., TOWLES, H., AND WELCH, G. 2000. Achieving color uniformity across multi-projector displays. Proceedings of IEEE Visualization, 117–124. MAJUMDER, A., JONES, D., MCCRORY, M., PAPKA, M. E., AND STEVENS, R. 2003. Using a camera to capture and correct spatial photometric variation in multi-projector displays. IEEE International Workshop on Projector-Camera Systems. MAJUMDER, A. AND STEVENS, R. 2002. LAM: Luminance attenuation map for photometric uniformity in projection based displays. Proceedings of ACM Virtual Reality and Software Technology, 147–154. MAJUMDER, A. AND STEVENS, R. 2004. Color nonuniformity in projection-based displays: Analysis and solutions. IEEE Trans. Visualiz. Comput. Graph. 10, 2 (March/April), 177–188. NAYAR, S. K., PERI, H., GROSSBERG, M. D., AND BELHUMEUR, P. N. 2003. IEEE International Workshop on Projector-Camera Systems. PAILTHORPE, B., BORDES, N., BLEHA, W., REINSCH, S., AND MORELAND, J. 2001. High-resolution display with uniform illumination. Proceedings Asia Display IDW, 1295–1298. RAIJ, A., GILL, G., MAJUMDER, A., TOWLES, H., AND FUCHS, H. 2003. Pixelﬂex2: A comprehensive, automatic, casually-aligned multi-projector display. IEEE International Workshop on Projector-Camera Systems. RASKAR, R. 1999. Immersive planar displays using roughly aligned projectors. In Proceedings of IEEE Virtual Reality 2000. 109–116. RASKAR, R., BROWN, M., YANG, R., CHEN, W., TOWLES, H., SEALES, B., AND FUCHS, H. 1999. Multi projector displays using camera based registration. Proceedings of IEEE Visualization, 161–168. RASKAR, R., WELCH, G., CUTTS, M., LAKE, A., STESIN, L., AND FUCHS, H. 1998. The ofﬁce of the future: A uniﬁed approach to image based modeling and spatially immersive display. In Proceedings of ACM SIGGRAPH, 168–176. SAMANTA, R., ZHENG, J., FUNKHOUSE, T., LI, K., AND SINGH, J. P. 1999. Load balancing for multi-projector rendering systems. In SIGGRAPH/Eurographics Workshop on Graphics Hardware. 107–116. STONE, M. C. 2001a. Color and brightness appearance issues in tiled displays. IEEE Comput. Graph. Applicat., 58–66. STONE, M. C. 2001b. Color balancing experimental projection displays. 9th IS&T/SID Color Imaging Conference, 342–347. VALOIS, R. L. D. AND VALOIS, K. K. D. 1990. Spatial Vision. Oxford University Press. YANG, R., GOTZ, D., HENSLEY, J., TOWLES, H., AND BROWN, M. S. 2001. Pixelﬂex: A reconﬁgurable multi-projector display system. Proceedings of IEEE Visualization, 167–174. Received May 2004; revised August 2004; accepted September 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects LIN SHI and YIZHOU YU University of Illinois at Urbana-Champaign This article addresses the problem of controlling the density and dynamics of smoke (a gas phenomenon) so that the synthetic appearance of the smoke (gas) resembles a still or moving object. Both the smoke region and the target object are represented as implicit functions. As a part of the target implicit function, a shape transformation is generated between an initial smoke region and the target object. In order to match the smoke surface with the target surface, we impose carefully designed velocity constraints on the smoke boundary during a dynamic ﬂuid simulation. The velocity constraints are derived from an iterative functional minimization procedure for shape matching. The dynamics of the smoke is formulated using a novel compressible ﬂuid model which can effectively absorb the discontinuities in the velocity ﬁeld caused by imposed velocity constraints while reproducing realistic smoke appearances. As a result, a smoke region can evolve into a regular object and follow the motion of the object, while maintaining its smoke appearance. Categories and Subject Descriptors: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling—Physically based modeling; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; I.6.8 [Simulation and Modeling]: Types of Simulation—Animation General Terms: Algorithms, Experimentation, Theory Additional Key Words and Phrases: Constrained animation, ﬂuid simulation, implicit functions, level sets, shape matching, shape transformations, velocity constraints 1. INTRODUCTION Amorphous but elegantly moving matters, such as clouds, fog, and smoke, give people plenty of space for imagination. We are excited when a cloud in the sunset sky assumes the approximate shape of an animal or some other real object. It is indeed an exhilarating event because of its rareness. For the same reason, ghosts and deities are usually described as manifesting themselves from smoke or clouds. A famous fairy tale has the following scene: As Aladdin rubbed the lamp to try to get a better look, the lamp came to life; the lamp launched a long, blue stream upward; the blue smoke rose toward the ceiling, and ﬁnally became an enormous, blue genie! We would like to develop techniques for digitally reproducing similar effects. Such techniques have many applications in the entertainment industry, especially in advertising and ﬁlm making. In some of the recent movies, there have been voxel water horses emerging from a ﬂooding river [Kapler 2002], and This work was supported by the National Science Foundation CCR-0132970 and start-up funds from the University of Illinois at Urbana Champaign. Authors’ address: Department of Computer Science, University of Illinois, 201 North Goodwin Avenue, Urbana, IL 61801; email: {linshi,yyz}@cs.uiuc.edu. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0140 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 140–164. Controllable Smoke Animation With Guiding Objects • 141 the mummy manifests itself from sand. Our goal in this article is to introduce methods that produce physically plausible motion for a gas phenomenon, which at the same time, assumes a recognizable static or dynamic shape. In the rest of the article, we choose smoke as a representative of such gas phenomena. Nevertheless, the approach introduced here is not only limited to smoke. Our goal in this article has the following implications: —the motion during two-way transitions between irregular smoke regions and regular object shapes should be natural and have realistic smoke appearances; —the global shape of the smoke should be able to approximate a static or moving object for an arbitrarily long period of time while maintaining its characteristic local structure and motion; —When smoke objects interact with each other or with the environment, the objects should exhibit the properties of smoke so that a strong wind or other regular objects can easily destroy the shape of such objects. An example with a smoke horse is shown in Figure 1(a) to illustrate this goal. Smoke consists of a collection of light-scattering tiny particles ﬂoating in the air. Creating dramatic effects is challenging since the smoke density in a ﬂuid medium always tends to drift from a nonuniform distribution to a uniform one. Solving the proposed problem requires the maximum level of control of this process while maintaining a believable appearance of smoke. When there is a conﬂict between controllability and physics rules, we choose to relax the physics rules since the desired effects can largely be considered a supernatural phenomenon. Our goal in this article is consistent with one of the general objectives of graphics research: the development of techniques that allow easy user-level control of the modeling and animation processes. This article presents an effective solution to the proposed problem. Our solution involves implicit functions (level sets) deﬁned for both smoke-object transitions and object motion. Since they are func- tions of both space and time, these level sets represent both the shape of the target objects as well as their evolution over time. These implicit functions serve as the underlying “storyboard” guiding the motion of the smoke. Thus, the problem becomes how to impose constraints on the motion of the smoke so that the smoke density distribution approximately matches these evolving level sets while the real- istic appearance of smoke is being maintained. This is actually a control problem that can be solved by a dynamic feedback process. The basic idea of our solution lies in the use of artiﬁcial feedback forces on the smoke so that subtle changes in the movement of the smoke reduces the shape discrepancy between the smoke and the target object. Such feedback forces are actually realized by velocity adjustments and constraints. They do not exist in the real world, and need to be carefully orchestrated to achieve the desired effects. Major contributions of this article include an overall framework for solving the proposed problem, an automatic scheme for target object matching based on velocity constraints imposed on the motion of the smoke, and an empirical compressible ﬂuid model for effectively integrating constraints into the velocity ﬁeld. These velocity constraints are derived from a shape matching functional. Simple but effective methods for smoke objects to interact with each other or the environment are also developed. 2. RELATED WORK The modeling of smoke and other gaseous phenomena has received much attention from the computer graphics community over the last two decades. Early models focused on a particular phenomenon and animated the smoke density directly without modeling its velocity [Gardner 1985; Perlin 1985; Perlin and Hoffert 1989; Ebert et al. 1998]. Additional details were added using solid textures whose parame- ters were animated over time. A common feature shared by these models and the voxel tool presented in Kapler [2002] is that they lack any dynamical feedback which is crucial to realistic animation. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 142 • L. Shi and Y. Yu Fig. 1. (a) An initial smoke blob evolves into a smoke horse. A wind blows the head away. When the wind recedes, the head grows back. (b) The underlying shape transformation for the smoke-horse transition. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 143 A more natural way to model the motion of smoke is to simulate the equations of ﬂuid dynamics. In recent work, relatively coarse grids were used in Foster and Metaxas [1997b] to produce nice smoke motion in three-dimensions. Their simulations are only stable if the time step is sufﬁciently small. To alleviate this problem and make the simulations more efﬁcient, Stam [1999] introduced a new model which is unconditionally stable and could be run at any speed in. This was achieved using a combination of a semi-Lagrangian advection scheme [Staniforth and Cote 1991] and implicit solvers. Vorticity conﬁnement and a higher-order interpolation technique were introduced in Fedkiw et al. [2001]. As a result, the simulations can keep ﬁner details on relatively coarse grids. Recently, this ﬂuid simulation method has been generalized to surfaces of arbitrary topology, including Catmull-Clark subdivision surfaces [Stam 2003] and triangle meshes [Shi and Yu 2004]. The level set method [Osher and Sethian 1988; Sethian 1999; Osher and Fedkiw 2001] is a robust computational method to track the surface of a volume of ﬂuid. In Foster and Fedkiw [2001], Foster and Fedkiw made important contributions to ﬂuid simulation and control through the introduction of a hybrid liquid volume model, combining implicit surfaces and massless marker particles. The work was further improved in terms of accuracy in Enright et al. [2002] by using particles on both sides of the interface between air and water. The level set method has also inspired interesting methods for fast surface reconstruction from unorganized points [Zhao et al. 2001], and geometric object editing [Museth et al. 2002]. What is important in these methods is the quality of the resulting static geometry instead of the dynamic surface evolution itself. In comparison, we are more concerned with the quality of the dynamics themselves for the realistic appearance of smoke. In terms of ﬂuid control, Stam [1996] applied multiple types of vector ﬁelds to control the global trajectories of ﬂuids. Foster and Metaxas [1997a] introduced embedded controllers that allow animators to specify and control a three-dimensional ﬂuid animation. In Foster and Fedkiw [2001] and Lamorlette and Foster [2002], animator-designed “fake” space curves and surfaces can be applied to control the motion and structures of ﬂuids. The tangents of the curves or the normals of the surfaces indicate the directions of motion. However, none of these attempts allows the user to enforce higher-level objectives, such as matching the shape of a still or moving object. A technique to make ﬂuids match regular shapes was ﬁrst reported in Yu and Shi [2002], which involves carefully designed force ﬁelds and a modiﬁed diffusion equation for smoke gathering. A similar approach that can produce results with improved visual quality was also taken by Fattal and Lischinski [2004]. Although being able to control the smoke density, the smoke gathering term in these techniques tends to generate “ghosting” effects, which means that the target shape looks like it is simply emerging from an amorphous cloud of smoke. In comparison, the level set method, taken by this article, can better control the smoke shape without introducing this type of artifact. A gradient-based iterative optimization approach for controlling ﬂuid simulations through user- speciﬁed objectives was introduced in Treuille et al. [2003] and McNamara et al. [2004]. The problem is cast as matching dynamically-evolved ﬂuid density with the speciﬁed density distributions at a set of keyframes. The elegant part of this approach is dynamically simulating the derivatives of the velocity ﬁeld in the same framework for simulating the velocity ﬁeld itself. However, since the derivatives of the velocity ﬁeld with respect to each control parameter need to be computed throughout a portion of an animation sequence, this approach is computationally expensive. The adjoint method was adopted in McNamara et al. [2004] to signiﬁcantly improve the efﬁciency of these derivative evaluations for each iteration. Since it is a gradient-based optimization, a number of iterations are still necessary before convergence, and there is also the peril of local minima. In comparison, the method introduced in this article is noniterative, and therefore, less expensive. In the aforementioned methods, once the ﬂuid density converges to the target shape, the ﬂuid either becomes static without dynamic appearances, or starts to evolve freely without any control. However, ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 144 • L. Shi and Y. Yu the method in this article can maintain interesting dynamic ﬂuid appearances around the target shapes for an arbitrarily long time. In addition, since we adopt distance transforms, our method is less likely to be trapped in local minima. Warping and morphing techniques [Wolberg 1990; Gomes et al. 1997] are very useful tools for trans- forming images and objects. Warping does not have a clear target image or object, while morphing usually produces transition between two objects. Sims [1992] introduced a technique for successively warping images using vector ﬁelds in. However, the results from the operation are not well controlled. 3D volume morphing methods [Hughes 1992; Lerios et al. 1995; Turk and O’Brien 1999; Alexa et al. 2000] can achieve well-behaved shape interpolation by considering both boundary and interior points. Actively moving implicit surfaces [Desbrun and Cani-Gascuel 1998] can be used to generate metamor- phosis between shapes without a correspondence. Most morphing techniques focus on the smoothness of the planned transition instead of its physical plausibility. In many experiments, we adopt the method in Turk and O’Brien [1999] to generate an underlying shape transformation guiding smoke-object shape transitions. 3. OVERVIEW In our method, we deﬁne an implicit function for the target object and try to drive an irregularly shaped smoke region so that a speciﬁc isosurface of the density distribution of the smoke closely matches the boundary (zero) level set of the target object. The implicit function for the target object is named the guiding implicit function and is denoted as D(x, t), where x represents a point in a 2D or 3D space, and the extra variable t means the target shape may move or deform over time. The density distribution of the simulated smoke is denoted as ρ(x, t). We use dynamic force feedback to actively inﬂuence the smoke isosurface so that the resulting dynamic surface, represented by ρ = {x|ρ(x, t)−τ = 0}, approximately matches the zero level set of the target object, D = {x|D(x, t) = 0}, where τ is a threshold used for deﬁning a boundary isosurface for the smoke region which is assumed to have internal densities higher than τ . Note that ρ(x, t) − τ = 0 is actually an implicit function for the smoke region. At every time step, given the shape discrepancy between the two isosurfaces, feedback forces are applied to the smoke boundary to reduce the amount of discrepancy. Since forces and accelerations are connected through Newton’s law, in practice, such feedback forces are actually realized by velocity adjustments. There are two essential stages in an animation where smoke evolves into an object. The ﬁrst stage involves a shape transition between the smoke region and the target object. In the second stage, the transformed smoke region needs to keep track of the object’s own nonrigid deformation or rigid-body motion. These two stages can be treated in a uniﬁed framework. The initial smoke region obtained by thresholding can be considered as an irregular object. 3D shape morphing techniques can be applied to generate a morph sequence between the initial smoke shape and the target object shape (Figure 1(b)). As a result, we can obtain an intermediate shape at any instance during the whole transition period. Thus, the shape transition between the smoke and the target object can be viewed as a nonrigid shape deformation, and any method designed for the second stage can be applied as well to this stage, where the morph sequence is used to generate the guiding implicit function which the smoke tries to match. Therefore, we only need to develop an approach for the smoke to track object motion. A static object is a special case for this problem. Nevertheless, tracking object motion using the smoke is by no means trivial. When there is tem- poral coherence and the frame-to-frame motion of the target object is small, it is possible to obtain velocity constraints on the boundary isosurface of the smoke by exploiting the local gradient ﬂow of the boundary. If a physical simulation of the smoke satisﬁes these boundary velocity constraints, the difference between the two boundary isosurfaces will be decreased. However, when there is fast ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 145 frame-to-frame motion, the target object from two consecutive frames might have a huge gap in posi- tion or orientation. Local gradient structure of the implicit functions cannot guarantee efﬁcient tracking results any more. When this happens, we use the target object to transport the smoke inside from its location in the previous frame, to its location in the current frame as if the target object is a con- tainer with a hard boundary. The hard boundary disappears once the smoke is in position again. Note that this step may not be physically realistic, and is specially designed to achieve the desired phenomenon. The overall solution to the proposed problem has the following components: —The animator needs to pick a target object and choose a smoke region from a simulation. A 3D morph sequence is created between the initial smoke shape and the target object. The target object for each frame is voxelized into a discrete implicit function. —For each frame, the system determines whether large motion has occurred by checking the amount of overlap between the two underlying object shapes at the previous and current frames. The strategy for smoke transportation is executed if the amount of overlap falls below a predeﬁned threshold. Small motion tracking is always performed, whether large motion occurred or not. —During each iteration, the system also simulates potential interactions among multiple smoke objects and interactions between a smoke object and its environment. The shape of a smoke region may be destroyed during such interactions, and a new morph sequence can be generated between the remaining smoke and the target object. 4. GUIDING OBJECTS The input to our system includes the target objects and their motion. When the objects are not static, the conﬁguration of the objects needs to be speciﬁed at every frame. The type of motion may include simple rigid-body motion, more complicated articulated body motion, or even nonrigid deformations. All target objects are internally represented as implicit functions. Our representation for the guiding implicit function D(x, t) is the signed distance function, which is zero on the object boundary and positive at the interior. The signed distance function has the advantage of conveniently providing the shortest distance between any point on the smoke boundary and the boundary of the guiding implicit function. Therefore, input objects need to be converted to this representation even if they are already given as implicit functions. Since smoke simulation will be performed on a volume grid, the smoke implicit function ρ(x, t) − τ is always directly represented on this discrete grid in the form of a density value at each voxel. The boundary isosurface of this function is obtained by labeling the voxels containing a density value close to the threshold τ . The signed distance function of a guiding object at a speciﬁc frame is also represented on the same volume grid by ﬁrst discretizing the object’s original representation, followed by a conversion to signed distance values. As discussed in Section 3, a part of the animation may involve a shape transition between the smoke region and the target object. An intermediate shape should be generated at each frame during this transition. This intermediate shape serves as the guiding object for that frame, and it needs to be represented as an implicit function as well. In practice, we apply 3D shape morphing and masking techniques (see Section 7.1) to generate a shape transition sequence which can produce such an in- termediate shape at any intermediate frame. These intermediate shapes should be represented as or converted to signed distance functions whenever they are needed. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 146 • L. Shi and Y. Yu 5. SHAPE MATCHING 5.1 Velocity Constraints for Small Motion Suppose at a certain time ti during simulation, the guiding implicit function is D(x, ti ). We would like to evolve the smoke density at the previous step, ρ(x, ti−1 ), so that the zero isosurface of the updated function, ρ(x, ti ) − τ , approximately matches the boundary of the guiding object. Since we are concerned with matching two isosurfaces, let us ﬁrst look at a criterion for measuring shape discrepancy. If we represent an object as a point set, two objects A and B exactly match each other if and only if both sets A − B and B − A are empty, which is also equivalent to that the volumes of both A − B and B − A are zero. The summed volumes of these two sets indicate the level of discrepancy between two shapes. Mathematically, we need to use characteristic functions of the shapes and integrals to represent these two volumes. Let us deﬁne χ D (x, ti ) = 1 if D(x, ti ) ≥ 0, and χ D (x, ti ) = 0, otherwise; and deﬁne χρ (x, ti−1 ) = 1 if ρ(x, ti−1 ) − τ ≥ 0, and χρ (x, ti−1 ) = 0, otherwise. The level of discrepancy between the two zero isosurfaces can be measured by the following integral, ev = χρ (x, ti−1 )(1 − χ D (x, ti ))d x + χ D (x, ti )(1 − χρ (x, ti−1 ))d x, (1) where the global minimum is zero and can be reached when the two characteristic functions coincide. Equation (1) can be simpliﬁed to χ D d x + χρ (1 − 2χ D )d x, where χρ is the shape variable, and χ D is the ﬁxed guiding shape for a speciﬁc frame since we would like ρ(x, ti−1 ) to approximate D(x, ti ). Therefore, reducing the shape discrepancy between the two is equivalent to minimizing the following functional, χρ (1 − 2χ D )d x. (2) According to calculus of variations [Gelfand and Fomin 1963], one can show that the ﬁrst variation of the integral in Equation (2), with respect to the smoke boundary surface, is simply based on the second part of its integrand and the normal directions of the smoke boundary. Therefore, the negative variational gradient minimizing the functional in Equation (2) with respect to the smoke boundary is as follows. δev ∇ρ = (1 − 2χ D ) , (3) δ ρ x=xb ∇ρ x=xb ∇ρ where ρ represents the smoke boundary, and ∇ρ represents the unit inward normal at the smoke boundary. Detailed discussion on the derivation of this variational gradient is presented in Appendix A. Consider a speciﬁc point xb on the boundary of the smoke, the sign of D(xb , ti ) indicates its location with respect to the guiding object. Note that the gradient of ρ(x, ti−1 ) at xb points to the interior of the smoke and is perpendicular to the boundary. Under the small motion assumption, the smoke and guiding object should have overlap. If xb is inside the guiding object, we need to move it slightly along the negative gradient direction; otherwise, move it along the positive gradient direction. Equation (3) means iteratively perturbing all the smoke boundary points simultaneously in this way would gradually decrease the costs in Equation (2) and Equation (1). Since a new smoke boundary is formed by the relocated points after each iteration, the gradient of the new boundary should be used for moving points in the subsequent iteration. This scheme for functional minimization resembles gradient descent for regular function minimization. As gradient descent, this scheme cannot always converge to the global minimum if the initial shape of the unknown is not sufﬁciently close to the target shape. For example, if the smoke region does not have overlap with the guiding object, this scheme would gradually shrink ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 147 the smoke region until it disappears and, at the same time, decrease the cost in Equation (1) to the volume of the guiding object which is actually a correct local minimum. In the current context, we can modify the above iterative minimization scheme to avoid the local minimum for two separate shapes. When D(xb , ti ) is negative, indicating xb is outside the guiding object, it should be moved in a direction in which the directional derivative of D(x, ti ) is positive to bring it closer to the boundary of the guiding object. Since this condition must be satisﬁed either by the positive or by the negative gradient direction of ρ(x, ti−1 ), we only need to choose the right one of these two instead of always following the positive gradient. We still keep the original scheme when xb is inside the guiding object. Since we would like to realistically evolve the boundary of ρ(x, ti−1 ) − τ into the target shape, the dynamic evolution should follow the above iterative procedure to minimize the integral in Equation (1), as well as follow the physics rules in smoke simulation as closely as possible. To achieve these goals, the smoke simulation should satisfy velocity constraints derived from the minimization procedure. Since Equation (3) shows a ﬁrst-order scheme, we should adopt its direction ﬁeld, but adjust its magnitude to improve stability. A velocity u on the smoke boundary can be decomposed into a normal component un and a tangential component ut . Based on Equation (3), the normal component un at a smoke boundary point xb , at time ti , is deﬁned to be ∇ρ un (xb , ti ) = Cn · min(d max , |D(xb , ti )|) · · msgn(xb , ti ), (4) ∇ρ where Cn is a constant scaling parameter, |D(xb , ti )| is the magnitude of the signed distance function, indicating how far away xb is from the current boundary of the guiding object, d max is a clamping upper bound for the distance, and msgn(xb , ti ) adjusts the direction of the velocity vector according to our modiﬁed scheme. Speciﬁcally, msgn(xb , ti ) = −1, if D(xb , ti ) > 0; msgn(xb , ti ) = sgn(∇ D · ∇ρ), if D(xb , ti ) ≤ 0. The incorporation of the term min(d max , |D(xb , ti )|) can alleviate overshooting when xb is already close to the boundary of the guiding object. However, this term should only be present in the normal component when the smoke region already has some overlap with the guiding object. Otherwise, it would reduce the velocity of the smoke to zero when it touches the boundary of the guiding object, and keep the smoke from entering the object. Various situations for setting up normal velocity constraints are summarized in Figure 2. Note that when the volume of the smoke differs from the object, these velocity constraints do not preserve mass. This is a tradeoff we need to make between physics and shape matching. Constraints on the tangential component are also crucial. Although the tangential component does not directly affect the shape of the level sets, it does affect the surrounding velocity ﬁeld. While a zero tangential component would make the smoke surface less alive, the numerical stability of the shape matching procedure may be compromised if it becomes overly large. The maximum allowable tangential component is actually dependent on the local geometry of the smoke surface. If the surface is ﬂat, the tangential displacement can be large without destroying the original shape while a highly curved surface is certainly more vulnerable. Therefore, the magnitude of the tangential component should have an upper bound related to the surface curvature. Since we would like to follow physical simulation as faithfully as possible, we can simply clamp the physically-generated tangential components against the upper bound when their magnitude becomes too large. Thus, the constrained tangential component is simply deﬁned to be Ct u∗ ut (xb , ti ) = min u∗ , t , (5) t K u∗ t ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 148 • L. Shi and Y. Yu Fig. 2. Normal velocity constraints on the smoke boundary for various situations. They are based on our revised minimization scheme for Equation (2). (a) The smoke region is completely outside the guiding object; (c) the smoke region partially overlaps with the guiding object; (b) & (d) a portion of the smoke boundary touches the boundary of the guiding object. The smoke velocity should not be affected in (b), while it should be reduced to zero in (d); (e) the smoke region is completely inside the guiding object; (f) the smoke region encloses the guiding object. where u∗ is the tangential component generated from a simulation, Ct is a constant parameter, and K is t the surface curvature. In practice, we use mean curvature. A robust implementation of mean curvature on a volume grid can be found in Museth et al. [2002]. Note that the normal and tangential velocity constraints are derived velocities for matching two shapes. They are different from those constraints deﬁned in Fedkiw et al. [2001] and Foster and Fedkiw [2001] for ﬂuid-object interaction and ﬂuid control. Our guiding objects are invisible “ghost” objects, and they do not directly interact with the smoke. 5.2 Velocity Constraints for Large Motion When the frame-to-frame motion of the guiding object becomes excessively large, the two instances of the guiding object at two consecutive frames may have little overlap, or no overlap at all. Although the matching scheme developed in the previous section can eventually converge, many iterations would be needed to actually reach convergence. For efﬁciency considerations, we directly transport the smoke from the location of the ﬁrst object instance to the second. During this direct transportation, the guiding object is assumed to have hard boundaries, and the part of the smoke that is already inside becomes trapped and moves together with the guiding object. Meanwhile, velocity constraints should still be imposed at the boundary and interior of the guiding object so that these constraints can bring along the surrounding region and generate turbulent ﬂows to create a ﬂuid appearance and “evidences” for the large motion. These constraints are not for transporting the smoke inside the guiding object. Suppose ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 149 a point on the guiding object moved from xi−1 to xi during two consecutive frames at time ti−1 and ti , a velocity constraint xi − xi−1 uL = ti − ti−1 should be imposed at xi−1 at time ti−1 . There is such a constraint for every voxel on and inside the guiding object at ti−1 . Note that the smoke transportation scheme in this section only accounts for large rigid-body motion and leaves nonrigid deformation to the velocity constraints in the previous section. The velocity constraints we have come up with need to be applied either at the boundary or interior of the smoke when the shape difference between the guiding object and the smoke region becomes sufﬁciently large. To make this statement more concrete, we are going to discuss in the following sections how to measure the shape difference and how to detect and evolve the smoke boundary. 5.3 Error Metrics We use two different error metrics for measuring the shape difference. The velocity constraints are only applied when the speciﬁc metric being used exceeds some threshold. The ﬁrst one is the volume discrepancy ev , deﬁned in Equation (1). The second metric is a generalization of the L p norm to object boundaries, 1/ p eL p = p D (x, t)d x , (6) ρ where p is positive, ρ is the boundary of the smoke, and D(x, t) is the distance function for the guiding object at time t. The latter two quantities have been deﬁned in Section 3. A special case of the L p norm is the L∞ norm, which is equivalent to e L∞ = max D(x, t). (7) x∈ ρ Although the velocity constraints were derived using the ﬁrst metric, both metrics have the same global minimum and can effectively reduce the shape difference. However, the dynamic behavior of the smoke under their respective control can be quite different, especially when p becomes large in the second metric. For instance, ev allows small portions of the smoke boundary to be far away from the guiding object as long as the integral remains small, while e L∞ keeps all the boundary points of the smoke within a certain distance from the guiding object. As a result, ev or e L p , with low p values, give rise to more realistic smoke appearances on the boundary and less clear object structures, while e L p , with high p values, becomes more appropriate when clear object structures are desirable. 5.4 Smoke Evolution and Boundary Detection The smoke implicit function φ = ρ − τ is evolved passively over time by the wind velocity ﬁeld {u}, which may be partially constrained. It can be easily shown [Osher and Sethian 1988] that the equation to update φ is as follows. φt + u · ∇φ = 0. (8) Except for the density threshold τ for boundary detection, this equation coincides with the advection equation for the smoke density in Fedkiw et al. [2001]. This equation can be solved either by the semi- Lagrangian method, or by the upwind scheme [Sethian 1999]. Note that both operate on the whole voxel grid instead of boundary voxels only. We have implemented both methods. Both of them can ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 150 • L. Shi and Y. Yu produce visually realistic results. Since a smoke boundary tends to evolve relatively slowly, the choice of a numerical method is not very critical. Our normal and tangential velocity constraints are positioned at the boundary of the smoke region. Enforcing these constraints requires the detection of the smoke boundary at every time step. This can be easily achieved since this smoke boundary is just the zero isosurface of the smoke implicit function. 6. SMOKE SIMULATION FOR CONSTRAINED VELOCITIES Since we exploit velocity constraints to achieve shape matching, a smoke simulation framework that can effectively incorporate hard-wired velocity constraints without producing visual discontinuities is desirable. Before we introduce our revised model for smoke simulation, let us ﬁrst have a look at the already adopted mathematical formulations for ﬂuid simulation in the graphics literature [Foster and Metaxas 1997b; Stam 1999; Yngve et al. 2000; Fedkiw et al. 2001]. 6.1 Previous Formulations The dynamics of a compressible ﬂuid can be modeled using the following Navier-Stokes equations: ∂ρ f = −∇ · (ρu), (9) ∂t ∂u ν ρf = −ρ f (u · ∇)u + ν∇ 2 u + H(u) + ρ f f − ∇ P, (10) ∂t 3 where ρ f is the density of the ﬂuid which is different from the smoke density, u represents the velocity ﬁeld, P is the pressure term, f is an external force ﬁeld, and ν is the kinematic viscosity of the ﬂuid. The x component of H(u) is ∇ · ( ∂u ), and the y and z components are deﬁned similarly. The ﬁrst equation ∂x arises from the conservation of mass, but the volume of the ﬂuid is compressible with increasing density for a decreasing volume. The second equation is for the conservation of momentum; the ﬁrst term on the right hand side is the convective term; the second and third terms model accelerations due to viscous forces; the last two terms model accelerations due to external forces and forces arising from the pressure gradient. These equations can effectively model high-speed velocity ﬁelds such as shock waves generated by explosions [Yngve et al. 2000]. However, a strict time step condition is necessary for stable numerical solutions. For liquids and low-speed gaseous phenomena, the compressibility effects are negligible, and conservation of mass becomes equivalent to conservation of volume. The assumption of incompressibility leads to the following equations which make more efﬁcient numerical methods possible. ∇ · u = 0, (11) ∂u 1 , = −(u · ∇)u − ∇ P + ν∇ 2 u + f, (12) ∂t ρf where the ﬁrst equation means the ﬂuid is volume-preserving, and the second one is similar to the corresponding equation in the aforementioned model for compressible ﬂuids. In Stam [1999], Stam de- veloped an unconditionally stable numerical method for these equations by adopting semi-Lagrangian tracing and the Helmholtz-Hodge Decomposition. If we focus on gases, the effects of viscosity are also negligible. Therefore, the diffusion terms in the above models can be left out. Simulating gaseous phenomena thus reduces to solving the following incompressible and inviscid Euler equations [Fedkiw et al. 2001]: ∇ · u = 0, (13) ∂u = −(u · ∇)u − ∇ P + f, (14) ∂t ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 151 where the factor ρ1f in Equation (12) has been integrated into the “pressure” P which is used to guarantee zero divergence, but does not represent actual pressure anymore. 6.2 An Empirical Equation for Compressible Gases The basic framework in this article is a dynamic feedback system where the velocity constraints deﬁned in Section 5 are dynamically updated every time step, according to the shape discrepancy between the smoke region and the underlying guiding object. The purpose of such velocity constraints is to reduce the amount of discrepancy. These velocity constraints have poor spatial and temporal coherence because constraints at spatially or temporally adjacent voxels may be quite different. In addition, the constraints are only imposed at the boundary of the smoke region which is essetially a thin layer in the simulation volume. Since these artiﬁcal velocity constraints do not exist in the real world, we argue that existing ﬂuid simulation methods cannot incorporate them without introducing artifacts. First, the incompressible scheme in Equations (13) and (14) does not work well. If we impose veloc- ity constraints which generate discontinuities in the velocity ﬁeld, concentrated high pressure tends to appear rapidly in regions nearby to guarantee zero divergence. Those sudden high pressures further in- ﬂuence surrounding velocities and generate temporal discontinuities and visual artifacts. Such artifacts have been observed in our experiments. Second, existing compressible schemes would not ﬁt our pur- pose very well, either. Simulating true compressibility is very expensive and requires a strict time step condition. On the other hand, an unphysical scheme called “artiﬁcial compressibility” was introduced in Chorin [1967]. It allows a certain degree of compressibility during a transition period and converges to incompressible ﬂuid simulation when a steady solution has been reached. However, this convergence is achieved over time, instead of within each time step, and is only guaranteed when external forces are absent. In our simulation, incoherent external forces are exerted constantly at every time step since we need to update the velocity constraints. Thus, the scheme in Chorin [1967] may not converge. In addition, the sound speed used in this scheme needs to be unusually small to prevent temporal discon- tinuities. An unreasonable sound speed further compromises the realism in the simulation results. The nature of our control scheme demands a new ﬂuid simulation technique which does not have to be physically accurate, but needs to produce results that are visually appealing. What we need is a simulation technique that approximates the behavior of incompressible ﬂuids while absorbing discontinuities where velocity constraints are present. We would also like the technique to be stable and efﬁcient. To achieve our goals, we propose an empirical scheme that does not strictly enforce incompressibility. Recall that the pressure P in Equation (14) can be estimated numerically using the Poisson equation [Stam 1999; Fedkiw et al. 2001] 1 ∇2 P = ∇ · u, (15) t where t is the size of the simulation time step. It has been shown in Stam [1999] that imposing a feedback force ﬁeld −∇ P to the velocity ﬁeld strictly enforces zero divergence. Our technique ﬁrst decomposes the pressure ﬁeld into two components, −µ t∇ 2 P and P = P + µ t∇ 2 P , where µ is a constant factor with units of area divided by time, and the negative sign in front of the ﬁrst component is due to the fact that a local maximum of P is typically a local minimum of ∇ 2 P . Note that the ﬁrst component is the negative Laplacian of the pressure ﬁeld, and the second one is a blurred version of the original pressure. This decomposition of the pressure ﬁeld, in turn, splits the feedback force ﬁeld into two components, µ∇( t∇ 2 P ) and −∇ P − µ∇( t∇ 2 P ). To allow a certain degree of compressibility, especially in the regions with velocity constraints, our scheme only applies the ﬁrst force component to the velocity ﬁeld immediately, while buffering the second one for later time steps. Thus, the discretized ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 152 • L. Shi and Y. Yu version of Equation (14) changes to u(t + t) − u(t) = −(u · ∇)u + µ∇( t∇ 2 P ) + f. (16) t Because of Equation (15), t∇ 2 P in Equation (16) can be replaced with ∇ · u. If we further replace the left-hand side of Equation (16) with a continuous partial derivative of the velocity ﬁeld, we arrive at the following single partial differential equation that is capable of simulating compressible gaseous phenomena: ∂u = −(u · ∇)u + µ∇(∇ · u) + f, (17) ∂t where 0 ≤ µ < ∞, with a typical value between 0 and 1. It is used to adjust the magnitude of the feedback force from the pressure ﬁeld to the velocity ﬁeld. Since our scheme only uses a ﬁltered version of the original pressure every time step, the feedback force from the pressure to the velocity ﬁeld is weakened. Nevertheless, this does not mean a portion of the pressure simply disappears, but mean that the release of the energy preserved in the rest of the pressure is delayed. In Appendix B, we show that, for a bounded workspace, without sources and sinks, ∇(∇ · u) = 0 everywhere is equivalent to ∇ · u = 0 everywhere. This result indicates that our new formulation of the feedback force from the pressure to the velocity ﬁeld can eventually reduce the divergence of the velocity to zero when there are no external forces. In addition, our formulation allows large time steps and does not involve a sound speed. While other decompositions are possible, the Laplacian operator in our pressure decomposition makes it particularly convenient to robustly obtain the components through a diffusion process, which will be discussed in Section 7.2. In practice, we have found that our formulation can effectively reproduce realistic ﬂuid motion as well as incorporate frequently inserted velocity constraints, without generating obvious visual discon- tinuities (see Section 8.1). 7. IMPLEMENTATION 7.1 Shape Transition 7.1.1 Variational Shape Morphing and Interpolation. As our ﬁrst option, we apply the shape trans- formation method introduced in Turk and O’Brien [1999] to generate a morph sequence between the shape of a smoke region and a target object. This method represents the whole morph sequence as a variational implicit function deﬁned in a space of n + 1 dimensions. This implicit function, which is based on radial basis functions, interpolates the source and target shapes deﬁned in an n-dimensional space and handles topological shape changes automatically. The extra dimension is aligned with the temporal axis so that the source and target shapes are n-dimensional slices of the implicit function at time zero and one, respectively. In our situation, we apply this method to obtain an interpolating 4D function (r, s) such that (r, 0) and (r, 1) reproduce the starting smoke shape and the target object, respectively. By ﬁxing s to a value between 0 and 1, (r, s) represents the analytic from of an inter- mediate shape. In practice, we found that the rate of shape transformation usually was not uniform and the target shape started to loom only when the time became very close to one, if the target shape was complicated and the parameter s was scheduled linearly with respect to the actual time scale. Therefore, we decided to warp the temporal axis and schedule s as a piecewise linear function of the actual time t used for an animation. The local slope of the piecewise linear function is used to adjust the transformation rate and make it perceptually more uniform. Better morph sequences can be obtained when the source and target objects are well aligned in terms of position and orientation. The amount of relative translation and rotation should then be uniformly distributed back across the frames of the ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 153 resulting sequence so that there is simultaneous deformation, translation, and rotation from frame-to- frame. It is desirable for the intermediate shapes to have an approximately constant volume since a gas usually does not signiﬁcantly change volume even when it is compressible. Therefore, we verify the volume at each intermediate frame, and apply morphological operations on the voxel grid to shrink or expand the shape. The same variational implicit functions for shape transformation can be used for surface interpolation [Carr et al. 2001; Turk and O’Brien 2002]. Therefore, they can be used for generating an analytic implicit function to approximate a polygonal mesh model by interpolating the vertices of the mesh. The interpolating implicit function and the polygonal mesh have the same dimensionality. Guiding objects originally represented as polygonal meshes are converted into implicit functions in this way. 7.1.2 Shape Masking. Shape masking is an easier-to-implement alternative for generating shape transitions. In this scheme, the target object can be simply a ﬁxed shape undergoing rigid-body motion. We also require an additional masking shape, M (x, t), whose scale, position, and orientation can all vary with time. The guiding shape at each time step is deﬁned to be the intersection between the target and masking objects. For example, the masking object can be a moving sphere with changing radius. At the beginning of a transition, the sphere has a tiny radius and does not have any overlap with the target object. The sphere then moves closer to the target object while increasing its radius so that the amount of overlap becomes larger. At the end, the sphere encompasses the target object and the transition is complete. One caveat with this scheme is that the volume of the guiding shape changes with time, which requires an increasing amount of smoke during the transition. 7.2 Numerical Smoke Simulation We have implemented numerical solutions for two formulations, our new compressible ﬂuid formula- tion and the incompressible and inviscid formulation in Fedkiw et al. [2001]. We also implemented enhancements that allow the integration of our velocity constraints. The details of these enhancements will be introduced in the next section. The implementation of the incompressible scheme consists of three basic steps [Fedkiw et al. 2001]: compute an intermediate ﬂuid velocity ﬁeld, {u∗ }, from Equation (14), ignoring the pressure term by ﬁrst adding external force times the time step, and then solving the advection part, (u · ∇)u, by using semi-Lagrangian tracing [Staniforth and Cote 1991]; obtain the pressure P by solving the Poisson equation, 1 ∇2 P = ∇ · u∗ ; (18) t where t is the size of the time step; ﬁnally, subtract the gradient of P from the intermediate velocity, u = u∗ − t∇ P. (19) To simulate our compressible ﬂuid formulation, we still keep the ﬁrst step unchanged, but revise the other two. Although our formulation does not involve pressure, directly solving Equation (17) may not be stable. Introducing pressure during the numerical process improves stability. Therefore, we actually solve Equations (15) and (16) instead. In the above steps, after obtaining the pressure by solving the Poisson equation, we apply a diffusion process, ∂P = µ∇ 2 P, (20) ∂t ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 154 • L. Shi and Y. Yu and solve for the new pressure, P , using a stable implicit method, P −P = µ∇ 2 P , (21) t where the Laplacian of the new pressure ∇ 2 P , instead of ∇ 2 P , appears on the right-hand side. The new pressure can be obtained by solving the sparse linear system arising from discretizing this equation. Finally, the intermediate velocity is updated as follows, u = u∗ + µ t∇(∇ 2 P ). (22) Note that solving the diffusion equation to obtain the new pressure is an extra step we performed during smoke simulation. We discretize the workspace into a ﬁnite volume grid. Keep all vector components on the faces of the voxels, and retain all the scalar ﬁelds at the center of each voxel. The velocity at any point inside a voxel can be obtained by linearly interpolating each component of the velocity vector separately. Additional implementation details about discretization and estimation of ﬁnite differences can be found in Fedkiw et al. [2001] and Stam [1999]. Our overall numerical solution involves multiple (around three) time steps between two consecutive frames. Small and large object motion tracking are carried out in separate time steps. There is at most one optional time step dealing with large motion between two frames. And there may be one or more time steps for small motion tracking since multiple iterations may be necessary to achieve good boundary matching. In the following, we introduce the details of these time steps in the context of the numerical solution for the incompressible formulation in Fedkiw et al. [2001]. It is straightforward to modify these details for our compressible ﬂuid formulation. A time step for small motion tracking has the following substeps: —Compute the intermediate ﬂuid velocity ﬁeld {u∗ }, detect the boundary of the smoke region using a density threshold, and set up velocity constraints at the boundary by modifying {u∗ }. —Solve the ﬁnal ﬂuid velocity ﬁeld using Equations (18) and (19), while satisfying the constraints. —Use the obtained velocity ﬁeld to evolve the smoke density distribution applying the semi-Lagrangian method. The optional time step for large motion between two frames is executed before the other time steps. It has the following substeps: —Obtain the intermediate ﬂuid velocity ﬁeld and set up velocity constraints for large motion. —Solve the ﬁnal ﬂuid velocity ﬁeld while satisfying the constraints. —Use the obtained velocity ﬁeld to evolve the smoke outside the guiding object. Use the guiding object to transport the smoke inside, and the transported smoke density overwrites existing density at each destination voxel. If we choose to apply our compressible ﬂuid formulation, the second substeps in both types of time steps should solve Equations (18), (21), and (22) instead. It is straightforward to apply additional vorticity conﬁnement [Fedkiw et al. 2001] in the second substeps as well since these substeps are basically the major part of a single forward simulation step. In our experiments, we always apply vorticity conﬁnement. 7.3 Enforcing Velocity Constraints Most of the time, we only set up and enforce velocity constraints at a selected subset of the voxels on the smoke boundary for small motion tracking. This process is initiated once the adopted error metric ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 155 (see Section 5.3) exceeds a prescribed threshold. A larger threshold produces more lively smoke motion, while a smaller threshold shows the object boundary more clearly. The voxels on the smoke boundary are sorted according to their distance to the guiding object. Voxels with larger distance are assigned higher priority. Velocity constraints are assigned to the boundary voxels with highest priority ﬁrst. If the shape discrepancy does not drop signiﬁcantly, additional voxels with the next level of priority become constrained. This process continues until the error drops below the threshold, or most of the boundary voxels are constrained. Once the error becomes sufﬁciently small, constrained voxels get released gradually in the reversed order. The desired velocity constraints are imposed after the computation of the intermediate velocity ﬁeld {u∗ }. Solving the Poisson equation may alter the velocities in these constraints and make it difﬁcult for the smoke to converge to the target shape if these constraints are not explicitly enforced. Enforcing these constraints means the constrained velocities should not be affected during this step. Equation (19) indicates that ∇ P should be zero at the constrained voxels. Therefore, the pressure p needs to satisfy such gradient constraints in addition to the Poisson equation. Using ﬁnite differences to discretize both left- and right-hand sides of Equation (18), we obtain a sparse linear system of equations, AP = b, where P is the vector of unknown pressures, and A is the coefﬁcient matrix with a sparse structure. A discrete gradient estimation at a voxel involves three or six of its direct neighbors, depending on whether central differences are being used. For symmetry consideration, when ∇ P = 0 at voxel (i, j, k), we impose that P (i, j, k) should equal the pressure at its six direct neighbors. For every equation involving P (i, j, k) in the linear system, we need to replace P (i, j, k) with the pressure at one of its direct neighbors. Eventually, we can eliminate P (i, j, k) from all the equations. Changes like this result in a linear system with a reduced number of unknown variables. Preconditioned conjugate gradient (PCG) with Incomplete Cholesky Factorization (ICF) [Golub and Loan 1996] is a natural choice to solve this sparse linear system. Since the smoke boundary surface is typically a closed surface, we avoid strictly enforcing boundary conditions everywhere on the surface because that would result in a singular matrix A if the volume inside the smoke region is not preserved. When it is necessary to cover the whole smoke boundary with constraints, we reduce the density of the constraints by setting a small distance threshold d t . Any voxels within that distance from a constrained voxel are not allowed to be constrained. We typically choose d t to be between 1 and 3 voxels. Alternatively, Dirichlet boundary conditions can be adopted by directly specifying pressure values on the smoke boundary. For example, one can force voxel (i, j, k) and all its direct neighbors to have zero pressure. 7.4 Smoke Transportation for Large Motion We assume that the frame-to-frame motion of the guiding object can be decomposed into a rigid-body transformation and a nonrigid deformation. Smoke transportation only handles large rigid transforma- tion and leaves small rigid transformation, as well as additional nonrigid deformation, to small motion tracking. To move the smoke to a new position and orientation speciﬁed by the rigid transformation, we apply backward mapping with a modiﬁed trilinear interpolation scheme. A voxel inside the transformed object takes the smoke density at a corresponding position in the original object. The density at the corresponding position should be interpolated only from the densities at those surrounding voxels that are also inside the original object to preserve the object boundary. 7.5 Thin Parts It is hard for the smoke surface to reach some of the thin parts of the guiding object. That is because the gradient and curvature estimations are inaccurate at these locations due to insufﬁcient sampling. We explicitly label those voxels that should have smoke but actually do not after a long time. If there is ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 156 • L. Shi and Y. Yu Fig. 3. A comparison between two smoke formulations. The ﬁrst row is generated by a model-enforcing incompressibility. The second row is generated by our model with compressibility. They are visually similar, which indicates our model can also produce realistic smoke appearances. Grid size 200 × 200 × 200. a smoke-ﬁlled voxel which is adjacent to one of those tagged voxels, we set a velocity constraint at the ﬁrst voxel and the velocity points to the tagged voxel. Thus, smoke can be propagated gradually into the thin parts. 7.6 Time Complexity Since we enforce shape matching by embedding velocity constraints into a conventional smoke simu- lation, the time complexity of our approach is slightly higher than the latter. The extra work at each time step includes obtaining the signed distance function of the guiding object and solving the diffusion equation for the pressure, among others. The complexity of the distance transform is O(n log m), where n is the number of voxels in the grid and m is the maximum number of voxels on the propagating front [Sethian 1999]. In the worst case, m = O(n), which usually does not occur unless the object boundary becomes close to a fractal surface. In all our experiments, the actual running time of the distance trans- form is less than an original smoke simulation step without constraints. The complexity of solving the sparse linear system arising from the diffusion equation is on the same order as that of solving the Poisson equation which is part of the original simulation. Solving these two equations is the most time- consuming step that occupies approximately 60% of the total running time. The complexity of all the other extra work is linearly proportional to the size of the voxel grid. Therefore, in practice, our complete algorithm maintains the same order of magnitude of the original complexity of smoke simulation. 8. EXPERIMENTAL RESULTS We have successfully tested our complete algorithm in a variety of examples. Some of the images from these examples are shown in Figure 1 and Figures 4–8. The target objects in these examples include both free-form objects and letters. Most of the target objects are converted to implicit functions from triangular meshes. If there are multiple objects in the same example, smoke simulation was performed on all of them simultaneously. Most of the simulations have been ﬁnished on an AMD 2100+ processor. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 157 Fig. 4. Two heart-shaped smoke objects collide and merge into one. Grid size 128 × 128 × 128. Fig. 5. Large motion tracking for a synthetic character. Constraints for large motion induce a velocity ﬁeld for the environment. There are a few sources on the ﬂoor to produce smoke for the environment. Grid size 128 × 128 × 128. A voxel grid of size between 64 × 64 × 64 and 200 × 200 × 200 has been adopted for the simulations. The examples took 15 seconds/frame on the lower end of the grid resolution and 60 seconds/frame on the higher end. The ﬁnal images were rendered by ray-tracing the smoke volume density distribution [Kajiya and von Herzen 1984]. The density threshold τ for smoke boundary detection is interactively determined at the beginning of the smoke-object shape transition and ﬁxed throughout a whole simulation. We have found that typically this threshold falls between 0.1 and 0.5. In the following, we specify other parameters used during experiments for a normalized grid with one unit length for each of its dimensions. In Equation (4), we typically set d max between 0.05 and 0.1, and set Cn around 25. In Equation (5), we set Ct around 2. These parameters were chosen with the assumption that the maximum velocity of the smoke is around 1 unit-length per-second. Obviously, by adjusting the maximum velocity, we can change the overall pace of the smoke. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 158 • L. Shi and Y. Yu Fig. 6. (a)-(c) A smoke cow generated by our method; (d)-(f) a smoke cow generated by the hard boundary scheme. Grid size 128 × 128 × 128. Fig. 7. The smoke switches back and forth between the shapes of a “check” and an “X”. Only one cycle is shown here. Grid size 128 × 128 × 128. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 159 Fig. 8. (a)-(d) Smoke rises to form the shape of four letters. The shapes start to disappear once the velocity constraints are released in (d). Grid size 128 × 128 × 128. 8.1 Validation of the Compressible Fluid Model We have validated our empirical ﬂuid model by visually comparing the results from our model with results from the model in Fedkiw et al. [2001]. We have performed two different types of comparisons. The ﬁrst set of comparisons are between smoke simulations without velocity constraints. They are meant to verify whether our model can be used for a general smoke simulation as opposed to the special application in this article. As a result, the visual quality of the smoke sequences generated by our model is comparable to those in Fedkiw et al. [2001]. It can produce realistic smoke appearances with rolling effects. Figure 3 shows one comparison in this category. The second type of comparisons are between simulations with our derived velocity constraints. We have conﬁrmed from these comparisons that our new model can integrate these velocity con- straints better with fewer visual artifacts. Introducing our velocity constraints into a model enforc- ing incompressibility would generate obvious artifacts since the nonzero divergence caused by the constraints inﬂuence surrounding velocities immediately. The reason our model can perform better is that it can absorb velocity discontinuities into pressure and release the energy in the pressure gradually. 8.2 Interaction with the Environment The smoke objects should interact with the environment like smoke. When there is a sufﬁciently strong wind, the animator can choose to release the velocity constraints in the inﬂuence region of the wind and let the smoke move freely. When there is an intervening real object, the animator can release the original velocity constraints in the contact region and let the smoke be controlled by the boundary conditions on the real object surface as in Fedkiw et al. [2001]. In the example shown in Figure 1(a), a strong wind blows away the head of a smoke horse. However, the head grows back when the wind recedes. This is because we keep the underlying target object unchanged, and the target object induces ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 160 • L. Shi and Y. Yu velocity constraints pointing towards the head region. In such examples, the animator needs to specify the spatial region where constraints should be released as well as the starting time and duration. In Figure 1(a), the user simply places a partitioning plane at the bottom of the horse’s neck. Constraints on the same side of the plane as the head are released for a short period of time. To preserve temporal coherence, the number of constraints can be decreased or increased gradually to produce a smooth transition. 8.3 Interaction between Smoke Objects When two smoke objects collide, their density distributions overlap and they should naturally merge into a larger smoke region since they do not have hard boundaries. Other scalar ﬁelds, such as temperature and color, can also be advected by semi-Lagrangian tracing and blended together in the same way as the density. The underlying guiding objects should either be merged or replaced with a new one which may require a new shape transition. Velocity constraints should only be imposed on the new boundary afterwards. In the example shown in Figure 4, two heart-shaped smoke objects merge into a larger one and their original colors get advected and blended together. The heart-shaped target object is deﬁned by a closed-form implicit function: (2x 2 + y 2 − z 2 − 1)3 − 0.1x 2 z 3 − y 2 z 3 = 0. 8.4 Large Motion To demonstrate large motion tracking, we use a synthetic character equipped with motion-captured data as the underlying guiding object. The limbs of this articulated character have large frame-to-frame motion so that a small amount of smoke escapes. We also put a few smoke sources on the ﬂoor. The results from this example are shown in Figure 5. In general, smoke does not move very fast. Its appearance would look less natural if we force the smoke to follow fast motion. Nevertheless, it can be clearly seen that the velocity constraints induced by the fast motion of the limbs creates an interesting velocity ﬁeld for the rest of the environment and the smoke around the character follows this velocity ﬁeld. 8.5 Tradeoff between Control and Smoke Appearance The number of velocity constraints on the smoke boundary is the most important factor affecting the realism of the smoke simulation because our velocity constraints are kinematic constraints that do not involve dynamics. The shape of the smoke region is free to evolve without any constraints, but is guaranteed to match the guiding object with dense constraints. Thus, we can achieve various levels of tradeoff between control and smoke appearance by changing the number of constraints. However, the user does not have direct control over the number of constraints which is automatically adjusted at each time step according to the error metrics, the error thresholds, and other parameters, as discussed in Section 5.3 and 7.3. There are no velocity constraints at all if the shape discrepancy is below the error threshold(s); otherwise, the number of constraints is increasing, and realism is partially given up for control. Therefore, in general, larger error thresholds allow more realistic smoke appearance, but less object structures. The parameter µ in Equation (17) can also affect the liveliness of the smoke. Larger µ’s provide stronger force feedback from the pressure, and typically lead to more lively motion. Figure 9 gives a comparison of the visual quality generated by different combinations of error metrics, thresholds, and µ. In this experiment, we chose to use both ev and e L∞ simultaneously. For ev , the threshold is set between 0% and 5% of the volume of the target object. For e L∞ , the error threshold is set between 0 and 4 voxels in a 128 × 128 × 128 grid. µ is set between 0.1 and 0.3. 8.6 Comparisons with other Control Schemes A simple solution to our problem is to perform smoke simulation in the volume enclosed by the boundary surface of the guiding implicit function. Obviously, the smoke is forced to follow the object when it moves. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 161 Fig. 9. A comparison of the visual quality of the smoke objects by different combinations of error metrics, thresholds, and the pressure diffusion parameter. (a) The error thresholds ev = 0%, e L∞ = 0 voxels, and the diffusion parameter µ = 0.1; (b) ev = 3%, e L∞ = 2.5 voxels, and µ = 0.1; (c) ev = 5%, e L∞ = 4 voxels, and µ = 0.3. Grid size 128 × 128 × 128. This solution is similar to our scheme for large motion, but the hard boundary used for large motion disappears once the smoke has been moved to the target location. There are a few limitations with this solution using a hard boundary. Even though the boundary surface is not visualized during ﬁnal rendering, it will still be obvious since the smoke stops or reﬂects at the boundary. A smooth, hard boundary for the smoke is not a realistic phenomenon. The second problem is that the smoke tends ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 162 • L. Shi and Y. Yu to be stationary, without interesting motion, inside a closed volume with a limited size. A side-by-side comparison is made between this simple scheme and our method in Figure 6. In the results generated by our method, the smoke close to the boundary of the target object still has natural motion and advection. Our control scheme also has advantages and differences if compared to the method in Treuille et al. [2003] and McNamara et al. [2004]. First, our scheme is more efﬁcient by introducing velocity con- straints instead of evaluating derivatives of the velocity ﬁeld with respect to all control parameters. The time complexity of our method is on the same order of magnitude as a single smoke simulation without shape matching. For example, Figure 7 shows an example where the smoke switches back and forth twice between the shapes of a “check” and an “X” on a 128 × 128 × 128 grid. It took only 4 hours 15 minutes to generate the 1250-frame sequence. Second, our scheme can maintain dynamic smoke appearances around a shape for an arbitrarily long time. Nevertheless, if we release the velocity con- straints, the smoke becomes free to evolve (Figure 8(d)). As shown in Figure 1(a), we can also release some of the constraints, and then regain control as we wish. A notable difference is that our algorithm is targeted at object shapes, while the method in Treuille et al. [2003] is designed for arbitrary density distributions at keyframes. A video including the aforementioned examples may be found at: http://www-sal.cs.uiuc.edu/~yyz/movie/smoke_obj.avi 9. CONCLUSIONS AND FUTURE WORK We have presented a novel technique to control the density and dynamics of smoke (a gas phenomenon) so that the synthetic appearance of the smoke (gas) resembles a still or moving object. The main focus has been on controllability and appearance. In order to match the smoke surface with the target sur- face, we represent both smoke and objects as implicit functions and impose carefully designed velocity constraints derived from a shape matching functional. An empirical compressible ﬂuid model has been introduced for effectively integrating constraints into the velocity ﬁeld while maintaining realistic ﬂuid appearances. The overall framework represents a signiﬁcant advance over previous methods for con- trolling ﬂuids. The implementation of our system is built upon recent advances in smoke simulation and shape transformation. We would like to extend this work to 2D images and video objects. The difference between a 3D object and a 2D image is that an object is a binary function, while an image has multiple levels of intensities or colors. Matching the smoke boundary with the object boundary should be extended to matching the level sets of smoke density with the level sets of image intensities or colors. Nevertheless, we expect the approach introduced here can be generalized to handle that circumstance by matching the boundaries of these level sets. Unlike 3D animated objects addressed in this article, the motion of video objects is unknown, and needs to be solved using computer vision techniques such as optical ﬂow [Beauchemin and Barron 1995; Black and Anandan 1996]. Once the motion of these objects has been estimated, the approach in this article can also be adapted to produce ﬂuid appearances for them. APPENDIX A. VARIATIONAL GRADIENT In this section, we provide an intuitive derivation of Equation (3) from Equation (2) in a 2D space. The derivation for a 3D space can be obtained similarly. The smoke boundary is ﬁrst discretized into a ﬁnite set of points. Consider one of the points xi shown in Figure 10. We need to perturb the position of xi to reduce the integral in Equation (2). Obviously, xi should move further into the interior of the target object. If xi moves along the local normal at xi by an inﬁnitesimal distance, Equation (2) is reduced by an amount equal to the area enclosed by the polygon xi−1 xi xi+1 xiN , where xiN represents the new ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Controllable Smoke Animation With Guiding Objects • 163 Fig. 10. Moving points on the smoke boundary along their respective normal directions can reduce the shape discrepancy between the smoke region and the target object most quickly. location of xi . If xi moves along some other arbitrary direction to xi by the same inﬁnitesimal distance, Equation (2) is reduced by an amount equal to the area enclosed by the polygon xi−1 xi xi+1 xi . Since xi xiN is perpendicular to the local boundary, when the lengths of xi−1 xi and xi xi+1 become sufﬁciently small, the area of xi−1 xi xi+1 xiN is guaranteed to be larger than the area of xi−1 xi xi+1 xi . That is, Equation (2) can be decreased most quickly by moving xi along the local normal of the smoke boundary. This also holds for all other points on the same boundary. By increasing the number of points in the boundary discretization, the distance between two adjacent points can be arbitrarily close to zero. By deﬁnition [Gelfand and Fomin 1963], the variational derivative of the functional in Equation (2), with respect to the geometry of the smoke boundary, exists. The variational derivative, with respect to a speciﬁc boundary point, is the unit normal vector at that point, multiplied by an appropriate sign given in Equation (3). B. AN EQUIVALENT CONDITION FOR ∇ · U = 0 We show that, for a bounded workspace , without sources and sinks, ∇(∇ · u) = 0 everywhere is equivalent to ∇ · u = 0 everywhere. Without loss of generality, suppose there exists a point x0 ∈ such that ∇ · u(x0 ) > 0. Since is bounded, (∇ · u)d x = 0. Therefore, there must be another point x1 ∈ such that ∇ · u(x1 ) < 0. Consider an arbitrary path between x0 and x1 . There must exist a point xm on that path such that ∇(∇ · u(xm )) = 0. Otherwise, ∇ · u(x0 ) = ∇ · u(x1 ). Thus, ∇ · u must be zero everywhere. REFERENCES ALEXA, M., COHEN-OR, D., AND LEVIN, D. 2000. As-rigid-as-possible shape interpolation. In SIGGRAPH 2000 Conference Proceedings. 157–164. BEAUCHEMIN, S. AND BARRON, J. 1995. The computation of optical ﬂow. ACM Comput. Surv. 27, 3, 433–467. BLACK, M. AND ANANDAN, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth ﬂow ﬁelds. Comput. Vision Image Understand. 63, 1, 75–104. CARR, J., MITCHELL, T., BEATSON, R., CHERRIE, J., FRIGHT, W., MCCALLUM, B., AND EVANS, T. 2001. Reconstruction and representation of 3d objects with radial basis functions. In SIGGRAPH 2001 Conference Proceedings. 67–76. CHORIN, A. 1967. A numerical method for solving incompressible viscous ﬂow problems. Computat. Phys. 2, 12–26. DESBRUN, M. AND CANI-GASCUEL, M.-P. 1998. Active implicit surface for animation. In Proceedings of Graphics Interface. EBERT, D., MUSGRAVE, K., PEACHY, D., PERLIN, K., AND WORLEY, S. 1998. Texturing and Modeling: A Procedural Approach, 2nd Ed. AP Professional. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 164 • L. Shi and Y. Yu ENRIGHT, D., MARSCHNER, S., AND FEDKIW, R. 2002. Animation and rendering of complex water surfaces. ACM Trans. Graph. 21, 3, 736–744. FATTAL, R. AND LISCHINSKI, D. 2004. Target-driven smoke animation. ACM Trans. Graph. 23, 3, 439–446. FEDKIW, R., STAM, J., AND JENSEN, H. 2001. Visual simulation of smoke. In SIGGRAPH 01 Conference Proceedings. 15–22. FOSTER, N. AND FEDKIW, R. 2001. Practical animation of liquids. In SIGGRAPH 2001 Conference Proceedings. 23–30. FOSTER, N. AND METAXAS, D. 1997a. Controlling ﬂuid animation. In Proceedings of Computer Graphics International. 178–188. FOSTER, N. AND METAXAS, D. 1997b. Modeling the motion of a hot, turbulent gas. In SIGGRAPH 97 Conference Proceedings. 181–188. GARDNER, G. 1985. Visual simulation of clouds. In Proc. of SIGGRAPH’85. 297–304. GELFAND, I. AND FOMIN, S. 1963. Calculus of Variations. Prentice-Hall. GOLUB, G. AND LOAN, C. V. 1996. Matrix Computations. The Johns Hopkins Unversity Press. GOMES, J., BEIER, T., COSTA, B., DARSA, L., AND VELHO, L. 1997. Warping and morphing of graphical objects. SIGGRAPH 97 Course Notes. HUGHES, J. F. 1992. Scheduled fourier volume morphing. In SIGGRAPH 92 Conference Proceedings. 43–46. KAJIYA, J. AND VON HERZEN, B. 1984. Ray tracing volume densities. Comput. Graph. (SIGGRAPH 84 Proceedings) 18, 3. KAPLER, A. 2002. Evolution of a vfx voxel tool. In SIGGRAPH 2002 Sketches & Applications, Conference Abstracts and Applications. 179. LAMORLETTE, A. AND FOSTER, N. 2002. Structural modeling of ﬂames for a production environment. In SIGGRAPH 02 Conference Proceedings. 729–735. LERIOS, A., GARFINKLE, C., AND LEVOY, M. 1995. Feature-based volume metamorphosis. In SIGGRAPH 95 Conference Proceed- ings. 449–456. MCNAMARA, A., TREUILLE, A., POPOVIC, Z., AND STAM, J. 2004. Fluid control using the adjoint method. ACM Trans. Graph. 23, 3, 447–454. MUSETH, K., BREEN, D., WHITAKER, R., AND BARR, A. 2002. Level set surface editing operators. ACM Trans. Graph. 21, 3, 330–338. OSHER, S. AND FEDKIW, R. 2001. Level set methods: an overview and some recent results. Computat. Phys. 169, 2. OSHER, S. AND SETHIAN, J. 1988. Fronts propagating with curvature dependent speed: Algorithms based on hamilton-jacobi formulations. Comput. Phys. 79, 12–49. PERLIN, K. 1985. An image synthesizer. In Proceedings of SIGGRAPH’85. 287–296. PERLIN, K. AND HOFFERT, E. 1989. Hypertexture. In Proceedings of SIGGRAPH’89. 253–262. SETHIAN, J. 1999. Level Set Methods and Fast Marching Methods. Cambridge University Press. SHI, L. AND YU, Y. 2004. Inviscid and incompressible ﬂuid simulation on triangle meshes. Comput. Animat. Virt. Worlds 15, 3-4, 173–181. SIMS, K. 1992. Choreographed image ﬂow. J. Visualiz. Comput. Animat. 3, 31–43. STAM, J. 1995. Multi-scale stochastic modeling of complex natural phenomena. Ph.D. Thesis, University of Toronto. STAM, J. 1999. Stable ﬂuids. In SIGGRAPH 99 Conference Proceedings. 121–128. STAM, J. 2003. Flows on surfaces of arbitrary topology. ACM Trans. Graph. 22, 3, 724–731. STANIFORTH, A. AND COTE, J. 1991. Semi-lagrangian integration schemes for atmospheric models: A review. Monthly Weather Review 119, 2206–2223. TREUILLE, A., MCNAMARA, A., POPOVIC, Z., AND STAM, J. 2003. Keyframe control of smoke simulations. ACM Trans. Graph. 22, 3, 716–723. TURK, G. AND O’BRIEN, J. 1999. Shape transformation using variational implicit functions. In SIGGRAPH 99 Conference Proceedings. 335–342. TURK, G. AND O’BRIEN, J. 2002. Modelling with implicit surfaces that interpolate. ACM Trans. Graph. 21, 4, 855–873. WOLBERG, G. 1990. Digital Image Warping. IEEE Computer Society Press. YNGVE, G., O’BRIEN, J., AND HODGINS, J. 2000. Animating explosions. In SIGGRAPH 2000 Conference Proceedings. 29–36. YU, Y. AND SHI, L. 2002. Object modeling and animation with smoke. Tech. Rep. No. UIUCDCS-R-2002-2262 (Engr. UILU-ENG-2002-1705), Computer Science, University of Illinois at Urbana-Champaign, http://www.cs.uiuc.edu/Dienst/UI/2.0/ Describe/ncstrl.uiuc cs/UIUCDCS-R-2002-2262 (Jan.). ZHAO, H.-K., OSHER, S., AND FEDKIW, R. 2001. Fast surface reconstruction using the level set method. In 1st IEEE Workshop on Variational and Level Set Methods. Received July 2003; revised May 2004, August 2004; accepted October 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion ˇ ´ ADNAN SULEJMANPASIC and JOVAN POPOVIC ´ Massachusetts Institute of Technology Adaptation of ballistic motion demands a technique that can make required adjustments in anticipation of ﬂight periods when only some physically consistent changes are possible. This article describes a numerical procedure that adjusts a physically consistent motion to fulﬁll new adaptation requirements expressed in kinematic and dynamic constraints. This iterative procedure reﬁnes the original motion with a sequence of minimal adjustments, implicitly favoring motions that are similar to the original performance, and transforming any input motion, including those that are difﬁcult to characterize with an objective function. In total, over twenty adaptations were generated from two recorded performances, a run and a jump, by varying foot placement, restricting muscle use, adding new environment constraints, and changing the length and mass of speciﬁc limbs. Categories and Subject Descriptors: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism—Animation; G.1.6 [Numerical Analysis]: Optimization—Constrained Optimization; G.1.7 [Numerical Analysis]: Ordinary Differential Equa- tions—Boundary value problems General Terms: Algorithms, Design Additional Key Words and Phrases: Animation with constraints, physically based animation, motion capture 1. INTRODUCTION Ballistic motions such as jumps, runs, and other acrobatic maneuvers consist of ﬂight periods during which performers are propelled by the force of gravity and momentum alone. In anticipation of such periods, performers execute speciﬁc actions to accomplish the desired outcome. For example, before a twisting jump, a performer bends down, bursts upward to propel himself, and simultaneously spins his body to generate the angular momentum that twists his body in the air. An adaptation technique must emulate this anticipation when adding a twist to a recorded jump performance. We describe a numerical method that creates such anticipation in ballistic motions of a full human ﬁgure with many (in our case, 42) degrees of freedom. This iterative algorithm modiﬁes a physically consistent input motion with a sequence of minimal modiﬁcations until all adaptation goals are met. Each adjustment modiﬁes the entire motion to anticipate the lack of control in ﬂight with changes at other times, when the appropriate control is available. As a result, lengthening the jump with kinematic constraints produces a motion with a deeper bend in the knees before the ﬁgure leaps from the ground.1 1 This approach is not required for motions without a ballistic period, such as walking and reaching; simpler kinematic techniques c are effective in these cases [Bruderlin and Williams 1995; Witkin and Popovi´ 1995]. This research was sponsored by the National Science Foundation and the NTT Corporation. Authors’ address: MIT CSAIL, The Stata Center, 32 Vassar Street, Cambridge, MA 02139; email: {adnans, jovan}@csail.mit.edu. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proﬁt or direct commercial advantage and that copies show this notice on the ﬁrst page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior speciﬁc permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036 USA, fax: +1 (212) 869-0481, or permissions@acm.org. c 2005 ACM 0730-0301/05/0100-0165 $5.00 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005, Pages 165–179. 166 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ Existing approaches in the literature employ optimization methods and caution of convergence difﬁ- culties, particularly for physically based problems with many degrees of freedom. Our method does not explicitly optimize any objective function. Instead, it searches for a motion that meets the constraints with a bias for motions that are similar to the original performance. The absence of the objective function allows it to transform any physically consistent motion, including biological motions that are typically difﬁcult to characterize with an objective function. We describe a simple scaling strategy for drastically improving the convergence of our iterative method. Unlike the previous methods, which simpliﬁed the state space or the governing dynamics equations, the scaling permits adaptations with restricted muscle use and with general interaction between the character and the environment. The same scaling strategy also improves the optimal trajectory formulations such as the method of spacetime constraints [Witkin and Kass 1988], which allows us to compare such optimizations with our method. 2. BACKGROUND Motion adaptation has emerged as an important computer graphics problem with the maturation of acquisition technology for archiving high-quality human motion. The ﬁrst methods broadened the applicability of acquired motion with the introduction of tools that displace (warp) motion to satisfy c new kinematic constraints [Bruderlin and Williams 1995; Witkin and Popovi´ 1995]. An animator can displace the foot in a few keyframes and the editing tool propagates the changes to the remaining frames with the interpolated displacements. Although this fast and simple approach is sufﬁcient in many cases, it only satisﬁes kinematic constraints at the displaced keyframes. If animators want to ensure that the feet do not go below the ground at any point in the animation, they have to introduce displacements at every frame, which defeats the purpose of using the automatic techniques in the ﬁrst place. Currently, these problems can be resolved with a technique that alternates between inverse kinemat- ics, which enforces speciﬁed constraints, and ﬁltering, which smooths the changes [Lee and Shin 1999]. A more general approach, and the one that is closer to our technique, is an optimal trajectory method that minimizes the difference from the original motion, while enforcing kinematic constraints through- out the motion [Gleicher 1997, 1998]. The motion retargeting techniques described in these papers do not generate physically consistent motions, but suggest applying a sequence of minimal modiﬁcations to the input motion. This is also an approach we use to adapt motions in a physically consistent manner. Optimal trajectory methods were introduced to computer graphics by Witkin and Kass [1988], who developed the spacetime constraints technique for motion synthesis. Dynamics is an integral part of that formulation, which showed that by optimizing power consumption, a numerical procedure could select a natural motion among the many physically valid alternatives. Extending these observations to generate natural motions for a full human ﬁgure proved difﬁcult [Cohen 1992; Liu et al. 1994] even as complex simulations of the human musculoskeletal system were veriﬁed by experimental data [Pandy and Anderson 2000]. The one exception was a technique for generating short motion transitions [Rose et al. 1996]. It seemed necessary to simplify the state space by reducing the number of degrees of freedom, or to approximate the governing dynamics equations. The ﬁrst technique for adaptation of ballistic motion projected the original motion onto a simple character, where it made the adjustments using the simpli- c ﬁed dynamics, before projecting the changes back onto the full human ﬁgure [Popovi´ and Witkin 1999]. However, the ﬁnal projection step does not produce a physically consistent ﬁnal motion. Our technique produces physically consistent motions without any simpliﬁcation. The governing physical laws could also be simpliﬁed with momentum constraints, which enforce c typical momentum proﬁles [Liu and Popovi´ 2002; Abe et al. 2004], or with aggregate-force constraints, which eliminate internal torques and constrain only the total (aggregate) force [Fang and Pollard 2003]. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 167 Fig. 1. The adaptation of a human broad jump (left ﬁgure) generates a new physically consistent jump (right ﬁgure) with a staggered takeoff and landing. An animator speciﬁes only the new foot placement to evoke this change. These simpliﬁcations yield efﬁcient adaptation methods at the cost of ignoring the internal muscle use. This implies that an animator is not allowed to restrict muscle usage and not allowed to specify an efﬁciency criterion that depends on such muscle activation. Another option is to reparameterize the degrees of freedom with principal components through sta- tistical analysis of variations observed in a speciﬁc behavior [Safonova et al. 2004]. The inferred low- dimensional representation facilitates convergence of optimal trajectory methods without simplifying dynamics. This approach complements our technique, but must be used carefully to construct low- dimensional subspaces that are capable of encoding motions that satisfy arbitrary kinematic constraints sc [Safonova et al. 2004; Sulejmanpaˇ i´ 2004]. Optimal trajectory methods are ideally suited for the adaptation of ballistic motions that require an- ticipatory planning. In scenarios in which anticipation plays a less prominent role, dynamic properties such as balancing or physically valid interaction with the environment might be more easily accom- plished with kinematic and dynamic ﬁltering [Zordan and Hodgins 1999; Yamane and Nakamura 2000; Tak et al. 2002; Shin et al. 2003]. These approaches are effective if the adjustments are small, and the required look-ahead is localized. When this is not the case, the simultaneous approach of optimal tra- jectory methods and our trajectory adaptation is a better choice. 3. ADAPTATION Adaptation reuses a recorded human motion by conforming it to make the required changes. The entire process resembles the standard keyframing techniques, which allow the animator to specify positions of hands, feet, or other end-effectors, with the added capability to enforce laws of physics, limit the use of speciﬁc muscles, and otherwise restrict the dynamics with constraints. Figure 1 shows an adaptation of a broad jump: the input motion, in which the performer jumps and lands with parallel footing, is transformed into a physically valid jump with staggered foot placement. 3.1 Formulation Adaptation computes the motion q, the internal muscle use f, and the external environment-reaction forces determined by the Lagrange multipliers λ. The solution satisﬁes the adaptation goals expressed in kinematic, K, and dynamic, D, constraints2 : K(q) = 0 (1) D(q, f, λ) = 0. 2 Inequality constraints, which are excluded from Equation (1) for simplicity, are also supported with an active-set method, which replaces the active inequality constraints with equalities. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 168 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ Here, the dynamic constraints contain both the governing physical equations and other restrictions on dynamic quantities such as ground-reaction forces and internal muscle usage. The constraints alone do not uniquely determine the desired trajectories. Instead, the transformation seeks to identify natural motions within the set of feasible trajectories that satisfy the constraints. Optimal trajectory methods accomplish this goal by maximizing the smoothness of muscle actuation c [Popovi´ and Witkin 1999] or the proximity to the input motion [Gleicher 1997]. Our method does not explicitly optimize any such criteria: instead, it modiﬁes input trajectories with a sequence of alterations until it constructs trajectories that meet the constraints. This choice allows us to transform any input motion, including those that are difﬁcult to characterize with an objective function, as is often the case with natural motions. Our method applies minimal alterations throughout the transformation process to keep the resulting motion close to the original input, thereby preserving its natural characteristics. Section 4.1 describes a gradient method for computing these alterations efﬁciently. 3.2 Dynamic Constraints Dynamic constraints ensure physical consistency and provide additional controls for animators. For example, animators can restrict the use of a speciﬁc muscle with inequality constraints on joint torques. The nonlinear equality constraints that enforce the laws of physics are hardest to fulﬁll and their formulation has a profound effect on the efﬁciency of the numerical solution. Their proper scaling improves convergence and enables adaptation of motions with many degrees of freedom. When internal torques are not restricted, this performance can be further improved with a formulation that eliminates the internal torques to reduce the order of constraints [Fang and Pollard 2003]. 3.2.1 Standard Formulation. The differential algebraic equations that express Newton’s laws are derived with the Euler-Lagrange equations from the Lagrangian L of the human ﬁgure: d ∂L ∂L ∂P dt ∂ qr˙ − ∂qr − λT ∂qr DT (q, f, λ) ≡ d ∂ L − ∂ L − λT ∂P − f i = 0, dt ∂ q˙i (2) ∂qi ∂qi P(q) where the index r enumerates the degrees of freedom in the unactuated root joint (global position and global orientation) and the index i enumerates the remaining degrees of freedom in joints actuated by muscle forces f i . For a human ﬁgure, the complexity of Euler-Lagrange equations and associated derivatives necessitates systematic evaluation [Liu 1996] and elimination of redundant computations with caching or a recursive formulation [Hollerbach 1980]. The algebraic equations P and the matching Lagrange multipliers λ deﬁne the dynamic interaction between the character and the environment.3 For example, during ground contact these constraints maintain the contact between the feet and the ground. Such holonomic constraints produce algebraic equations that depend only on the kinematic quantities, but they are still a component of the dynamics constraints because of the associated multiplier terms λT ∂P , which describe the exchange of forces ∂q between the character and the environment. In contrast, pure kinematic constraints are separated from the dynamic constraints because they do not have the associated multiplier terms. 3.2.2 Low-Order Formulation. When internal torques f i are not bound by the objective function (to select a natural motion) or by the constraints (to restrict the use of a muscle), the torques are free 3 Another option is to select an independent set of reduced coordinates for which the environment constraints are satisﬁed by construction. Section 3.2.3 discusses the two design choices in more detail. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 169 variables, and can be eliminated from the optimization problem: ∂L ∂L ∂P d ∂ qr − ∂qr − λT ∂qr DG (q, λ) ≡ dt ˙ = 0. (3) P(q) ∂P On the ground, the total force on the body is given by the ground reaction force λT ∂qr . These reaction forces are a function of the root position and the angle values of all other joints between the root and the contact point. In ﬂight, there are no environment constraints and the total force is zero. Eliminating optimization variables and lowering the order of dynamic constraints improves the ef- ﬁciency of numerical solutions—often by an order of magnitude—but might have an adverse effect on the smoothness of the resulting motion as freely varying torques may cause the body limbs to jerk un- desirably. In many cases, minimizing the total change made to joint angles and multipliers corrects this problem. Additionally, the torque constraints from Equation (2) can be reinserted and their smoothness controlled by the number of control points in the spline parameterization. Even when the visual quality of the ﬁnal motion requires optimizing every joint-angle, torque, and multiplier value, the efﬁciency of adaptation with the low-order constraints of Equation (3) enables rapid prototyping before the ﬁnal reﬁnements and smoothing are made. 3.2.3 Discussion. The low-order dynamics constraints are equivalent to the aggregate-force con- straints introduced by Fang and Pollard [2003]. Fang and Pollard use reduced coordinates to enforce environment constraints explicitly. In contrast, our formulation adds Lagrange multipliers to determine the forces needed to enforce the environment constraints. Baraff [1996] summarizes the differences be- tween the reduced and multiplier formulations for general applications in computer graphics. For the adaptation problem, we prefer the multiplier formulation for its explicit modeling of reaction forces and for its systematic treatment of all environment constraints P, especially in the presence of cyclic dependencies (e.g., both feet constrained to be on the ground simultaneously). The aggregate-force constraints are similar to the momentum constraints [Liu and Popovi´ 2002]. c In ﬂight, they are identical and equivalently state that the total angular momentum remains constant while the center of mass follows a parabolic trajectory. On the ground, or whenever the human ﬁgure interacts with the environment, the momentum constraints employ characteristic momentum patterns, without modeling the reaction forces. These momentum patterns emerge naturally from our aggregate- force formulation. In addition to this generality, our formulation exposes the reaction forces and enables their use in objective functions (e.g., to match the impact forces in the original motion or to reduce them for a “softer” run) and constraints (e.g., to keep the impact forces within a friction cone [Fang and Pollard 2003]). The two expressions for dynamics, with torques DT and without torques DG , deﬁne the two extremes in a range of possibilities. Intermediate formulations, which include some but not all torques, can exploit the beneﬁts of the reduced formulation, which generates results rapidly but has limited applicability, and the beneﬁts of the full formulation, which generates the best results and applies to all adaptation problems. For example, if the adaptation requires a jump with an injured ankle, the values of the ankle torques can be restricted and added to the optimization along with the Euler-Lagrange equations for the corresponding degrees of freedom. Or, if some of the limbs jerk undesirably, their torques can be included and their change minimized along with the modiﬁcations to joint angles and multipliers. 4. NUMERICAL SOLUTION Our numerical method approximates trajectories with the cubic B-spline expansion [Cohen 1992] and computes the expansion coefﬁcients with collocation by fulﬁlling kinematic and dynamic constraints at ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 170 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ prescribed time points. Because of the large number of unknowns, we employ a gradient method that computes the adjustments by solving a sequence of quadratic subproblems with linearized constraints. Proper scaling of constraint equations ensures efﬁcient convergence of this method. 4.1 Gradient Method Our adaptation technique selects a natural motion with a sequence of minimal modiﬁcations that iteratively modify the original motion until all adaptation goals are met. The gradient method alternates between computing the direction vector di of smallest magnitude, and computing the step size αi ∈ (0, 1] for the modiﬁcation in this direction. The small size of the individual direction vectors ensures that the cumulative change is small, even though the proximity to the original motion is never enforced explicitly. At each iteration, the direction for the next modiﬁcation di is a minimum-norm solution of an un- derdetermined linear system produced by a linearization of the kinematic and dynamic constraints C = (K, D): di = arg min d 2 d ∂C (4) subject to (xi ) d = 0. C(xi ) + ∂x The current iterate xi and the adjustment di contain the coefﬁcients for joint angles, torques, and multipliers. In effect, the solution to this quadratic problem computes the smallest change in joint angles, reaction forces, and muscle actuations. The dimensions of the Jacobian matrix ∂C/∂x restricts the choice of numerical solutions to tech- niques that can exploit its sparsity. The Conjugate-Gradient algorithm converges slowly because the corresponding normal equations are poorly conditioned and do not improve with diagonal precondi- tioning. A faster solution can be derived by explicit construction of the null-space basis with Q-less QR factorization of the sparse Jacobian matrix [Demmel and Higham 1993]. In our implementation, we chose the quadratic programming technique SQOPT [Gill et al. 1997] to support inequality constraints with sparse LU factorization of the Jacobian matrix. The minimum-norm solution deﬁnes the direction for the line search, which computes the step size αi for the modiﬁcation that minimizes the distance of the next iterate xi+1 from the adaptation constraints4 : αi = arg min C(xi + αdi ) 2 , α . (5) xi+1 = xi + αi di . The iteration stops once the constraints are satisﬁed with the desired accuracy. Although the magnitude of the total adjustment could be improved by additional steps along the constraints surface, these improvements have negligible effect on the visual quality of the resulting motion. A similar effect was also observed in adaptation problems without dynamics [Gleicher 2004]. 4.2 Scaling Under restrictive theoretical assumptions, some gradient methods can be shown to produce the same sequence of iterates regardless of the scaling. In practice, this scale invariance cannot be achieved and proper scaling is essential to resolve the difﬁculties in the conditioning of difﬁcult nonlinear problems [Gill et al. 1989]. In our experiments, proper scaling of joint angles, torques, and multipliers improved efﬁciency and prevented divergence. 4 The same approach would accommodate a weighted norm, but our scaling made such norm unnecessary for our experiments. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 171 A simple physical pendulum, weighing 70 kg (a typical human weight) and 1.70 m long (a typical human height), demonstrates the effect of scaling on the computation of physically valid trajectories with collocation and gradient methods. Without scaling, the computation of the pendulum trajectory requires 129 iterations and 11.6 seconds of computation time. With the simple scaling procedure de- scribed in this section, the computation of identical trajectories requires 24 iterations and 1.96 seconds. In both instances, the parameters of the gradient procedure are identical: the pendulum starts in a conﬁguration perpendicular to the direction of gravity and swings for 0.6 s, ending just before the back swing. On adaptation problems, the effect is even more drastic: improper scaling prevents convergence without excessively small steps, which can extend the computation of several seconds to as long as several hours. The problem can be traced back to the discrepancy in the range of the state variables, the torques, and the Lagrange multipliers, which in turn affects the scaling of the Jacobian matrices. The simplest solution is to scale the mass density of each limb by a uniform constant factor s, and to solve for the new joint angles q , torques f , and Lagrange multipliers λ . This scaling changes the Lagrangian L = sL and the expression of the dynamics in terms of the unscaled Lagrangian L: ∂L ∂L ∂P d s dt ∂ qr ˙ − s ∂q − λ T ∂q r r DT (q , f , λ ) ≡ s d ∂L − ∂L s ∂q −λT ∂P − fi . (6) dt ˙ ∂ qi i ∂qi P(q ) As division by the factor s reveals, this scales the values of torques and Lagrange multipliers without changing the joint-angle trajectories: f λ DT (q , f , λ ) = DT q, , . (7) s s Loss of accuracy, which could prevent accurate speciﬁcation of kinematic constraints, is one possible drawback of this simple scaling transformation. We avoid this problem in our experiments by empirically choosing the scale factor s = 0.001. We found that larger values of 0.1 and 0.01 degraded convergence, and smaller values of 0.0001 and less interfered with our internal constant, which determines the feasibility of each constraint. The same scaling factor worked in all of our experiments, but should the loss of accuracy become problematic with this simple strategy, proper scaling could also be established by determining the range of torque and multiplier forces precisely [Gill et al. 1989]. Scaling is not as critical for alternative formulations with momentum constraints [Liu and Popovi´ c 2002] and constrained aggregate forces [Fang and Pollard 2003]. Momentum constraints do not compute Lagrange multipliers as they do not model ground-reaction forces. The aggregate-force constraints use reduced coordinates, which do not employ Lagrange multipliers. This leaves only joint-angle trajectories for unknowns, as internal torques are also not computed in either formulation. Because joint angles have identical units, the problem is scaled automatically, but this convenience prevents these alternatives from solving adaptation problems that must restrict muscle forces. 4.3 Initialization ¯ The initial values q, f, and λ are estimated by ﬁtting the joint-angle trajectories q in the original performance. This initialization solves an optimal trajectory problem with the least squares objective ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 172 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ function: arg min q(t) − q(t) 2 dt ¯ q(t),f(t),λ(t) (8) subject to K(q) = 0 D(q, f, λ) = 0. A solution to a similar optimization problem was previously used to estimate the parameters of a c simpliﬁed character from the original performance [Popovi´ and Witkin 1999]. The key difference is that our estimation computes joint angles, torques, and multipliers for the entire human ﬁgure using the full dynamics of Equation (2), without any simpliﬁcation. For this optimization, the torques and multipliers are initially set to zero, and the joint angles are computed with cubic B-spline interpolation of joint angles in the original performance. Initialization is performed only once for each recorded performance. Subsequently, the estimated trajectories are stored with the motion and used for every adaptation problem. The initialization assumes that the original performance is a physically valid motion. When it is not, the initialization might not successfully compute the torques and the multipliers required by our adaptation technique. Although not explored in this article, a different initialization procedure might c eliminate this requirement by consulting a stored database of transition poses [Liu and Popovi´ 2002], or by restricting the ground-reaction forces [Fang and Pollard 2003]. 5. RESULTS In our adaptation experiments, we varied foot placement, restricted the use of speciﬁc muscles, intro- duced new environment constraints, added additional weights, and changed the skeleton dimensions to generate over twenty adaptations from two performances: a run and a jump. The distinctive attributes of the original motion were visibly preserved in all examples. In these experiments, we needed only a few constraints to indicate, the desired adjustments. In contrast, kinematic techniques would require many manually speciﬁed constraints to ensure realism of resulting motions. We relied primarily on point constraints to indicate placement of feet and hands. For aesthetic reasons, cone constraints were used to enforce natural joint-angle limits and to prevent interpenetration of limbs. Full pose constraints were used sparingly because they were not necessary to generate high-quality adaptations, and because they are, in general, difﬁcult to specify. On occasion, it was easier to prevent limb intersections with a full pose constraint than with a cone constraint for each joint. The ﬁgures include a visual illustration of almost all kinematic constraints required to generate the motion shown. The only exceptions are inequality constraints (feet above the ground, feet clearing the hurdle, or hand above the rim) and orientation constraints, which speciﬁed starting and ﬁnal body orientation for the twist jumps. 5.1 Discretization Our implementation uses a collocation method to enforce the constraints at designated discrete time points, which are also called collocation points. During takeoff, landing, and other periods with contact between the character and the environment, the environment constraints maintain the contact and the Lagrange multipliers determine the associated reaction forces. These multipliers and environment constraints are not part of the dynamic constraints evaluated on collocation points in periods of free ﬂight. The unknowns, which include the joint-angles, torques, and multipliers, are also discretized. The Lagrange multipliers λ are sampled at the collocation points, and the joint-angle q and torque trajec- tories f are parameterized with cubic B-spline curves. Because the number of multiplier samples and ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 173 Table I. unknowns dynamic constraints Motion q c. p. f c. p. λ samples reduced full reduced full Jump 24 30 20 1128 2388 288 1296 Run 30 38 20 1320 2916 276 1572 Our implementation approximates each joint-angle q and torque trajectory f with a cubic B-spline curve. The Lagrange multipliers λ are sampled at prescribed points that enforce the environment constraints and are not approximated by a cubic B-spline curve. Our adaptation can use the full dynamics of Equation (2), or the reduced formulation of Equation (3). If the internal torques are not smoothed or restricted, then the reduced formulation enables faster convergence. Fig. 2. Motion adaptation adjusts motions of high-dimensional human characters. A skeleton conﬁguration with 42 degrees of freedom was used in our adaptation experiments. The mass of each limb was assigned as a percentage of the total body mass. B-spline control points determines the total number of unknowns,5 the appropriate setting is chosen through trial-and-error to minimize the total number of unknowns. In general, we begin by matching the control points, and collocation points, and then reduce the number of control points until we reach the minimum necessary to avoid an overconstrained formulation. Table I lists the number of samples and control points used in our experiments. 5.2 Initialization The original performances were captured in a motion-capture studio. Standard commercial tools [Vicon 2003] were used to compute the limb lengths for the 42 degree-of-freedom character shown in Figure 2. The mass of each limb is computed as a percentage of the total weight with the ratios shown in Figure 2. These ratios were manually adjusted for the limbs in our skeleton from the weight ratios reported in a standard biomechanical reference [Winter 1990]. Given the inferred physical parameters and the original motion, the initialization (Section 4.3) es- timates the joint-angle trajectories that are physically consistent with internal muscle use and the environmental reaction forces. For this process, the environmental constraints and the duration of each ground-contact and free-ﬂight stage were identiﬁed manually by selecting the appropriate frames in the original performance. 5 In a jump, for example, the unknowns for the full dynamics of Equation (2) include 24 × 42 = 1008 joint-angle control points (24 for each of 42 degrees of freedom), 30 × 42 = 1260 torque control points, and 20 × 6 = 120, lagrange multiplier samples (6 multipliers for location of both feet in each of 20 time samples). The reduced dynamics of Equation (3) have fewer unknowns because they do not include torques. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 174 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ Fig. 3. All motions were generated by adapting two performances: a human jump (a) and a run (j). The ﬁgures visualize the constraints required for each adaptation. Red spheres represent point constraints, red cones the joint-limit constraints visualized by the limits drawn around the corresponding limb, and the transparent gray boxes identify pose constraints, which constrain the values of all joints in the skeleton. The shin is colored purple when the knee is constrained to a ﬁxed value. A line extending from a knee shows the maximum allowed range for a knee joint. Some inequality and orientation constraints are not shown. 5.3 Jumps The original jump, after initialization, is shown in Figure 3(a). Our adaptation process transforms this jump into new motions that meet the desired constraints. In Figure 1, for example, the constraints specify new foot placement, and the adaptation produces a jump with staggered takeoff and landing. In Figure 3(b), the constraints maintain the foot placement in the original jump and add a briefcase ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 175 to increase the weight of the right arm. The adaptation lowers the motion of the heavy right arm and compensates by raising the lighter left arm higher. This motion is then further reﬁned in Figure 3(c), with a constraint that limits the torque applied around the right ankle joint. This upper bound simulates the effect of an injured right ankle. The healthy left leg carries more weight throughout the takeoff and trails the injured right leg to generate momentum needed for the liftoff. In the air, the same leg hurries forward to absorb most of the landing impact. In Figure 3(d), a similar upper bound constrains the torque applied at the right shoulder joint. As a result, the limp right arm dangles throughout the motion and the upper body compensates to swing it forward. As with a briefcase jump, the healthy arm is lifted higher to increase the vertical momentum. In Figures 3(e) and 3(f), the constraints enforce the pelvis orientation, along with the foot placement. This produces twisting jumps: ﬁrst in 3(e) with the original takeoff and modiﬁed landing, and then in 3(f) with the modiﬁed takeoff and the original landing. In the second example, we needed to adjust the original jump to rotate the takeoff stage. This manual adjustment is a limitation that’s revisited in Section 5.6. Still, such adjustments are simple to perform because our adaptation process resolves any discontinuities such as the gap between the takeoff stage and the remaining motion. In Figure 3(g), a new environment constraint replaces the ground with a high bar. In the new “landing” stage, the feet are unconstrained and the constraints maintain the contact between the hands and the high bar. As a result, the character jumps, reaches for the bar, and swings while holding onto it. To initialize the optimization, we only use the ﬁrst half of the original jump, from the start to the ﬂight apex. The last frame of this sequence was extended, as a still frame, for the duration of the swinging stage. The adaptation adjusts the entire motion to produce a physically consistent motion that satisﬁes new environment constraints. Both one-legged hops shown in Figure 3(h) and 3(i) were produced with adaptations that maintain a ﬁxed angle in the right knee. The ﬁrst adaptation mimics the broad jump to create a character that absorbs most of the impact with torques applied at the hip and knee joints. The second adaptation produces a more typical one-legged landing by enforcing additional constraints that restrict the torques applied at these two joints. As a result, the character avoids excessive strain by landing with an extended leg. 5.4 Runs The original run is shown in Figure 3(j). Our adaptation produces a hurdle jump shown in Figure 3(k) by meeting the speciﬁed constraints: increased distance between the ground contacts and height required to clear the hurdle. New distances are enforced by displacing all footprints. This generates bouncier runs with longer strides. The hurdle constraint speciﬁes a lower bound on the height attained by the left and right toe. The dunk motion in Figure 3(l) is similar to the high-bar example. A new environment constraint enforces the contact with the rim. Just as in the high-bar example, the last frame of the run cycle was extended, as a still frame, for the duration of the swing stage. We’ve also modiﬁed the skeleton by shortening its legs. The adaptation process compensates by producing a more explosive motion. Cone constraints were added to prevent interpenetration of limbs. In Figure 3(m), the adaptation produces a cross-step run with the proper step-to-step lean by satisfying the displaced footprints. 5.5 Comparison We compared the efﬁciency of our adaptation method with the optimal trajectory method that employed sequential quadratic programming (SQP) [Gill et al. 2002]. SQP is an iterative descent technique that computes the optimal trajectory by minimizing the merit function along search directions given by solutions to quadratic programming subproblems. The merit function balances the competing goals of improving the objective function and remaining on the nonlinear constraint surface, while the quadratic ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 176 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ Fig. 4. The timing chart compares the efﬁciency of our adaptation method—with and without reduced dynamics—against the best performing optimal trajectory method that maximized the smoothness of muscle actuation. The corpus of problems includes all adaptations on which all three formulations were applicable. The adaptations were timed on a single-processor 1.7 GHz Pentium III workstation with 1 GB of main memory. In general, the visual quality of each adaptation was similar, but on occasion, the adaptation with reduced dynamics produced jerky motions. subproblems linearize constraints and approximate the merit function with a quadratic expansion around the current iterate. Just as with our gradient method, proper scaling and initialization ensured that the SQP method solves adaptation problems for a full human ﬁgure. The SQP method uses an objective function to identify a unique motion that satisﬁes speciﬁed con- straints. Prior literature proposed several objective functions with the aim of selecting motions that ˙ appear most natural. The synthesis applications optimized power consumption ( f· θ 2 dt) [Witkin and 2 Kass 1988], torque output ( f dt) [Liu et al. 1994], torque smoothness ( f ¨ 2 dt) [Popovi´ and Witkin c ¨ 1999], and kinematic smoothness ( θ 2 dt) [Fang and Pollard 2003], while the adaptation applications c minimized either joint-angle displacement [Gleicher 1997], or mass displacement [Popovi´ and Witkin 1999] from the original motion. In our experiments with motions from Figures 4 and 3, optimization of torque smoothness consistently generated better results than any other objective function. The power-consumption objective, which selects the most efﬁcient motion, produced natural motions on some adaptation problems, but performed poorly in many cases, especially during adaptation of running motions. The most efﬁcient motions frequently lead to interpenetration of limbs (between the arms and the torso or the two legs), which could not be resolved with joint-limit constraints. Optimization of torque output demonstrated identical problems and also produced jerky motions unless used in combination with the kinematic-smoothness objective. Similarly, the mass displacement objective, a kinematic analogue of the power consumption, had to be applied in conjunction with kinematic smoothness to prevent the human ﬁgure from jumping with a still body, and using only its ankles. We opted not to combine multiple objective functions because the quality of the resulting motion became highly dependent on weights assigned to each component in the objective function. Optimization of kinematic smoothness alone produced motions that are smooth, but not natural. For example, during the jump takeoff, the body curled up into an ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 177 unnatural position from which it would smoothly, but unnaturally, unwind to gain the appropriate momentum. Figure 4 compares our adaptation technique (full and reduced dynamics) with the best performing optimal formulation that maximizes the smoothness of internal torques. Note that proper scaling of optimization variables permits efﬁcient convergence of the optimal formulation even for the skeleton with many degrees of freedom. With reduced dynamics, our adaptation technique converges faster than the optimal formulation, by an order of magnitude on almost all problems. This improvement is smaller with the full dynamics, which suggests that adaptation with reduced and full dynamics is complementary. Once the desired constraints are met in rapid prototyping with the reduced dynamics, the full dynamics can be used to generate smoother motions. 5.6 Limitations The extent of any adaptation has its limits. If the difference between the initial and the ﬁnal motion is very large, motion adaptation will not be able to converge to a natural motion. For example, although we could produce a jump with a half-twist directly from the original motion, the jump with a full- twist required that we manually rotate the takeoff and the landing stages. Our technique, however, eliminates the most tedious work. It removes the discontinuities between the ﬂight and the transformed ground stages, and adjusts them to add the required anticipation. The success of this simple approach suggests that a combination of sequencing methods (e.g., Kovar et al. [2002]) and motion adaptation could further extend the value of the original motion. Optimization is sensitive to discretization of joint-angle and torque trajectories. If too few control points are chosen for the cubic B-spline approximation, the optimization might not converge because it cannot satisfy the dynamic constraints. On the other hand, with too many control points, the solution will satisfy the dynamics at prescribed time points, but will not look physically correct in between. Often, trial-and-error is the best way to determine the settings. Table I shows the settings used for the adaptation of jumps and runs. 6. CONCLUSION Motion adaptation with dynamics simpliﬁes the generation of very complex human motions. An ani- mator need specify only a sparse set of keyframes to deﬁne the goals for the motion, and the adaptation will generate a new natural motion automatically. This method, like previously published optimal tra- jectory techniques, beneﬁts from restrictions imposed by the dynamics of ballistic motion—the same restrictions that make manual adjustments particularly tedious. Although adaptation in the previously published work relies on the simpliﬁcation of dynamics constraints, this article demonstrates that such simpliﬁcations are not necessary. This generalization allows for the speciﬁcation of torque-based ob- jective functions and torque-based constraints and broadens the set of motions that can be generated with an adaptation approach. The adaptation problem provides a common ground for evaluation and comparison of optimal trajec- tory methods. In this article, we deﬁne a numerical solution that adapts a motion with a biased search for trajectories that satisfy the adaptation goals. The search favors the motions that are close to the original performance to preserve the distinctive attributes of the original performance. Our method is compared to an explicit optimization of muscle actuations on an identical set of adaptation problems. The results show that our method efﬁciently produces visually indistinguishable solutions. In the fu- ture, we would like to improve the robustness with automated selection of collocation points, and to explore techniques that use sequencing methods (e.g., Kovar et al. [2002]) to construct a better initial guess for each adaptation problem. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. 178 • sc c A. Sulejmanpaˇ i´ and J. Popovi´ REFERENCES ´ ABE, Y., LIU, C. K., AND POPOVIC, Z. 2004. Momentum-based parameterization of dynamic character motion. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. At press. BARAFF, D. 1996. Linear-time dynamics using lagrange multipliers. In Proceedings of SIGGRAPH 96. Computer Graphics Proceedings, Annual Conference Series. 137–146. BRUDERLIN, A. AND WILLIAMS, L. 1995. Motion signal processing. In Proceedings of SIGGRAPH 95. Computer Graphics Pro- ceedings, Annual Conference Series. 97–104. COHEN, M. F. 1992. Interactive spacetime control for animation. In Comput. Graph. (Proceedings of SIGGRAPH 92). Annual Conference Series. ACM, 293–302. DEMMEL, J. W. AND HIGHAM, N. J. 1993. Improved error bounds for underdetermined system solvers. SIAM J. Matrix Anal. Applicat. 14, 1–14. FANG, A. C. AND POLLARD, N. S. 2003. Efﬁcient synthesis of physically valid human motion. ACM Trans. Graph. 22, 3 (July), 417–426. GILL, P. E., MURRAY, W., AND SAUNDERS, M. A. 1997. User’s guide for SQOPT 5.3: A fortran package for large-scale linear and quadratic programming. Tech. Rep. NA 97-4, University of California, San Diego. GILL, P. E., MURRAY, W., AND SAUNDERS, M. A. 2002. SNOPT: An SQP algorithm for large-scale constrained optimization. SIAM J. Optimiz. 12, 4, 979–1006. GILL, P. E., MURRAY, W., AND WRIGHT, M. H. 1989. Practical Optimization. Academic Press, London. GLEICHER, M. 1997. Motion editing with spacetime constraints. In Symposium on Interactive 3D Graphics. 139–148. GLEICHER, M. 1998. Retargeting motion to new characters. In Proceedings of SIGGRAPH 98. Computer Graphics Proceedings, Annual Conference Series. 33–42. GLEICHER, M. 2004. Personal communication. HOLLERBACH, J. M. 1980. A recursive lagrangian formulation of manipulator dynamics and a comparative study of dynamics formulation complexity. IEEE Trans. Syst., Man, Cybernet. 10, 11 (Nov.), 730–736. KOVAR, L., GLEICHER, M., AND PIGHIN, F. 2002. Motion graphs. In Comput. Graph. (Proceedings of SIGGRAPH 2002). Annual Conference Series. ACM SIGGRAPH, 473–482. LEE, J. AND SHIN, S. Y. 1999. A hierarchical approach to interactive motion editing for human-like ﬁgures. In Proceedings of SIGGRAPH 99. Computer Graphics Proceedings, Annual Conference Series. 39–48. ´ LIU, C. K. AND POPOVIC, Z. 2002. Synthesis of complex dynamic character motion from simple animations. ACM Trans. Graph. 21, 3 (July), 408–416. LIU, Z. 1996. Efﬁcient animation techniques balancing both user control and physical realism. Ph.D. thesis, Princeton University. LIU, Z., GORTLER, S. J., AND COHEN, M. F. 1994. Hierarchical spacetime control. In Proceedings of SIGGRAPH, 94. Computer Graphics Proceedings, Annual Conference Series. 35–42. PANDY, M. G. AND ANDERSON, F. C. 2000. Dynamic simulation of human movement using large-scale models of the body. In Proceedings of the IEEE International Conference on Robotics and Automation. 676–681. ´ POPOVIC, Z. AND WITKIN, A. P. 1999. Physically based motion transformation. In Proceedings of SIGGRAPH 99. Computer Graphics Proceedings, Annual Conference Series. 11–20. ROSE, C. F., GUENTER, B., BODENHEIMER, B., AND COHEN, M. F. 1996. Efﬁcient generation of motion transitions using spacetime constraints. In Proceedings of SIGGRAPH 96, Computer Graphics. Annual Conference Series. 147–154. SAFONOVA, A., HODGINS, J., AND POLLARD, N. 2004. Synthesizing physically realistic human motion in low-dimensional, behavior- speciﬁc spaces. ACM Trans. Graph. 23, 3 (Aug.), 514–521. SHIN, H. J., KOVAR, L., AND GLEICHER, M. 2003. Physical touchup of human motions. In Proceedings 11th Paciﬁc Conference on Computer Graphics and Applications. 194–203. ˇ ´ SULEJMANPASIC, A. 2004. Adaptation of performed ballistic motion. M.S. thesis, Massachusetts Institution of Technology. TAK, S., YOUNG SONG, O., AND KO, H.-S. 2002. Spacetime sweeping: An interactive dynamic constraints solver. In Proceedings of the Computer Animation (CA 2002). IEEE Computer Society, 261–270. VICON. 2003. Vicon iQ Reference Manual. Vicon Motion Systems Inc., Lake Forest, CA. WINTER, D. A. 1990. Biomechanics and Motor Control of Human Movement, 2nd Ed. John Wiley and Sons, Inc., New York. WITKIN, A. AND KASS, M. 1988. Spacetime constraints. In Computer Graphics (Proceedings of SIGGRAPH 88), Vol. 22. 159–168. ´ WITKIN, A. AND POPOVIC, Z. 1995. Motion warping. In Proceedings of SIGGRAPH 95. Computer Graphics Proceedings, Annual Conference Series. 105–108. ACM Transactions on Graphics, Vol. 24, No. 1, January 2005. Adaptation of Performed Ballistic Motion • 179 YAMANE, K. AND NAKAMURA, Y. 2000. Dynamics ﬁlter—concept and implementation of on-line motion generator for human ﬁgures. In Proceedings of the IEEE International Conference on Robotics and Automation. 688–694. ZORDAN, V. B. AND HODGINS, J. K. 1999. Tracking and modifying upper-body human motion data with dynamic simulation. In Computer Animation and Simulation ’99. Recieved March 2004; revised September 2004; accepted September 2004 ACM Transactions on Graphics, Vol. 24, No. 1, January 2005.

DOCUMENT INFO

Shared By:

Categories:

Tags:
Feature-Based, Surface, Parameterization, Texture, Mapping

Stats:

views: | 18 |

posted: | 8/23/2011 |

language: | English |

pages: | 179 |

Description:
Feature-Based Surface Parameterization and Texture Mapping

OTHER DOCS BY docstocfun

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.