Virtual Populace Brain Springs: Fast Physics for Large Crowds in WALL•E Paul Kanyuk ■ Pixar Animation Studios C omputer animation is a developed, thriving field that has benefitted greatly from the techniques of traditional animation and its insights into what elements of motion create appealing characters. Pixar Animation Studios introduced the adaptation of such elements as squash, stretch, follow-through, and anticipation to the field,1 which guide some of the most successful computer animation to date. However, these elements are often neglected in crowd simulations, where rendering challenges, group behavior, and appearance variaA major challenge of making tion are the dominant focus. WALL•E was creating believable (For more information, see the physics for human and robot sidebar “Related Work in Crowd crowds. To do this, Pixar Simulation.”) technical directors combined We’ve developed a set of techa custom spring-physics niques that help incorporate key system in the Massive software elements of appealing character animation into crowd simulaplatform with traditional tions. In particular, we’ve devised simulation methods. The a procedural noise-based system performance was fast enough for creating “hover” motion, a to scale for large crowds and simple technique for anticipatmaintain interactivity for ing turns, and most significant, previews. brain springs, an efficient physics approximation for spring-based motion. We’ve applied these techniques to various scenarios, ranging from thousands of flying robots to crowds tumbling on a tilting spaceship (Figure 1 shows one example). These scenarios were varied enough to require extensive iteration on large crowds, necessitating both efficiency and interactivity. Conventions Before we explain our techniques, a frame of reference for translation and rotation is necessary. From a crowd agent’s frame of reference, let the 24 July/August 2009 y-axis be up, the z-axis be forward, and the x-axis be left and right. Rotation around the z-axis is banking, around the x-axis is leaning, and around the y-axis is turning. Also, we created many of the following methods using Massive, commercially available crowd software. (See the sidebar “Massive and WALL•E” for implementation details.) Physics Driving Motion A defining aspect of crowd locomotion in WALL•E is the characters’ ability to effortlessly float above the ground. This applied to robots flying through hallways as well as humans leisurely drifting on levitating lounge chairs. We broke this look down into three main elements: hover, anticipation, and reaction. Hover This element is the naturalistic oscillation in translation and rotation that keeps a floating agent from looking abnormally static (as if balanced between slowly varying forces). (For an example of this element, see Video 1; all videos mentioned in this article are at http://computingnow.computer.org/ wall-e.) We first attempted to create this effect by using sine waves at different amplitudes and frequencies that vary over time, for each root degree of freedom (x, y, and z translations and rotations). Simply tuning the amplitudes and frequencies appeared to work at first, but the simulation often degenerated after a few seconds and yielded “offkilter” motion (see Video 2). To solve the problem, we first had our animators keyframe a pleasing hover cycle, to serve as a reference and a model for a procedural solution. The animators used sinusoidal curves for each degree of freedom, too; however, this animated cycle’s key feature was the phase relationship between Published by the IEEE Computer Society 0272-1716/09/$25.00 © 2009 IEEE the sine waves. One set of phases can create the impression of a gallop, whereas another can create a bounce, for example. We adjusted our procedural model to maintain the phase relationships that the animators assigned, and let only those parameters vary that preserved relative phase (overall amplitude or frequency, wave shaping, amplitude per degreeof-freedom, and so on). Varying these parameters from agent to agent would cause the hovers to remain distinct over time and impart a different “affect.” (For example, a small, energetic cleaning robot would have a very-high-frequency hover, whereas a large, lumbering cargobot would have broader noise curves.) Parameterizing our noise in this way let animators fine-tune our characters’ hover personalities (see Video 3). Anticipation This element is a flying agent’s tendency to bank into a turn before reaching it, which is a natural reaction for rapid cornering. Craig Reynolds described a similar effect in which the agent’s acceleration vector determines its banking magnitude and direction.2 That technique, however, is reactive, in which the agent rotates only as the turn is in progress. Anticipation requires executing this banking motion before the actual turn, which gives the agent a sense of volition and intent. We modeled this effect in our agent “brains” (See the “Massive and WALL•E” sidebar) by looking ahead of each character’s path and measuring the difference in path orientation, which drove bank rotation (see Figure 2). Owing to our agents’ extreme speeds and frequency of turns, this signal was very jagged. We oversampled and filtered the signal as a postprocess, using a basic quaternion filter to achieve adequate smoothing. The result is a flying agent that appears to “carve” into the turns, showing a sense of awareness (see Video 4). Reaction This element is the natural motion resulting from physical forces being applied to the character, as opposed to the character’s intent at that time. These effects can be well modeled using spring dynamics. Rather than use a physical simulator, however, we implemented this system within our characters’ Massive “brains,” using fuzzy-logic nodes (see “Implementing Brain Springs” in the “Massive and WALL•E” sidebar). Our fake physics was successful because it was stable, allowed for interactive playback with large numbers of characters, and was easily parameterized to provide controls based on the distance to the camera and other metrics. The brain spring, our system’s key unit, converts an arbitrary input—typically a physical signal such as acceleration or velocity—into an approximation of damped simple harmonic motion that’s then connected to an animation output. For example, Figure 1. Robot crowds reacting and hovering in the movie WALL•E. Throughout much of the film, crowds of robots and humans on hover chairs play an important role in the environment and action. (© 2007 Disney/ Pixar. All rights reserved.) Look-ahead Theta Figure 2. The angle theta is the difference between the agent’s orientation and intended path at the current location, and that of the path a certain distance ahead. This angle drives banking before turning. IEEE Computer Graphics and Applications 25 Virtual Populace Related Work in Crowd Simulation I n feature films, animated and otherwise, crowd simulation is an area of ongoing research and development. The predominant direction has been refinement of agent behavior and decision-making. Early research modeled flocking behavior through simple local rules, which provided compelling navigation for groups of fish or birds.1 Later developments added knowledge and learning.2 Other research has explored diverse additions to behavioral models considering factors such as sociology3 and continuum dynamics.4 However, behavior—navigation and motion choice—is only one aspect of compelling crowd simulation. Dynamics and the impression of real physics are key in making animated characters believable but are often overlooked for large crowds. This is especially important for animated films, where exaggeration of dynamics is the stylistic norm. Pixar Animation Studios describes how 2D animation principles such as exaggeration and followthrough can enhance a 3D character’s believability.5 One can model many of these nuances through the dynamics of a spring (damped simple harmonic motion), and considerable research has focused on techniques for this modeling. Physics simulation is common but often proves too slow for real-time and crowd applications. “Fake” physics techniques that avoid simulation yet model this behavior are preferable for crowds. Two examples are a nondynamic technique for rope and spring motion6 and “Wiggly Splines,” a novel spline interpolation scheme that models spring oscillation.7 Another fake-physics technique is NeuroAnimators, which use trained neural networks to emulate physical systems.8 Many of these techniques, however, are cumbersome for crowd simulation, NeuroAnimators owing to the neural-net training, and others because of lack of speed or specificity issues. One class of techniques that does work well for crowds is motion signal processing, which uses filters to process existing motion data to generate an effect.9 Such techniques are often used in computer puppetry to take live performance input data and create more complex output. Examples of this application include Protozoa’s Alive! (described at www.awn.com/mag/issue5.03/5.03pages/ dannacherdot.php3) and Mike the Talking Head (http:// mambo.ucsc.edu/psl/mike.html). One motion-signal-processing technique creates oscillatory motion using an infinite-impulse response filter that can easily operate in real time on vast amounts of motion data.10 Our brain-springs technique (see the main article) furthers the state of the art in this area by modeling damped simple harmonic motion in crowd agents through a highly flexible motion-signal-processing method. With brain springs, animators can create efficient, controllable, and believable spring physics in crowd animation. References 1. C.W. Reynolds, “Flocks, Herds and Schools: A Distributed Behavioral Model,” Proc. Siggraph, ACM Press, 1987, pp. 25–34. 2. J. Funge, X. Tu, and D. Terzopoulos, “Cognitive Modeling: Knowledge, Reasoning and Planning for Intelligent Characters,” Proc. Siggraph, ACM Press, 1999, pp. 29–38. 3. S.R. Musse and D. Thalmann, “A Model of Human Crowd Behavior: Group Inter-relationship and Collision Detection Analysis,” Proc. 1997 Eurographics Workshop Computer Animation and Simulation, Eurographics Assoc., 1997, pp. 39–52. 4. A. Treuille, S. Cooper, and Z. Popovi, “Continuum Crowds,” ACM Trans. Graphics (Proc. Siggraph), vol. 25, no. 3, 2006, pp. 1160–1168. 5. J. Lasseter, “Principles of Traditional Animation Applied to 3D Computer Animation,” Proc. Siggraph, 1987, ACM Press, pp. 35–44. 6. R. Barzel, “Faking Dynamics of Ropes and Springs,” IEEE Computer Graphics and Applications, vol. 17, no. 3, 1997, pp. 31–39. 7. M. Kass and J. Anderson, “Animating Oscillatory Motion with Overlap: Wiggly Splines,” ACM Trans. Graphics (Proc. Siggraph), vol. 27, no. 3, 2008, pp. 1–8. 8. R. Grzeszczuk, D. Terzopoulos, and G. Hinton, “Neuro­ Animator: Fast Neural Network Emulation and Control of Physics-Based Models,” Proc. Siggraph, ACM Press, 1998, pp. 9–20. 9. A. Bruderlin and L. Williams, “Motion Signal Processing,” Proc. Siggraph, ACM Press, 1995, pp. 97–104. 10. P.C. Litwinowicz, “Inkwell: A 2-D Animation System,” Proc. Siggraph, ACM Press, 1991, pp. 113–122. when an agent quickly stops, we might want it to lean forward and back in a spring-like manner, eventually coming to rest (see Figure 3). To achieve this effect, we used a brain spring to convert translational acceleration into damped harmonic motion driving rotation around the agent’s x-axis (leaning). Expanding on this, our typical character’s brain contained springs for translational velocity and acceleration for leaning, and rotational velocity and acceleration for 26 July/August 2009 banking. For example, we can filter the rotationalacceleration signal with a brain spring and add it to the agent’s z-axis rotation to have the agent bank left and right in a spring-like manner owing to turn accelerations and decelerations. Figure 4 diagrams these relationships; for a demonstration of the effect on a simple robot turning, stopping, and starting, see Video 5. At its core, the brain spring model uses signal processing to shape its input. One related example used Massive and WALL•E T o execute the motion synthesis of our crowd agents, we used Massive, an agent-based crowd simulation system by Massive Software (www.massivesoftware.com). With Massive, we created small motion programs called “brains” to implement the techniques described in the main article. A sketch presented at Siggraph 2007 describes how we integrated Massive into our animation pipeline.1 use an unclamped “or” node so that the values aren’t clipped at 0 to 1. Usually, it’s a good idea not to connect the summation’s output directly to the agent curves but to the joint just after the root joint that affects the whole agent. Once the brain spring is working, make a macro out of the output and sum nodes so that you can easily reuse it. Motion Blending Bots on Paths The robot crowds in WALL•E traveled mainly on predetermined paths in the spaceship Axiom. In Massive, this is most easily implemented with the lanes feature. Lanes conveniently provide a “look ahead” function that’s suitable for calculating the angle necessary for computing “anticipation banking” (see the section “Anticipation” in the main article). Implementing Brain Springs To implement a brain spring in Massive, use an ouput node for each bounce (half oscillation), and enter the filter width and delay into the filter and delay fields. To sum the bounces, filtering to add overshoot and undershoot to animation signals.3 Our technique builds on these observations and uses filtering to approximate damped simple harmonic motion. We model each half of an oscillation (or bounce) as a filtered, shifted, and scaled version of the original signal, where increasing filter widths, increasing delays, and decreasing scales at alternating signs are used as we accumulate the bounces to create the desired pattern. Figure 5a shows this process’s mathematical formulation; Figure 5b provides a more intuitive illustration. It shows how we progressively filter, scale, and shift an initial negative acceleration input to form each half bounce of a two-oscillation spring. We can tune a brain spring’s parameters to achieve a range of motions from small, highfrequency oscillation to broad, slow oscillation (see Figure 6). This system provides convincing reaction of the root transforms for each character. However, many of our robots had limbs that would naturally react to forces differently than the body as a whole. To model such limbs’ overlapping motion, we applied the same brain springs to individual joint rotations with varying amplitudes and frequencies. We added more filtering and delay to the brain springs as the joints got further down the skeletal chain (see Figure 7). The resulting agent would then react to a sudden force first at the root level, with springy motion propagating up the skeleton toward each limb’s tip, providing convincing reaction and overlap of motion (see Video 6). This term refers to interpolating different animation data to create a new animation that’s a mix of the original data.2 In Massive, this is simply called a blend and is useful for modifying animation cycles in crowd simulation. References 1. D. Ryu and P. Kanyuk, “Rivers of Rodents: An AnimationCentric Crowds Pipeline for Ratatouille,” Tech. Memo 07-02, Pixar, 2007. 2. A. Bruderlin and L. Williams, “Motion Signal Processing,” Proc. Siggraph, ACM Press, 1995, pp. 97–104. Agent rotates around x-axis Moving forward Large negative translational acceleration Physics Driving Acting The climactic sequence in WALL•E involved a scene with complex physical interaction between thousands of characters and their environment (we couldn’t fake this one with signal processing). In particular, the spaceship Axiom’s human passengers fall from their levitating chairs, bounce off one another and their remaining chairs, which pivot like bar stools, to collect on the ship’s side. It initially seemed best to model the agents’ overall motion with a rigid-body simulation. Accordingly, we integrated a rigid-body simulation engine into our pipeline (see the “Open Dynamics Engine” sidebar). This provided our characters’ root transformations, on top of which we layered motion cycles and procedural animation. To do so, Figure 3. An agent moving forward comes to a stop and leans forward. Note how the “large negative translational acceleration” signal causes rotation around the x-axis. To set up this relationship, we use a brain spring. IEEE Computer Graphics and Applications 27 Virtual Populace Translational velocity in z Brain spring Rx offset Open Dynamics Engine T (Lean) Translational acceleration in z Brain spring Rx offset Brain spring Rz offset (Lean bounce) Rotational velocity around y he rigid-body simulator we integrated into our pipeline is the freely available Open Dynamics Engine (ODE). For the tilting simulation, we represented the ship geometry with a smoothed height map, and the agents as combinations of a cube and sphere. The simple collision bodies yielded more-stable simulations than arbitrary meshes yield. For a demo of a simplified rigid-body simulation, see Video 7 at http://computingnow.computer.org/wall-e. (Banking) Rotational acceleration around y Brain spring Rz offset (Banking bounce) Figure 4. Relationships between physical inputs and animation outputs using brain springs. Rx stands for rotation around the x-axis; Rz stands for rotation around the z-axis. Altogether, the figure shows four springs: two to drive leaning and two to drive banking. bspring ( x ) = ∑ filter (x , w , d ) ∗ a i i i i ∈[0 ,n ] a is a set of n amplitudes with monotonically decreasing magnitudes and alternating signs—for example, n = 4, a = [-2, 1, -0.5, 0.25]. w is a set of n filter widths with monotonically increasing values—for example, n = 4, w = [0.5, 1, 2, 4]. d is a set of n delays with monotonically increasing values—for example, n = 4, d = [0, 0.1, 0.2, 0.4]. (a) First bounce Third bounce Fourth bounce Second bounce Rapid negative acceleration (complete stop) (b) Figure 5. Brain springs: (a) the mathematical formulation and (b) the corresponding graph of an example, which provides a more intuitive illustration. The brain spring filters the original signal (red) four times to create two oscillations. 28 July/August 2009 we used the same brain springs that we designed for reaction. In addition to creating simple reaction rotations, the springs’ outputs drove motion blending between custom animation cycles (for more details, see “Motion Blending” in the “Massive and WALL• ­E” sidebar). So, the physical forces measured by the springs appeared to influence the character’s acting. For instance, for a single acting clip—“panicked flailing”—our animators produced a version under the influence of forces from the six axis directions (see Video 8). Then, on the basis of the translational acceleration’s magnitude and direction, the characters blended their flailing accordingly (see Figure 8). That is, if an agent received a negative z translational velocity, its animation clip was blended with the version being pushed backward, on the basis of the signal filtered through a brain spring. For example, Video 9 shows an agent being pushed backward in a simple simulation without blending animation clips. Video 10 shows that same simulation with translational velocity driving a motion blend with the clips shown in Figure 8, and is noticeably more realistic. This let us use the rigid-body simulation’s results to influence our agents’ acting, bringing the crowd to life. Once we simulated this motion, we applied a soft-body simulation to the close-up characters requiring it. For information on a workflow demo for the climactic scene in WALL•E, see the “Shot Progression” sidebar. Results We developed these fake-physics techniques to create motion for crowd simulation that was compelling, easy to control, and cheap. You can judge for yourself by watching WALL•E, but we’ve also provided a short demonstration of our crowd robots in flight (see Video 17). The robots clearly aren’t just following a path but are hovering, anticipating their turns, and reacting to physical forces in a spring-like manner. Bottom heavy— stiff spring Second bounce Moving forward Bounce 2 First bounce Bounce 1 Large negative translational acceleration A1: 180 A2: –40 Rapid negative acceleration F1: 0.25 F2: 0.50 Moving forward Second bounce Bounce 1 Large negative translational acceleration A1: –160 A2: 80 Brain springs are also remarkably easy to control, tune, and adapt for novel tasks. At one point in the film’s production, the director asked for a group of crowd robots to come to a complete stop and bounce, yet begin their bounce before the stop for exaggeration. This would be impossible for a traditional physics simulation because the desired result isn’t physically plausible. However, with brain springs, this is a small modification. Rather than use the real accelerations to drive the brain springs, we painted a custom texture map on the ground in Massive, containing bright pixels where the agents should begin to bounce. The agents queried this texture map and remapped its information into the brain springs to trigger early bounces (see Video 18). So, brain springs can also filter nonphysics signals, allowing for highly customizable behavior. As for speed and interactivity, we tested performance by running several 1,000-frame, 1,000-agent robot flight simulations with and without brain springs (four springs as described in Figure 4). With brain springs, the time increase was 287 seconds on average, giving a perframe, per-agent cost of 0.000287 seconds. This cost was negligible compared to the other simulation aspects (for example, posing, ray casting, and OpenGL drawing time). So, brain springs proved highly interactive. D2: 0.250 First bounce Top heavy— loose spring Bounce 2 D1: 0.125 Rapid negative acceleration F1: 0.50 F2: 1.0 D1: 0.250 D2: 0.500 W e believe this research can easily be generalized to crowd simulation tasks other than robots for WALL•E. Characters with more complicated skeletal hierarchies, such as animals, could benefit from springy motion overlap in limbs using the skeletal-chain extension of brain springs. Also, hovering or driving characters, common in interactive games, could benefit from our techniques’ cheap cost and ease of implementation. Potentially, brain springs could find use outside computer graphics as a general filter for arbitrary signals, although we expect the most common applications will be crowds and games. Acknowledgments I thank Pixar Animation Studios and Massive Software for support. References 1. J. Lasseter, “Principles of Traditional Animation Applied to 3D Computer Animation,” Proc. Siggraph, 1987, ACM Press, pp. 35–44. 2. C.W. Reynolds, “Flocks, Herds and Schools: A Distributed Behavioral Model,” Proc. Siggraph, ACM Press, 1987, pp. 25–34. 3. J. Wang et al., “The Cartoon Animation Filter,” ACM IEEE Computer Graphics and Applications Figure 6. Two brain spring configurations and the resulting effect on a translating flying agent quickly stopping. A1, F1, and D1 are the amplitude scale, filter width, and delay of the first half oscillation; A2, F2, and D2 are those for the second half oscillation. F and D values are in seconds. (a) A “bottomheavy stiff spring” effect. The agent first leans forward when stopping (indicating the center of mass is in the agent’s lower half), then quickly leans back with low magnitude and high frequency. (b) A “topheavy loose spring” effect. The agent first leans back (bottom swings forward) when stopping (indicating the center of mass in the agent’s upper half), then slowly leans forward with high magnitude and low frequency. 29 Virtual Populace Brain spring 2× Clavicle Rx offset Brain spring 4× Shoulder Rx offset Translational acceleration in z Brain spring 8× Elbow Rx offset Translational acceleration in z Brain spring 16× Hand Rx offset Translational acceleration in z Translational acceleration in z Elbow (8× filter width and delay) Hand: (16× ...) Shoulder (4× filter width and delay) Clavicle (2× filter width and delay) Large negative translational acceleration Moving forward Figure 7. Modeling a robot limb’s overlapping motion: (a) a flowchart showing how the input motion data is related to animation outputs through brain springs and (b) the corresponding schematic. The skeletal hierarchy (blue) experiences a large negative translational acceleration, and the brain springs (orange) applied to each joint rotate the bones. The larger filter widths and delays cause each successive joint to react more gradually to the force, creating overlapping motion in the limb. Trans. Graphics (Proc. Siggraph), vol. 25, no. 3, 2006, pp. 1169–1173. Blend into upward-impact animation clip bspring Acceleration downward Acceleration right bspring bspring Blend into left-impact animation clip Acceleration left bspring Acceleration upward Blend into right-impact animation clip Blend into downward-impact animation clip Figure 8. How forces from each direction drive motion blends. In practice, this let us give the impression that each agent’s acting was being influenced by physical forces. Paul Kanyuk is a technical director at Pixar Animation Studios, with credits on the films Cars, Ratatouille, WALL•E, and Up. His specialty is crowd simulation, and he’s responsible for the procedural animation of numerous crowd spectacles, including the hordes of rats in Ratatouille, the deluge of falling passengers in WALL•E, and the vicious pack of dogs in Up. He also teaches courses in RenderMan and crowd simulation at the Academy of Art University in San Francisco. Kanyuk has a BS Eng. in digital media design from the University of Pennsylvania. He’s a member of ACM Siggraph and Tau Beta Pi. Contact him at [email protected] Shot Progression T o show how our physics-based acting workflow looks in action, six videos at http:// computingnow.computer.org/wall-e present a rough shot progression for the scene in WALL•E where the Axiom tilts and the passengers slide across the deck (see the section “Physics Driving Acting” in the main article). Video 11 shows part of the rigid-body simulation used to generate 30 July/August 2009 the final animation. Video 12 shows the acting of individual agent types independent of the rigidbody simulation. Video 13 shows the noisy simulation with a subset of the final agents with acting layered on top. Video 14 shows the agent positions after filtering and smoothing. Video 15 shows the unlit shot with final animation and cleanup. Video 16 shows the final result with lighting.