Game Programming Gems 8

Copyright

Game Programming Gems 8 Edited by Adam Lake Publisher and General Manager, Course Technology PTR: Stacy L. Hiquet Associate Director of Marketing: Sarah Panella Manager of Editorial Services: Heather Talbot Marketing Manager: Jordan Castellani Senior Acquisitions Editor: Emi Smith Project and Copy Editor: Cathleen D. Small Interior Layout: Shawn Morningstar Cover Designer: Mike Tanamachi CD-ROM Producer: Brandon Penticuff Indexer: Katherine Stimson Proofreader: Heather Urschel © 2011 Course Technology, a part of Cengage Learning. ALL RIGHTS RESERVED. No part of this work covered by the copyright herein may be reproduced, transmitted, stored, or used in any form or by any means graphic, electronic, or mechanical, including but not limited to photocopying, recording, scanning, digitizing, taping, Web distribution, information networks, or information storage and retrieval systems, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the publisher. For product information and technology assistance, contact us at Cengage Learning Customer & Sales Support, 1-800-354-9706 For permission to use material from this text or product, submit all requests online at cengage.com/permissions Further permissions questions can be emailed to

For product information and technology assistance, contact us at Cengage Learning Customer & Sales Support, 1-800-354-9706 [email protected]

All trademarks are the property of their respective owners. Cover image used courtesy of Valve Corporation. All other images © Cengage Learning unless otherwise noted. Library of Congress Control Number: 2010920327 ISBN-10: 1-58450-702-0 eISBN-10: 1-43545-771-4 Course Technology, a part of Cengage Learning 20 Channel Center Street Boston, MA 02210 USA Cengage Learning is a leading provider of customized learning solutions with office locations around the globe, including Singapore, the U nited Kingdom, Australia, Mexico, Brazil, and Japan. Locate your local office at: international.cengage.com/region Cengage Learning products are represented in Canada by Nelson Education, Ltd. For your lifelong learning solutions, visit courseptr.com Visit our corporate website at cengage.com Printed in the United States of America 1 2 3 4 5 6 7 12 1 1 10

Preface Welcome to the eighth edition of the Game Programming Gems series, started by Mark DeLoura in 2000. The first edition was inspired by Andrew Glassner‘s popular Graphics Gems series. Since then, other Gems series have started, including AI Gems and a new series focused on the capabilities of programmable graphics, the ShaderX series. These tomes serve as an opportunity to share our experience and best practices with the rest of the industry. Many readers think of the Game Programming Gems series as a collection of articles with sections that target specialists. For me, I‘ve read through them as a way to get exposure to the diverse subsystems used to create games and stay abreast of the latest techniques. For example, I may not be a specialist in networking, but reading this section will often enlighten and stimulate connections that I may not have made between areas in which I have expertise and ones in which I do not.

One statement I‘ve heard recently regarding our industry is the idea that we now have all the horsepower we need to create games, so innovations by hardware companies are not needed. I believe this argument is flawed in many ways. First, there are continued advancements in graphical realism in academia, in R&D labs, and in the film industry that have yet to be incorporated into our real-time pipelines. As developers adopt these new features, computational requirements of software will continue to increase. Second, and the more important issue, is that this concept of play isn‘t entirely correct—the very notion of what gaming serves from an anthropological perspective. Play is fundamental, not just to the human condition, but to the sentient condition. We invent interactive experiences on any platform, be it a deck of cards, a set of cardboard cutouts, or a next-gen PC platform with multi-terabyte data and multi-threaded, multi-gigahertz, multi-processor environments. It‘s as natural as the pursuit of food. This play inspires real-world applications and pushes the next generation of platform requirements. It enables affordability of ever-increased computational horsepower in our computing platforms. The extension of gaming into other arenas, mobile and netbook platforms, serves only to prove the point. While the same ideas and themes may be used in these environments, the experience available to the player is different if the designer is to leverage the full capabilities and differentiating features of the platform. There is an often-chanted ―ever increasing cost of game development‖ quote for console and PC platforms. In the same breath, it‘s alluded that this spiral of cost cannot continue. I believe these issues are of short-term concern. If there is a community willing to play, our economies will figure out a way to satisfy those needs. This will open up new opportunities for venture capital and middleware to reduce those platform complexities and cross-industry development costs, fueling the next generation of interactive experiences. I do believe the process has changed and will continue to evolve, but game development will continue to thrive. Will there be 15 first-person military simulations on a single platform? Perhaps not, but will there continue to be compelling multiplayer and single-player experiences? I believe so. The ingenuity of the game developer, when brought to the task of leveraging new incarnations of silicon, will continue to create enriching interactive experiences for everincreasing audiences. Finally, I‘d like to take a moment to address another issue often mentioned in the press. In November 2009, the Wall Street Journal ran an article by Jonathan V. Last from the Weekly Standard discussing the social implications of gaming. The majority of his article, ―Videogames—Not Only for the Lonely,‖ was making this observation in the context of a holiday gathering of family members of many generations sharing experiences with their Nintendo Wii. Near the end of the article, he refers to the fact that ―the shift to videogames might be lamenting if it meant that people who would otherwise be playing mini-golf or Monopoly were sealing themselves off and playing Halo 3 death matches across the Internet.‖ Much to the contrary, I have personally spent many quality multiplayer hours interacting socially with longtime friends when playing multiplayer games. A few days ago, I was having a conversation with an acquaintance who was thrilled that she could maintain her relationship with her brother on the East Coast by playing World of Warcraft with him. Ultimately, whether we are discussing our individual game experiences with others or interacting directly while playing, games do what they have always done across generations and platforms—they bring us together with shared experiences, whether it be cardboard cutouts, a deck of cards, or multiplayer capture the flag. Despite the overall informed message of the article, the writer encouraged a myth I see repeated in the mainstream press by those out of touch with the multiplayer, socially interactive game experiences that are common today, including Halo 3.

Overview of Content The graphics section in this edition covers several topics of recent interest, leveraging new features of graphics APIs such as Compute Shader, tessellation using DirectX 11, and two gems on the implementation details of Screen Space Ambient Occlusion (SSAO). In the

physics and animation section, we have selected a number of gems that advance beyond the basics of the topics such as IK solvers or fluid simulation in general. Instead, these gems go deeper with improvements to existing published techniques based on real-world experience with the current state of the art—for example, a simple, fast, and accurate IK solver, leveraging swarm systems for animation, and modeling air and fluid resistance. Artificial intelligence, AI, is one of the hottest areas in game development these days. Game players want worlds that don‘t just look real, but that also feel and act real. The acting part is the responsibility of the AI programmer. Gems in the AI section are diverse, covering areas such as decision making, detailed character simulation, and player modeling to solve the problem of gold farm detection. The innovations discussed are sure to influence future gems. In the general programming section, we have a number of tools to help with the development, performance, and testing of our game engines. We include gems that deal with multi-threading using Intel‘s Thread Building Blocks, an open-source multithreading library, memory allocation and profiling, as well as a useful code coverage system used by the developers at Crytek. The gems in the networking and multiplayer section cover architecture, security, scalability, and the leveraging of social networking applications to create multiplayer experiences. The audio section had fewer submissions than in past years. Why is this? Is the area of audio lacking in innovation? Has it matured to the point where developers are buying offthe-shelf components? Regardless, we‘ve assembled a collection of gems for audio that we think will be of interest. In one of the articles in the audio section, we discuss a relatively new idea—the notion of real-time calculation of the audio signal based on the actual physics instead of using the traditional technique of playing a pre-recorded processed sound. As games become more interactive and physics driven, there will be a corresponding demand for more realistic sound environments generated by such techniques enabled with the increasing computational horsepower Moore‘s Law continues to deliver to game developers. I‘m excited to introduce a new section in this edition of Game Programming Gems 8 that I‘m calling ―General Purpose Computing on GPUs.‖ This is a new area for the Gems series, and we wanted to have a real-world case study of a game developer using the GPU for nongraphics tasks. We‘ve collected three gems for this section. The first is about OpenCL, a new open standard for programming heterogeneous platforms of today, and we also have two gems that leverage PhysX for collision detection and fluid simulation. The PhysX components were used in Batman: Arkham Asylum by Rocksteady Studios Ltd. As the computing capabilities of the platform evolve, I expect game developers will face the decision of what to compute, where to compute, and how to manage the data being operated upon. These articles serve as case studies in what others have done in their games. I expect this to be an exciting area of future development. While we all have our areas of specialty, I think it‘s fair to say game developers are a hungry bunch, with a common desire to learn, develop, and challenge ourselves and our abilities. These gems are meant to insprire, enlighten, and evolve the industry. As always, we look forward to the contributions and feedback developers have when putting these gems into practice. Adam Lake [email protected]

About the Cover Image © Valve Corporation

The cover of Game Programming Gems 8 features the Engineer from Valve‘s Team Fortress 2. With their follow-up to the original class-based multiplayer shooter Team Fortress, Valve chose to depart from the typical photorealistic military themes of the genre. Instead, they employed an ―illustrative‖ non-photorealistic rendering style, reminiscent of American commercial illustrators of the 1920s. This was motivated by the need for players to be able to quickly visually identify each other‘s team, class, and weapon choices in the game. The novel art style and rendering techniques of Team Fortress 2 allowed Valve‘s designers to visually separate the character classes from each other and from the game‘s environments through the use of strong silhouettes and strategic distribution of color value.

CD-ROM Downloads If you purchased an ebook version of this book, and the book had a companion CD-ROM, we will mail you a copy of the disc. Please send [email protected] the title of the book, the ISBN, your name, address, and phone number. Thank you.

Acknowledgments I‘d like to take a moment to acknowledge the section editors that I worked with to create this tome. They are the best and brightest in the industry. The quality of submissions and content in this book is a testament to this fact. They worked incredibly hard to bring this book together, and I thank them for their time and expertise. Also, I appreciate the time and patience that Emi Smith and Cathleen Small at Cengage Learning have put into this first-time book editor. They were essential in taking care of all the details necessary for publication. Finally, I‘d like to acknowledge the artists at Valve who provided the cover image for this edition of Game Programming Gems. I have been blessed to have had exposure to numerous inspirational individuals— friends who refused to accept norms, parents who satiated my educational desires, teachers willing to spend a few extra minutes on a random tangent, instructors to teach not just what we know about the world, but also to make me aware of the things we do not. Most importantly, I want to acknowledge my wife, Stacey Lake, who remained supportive while I toiled away in the evenings and weekends for the better part of a year on this book. I dedicate these efforts to my mother, Amanda Lake. I thank her for teaching me that education is an enjoyable lifelong endeavor.

Contributors Full bios for those contributors who submitted one can be found at www.courseptr.com/ downloads. Contributors to this book include: Dr. Doug Binks, D.Phil. Udeepta Bordoloi Igor Borovikov Cyril Brom Eric Brown Phil Carlisle Michael Dailly Peter Dalton Kevin Dill Jean-Francois Dube Dominic Filion Marco Fratarcangeli Nico Galoppo Benedict R. Gaster Gero Gerber Robert Jay Gould Neil Gower Joshua Grass, Ph.D. Hunter Hale Mark Harris Thomas Hartley Kevin He Claus Höfele Allen Hux Peter Iliev

Matthew Jack Aleksey Kadukin Nikhil S. Ketkar Hyunwoo Ki Adam Lake Michael Lewin Chris Lomont, Ph.D. Ricky Lung Khaled Mamou Dave Mark Quasim Mehdi Krzysztof Mieloszyk Jason Mitchell Ben Nicholson Ian Ni-Lewis Mat Noguchi Borut Pfeifer Brian Pickrell Tomas Poch Steve Rabin Mike Ramsey B. Charles Rasco, Ph.D. João Lucas G. Raza Aurelio Reis Zhimin Ren Marc Romankewicz Dario Sancho Rahul Sathe

Simon Schirm Brian Schmidt Ondřej Šerý Philip Taylor Richard Tonge Steven Tovey Gabriel Ware Ben Wyatt G. Michael Youngblood Jason Zink Robert Zubek

Section 1: Graphics Introduction Fast Font Rendering with Instancing Principles and Practice of Screen Space Ambient Occlusion Multi-Resolution Deferred Shading View Frustum Culling of Catmull-Clark Patches in DirectX 11 Ambient Occlusion Using DirectX Compute Shader Eye-View Pixel Anti-Aliasing for Irregular Shadow Mapping Overlapped Execution on Programmable Graphics Hardware Techniques for Effective Vertex and Fragment Shading on the SPUs

Introduction Jason Mitchell, Valve [email protected] In this edition of the Game Programming Gems series, we explore a wide range of important real-time graphics topics, from lynchpin systems such as font rendering to cutting-edge hardware architectures, such as Larrabee, PlayStation 3, and the DirectX 11 compute shader. Developers in the trenches at top industry studios such as Blizzard, id,

Bizarre Creations, Nexon, and Intel‘s Advanced Visual Computing group share their insights on optimally exploiting graphics hardware to create high-quality visuals for games. To kick off this section, Aurelio Reis of id Software compares several methods for accelerating font rendering by exploiting GPU instancing, settling on a constant-bufferbased method that achieves the best performance. We then move on to two chapters discussing the popular image-space techniques of Screen Space Ambient Occlusion (SSAO) and deferred shading. Dominic Filion of Blizzard Entertainment discusses the SSAO algorithms used in StarCraft II, including novel controls that allowed Blizzard‘s artists to tune the look of the effect to suit their vision. Hyunwoo Ki of Nexon then describes a multi-resolution acceleration method for deferred shading that computes low-frequency lighting information at a lower spatial frequency and uses a novel method for handling high-frequency edge cases. For the remainder of the section, we concentrate on techniques that take advantage of the very latest graphics hardware, from DirectX 11‘s tessellator and compute shader to Larrabee and the PlayStation 3. Rahul Sathe of Intel presents a method for culling of Bezier patches in the context of the new DirectX 11 pipeline. Jason Zink then describes the new DirectX 11 compute shader architecture, using Screen Space Ambient Occlusion as a case study to illustrate the novel aspects of this new hardware architecture. In a pair of articles from Intel, Nico Galoppo and Allen Hux describe a method for integrating anti-aliasing into the irregular shadow mapping algorithm as well as a software task system that allows highly programmable systems such as Larrabee to achieve maximum throughput on this type of technique. We conclude the section with Steven Tovey‘s look at the SPU units on the PlayStation 3 and techniques for achieving maximum performance in the vehicle damage and light pre-pass rendering systems in the racing game Blur from Bizarre Creations.

1.1. Fast Font Rendering with Instancing Aurelio Reis, id Software [email protected] Font rendering is an essential component of almost all interactive applications, and while techniques exist to allow for fully scalable vector-based font rendering using modern GPUs, the so-called ―bitmap font‖ is still the most versatile, efficient, and easy-to-implement solution. When implemented on typical graphics APIs, however, this technique uses runtime updated vertex buffers to store per-glyph geometry, resulting in inefficient rendering performance by potentially stalling the graphics pipeline. By leveraging efficient particle system rendering techniques that were developed previously, it is possible to render thousands of glyphs in a single batch without ever touching the vertex buffer. In this article, I propose a simple and efficient method to render fonts utilizing modern graphics hardware when compared to other similar methods. This technique is also useful in that it can be generalized for use in rendering other 2D elements, such as sprites and graphical user interface (GUI) elements.

Text-Rendering Basics The most common font format is the vector-based TrueType format. This format represents font glyphs (in other words, alphabetic characters and other symbols) as vector data, specifically, quadratic Bezier curves and line segments. As a result, TrueType fonts are compact, easy to author, and scale well with different display resolutions. The downside of a vector font, however, is that it is not straightforward to directly render this type of data on

graphics hardware. There are, however, a few different ways to map the vector representation to a form that graphics hardware can render. One way is to generate geometry directly from the vector curves, as shown in Figure 1.1.1. However, while modern GPUs are quite efficient at rendering large numbers of triangles, the number of polygons generated from converting a large number of complex vector curves to a triangle mesh could number in the tens of thousands. This increase in triangle throughput can greatly decrease application performance. Some optimizations to this way of rendering fonts have been introduced, such as the technique described by Loop and Blinn in which the polygonal mesh consists merely of the curve control points while the curve pixels are generated using a simple and efficient pixel shader [Loop05]. While this is a great improvement over the naive triangulation approach, the number of polygons generated in this approach is still prohibitively high on older graphics hardware (and that of the current console generation—the target of this article). Figure 1.1.1. Vector curves converted into polygonal geometry.

Because of these limitations, the most common approach relies on rasterizing vector graphics into a bitmap and displaying each glyph as a rectangle composed of two triangles (from here on referred to as a quad), as shown in Figure 1.1.2. A font texture page is generated with an additional UV offset table that maps glyphs to a location in that texture very similar to how a texture atlas is used [NVIDIA04]. The most obvious drawback is the resolution dependence caused by the font page being rasterized at a predefined resolution, which leads to distortion when rendering a font at a non-native resolution. Additional techniques exist to supplement this approach with higher quality results while mitigating the resolution dependence that leads to blurry and aliased textures, such as the approach described by [Green07]. Overall, the benefits of the raster approach outweigh the drawbacks, because rendering bitmap fonts is incredibly easy and efficient. Figure 1.1.2. A font page and a glyph rendered on a quad.

To draw glyphs for a bitmap font, the program must bind the texture page matching the intended glyph set and draw a quad for each glyph, taking into account spacing for kerning or other character-related offsets. While this technique yields very good performance, it can still be inefficient, as the buffers containing the geometry for each batch of glyphs must be continually updated. Constantly touching these buffers is a sure way to cause GPU stalls,

resulting in decreased performance. For text- or GUI-heavy games, this can lead to an unacceptable overall performance hit.

Improving Performance One way to draw the glyphs for the GUI is to create a GUI model that maintains buffers on the graphics card for drawing a predefined maximum number of indexed triangles as quads. Whenever a new glyph is to be drawn, its quad is inserted into a list, and the vertex buffer for the model is eventually updated with the needed geometry at a convenient point in the graphics pipeline. When the time comes to render the GUI model, assuming the same texture page is used, only a single draw call is required. As previously mentioned, this buffer must be updated each frame and for each draw batch that must be drawn. Ideally, as few draw batches as possible are needed, as the font texture page should contain all the individual glyphs that would need to be rendered, but on occasion (such as for highresolution fonts or Asian fonts with many glyphs), it‘s not possible to fit them all on one page. In the situation where a font glyph must be rendered from a different page, the batch is broken and must be presented immediately so that a new one can be started with the new texture. This holds true for any unique rendering states that a glyph may hold, such as blending modes or custom shaders. Lock-Discard The slowest part of the process is when the per-glyph geometry must be uploaded to the graphics card. Placing the buffer memory as close to AGP memory as possible (using API hints) helps, but locking and unlocking vertex buffers can still be quite expensive. To alleviate the expense, it is possible to use a buffer that is marked to ―discard‖ its existing buffer if the GPU is currently busy with it. By telling the API to discard the existing buffer, a new one is created, which can be written to immediately. Eventually, the old buffer is purged by the API under the covers. This use of lock-discard prevents the CPU from waiting on the GPU to finish consuming the buffer (for example, in the case where it was being rendered at the same time). You can specify this with the D3DLOCK_DISCARD flag in Direct3D or by passing a NULL pointer to glBufferDataARB and then calling glMapBufferARB(). Be aware that although this is quite an improvement, it is still not an ideal solution, as the entire buffer must be discarded. Essentially, this makes initiating a small update to the buffer impossible. Vertex Compression Another step in improving performance is reducing the amount of memory that needs to be sent to the video card. The vertex structure for sending a quad looks something like this and takes 28 bytes per vertex (and 112 bytes for each quad):

struct GPU_QUAD_VERTEX_POS_TC_COLOR { D3DXVECTOR4 Position; D3DXVECTOR2 Texcoord; D3DCOLOR Color; };

Since the bandwidth across the AGP bus to the video card is not infinite, it is important to be aware of how much memory is being pushed through it. One way to reduce the memory costs is to use an additional vertex stream to update only that information that has changed on a per-frame basis. Unfortunately, the three essential quad attributes (position, texture

dimensions, and color) could be in a state of constant flux, so there is little frame-to-frame coherency we can exploit. There is one very easy way to reduce at least some of the data that must be sent to the video card, however. Traditionally, each vertex represents a corner of a quad. This is not ideal, because this data is relatively static. That is, the size and position of a quad changes, but not the fact that it is a quad. Hicks describes a shader technique that allows for aligning a billboarded quad toward the screen by storing a rightFactor and upFactor for each corner of the billboard and projecting those vertices along the camera axes [Hicks03]. This technique is attractive, as it puts the computation of offsetting the vertices on the GPU and potentially limits the need for vertex buffer locks to update the quad positions. By using a separate vertex stream that contains unique data, it is possible to represent the width and height of the quad corners as a 4D unsigned byte vector. (Technically, you could go as small as a Bool if that was supported on modern hardware.) In the vertex declaration, it is possible to map the position information to specific vertex semantics, which can then be accessed directly in the vertex shader. The vertex structure would look something like this:

struct GPU_QUAD_VERTEX { BYTE OffsetXY[ 4 ]; };

Although this may seem like an improvement, it really isn‘t, since the same amount of memory must be used to represent the quad attributes (more so since we‘re supplying a 4byte offset now). There is an easy way to supply this additional information without requiring the redundancy of all those additional vertices.

Instancing Quad Geometry If you‘re lucky enough to support a Shader Model 3 profile, you have hardware support for some form of geometry instancing. OpenGL 2.0 has support for instancing using pseudoinstancing [GLSL04] and the EXT_draw_instanced [EXT06] extension, which uses the glDrawArraysInstancedEXT and glDrawElementsInstancedEXT routines to render up to 1,024 instanced primitives that are referenced via an instance identifier in shader code. As of DirectX 9, Direct3D also supports instancing, which can be utilized by creating a vertex buffer containing the instance geometry and an additional vertex buffer with the perinstance data. By using instancing, we‘re able to completely eliminate our redundant quad vertices (and index buffer) at the cost of an additional but smaller buffer that holds only the per-instance data. This buffer is directly hooked up to the vertex shader via input semantics and can be easily accessed with almost no additional work to the previous method. While this solution sounds ideal, we have found that instancing actually comes with quite a bit of per-batch overhead and also requires quite a bit of instanced data to become a win. As a result, it should be noted that performance does not scale quite so well and in some situations can be as poor as that of the original buffer approach (or worse on certain hardware)! This is likely attributed to the fact that the graphics hardware must still point to this data in some way or another, and while space is saved, additional logic is required to compute the proper vertex strides.

Constant Array Instancing Another way to achieve similar results with better performance is to perform shader instancing using constant arrays. By creating a constant array for each of the separate quad attributes (in other words, position/size, texture coordinate position/size, color), it is possible to represent all the necessary information without the need for a heavyweight vertex structure. See Figure 1.1.3. Figure 1.1.3. A number of glyphs referencing their data from a constant array.

Similar to indexed vertex blending (a.k.a. matrix palette skinning), an index is assigned for each group of four vertices required to render a quad, as shown in Figure 1.1.4. To get the value for the current vertex, all that is needed is to index into the constant array using this value. Because the number of constants available is usually below 256 on pre–Shader Model 4 hardware, this index can be packed directly as an additional element in the vertex offset vector (thus requiring no additional storage space). It‘s also possible to use geometry instancing to just pass in the quad ID/index in order to bypass the need for a large buffer of four vertices per quad. However, as mentioned previously, we have found that instancing can be unreliable in practice. Figure 1.1.4. A quad referencing an element within the attribute constant array.

This technique yields fantastic performance but has the downside of only allowing a certain number of constants, depending on your shader profile. The vertex structure is incredibly compact, weighing in at a mere 4 bytes (16 bytes per quad) with an additional channel still available for use:

struct GPU_QUAD_VERTEX { BYTE OffsetXY_IndexZ[ 4 ]; };

Given the three quad attributes presented above and with a limit of 256 constants, up to 85 quads can be rendered per batch. Despite this limitation, performance can still be quite a bit better than the other approaches, especially as the number of state changes increases (driving up the number of batches and driving down the number of quads per batch).

Additional Considerations I will now describe some small but important facets of font rendering, notably an efficient use of clip-space position and a cheap but effective sorting method. Also, in the sample code for this chapter on the book‘s CD, I have provided source code for a texture atlasing solution that readers may find useful in their font rendering systems. Sorting

Fonts are typically drawn in a back-to-front fashion, relying on the painter‘s algorithm to achieve correct occlusion. Although this is suitable for most applications, certain situations may require that quads be layered in a different sort order than that in which they were drawn. This is easily implemented by using the remaining available value in the vertex structure offset/index vector as a z value for the quad, allowing for up to 256 layers. Clip-Space Positions To save a few instructions and the constant space for the world-view-projection matrix (the clip matrix), it‘s possible to specify the position directly in clip-space to forego having to transform the vertices from perspective to orthographic space, as illustrated in Figure 1.1.5. Clip-space positions range from –1 to 1 in the X and Y directions. To remap an absolute screen-space coordinate to clip space, we can just use the equation [cx = –1 + x * (2 / screen_width)], [cy = 1 – y * (2 / screen_height)], where x and y are the screen-space coordinates up to a max of screen_width and screen_height, respectively. Figure 1.1.5. A quad/billboard being expanded.

Texture Atlasing On the book‘s CD, I have provided code for a simple virtual texture system that uses atlases to reduce batches. This system attempts to load an atlased version of a texture if possible and otherwise loads a texture directly from disk. There are some switches (documented in the code) that demonstrate how to turn this system on and off to demonstrate how important it can be toward reducing the number of batches and maintaining a high level of performance.

Future Work

The techniques demonstrated in this chapter were tailored to work on current console technology, which is limited to Shader Model 3. In the future, I would like to extend these techniques to take advantage of new hardware features, such as Geometry Shaders and StreamOut, to further increase performance, image fidelity, and ease of use.

Demo On the accompanying disc, you‘ll find a Direct3D sample application that demonstrates each of the discussed techniques in a text- and GUI-rich presentation. Two scenes are presented: One displays a cityscape for a typical 2D tile-based game, and the other displays a Strange Attractor simulation. In addition, there is an option to go overboard with the text rendering. Feel free to play around with the code until you get a feel for the strengths and weaknesses of the different approaches. The main shader file (Font.fx) contains the shaders of interest as well as some additional functionality (such as font anti-aliasing/filtering). Please note that certain aspects (such as quad expansion) were made for optimum efficiency and not necessarily readability. In general, most of the code was meant to be very accessible, and it will be helpful to periodically cross-reference the files GuiModel.cpp and Font.fx.

Conclusion In this gem, I demonstrated a way to render font and GUI elements easily and efficiently by taking advantage of readily available hardware features, such as instancing, multiple stream support, and constant array indexing. As a takeaway item, you should be able to easily incorporate such a system into your technology base or improve an existing system with only minor changes.

References [EXT06] ―EXT_draw_instanced.‖ 2006. Open GL. n.d. . [GLSL04] ―GLSL Pseudo-Instancing.‖ 17 Nov. 2004. NVIDIA. n.d. . [Green07] Green, Chris. ―Improved Alpha-Tested Magnification for Vector Textures and Special Effects.‖ Course on Advanced Real-Time Rendering in 3D Graphics and Games. SIGGRAPH 2007. San Diego Convention Center, San Diego, CA. 8 August 2007. [Hicks03] Hicks, O‘Dell. ―Screen-aligned Particles with Minimal VertexBuffer Locking.‖ ShaderX2: Shader Programming Tips and Tricks with DirectX 9.0. Ed. Wolfgang F. Engel. Plano, TX: Wordware Publishing, Inc., 2004. 107–112. [Loop05] Loop, Charles and Jim Blinn. ―Resolution Independent Curve Rendering Using Programmable Graphics Hardware.‖ 2005. Microsoft. n.d. .

[NVIDIA04] ―Improve Batching Using Texture Atlases.‖ 2004. NVIDIA. n.d. .

1.2. Principles and Practice of Screen Space Ambient Occlusion Dominic Filion, Blizzard Entertainment [email protected] Simulation of direct lighting in modern video games is a well-understood concept, as virtually all of real-time graphics has standardized on the Lambertian and Blinn models for simulating direct lighting. However, indirect lighting (also referred to as global illumination) is still an active area of research with a variety of approaches being explored. Moreover, although some simulation of indirect lighting is possible in real time, full simulation of all its effects in real time is very challenging, even on the latest hardware. Global illumination is based on simulating the effects of light bouncing around a scene multiple times as light is reflected on light surfaces. Computational methods such as radiosity attempt to directly model this physical process by modeling the interactions of lights and surfaces in an environment, including the bouncing of light off of surfaces. Although highly realistic, sophisticated global illumination methods are typically too computationally intensive to perform in real time, especially for games, and thus to achieve the complex shadowing and bounced lighting effects in games, one has to look for simplifications to achieve a comparable result. One possible simplification is to focus on the visual effects of global illumination instead of the physical process and furthermore to aim at a particular subset of effects that global illumination achieves. Ambient occlusion is one such subset. Ambient occlusion simplifies the problem space by assuming all indirect light is equally distributed throughout the scene. With this assumption, the amount of indirect light hitting a point on a surface will be directly proportional to how much that point is exposed to the scene around it. A point on a plane surface can receive light from a full 180-degree hemisphere around that point and above the plane. In another example, a point in a room‘s corner, as shown in Figure 1.2.1, could receive a smaller amount of light than a point in the middle of the floor, since a greater amount of its ―upper hemisphere‖ is occluded by the nearby walls. The resulting effect is a crude approximation of global illumination that enhances depth in the scene by shrouding corners, nooks, and crannies in a scene. Artistically, the effect can be controlled by varying the size of the hemisphere within which other objects are considered to occlude neighboring points; large hemisphere ranges will extend the shadow shroud outward from corners and recesses. Figure 1.2.1. Ambient occlusion relies on finding how much of the hemisphere around the sampling point is blocked by the environment.

Although the global illumination problem has been vastly simplified through this approach, it can still be prohibitively expensive to compute in real time. Every point on every scene surface needs to cast many rays around it to test whether an occluding object might be blocking the light, and an ambient occlusion term is computed based on how many rays were occluded from the total amount of rays emitted from that point. Performing arbitrary ray intersections with the full scene is also difficult to implement on graphics hardware. We need further simplification.

Screen Space Ambient Occlusion What is needed is a way to structure the scene so that we can quickly and easily determine whether a given surface point is occluded by nearby geometry. It turns out that the standard depth buffer, which graphics engines already use to perform hidden surface removal, can be used to approximate local occlusion [Shanmugam07, Mittring07]. By definition, the depth buffer contains the depth of every visible point in the scene. From these depths, we can reconstruct the 3D positions of the visible surface points. Points that can potentially occlude other points are located close to each other in both screen space and world space, making the search for potential occluders straightforward. We need to align a hemisphere around each point‘s upper hemisphere as defined by its normal. We will thus need a normal buffer that will encode the normal of every corresponding point in the depth buffer in screen space. Rather than doing a full ray intersection, we can simply inspect the depths of neighboring points to establish the likelihood that each is occluding the current point. Any neighbor whose 2D position does not fall within the 2D coverage of the hemisphere could not possibly be an occluder. If it does lie within the hemisphere, then the closer the neighbor point‘s depth is to the target point, the higher the odds it is an occluder. If the neighbor‘s depth is behind the point being tested for occlusion, then no occlusion is assumed to occur. All of these calculations can be performed using the screen space buffer of normals and depths, hence the name Screen Space Ambient Occlusion (SSAO). At first glance, this may seem like a gross oversimplification. After all, the depth buffer doesn‘t contain the whole scene, just the visible parts of it, and as such is only a partial reconstruction of the scene. For example, a point in the background could be occluded by an object that is hidden behind another object in the foreground, which a depth buffer would

completely miss. Thus, there would be pixels in the image that should have some amount of occlusion but don‘t due to the incomplete representation we have of the scene‘s geometry. Figure 1.2.2. SSAO Samples neighbor points to discover the likelihood of occlusion. Lighter arrows are behind the center point and are considered occluded samples.

It turns out that these kinds of artifacts are not especially objectionable in practice. The eye focuses first on cues from objects within the scene, and missing cues from objects hidden behind one another are not as disturbing. Furthermore, ambient occlusion is a lowfrequency phenomenon; what matters more is the general effect rather than specific detailed cues, and taking shortcuts to achieve a similar yet incorrect effect is a fine tradeoff in this case. Discovering where the artifacts lie should be more a process of rationalizing the errors than of simply catching them with the untrained eye. From this brief overview, we can outline the steps we will take to implement Screen Space Ambient Occlusion. We will first need to have a depth buffer and a normal buffer at our disposal from which we can extract information. From these screen space maps, we can derive our algorithm. Each pixel in screen space will generate a corresponding ambient occlusion value for that pixel and store that information in a separate render target. For each pixel in our depth buffer, we extract that point‘s position and sample n neighboring pixels within the hemisphere aligned around the point‘s normal. The ratio of occluding versus non-occluding points will be our ambient occlusion term result. The ambient occlusion render target can then be blended with the color output from the scene generated afterward. I will now describe our Screen Space Ambient Occlusion algorithm in greater detail.

Generating the Source Data The first step in setting up the SSAO algorithm is to prepare the necessary incoming data. Depending on how the final compositing is to be done, this can be accomplished in one of two ways. The first method requires that the scene be rendered twice. The first pass will render the depth and normal data only. The SSAO algorithm can then generate the ambient occlusion output in an intermediate step, and the scene can be rendered again in full color. With this

approach, the ambient occlusion map (in screen space) can be sampled by direct lights from the scene to have their contribution modulated by the ambient occlusion term as well, which can help make the contributions from direct and indirect lighting more coherent with each other. This approach is the most flexible but is somewhat less efficient because the geometry has to be passed to the hardware twice, doubling the API batch count and, of course, the geometry processing load. A different approach is to render the scene only once, using multiple render targets bound as output to generate the depth and normal information as the scene is first rendered without an ambient lighting term. SSAO data is then generated as a post-step, and the ambient lighting term can simply be added. This is a faster approach, but in practice artists lose the flexibility to decide which individual lights in the scene may or may not be affected by the ambient occlusion term, should they want to do so. Using a fully deferred renderer and pushing the entire scene lighting stage to a post-processing step can get around this limitation to allow the entire lighting setup to be configurable to use ambient occlusion per light. Whether to use the single-pass or dual-pass method will depend on the constraints that are most important to a given graphics engine. In all cases, a suitable format must be chosen to store the depth and normal information. When supported, a 16-bit floating-point format will be the easiest to work with, storing the normal components in the red, green, and blue components and storing depth as the alpha component. Screen Space Ambient Occlusion is very bandwidth intensive, and minimizing sampling bandwidth is necessary to achieve optimal performance. Moreover, if using the single-pass multi-render target approach, all bound render targets typically need to be of the same bit depth on the graphics hardware. If the main color output is 32-bit RGBA, then outputting to a 16-bit floating-point buffer at the same time won‘t be possible. To minimize bandwidth and storage, the depth and normal can be encoded in as little as a single 32-bit RGBA color, storing the x and y components of the normal in the 8-bit red and green channels while storing a 16-bit depth value in the blue and alpha channels. The HLSL shader code for encoding and decoding the normal and depth values is shown in Listing 1.2.1. Listing 1.2.1. HLSL code to decode the normal on subsequent passes as well as HLSL code used to encode and decode the 16-bit depth value

// Normal encoding simply outputs x and y components in R and G in // the range 0.. 1 float3 DecodeNormal( float2 cInput ) { float3 vNormal.xy = 2.0f * cInput.rg - 1.0f; vNormal.z = sqrt(max(0, 1 - dot(vNormal.xy, vNormal.xy))); return vNormal; } // Encode depth to B and A float2 DepthEncode( float fDepth ) { float2 vResult; // Input depth must be mapped to 0..1 range fDepth = fDepth / p_fScalingFactor; // R = Basis = 8 bits = 256 possible values // G = fractional part with each 1/256th slice vResult.ba = frac( float2( fDepth, fDepth * 256.0f )); return vResult;

} float3 DecodeDepth( float4 cInput ) { return dot ( cInput.ba, float2( 1.0f, 1.0f / 256.0f ) * p_fScalingFactor; }

Sampling Process With the input data in hand, we can begin the ambient occlusion generation process itself. At any visible point on a surface on the screen, we need to explore neighboring points to determine whether they could occlude our current point. Multiple samples are thus taken from neighboring points in the scene using a filtering process described by the HLSL shader code in Listing 1.2.2. Listing 1.2.2. Screen Space Ambient Occlusion filter described in HLSL code

// i_VPOS is screen pixel coordinate as given by HLSL VPOS interpolant. // p_vSSAOSamplePoints is a distribution of sample offsets for each sample. float4 PostProcessSSAO( float 3 i_VPOS ) { float2 vScreenUV; //←This will become useful later. float3 vViewPos = 2DToViewPos( i_VPOS, vScreenUV); half fAccumBlock = 0.Of; for (inti = 0; i < iSampleCount; i++ ) { float3 vSamplePointDelta = p_vSSAOSamplePoints[i]; float fBlock = TestOcclusion( vViewPos, vSamplePointDelta, p_fOcclusionRadius, p_fFullOcclusionThreshold, p_fNoOcclusionThreshold, p_fOcclusionPower ) ) fAccumBlock += fBlock; } fAccumBlock / = iSampleCount; return 1.Of - fAccumBlock; }

We start with the current point, p, whose occlusion we are computing. We have the point‘s 2D coordinate in screen space. Sampling the depth buffer at the corresponding UV coordinates, we can retrieve that point‘s depth. From these three pieces of information, the

3D position of the point within can be reconstructed using the shader code shown in Listing 1.2.3. Listing 1.2.3. HLSL shader code used to map a pixel from screen space to view space

// vRecipDepthBufferSize = 1.0 / depth buffer width and height in pixels. // p_vCameraFrustrumSize = Full width and height of camera frustum at the // camera's near plane in world space. float2 p_vRecipDepthBufferSize; float2 p_vCameraFrustrumSize; float3 2DPosToViewPos( float3 i_VPOS, out float2 vScreenUV ) { float2 vViewSpaceUV = i_VPOS * p_vRecipDepthBufferSize; vScreenUV = vViewSpaceUV; // From 0..1 vViewSpaceUV // From 0..2 vViewSpaceUV vViewSpaceUV

to to 0..2 = vViewSpaceUV * float2( 2.Of, -2.Of ); to-1..1 = vViewSpaceUV + float2( -1.0f , 1.Of ); = vViewSpaceUV * p_vCameraFrustrumSize * 0.5f;

return float3( vViewSpaceUV.x, vViewSpaceUV.y, 1.0f ) * tex2D( p_sDepthBuffer, vScreenUV ).r; }

We will need to sample the surrounding area of the point p along multiple offsets from its position, giving us n neighbor positions qi. Sampling the normal buffer will give us the normal around which we can align our set of offset vectors, ensuring that all sample offsets fall within point p‘s upper hemisphere. Transforming each offset vector by a matrix can be expensive, and one alternative is to perform a dot product between the offset vector and the normal vector at that point and to flip the offset vector if the dot product is negative, as shown in Figure 1.2.3. This is a cheaper way to solve for the offset vectors without doing a full matrix transform, but it has the drawback of using fewer samples when samples are rejected due to falling behind the plane of the surface of the point p. Figure 1.2.3. Samples behind the hemisphere are flipped over to stay within the hemisphere.

Each neighbor‘s 3D position can then be transformed back to screen space in 2D, and the depth of the neighbor point can be sampled from the depth buffer. From this neighboring depth value, we can establish whether an object likely occupies that space at the neighbor point. Listing 1.2.4 shows shader code to test for this occlusion. Listing 1.2.4. HLSL code used to test occlusion by a neighboring pixel

float TestOcclusion( float3 vViewPos, float3 vSamplePointDelta, float fOcclusionRadius, float fFullOcclusionThreshold, float fNoOcclusionThreshold, float fOcclusionPower ) { float3 vSamplePoint = vViewPos + fOcclusionRadius * vSamplePointDelta; float2 vSamplePointUV; vSamplePointUV = vSamplePoint.xy / vSamplePoint.z; vSamplePointUV = vSamplePointUV / p_vCameraSize / 0.5f; vSamplePointUV = vSamplePointUV + float2( 1.0f, -1.0f ); vSamplePointUV = vSamplePointUV * float2( 0.5f, -0.5f ); float fSampleDepth = tex2D( p_sDepthBuffer, vSamplePointUV ).r; float fDistance = vSamplePoint.z - fSampleDepth; return OcclusionFunction( fDistance, fFullOcclusionThreshold, fNoOcclusionThreshold, fOcclusionPower ); }

We now have the 3D positions of both our point p and the neighboring points qi. We also have the depth di of the frontmost object along the ray that connects the eye to each neighboring point. How do we determine ambient occlusion? The depth di gives us some hints as to whether a solid object occupies the space at each of the sampled neighboring points. Clearly, if the depth di is behind the sampled point‘s depth, it cannot occupy the space at the sampled point. The depth buffer does not give us the thickness of the object along the ray from the viewer; thus, if the depth of the object is anywhere in front of p, it may occupy the space, though without thickness information, we can‘t know for sure. We can devise some reasonable heuristics with the information we do have and use a probabilistic method. The further in front of the sample point the depth is, the less likely it is to occupy that space. Also, the greater the distance between the point p and the neighbor point, the lesser the occlusion, as the object covers a smaller part of the hemisphere. Thus, we can derive some occlusion heuristics based on: The difference between the sampled depth di and the depth of the point qi The distance between p and qi For the first relationship, we can formulate an occlusion function to map the depth deltas to occlusion values. If the aim is to be physically correct, then the occlusion function should be quadratic. In our case we are more concerned about being able to let our artists adjust the occlusion function, and thus the occlusion function can be arbitrary. Really, the occlusion function can be any function that adheres to the following criteria: Negative depth deltas should give zero occlusion. (The occluding surface is behind the sample point.) Smaller depth deltas should give higher occlusion values. The occlusion value needs to fall to zero again beyond a certain depth delta value, as the object is too far away to occlude. For our implementation, we simply chose a linearly stepped function that is entirely controlled by the artist. A graph of our occlusion function is shown in Figure 1.2.4. There is a full-occlusion threshold where every positive depth delta smaller than this value gets complete occlusion of one, and a no-occlusion threshold beyond which no occlusion occurs. Depth deltas between these two extremes fall off linearly from one to zero, and the value is exponentially raised to a specified occlusion power value. If a more complex occlusion function is required, it can be pre-computed in a small ID texture to be looked up on demand. Figure 1.2.4. SSAO blocker function.

Listing 1.2.5. HLSL code used to implement occlusion function

float OcclusionFunction( float fDistance, float fNoOcclusionThreshold, float fFullOcclusionThreshold, float fOcclusionPower ) { const c_occlusionEpsilon = 0.01f; if ( fDistance > c_ occlusionEpsilon ) { // Past this distance there is no occlusion. float fNoOcclusionRange = fNoOcclusionThreshold fFullOcclusionThreshold; if ( fDistance < fFullOcclusionThreshold ) return 1.0f; else return max(1.0f – pow(( ( fDistance – fFullOcclusionThreshold ) / fNoOcclusionRange, fOcclusionPower ) ) ,0.0f ); } else return 0.0f; }

Once we have gathered an occlusion value for each sample point, we can take the average of these, weighted by the distance of each sample point to p, and the average will be our ambient occlusion value for that pixel.

Sampling Randomization

Sampling neighboring pixels at regular vector offsets will produce glaring artifacts to the eye, as shown in Figure 1.2.5. Figure 1.2.5. SSAO without random sampling.

To smooth out the results of the SSAO lookups, the offset vectors can be randomized. A good approach is to generate a 2D texture of random normal vectors and perform a lookup on this texture in screen space, thus fetching a unique random vector per pixel on the screen, as illustrated in Figure 1.2.6 [Mittring07]. We have n neighbors we must sample, and thus we will need to generate a set of n unique vectors per pixel on the screen. These will be generated by passing a set of offset vectors in the pixel shader constant registers and reflecting these vectors through the sampled random vector, resulting in a semirandom set of vectors at each pixel, as illustrated by Listing 1.2.6. The set of vectors passed in as registers is not normalized—having varying lengths helps to smooth out the noise pattern and produces a more even distribution of the samples inside the occlusion hemisphere. The offset vectors must not be too short to avoid clustering samples too close to the source point p. In general, varying the offset vectors from half to full length of the occlusion hemisphere radius produces good results. The size of the occlusion hemisphere becomes a parameter controllable by the artist that determines the size of the sampling area. Figure 1.2.6. Randomized sampling process.

Listing 1.2.6. HLSL code used to generate a set of semi-random 3D vectors at each pixel

float3 reflect( float 3 vSample, float3 vNormal ) { return normalize ( vSample – 2.0f * dot( vSample, vNormal ) * vNormal ); } float3x3 MakeRotation( float fAngle, float3 vAxis ) { float fS; float fC; sincos( fAngle, fS, fC ); float fXX = vAxis.x * vAxis.x; float fYY = vAxis.y * vAxis.y; float fZZ = vAxis.z * vAxis.z; float fXY = vAxis.x * vAxis.y; float fYZ = vAxis.y * vAxis.z; float fZX = vAxis.z * vAxis.x; float fXS = vAxis.x * fS; float fYS = vAxis.y * fS; float fZS = vAxis.z * fS; float fOneC = 1.0f - fC; float3x3 result = float3x3( fOneC * fXX + fC, fOneC * fXY + fZS, fOneC * fZX fYS, fOneC * fXY - fZS,

fOneC * fYY +

fC, fOneC * fYZ +

fOneC * fZX + fYS,

fOneC * fYZ - fXS, fOneC * fZZ +

fXS, ); return result; }

fC

float4 PostProcessSSAO( float3 i_VPOS ) { ... const float c_scalingConstant = 256.0f; float3 vRandomNormal = ( normalize( tex2D( p_sSSAONoise, vScreenUV * p_vSrcImageSize / c_scalingConstant ).xyz * 2.0f – 1.0f ) ); float3x3 rotMatrix = MakeRotation( 1.0f,vNormal ); half fAccumBlock = 0.0f; for ( int i = 0; i < iSampleCount; i++ ) { float3 vSamplePointDelta = reflect( p_vSSAOSamplePoints[i], vRandomNormal ); float fBlock = TestOcclusion( vViewPos, vSamplePointDelta, p_fOcclusionRadius, p_fFullOcclusionThreshold, p_fNoOcclusionThreshold, p_fOcclusionPower ) ) { fAccumBlock += fBlock; } ... }

Ambient Occlusion Post-Processing As shown in Figure 1.2.7, the previous step helps to break up the noise pattern, producing a finer-grained pattern that is less objectionable. With wider sampling areas, however, a further blurring of the ambient occlusion result becomes necessary. The ambient occlusion results are low frequency, and losing some of the high-frequency detail due to blurring is generally preferable to the noisy result obtained by the previous steps. Figure 1.2.7. SSAO term after random sampling applied. Applying blur passes will further reduce the noise to achieve the final look.

To smooth out the noise, a separable Gaussian blur can be applied to the ambient occlusion buffer. However, the ambient occlusion must not bleed through edges to objects that are physically separate within the scene. A form of bilateral filtering is used. This filter samples the nearby pixels as a regular Gaussian blur shader would, yet the normal and depth for each of the Gaussian samples are sampled as well. (Encoding the normal and depth in the same render targets presents significant advantages here.) If the depth from the Gaussian sample differs from the center tap by more than a certain threshold, or the dot product of the Gaussian sample and the center tap normal is less than a certain threshold value, then the Gaussian weight is reduced to zero. The sum of the Gaussian samples is then renormalized to account for the missing samples. Listing 1.2.7. HLSL code used to blur the ambient occlusion image

// i_UV : UV of center tap // p_fBlurWeights Array of gaussian weights // i_GaussianBlurSample: Array of interpolants, with each interpolants // packing 2 gaussian sample positions. float4 PostProcessGaussianBlur( VertexTransport vertOut ) { float2 vCenterTap = i_UV.xy; float4 cValue = tex2D( p_sSrcMap, vCenterTap.xy ); float4 cResult = cValue * p_fBlurWeights[0]; float fTotalWeight = p_fBlurWeights[0]; // Sample normal & depth for center tap float4 vNormalDepth = tex2D( p_sNormalDepthMap, vCenterTap.xy ).a; for ( int i = 0; i < b_iSampleInterpolantCount; i++ ) { half4 cValue = tex2D( p_sSrcMap, i_GaussianBlurSample[i].xy ); half fWeight = p_fBlurWeights[i * 2 + 1];

float4 vSampleNormalDepth = tex2D( p_sNormalDepthMap, i_GaussianBlurSample[i].xy ); if ( dot( vSampleNormalDepth.rgb, vNormalDepth.rgb) < 0.9f || abs( vSampleNormalDepth.a – vNormalDepth.a ) > 0.01f ) fWeight = 0.0f; cResult += cValue * fWeight; fTotalWeight += fWeight; cValue = tex2D( p_sSeparateBlurMap, INTERPOLANT_GaussianBlurSample[i].zw ) ; fWeight = p_fBlurWeights[i * 2 + 2]; vSampleNormalDepth = tex2D( p_sSrcMap, INTERPOLANT_GaussianBlurSample[i].zw ) ; if ( dot( vSampleNormalDepth.rgb, vNormalDepth .rgb < 0.9f ) || abs( vSampleNormalDepth.a – vNormalDepth.a ) > 0.01f ) fWeight = 0.0f; cResult += cValue * fWeight; fTotalWeight += fWeight; } // Rescale result according to number of discarded samples. cResult *= 1.0f / fTotalWeight; return cResult; }

Several blur passes can thus be applied to the ambient occlusion output to completely eliminate the noisy pattern, trading off some higher-frequency detail in exchange. Figure 1.2.8. Result of Gaussian blur.

Handling Edge Cases The offset vectors are in view space, not screen space, and thus the length of the offset vectors will vary depending on how far away they are from the viewer. This can result in using an insufficient number of samples at close-up pixels, resulting in a noisier result for these pixels. Of course, samples can also go outside the 2D bounds of the screen. Naturally, depth information outside of the screen is not available. In our implementation, we ensure that samples outside the screen return a large depth value, ensuring they would never occlude any neighboring pixels. This can be achieved through the ―border color‖ texture wrapping state, setting the border color to a suitably high depth value. To prevent unacceptable breakdown of the SSAO quality in extreme close-ups, the number of samples can be increased dynamically in the shader based on the distance of the point p to the viewer. This can improve the quality of the visual results but can result in erratic performance. Alternatively, the 2D offset vector lengths can be artificially capped to some threshold value regardless of distance from viewer. In effect, if the camera is very close to an object and the SSAO samples end up being too wide, the SSAO area consistency constraint is violated so that the noise pattern doesn‘t become too noticeable.

Optimizing Performance Screen Space Ambient Occlusion can have a significant payoff in terms of mood and visual quality of the image, but it can be quite an expensive effect. The main bottleneck of the algorithm is the sampling itself. The semi-random nature of the sampling, which is necessary to minimize banding, wreaks havoc with the GPU‘s texture cache system and can become a problem if not managed. The performance of the texture cache will also be very dependent on the sampling area size, with wider areas straining the cache more and yielding poorer performance. Our artists quickly got in the habit of using SSAO to achieve a

faked global illumination look that suited their purposes. This required more samples and wider sampling areas, so extensive optimization became necessary for us. One method to bring SSAO to an acceptable performance level relies on the fact that ambient occlusion is a low-frequency phenomenon. Thus, there is generally no need for the depth buffer sampled by the SSAO algorithm to be at full-screen resolution. The initial depth buffer can be generated at screen resolution, since the depth information is generally reused for other effects, and it potentially has to fit the size of other render targets, but it can thereafter be downsampled to a smaller depth buffer that is a quarter size of the original on each side. The downsampling itself does have some cost, but the payback in improved throughput is very significant. Downsampling the depth buffer also makes it possible to convert it from a wide 16-bit floating-point format to a more bandwidth-friendly 32-bit packed format.

Fake Global Illumination and Artistic Styling If the ambient occlusion hemisphere is large enough, the SSAO algorithm eventually starts to mimic behavior seen from general global illumination; a character relatively far away from a wall could cause the wall to catch some of the subtle shadowing cues a global illumination algorithm would detect. If the sampling area of the SSAO is wide enough, the look of the scene changes from darkness in nooks and crannies to a softer, ambient feel. This can pull the art direction in two somewhat conflicting directions: on the one hand, the need for tighter, high-contrast occluded zones in deeper recesses, and on the other hand, the desire for the larger, softer, ambient look of the wide-area sampling. One approach is to split the SSAO samples between two different sets of SSAO parameters: Some samples are concentrated in a small area with a rapidly increasing occlusion function (generally a quarter of all samples), while the remaining samples use a wide sampling area with a gentler function slope. The two sets are then averaged independently, and the final result uses the value from the set that produces the most (darkest) occlusion. This is the approach that was used in StarCraft II. Figure 1.2.9. SSAO with different sampling-area radii.

The edge-enhancing component of the ambient occlusion does not require as many samples as the global illumination one, thus a quarter of the samples can be assigned to crease enhancement while the remainder are assigned for the larger area threshold. Though SSAO provides for important lighting cues to enhance the depth of the scene, there was still a demand from our artist for more accurate control that was only feasible through the use of some painted-in ambient occlusion. The creases from SSAO in particular cannot reach the accuracy that using a simple texture can without using an enormous amount of samples. Thus the usage of SSAO does not preclude the need for some static ambient occlusion maps to be blended in with the final ambient occlusion result, which we have done here. Figure 1.2.10. Combined small- and large-area SSAO result.

For our project, complaints about image noise, balanced with concerns about performance, were the main issues to deal with for the technique to gain acceptance among our artists. Increasing SSAO samples helps improve the noise, yet it takes an ever-increasing number of samples to get ever smaller gains in image quality. Past 16 samples, we‘ve found it‘s more effective to use additional blur passes to smooth away the noise pattern, at the expense of some loss of definition around depth discontinuities in the image.

Transparency It should be noted the depth buffer can only contain one depth value per pixel, and thus transparencies cannot be fully supported. This is generally a problem with all algorithms that rely on screen space depth information. There is no easy solution to this, and the SSAO process itself is intensive enough that dealing with edge cases can push the algorithm outside of the real-time realm. In practice, for the vast majority of scenes, correct ambient occlusion for transparencies is a luxury that can be skimped on. Very transparent objects will typically be barely visible either way. For transparent objects that are nearly opaque, the choice can be given to the artist to allow some transparencies to write to the depth buffer input to the SSAO algorithm (not the z-buffer used for hidden surface removal), overriding opaque objects behind them.

Final Results Color Plate 1 shows some results portraying what the algorithm contributes in its final form. The top-left pane shows lighting without the ambient occlusion, while the top-right pane shows lighting with the SSAO component mixed in. The final colored result is shown in the bottom pane. Here the SSAO samples are very wide, bathing the background area with an effect that would otherwise only be obtained with a full global illumination algorithm. The SSAO term adds depth to the scene and helps anchor the characters within the environment. Color Plate 2 shows the contrast between the large-area, low-contrast SSAO sampling component on the bar surface and background and the tighter, higher-contrast SSAO samples apparent within the helmet, nooks, and crannies found on the character‘s spacesuit.

Conclusion This gem has described the Screen Space Ambient Occlusion technique used at Blizzard and presented various problems and solutions that arise. Screen Space Ambient Occlusion offers a different perspective in achieving results that closely resemble what the eye expects from ambient occlusion. The technique is reasonably simple to implement and amenable to artistic tweaks in real time to make it ideal to fit an artistic vision.

References

[Bavoil] Bavoil, Louis and Miguel Sainz. ―Image-Space Horizon-Based Ambient Occlusion.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. Section 6.2. [Bavoil09] Bavoil, Louis and Miguel Sainz. ―Multi-Layer Dual-Resolution Screen-Space Ambient Occlusion.‖ 2009. NVIDIA. n.d. . [Bavoil08] Bavoil, Louis and Miguel Sainz. ―Screen Space Ambient Occlusion.‖ Sept. 2008. NVIDIA. n.d. . [Fox08] Fox, Megan. ―Ambient Occlusive Crease Shading.‖ Game Developer. March 2008. [Kajalin] Kajalin, Vladimir. ―Screen Space Ambient Occlusion.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. Section 6.1. [Lajzer] Lajzer, Brett and Dan Nottingham. ―Combining Screen-Space Ambient Occlusion and Cartoon Rendering on Graphics Hardware.‖ n.d. Brett Lajzer. n.d.. [Luft06] Luft, Thomas, Carsten Colditz, and Oliver Deussen. ―Image Enhancement by Unsharp Masking the Depth Buffer.‖ Course on Non-Photorealistic Rendering. SIGGRAPH 2006. Boston Convention and Exhibition Center, Boston, MA. 3 August 2006. [Mittring07] Mittring, Martin. ―Finding Next Gen—CryEngine 2.0.‖ Course on Advanced RealTime Rendering in 3D Graphics and Games. SIGGRAPH 2007. San Diego Convention Center, San Diego, CA. 8 August 2007. [Pesce] Pesce, Angelo. ―Variance Methods for Screen-Space Ambient Occlusion.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. Section 6.7. [Ritschel09] Ritschel, Tobias, Thorsten Grosch, and Hans-Peter Seidel. ―Approximating Dynamic Global Illumination in Image Space.‖ 2009. Max Planck Institut Informatik. n.d. . [Sains08] Sains, Miguel. ―Real-Time Depth Buffer Based Ambient Occlusion.‖ Game Developers Conference. Moscone Center, San Francisco, CA. 18–22 February 2008. [Shamugan07] Shanmugam, Perumaal and Okan Arikan. ―Hardware Accelerated Ambient Occlusion Techniques on GPUs.‖ 2007. Google Sites. n.d.. [Sloan07] Sloan, Peter-Pike, Naga K. Govindaraju, Derek Nowrouzezahrai, and John Snyder. ―Image-Based Proxy Accumulation for Real-Time Soft Global Illumination.‖ Pacific Graphics Conference. The Royal Lahaina Resort, Maui, Hawaii. 29 October 2007. [Tomasi98] Tomasi, Carlo and Roberto Manduchi. ―Bilateral Filtering for Gray and Color Images.‖ IEEE International Conference on Computer Vision. Homi Bhabha Auditorium, Bombay, India. 7 January 1998.

1.3. Multi-Resolution Deferred Shading Hyunwoo Ki, INNOACE Co., Ltd [email protected] Recently, deferred shading has become a popular rendering technique for real-time games. Deferred shading enables game engines to handle many local lights without repeated geometry processing because it replaces geometry processing with pixel processing [Saito90, Shishkovtsov05, Valient07, Koonce07, Engel09, Kircher09]. In other words, shading costs are independent of geometric complexity, which is important as the CPU cost of scene-graph traversal and the GPU cost of geometry processing grows with scene complexity. Despite this decoupling of shading cost from geometric complexity, we still seek to optimize the pixel processing necessary to handle many local lights, soft shadows, and other per-pixel effects. In this gem, we present a technique that we call multi-resolution deferred shading, which provides adaptive sub-sampling using a hierarchical approach to shading by exploiting spatial coherence of the scene. Multi-resolution deferred shading efficiently reduces pixel shading costs as compared to traditional deferred shading without noticeable aliasing. As shown in Figure 1.3.1, our technique allows us to achieve a significant improvement in performance with negligible visual degradation relative to a more expensive full-resolution deferred shading approach. Figure 1.3.1. Deferred shading (left: 20 fps), multi-resolution deferred shading (center: 38 fps), and their difference image (right). There are 40 spot lights, including fuzzy shadows (1024×1024 pixels with 24 shadow samples per pixel).

Deferred Shading Unlike traditional forward rendering approaches, deferred shading costs are independent of scene complexity. This is because deferred shading techniques store geometry information in textures, often called G-buffers, replacing geometry processing with pixel processing [Saito90, Shishkovtsov05, Valient07, Koonce07]. Deferred shading techniques start by rendering the scene into a G-buffer, which is typically implemented using multiple render targets to store geometry information, such as positions, normals, and other quantities instead of final shading results. Next, deferred shading systems render a screen-aligned quad to invoke a pixel shader at all pixels in the output image. The pixel shader retrieves the geometry information from the G-buffer and performs shading operations as a post process. Naturally, one must carefully choose the data formats and precise quantities to store in a G-buffer in order to make the best possible use of both memory and memory bandwidth. For example, the game Killzone 2 utilizes four buffers containing lighting accumulation and intensity, normal XY in 16-bit floating-point format, motion vector XY, specular and diffuse albedo, and sun occlusion [Valient07]. The Z component of the normal is computed from normal XY, and position is computed from depth

and pixel coordinates. These types of encodings are a tradeoff between decode/encode cost and the memory and memory bandwidth consumed by the G-buffer. As shown in Color Plate 3, we simply use two four-channel buffers of 16-bit floating-point precision per channel without any advanced encoding schemes for ease of description and implementation. The first of our buffers contains view-space position in the RGB channels and a material ID in the alpha channel. The other buffer contains view-space normal in the RGB channels and depth in the alpha channel. We could also use material buffers that store diffuse reflectance, specular reflectance, shininess, and so on. However, material buffers are not necessary if we separate lighting and material phases from the shading phase using light pre-pass rendering [Engel09]. Unlike traditional deferred shading, light pre-pass rendering first computes lighting results instead of full shading. This method can then incorporate material properties in an additional material phase with forward rendering. Although this technique requires a second geometry rendering pass, such separation of lighting and material phases gives added flexibility during material shading and is compatible with hardware multi-sample antialiasing. A related technique, inferred lighting, stores lighting results in a single lowresolution buffer instead of the full-resolution buffer [Kircher09]. To avoid discontinuity problems, this technique filters edges using depth and object ID comparison in the material phase. As we will describe in the next section, our technique is similar to inferred lighting, but our method finds discontinuous areas based on spatial proximity and then solves the discontinuity problems using a multi-resolution approach during the lighting (or shading) phase.

Multi-Resolution Deferred Shading Although deferred shading improves lighting efficiency, computing illumination for every pixel is still expensive, despite the fact that it is often fairly low frequency. We have developed a multi-resolution deferred shading approach to exploit the low-frequency nature of illumination. We perform lighting in a lower-resolution buffer for spatially coherent areas and then interpolate results into a higher-resolution buffer. This key concept is based upon our prior work [Ki07a]. Here, we generalize this work and improve upon it to reduce aliasing. The algorithm has three steps, as shown in Color Plate 4: geometry pass, multi-resolution rendering pass, and composite pass. The geometry pass populates the G-buffers. Our technique is compatible with any sort of G-buffer organization, but for ease of explanation, we will stick with the 8-channel G-buffer layout described previously. The next step is multi-resolution rendering, which consists of resolution selection (non-edge detection), shading (lighting), and interpolation (up-sampling). We allocate buffers to store rendering results at various resolutions. We call these buffers R-buffers, where the ―R‖ stands for ―Result‖ or ―Resolution.‖ In this chapter, we will use three R-buffers: full resolution, quarter resolution, and 1/16th resolution (for example, 1280×1024, 640×512, and 320×256). If the full-resolution image is especially high, we could choose to decrease the resolutions of the R-buffers even more drastically than just one-quarter resolution in each step. Multi-resolution rendering uses rendering iterations from lower-resolution to higherresolution R-buffers. We prevent repeated pixel processing by exploiting early-Z culling to skip pixels processed in earlier iterations using lower-resolution R-buffers [Mitchell04]. To start shading our R-buffers, we set the lowest-resolution R-buffer as the current render target and clear its depth buffer with one depth (farthest). Next, we determine pixels being rendered in this resolution by rendering a screen-aligned quad with Zi = 1.0 – i * 0.1, where i is the current iteration, writing only depth. During this pass, the pixel shader reads geometry information from mip-mapped versions of our G-buffers and estimates spatial

proximity for non-edge detection. To estimate spatial proximity, we first compare the current pixel‘s material ID with the material IDs of neighboring pixels. Then, we compare the difference of normal and depth values using tunable thresholds. If spatial proximity is low for the current pixel, we should use a higher-resolution R-buffer for better quality, and thus we discard the current pixel in the shader to skip writing Z. After this pass, pixels whose spatial proximity is high (in other words, non-edge) in the current resolution contain meaningful Z values because they were not discarded. The pixels whose spatial proximity is low (in other words, edges) still have farthest Z values left over from the initial clear. We then perform shading (or lighting) by rendering a screen-aligned quad with Zi = 1.0 – i * 0.1 again, but the Z function is changed to Equal. This means that only spatially coherent pixels in this resolution will pass the Z-test, as illustrated in Color Plate 4. In the pixel shader, we read geometric data from G-buffers and compute illumination as in light prepass rendering. On a textured surface, such as wall and floor, although spatial proximity between neighboring pixels is high, these pixel colors are often different. Such cases can cause serious aliasing in the resulting images. To solve this problem, we store only lighting results instead of full shading results into R-buffers, and we handle material properties with stored illumination in R-buffers in the composite pass. After shading, we copy the current shading/lighting results and depth to the next higherresolution R-buffer, allowing the hardware‘s bilinear units to do a simple interpolation as we up-sample. We have found that bilinear filtering is adequate, though we could use bi-cubic filtering or other higher-order filtering for better quality. We repeat the process described above at the next higher resolution, estimating spatial proximity and writing Z and computing illumination until we reach the full-resolution Rbuffer. A full-screen quad is drawn three times per iteration. If a given pixel was shaded on a prior iteration in a lower-resolution R-buffer, that pixel is not shaded again at the higher resolution due to early-Z culling. In this way, we are able to perform our screen-space shading operations at the appropriate resolution for different regions of the screen. In Figure 1.3.2, we visualize the distribution of pixels shaded at each level of our hierarchy. Figure 1.3.2. Visualization of hierarchical pixel processing.

Non-black pixels were shaded in the first pass at 1/16th resolution as in the image on the left. The middle image shows the pixels shaded in the second iteration at one-quarter resolution, and only the pixels in the image on the right were shaded at full image resolution.

Because this approach exploits image scaling from low resolution to high resolution with interpolation, discontinuity artifacts can appear at boundaries of lighting or shadows. We address this issue during the multi-resolution rendering phase. We write 1.0 to the alpha channel of R-buffer pixels that are lit; otherwise, we write zero. If pixels are lit by the same lights (or the same number of lights), their neighbors‘ alpha values will be equal. Therefore, we interpolate these pixels to a higher-resolution buffer. Otherwise, we consider these pixels within the boundary, and thus we discard them in the interpolation pass (see Figure 1.3.3). We can handle shadow boundaries similarly.

Figure 1.3.3. A boundary-check algorithm. If a pixel is lit by a light, we add one alpha for this pixel in the lighting phase. In the interpolation pass, we consider pixels that are in boundaries whose neighbor pixels’ alpha values are different to others, and thus we use a higher-resolution buffer without interpolation.

If shadow color is neither zero nor one (in other words, penumbra), we also set a pixel alpha to zero and thus discard it in the interpolation work. In the composite pass, we render a screen-aligned quad, reading shading results from the full-resolution R-buffer and material properties such as albedo to compute the final shading result. We could draw scene geometry instead of drawing a screen quad for MSAA, similar to light pre-pass rendering. In contrast to traditional deferred shading and light pre-pass rendering, multi-resolution deferred shading reduces rendering costs for low-frequency pixels. Our multi-resolution deferred shading is also more efficient than inferred lighting due to the hierarchical approach. Multi-resolution deferred shading can also be used for other rendering techniques, such as the GPU-based light clustering technique for diffuse interreflection and subsurface light diffusion called Light Pyramids [Ki08]. The Light Pyramids technique stores first-bounced lights in shadow maps and groups them by considering their angular and spatial similarity. Although such light clustering dramatically reduces the number of lights, it still requires hundreds of lights for each pixel. Figure 1.3.4 shows an example of a combination of Light Pyramids and multi-resolution deferred shading. Thanks to our pixel clustering, we achieved a performance improvement of approximately 1.5 to 2.0 times without noticeable quality loss. As pixel processing increases in complexity—for example, using higher resolution or using more lights—the relative performance improvement also increases. Figure 1.3.4. Indirect illumination using Light Pyramids [Ki08] based on traditional deferred shading (left) and multi-resolution deferred shading (right: 1.7 times faster).

Conclusion and Future Work We have presented a multi-resolution deferred shading technique that performs lighting and shading computations at appropriate screen-space frequency in order to improve the efficiency of deferred shading without aliasing. In the future, we would also like to develop even more efficient resolution-selection algorithms, and we also seek to handle a wider variety of surface reflection models. We also hope to integrate transparent rendering of inferred lighting into our method. We believe that our method could be applied for not only lighting but also other rendering operations with high per-pixel overhead, such as per-pixel displacement mapping [Ki07b].

References [Engel09] Engel, Wolfgang. ―Designing a Renderer for Multiple Lights: The Light Pre-Pass Renderer.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. 655-666. [Ki08] Ki, Hyunwoo. ―A GPU-Based Light Hierarchy for Real-Time Approximate Illumination.‖ The Visual Computer 24.7–9 (July 2008): 649–658. [Ki07a] Ki, Hyunwoo. ―Hierarchical Rendering Techniques for Real-Time Approximate Illumination on Programmable Graphics Hardware.‖ Master‘s Thesis. Soongsil University, 2007. [Ki07b] Ki, Hyunwoo and Kyoungsu Oh. ―Accurate Per-Pixel Displacement Mapping using a Pyramid Structure.‖ 2007. Hyunwoo Ki. n.d. . [Kircher09] Kircher, Scott and Alan Lawrance. ―Inferred Lighting: Fast Dynamic Lighting and Shadows for Opaque and Translucent Objects.‖ Course on 3D and the Cinematic in Games. SIGGRAPH 2009. Ernest N. Morial Convention Center, New Orleans, LA. 6 August 2009. [Koonce07] Koonce, Rusty. ―Deferred Shading in Tabula Rasa.‖ GPU Gems 3. Ed. Hurbert Nguyen. Kendallville, KY: Addison-Wesley, 2007. 429–458.

[Mitchell04] Mitchell, Jason and Pedro Sander. ―Applications of Explicit Early-Z Culling.‖ Course on Real-Time Shading. SIGGRAPH 2004. Los Angeles Convention Center, Los Angeles, CA. 8 August 2004. [Saito90] Saito, Takafumi and Tokiichiro Takahashi. ―Comprehensible Rendering of 3-D Shapes.‖ ACM SIGGRAPH Computer Graphics 24.4 (August 1990): 197–206. [Shishkovtsov05] Shishkovtsov, Oles. ―Deferred Shading in S.T.A.L.K.E.R.‖ GPU Gems 2: Programming Techniques for High-Performance Graphics and General Purpose Computation. Ed. Matt Pharr. Kendallville, Ky: Addison-Wesley, 2005. 143–166. [Valient07] Valient, Michal. ―Deferred Rendering in Killzone 2.‖ Develop Conference 2007. Brighton Hilton Metropole, Brighton, England, UK. 25 July 2007.

1.4. View Frustum Culling of Catmull-Clark Patches in DirectX 11 Rahul P. Sathe, Advanced Visual Computing, Intel Corp [email protected] DirectX 11 has introduced hardware tessellation in order to enable high geometric detail without increasing memory usage or memory bandwidth demands. Higher-order surface patches with displacements are of prime interest to game developers, and we would like to render them as efficiently as possible. For example, we would like to cull subdivision surface patches (instead of the resulting triangles) that will not affect the final image. Culling a given patch avoids higher-order surface evaluation of domain points in that patch as well as processing of the triangles generated for the patch. The nature of higher-order surface patches coupled with displacements and animation make the process of culling them nontrivial, since the exact geometric bounds are not known until well after the opportunity to cull a given patch. In this chapter, we will present an algorithm that evaluates conservative bounding boxes for displaced approximate Catmull-Clark subdivision surface patches at run time, allowing us to perform view frustum culling on the patches. With this method, we achieve performance improvement with minimal overhead.

Background Before describing our culling strategy, we must review the fundamentals of Catmull-Clark subdivision surfaces, displacement mapping, and the methods that are currently in use to approximate Catmull-Clark subdivision surfaces on DirectX 11. Displaced Subdivision Surfaces and Catmull-Clark Surfaces Catmull-Clark subdivision surfaces have become an increasingly popular modeling primitive and have been extensively used in offline rendering [DeRose98]. In general, subdivision surfaces can be described as recursive refinement of a polygonal mesh. Starting with a coarse polygonal mesh M0, one can introduce new vertices along the edges and faces and update the connectivity to get a mesh M1, and repeat this process to get meshes M2, M3, and so on. In the limit, this process approaches a smooth surface S. This smooth surface S is called the subdivision limit surface, and the original mesh M0 is often referred to as the control mesh. The control mesh consists of vertices connected to each other to form edges and faces. The number of other vertices that a given vertex is connected to directly by shared edges is called the valence of a vertex. In the realm of Catmull-Clark subdivision surfaces, a vertex is called a regular or ordinary vertex if it has a valence of four. If the valences of all of the

vertices of a given quad are four, then that quad is called an ordinary quad or an ordinary patch. The faces that have at least one vertex that is not valence four are called extraordinary faces (or patches). Approximate Catmull-Clark Subdivision Surfaces Recently, Loop and Schaefer introduced a hardware-friendly method of rendering Approximate Catmull Clark (ACC) subdivision surfaces, which maps very naturally to the DirectX 11 pipeline [Loop08]. At its core, the ACC scheme maps each quadrilateral from the original control mesh to a bi-cubic Bezier patch. Loop and Schaefer show that, for ordinary patches, the bi-cubic Bezier corresponds exactly to the Catmull-Clark limit surface. Extraordinary patches do not correspond exactly to the limit surface, but Loop and Schaefer decouple the patch description for position attributes and normal attributes in order to reduce the visual impact of the resulting discontinuities. To do this, for extraordinary patches, ACC generates separate normal and bi-tangent patches in order to impose GN continuity at patch boundaries. The word ―approximate‖ in ACC has its roots in the fact that these extraordinary patches are GN continuous, and this GN continuity only guarantees the same direction of partial derivatives but not the magnitudes across the patch boundaries. The ACC scheme describes the normals and bi-tangents using additional Bezier patches, which results in a continuous normal field even across edges of extraordinary patches. Displacement Although it is very empowering to be able to generate smooth surfaces from polygonal meshes procedurally, such smooth surfaces are rarely encountered in real life and lack realism without additional high-frequency geometric detail. This is where displacement maps come into the picture. Displacement maps are simply textures that can be used to store geometric perturbations from a smooth surface. Although normal maps and displacement maps have the similar effect of adding high-frequency detail, the difference is notable around the silhouettes of objects. A normal mapped object‘s silhouette lacks geometric detail because only per-pixel normals are perturbed and not the underlying geometry, as illustrated in Figure 1.4.1. To add this high-frequency detail, displacement maps can be applied to subdivision surfaces. Figure 1.4.1. Normal mapping versus displacement mapping.

DirectX 11 Pipeline DirectX 11 has introduced three new stages to the graphics pipeline to enable dynamic on chip tessellation, as shown in Figure 1.4.4. The two new programmable pipeline stages are the hull shader and the domain shader. Between these two programmable stages lies a new fixed function stage, the tessellator. Fortunately for us, ACC and Direct3D 11 were designed with each other in mind, and there is a natural mapping of the ACC algorithm onto the Direct3D 11 pipeline.

Hull Shader As illustrated in Figure 1.4.1, the new hull shader stage follows the traditional vertex shader. In a typical implementation of ACC on Direct3D 11, the vertex shader is responsible for performing animation of the control mesh vertices. In the hull shader, each quadrilateral‘s four vertices and its one-ring neighborhood are gathered from the output of the vertex shader. These vertices are used to define the control points of a bi-cubic Bezier patch. This basis conversion process that generates the Bezier patch control points is SIMD friendly, and every output control point can be calculated independently of others. In order to exploit this opportunity for parallelism, this control point phase of the hull shader is invoked once per control point. In the case of ACC, the basis conversion process depends on the topology of the incoming patch, but the output control points are always a 4×4 Bezier control mesh. Please refer to the sample code on the CD. Figure 1.4.2. Basis conversion for an irregular patch.

In addition to the computation of the Bezier control points, the hull shader can optionally calculate edge tessellation factors in order to manage level of detail. One can assign arbitrary tessellation factors to the edges of a patch (within some constraints, defined by the DirectX 11 tessellator specifications). Because the hull shader is programmable, one can choose any metric to calculate edge tessellation factors. Typical metrics may include screen space projection, proximity to silhouette, luminosity reaching the patch, and so on. The calculation of each edge tessellation factor is typically independent of the others, and hence the edge tessellation factors can also be computed in parallel in a separate phase of the hull shader called the fork phase. The final stage of hull shader is called the join phase (or patch constant phase) and is a phase in which the shader can efficiently compute data that is constant for the entire patch. This stage is of most interest to us in this chapter. Tessellator The tessellator accepts edge LODs of a patch and other tessellator-specific states that control how it generates domain locations and connectivity. Some of these states include patch topology (quad, tri, or isoline), inside reduction function (how to calculate inner tessellation factor(s) using outer tessellation factors), one-axis versus two-axis reduction (whether to reduce only one inner tessellation factor or two—once per each domain axis), and scale (how much to scale inner LOD). The tessellator feeds domain values to the domain shader and connectivity information to the rest of the pipeline via the geometry shader. Domain Shader In the case of quadrilateral patch rendering, the domain shader is invoked at domain values (u,v) determined by the tessellator. (In the case of triangular patches, the barycentric

coordinates (u,v,w); w = 1 – u – v are used.) Naturally, the domain shader has access to output control points from the hull shader. Typically, the domain shader evaluates a higherorder surface at these domain locations using the control points provided by the hull shader as the basis. After evaluating the surface, the domain shader can perform arbitrary operations on the surface position, such as displacing the geometry using a displacement map. In ACC, we evaluate position using bi-cubic polynomials for a given (u,v). Our domain shader interpolates texture coordinates (s,t) from the four vertices using bilinear interpolation to generate the texture coordinates for the given (u,v). We also optionally sample a displacement map at these interpolated texture coordinates. As mentioned earlier, normal calculation is different for ordinary and extraordinary patches. For ordinary patches, we just calculate d/du and d/dv of the position and take the cross-product. For extraordinary patches, we evaluate tangent and bi-tangent patches separately and take their cross-product.

Culling The mapping of ACC to the DirectX 11 pipeline that we have described allows us to render smooth surfaces with adaptive tessellation and displacement mapping, resulting in a compelling visual quality improvement while maintaining a modest memory footprint. At the end of the day, however, we are still rendering triangles, and the remaining stages of the graphics pipeline are largely unchanged, including the hardware stages that perform triangle setup and culling. This means that we perform vertex shading, domain shading, tessellation, and hull shading of all patches submitted to the graphics pipeline, including those patches that are completely outside of the view frustum. Clearly, this provides an opportunity for optimization. The main contribution of this chapter is a method for frustum culling patches early in the pipeline in order to avoid unnecessary computations. Of course, we must account for mesh animation and displacement, both of which deform a given patch in a way that complicates culling. An elegant generalized solution to surface patch culling has been proposed by Hasselgren et al. that generates culling shaders, looking at domain shaders using Taylor Arithmetic [Hasselgren09]. This article proposes a simplified version of ideas discussed in their work to cull the approximate Catmull-Clark patches against view frustum.

Pre-Processing Step We perform a pre-processing step on a given control mesh and displacement map in order to find the maximum displacement for each patch. Please note, although the positions are evaluated as bi-cubic polynomials using the new basis, the texture coordinates for those points are the result of bilinear interpolation of texture coordinates of the corners. This is due to the fact that the local (per-patch) uv-parameterization used to describe the CatmullClark surface and the global uv-parameterization done while creating the displacement map are linearly dependent on each other. Figure 1.4.3 shows one such patch. This linear dependence means that straight lines u = 0, v = 0, u = 1, and v = 1 in the patch parameterization are also straight lines in the global parameterization. Due to this linear relationship, we know the exact area in the displacement map from which the displacements will be sampled in the domain shader for that patch. The maximum displacement in the given patch can be found by calculating the maximum displacement in the region confined by patch boundaries in the displacement map. Even if the displacement map stores vectorvalued displacements, the mapping is still linear, so we can still find the magnitude of the maximum displacement for a given patch. Based on this, we can create a buffer for the entire mesh that stores this maximum displacement per patch.

Figure 1.4.3. Mapping between global (s-t) and local (u-v) parameterization is linear. The figure on the left shows (u,v) parameterization that is used for patch evaluation. The figure on the right shows the global parameterization (s,t) that was used while unwrapping original mesh. Bold lines correspond to u=0, v=0, u=1, and v=1 lines in the figure on the left.

Run-Time Step At run time, the patch vertices of the control mesh go through the vertex shader, which animates the control mesh. The hull shader then operates on each quad patch, performing the basis transformation to Bezier control points. One convenient property of Bezier patches is that they always stay within the convex hull of the control mesh defining the patch. Using the maximum displacement computed previously, we can move the convex hull planes of a given patch outward by the maximum displacement, resulting in conservative bounds suitable for culling a given patch. Although moving the convex hull planes out by the max displacement may give tighter bounds compared to an axis-aligned bounding box (AABB) for the control mesh, calculating the corner points can be tricky because it requires calculation of plane intersections. It is simpler and more efficient to compute an AABB of the control mesh and offset the AABB planes by the maximum displacement. In Figure 1.4.5, we show a 2D representation of this process for illustration. Dotted black lines represent the basis-converted Bezier control mesh. The actual Bezier curve is shown in bold black, displacements along the curve normal (scalar valued displacements) are shown in solid gray, and the maximum displacement for this curve segment is denoted as d. An AABB for the Bezier curve is shown in dashed lines (the inner bounding box), and the conservative AABB that takes displacements into account is shown in dashed and dotted lines (the outer bounding box). Figure 1.4.4. The DirectX11 pipeline. Normally, triangles get culled after primitive assembly, just before rasterization. The proposed scheme culls the patches in the

hull shader, and all the associated triangles from that patch get culled as a result, freeing up compute resources.

Figure 1.4.5. Conservative AABB for a displaced Bezier curve. The Bezier curve is shown in bold black, the control mesh in dotted lines, and displacements in solid gray lines. AABB for the Bezier curve without displacements is shown in dashed lines (inner bounding box), and conservative AABB for the displaced Bezier curve is shown in dashed and dotted lines (outer bounding box).

As you can see, the corners of inner and outer enclosures are more than d distance apart, so we are being more conservative than we need to be for the ease and speed of computation.

At this point, we have a conservative patch AABB that takes displacements into account. If the AABB for a patch is outside the view frustum, we know that the entire patch is outside the view frustum and can be safely culled. If we make the view frustum‘s plane equations available as shader constants, then our shader can test the AABB using in-out tests for view frustum. Alternatively, one can transform the AABB into normalized device coordinates (NDC), and the in-out tests can be done in NDC space. In-out tests in NDC space are easier than world space tests because they involve comparing only with +1 or –1. If the AABB is outside the view frustum, we set the edge LODs for that patch to be negative, which indicates to the graphics hardware that the patch should be culled. We perform the culling test during the join phase (a.k.a. patch constant phase) of the hull shader because this operation only needs to be performed once per patch.

Performance For each culled patch, we eliminate unnecessary tessellator and domain shader work for that patch. All patches, whether or not they‘re culled, take on the additional computational burden of computing the conservative AABB and testing against the view frustum. When most of the character is visible on the screen (for example, Figure 1.4.9 (a)), culling overhead is at its worst. Figure 1.4.6 shows that, even in this case, culling overhead is minimal and is seen only at very low levels of tessellation. At LOD=3, the gains due to culling a very small number of patches (around the character‘s feet) start offsetting the cycles spent on culling tests. Figure 1.4.6. Culling overhead is the worst when nothing gets culled. Culling overhead is minimal except at very low levels of tessellation. ―NO CULL‖ indicates the fps measured when no culling code was running. ―CULL Overhead‖ shows the fps measured when culling code was running in the patch constant phase of shaders.

When about half of the patches in our test model are outside of the view frustum (see Figure 1.4.9 (b)), the overhead of the AABB computations is offset by the gains from culling the offscreen patches. The gains from culling patches are more noticeable at higher levels of tessellation. This is shown graphically in Figures 1.4.7 and 1.4.8. Figure 1.4.7 shows how

fps changes with the edge tessellation factor (edge LOD) when about half of the patches are culled. As you can see, at moderate levels of tessellation, we strike the balance between benefits of the proposed algorithm at increased level of detail. Figure 1.4.8 shows the same data as percentage speed-up. Figure 1.4.7. Culling benefits go up with the level of tessellation, except at the super-high levels of tessellation where culling patches doesn’t help. At moderate levels of tessellation, we get benefits of the proposed algorithm and still see high geometric details.

Figure 1.4.8. Culling benefits shown as percentage increase in fps against edge LODs (edge tessellation factor).

Figure 1.4.9. Screenshots showing our algorithm in action. We saw about 8.9 fps for the view on the left and 15.1 fps for the view on the right on the ATI Radeon 5870. Increase in the frame rate was due to view frustum culling patches.

We performed all our tests on the ATI Radeon 5870 card, with 1 GB GDDR. The benefits of this algorithm increase with domain shader complexity and tessellation level, whereas the per-patch overhead of the culling tests remains constant. It is easy to imagine an application strategy that first tests an object‘s bounding box against the frustum to determine whether patch culling should be performed at all for a given object, thus avoiding the culling overhead for objects that are known to be mostly onscreen.

Conclusion We have presented a method for culling Catmull-Clark patches against the view frustum using the DirectX 11 pipeline. Applications will benefit the most from this algorithm at moderate to high levels of tessellation. In the future, we would like to extend this technique to account for occluded and back-facing patches with displacements.

References [DeRose98] DeRose, Tony, Michael Kass, and Tien Truong. ―Subdivision Surfaces in Character Animation.‖ Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. 1998. ACM SIGGRAPH. n.d. . [Hasselgren09] Hasselgren, Jon, Jacob Munkberg, and Tomas Akenine-Möller. ―Automatic Pre-Tessellation Culling.‖ ACM Transactions on Graphics 28.2 (April 2009): n.p. ACM Portal. [Loop08] Loop, Charles and Scott Schaefer. ―Approximating Catmull-Clark Subdivision Surfaces with Bicubic Patches.‖ ACM Transactions on Graphics 27.1 (March 2008): n.p. ACM Portal. [Microsoft09] Microsoft Corporation. DirectX SDK. August 2009. [Reif95] Reif, Ulrich. ―A Unified Approach to Subdivision Algorithms Near Extraordinary Vertices.‖ Computer Aided Geometric Design 12.2 (March 1995): 153–174. ACM Portal. [Stam98] Stam, Jos. ―Exact Evaluation of Catmull-Clark Subdivision Surfaces at Arbitrary Parameter Values.‖ Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (1998): 395–404. ACM Portal.

[Zorin2000] Zorin, Dennis and Peter Schroder. ―Subdivision for Modeling and Animation.‖ SIGGRAPH. 2000. 85–94.

1.5. Ambient Occlusion Using DirectX Compute Shader Jason Zink [email protected] Microsoft has recently released DirectX 11, which brings with it significant changes in several of its APIs. Among these new and updated APIs is the latest version of Direct3D. Direct3D 11 provides the ability to perform multi-threaded rendering calls, a shader interface system for providing an abstraction layer to shader code, and the addition of several new programmable shader stages. One of these new shader stages is the compute shader, which provides a significantly more flexible processing paradigm than was available in previous iterations of the Direct3D API. Specifically, the compute shader allows for a controllable threading model, sharing memory between processing threads, synchronization of primitive functions, and several new resource types to allow read/write access to resources. This gem will provide an introduction to the compute shader and its new features. In addition, we will take an in-depth look at a Screen Space Ambient Occlusion (SSAO) algorithm implemented on the compute shader to show how to take advantage of this new processing paradigm. We will examine the SSAO algorithm in detail and provide a sample implementation to demonstrate how the compute shader can work together with the traditional rendering pipeline. Finally, we will wrap up with a discussion of our results and future work.

The Compute Shader Before we begin to apply the compute shader‘s capabilities to a particular problem domain, let‘s take a closer look at the compute shader itself and the general concepts needed to program it. Overview The compute shader is a new programmable shader stage that is actually not simply inserted into the traditional rendering pipeline like some of the other new DirectX 11 pipeline stages discussed in Sathe‘s Gem 1.4. Rather, the compute shader is conceptually a standalone processing element that has access to the majority of the functionality available in the common shader core, but with some important additional functionality. The two most important new mechanics are fine-grained control over how each thread is used in a given shader invocation and new synchronization primitives that allow threads to synchronize. The threads also have read/write access to a common memory pool, which provides the opportunity for threads to share intermediate calculations with one another. These new capabilities are the basic building blocks for advanced algorithms that have yet to be developed, while at the same time allowing for traditional algorithms to be implemented in different ways in order to achieve performance improvements. Compute Shader Threading Model To use the compute shader, we need to understand its threading model. The main concept is that of a Thread Group. A Thread Group defines the number of threads that will be executing in parallel that will have the ability to communicate with one another. The threads within the Thread Group are conceptually organized in a 3D grid layout, as shown in Figure

1.5.1, with the sizes along each axis of the grid determined by the developer. The choice of the layout provides a simple addressing scheme used in the compute shader code to have each thread perform an operation on a particular portion of the input resources. When a particular thread is running, it executes the compute shader code and has access to several system value input attributes that uniquely identify the given thread. Figure 1.5.1. Thread Groups visualized as a 3D volume.

To actually execute the compute shader, we tell the API to execute a given number of Thread Groups via the Dispatch method, as illustrated in Figure 1.5.2. Figure 1.5.2. Visualization of the Dispatch method.

With these two layout definitions in mind, we can look at how they affect the addressing scheme of the compute shader. The following list of system values is available to the compute shader:

SV_GroupID. This system value identifies the Thread Group that a thread belongs to with a 3-tuple of zero-based indices. SV_GroupThreadID. This system value identifies the thread index within the current Thread Group with a 3-tuple of zero-based indices. SV_DispatchThreadID. This system value identifies the current thread identifier over a complete Dispatch call with a 3-tuple of zero-based indices. SV_GroupIndex. This system value is a single integer value representing a flat index of the current thread within the group. The individual threads running the compute shader have access to these system values and can use the values to determine, for example, which portions of input to use or which output resources to compute. For example, if we wanted a compute shader to perform an operation on each pixel of an input texture, we would define the thread group to be of size (x, y, 1) and call the Dispatch method with a size of (m, n, 1) where x*m is the width of the image and y*n is the height of the image. In this case, the shader code would use the SV_DispatchThreadID system value to determine the location in the input image from which to load data and where the result should be stored in the output image. Figure 1.5.3 illustrates one way in which a 2D workload might be partitioned using this method. In this example, we have an image with a size of 32×32 pixels. If we wanted to process the image with a total of 4×4 (m = 4, n = 4) Thread Groups as shown, then we would need to define the Thread Groups to each have 8×8 (x = 8 and y = 8) threads. This gives us the total number of threads needed to process all 32×32 (x*m and y*n) pixels of the input image.

Figure 1.5.3. Visualization of Thread Group distribution for a 2D workload, where the number of Thread Groups (m = 4, n = 4) and the number of threads (x = 8, y =

Compute Shader Thread Interactions In addition to providing an easy-to-use thread addressing scheme, the compute shader also allows each Thread Group to declare a block of Group Shared Memory (GSM). This memory is basically defined as an array of variables that are accessible to all of the threads in the Thread Group. The array itself can be composed of any native data types as well as structures, allowing for flexible grouping of data. In practice, the Group Shared Memory is expected to be on-chip register-based memory that should be significantly faster to access than general texture memory, which can have unpredictable performance depending on access patterns. Similar to CPU-based multi-threaded programming, when you have multiple threads reading and writing to the same area of memory there is the potential that the same memory can

be accessed simultaneously by more than one thread. To provide some form of control over the sequences of access, the compute shader introduces several atomic functions for thread synchronization. For example, there is an atomic function for adding called InterlockedAdd. This can be used to have all threads perform a test sequence and then use the InterlockedAdd function to increment a variable in the Group Shared Memory to tabulate an overall number of test sequences that produce a particular result. Another atomic function is the InterlockedCompareExchange function, which compares a shared variable with one argument and sets the variable to a second argument if the variable has the same value as the first argument. This provides the basic building blocks of creating a mutex system in the compute shader, where a shared variable serves as the mutex. Each thread can call this function on the mutex variable and only take action if it is able to update the variable to its own identifier. Since the compute shader is intended to provide massively parallel execution, a mutex is not really a preferred choice, but in some situations it may be a desirable avenue to follow, such as when a single resource must be shared across many threads. The Direct3D 11 documentation can be referenced for a complete list of these atomic functions and how they can be used. Also similar to CPU-based multi-threaded programming is the fact that it is more efficient to design your algorithms to operate in parallel while minimizing the number of times that they must synchronize data with one another. The fastest synchronization operation is the one that you don‘t have to perform! Compute Shader Resources New resource types introduced in Direct3D 11 include Structured Buffers, Byte Address Buffers, and Append/Consume Buffers. Structured Buffers provide what they sound like—ID buffers of structures available in your shader code. The Byte Address Buffers are similar, except that they are a general block of 32-bit memory elements. The Append/Consume Buffers allow for stack/queue-like access to a resource, allowing the shader to consume the elements of a buffer one at a time and append results to an output buffer one at a time. This should also provide some simplified processing paradigms in which the absolute position of an element is less important than the relative order in which it was added to the buffer. To further facilitate the compute shader‘s parallel-processing capabilities, Direct3D 11 provides a new resource view called an Unordered Access View (UAV). This type of view allows the compute shader (as well as the pixel shader) to have read and write access to a resource, where any thread can access any portion of the resource. This is a big departure from the traditional shader resource access paradigm; typically, a shader can only read from or write to a given resource during a shader invocation, but not both. The UAV can be used to provide random access to both the new and existing resource types, which provides significant freedom in designing the input and output structure of compute shader–based algorithms. With a general understanding of the new capabilities of the compute shader, we can now take a look at a concrete example in order to better understand the details. We will discuss the general concepts of the SSAO algorithm and then describe how we can use the compute shader‘s features to build an efficient implementation of the technique.

Screen Space Ambient Occlusion Screen Space Ambient Occlusion is a relatively recently developed technique for approximating global illumination in the ambient lighting term based solely on the

information present in a given frame‘s depth buffer [Mittring07]. As described in detail in Gem 1.2 by Filion, an approximate amount of ambient light that reaches a given pixel can be computed by sampling the area around the pixel in screen space. This technique provides a convincing approximation to global illumination and performs at a usable speed for high-end applications. The quality of the algorithm depends on the number of samples and subsequent calculations that are performed for each pixel. In the past few years, a variety of techniques have been proposed to modify the general SSAO algorithm with varying levels of quality versus performance tradeoffs, such as HBAO [Bavoil09a] and SSDO [Ritschel09]. While these new variants of the original algorithm provide improvements in image quality or performance, the basic underlying concepts are shared across all implementations, and hence the compute shader should be applicable in general. We will now review some of these recent SSAO techniques and discuss several areas of the underlying algorithm that can benefit from the compute shader‘s new capabilities. Then we will look at an implementation that takes advantage of some of these possible improvements. SSAO Algorithm Ambient occlusion techniques have been around for some time and have found uses primarily in offline rendering applications [Landis02]. The concept behind these techniques is to utilize the geometric shape of a model to calculate which portions of the model would be more likely to be occluded than others. If a given point on a model is located on a flat surface, it will be less occluded than another point that is located at a fold in the surface. This relationship is based on the following integral for the reflected radiance:

In this integral, Lin is the incident radiation from direction ω, and the surface normal vector is n. This integral indicates that the amount of light reflected at a given surface point is a function of the incident radiance and the angle at which it reaches that point. If there is nearby geometry blocking some portion of the surface surrounding the surface point, then we can generally conclude that less radiant energy will reach the surface. With this in mind, the ambient lighting term can be modulated by an occlusion factor to approximately represent this geometric relationship. One way to perform this geometric calculation is to project a series of rays from each surface point being tested. The amount of occlusion is then calculated depending on the number of rays that intersect another part of the model within a given radius from the surface point. This effectively determines how much ―background‖ light can reach that point by performing the inverse operation of the radiance integral described previously. Instead of integrating the incident radiance coming into that point over the surface of a hemisphere, we shoot rays out from the surface point over the hemisphere to test for occlusion within the immediate area. The overall occlusion factor is then calculated by accumulating the ray test results and finding the ratio of occluded rays versus non-occluded rays. Once it is calculated, this occlusion factor is then stored either per vertex or per pixel in a texture map and is used to modulate the ambient lighting term of that object when rendered. This produces a rough approximation of global illumination. Figure 1.5.4 demonstrates this ray casting technique. Figure 1.5.4. Side profile of a ray casting technique for approximating occlusion.

This technique works quite well for static scenes or individual static geometric models, but the pre-computation requirements are not practical for dynamic geometry, such as skinned meshes. Several alternative techniques have been suggested to allow for dynamic ambient occlusion calculations, such as [Bunnell05], which generalizes the geometric object into disks to reduce the computational complexity of the occlusion calculations. This allows realtime operation of the algorithm, but it still requires some pre-processing of the models being rendered to determine where to place the disks in the approximated models. In addition, the cost of performing the occlusion calculation scales with increased scene complexity. The Screen Space Ambient Occlusion algorithm provides an interesting alternative technique for determining an approximate occlusion value. Instead of computing an occlusion value based on the geometric representation of a scene by performing ray casting, the occlusion calculation is delayed until after the scene has been rasterized. Once the scene has been rasterized, an approximated amount of occlusion is determined by inspecting the contents of the scene‘s depth buffer only—the geometric queries are carried out on the depth buffer instead of on the geometric models. This effectively moves the operation from an object space operation to a screen space operation—which is one of the major benefits of this algorithm. Since it operates at the screen space level, the algorithm‘s performance is less sensitive to the amount of geometry being rendered and is more sensitive to the resolution of the buffers being used. The scene‘s depth buffer can be obtained by utilizing the actual Z-buffer used during rendering, by performing a separate rendering pass that writes the linear depth to a render target, or by using the depth information from a deferred rendering G-buffer. Regardless of how the buffer is generated, the algorithm performs a processing pass that uses the depth buffer as an input and generates an output texture that holds the occlusion values for the entire visible scene. Each pixel of the output is calculated using the depth information within a given radius of its local area, which can be considered an approximation to ambient occlusion. I will refer to this output in the remainder of this document as the occlusion buffer. When the final scene rendering is performed, the occlusion buffer is sampled based on screen space location and used to modulate the ambient term of each object in the final scene. SSAO Algorithm Details Screen Space Ambient Occlusion has provided a significant improvement over previous ambient occlusion algorithms. Due to the fact that the algorithm runs after a scene is rendered, it focuses the processing time on only the portion of the scene that is visible for

the current frame, saving a significant amount of computation and allowing the algorithm to be run in real-time applications without pre-computation. However, the use of the depth buffer also introduces a few obstacles to overcome. There is the potential that some occluders will not be visible in the depth buffer if there is another object in front of it. Since the depth buffer only records one depth sample per pixel, there is no additional information about the occluders behind the foreground object. This is typically handled by defaulting to zero occlusion if the depth sample read from the depth buffer is too far away from the current pixel being processed. If a more accurate solution is needed, depth peeling can be used to perform multiple occlusion queries, as described in [Bavoil09b]. Additionally, if an object is offscreen but is still occluding an object that is visible onscreen, then the occlusion is not taken into account. This leads to some incorrect occlusion calculations around the outer edge of the image, but solutions have also been proposed to minimize or eliminate these issues. One possibility is to render the depth buffer with a larger field of view than the final rendering to allow objects to be visible to the algorithm around the perimeter of the view port [Bavoil09a]. Another issue with the algorithm is that a relatively large number of samples needs to be taken in order to generate a complete representation of the geometry around each pixel. If performance were not a concern, we could sample the entire area around the pixel P in a regular sampling pattern, but in real-time applications this quickly becomes impractical. Instead of a regular sampling pattern, a common solution is to use a sparse sampling kernel to choose sampling points around the current pixel. This roughly approximates the surrounding area, but the decreased sampling rate may miss some detail. To compensate for the decreased sampling, it is common to use a stochastic sampling technique instead. By varying the sampling kernel shape and/or orientation for each pixel and then sharing the results between neighboring pixels, an approximation to the more expensive regular sampling pattern can be achieved. Since a typical 3D scene is composed of groups of connected triangles, the majority of the contents of the depth buffer will contain roughly similar depth values in neighborhoods of pixels except at geometric silhouette edges. The variation of the sampling kernel between pixels in combination with this spatial coherence of the depth buffer allows us to share a combined larger number of sample results per pixel while reducing the overall number of calculations that need to be performed. This helps to effectively widen the sampling kernel, but it also introduces some additional high-frequency noise into the occlusion buffer. To compensate for this effect, it is common to perform a filtering pass over the entire occlusion buffer that blurs the occlusion values without bleeding across object boundaries. This type of a filter is referred to as a bilateral filter, which takes into account both the spatial distance between pixels and the intensity values stored in neighboring pixels when calculating the weights to apply to a sample [Tomasi98]. This allows the filter to remove high-frequency noise and at the same time preserve the edges that are present in the occlusion buffer. In addition, the randomization process can be repeated over a small range to facilitate easier filtering later on. Figures 1.5.5 and 1.5.6 show ambient occlusion results before and after bilateral filtering. Figure 1.5.5. A sample scene rendered without bilateral filtering.

Figure 1.5.6. A sample scene after bilateral filtering.

As mentioned before, the algorithm is performed after rasterization, meaning that its performance is directly related to the screen resolution being used. In fact, this dependency on screen resolution has been exploited to speed up the algorithm as described in Gem 1.3. The depth buffer and/or the occlusion buffer can be generated at a decreased resolution. If the screen resolution is decreased by a factor of 2 in the x and y directions, there is an overall factor of 4 reduction in the number of occlusion pixels that need to be calculated. Then the occlusion buffer can either be upsampled with a bilateral filter or just directly used at the lower resolution. This strategy can still lead to fairly pleasing results, since the contents of the occlusion buffer are relatively low frequency. SSAO Meets the Compute Shader When looking at the block diagram of the SSAO algorithm in Figure 1.5.7, we can begin to compare these high-level operations with the new capabilities of the compute shader to see how we can build a more efficient implementation. We will now go over the steps of the algorithm and discuss potential strategies for mapping to the compute shader. Figure 1.5.7. Block diagram of the SSAO algorithm.

Calculation Setup The first step shown in the block diagram is to initialize the computations for the current pixel. This entails sampling the depth buffer to obtain the pixel‘s depth. One of the benefits of having a Group Shared Memory that can be shared by all threads in a Thread Group is the possibility to share texture samples among the entire Thread Group. Because the shared memory is supposed to be significantly faster than a direct texture sample, if each thread requests a depth sample to initialize its own calculations, then it can also write that depth value to the shared memory for use later on by other threads. The net effect of every thread in a Thread Group doing this is to have a copy of the complete local depth data in the Group Shared Memory. Later, as each thread begins calculating the relative occlusion against the local area, it can read the needed depth values from the Group Shared Memory instead of directly loading from texture memory. Figure 1.5.8 shows this process. Figure 1.5.8. Comparison of directly sampling versus using the Group Shared Memory for cached sampling.

There are a few additional notes to consider on this topic, however. There is some overhead associated with reading the depth values and then storing them to the Group Shared Memory. In addition, the texture cache can often provide very fast results from memory sample requests if the result was in the cache. Thus, depending on the hardware being run and the patterns and frequency of memory access, it may or may not provide a speed increase to use the Group Shared Memory in practice. Randomize Sampling Kernel The next step in the SSAO block diagram is to somehow randomize the sampling kernel that will be used to later look up the surrounding area. This is typically done by acquiring a random vector and then performing a ―reflect‖ operation on each of the sampling kernel vectors around the random vector. Probably the most common way to acquire this vector is to build a small texture with randomized normal vectors inside. The shader can load a single normalized reflection vector based on the screen space position of the pixel being processed [Kajalin09]. This makes removing the ―salt-and-pepper‖ noise easier in the filtering stage of the algorithm. In the past, SSAO was performed in the pixel shader, which means that the pixel shader required a screen space position as a fragment attribute to be passed by the vertex or geometry shader. The compute shader can help to simplify this operation somewhat. By utilizing the Dispatch ID system value, we can automatically receive the integer ID of each pixel being processed in our compute shader code. To create our repeating pattern of reflection vectors in screen space, we can simply perform a bitwise AND operation on the least significant bits of the dispatch ID—in other words, if we wanted to repeat every 4×4 block of pixels, we would mask off all but the two least significant bits of the ID. In fact, we can even store the randomized vectors as an array of constants in our shader. This eliminates the need for a texture sample and the repeating texture of normalized reflection vectors altogether. Of course this is predicated on the fact that we don‘t use too many vectors, but we could always use the standard approach if that is needed. Acquire Depth Data

Once the sampling kernel has been randomized, we can acquire each individual depth sample. In a traditional SSAO algorithm, this is done with a sampler that uses the x and y coordinates of the current sampling kernel vector to offset from the current pixel location. Since the sampling kernel has been pseudo-randomized, there is a potential for reduced texture cache efficiency if the sampling kernel width is large enough. If we utilize the Group Shared Memory as described previously, then the depth values that we need to acquire could already be available in the GSM. However, there are several points to consider before embarking on this strategy as well. Since the Thread Group will only be operating on one block of the depth data at a time—for example, a 16×16 block—then we need to consider what happens at the edges of that block. The pixels along the outer edges of the block will need access to the depth samples within our sampling radius, and they would not already be pre-loaded. This provides a choice—we could either pre-load a larger portion of the depth buffer to include the surrounding area or we could dynamically check to see whether the data has been loaded to the GSM yet, and, if not, then directly get it from the depth buffer. Both options could have performance penalties. Pre-loading large bands of depth data around each block may end up increasing the number of depth samples to the point that it would be just as efficient to perform the sampling in the traditional manner. If we dynamically decide whether or not to fetch data from the depth buffer, then we could perform a large number of dynamic branches in the shader, which could also be detrimental to performance. These factors need to be weighed against the increased access speed provided by using the GSM instead of direct sampling. With the texture cache providing similar fast access for at least a portion of the texture samples, it is altogether possible that the standard approach would be faster. Of course, any discussion of texture cache performance depends on the hardware that the algorithm is running on, so this should be tested against your target platform to see which would be a better choice. The other point to consider with using the GSM is that there is no native support for bilinear filtering of the GSM data. If you wanted to filter the depth values for each depth sample based on the floating-point values of the kernel offset vector, then you would need to implement this functionality in the shader code itself. However, since the depth buffer contains relatively low-frequency data, this is not likely to affect image quality in this case. Perform Partial Occlusion Calculation (per Sample) Once we have obtained a depth sample to compare to our current pixel depth, we can move to the partial occlusion calculations. In this step, we determine whether our sample depth causes any occlusion at the current pixel. There are many different varieties of calculations available to perform here, from a binary test of the sample point being above or below the kernel offset vector [Kajalin09] all the way up to a piecewise defined function read from a texture [Filion08]. Regardless of how the calculation is performed, there is an interesting possibility that the compute shader introduces if the calculation is only a function of the depth delta—sharing occlusion calculations between pixels. If we call our current pixel point P and our current sample point S, then the occlusion caused at point P by point S is inherently related to the inverse occlusion at point S by point P. Since the compute shader can perform scatter operations, a single thread can calculate the occlusion for one pair of locations and then write the result to point P and the inverse of the calculation to point S. This would save the number of required calculations by nearly a factor of 2, but it would also introduce the need for some type of communication mechanism to get the values to both occlusion buffer values. Since there is the possibility that multiple pixels would be trying to write a result to the same pixel, we could attempt to use the atomic operations for updating the values, but this could lead to a large number of synchronization events between threads. At the same time, these occlusion values can be accumulated in the GSM

for fast access by each thread. Again, the cost of the synchronization events will likely vary across hardware, so further testing would be needed to see how much of a benefit could come from this implementation. Perform Complete Occlusion Calculation The final step in this process is to calculate the final occlusion value that will end up in the occlusion buffer for use in the final rendering. This is normally done by performing a simple average of all of the partial occlusion calculations. In this way, we can scale the number of samples used to calculate the occlusion according to the performance level of the target hardware. As described earlier, there is typically some form of a bilateral filter applied to the occlusion buffer after all pixels have a final occlusion value calculated. In general, filtering is one area that could potentially see huge benefits from compute shader implementations. Since filtering generally has an exact predetermined access pattern for the input image, the Group Shared Memory can directly be used to pre-load the exact texture data needed. This is especially beneficial when implementing 2D separable filters due to the ability to perform the filtering pass in one direction, store the result into the GSM, then perform the second filtering pass in the other direction over the values in the GSM without ever writing the results back to the output buffer in between steps. Even though the bilateral filter is nonseparable, it has been shown that a decent approximation of it can be achieved with a separable implementation [Pham05].

Compute Shader Implementation Details After reviewing some of the new features available in the compute shader and how they can be used with the SSAO algorithm, we can now look at a sample implementation. Since the compute shader techniques are relatively new, the focus of this implementation will be to demonstrate some of its new features and draw some conclusions about appropriate-use cases for them. These features are described briefly here, with additional detail provided in the following sections. This implementation will utilize two different-size thread groups, 16×16 and 32×32, to generate the occlusion buffer. Using two different sizes will allow us to see whether the Thread Group size has any effect on the performance of the algorithm. We will also demonstrate the use of the GSM as a cache for the depth values and compare how well this tactic performs relative to directly loading samples from the depth buffer. In addition to using the GSM, we also utilize the Gather sampling function for filling the GSM with depth values to see whether there is any impact on overall performance. The randomization system will utilize one of the new thread addressing system values to select a reflection vector, eliminating the need for a randomization texture. After the occlusion buffer has been generated, we will utilize a separable version of the bilateral filter to demonstrate the ability of the compute shader to efficiently perform filtering operations. Implementation Overview The process is started by rendering a linear depth buffer at full-screen resolution with the traditional rendering pipeline. Stored along with the depth value is the view space normal vector, which will be used during the occlusion calculations. This depth/normal buffer serves as the primary input to the compute shader to calculate a raw, unfiltered occlusion buffer. Finally, we use the depth/normal buffer and the raw occlusion buffer to perform separable bilateral filtering to produce a final occlusion buffer suitable for rendering the scene with the standard rendering pipeline.

Depth/Normal Buffer Generation The depth/normal buffer will consist of a four-component floating-point texture, and each of the occlusion buffers will consist of a single floating-point component. The depth/normal vectors are generated by rendering the linear view space depth and view space normal vectors into the depth/normal buffer. The depth value is calculated by simply scaling the view space depth by the distance to the far clipping plane. This ensures an output in the range of [0,1]. The normal vector is calculated by transforming the normal vector into view space and then scaling and biasing the vector components. Listing 1.5.1 shows the code for doing so. Listing 1.5.1. Generation of the view space depth and normal vector buffer

output.position = mul( float4( v.position, 1.0f ) ,WorldViewProjMatrix ); float3 ViewSpaceNormals = mul( float4( v.normal, 0.0f ) ,WorldViewMatrix ).xyz; output.depth.xyz = ViewSpaceNormals * 0.5f + 0.5f; output.depth.w = output.position.w / 50.0f;

Depending on the depth precision required for your scene, you can choose an appropriate image format—either 16 or 32 bits. This sample implementation utilizes 16-bit formats. Raw Occlusion Buffer Generation Next, we generate the raw occlusion buffer in the compute shader. This represents the heart of the SSAO algorithm. As mentioned earlier, we will utilize two different Thread Group sizes. The occlusion calculations will be performed in Thread Groups of size 16×16×1 and 32×32×1. Since we can adjust the number of Thread Groups executed in the application‘s Dispatch call, either Thread Group size can be used to generate the raw occlusion buffer. However, if there is any performance difference between the two Thread Group sizes, this will provide some insight into the proper usage of the compute shader. Regardless of the size of the Thread Groups, each one will generate one portion of the raw occlusion buffer equivalent to its size. Each thread will calculate a single pixel of the raw occlusion buffer that corresponds to the thread‘s Dispatch thread ID system value. This Dispatch thread ID is also used to determine the appropriate location in the depth/normal buffer to load. The depth value and normal vector are loaded from the texture and converted back into their original formats for use later. Depth Value Cache with the GSM We will also set up the compute shader to cache local depth values in the GSM. Once the depth values of the surrounding area are loaded into the GSM, all subsequent depth sampling can be performed on the GSM instead of loading directly from texture memory. Before we discuss how to set up and use the GSM, we need to consider the desired layout for the data. Since we are utilizing two different Thread Group sizes, we will specify a different layout for each. Each of the Thread Groups requires the corresponding depth region that it represents to be present in the GSM. In addition, the area surrounding the Thread Group‘s boundary is also needed to allow the occlusion calculations for the border pixels to be carried out correctly. This requires each thread to sample not only its own depth/normal vector, but also some additional depth values to properly load the GSM for

use later. If we stipulate that each thread will load four depth values into the GSM, then our 16×16 thread group will provide a 32×32 overall region in the GSM (the original 16×16 block with an 8-pixel boundary). The 32×32 Thread Group size will provide a 64×64 region (the original 32×32 block with a 16-pixel boundary). Fortunately, the Gather instruction can be utilized to increase the number of depth values that are sampled for each thread. The Gather instruction returns the four point-sampled single component texture samples that would normally have been used for bilinear interpolation—which is perfect for pre-loading the GSM since we are using only single component depth values. This effectively increases the number of depth samples per texture instruction by a factor of 4. If we use each thread to perform a single Gather instruction, then we can easily fill the required areas of 32×32 and 64×64. The required samples are obtained by having each thread perform the Gather instruction and store the results in the GSM for all other threads within the group to utilize. This is demonstrated in Listing 1.5.2. Listing 1.5.2. Declaring and populating the Group Shared Memory with depth data

#define USE_GSM #ifdef USE_GSM // Declare enough shared memory for the padded thread group size groupshared float LoadedDepths[padded_x][padded_y]; #endif int3 OffsetLocation = int3( GroupID.x*size_x - kernel_x, GroupID.y*size_y kernel_y, 0 ); int3 ThreadLocation = GroupThreadID * 2; float2 fGatherSample; fGatherSample.x = ((float)GroupID.x * (float)size_x (float)kernel_x + (float)GroupThreadID.x * 2.0f xres; fGatherSample.y = ((float)GroupID.y * (float)size_y (float)kernel_y + (float)GroupThreadID.y * 2.0f yres;

– ) / – ) /

float4 fDepths = DepthMap.GatherAlpha( DepthSampler, fGatherSample + float2( 0.5f / (float)xres, 0.5f / (float)yres ) ) * zf; LoadedDepths[ThreadLocation.x][ThreadLocation.y] = fDepths.w; LoadedDepths[ThreadLocation.x+1][ThreadLocation.y] = fDepths.z; LoadedDepths[ThreadLocation.x+1][ThreadLocation.y+1] = fDepths.y; LoadedDepths[ThreadLocation.x][ThreadLocation.y+1] = fDepths.x; GroupMemoryBarrierWithGroupSync();

The number of depth values loaded into the GSM can be increased as needed by having each thread perform additional Gather instructions. The Group Shared Memory is defined as a 2D array corresponding to the size of the area that will be loaded and cached. After all of the depth values have been loaded, we introduce a synchronization among threads in the Thread Group with the GroupMemoryBarrierWithGroupSync() intrinsic function. This function ensures that all threads have finished writing to the GSM up to this point in the compute shader before continuing execution. A compile-time switch is provided in the sample code to allow switching between filling the GSM to use the cached depth values or to directly access the depth texture. Since the GSM has the potential to improve the sampling performance depending on the access pattern, this will allow an easy switch between techniques for a clear efficiency comparison. Next, we initialize the randomization of the sampling kernel with the lowest four bits of the Dispatch thread ID x and y coordinates, as shown in Listing 1.5.3. The lowest four bits in each direction are used to select a reflection vector from a 2D array of rotation vectors, which are predefined and stored in a constant array. This eliminates the need for a separate texture and range expansion calculations, but it requires a relatively large array to be loaded when the compute shader is loaded. After it is selected, the reflection vector is then used to modify the orientation of the sampling kernel by reflecting each of the kernel vectors about the reflection vector. This provides a different sampling kernel for each consecutive pixel in the occlusion buffer. Listing 1.5.3. Definition of the sampling kernel and selection of the randomization vector

const float3 kernel[8] { normalize( float3( normalize( float3( normalize( float3( normalize( float3( normalize( float3( normalize( float3( normalize( float3( normalize( float3( };

= 1, 1, 1 -1,-1,-1 -1,-1, 1 -1, 1,-1 -1, 1, 1 1,-1,-1 1,-1, 1 1, 1,-1

)), )), )), )), )), )), )), ))

const float3 rotation[16][16] = { { {...},{...},{...},{...}, ... } }; int rotx = DispatchThreadID.x & 0xF; int roty = DispatchThreadID.y & 0xF; float3 reflection = rotation[rotx][roty];

With a random reflection vector selected, we can begin the iteration process by sampling a depth value at the location determined by the randomized sampling kernel offsets. The

sample location is found by determining the current pixel‘s view space 3D position and then adding the reoriented sampling kernel vectors as offsets from the pixel‘s location. This new view space position is then converted back to screen space, producing an (x, y) coordinate pair that can then be used to select the depth sample from either the GSM or the depth/normal texture. This is shown in Listing 1.5.4. Listing 1.5.4. Sampling location flipping and re-projection from view space to screen space

float3 vRotatedOffset = reflect( kernel[y], rotation[rotx][roty] ); float fSign = dot( fPixelNormal, vRotatedOffset ); if ( fSign < 0.0f ) vFlippedOffset = -vFlippedOffset; float3 Sample3D = PixelPosVS + vFlippedOffset * scale; int3 newoffset = ViewPosToScreenPos( Sample3D ); #ifndef USE_GSM float fSample = DepthMap.Load( iNewOffset ).w * zf; #else float fSample = LoadDepth( iNewOffset - OffsetLocation ); #endif

The pixel‘s view space normal vector is used to determine whether the kernel offset vector points away from the current pixel. If so, then the direction of the offset vector is negated to provide an additional sample that is more relevant for determining occlusion. This provides additional samples in the visible hemisphere of the pixel, which increases the usable sample density for the pixel. The final screen space sample location is then used to look up the depth sample either directly from the texture or from the GSM by calling the LoadDepth() function. After the depth has been loaded, the occlusion at the current pixel from this sample is calculated. The calculation that is used is similar to the one presented in [Filion08] and [Lake10], using a linear occlusion falloff function raised to a power. This produces a smooth gradual falloff from full occlusion to zero occlusion and provides easy-to-use parameters for adjusting the occlusion values. The partial occlusion calculation is repeated for a given number of samples, implemented as a multiple of the number of elements in the sampling kernel. In this implementation, the number of samples can be chosen in multiples of eight. All of these individual occlusion values are averaged and then stored in the raw occlusion buffer for further processing. Separable Bilateral Filter The final step in our occlusion value generation is to perform the bilateral blur. As described earlier, we are able to use a separable version of the filter, even though it is not perfectly accurate to do so. The bilateral filter passes are implemented in the compute shader, with each of the separable passes being performed in an individual dispatch call. Since we are only processing one direction at a time, we will first use one Thread Group for each row of the image and then process the resulting image with one Thread Group for each column of the image. In this arrangement, we can load the entire contents of a Thread Group‘s row or

column into the GSM, and then each thread can directly read its neighbor values from it. This should minimize the cost of sampling a texture for filtering and allow larger filter sizes to be used. This implementation uses 7×7 bilateral filters, but this can easily be increased or decreased as needed. Listing 1.5.5 shows how the separable filter pass loads its data into the GSM. Listing 1.5.5. Loading and storing the depth and occlusion values into the GSM for the horizontal portion of a separable bilateral filter

// Declare enough shared memory for the padded group size groupshared float2 horizontalpoints[totalsize_x]; ... int textureindex = DispatchThreadID.x + DispatchThreadID.y * totalsize_x; // Each thread will load its own depth/occlusion values float fCenterDepth = DepthMap.Load(DispatchThreadID).w; float fCenterOcclusion = AmbientOcclusionTarget[textureindex].x; // Then store them in the GSM for everyone touse horizontalpoints[GroupIndex].x = fCenterDepth; horizontalpoints[GroupIndex].y = fCenterOcclusion; // Synchronize all threads GroupMemoryBarrierWithGroupSync();

One thread is declared for each pixel of the row/column, and each thread loads a single value out of the raw occlusion buffer and stores that value in the GSM. Once the value has been stored, a synchronization point is used to ensure that all of the memory accesses have completed and that the values that have been stored can be safely read by other threads. The bilateral filter weights consist of two components: a spatially based weighting and a range-based weighting. The spatial weights utilize a fixed Gaussian kernel with a size of 7 taps in each direction. A separate Gaussian weighting value is calculated based on the difference between the center pixel and each of the samples to determine the weighting to apply to that sample. Modifying the sigma values used in the range-based Gaussian allows for easy adjustment of the range-filtering properties of the bilateral filter. Listing 1.5.6 shows how this calculation is performed. Listing 1.5.6. Horizontal portion of a separable bilateral filter in the compute shader

const float avKernel7 = { 0.004431f, 0.05402f, 0.2420f, 0.3990f, 0.2420f, 0.05402f, 0.004431f }; const float rsigma = 0.0051f; float fBilateral = 0.0f; float fWeight = 0.0f; for ( int x = -3; x <= 3; x++ )

{ int location = GroupIndex + x; float fSampleDepth = horizontalpoints[location].x; float fSampleOcclusion = horizontalpoints[location].y; float fDelta = fCenterDepth - fSampleDepth; float fRange = exp( ( -1.0f * fDelta * fDelta ) / ( 2.0f * rsigma * rsigma ) ); fBilateral += fSampleOcclusion * fRange * avKernel[x+3]; fWeight += fRange * avKernel[x+3]; } AmbientOcclusionTarget[textureindex] = fBilateral / fWeight;

Finally, once both passes of the bilateral filter are performed, the values written to the final output buffer can be used to supply the ambient lighting term of an output image. The value stored in the occlusion buffer represents visibility, and thus can be directly used as the ambient lighting term without modification. The sample implementation provides its output using only the occlusion value—no other lighting is applied to the scene.

Results Figure 1.5.9 shows the end result of our compute shader–based implementation of the algorithm. All images and performance numbers were generated on an AMD 57xx series GPU. The following image was generated using 32 depth samples for each occlusion pixel and with the 7×7 separable bilateral filter applied twice. Figure 1.5.9. Final results of the compute shader SSAO implementation.

To gain some insight into how well the new implementation techniques perform, we can review the overall frame time for each of our optional configurations. Table 1.5.1 provides

the performance metrics for the two Thread Group sizes for varying numbers of samples used in the occlusion calculation. These frames-per-second figures were generated with the direct sampling technique on a 640×480 render target with no bilateral filtering applied. Table 1.5.1. NO GSM 8

16

24

32

40

48

56

64

16×16

947 608 437 349 288 241 212 188

32×32

961 610 445 350 288 245 214 187

The two Thread Group sizes produce nearly identical performance numbers. This indicates that the Thread Group size does not have a significant impact on this type of iterative algorithm. When considering the actual frame time for each test (the inverse of the fps), we see a linear increase in frame time for each of the additional sets of samples. In comparison, Table 1.5.2 provides the same metrics generated with the GSM caching technique. Table 1.5.2. W/GSM 8

16

24

32

40

48

56

64

16×16

811 699 531 426 357 306 269 240

32×32

764 625 485 391 330 286 252 225

Table 1.5.2 shows a different performance characteristic. When compared to the direct loading technique, we see the GSM technique performs slower at the 8-sample level. However, for all sample levels above this, the GSM technique significantly outperforms the direct sampling method. For all of these higher sampling levels, we see a similar linear increase in frame time but a smaller slope than the direct sampling method. Figure 1.5.10 shows the frame times for the four different cases. Figure 1.5.10. Comparison of frame times with and without the GSM as a cache.

The slower performance with the GSM at lower sampling rates can be attributed to the overhead of loading and storing all of the additional depth data. However, there is a clear performance gain for each additional sample used in the occlusion calculation. With this performance advantage also comes some limitations. In both Thread Group sizes, we defined a fixed border size. In some cases, when a pixel is close to the viewer, the offset vector can produce a screen space offset much larger than this border size. This can be overcome either by scaling the size of the sampling kernel according to the distance from the camera or by dynamically determining whether the sample location is available in the GSM and directly loading the sample if needed.

Conclusion and Future Work In this chapter, we have applied the compute shader to the Screen Space Ambient Occlusion algorithm and discussed the implications of various implementation choices. This implementation provides a basic framework upon which further proposed extensions can be implemented relatively easily. Additional research can be directed at sharing partial occlusion values between neighboring pixels for each occlusion calculation, which is now possible due to the scatter capabilities of the compute shader. In addition, further exploration on the use of the GSM as a caching mechanism for regional depth averages could be investigated. Finally, there have been several recent findings using multi-resolution rendering solutions for SSAO, which should also benefit from the compute shader implementations.

References [Bavoil09a] Bavoil, Louis and Miguel Sainz. ―Image-Space Horizon-Based Ambient Occlusion.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. Section 6.2.

[Bavoil09b] Bavoil, Louis and Miguel Sainz. ―Multi-Layer Dual-Resolution Screen-Space Ambient Occlusion.‖ 2009. SlideShare. n.d. . [Bunnell05] Bunnell, Michael. ―Dynamic Ambient Occlusion and Indirect Lighting.‖ 2005. Gamedev.net. n.d. . [Filion08] Filion, Dominic and Rob McNaughton. ―Effects and Techniques.‖ Course on Advances in Real-Time Rendering in 3D Graphics and Games Course. SIGGRAPH 2008. Los Angeles Convention Center, Los Angeles, CA. 11 August 2008. [Kajalin09] Kajalin, Vladimir. ―Screen Space Ambient Occlusion.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang F. Engel. Boston: Charles River Media, 2009. Section 6.1. [Lake10] Lake, Adam, ed. Game Programming Gems 8. Boston: Charles River Media, 2010. [Landis02] Landis, Hayden. ―Production-Ready Global Illumination.‖ Course notes on RenderMan in Production. SIGGRAPH 2002. Henry B. Gonzalez Convention Center, San Antonio, TX. 21 July 2002. Chapter 5. [Mittring07] Mittring, Martin. ―Finding Next Gen – CryEngine 2.0.‖ Course notes on Advanced Real-Time Rendering in 3D Graphics and Games. SIGGRAPH 2007. San Diego Convention Center, San Diego, CA. 8 August 2007. 97–121. [Pham05] Pham, T.Q. and L.J. van Vliet. ―Separable Bilateral Filtering for Fast Video Processing.‖ IEEE International Conference on Multimedia and Expo. Amsterdam, The Netherlands. July 2005. [Ritschel09] Ritschel, Tobias, Thorsten Grosch, and Hans-Peter Seidel. ―Approximating Dynamic Global Illumination in Image Space.‖ Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (2009): 75–82. [Tomasi98] Tomasi, Carlo and Roberto Manduchi. ―Bilateral Filtering for Gray and Color Images.‖ Proceedings of the Sixth International Conference on Computer Vision. (1998): 839–846.

1.6. Eye-View Pixel Anti-Aliasing for Irregular Shadow Mapping Nico Galoppo, Intel Advanced Visual Computing (AVC) [email protected] The irregular shadows algorithm (also known as Irregular Z-Buffer shadows) combines the image quality and sampling characteristics of ray-traced shadows with the performance advantages of depth buffer–based hardware pipelines [Johnson04]. Irregular shadows are free from aliasing from the perspective of the light source because the occlusion of each eye-view sample is evaluated at sub-pixel precision in the light view. However, irregular shadow mapping suffers from pixel aliasing in the final shadowed image due to the fact that shadow edges and high-frequency shadows are not correctly captured by the resolution of the eye-view image. Brute-force super-sampling of eye-view pixels decreases shadow aliasing overall but incurs impractical memory and computational requirements. In this gem, we present an efficient algorithm to compute anti-aliased occlusion values. Rather than brute-force super-sampling all pixels, we propose adaptively adding shadow

evaluation samples for a small fraction of potentially aliased pixels. We construct a conservative estimate of eye-view pixels that are not fully lit and not fully occluded. Multiple shadow samples are then inserted into the irregular Z-buffer based on the footprint of the light-view projection of potentially aliased pixels. Finally, the individual shadow sample occlusion values are combined into fractional and properly anti-aliased occlusion values. Our algorithm requires minimal additional storage and shadow evaluation cost but results in significantly better image quality of shadow edges and improved temporal behavior of highfrequency shadow content. Previously, architectural constraints of traditional GPUs have inhibited per-frame construction and traversal of irregular data structures in terms of both performance and programmer flexibility. Our implementation of anti-aliased irregular shadow mapping exploits many strengths of the Larrabee architecture, one of which is the ability to write to run-time computed addresses in global memory space. Additionally, we were able to do so using the conventional C programming model and incorporate the adaptive nature of our technique with little effort. In comparison, traditional GPU architectures do not offer programming semantics for such global scatter operations, or they do so at extremely low performance due to their highly specialized but constrained (localized) memory hierarchy (for example, the CUDA programming model), in the worst case falling back to main memory writes [Sintorn08, Baumann05].

Background and Problem: Shadow Edge Aliasing In this section, we‘ll describe the characteristics of various popular shadow generation algorithms and how they cope with different forms of aliasing, and we‘ll describe the problem of screen-space shadow edge aliasing, which affects many current algorithms. Pixel-Perfect Shadows with the Irregular Z-Buffer Conventional shadow mapping renders the scene from the eye and the light, and in the final compositing pass, the two views are compared to identify points that are in shadow [Williams78]. Light-view aliasing results from misalignment of these two views, as shown in Figure 1.6.1(a). There are several variants of shadow mapping that reduce but do not eliminate sampling and self-shadowing artifacts [Fernando01, Stamminger02, Sen03, Lloyd08, Lefohn07], because none of them resolves the fundamental mismatch in sampling patterns between the eye and light views, which is the root cause of most shadow mapping artifacts. Figure 1.6.1. Conventional versus irregular shadow mapping. In conventional shadow mapping (left), both the eye-view and light-view images are rendered with the classic Z-buffer, leading to a mismatch between the desired and actual sample locations in the shadow map. Irregular shadow mapping (right) avoids this mismatch by rendering the light-view image with the irregular Z-buffer.

Irregular shadow mapping addresses the root cause of visual artifacts in conventional shadow mapping by basing the light-view sampling pattern on the positions of pixels in the eye-view raster and their corresponding depth values, therefore perfectly aligning the compared occluder surface point with the projection of the shadow sample, as illustrated in Figure 1.6.1(b) [Johnson04, Johnson05]. The density of shadow samples varies significantly across the image plane (as seen in Figure 1.6.2), which illustrates the need for an irregular data structure during the light pass. Figure 1.6.2. The classic Z-buffer (a) samples a scene at regularly spaced points on the light image plane. The irregular Z-buffer (b) samples a scene at arbitrary points on the light image plane. Irregular shadow mapping (d) eliminates aliasing artifacts typically associated with conventional shadow mapping (c).

Irregular shadow mapping utilizes the irregular Z-buffer in this context. This data structure explicitly stores all of the sample locations in a two-dimensional spatial data structure rather than implicitly representing them with a regular pattern. The data structure can be any spatial data structure that supports efficient range queries, such as a k-d tree or a grid. Just as in conventional shadow mapping, irregular shadow mapping projects triangles onto the light-view image plane one at a time and then determines which samples lie inside a triangle. Unlike conventional shadow mapping, this determination is made by querying the irregular Z-buffer. Finally, for each sample inside a triangle, irregular shadow mapping performs the standard depth comparison and updates the sample‘s occlusion value.

Note that when a conventional rasterizer is used during light-view projection of occluder triangles, it is necessary to scan-convert expanded triangles to ensure fragments will be generated for any cell touched by the unexpanded triangle (also known as conservative rasterization [Akenine-Möller05]), since irregular Z-buffer samples may lie anywhere within the cell bounds, as illustrated in Figure 1.6.3. For reference, [Hasselgren05] describes a shader implementation with example code. On the other hand, the advantage of a software rasterizer (for example, on Larrabee) is that a special rasterization path can be implemented to apply custom rasterization rules that enable conservative rasterization directly without triangle expansion. Figure 1.6.3. Conservative rasterization versus conventional rasterization. Scanconverted triangles have to be expanded during light-view projection to ensure fragments will be generated for any cell touched by the unexpanded triangle (shaded cells), since irregular Z-buffer samples (circles) may lie anywhere within the pixel bounds.

Eye-View Aliasing While irregular shadow mapping is free of light-view aliasing, it still suffers from eye-view aliasing of pixels, as illustrated in Figure 1.6.4. Such aliasing is a common problem in computer graphics. For example, it is also encountered in ray casting with a single eye ray per pixel. The problem is that thin geometry (high-frequency screen content) cannot be captured by a single ray, because the rays of two neighboring pixels may miss some geometry even though the geometry projects to part of those pixels. Similarly, in the case of shadows in a rasterizer, it is possible that a surface point projected to the center of an eye-view pixel is lit, but the entire area of the pixel is not lit. This phenomenon, known as eye-view shadow aliasing, is caused by the fact that a single bit occlusion value is not sufficient to represent the occlusion value of aliased pixels. Anti-aliased occlusion values are fractional values that represent the fraction of the total pixel area that is lit. Recently, a few novel shadow mapping techniques [Brabec01, Lauritzen06, Salvi08] have addressed this problem and provide good solutions for eye-view aliasing but still expose light-view aliasing. Figure 1.6.4. The thin geometry in the tower causes eye-view aliasing of the projected shadow. Note that some of the tower’s connected features are disconnected in the shadow.

The most obvious approach to produce anti-aliased shadows with irregular shadow mapping is super-sampling of the entire screen by generating and evaluating multiple shadow samples for each eye-view pixel. The anti-aliased occlusion value for a pixel is then simply the average of the individual sample occlusion values. While this brute-force approach certainly works, as illustrated in Figure 1.6.5, the computational and storage costs quickly become impractical. Data structure construction, traversal times, and storage requirements of the irregular Z-buffer are proportional to the number of shadow samples, making realtime performance impossible on current hardware for even as little as four shadow samples per pixel. Figure 1.6.5. A four-times super-sampled irregular shadow mapping result image of the tower scene.

The recent method by [Robison09] provides a solution to compute anti-aliased shadows from the (aliased) output of irregular shadow mapping, but in essence it is also a bruteforce approach in screen space that does not exploit the irregular Z-buffer acceleration structure and is therefore at a computational disadvantage compared to our approach.

Solution: Adaptive Multi-Sampling of Irregular Shadows We observed in Figure 1.6.5 that accumulating shadow evaluation results of multiple samples per eye-view pixels provides a nice anti-aliased shadow and that potentially shadow-aliased pixels are those pixels that lie on a projected shadow edge. Therefore, we propose an efficient algorithm for anti-aliased irregular shadow mapping by adaptive multisampling of only those pixels that potentially lie on a shadow edge. Since only a marginal fraction of all screen pixels are shadow-edge pixels, this approach results in substantial gains in computational and storage costs compared to the brute-force approach. Essentially, our method is an extension of the original irregular shadow mapping algorithm, where the irregular Z-buffer acceleration structure remains a light space–oriented acceleration structure for the projected eye-view shadow samples. However, during irregular Z-buffer construction, potential shadow edge pixels are detected using a conservative shadow edge stencil buffer. Such pixels generate multiple shadow samples distributed over the pixel‘s extent and are inserted in the irregular Z-buffer (shadow sample splatting). Non-shadow-edge pixels are treated just as in the original irregular shadow mapping algorithm—a single shadow sample is sufficient to detect the occlusion value of the entire pixel. In the final shadow evaluation step, shadow occlusion values are averaged over each eye-view pixel‘s sample, resulting in a properly anti-aliased fractional occlusion value. This value approximates the fraction of the pixel‘s area that is occluded, and it goes toward the true value in the limit as the number of samples per pixel increases. Algorithm: Anti-Aliased Irregular Shadow Mapping We will now give an overview of the complete algorithm to provide structure to the remainder of the algorithm description in this section. Then we describe how to determine which pixels are potentially aliased by constructing a conservative shadow edge stencil buffer and how to splat multiple samples into the irregular Z-buffer efficiently. Finally, we put it all together and present the complete algorithm in practice. We can formulate our approach in the following top-level description of our algorithm: 1. Render the scene conservatively from the light‘s point of view to a variance shadow map. 2. Render the scene from the eye point to a conventional Z-buffer—depth values only (gives points P0). 3. Construct a conservative shadow edge stencil buffer using a variance shadow map and light-space projection of P0. 4. Using the stencil in Step 3, generate N extra eye-view samples Pi for potential shadow edge pixels only. 5. Transform eye-view samples Pi to light space P‘i (shadow sample splatting). 6. Insert all samples P‘i in the irregular Z-buffer. 7. Render the scene from the light‘s point of view while testing against samples in the irregular Z-buffer, tagging occluded samples. 8. Render the scene from the eye point, using the result from Step 7 and the conservative shadow edge stencil buffer. Multi-sampled eye-view pixels accumulate shadow sample

values into a fractional (anti-aliased) shadow value.

Conservative Shadow Edge Stencil Buffer To adaptively add shadow samples at shadow-edge pixels, we construct a special stencil buffer that answers the following question: Is there any chance that this eye-space pixel is partially occluded by geometry in this light-space texel? We call this stencil buffer the conservative shadow edge stencil buffer. Giving an exact answer to the aforementioned question is impossible because it is essentially solving the shadowing problem. However, we can use a probabilistic technique to answer the question conservatively with sufficient confidence. A conservative answer is sufficient for our purpose, since multi-sampling of nonshadow-edge pixels does not alter the correctness of the result—it only adds some extra cost. Obviously, we strive to make the stencil buffer only as conservative as necessary. We employ a technique called Variance Shadow Mapping [Lauritzen06]. Variance shadow maps encode a distribution of depths at each light-space texel by determining the mean and variance of depth (the first two moments of the depth distribution). These moments are constructed through mip-mapping of the variance shadow map. When querying the variance shadow map, we use these moments to compute an upper bound on the fraction of the distribution that is more distant than the surface being shaded, and therefore this bound can be used to cull eye-view pixels that have very little probability to be in shadow. In particular, the cumulative distribution function F(t) = P(x ≥ t) can be used as a measure of the fraction of the eye-view fragment that is lit, where t is the distance of the eye-view sample to the light, x is the occluder depth distribution, and P stands for the probability function. While we cannot compute this function F(t) exactly, Chebyshev‘s inequality gives an upper bound:

The upper bound Pmax (t) and the true probability Plit (t) are depicted in Figure 1.6.6. Thus, we can determine that it is almost certain that a projected eye-view sample with light depth t is in shadow (for example, with 99-percent certainty) by comparing Pmax to 1% (Pmax < implies Plit (t) < 0.01). Figure 1.6.6. Pin shadow (t) and Plit (t), in addition to their conservative upper bounds Pmax (t) and p′max (t).

Conversely, we can use the same distribution to construct a bound to cull eye-view pixels that have very high probability of being lit:

In summary, the conservative shadow edge stencil buffer can be constructed in the following steps: 1. Render the scene from the light‘s point of view, writing out depth x and depth squared x2 to a variance shadow map texture (VSM). 2. Mip-map the resulting texture, effectively computing E(x) and E(x2), the mean and variance of the depth distribution. 3. Render the scene from the eye point, computing for each sample: a. The light depth t by projection of the sample to light space. b. E(x) and E(x2) by texture-sampling the mip-mapped VSM with the appropriate filter width, determined by the extent of the light projection of the pixel area. c. μ = E(x), ζ2 = E(x2) – E(x) and Pmax (t), p′max (t) 4. Compare Pmax (t) and p′max (t) to a chosen threshold (for example, 1 percent). Set the stencil buffer bit if either one is smaller than the threshold.

These steps can be implemented in HLSL shader pseudocode, as shown in Listing 1.6.1. Listing 1.6.1. Conservative shadow edge stencil buffer construction HLSL shader

float2 ComputeMoments(float Depth) { // Compute first few moments of depth float2 Moments; Moments.x = Depth; Moments.y = Depth * Depth; return Moments; } float ChebyshevUpperBound( float2 moments, float mean, float minVariance) { // Compute variance float variance = max( minVariance, moments.y - (moments.x * moments.x)); float d = mean - moments.x; float pMax = variance / (variance + (d * d)); // One-tailed Chebyshev's Inequality

return (mean <= moments.x ? 1.0f : pMax); } bool IsPotentialShadowEdge(float2 texCoord, float2 texCoordDX, float2 texCoordDY, float depth) { float4 occluderData; // Variance Shadow Map mip-mapped LOD tex lookup occluderData = texShadowMap.SampleGrad( sampShadowMap, texCoord, texCoordDX, texCoordDY); float2 posMoments = occluderData.xy; // Minimum variance to take account for variance // across entire pixel float gMinVariance = 0.000001f; float pMaxLit = ChebyshevUpperBound( posMoments, depth, gMinVariance); float pMaxShadow = ChebyshevLowerBound( posMoments, depth, gMinVariance); if (PMaxLit < .01 || PMaxShadow < .01) { return true; } return false; }

Note that while conventional rasterization of the scene from the light‘s point of view in Step 1 earlier is sufficient for generating a conventional variance shadow map, it is not sufficient for generating our conservative stencil buffer. Conventional rasterization does not guarantee that a primitive‘s depth contributes to the minimum depth of each light-view texel that it touches. Hence, the conservativeness of the stencil buffer would not be preserved as illustrated in Figure 1.6.7, which depicts potential shadow-edge pixels in overlay but misses quite a few due to the low resolution of the variance shadow map. Figure 1.6.7. Conservative shadow edge stencil map with regular rasterization. Many potential shadow-edge pixels (overlay) are missed due to low resolution of the variance shadow map.

To preserve conservativeness, it is required to perform conservative rasterization in the light-view render of Step 1, just as we do during the light-view render of irregular shadow mapping illustrated in Figure 1.6.3. Figure 1.6.8 depicts correctly detected potential shadow-edge pixels in overlay, regardless of the variance shadow map resolution. Figure 1.6.8. Conservative shadow edge stencil map with conservative rasterization. All potential shadow-edge pixels are detected (overlay), regardless of the variance shadow map resolution.

Shadow Sample Splatting In the irregular Z-buffer construction phase, when the time comes to generate additional samples for potentially aliased pixels, as defined by the conservative shadow edge stencil buffer, we insert the eye-view samples in each light-view grid cell that is touched by the pixel samples. We call this process shadow sample splatting, because we conceptually splat the projection of the pixel footprint into the light space grid data structure. This process is as follows: 1. In addition to light view coordinates of the eye-view pixel center, also generate multiple samples per eye-view pixel. We have achieved very good results with

rotated grid 4× multi-sampling, but higher sample rates and even jittered sampling strategies can be used to increase the quality of the anti-aliasing. 2. Project all samples into light space as in the original irregular shadow algorithm. 3. Insert all samples into the irregular Z-buffer as in the conventional irregular shadowing algorithm. Potentially multiple light grid cells are touched by the set of samples of a pixel.

Results and Discussion We will now show the results of our algorithm for two different scenes. The first scene consists of a tower construction with fine geometry casting high-frequency shadows onto the rest of the scene. The second scene is the view of a fan at the end of a tunnel, viewed from the inside. The fan geometry casts high-frequency shadows on the inside walls of the tunnel. The tunnel walls are almost parallel to the eye and light directions, a setup that is particularly hard for many shadow mapping algorithms. Irregular shadow mapping shows its strength in the tunnel scene because no shadow map resolution management is required to avoid light-view aliasing. However, severe eye-view aliasing artifacts are present for the single-sample irregular shadow algorithm (see Figures 1.6.10(a) and 1.6.11(a)). Figure 1.6.9 illustrates the result of computing the conservative shadow edge stencil buffer on both scenes: Potential shadow edge pixels are rendered with an overlay. Figure 1.6.10 compares single-sample irregular shadows with 4× rotated grid multi-sampling on potential shadow edge pixels only. Note the significant improvement in the tower shadow, where many disconnected features in the shadow are now correctly connected in the improved algorithm. Figure 1.6.11 illustrates the same comparison for the tunnel scene. There is a great improvement in shadow quality toward the far end of the tunnel, where highfrequency shadows cause significant aliasing when using only a single eye-view shadow sample. Figure 1.6.9. Result of the conservative shadow edge stencil buffer on the tower (a) and tunnel (b) scenes. Potential shadow edge pixels are rendered with an overlay.

Figure 1.6.10. Tower scene: (a) Single-sample irregular shadows and (b) 4× rotated grid multi-sampling on potential shadow edge pixels only. Note the significant improvement in the tower shadow, where many disconnected features in the shadow are now correctly connected in the improved algorithm.

Figure 1.6.11. Tunnel scene: (a) Single-sample irregular shadows and (b) 4× rotated grid multi-sampling on potential shadow edge pixels only. There is a great improvement in shadow quality toward the end of the tunnel, where highfrequency shadows caused significant anti-aliasing when using only a single eyeview shadow sample.

Implementation Details On Larrabee, we have implemented Steps 2 and 3 of our algorithm in an efficient postprocess over all eye-view pixels in parallel. However, Step 3 is identical to the conventional irregular shadowing algorithm; therefore, it could be implemented as in [Arvo07] as well. Conceptually, we use a grid-of-lists representation for the irregular Z-buffer. This representation is well-suited to parallel and streaming computer architectures and produces high-quality shadows in real time in game scenes [Johnson05]. The following chapter of this book [Hux10], in particular Figure 1.7.1, explains our grid-of-lists representation and its construction in more detail. Finally, our solution was implemented in a deferred renderer, but it could also be implemented in a forward renderer with a few modifications. Compute Requirements Since only a marginal fraction of all screen pixels are shadow-edge pixels, this approach results in substantial gains in computational and storage costs compared to the brute-force

approach. Compared to the single-sample irregular shadow maps, the additional computational cost is relatively small. For example, let‘s assume the number of potential shadow-edge pixels is ~10 percent of all eye-view pixels, and that we generate N additional samples per potential shadow-edge pixel. Since data structure construction and traversal times are proportional to the number of shadow map samples, this means an additional cost of 10N percent for anti-aliasing. Additionally, there is an extra cost associated with creating the conservative shadow edge stencil buffer. In our implementation inside a deferred rendering, much of the required information was already computed—therefore, that extra cost is small. However, our algorithm does require an extra light-space pass, per light, to capture the depth distribution into the variance shadow map. Storage Requirements The storage cost is the same as the standard irregular Z-buffer, proportional to the number of samples. Again, for 10-percent extra samples, 10N-percent extra storage is required, depending on the implementation. Storage of the stencil buffer requires only 1 bit per eyeview pixel and can easily be packed into one of the existing eye-view buffers of the irregular shadowing algorithm. Future Work Going forward, we would like to investigate the benefits of merging our algorithm that adaptively samples potential shadow edge pixels multiple times with conventional multisampling techniques that adaptively sample the geometry silhouette pixels multiple times— for best performance, preferably through the use of common data structures and shared rendering passes. Additionally, it should be fairly straightforward to extend our approach to soft irregular shadow mapping, where the concept of anti-aliasing is implicit, as both the soft and hard irregular shadow mapping algorithms share the same algorithmic framework [Johnson09]. For soft shadows, one may envision extending the conservative stencil to shadow penumbra detection.

Conclusion The main advantage of irregular shadow maps with respect to conventional shadow maps is that they bear no light-view aliasing. However, irregular shadow maps are affected by eyeview aliasing of the shadow result. Recent pre-filterable shadow mapping algorithms and brute-force eye-view techniques have provided solutions for anti-aliased shadows, but none of them exploits the irregular Z-buffer acceleration structure directly. The method in this chapter is an extension of irregular shadow mapping, exploits the same irregular data structure, and is therefore the first algorithm to produce anti-aliased shadows by means of adaptive multi-sampling of irregular shadow maps, while keeping all its other positive characteristics, such as pixel-perfect ray-traced quality shadows and the complete lack of light-view aliasing.

Acknowledgements We‘d like to thank the people at the 3D Graphics and Advanced Rendering teams at the Intel Visual Computing group for their continued input while developing the methods described here. Many thanks go out in particular to Jeffery A. Williams for providing the art assets in the tower and tunnel scenes and to David Bookout for persistent support and feedback.

References [Akenine-Möller05] Akenine-Möller, Tomas and Timo Aila. ―Conservative and Tiled Rasterization Using a Modified Triangle Setup.‖ Journal of Graphics, GPU, and Game Tools 10.3 (2005): 1–8. [Arvo07] Arvo, Jukka. ―Alias-Free Shadow Maps using Graphics Hardware.‖ Journal of Graphics, GPU, and Game Tools 12.1 (2007): 47–59. [Baumann05] Baumann, Dave. ―ATI Xenos: Xbox 360 Graphics Demystified.‖ 13 June 2005. Beyond 3D. n.d. http://www.beyond3d.com/content/articles/4/>. [Brabec01] Brabec, Stefan and Hans-Peter Seidel. ―Hardware-Accelerated Rendering of Antialiased Shadows with Shadow Maps.‖ Proceedings of the International Conference on Computer Graphics (July 2001): 209. ACM Portal. [Fernando01] Fernando, Randima, Sebastian Fernandez, Kavita Bala, and Donald P. Greenberg. ―Adaptive Shadow Maps.‖ Proceedings of the 28th Annual Conference on Computer Graphics and interactive Techniques (2001): 387–390. ACM Portal. [Hasselgren05] Hasselgren, Jon, Tomas Akenine-Möller, and Lennart Ohlsson. ―Conservative Rasterization.‖ GPU Gems 2. 2005. NVIDIA. n.d. . [Hux10] Hux, Allen. ―Overlapped Execution on Programmable Graphics Hardware.‖ Game Programming Gems 8. Ed. Adam Lake. Boston: Charles River Media, 2010. [Johnson04] Johnson, Gregory S., William R. Mark, and Christopher A. Burns. ―The Irregular Z-Buffer and its Application to Shadow Mapping.‖ April 2004. The University of Texas at Austin. n.d. . [Johnson05] Johnson, Gregory S., Juhyun Lee, Christopher A. Burns, and William R. Mark. ―The Irregular Z-Buffer: Hardware Acceleration for Irregular Data Structures.‖ ACM Transactions on Graphics 24.4 (October 2005): 1462–1482. ACM Portal. [Johnson09] Johnson, Gregory S., Allen Hux, Christopher A. Burns, Warren A. Hunt, William R. Mark, and Stephen Junkins. ―Soft Irregular Shadow Mapping: Fast, High-Quality, and Robust Soft Shadows.‖ Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games. (2009): 57–66. ACM Portal. [Lauritzen06] Lauritzen, Andrew and William Donnelly. ―Variance Shadow Maps.‖ Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games. (2006): 161– 165. ACM Portal. [Lefohn07] Lefohn, Aaron E., Shubhabrata Sengupta, and John D. Owens. ―ResolutionMatched Shadow Maps.‖ ACM Transactions on Graphics 26.4 (Oct. 2007): 20. ACM Portal. [Lloyd08] Lloyd, D. Brandon, Naga K. Govindaraju, Cory Quammen, Steven E. Molnar, and Dinesh Manocha. ―Logarithmic Perspective Shadow Maps.‖ ACM Transactions on Graphics 27.4 (Oct. 2008): 1–32. ACM Portal. [Robison09] Robison, Austin, and Peter Shirley. ―Image Space Gathering.‖ Proceedings of the Conference on High Performance Graphics 2009 (2009): 91–98. ACM Portal.

[Salvi08] Salvi, Marco. ―Rendering Filtered Shadows with Exponential Shadow Maps.‖ ShaderX6: Advanced Rendering Techniques. Ed. Wolfgang Engel. Boston: Charles River Media, 2008. 257–274. [Sen03] Sen, Pradeep, Mike Cammarano, and Pat Hanrahan. ―Shadow Silhouette Maps.‖ ACM Transactions on Graphics 22.3 (July 2003): 521–526. ACM Portal. [Sintorn08] Sintorn, Erik, Elmar Eisemann, and Ulf Assarsson. ―Sample Based Visibility for Soft Shadows using Alias-free Shadow Maps.‖ Computer Graphics Forum: Proceedings of the Eurographics Symposium on Rendering 2008 27.4 (June 2008): 1285–1292. [Stamminger02] Stamminger, Marc and George Drettakis. ―Perspective Shadow Maps.‖ Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (2002): 557–562. ACM Portal. [Williams78] Williams, Lance. ―Casting Curved Shadows on Curved Surfaces.‖ ACM SIGGRAPH Computer Graphics 12.3 (Aug. 1978): 270–274. ACM Portal.

1.7. Overlapped Execution on Programmable Graphics Hardware Allen Hux, Intel Advanced Visual Computing (AVC) [email protected] Some graphics algorithms require data structure construction and traversal steps that do not map well to constrained graphics pipelines. Additionally, because of the dependencies between rendering and non-rendering passes, much (or all) of the compute power of the device may go idle between steps of a given algorithm. In this gem, we examine techniques for executing non-rendering algorithms concurrently with traditional rendering on programmable graphics hardware, such as Larrabee. Such programmable graphics devices enable fine-grained signaling and event graphs, allowing algorithmic stages to ―overlap.‖ As a working model, we present an implementation of Irregular Z-Buffer (IZB) shadows—an algorithm requiring both standard rendering passes (for example, depth-only pre-pass) and parallelized data structure construction. Identification of rendering and non-rendering work that is not dependent reveals opportunities to remove the stalls that currently occur when switching between the two types of workloads. The APIs discussed in this article are examples and not necessarily representative of APIs provided with a particular product.

Introduction to Irregular Z-Buffer Shadows The simplest shadow mapping algorithm requires two passes. First, render the scene from the light view to get a light-view depth buffer (at one resolution). Second, render the scene from the eye view (at the same or different resolution). For each point, test the visibility of that point by projecting it into the light view and comparing its depth to the one found in the first pass. If the eye-view depth is greater, then something must be between the light and the point being tested, and therefore it must be in shadow. This algorithm fits entirely within conventional rendering pipelines [Williams78]. The problem with this approach is aliasing: The resolution of the light-view plane will never precisely match the sampling frequency when projecting from the eye view, resulting in visual artifacts. As described in the previous gem, an irregular Z-buffer stores precise eyeview positions in the light-view grid, enabling accurate shadow determination from the eye view [Johnson04, Johnson09, Galoppo10]. Shadow mapping with an irregular Z-buffer is a multi-step process involving three rendering passes interleaved with two non-rendering steps.

1. Render the scene from the eye view, depth only. 2. Transform the eye-view points to light view. a. For each point, atomic increment the corresponding pixel in the light-view plane. (A bigger plane improves parallelism at the cost of memory.) b. Parallel prefix sum the indices, resulting in a mapping from each eye-view receiver to a light-view pixel. c. Scatter the light-view values into a light-view-friendly 1D structure. Indices into the 1D structure are stored in the light-view plane. 3. Render the scene from the light view. Instead of executing a traditional pixel shader, test the triangle bounds against the points in the data structure (referring to the indices from Step 2c). Points that are inside and behind the triangle are in shadow. Set a bit in the data structure marking this point as in shadow (occluded). 4. Create a standard shadow map by traversing the data structure and scattering out the occlusion value to a traditional 2D image. We call this the ―deswizzle‖ step. 5. Render the scene from the eye view again, using the shadow map.

For our purposes, this serves as an example of an algorithm that has some rasterization steps intermingled with some algorithmic work one might normally implement in C++. The second step in particular can result in quite a bit of idle hardware, because it requires a dependency chain of parallel and serial workloads. Assuming we have a graphics system capable of performing the compute steps of IZB (shown in Figure 1.7.1) and executing a pixel shader capable of traversing the IZB data structure, we get the simple dependency graph of tasks shown in Figure 1.7.2. Each stage in the graph cannot start until the last thread completes the last task of the prior stage. Figure 1.7.1. Building the IZB data structure, effectively a grid of lists. Regular points from the eye view (top left) are transformed into the light view (top middle). A count of the number of pixels is kept in the light grid (bottom left). A prefix sum of the counters in the light grid results in offsets (bottom right) into the 1D data structure (top right). Point data is scattered to the 1D data structure, including position and occlusion state (initially 0). The number of points in a lightview pixel can be determined by subtracting the current pixel value from the value to the right. The offset table combined with the 1D data structure forms the gridof-lists IZB data structure.

Figure 1.7.2. IZB dependency graph. (Tasks are completed in the order they are submitted.)

Because there is almost always a ―long pole‖ that determines the duration of a stage in the algorithm, nearly every thread in the system experiences some amount of idle time. This is suggested by the activity bars in Figure 1.7.2. (Imagine these correspond to the execution time on a system with four threads.) In the remainder of the gem, we will describe how we can reclaim some of those lost cycles on a programmable architecture, such as Larrabee.

Overview of Programmable Graphics Hardware As graphics devices become more general, they can be viewed as many-core compute devices with threads that can communicate amongst themselves (for example, via global variables and atomic instructions). Consider a representative programmable graphics architecture, Larrabee. Larrabee consists of many in-order cores on a single chip, each executing four threads in round-robin fashion, with an extended instruction set supporting 16-element vectors [Seiler08]. The cores are connected by a very high-bandwidth ring that maintains cache coherency. Several hardware texture samplers are distributed around the ring, as well as connections to GDDR. Traditional graphics APIs (DirectX, OpenGL) could be implemented on Larrabee as a typical process running within a relatively conventional operating system [Abrash09a].

A Larrabee architecture device could be an add-in adapter discrete from the traditional host CPU, it could be on-chip or on-die with the host CPU, or it could be the only processor in the system. For the purposes of this gem, we ignore the transport mechanism that enables our programs to run on the device, but it is important to realize that the techniques and code that follow are designed to execute directly on an architecture such as Larrabee. Implementing an efficient rasterizer within a many-threaded, 16-wide SIMD platform is beyond the scope of this chapter, but we can summarize it here. Maximizing parallelism is the main design goal. (Keeping the cores busy will be a recurring theme.) The approach described by Abrash [Abrash09b] is to use a binning architecture where the bin dimensions are chosen such that the data accessed by each core (the depth buffer format and the pixel format) does not exceed the cache size of the core (256 KB L2, 32 KB L1 in the current Larrabee architecture). Done properly, operations such as depth tests would require no offcore bandwidth (neither ring nor GDDR accesses)—unless, of course, something wants to access that depth buffer later, which merely requires a one-time write at the end. Remember that textures are accessed via hardware texture samplers, which have their own caches. Despite the attention to bandwidth, the primary motivation for binning is to produce a lot of independent work that can be executed in parallel. A programmable graphics device, such as a software rasterizer, would build a dependency graph of rendering tasks to complete (which we will call a render graph). We can expect rendering tasks to be roughly divided between front end (for example, vertex processing/binning) and back end (rasterization/pixel shading). A dependency graph enables independent tasks to run concurrently; for example, if a core completes pixel shading within a bin, it can work on a different bin or begin vertex shading for the next frame. Dependencies can be defined in terms of resources—for example, a render task may signal that it has completed writing to a render target resource (write dependency) or wait to start until a shadow map resource (texture) is ready for use (read dependency). The task affected (front end or back end) is a function of whether the resource is bound to the vertex shader or pixel shader. Commands can be inserted within the graph to be executed when a resource is ready—for example, CopySubResource of a render target. We can also create nodes in the graph that are signaled by outside events, such as direct memory access (DMA) transfer completions. We will discuss dependency graphs in more detail in the subsequent sections.

Overview of Task-Based Parallelism Non-rendering algorithms hoping to effectively use many-core architectures require an efficient tasking system, such as Cilk [Blumofe95]. Such a task system leverages a thread pool (where the number of software threads is less than or equal to the number of hardware threads) to avoid operating system overhead from switching between threads. A good tasking system also provides the following features: The ability to create individual tasks or sets of tasks (task sets). A task set calls the same task function a user-defined number of times (in parallel). Tasks and task sets may depend upon each other. Specifically, a task or task set will not start until the tasks or task sets it depends on have completed. Tasks and task sets may also depend upon user-defined events. Events can be signaled by simply calling a NotifyEvent API. An efficient work-stealing scheduler to automatically spawn and load-balance among independent tasks. Following is an example of what such a task API might look like. Tasks call a user function with user data. Task sets call a user function with user data and a number indicating which instance this is (0..numTasks-1).

// create an individual work item typedef void (* const TaskFunction) (void* in_pTaskFunctionArg); SyncObject CreateTask( TaskFunction in_pTaskFunc, void* in_pData, SyncObject* in_pDependencies, int in_numDependencies); // create work items that do a subset of work typedef void (* const TaskSetFunction) (void* in_pTaskSetFunctionArg, int in_taskIndex, int in_taskSetSize); SyncObject CreateTaskSet( int in_numTasksInSet, TaskSetFunction in_pTaskFunc, void* in_pData, SyncObject* in_pDependencies, int in_numDependencies); SyncObject CreateEvent(bool initialState);

Since the task system itself is software, common-sense performance heuristics apply: The amount of work done in the task should be sufficient to compensate for the overhead of the tasking system, which includes the function call into the task as well as some amount of communication to synchronize with internal task queues. For example, to perform an operation across a 1024×1024 image, don‘t create one million tasks. Instead, create a multiple of the number of hardware threads in the system. For a system with 100 hardware threads, 400 or 500 equally sized tasks would give the task system some opportunity to greedily load balance, while each task of approximately 2,000 pixels would give good opportunities for prefetching and loop unrolling. (A Larrabee optimized routine would operate on 16 pixels at a time, hence a loop of only about 100 iterations.) Our experiments on desktop x86 machines show that tasks of a few thousand clocks each achieve 90 percent or better overall efficiency. Efficient graphics processing requires thread affinity knowledge. That is, the rasterizer assigns threads to cores with the expectation that those threads will share render target data in their cache. Non-rendering tasks typically function more opportunistically, executing whenever a thread becomes idle. Hence, we design our tasks to be independent of the threads they may execute on. Since the hardware is programmable, finer control is possible but adds complexity. Even with equal-sized tasks, in a machine with that many threads, contention for resources (caches and GDDR bandwidth) will cause tasks to have variable durations. For very irregular workloads, such as irregular Z-buffer, optimal performance may require thousands of tasks—it all depends on the algorithm.

Combining Render Graphs and Task Graphs The ability to create non-rendering task sets that depend on, or are dependencies of, rendering tasks is the key to achieving maximum hardware utilization for these mixedusage algorithms. To do this, we need a way to interact with the rendering dependency graph from user code. Following is a method to inject a non-rendering task into the render graph, referring to the current render context (analogous to a DirectX or OpenGL context),

a function to call when the dependencies are met, user data to pass to the function, and a list of all the resource dependencies.

void CreateRenderTask(in_pRenderContext, in_pUserFunc, in_pUserData, in_pReadDependencies, in_numReadDependencies, in_pWriteDependencies, in_numWriteDependencies, out_pRenderTask);

We then need a way to notify the render system that the render task and its read and write dependencies, as declared above, are complete and available.

void NotifyRenderTaskComplete(in_pRenderTask);

Now we have a way for tasks created using our task system to define dependencies with rendering tasks and for rendering tasks to very finely interact with our task system. For example, if a render pass has pixel shaders bound to a resource declared as a write dependency of a user task, the front end of the render pass can start (transform/ bin), but the back end cannot (rasterize/pixel shade). We need a little helper glue to efficiently communicate between ―render‖ work and the tasks created for our ―client‖ work. Following, we show how an event that waits on a task set can call NotifyRenderTaskComplete() to enable dependent render work. We also show how render work can cause a callback declared in CreateRenderTask() to signal an event, thereby starting dependent client work. On the left side of Figure 1.7.3, we show how a render pass can be made to wait on client work. First, create a task set that does some client work, such as builds a data structure. Next, create a render task with the data structure (resource) written to by the task set as a write dependency, no read dependencies, and no callback. Then, create a render pass with the data structure resource as a read dependency. Finally, create a task that depends on the task set that will call NotifyRenderTaskComplete(). The render pass cannot start until the task set is complete. Figure 1.7.3. Detail of connecting client task sets to render passes via render tasks.

On the right side of Figure 1.7.3, we show how client work can be made dependent on a render pass. First, create an event that we will signal and a task set that depends on the event. (It would do work on the render target.) Next, create the render pass, which in this case writes to a render target (write dependency). Finally, create a render task with the render target as a read dependency and a callback that will set the event. The task set cannot start until the render pass is complete.

Combined Dependency Graph To reduce the idle time, we need to build a complete dependency graph including both rendering and non-rendering tasks. Below, we work with the following constraints: Tasks or task sets must have their dependencies described at creation time. Tasks or tasks sets can depend on tasks, task sets, or events. These constraints force us to work from the end of the algorithm backwards. Since a task will start immediately if it has no dependencies, we create events in a not-signaled state to act as gates for the task sets. 1. Create a build data structure event (not signaled).

2. 3. 4. 5.

Create a build data structure task set that depends on event (1). Create a deswizzle event (not signaled). Create a deswizzle task set that depends on event (3). Create a light-view render pass where: a. Rasterization depends on resource from task set (2). b. Render target resource complete signals event (3). 6. Create a final eye-view render pass where rasterization depends on the shadow map resource from task set (4). 7. Create a depth-only eye-view render pass that signals event (1) when its render target resource is complete. As soon as we complete the seventh step, creating the depth-only eye-view render with no dependencies, the whole algorithm will fall into place. As shown in Figure 1.7.4, when the depth-only pass (7) completes, it signals the build event (1), which enables the build task set (2) to start. Completion of the build task set (2) enables the light-view rasterization (5) to start. (Transform and binning should already be complete.) When the light-view render (5) completes, it signals the deswizzle event (3), which enables the deswizzle task set (4) to start. When the deswizzle task set (4) completes, it enables the final eye-view rasterization (6) to start. (Transform and binning should already be complete.) Figure 1.7.4. Order of creation of events, task sets, and render graph nodes for overlapped execution of IZB.

Idle-Free Irregular Z-Buffer Shadows Figure 1.7.5 shows the naïve linear dependency graph of the IZB algorithm discussed earlier, showing the render stages expanded into front end (transform + binning) and back end (rasterization + pixel shading) for a total of eight stages, or task sets. Figure 1.7.5. IZB dependency graph with render stages expanded into front and back end.

Figure 1.7.6 shows how threads that would otherwise have become idle can instead work on non-dependent front-end rendering tasks if we start all three render passes immediately. This overlapped execution is especially helpful for improving the performance of our irregular shadow-mapping tasks, allowing us (in this example) to fully hide the cost of the front-end rendering tasks. Another way to interpret this is that the compute part of irregular shadow mapping is essentially free when overlapped with rendering. Figure 1.7.6. IZB with dependency graph integrated with flexible rendering pipeline. Xform/bin tasks can complete as threads become available from nonrendering tasks, filling in gaps in execution.

This demonstrates another advantage of programmable hardware: Maximum performance is achieved by enabling flexible hardware to execute whatever tasks are available, rather than by partitioning the hardware into dedicated islands of computation. Compare this to the early days of graphics devices with dedicated pixel shader and vertex shader hardware: When there was more vertex work than pixel work, the idle pixel shading hardware could not be reconfigured to help out. In modern architectures, graphics processors dynamically load balance across all execution units. On a programmable architecture such as Larrabee, this load balancing can be controlled by the programmer, enabling the overlap of rendering and non-rendering tasks.

Conclusion

Maximizing performance on modern, many-core platforms requires identifying independent work to be executed in parallel. Non-graphics workloads leverage a system of dependent tasks and task sets to manage parallel computation. On programmable graphics devices such as Larrabee, a similar system of task dependencies can be used to identify independent and dependent graphics work—for example, binning versus pixel shading. By connecting these graphs, we can further exploit available execution units by exposing more opportunities for independent tasks to run concurrently.

References [Abrash09a] Abrash, Michael. ―A First Look at the Larrabee New Instructions (LRBni).‖ Dr. Dobb‘s. 1 April 2009. . [Abrash09b] Abrash, Michael. ―Rasterization on Larrabee: A First Look at the Larrabee New Instructions (LRBni) in Action.‖ Intel. March 2009. . [Blumofe95] Blumofe, Robert D., Christopher F. Joerg, Bradley C. Kuszmaul, Charles E. Leiserson, Keith H. Randall, and Yuli Zhou. ―Cilk: An Efficient Multithreaded Runtime System.‖ Proceedings of the Fifth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (July 1995): 207–216. [Galoppo10] Galoppo, Nico. ―Eye-view Pixel Anti-Aliasing for Irregular Shadow Mapping.‖ Game Programming Gems 8. Ed. Adam Lake. Boston: Charles River Media, 2010. [Johnson04] Johnson, Gregory S., William R. Mark, and Christopher A. Burns. ―The Irregular Z-Buffer and its Application to Shadow Mapping.‖ April 2004. The University of Texas at Austin. n.d. . [Johnson09] Johnson, Gregory S., Allen Hux, Christopher A. Burns, Warren A. Hunt, William R. Mark, and Stephen Junkins. ―Soft Irregular Shadow Mapping: Fast, High-Quality, and Robust Soft Shadows.‖ Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games. (2009): 57–66. ACM Portal. [Seiler08] Seiler, Larry, et al. ―Larrabee: A Many Core x86 Architecture for Visual Computing.‖ ACM Transactions on Graphics 27.3 (Aug. 2008): n.p. ACM Portal. [Williams78] Williams, Lance. ―Casting Curved Shadows on Curved Surfaces.‖ ACM SIGGRAPH Computer Graphics 12.3 (Aug. 1978): 270–274. ACM Portal.

1.8. Techniques for Effective Vertex and Fragment Shading on the SPUs Steven Tovey, Bizarre Creations Ltd. [email protected] When the Cell Broadband Engine was designed, Sony and the other corporations in the STI coalition always had one eye on the Cell‘s ability to support a GPU in its processing activities [Shippy09]. The Cell has been with us for three years now, and like any new piece of hardware, it has taken time for developers to understand the best ways of pushing the hardware to its limits. The likes of Mike Acton and the Insomniac Games Technology Team

have been instrumental in pushing general development and coding strategies for the Cell forward, but there has been little discussion about ways that the SPUs can support a GPU in its processing activities specifically. This chapter aims to introduce fundamental techniques that can be employed when developing code for the CBE that will allow it to aid the GPU in performing rendering tasks.

The CBE as Part of a Real-World System Understanding Cell‘s place in a real-world system is useful to our discussion, and, as such, we will use Sony‘s PlayStation 3 as our case study. PlayStation 3 contains the Cell Broadband Engine, which was developed jointly by Sony Computer Entertainment, Toshiba Inc., and IBM Corp. [Shippy09, Möller08, IBM08]. The Cell forms part of the overall architecture of the console along with the Reality Synthesizer, RSX, and two types of memory. Figure 1.8.1 shows a high-level view of the architecture. Figure 1.8.1. The PlayStation 3 architecture (illustration modeled after [Möller08, Perthuis06]).

The Cell contains two distinctly different types of processor: the PowerPC Processing Element (PPE) and the Synergistic Processing Element (SPE). The PPE is essentially the brains of the chip [Shippy09] and is capable of running an operating system in addition to coordinating the processing activities of its counterpart processing elements, the SPEs. Inside PlayStation 3, there are eight SPEs. However, to increase chip yield, one is locked out, and Sony reserves another for their operating system, leaving a total of six SPEs available for application programmers. All processing elements in the Cell are connected by a token-ring bus, as shown in Figure 1.8.2. Figure 1.8.2. The Cell Broadband Engine (modeled after [IBM08]).

Because the SPEs are the main focus of this chapter, they are discussed in much greater detail in the forthcoming sections.

The SPEs Each Synergistic Processing Element is composed of two major components: the Synergistic Processing Unit (SPU) and the Memory Flow Controller (MFC). The SPU Detailed knowledge of the SPU instruction set and internal execution model are critical to achieving peak performance on the PlayStation 3. In the following sections, we will highlight some important facets of this unique processor. The Synergistic Execution Unit and SPU ISA The Synergistic Execution Unit (SXU), part of the SPU, is responsible for the execution of instructions. Inside the SXU are two pipelines: the odd pipeline and the even pipeline. Instructions are issued to exactly one of these pipelines, depending on the group the issued instruction falls into (see Table 1.8.1). The SXU supports the dual issue of instructions (one from each pipeline) if and only if a very strict set of requirements is met. We will discuss these requirements in detail later. Table 1.8.1. A List of Instruction Groups Together with Their Associated Execution Pipes and Latencies Instruction Group

Latency Pipeline (Cycles)

Issue (Cycles)

Single precision floating-point operations

EVEN

6

1

Double precision floating-point operations

EVEN

7

6

Integer multiplies, integer/float conversions, and interpolation

EVEN

7

1

Immediate loads, logical operations, integer addition/subtraction, carry/borrow generate

EVEN

2

1

Element-wise rotates and shifts, special byte operations

EVEN

4

1

Loads and stores, branch hints, channel operations

ODD

6

1

Shuffle bytes, qword rotates and shifts, estimates, gather, selection mask formation and branches

ODD

4

1

The SPU has a particularly large register file to facilitate the execution of pipelined, unrolled code without the need for excessive register spilling. Unlike its counterpart, the PPE, the register file of the SPU is unified. That is, floating-point, integer, and vector operations act on the same registers without having to move through memory. As the SPU is a vector

processing unit at its heart, its Instruction Set Architecture (ISA) is designed specifically for vector processing [IBM08a]. All 128 of the SPU‘s registers are 16 bytes in size, allowing for up to four 32-bit floating-point values or eight 16-bit integers to be processed with each instruction. While a full analysis of the SPU‘s ISA is beyond the scope of this gem, there are a number of instructions worth discussing in greater detail that are particularly important for efficient programming of the SPU. The first of these instructions is selb, or ―select bits.‖ The selb instruction performs branchless selection on a bitwise basis and takes the form selb rt, ra, rb, rm. For each bit of a quadword, this instruction uses the mask register (rm) to determine which bits of the source registers (ra and rb) should be placed in the corresponding bits of the target register (rt). Comparison instructions all return a quadword selection mask that can be used with selb[1]. The fsmbi instruction is also very useful for efficiently constructing a selection mask for use with selb. [1]

The shuffle bytes instruction, shufb, is the key instruction in data manipulation on the SPU. The shufb instruction takes four operands, all of which are registers. The first operand, rt, is the target register. The next two operands, ra and rb, are the two quadwords that will be manipulated by the quadword pattern from the fourth operand, rp. The manipulations controlled by this fourth operand, known as the shuffle pattern, are particularly interesting. A shuffle pattern is a quadword value that works on a byte level. Each of the 16 bytes in the quadword controls the contents of the corresponding byte in the target register. For example, the 0th byte of the pattern quadword controls the value that will ultimately be placed into the 0th byte of the target register, the 1th byte controls the value of the 1th byte placed into the target register, and so on, for all 16 bytes of the quad word. Listing 1.8.1 provides an example shuffle pattern. Listing 1.8.1. An example shuffle pattern

const vector unsigned char _example1 = { 0x00, 0x11, 0x02, 0x13, 0x04, 0x15, 0x06, 0x17, 0x08, 0x19, 0x0a, 0x1b, 0x0c, 0x1d, 0x0e, 0x1f };

The above pattern performs a perfect shuffle, but on a byte level. (The term ―perfect shuffle‖ typically refers to the interleaving of bits from two words.) The lower 4 bits of each byte can essentially be thought of as an index into the bytes of the first or second operand quadword. Similarly, the upper 4 bits can be thought of as an index into the registers referred to in the instruction‘s operands. Since there are only two, we need only concern ourselves with the LSB of this 4-bit group—in other words, 0x0x (where x denotes some other value of the lower 4 bits of the byte) would index into the contents of the ra register, and 0x1x would access the second. It is worth noting that there are special case values that can be used to load constants with shufb; an interested reader can refer to [IBM08a] for details. A further example in Listing 1.8.2 will aid us in our understanding. Listing 1.8.2. An example of using shufb

const vector unsigned char _example2 = { 0x00, 0x01, 0x02, 0x03, 0x14, 0x15, 0x16, 0x17, 0x08, 0x09, 0x0a, 0x0b, 0x1c, 0x1d, 0x1e, 0x1f }; qword pattern = (const qword)_example2; qword ra = si_ilhu(0x3f80); // ra contains: 1.0f, 1.0f, 1.0f, 1.0f qword rb = si_ilhu(0x4000); // rb contains: 2.0f, 2.0f, 2.0f, 2.0f // result contains: 1.0f, 2.0f, 1.0f, 2.0f qword result = si_shufb(ra, rb, pattern);

In many programs, simply inlining of shuffle patterns for data manipulation requirements will suffice, but since the terminal operand to shufb is simply a register, there is nothing to stop you from computing the patterns dynamically in your program or from forming them with the constant formation instructions (as should be preferred when lower latency can be achieved than the 6-cycle load from the local store). As it turns out, dynamic shuffle pattern computation is actually critical to performing unaligned loads from the local store in a vaguely efficient manner, as we shall see later. In-depth details of the SPU ISA can be found in [IBM08a]. Local Store and Memory Flow Controller As previously mentioned, each of the SPUs in the Cell is individually endowed with its own memory, known as its local store. The local store is (at least on current implementations of the CBE) 256 KB in size and can essentially be thought of as an L1 cache for the Synergistic Execution Unit. Data can be copied into and out of the local store by way of the DMA engine in the MFC, which resides on each SPE and acts asynchronously of the SXU. Loads and stores to and from the local store are always 16-byte aligned and sized. Hence, processing data smaller than 16 bytes requires use of a less-than-efficient load-modify-store pattern. Accesses to the local store are arbitrated by the SPU Store and Load unit (SLS) based on a priority; the DMA engine always has priority over the SXU for local store accesses. Each DMA is part of a programmer-specified tag group. This provides a mechanism for a programmer to poll the state of the MFC to find out if a specific DMA has completed. A tag group is able to contain multiple DMAs. The tag group is denoted by a 5-bit value internally, and, as such, the MFC supports 32 distinct tag groups [Bader07]. The DMA queue (DMAQ) is 16 entries deep in current implementations of the CBE.

Data Management In many ways, the choice of data structure is more important than the efficiency of the operations that must be performed on it. In the following sections, we will describe a variety of data management strategies and their tradeoffs in the context of the SPU. Multi-Buffering

All graphics programmers will be familiar with the concept of a double buffer. The multibuffer is simply a term that generalizes the concept to an arbitrary number of buffers. In many cases two buffers will be sufficient, but sometimes a third buffer will be required to effectively hide the latency of transfers to and from the effective address space. Figure 1.8.3 shows the concept of multi-buffering. Figure 1.8.3. Multi-buffering data to hide latency (modeled after [Bader07]).

Bader suggests that each buffer should use a separate tag group in order to prevent unnecessary stalling of the SPU waiting for data that will be processed sometime in the future. Barriers and fences should be used to order DMAs within a tag group and the DMA queue, respectively [Bader07]. Multi-buffering can yield significant performance increases, but it does have a downside. Because the buffers are resident in the local store, it does mean that SPE programs must be careful not to exceed the 256-KB limit. Using a reasonable size for each of the buffers in your multi-buffer (about 16 KB) in order to allow the SPU to process several vertices or pixels before requiring more data from the main address space is a fine strategy. However, the pointer wrangling can become a little complicated if one‘s goal is to support a list of arbitrarily sized (and hence aligned) vertex formats. Conversely, alignments do tend to be a little more favorable and can be easily controlled by carefully selecting a reasonably sized unit of work when processing pixels. Structure-of-Arrays versus Array-of-Structures The design of data is paramount when hoping to write performant software for the SPU. Since the SPU is a SIMD vector processor, concepts familiar to those who have programmed with other vector ISAs, such as SSE on Intel chips, Altivec on PowerPC chips, or even the VU on the PlayStation 2, are immediately transferable to the SPU. One such concept is parallel array data layout, better known as Structure-of-Arrays (SOA). By laying data out in a format that is the transpose of its natural layout (Array-of-Structures), as can be seen in Figure 1.8.4, a programmer is often able to produce much more efficient code (most notably in those cases where vectorized data is interacting with scalar data).

Figure 1.8.4. An Array-of-Structures layout on the left is transposed into a Structure-of-Arrays layout (illustration modeled after [Tovey10]).

The benefits of using an SOA layout are substantial in a lot of common cases. Listing 1.8.3 illustrates this by way of computing the squared length of a vector. Listing 1.8.3. Two versions of a function to calculate the squared length of a vector. The first assumes Array-of-Structures data layout, and the second Structure-of-Arrays layout.

// Version 1: AOS mode - 1 vector, ~18 cycles. qword dot_xx = si_fm(v, v); qword dot_xx_r4 = si_rotqbyi(dot_xx, 4); dot_xx = si_fa(dot_xx, dot_xx_r4); qword dot_xx_r8 = si_rotqbyi(dot_xx, 8); dot_xx = si_fa(dot_xx, dot_xx_r8); return si_to_float(dot_xx); // Version 2: SOA mode - 4 vectors, ~8 cycles. qword dot_x = si_fm(x, x); qword dot_y = si_fma(y, y, dot_x); qword dot_z = si_fma(z, z, dot_y); return dot_z;

Branch-Free DMAs The cost of badly predicted branches on the SPU is quite significant. Given that the SPU does not contain any dedicated branch prediction hardware[2], the burden of responsibility falls squarely on the shoulders of the programmer (or in the majority of cases, the compiler). There are built-in language extensions available in most SPU compilers that allow the programmer to supply branch hints, but such things assume that you have sufficient time in order to make the prediction (that is, more than 11 cycles) and that the branch is intrinsically predictable, which may not be the case. It is therefore recommended that programmers avoid branches entirely [Acton08]. Others have discussed this topic at length [Acton08, Kapoulkine09], so I will refrain from doing so here; however, I do wish to touch upon one common case where branch avoidance is not entirely obvious but is entirely trivial. [2]

The SXU adopts the default prediction strategy that all branches are not taken.

IBM‘s SDK provides several MFC functions to initiate DMA without resorting to the manual writing of registers[3]. An unfortunate side effect of such functions is that they seem to actively encourage code such as that presented in Listing 1.8.4.

[3]

SPU-initiated DMAs are performed by the writing of special-purpose registers in the MFC using the

wrch instruction. There are six such registers that must be written in order to initiate a DMA. These may be written arbitrarily as long as the command register is written terminally [IBM08].

Listing 1.8.4. All-too-often encountered code to avoid issuing unwanted DMAs

if(si_to_uint(counter)> 0) ) mfc_put(si_to_uint(lsa), si_to_uint(ea), si_to_uint(size), si_to_uint(tag));

However, a little knowledge of the MFC can help avoid the branch in this case. The MFC contains the DMA queue (DMAQ). This queue contains SPU-initiated commands to the MFC‘s DMA engine. Similar to a CPU or GPU, the MFC supports the concept of a NOP. A NOP is an operation that can be inserted into the DMAQ but doesn‘t result in any data being transferred. A NOP for the MFC is denoted by any DMA command being written that has zero size. The resulting code looks something like Listing 1.8.5. Listing 1.8.5. Branch-free issue of DMA

qword cmp_mask qword cmp qword dma_size

= si_cgti(counter, 0x0); = si_andi(cmp_mask, 0x1); // bottom bit only. = si_mpy(size, cmp); // size < 2^16 mfc_put(si_to_uint(lsa), si_to_uint(ea), si_to_uint(dma_size), si_to_uint(tag));

Unfortunately, the hardware is not smart enough to discard zero-sized DMA commands immediately upon the command register being written, and these commands are inserted into the 16-entry DMAQ for processing. The entry into the queue is immediately discarded when the DMA engine attempts to process this element of the queue. However, this causes a subtle downside to the employment of this technique for branch avoidance. SPE programs that issue a lot of DMAs can quickly back up the DMAQ, and issuing a zero-sized DMA can stall the SPU while it flushes the entire DMAQ. Luckily, this state of affairs can be almost entirely mitigated by a well-designed SPE program, which issues fewer, but larger DMAs.

Vertex/Geometry Shading The SPUs can also lend a hand in various vertex processing tasks and, because of their general nature, can help overcome some of the shortcomings of the GPU programming model. In Blur, we were able to use the SPU to deal with awkward vertex sizes and to optimize the vehicle damage system. Handling Strange Alignments When Multi-Buffering

Vertex data comes in all shapes and sizes, and, as a result, multi-buffering this type of data presents some challenges. When vertex buffers are created, contiguous vertices are packed tightly together in the buffer to both save memory and improve the performance of the pretransform cache on the GPU. This presents an SPU programmer with a challenge when attempting to process buffers whose per-vertex alignment may not be a multiple of 16 bytes. This is a problem for two reasons. First, the DMA engine in the MFC transfers 1, 2, 4, 8, or multiples of 16 bytes, meaning that we must be careful not to overwrite parts of the buffer that we do not mean to modify. Second, loads and stores performed by the SXU itself are always 16-byte aligned [IBM08]. There are a lot of cases where a single vertex will straddle the boundary of two multibuffers, due to vertex structures that have alignments that are sub-optimal from an SPU processing point of view. The best way of coding around this problem is to simply copy the end of a multi-buffer to its nearest 16-byte boundary into the start of the second multibuffer and offset the pointer to the element you are currently processing. This means that when the second multi-buffer is transferred back to the main address space, it will not corrupt the vertices you had previously processed and transferred out of the first multibuffer, as shown in Figure 1.8.5. Listing 1.8.6 contains code demonstrating how to handle unaligned loads from the local store. Figure 1.8.5. Avoid buffer corruption by copying a small chunk from the end of one multi-buffer into the start of another.

Case Study: Car Damage in Blur The car damage system in Blur works by manipulating a lattice of nodes that roughly represent the volume of the car. The GPU implementation makes use of a volume texture containing vectors representing the offset of these nodes‘ positions from their original positions. This is then sampled based on the position of a vertex being processed relative to a volume that loosely represents the car in order to calculate position and normal offsets (see Figure 1.8.6). The texture is updated each time impacts are applied to the lattice, or when the car is repaired. Figure 1.8.6. Position and normal offsets are applied to each vertex based on deltas stored in a volume texture.

The GPU performs the deformation every frame because the damage is stateless and a function of the volume texture and the undamaged vertex data. Given the amount of work involved and the additional performance hit from sampling textures in the vertex unit, the performance of rendering cars in Blur was heavily vertex limited. This was something we wanted to tackle, and the SPUs were useful in doing so. Porting the entire vertex shader to the SPU was not practical given the timeframe and memory budgets, so instead we focused on moving just the damage calculations to the SPUs. This meant that the car damage vertex processing would only occur when damage needed to be inflicted on the car (instead of every frame with the equivalent GPU implementation), and it would greatly reduce the complexity of the vertex shader running on the GPU. The damage offsets are a function of the vertex‘s position and the state of the node lattice. Given the need for original position, we must transfer the vertex data for the cars to the local store via DMA and read the position data corresponding to each vertex. This is done using a multi-buffering strategy. Because different components of the car utilize different materials (and hence have different vertex formats), we were also forced to a variety of vertex alignments as described earlier. With the vertex data of the car in the SPU local store, we are able to calculate a position and normal offset for each vertex and write these out to a separate vertex buffer. Each of these values is stored as a float4, which means the additional vertex stream has a stride of 32 bytes per vertex. An astute GPU programmer will notice the potential to pack this data into fewer bits to improve cache utilization. This is undesirable, however. The data in its 32-bytes-per-vertex form is ideal for the DMA engine because the MFC natively works in 16-byte chunks, meaning from the point of view of other processing elements (in our case, the GPU), a given vertex is either deformed or it is not. This is one of the tradeoffs made to mitigate the use of a double buffer. Color Plate 5 has a screen-shot of this technique. To GPU Types and Back Again For the most part, GPUs do their best to support common type formats found in CPUs. The IEEE754 floating-point format is (for better or worse) the de facto floating-point standard on pretty much all modern hardware that supports floating point[4]. [4]

Ironically, the SPUs do not offer full IEEE754 support, but it‘s very close.

However, in addition to the IEEE754 standard 32-bit floats and 64-bit doubles, most shading languages offer a 16-bit counterpart known as half. The format of the half is not defined by any standard, and, as such, chip designers are free to implement their own floating-point formats for this data type on their GPUs. Fortunately, almost all GPU vendors have adopted the half format formalized by Industrial Light & Magic for their OpenEXR HDR file format [ILM09]. This format uses a single bit to denote the sign of the number, 5 bits for the exponent, and the remaining 10 bits for its mantissa or significand. Since the half type is regrettably absent from the C98 and C++ standards, it falls to the programmer to write routines to convert to other data types. Acton has made available an entirely branch-free version of these conversion functions at [Acton06]. For the general case, you would be hard-pressed to better Acton‘s code (assuming you don‘t have the memory for a lookup table as in [ILM09]). However, in many constrained cases, we have

knowledge about our data that allows us to omit support for floating-point special cases that require heavyweight conversion logic (NaNs and de-normalized numbers). Listing 1.8.6 contains code to convert between an unaligned half4 and float4 but omits support for NaNs. This is an optimization that was employed in Blur‘s damage system. The inverse of this function is left as an exercise for the reader. Listing 1.8.6. Code to convert an unaligned half4 to a qword

static inline const qword ld_float16_4(void * __restrict__ addr) { const vector unsigned char _loader = { 0x80, 0x80, 0x00, 0x01, 0x80, 0x80, 0x02, 0x03, 0x80, 0x80, 0x04, 0x05, 0x80, 0x80, 0x06, 0x07 }; const vector unsigned char _shft = { 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f }; qword qword qword qword

target val_lo val_hi sign_bit_mask sign_bit_mask qword mant_bit_mask mant_bit_mask qword expo_bias qword loader qword shft qword offset qword lo_byte_pat qword offset_pat lo_byte_pat); qword mod_shuf qword val mod_shuf); qword result aligned qword sign_bit sign_bit qword significand significand qword is_zero_mask expo_bias qword exponent_bias qword final_result return final_result; }

= = = = = = = = = = = = =

si_from_ptr(addr); si_lqd(target, 0x00); si_lqd(target, 0x10); si_ilhu(0x0); si_iohl(sign_bit_mask, 0x8000); si_ilhu(0x0); si_iohl(mant_bit_mask, 0x7fff); si_ilhu(0x3800); (const qword)_loader; (const qword)_shft; si_andi(target, 0x0f); si_ilh(0x0303); si_shufb(offset, offset,

= si_a(shft, offset_pat); = si_shufb(val_lo, val_hi, = si_shufb(val, val, loader); // = = = = = = = =

si_and(result, sign_bit_mask); si_shli(sign_bit, 0x10); si_and(result, mant_bit_mask); si_shli(significand, 0xd); si_cgti(significand, 0x0); si_and(is_zero_mask, expo_bias); si_a(significand, expo_bias); si_or(exponent_bias, sign_bit);

Benefits versus Drawbacks Processing vertex data on the SPUs has a number of advantages; one of the most significant is that the rigidity of the GPU‘s processing model is largely circumvented as you are performing processing on a general-purpose CPU. Access to mesh topology is supported, but one must be careful that these accesses do not introduce unwanted stalls as the data is fetched from the main address space. In addition, since we are using a CPU capable of general-purpose program execution, we are able to employ higher-level optimization tactics, such as early outs or faster code paths, which would be tricky or impossible under the rigid processing model adopted by GPUs. The ability to split workloads between the SPUs and the GPU is also useful in striking the ideal balance for a given application. As with most things in graphics programming, there are some tradeoffs to be made. Vertex processing on the SPU can in many cases require that vertex buffers are double buffered, meaning a significantly increased memory footprint. The situation is only aggravated if there is a requirement to support multiple instances of the same model. In this case, each instance of the base model may also require a double buffer. This can be mitigated to some extent by carefully designing the vertex format to support atomic writes of individual elements by the DMA engine, but the practicality of this is highly application-specific and certainly doesn‘t work in the case of instances. Clever use of a ring buffer can also solve this problem to some extent, but it introduces additional problems with SPU/GPU inter-processor communication.

Fragment Shading Fragment shading in the traditional sense is heavily tied to the output of the GPU‘s rasterizer. Arbitrarily ―hooking into‖ the graphics pipeline to have the SPUs perform generalpurpose fragment shading with current generations of graphics hardware is effectively impossible. However, performing the heavy lifting for certain types of fragment shading that do not necessarily require the use of the rasterizer, or even helping out the GPU with some pre-processing as in [Swoboda09], is certainly feasible and in our experience has yielded significant performance benefits in real-world applications [Tovey10]. This section discusses some of the techniques that will help you get the most out of the SPUs when shading fragments. Batch! Batch! Batch! It might be tempting with initial implementations of pixel processing code on the SPU to adopt the approach of video hardware, such as the RSX. RSX processes pixels in groups of four, known as quads [Möller08]. For sufficiently interleavable program code—in other words, program code that contains little dependency between operations that follow one another—this may be a good approach. However, in our experience, larger batches can produce better results with respect to pixel throughput because there is a greater volume of interleavable operations. Too few pixels result in large stalls between dependant operations, time that could be better spent performing pixel shading, while larger batches cause high register pressure and ultimately spilling. Moreover, in many applications that have a fixed setup cost for each processing batch, you are doing more work for little to no extra setup overhead. So, what is the upper bound on the number of pixels to process in a single batch of work? Can we simply process the entire buffer at once? The answer to this is not obvious and depends on a number of factors, including the complexity of your fragment program and the number of intermediate values that you have occupying registers at any one time. Typically, the two are inextricably linked.

As mentioned earlier, the SXU contains 128 registers, each 16 bytes in size. It is the task of the compiler to multiplex all live variables in your program onto a limited register file[5]. When there are more live variables than there are registers—in other words, when register pressure is high—the contents of some or all of the registers (depending on the size of the register file) have to be written back to main memory and restored later. This is known as spilling registers. The more pixels one attempts to process in a batch, the higher the register pressure for that function will be, and the likelihood that the compiler will have to spill registers back to the stack becomes greater. Spilling registers can become very expensive if done to excess. The optimum batch size is hence the largest number of pixels that one can reasonably process without spilling any registers back to the local store and without adding expense to the setup code for the batch of pixels. [5]

The process of mapping multiple live variables onto a limited register file is known as register coloring. Register coloring is a topic in its own right, and we will not cover it in detail here.

Pipeline Balance Is Key! An efficient, well-written program will be limited by the number of instructions issued to the processor. Those processors with dual-issue capabilities, such as the SPU, have the potential to dramatically decrease the number of cycles that a program consumes. Pipeline balance between the odd and even execution pipelines is critical to achieving good performance with SPU programs. We will now discuss the requirements for instruction dualissue and touch briefly on techniques to maximize instruction issue (through dual-issue) for those programmers writing in assembly. The SPU can dual-issue instructions under a very specific set of circumstances. Instructions are fetched in pairs from two very small instruction buffers [Bader07], and the following must all be true if dual-issue is to occur: The instructions in the fetch group must be capable of dispatch to separate execution pipelines. The alignment of the instructions must be such that the even pipeline instruction occupies an even-aligned address in the fetch group, and the odd pipeline in the odd-aligned address. Finally, there must be no dependencies either between the two instructions in the fetch group or between any one of the instructions in the fetch group and another instruction currently being executed in either of the pipelines. Programmers writing code with intrinsics rarely need to worry about instruction alignment. The addition of nops and lnops in intrinsic form does not typically help the compiler to better align your code for dual-issue, and, in many cases, the compiler will do a reasonable job of instruction balancing. However, if you‘re programming in assembly language, the use of nop (and its odd-pipeline equivalent, lnop) will be useful in ensuring that code is correctly aligned for dual-issue. Of course, care must be taken not to overdo it and actually make the resulting code slower. A good rule of thumb is never to insert more than two nops/lnops. Case Study: Light Pre-Pass Rendering in Blur Light pre-pass rendering is a variant of deferred shading first introduced by Wolfgang Engel on his blog [Engel08] and later in [Engel09, Engel09a] at the same time it was derived independently by Balestra et al. for use in Uncharted: Drake‘s Fortune [Balestra08]. The techniques behind light pre-pass rendering are well understood and are discussed elsewhere [Engel08, Balestra08, Engel09, Engel09a, Tovey10], so a brief summary will suffice here. As with all deferred rendering, the shading of pixels is decoupled from scene complexity by rendering out ―fat‖ frame buffers for use in an image space pass [Deering88, Saito90]. Light pre-pass rendering differs slightly from traditional deferred shading in that only the data

required for lighting calculations is written to the frame buffer during an initial rendering pass of the scene. This has several advantages, including a warm Z-buffer and a reduced impact on bandwidth requirements, at the expense of rendering the scene geometry twice. Because one of the main requirements for the new engine written for Blur was that it should be equipped to handle a large number of dynamic lights, the light pre-pass renderer was a very attractive option. After implementing a light pre-pass renderer for Blur (which ran on the RSX), it became apparent that we could get significant performance gains from offloading the screen-space lighting pass to the SPUs[6]. [6]

Coincidentally, it was around this time that Matt Swoboda presented his work in a similar area, in which he moved a fully deferred renderer to the SPUs in [Swoboda09]; Matt‘s work and willingness to communicate with us was useful in laying the ground work for our implementation in Blur.

The lighting calculations in Blur are performed on the SPU in parallel with other nondependent parts of the frame. This means that as long as we have enough rendering work for the RSX, the lighting has no impact on the latency of a frame. Processing of the lighting buffer is done in tiles, the selection of which is managed through the use of the SPE‘s atomic unit. When the tiles are processed, the RSX is free to access the lighting buffer during the rendering of the main pass. The results of our technique are shown in Color Plate 6 and discussed in greater detail in [Swoboda09, Tovey10]. Benefits versus Drawbacks The SPUs are powerful enough to perform fragment processing. This has been demonstrated by developers with deferred shading, post-processing, and so on [Swoboda09, van der Leeuw09, Tovey10]. While general-purpose fragment shading is not possible, it is possible to perform a plethora of image-space techniques on the SPUs, including motion blur, depth of field, shadowing, and lighting. Parallelization with other nonrelated rendering work on the GPU can provide an extra gain if one‘s goal is to minimize frame latency. Such gains can even be made without the expense of an increased memory footprint. Rasterization on the SPUs has been achieved by a number of studios with good results, but the use cases for this technique are somewhat restricted, usually being reserved for occlusion culling and the like rather than general-purpose rendering. Rasterization aside, the most serious drawback to performing fragment shading on the SPUs is the lack of dedicated texture-mapping hardware. Small textures may be feasible, as they will fit in the limited local store, but for larger textures or multiple textures, software caching is currently considered to be the best approach [Swoboda09].

Further Work Due to the highly flexible nature of the SPUs in augmenting the processing power of the GPU, it is hard to suggest avenues of further work with any certainty. However, there are a few significant challenges that warrant additional research efforts in order to further improve the feasibility of some graphics techniques on the SPUs. Texture mapping is one such avenue of research. Currently, the best that has been done is the use of a good software cache [Swoboda09] to try and minimize the latency of texture accesses from the SPUs. Taking inspiration from other convergent architectures, namely Intel Larrabee [Seiler08], we believe that the employment of user-level threads on the SPUs as a mechanism for hiding latency could certainly go some way to helping the prohibitively slow texture access speeds currently endured by graphics programmers seeking to help the GPU along with the SPUs. Running two to four copies of the same SPU program (albeit with offline modifications to the program‘s byte code) could allow a programmer to trade space

in the local store for processing speed. The idea is simple: Each time a DMA is initiated, the programmer performs a lightweight context-switch to another version of the program residing in the local store, which can be done cheaply if the second copy does not make use of the same registers. The hope is that by the time we return the original copy, the data we requested has arrived in the local store, allowing us to process it without delay. Such a scheme would impose some limitations but could be feasible for small-stream kernels, such as shaders.

Conclusion The SPUs are fast enough to perform high-end vertex and fragment processing. While they are almost certainly not going to beat the GPU in a like-for-like race (in other words, the implementation of a full graphics pipeline), they can be used in synergy with the GPU to supplement processing activities traditionally associated with rendering. The option to split work between the two processing elements makes them great tools for optimizing the rendering of specific objects in a scene. The deferred lighting and car damage systems in Blur demonstrate the potential of the SPUs to work harmoniously with the GPU to produce impressive results. Looking to the future, the ever-growing popularity and prevalence of deferred rendering techniques in current generations of hardware further empower the SPUs to deliver impressive improvements to the latency of a frame and allow game developers to get closer to synthesizing reality than ever before.

Acknowledgements I would like to thank the supremely talented individuals of the Bizarre Creations Core Technologies Team for being such a great bunch to work with, with special thanks reserved for Steve McAuley for being my partner in crime with our SPU lighting implementation. Thanks also go to Andrew Newton and Neil Purvey at Juice Games for our numerous discussions about SPU coding, to Matt Swoboda of SCEE R&D for our useful discussions about SPU-based image processing, and to Wade Brainerd of Activision Central Technology for his helpful comments, corrections, and suggestions. Last but not least, thanks also to Jason Mitchell of Valve for being an understanding and knowledgeable section editor!

References [Acton06] Acton, Mike. ―Branch-Free Implementation of Half Precision Floating Point.‖ CellPerformance. 17 July 2006. Mike Acton. 2 July 2009. . [Acton08] Acton, Mike and Eric Christensen. ―Insomniac SPU Best Practices.‖ Insomniac Games. 2008. Insomniac Games. 2 July 2009. . [Bader07] Bader, David A. ―Cell Programming Tips & Techniques.‖ One-Day IBM Cell Programming Workshop at Georgia Tech. 6 Feb. 2007. Georgia Tech College of Computing. 2 July 2009. .

[Balestra08] Balestra, Christophe and Pal-Kristian Engstad. ―The Technology of Uncharted: Drake‘s Fortune.‖ Game Developers Conference. 2008. Naughty Dog Inc. n.d. . [Deering88] Deering, Michael, et al. ―The Triangle Processor and Normal Vector Shader: A VLSI System for High Performance Graphics.‖ Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques (1988): 21–30. ACM Portal. [Engel08] Engel, Wolfgang. ―Light Pre-Pass Renderer.‖ Diary of a Graphics Programmer. 16 March 2008. Blogspot.com. 4 July 2009. . [Engel09] Engel, Wolfgang. ―Designing a Renderer for Multiple Lights: The Light Pre-Pass Renderer.‖ ShaderX7: Advanced Rendering Techniques. Ed. Wolfgang Engel. Boston: Charles River Media, 2009. 655–666. [Engel09a] Engel, Wolfgang. ―The Light Pre-Pass Renderer Mach III.‖ To appear in proceedings of ACM SIGGRAPH09, 2009. [IBM08] ―Cell Broadband Engine Programming Handbook.‖ IBM. 19 April 2006. IBM. n.d. https://www01.ibm.com/chips/techlib/techlib.nsf/techdocs/7A77CCDF14FE70D5852575CA0074E8ED>. [IBM08a] ―Synergistic Processing Unit Instruction Set Architecture.‖ IBM. 27 Jan. 2007. IBM. n.d. [IBM09] ―The Cell Project at IBM Research.‖ IBM. n.d. IBM. 4 July 2009. . [ILM09] ―OpenEXR.‖ OpenEXR. n.d. Lucas Digital Limited. 4 July 2009. . [Kapoulkine09] Kapoulkine, Arseny. ―View frustum culling optimization—never let me branch.‖ What Your Mother Never Told You About Graphics Development. 1 March 2009. Blogspot.com. 21 July 2009. . [Möller08] Akenine-Möller, Thomas, Eric Haines, and Naty Hoffman. Real-Time Rendering, 3rd Edition. Wellesley, MA: A.K. Peters, Ltd, 2008. [Perthuis06] Perthuis, Cedric. ―Introduction to the Graphics Pipeline of the PS3.‖ Eurographics 2006. Austrian Academy of Sciences, Vienna, Austria. 6 Sept. 2006. [Saito90] Saito, Takafumi and Tokiichiro Takahashi. ―Comprehensible Rendering of 3-D Shapes.‖ ACM SIGGRAPH Computer Graphics 24.4 (1990): 197-206. ACM Portal. [Seiler08] Seiler, Larry, et al. ―Larrabee: A Many Core x86 Architecture for Visual Computing.‖ ACM Transactions on Graphics 27.3 (Aug. 2008): n.p. ACM Portal. [Shippy09] Shippy, David and Mickie Phipps. The Race for a New Games Machine: Creating the Chips Inside The New Xbox360 & The Playstation 3. New York: Citadel Press, 2009. [Swoboda09] Swoboda, Matt. ―Deferred Lighting and Post Processing on PLAYSTATION®3.‖ Game Developers Conference. 2009. Sony Computer Entertainment Eurpoe, Ltd. n.d.

. [Tovey10] Tovey, Steven and Steven McAuley. ―Parallelized Light Pre-Pass Rendering with the Cell Broadband Engine™.‖ GPU Pro: Advanced Rendering Techniques. Natick, MA: A K Peters Ltd., 2010. [van der Leeuw09] van der Leeuw, Michiel. ―The PLAYSTATION3‘s SPUs in the Real World— KILLZONE2 Case Study.‖ Game Developers Conference 2009. Moscone Center, San Francisco, CA. 25 March 2009.

Section 2: Physics and Animation Introduction A Versatile and Interactive Anatomical Human Face Model Curved Paths for Seamless Character Animation Non-Iterative, Closed-Form, Inverse Kinematic Chain Solver (NCF IK) Particle Swarm Optimization for Game Programming Improved Numerical Integration with Analytical Techniques What a Drag: Modeling Realistic Three-Dimensional Air and Fluid Resistance Application of Quasi-Fluid Dynamics for Arbitrary Closed Meshes Approximate Convex Decomposition for Real-Time Collision Detection

Introduction Jeff Lander, Darwin 3D, LLC [email protected] Game creation as a business and an art has become much more mature, in years of experience, in complexity, and in controversy of the material covered. For the most part, long gone are the days when text adventures, simple frame flipping, and sprite-based animation ruled the top-ten lists of gamers‘ hearts. Our games need to be much more real and complex to compete with the ever-increasing expectations of our audience. Nowhere is this more evident than in the character performances and physical simulations of real (or imaginary) worlds. Players exposed to the amazing worlds that television and filmmakers can create with visual effects rightly believe that their games should reflect these advances and expectations as well. We need to bring our characters to life. We need to create worlds where the rules and systems that govern the reality have a basis in physical realism and have a consistency that is at once familiar and exciting for our players. We are past the point where players can be amazed by a simple animation clip of a character running or watching a ball bounce on the ground. They have seen that all before. They now expect a game‘s characters to react to the worlds around them in an intelligent way, as well as interact with the world in a way that models the physical interactions in our real-life experiences.

The gems in this section represent the intersection of the physical interactions and animated performances that we need in order to bring more of the illusion of life to our characters and worlds. In ―A Versatile and Interactive Anatomical Human Face Model,‖ Marco Fratarcangeli discusses how to bring more realistic movement to facial animation to directly attack the problems with facial performance. This gem models the underlying facial animation systems with physical simulation. In this same way, physical simulation is used in the gems ―Application of Quasi-Fluid Dynamics for Arbitrary Closed Meshes‖ and ―What a Drag: Modeling Realistic Three-Dimensional Air and Fluid Resistance‖ to improve the realism in our interactive worlds. ―Particle Swarm Optimization for Game Programming‖ discusses applying easy-to-use particle simulation techniques to a variety of optimization problems. Much of this is pretty advanced stuff. We will not be discussing how to play back an animation on a hierarchical character or how to simulate and detect a collision between two objects. You are expected to be masters of that kind of low-level system by now. We are attacking larger problems now. For example, it is no longer sufficient for our characters to follow a simple piecewise linear path when moving across a space. That simply looks too mechanical and robotic. In ―Curved Paths for Seamless Character Animation,‖ Michael Lewin discusses how it is necessary to smooth the results from our AI pathfinding systems to create a movement path that follows a much more realistic curve, while still avoiding all the obstacles that may be in the way. Adding to our character performance improvements, in Philip Taylor‘s ―Non-Iterative, Closed-Form, Inverse Kinematic Chain Solver,‖ the existing iterative IK techniques are improved with an easy-to-understand, closed-form solution. It is also not enough to take for granted little code snippets for numerical integration we have seen posted on the net. It is important now for us to have a deeper understanding of what is going on when we use something such as Euler integration for a physical simulation and why it is important that we understand the error inherent in these algorithms. ―Improved Numerical Integration with Analytical Techniques‖ looks directly at these issues and proposes methods to increase the accuracy in our simulations. As we continue to create amazing new projects and push the envelope for what is possible to do in a game, I believe some of the most important steps forward will come at this intersection of animation and physics. As our animated characters become more physically aware and grounded in our game environments, and our simulated worlds become inhabited by these responsive and emotional characters, our games will make huge leaps forward in connecting to our audience. I hope these gems provoke some new ideas and encourage you all to inject just a little more life into your virtual creations.

2.1. A Versatile and Interactive Anatomical Human Face Model Marco Fratarcangeli [email protected] In a compelling virtual world, virtual humans play an important role in improving the illusion of life and interacting with the user in a natural way. In particular, face motion is crucial to represent a talking person and convey emotive states. In a modern video game, approximately 5 to 10 percent of the frame cycle is devoted to the animation and rendering of virtual characters, including face, body, hair, cloth, and interaction with the surrounding virtual environment and with the final user. Facial blend shapes are commonly adopted in the industry to efficiently animate virtual faces. Each blend shape represents a key pose of the face while performing an action (for example, a smiling face or raised eyebrows). By interpolating the blend shapes with each other, as depicted in Figure 2.1.1, the artist can achieve a great number of facial expressions in real time at a negligible computational cost. Figure 2.1.1. Blend shapes interpolation.

However, creation of the facial blend shapes is a difficult and time-consuming task, and, more likely than not, it is still heavily dependant on the manual work of talented artists. In this gem, I share principles and ideas for a tool that assists the artist in authoring humanoid facial blend shapes. It is based on a physical simulation of the human head, which is able to mimic the dynamic behavior of the skull, passive tissues, muscles, and skin. The idea is to let the artist design blend shapes by simply adjusting the contraction of the virtual muscles and rotating the bony jaw. The anatomical simulation is fast enough to feed back the results in real time, allowing the artist to tune the anatomical parameters interactively.

Overview Our objective is to build a virtual model of the human head that simulates its anatomy and can be adapted to simulate the dynamics of different facial meshes. The artist models a static face mesh, and then the anatomical simulation is used to generate its blend shapes. The goal of the simulation is to create shapes that generate realistic motion. The exact accuracy of the individual muscles in a real anatomical model is not the goal. The anatomical elements are simple yet expressive and able to capture the dynamics of a real head. The anatomical simulation must also be fast enough to be interactive (in other words, run at least at 30 fps) to allow the artist to quickly sketch, prototype, tune, and, where needed, discard facial poses. The basic anatomical element is the skull. The anatomical model is not bound to the polygonal resolution or to the shape of the skull mesh; we require only that the skull mesh has a movable jaw. On top of the skull, the artist may design several layers of muscles and passive tissue (such as the fat under the cheeks), the so-called muscle map. The skull and the muscle map form the musculoskeletal structure that can be saved and reused for different faces. The musculoskeletal structure is morphed to fit the shape of the target face. Then, the face is bound to the muscles and to the skull, and thus it is animated through a simple and efficient numerical integration scheme.

Numerical Simulation The anatomical model is composed of different parts, most of which are deformable bodies, such as muscles, fat, and the skin. The dynamics algorithm must be stable enough to allow interaction among these parts, it must be computationally cheap to carry out the computation at an interactive rate, and it must be controllable to allow precise tuning of the muscles‘ contractions. To meet these requirements, we will use Position Based Dynamics (PBD), a method introduced in [Müller06] and recently embedded in the PhysX and Bullet engines. A less formal (although limited) description was introduced in [Jakobsen03] and employed in the popular game Hitman: Codename 47.

For an easy-to-understand and complete formulation of PBD, as well as other useful knowledge about physics-based animation, you can review the publicly available SIGGRAPH course [Müller08]. In this gem, I describe PBD from an implementation point of view and focus on the aspects needed for the anatomical simulation. In most of the numerical methods used in games, the position of particles is computed starting from the forces that are applied to the physical system. For each integration step at a given time, we obtain velocities by integrating forces, and eventually we obtain the particle‘s position by integrating velocities. This means that, in general, we can only influence a particle‘s position through forces. PBD works in a different way. It is based on a simple yet effective concept: The current position of particles can be directly set according to a customizable set of geometrical constraints , where is the set of particles involved in the simulation. Because the constraints could cause the resulting position to change slightly, the velocity must be recalculated using the new position and the position at the previous time step. As the position is computed, the velocity is adjusted according to the current position and the position at the previous time step. The integration steps are: (1) for each particle i do (2) for each particle i do (3) loop nbIterations times (4) solve (5) for each particle i (6) For example, let us consider the simple case of two particles traveling in the space, which must stick to a fixed distance d from each other. In this case, there is only one constraint C, which is: Equation 1

Assuming a particle mass m = 1kg, Step (3) in the Algorithm 1 is solved by:

In this example, if a force is applied to force, and it will move. Then, both maintain the given distance d.

and

, it will gain acceleration in the direction of the will be displaced in Step (3) in order to

Given , finding the change in position Δpi for each particle is not difficult; however, it requires some notions of vectorial calculus, which are outside the scope of this gem. The process is explained in [Müller06, Müller08] and partly extended in [Müller08b]. Steps (3) and (4) solve the set of constraints in an iterative way. That is, each constraint is solved separately, one after the other. When the last constraint is solved, the iteration starts again from the first one, and the loop is repeated nbIterations times, eventually converging to a solution. Then, the velocity is accommodated in order to compensate the change in position Δpi. In general, using a high value for nbIterations improves the precision of the solution and the stiffness of the system, but it slows the computation. Furthermore, an exact solution is not always guaranteed because the solution of a constraint may violate another constraint. This issue is partly solved by simply multiplying the change in position Δpi by a scalar constant K∊[0,..,1], the so-called constraint stiffness. For example, choosing k< for a distance constraint leads to a dynamics behavior similar to the one of a soft spring. Using soft constraints leads to soft dynamics and improves drastically the probability of finding an acceptable (and approximated) solution for the constraint set. For example, think about two rigid spheres that must be accommodated inside a cube with a diagonal length smaller than the sum of the diameters of the two spheres: The spheres simply will not fit. However, if the spheres were soft enough, they would change shape and eventually find a steady configuration to fit in the cube. This is exactly how soft constraints work: If they are soft enough, they will adapt and converge toward a steady state. In my experiments, I used a time step of 16.6 ms and found that one iteration is enough in most of the cases to solve the set of constraints. Rarely did I use more iterations, and never beyond four. Similar to the distance constraint, other constraints can be formulated considering further geometric entities, such as areas, angles, or volumes. The set of constraints defined over the particles defines the dynamics of a deformable body represented by the triangulated mesh. I provide the source code for the distance, bending, triangular area, and volume constraints on the companion CD-ROM. You are encouraged to experiment with PBD and build deformable bodies following the examples in the source code. PBD has several advantages: The overshooting problem typical of force-driven techniques, such as mass-spring networks, is avoided. You can control exactly the position of a subset of particles by applying proper constraints; thus, the remaining particles will displace accordingly. PBD scales up very well with spatial dimensions because the constraint stiffness parameter is an adimensional number, without unit of measure.

Building the Anatomical Model We begin the process of building our virtual head by building up the low-level pieces that make up the foundation of the physical motion. To do this, we take an anatomical approach. The Skull

The skull is the basis upon which the entire computational model of the face is built. It is represented by a triangulated mesh chosen by the artist. Figure 2.1.2 illustrates an example of the mesh used in our prototype. The skull mesh is divided in two parts: the fixed upper skull and the movable mandible. The latter will move by applying the rigid transformation depicted in Figure 2.1.2. Figure 2.1.2. (1) Example skull mesh, (2) jaw lateral slide, (3) jaw opening, (4) jaw protruding.

Interactive Sketching of the Muscles In our model, muscles are represented by rectangular parallelepipeds, which are deformed to match the shape of the facial muscles. To define the shape of the muscles, we draw a closed contour directly on the skull surface and the already-made muscles. Figure 2.1.3 shows an example of the definition of the initial shape of a muscle in rest state. The closed contour is defined upon underlying anatomical structures—in this case, the skull. Then, a hexahedral mesh M is morphed to fit the contour. M is passively deformed as the skull moves. Figure 2.1.3. Defining the shape of a muscle on the skull.

The closed contour is drawn through a simple ray-casting algorithm. The position of the pointing device (I used a mouse) is projected into the 3D scene, and a ray is cast from the near plane of the frustum to the far plane, as shown in Figure 2.1.4. Figure 2.1.4. Casting a ray to determine the intersection point.

The intersection points form the basis of the muscle geometry, the so-called action lines. An action line is a piecewise linear curve lying on at least one mesh. The purpose of the action lines is twofold: (1) They define the bottom contour of the muscle geometry during the simulation, and (2) they provide a mechanism to control the active contraction of the muscle itself. A surface point is a point sp in S, where S is the surface represented by the mesh. A surface point sp is uniquely described by the homogeneous barycentric coordinates (t1, t2, t3) with respect to the vertices A1A2A3 of the triangular facet to which it belongs, as shown in Figure 2.1.5. Figure 2.1.5. (a) A surface point sp in A1A2A3 is defined by the homogeneous barycentric coordinates (t1, t2, t3) with respect to the triangle vertices. (b) When the triangle deforms, the triple (t1, t2, t3) does not change, and sp is updated to sp’.

The relevant attributes of a surface point are position and normal; both of them are obtained through the linear combination of the barycentric coordinates with the corresponding attributes of the triangle vertices. When the latter displace due to a deformation of the triangle, the new position and normal of sp are updated and use the new attributes of the vertices (Figure 2.1.5 (b)). Each linear segment of the action line is defined by two surface points; thus, an action line is completely described by the ordered list of its surface points. Note that each single surface point may belong to a different surface S. So, for example, an action line may start on a surface, continue on another surface, and finish

on a third surface. When the underlying surfaces deform, the surface points displace, and the action line deforms accordingly. Soft Model for a Facial Muscle Starting from a triangulated, rectangular parallelepiped (or hexahedron), each vertex is considered as a particle with mass m = 1; particles are connected with each other to form a network of distance constraints, as shown in Figure 2.1.6. Figure 2.1.6. Distance constraints used in the muscle model.

These constraints replicate some aspects of the dynamic behavior of a real face muscle, in particular resistance to in-plane compression, shearing, and tension stresses. Note that distance constraints are placed over the surface of the hexahedron, not internally. For completing the muscle model, we add bending constraints among the triangular faces of the mesh to conserve superficial tension. We also add a further volume constraint over all the particles, which makes the muscle thicker due to compression and thinner due to elongation. The Muscle Map Using the interactive editor to sketch the shape of the soft tissues over the skull, we define the structure made up of intertwined muscles, cartilage, and facial tissue, mostly fat. The muscles are organized in layers. Each layer influences the deformation of the layers on top of it, but not those underlying it. See Figure 2.1.7. Figure 2.1.7. Different layers forming the muscle map used in the experiments.

The muscle map comprises 25 linear muscles and one circular muscle. This map does not represent the real muscular structure of the human head; this is due to the simulated muscle model, which has simplified dynamics compared to the real musculoskeletal system. However, even though there may be not a one-to-one mapping with the muscle map in a real head, this virtual muscle map has been devised to mimic all the main expressive functionalities of the real one. For instance, on the forehead area of a real head, there is a single large, flat sheet muscle, the frontalis belly, which causes almost all the motion of the eyebrows. In the virtual model, this has been represented by two separate groups of muscles, each one on a separate side of the forehead. Each group is formed by a flat linear muscle (the frontalis) and on top of it two additional muscles (named, for convenience, frontalis inner and frontalis outer). On top of them, there is the corrugator, which ends on the nasal region of the skull. Combining these muscles, the dynamics of the real frontalis belly are reproduced with a satisfying degree of visual realism, even though the single linear muscle models have simple dynamics compared to the corresponding real ones. Each simulated muscle is linked to the underlying structures through position constraints following the position of surface points. Thus, when an anatomical structure deforms, the entire set of surface points lying on it moves as well, which in turn influences the motion of the above linked structures. For instance, when the jaw, which is part of the deepest layer, rotates, all the deformable tissues that totally or partially lie on it will be deformed as well, and so on, in a sort of chain reaction that eventually arrives at the skin. Active contraction of a muscle is achieved by simply moving the surface points along the action lines. Given that the bottom surface of the muscles is anchored to the surface points through position constraints, when the latter move, the muscle contracts or elongates, depending on the direction of motion of the surface points. Figure 2.1.8. The example muscle map is deformed by rotating the jaw and contracting the frontalis belly. Note how all the above muscles are properly deformed.

Morphing the Muscle Map into the Target Face Mesh Once the skull and the muscle map are ready, they can be morphed to fit inside the target facial mesh, which represents the external skin. The morphing is done through an interpolation function, which relies on the so-called Radial Basis Functions [Fang96]. We define two sets of 3D points, P and Q. P is a set of points defined over the surface of the skull and the muscles, and Q is a set of points defined over the skin mesh. Each point of P

corresponds to one, and only one, point of Q. The position of the points in P and Q are illustrated in Figure 2.1.9 (a) and must be manually picked by the artist. These positions have been proven to be effective for describing the shape of the skull and of the human face in the context of image-based coding [Pandzic and Forchheimer02]. Given P and Q, we find the interpolation function G(p), which transforms a point pi in P and a point qi in Q. Once G(p) is defined, we apply it to all the vertices of the skull and muscle meshes, fitting them in the target mesh. Figure 2.1.9. (a) The set of points P and Q picked on the skull and on the face mesh, respectively. (b) The outcome of the morphing technique.

Finding the interpolation function G(p) requires solving a system of linear independent equations. For each couple 〈pi,qi〉, where pi is in P and qi is the corresponding point in Q, we set an equation like:

Where n is the number of points in each set, di is the distance of pi from pj, ri is a positive number that controls the ―stiffness‖ of the morphing, and hi is the unknown. Solving the system leads to the values of hi, i = 1, .., n, and thus G(p):

This particular form of Radial Basis Functions is demonstrated to always have a solution, it is computationally cheap, and it is particularly effective when the number of points in P and Q is scarce. Figure 2.1.9 (b) shows an example of fitting the skull into a target skin mesh. Skin

Skin is modeled as a deformable body; its properties are defined by geometrical constraints in a similar way to the muscles. The skin is built starting from the target face mesh provided in input. Each vertex in the mesh is handled as a particle with a mass, which is set to 1.0. After the skull and the muscle map are fitted onto the skin mesh, further constraints are defined to bind the skin to the underlying musculoskeletal structure. For each particle p in the skin mesh, a ray is cast along the normal that is toward the outer direction. In fact, after the fitting, portions of some muscles may stay outside the skin. By projecting in the outer direction, the skin vertices are first bound to these muscles. If no intersection is found, then another ray is cast in the opposite direction of the normal, toward the inner part of the head. The ray is tested against the muscles from the most superficial to the deepest one. If the ray does not intersect any muscle, then the skull is tested. The normal of the skin particle is created by averaging the normals of the star of faces to which the particle belongs. If an intersection is found, then it is defined as a surface point sp on the intersected triangular face in the position where the ray intersects the face. A particle q is added to the system, and it is bound through a position constraint to sp. A stretching constraint is placed among the particles p and q. When the skull and the muscles move, the position of the surface points will change accordingly. The set of added particle q is updated as well because it is bound to the surface points through the corresponding position constraint and will displace the skin particles, whose final motion will depend also on the other involved constraints.

Conclusion Although not very accurate from the point of view of biomechanics, the presented anatomical model is able to simulate convincing facial poses, including macro-wrinkles, which can be used as blend shapes, as shown in Color Plate 7. The model is stable, robust, and controllable, which is critical in interactive tools for producing video game content. The anatomical model is adaptive enough to animate directly the face mesh provided by the artist, thus it does not lead to the artifacts associated with motion retargeting. It does not require expensive hardware, and it may run on a consumer-class PC while still providing interactive feedback to the artist.

References [Fang96] Fang, Shiaofen, Raghu Raghavan, and Joan T. Richtsmeier. ―Volume Morphing Methods for Landmark Based 3D Image Deformation.‖ International Symposium on Medical Imaging. 2710 (1996): 404–415. [Fratarcangeli08]. Fratarcangeli, Marco. ―A Computational Musco-Skeletal Model for Animating Virtual Faces.‖ Ph.D. thesis, Universitá degli Studi di Roma ―La Sapienza.‖ 2008. [Pandzic and Forchheimer02] Pandzic, Igor S. and Robert Forchheimer. MPEG-4 Facial Animation—The Standard, Implementation and Applications, 1st ed. John Wiley & Sons, 2002. [Jakobsen03] Jakobsen, T. ―Advanced Character Physics.‖ 2003. Gamasutra. n.d. . [Müller06] Müller, M., B. Heidelberger, M. Hennix, and J. Ratcliff. ―Position Based Dynamics.‖ J. Vis. Commun. 18.2 (2007): 109–118.

[Müller08] Müller M., D. James, J. Stam, and N. Thuerey. ―Real Time Physics.‖ 2008. Matthias Muller. n.d. . [Müller08b] Müller, M. ―Hierarchical Position Based Dynamics.‖ Proceedings of Virtual Reality Interactions and Physical Simulations. Grenoble, 2008.

2.2. Curved Paths for Seamless Character Animation Michael Lewin [email protected] The latest generation of consoles brings with it the promise of almost lifelike graphics and animation. Yet for all the beautiful artwork, we are still seeing artifacts, such as character sliding, that break the illusion of the virtual world. Sliding occurs when we apply any extra rotation or translation to an animation that does not correspond to the way the character‘s limbs are moving. This causes movement in which the feet are not planted firmly on the ground, such as running on the spot, shifting left or right while walking, rotating unnaturally while walking or standing, or sliding upwards or downwards while walking on stairs. A tension exists between our desire for realistic human animation and our need to maintain precise control of the character‘s position, orientation, and velocity. In many projects, there is not sufficient synergy between the AI and animation layers to satisfy both requirements simultaneously. This gem presents a general way to adapt pathfinding techniques to better interact with animation selection. We developed a technique at Sony Computer Entertainment that uses cubic Bezier curves to allow the AI system to be more closely coupled with dynamics and animation, considering the constraints of the character‘s movement.

Related Work [Johnson06] describes a method for fitting Bezier curves inside the cells of a navigation mesh to create a smooth path. This gem describes in a more general way that Bezier curves can be fitted to any piecewise linear path.

Finding a Path All characters need to move around their environment, which usually involves fast-paced dynamic changes and contains unpredicted obstacles. This problem is just as relevant to robot motion planning in the real world as it is to character movement in simulations such as games, and there is a wealth of literature on the subject applied to both domains. The vast majority of pathfinding solutions create a piecewise linear path first and then fit a curved path to it. There are a few examples where this intermediate step is not needed, such as using potential fields [Ratering95], but they do not give the same level of control as other methods. I will not consider them further in this article. There are many different choices of representation when constructing a piecewise linear path. The A* algorithm is usually applied because it is simple, fast, and guaranteed to be optimal. Where techniques for generating pathfinding solutions differ lies in how to represent the physical space. It can be represented as a set of connected cells or points. I briefly explain some of the main techniques; a thorough review can be found in [Tozour03]

or [Latombe91], and an edifying demo can be downloaded at [Tozour05]. Figure 2.2.1 shows an example that compares the results of using the various methods. Figure 2.2.1. The various methods of creating a piecewise linear path: (a) regular grid, (b) quad tree, (c) navigation mesh, (d) Voronoi diagram, and (e) visibility graph.

The simplest method is to divide up the space into a grid of regular rectangular cells with a resolution that is small compared to the size of obstacles. A slightly more sophisticated

version of this is to use a quad tree instead of identical rectangles, so that areas of different resolution are defined as needed. Navigation meshes, as described in [Snook00, Tozour02], are an increasingly popular alternative to the grid solutions described previously. They partition the space using a variety of irregular polygons so that each cell is either entirely filled by an obstacle or entirely empty. The partitioning may be performed by hand when the environment is authored, or it can be generated automatically and then hand-edited later. [Axelrod08, Marden08] describe algorithms for dynamically recalculating the mesh to account for moving obstacles. Voronoi diagrams are on first appearance similar to navigation meshes, but they yield quite different paths. Obstacles are represented as a finite set of points, and the space is partitioned such that each point is inside a region, where that region is defined as the set of points closer to that obstacle point than any other. The edges of the resulting diagram can define paths around the space provided that all edges that pass through an obstacle are removed first. Visibility graphs are different from the previously described methods because they generate a set of waypoints rather than partitioning the space into cells. The graph is defined by points in the environment, in which pairs of points are connected by an edge if they can be joined by a straight line without intersecting any obstacles. The points are selected to be close to the corners of obstacles (this is known as a corner graph), so that the character can pass close to obstacles without colliding with them. A good demonstration of the technique can be seen at [Paluszewski]. These methods are shown in Figure 2.2.1.

Smoothing the Path The AI programmer‘s challenge is to create a piecewise linear path through the environment so that the character appears as lifelike and natural as possible, which means it cannot include jagged edges. Piecewise polynomial parametric curves, more commonly known as splines, are well suited to this task, and in particular cubic Bezier curves. This is a parametric curve of the form:

As Figure 2.2.2 illustrates, the control points P0 and P3 define the start and end points, respectively, while the control points P1 and P2 define the curvature at the end points. The vectors P1-P0 and P3-P2 are referred to as the start and end velocity, respectively, because they define the tangent and curvature at the two end points. In practical terms, this means we can adjust their direction to control the character‘s initial and final direction of motion, and we can adjust their magnitude to control the curvature of the curve. Another useful property is that the curve will always lie within the boundary of the quadrilateral convex hull defined by the four control points. Figure 2.2.2. Bezier curves are well suited to path fitting. The four control points define the curve’s position, shape, and curvature. The curve will always be contained inside the convex hull defined by the control points.

Having constructed a piecewise linear path, we can fit cubic Bezier curves to it for a smooth path. By choosing the control points so that for each consecutive pair of curves the output direction of the first matches the input direction of the second, we achieve continuity and smoothness at the endpoints. Our objective is to adhere to the linear path only as much as necessary. If the character‘s momentum is high and a large sweeping curve looks most natural, and provided there are no obstacles in the way of that path, we want to select such a path rather than sticking rigidly to the linear path. However, if the path is very close to obstacles and there is little room to maneuver, we want a path that is as curved as possible but still avoids those obstacles. The solution to this problem comes from an unlikely source: The open source graph plotting software Graphviz [Graphviz] makes use of the same technique for drawing edges between its nodes [Dobkin97]. The inspiration comes from a 1990 article in the first Graphics Gems book that was presented as a way to render fonts using vector graphics [Schneider90, Schneider90_2]. The basic principle is described by the following pseudocode:

Fitcurve(startPos, endPos, startVel, endVel) create a Bezier curve from startPos to endPos, using startVel and endVel to define the control points while the curve intersects an obstacle: reduce startVel and endVel recalculate the Bezier curve if startVel and endVel reach a user-defined minimum: break if the curve still intersects an obstacle: divide the path in two by choosing a point newPos along path set newVel using the vector between two neighbors of newPos Fitcurve(startPos, newPos, startVel, newVel) Fitcurve(newPos, endPos, newVel, endVel)

The whole process of calculating the piecewise linear path and fitting the curved path to it is fast enough to run in a single frame. The path can therefore be recalculated whenever a new obstacle renders the current path unfeasible. Note that the above recursive algorithm is guaranteed to terminate with a valid path because, in the limiting case, we are left with the same piecewise linear path that we began with.

Figure 2.2.3 shows an example. A piecewise linear path is created that avoids an obstacle. Then a single cubic Bezier curve is fitted to it, which takes into account the character‘s starting momentum. But this curve intersects with another obstacle, so a new path is created, using two cubic Bezier curves, that passes through another point on the piecewise linear path. Figure 2.2.3. Fitting a curve that accounts for character momentum. A single cubic Bezier curve A (light dashed line) is fitted to the piecewise linear curve (light filled line). But this passes through another obstacle, so a new path B (heavy dashed line) is constructed from two cubic Bezier curves that passes through an intermediate point on the piecewise linear path.

In order for a character to walk along the Bezier path, we need to choose the maximum and minimum velocities for the curve. Consider the distance d from the start point to the end point of a single Bezier curve. If the sum of the magnitudes of the start and end velocities exceeds d, the resulting curve can contain a loop. Usually we do not want this, so we can impose d/2 as a maximum. A value of zero works perfectly well for a minimum, but we can also use this as an opportunity to impose constraints based on the character‘s initial momentum, so that sharp turns are not allowed if the character is moving fast. Unfortunately, the character‘s physical velocity is not identical to the concept of velocity in the Bezier curve. This is because the latter is related to the curve‘s internal parametrization and therefore varies with curvature and curve length. Nonetheless, good results can be achieved by imposing a minimum kv/d, where v is the character‘s speed and k is a constant calculated empirically.

Animation Selection Thus we have created a smooth path to the end point that avoids any obstacles in the environment. All that remains is to select the appropriate animation for the character that will take it along this path. There is a wealth of literature on this complicated topic, using techniques such as motion graphs and inverse kinematics, which are beyond the scope of this article. Instead, I will present a simple solution using animation blending that gives a satisfactory degree of accuracy. To test the system, I created a character with a family of hand-designed animations for each different gait (for example, walking, running, sprinting). Within each family, all the animations had the same duration and path length but different amounts of rotation (for example, 0, 22.5, 45, 90 degrees). In this way, we can generate motion with any amount of rotation by blending two of the animations together. Linear blending of two or more animations simply means averaging the position and rotation of the skeleton joints. This is done separately for each frame of the animation. The resulting output is therefore a mixture of the inputs. This will only look natural when the animations are quite similar to begin with, as is the case here. Figure 2.2.4 shows an example.

Figure 2.2.4. A family of animations that can be blended together. Each has the same path length but a different amount of rotation (0, 22.5, 45, and 90 degrees).

We can chain the animations together because each one begins and ends at the same point in the character‘s stride. We can transition seamlessly between different gaits using transition animations, authored in the same way as a family of animations with different degrees of rotation. As one animation ends, a new one must be selected. It is possible at this moment that the character‘s position and orientation do not perfectly match the curved path. We could choose an animation that will return the character to the path, but if his orientation is wrong, this will result in a zigzagging motion as the character veers too far one way with one step and then too far the other way with the next. It is better to ensure the character‘s final orientation matches the tangent to the path, even if this means there is some error in his position. The method described gave sufficiently accurate results with a wide range of walking and running gaits; Figure 2.2.5 shows an example. This technique is limited, however, as Figure 2.2.6 shows. The animations we can create by blending in this way always describe a circular arc. All we can do is set the curvature of this arc. Some portions of a Bezier curve, however, cannot be described by a circular arc, and this is what causes the character to deviate from the path. For this reason, more sophisticated animation selection techniques would work better. Figure 2.2.5. There is not much error between the dotted line, made of circular arcs, and the full line, made of four cubic Bezier curves. Each dot represents the start of a new arc.

Figure 2.2.6. (a) A possible blended output: a circular arc with a rotation of 67.5 degrees. (b) An impossible output: No single circular arc can fit it.

Conclusion In conclusion, this gem has presented a simple and effective way to generate a curved path from any piecewise linear path that facilitates animation selection and promotes interaction between the AI and animation layers of a character. Future work should look at applying this to common physical constraints, such as enforcing a run-up before jumping or picking up an object. An important future extension to this work is to address how best to manage features of a three-dimensional terrain, such as stairs, gaps, and ledges.

References

[Axelrod08] Axelrod, Ramon. ―Navigation Graph Generation in Highly Dynamic Worlds.‖ AI Game Programming Wisdom 4. Boston: Charles River Media, 2008. [Dobkin97] Dobkin, David P., et al. ―Implementing a General-Purpose Edge Router.‖ Proceedings of Graph Drawing 1997: 262–271. [Graphviz] Graphviz ―Spline-o-matic.‖ n.d. Graphviz. n.d. . [Johnson06] Johnson, Geraint. ―Smoothing a Navigation Mesh Path.‖ AI Game Programming Wisdom 3. Boston: Charles River Media, 2006. [Latombe91] Latombe, Jean-Claude. Robot Motion Planning. Kluwer Academic Publishers, 1991. [Marden08] Marden, Paul. ―Dynamically Updating a Navigation Mesh via Efficient Polygon Subdivision.‖ AI Game Programming Wisdom 4. Boston: Charles River Media, 2008. [Paluszewski] Paluszewski, Martin. ―Robot Motion Planning (applet).‖ n.d. University of Copenhagen. n.d. . [Ratering95] Ratering, Steven and Maria Gini. ―Robot Navigation in a Known Environment with Unknown Moving Obstacles.‖ Autonomous Robots. 1.1 (June 1995): n.p. [Schneider90] Schneider, Philip J. ―An Algorithm for Automatically Fitting Digitized Curves.‖ Graphics Gems. Academic Press Professional, Inc., 1990. [Schneider90_2] Schneider, Philip J. ―A Bezier Curve-Based Root-Finder.‖ Graphics Gems. Academic Press Professional, Inc., 1990. [Snook00] Snook, Greg. ―Simplified 3D Movement and Pathfinding Using Navigation Meshes.‖ Game Programming Gems. Boston: Charles River Media, 2000. [Tozour02] Tozour, Paul. ―Building a Near-Optimal Navigation Mesh.‖ AI Game Programming Wisdom. Boston: Charles River Media, 2002. [Tozour03] Tozour, Paul. ―Search Space Representations.‖ AI Game Programming Wisdom 2. Boston: Charles River Media, 2003. [Tozour05] Tozour, Paul. ―Pathfinding Algorithms & Search Space Representations Demo.‖ 16 July 2005. .

2.3. Non-Iterative, Closed-Form, Inverse Kinematic Chain Solver (NCF IK) Philip Taylor [email protected] Inverse kinematics (IK) has many uses in games. A primary use is to control the limbs of characters—to fit the pose of the character to the terrain it is standing on or to pin a foot while walking to reduce foot sliding, as described in [Forsyth04]. For many characters, a simple two-bone solver is all that is required because, as in the case of human characters, there are only two major bones in a chain that need to be modified.

The problem of solving a two-bone chain is often reduced to a two-dimensional problem by constraining the solution to lie on a plane, usually defined by the root of the chain, the IK goal, and an up-vector position value. These two-bone solutions are considered ―closed form‖ because the solution can be found using trigonometry [Lander98]. When it comes to chains with more than two bones, there are several well-known algorithms used to solve this problem. Coordinate Cyclic Descent (CCD), Jacobian Transpose, or Pseudo-Inverse are commonly used. These algorithms suffer from performance issues because multiple iterations are required to converge on a solution. Without sufficient iterations, the chain may not reach the goal within acceptable limits, and the chain may exhibit irregular movements between frames, causing visual artifacts. Additionally, they lack precise control over the resulting shape of the chain. In this gem, I present a new method for solving inverse kinematics on chains comprising any number of bones by solving each bone separately in two dimensions. This solution is non-iterative in that only a single evaluation per bone is required, while guaranteeing a correct solution if the goal is within reach of the chain. The algorithm does not require extra parameters, such as up-vector position values or preferred angles. Instead, the algorithm attempts to preserve the initial shape of the chain while reaching for the goal, maintaining the integrity of any pose or animation data that was present on the chain prior to the IK solver‘s evaluation.

Context Consider the context of the character within a game. We are usually not building a set of rotations to define the chain pose; rather, we are modifying a set of orientations to more closely fit a world space constraint. During the evaluation of a game scene, the animation system is sampled to provide a local space transform per bone, and these local space pose transforms get concatenated together to build a global space pose. The pose at this point in the engine‘s evaluation is artist defined and part of what describes the style and personality of the character. Maintaining the integrity of the character‘s pose, or motion, while introducing new constraints, such as foot planting or lever pulling, is desirable as significant investment has been made in defining these motions. Any modifications made to the pose should be minimized to avoid breaking the original intent of the motion. In this gem, the pose defined by the animation system is referred to as the forward kinematic pose, or FK pose. Forward kinematic refers to the way that the pose was built, by accumulating local space transforms down through the hierarchy to generate global space transforms.

Bone Definition For the purpose of this gem, a bone is defined as containing a position and an orientation. The position is represented as a vector triple, and the orientation is represented as a quaternion. As illustrated in Figure 2.3.1, we refer to the vector that runs along its length as the bone length vector, and the length of a bone is defined as the length of this vector. By convention, bones are aligned with their local X-axis. A chain is defined as a linear hierarchy of bones, usually lined end to end. Figure 2.3.1. Bone definition.

IK Goal Position Without considering hands, feet, or any other child bones of the chain, a solution is defined as solving the chain such that the tip of the last bone touches, or comes as close to touching as possible, some predetermined IK goal position. The exact location of this IK goal position is defined by the engine and is not within the scope of this gem. The animation engine might be locking the foot to a plant position over the course of a character‘s step, or the hand of a character could be constrained to the steering wheel of a vehicle. In both cases, we have a bone chain with animation applied and a desired goal position, which is defined by the character‘s environment. One-Bone IK The simplest chain would only comprise one bone, and the closest valid solution would be to align the bone with the goal. While this example may appear trivial, the rest of this gem builds upon this concept. The vector between the bone and the goal is called the bone-to-goal vector, and we refer to the desired angle between the bone length vector and the bone-to-goal vector as the IK angle. When solving a chain comprising one bone, the best that can be achieved is to adjust the bone‘s pose such that the bone length vector is aimed at the IK goal, as shown in Figure 2.3.2. This is done by incrementing the rotation using the angle between the bone length vector and the bone-to-goal vector. The axis of rotation is perpendicular to both the bone length vector and the bone-to-goal vector. This rotation is the shortest arc of rotation that will transform the bone vector onto the bone-to-goal vector. Computing this quaternion directly is described in the article by [Melax06]. Figure 2.3.2. Aiming a bone at an IK goal position.

Two-Bone IK When an additional bone is added into the system, the mathematics becomes more complex. Aligning the Chain to the IK Goal Maintaining the shape of the limb while solving IK requires that each bone retain the respective transforms with bones prior to it and after it in the chain. For example, if the bones in the chain all lie on one plane prior to solving, then after solving the bones should all still lie on a single plane. The first step to solving a chain is to offset the orientation of each bone by the shortest arc between the root of the chain and the chain tip, and the root of the chain and the IK goal position, as shown in Figure 2.3.3. This step ensures that all further deformation will occur on the same plane with respect to the overall chain shape, as the original chain pose describes. This step also has the effect of biasing most of the deformation to the first joint in the chain. For most characters‘ limbs this is desirable, but joint limits can be imposed to restrict this deformation. See the ―Future Work‖ section. Figure 2.3.3. Aligning the chain to the IK goal.

We first apply the overall alignment to each bone and then solve using the appropriate method for that bone.

Calculating Bone 0 in a Two-Bone Chain The vector from a bone to the tip of the chain prior to solving is called the bone-to-chain-tip vector, and the angle between the bone length vector and the bone-to-chain tip is the FK angle, as shown in Figure 2.3.4. To calculate the IK angle for Bone 0, we use the law of cosines equation providing the parameters a, b, and c. Equation 1

Figure 2.3.4. Solving the first bone in a two-bone chain.

Once we have calculated the IK angle, we subtract this from the current angle between the bone length vector and the bone-to-goal vector to get a delta angle, which we will use to modify the bone‘s pose. The axis around which we rotate the bone is the cross product of the bone-to-goal vector and the bone-to-chain-tip vector. We then modify the bone‘s orientation by incrementing the rotation using the axis and the delta angle. Solving the Last Bone Before any child bone can be solved, its new global position must be calculated by transforming the bone‘s length vector using the new orientation of its parent and adding it to the parent‘s position. Bone 1, or the last bone in any chain, is solved using the one-bone solution described previously in the section ―One-Bone IK.‖ Three-Bone IK Solver

I have shown that a two-bone problem can be decomposed into two types of bones that are each solved using an appropriate method. The first thing to consider about a three-bone chain is that it encapsulates the two-bone chain described previously with one extra bone at the start of the chain, as shown in Figure 2.3.5. Therefore, once a solution can be found for Bone 0, then bones one and two can be derived using the previously described methods. Figure 2.3.5. Three-bone chain.

Maximum Bone Angles Consider that the initial pose of the chain defines an ideal pose for the limb, and any modification of the limb pose must be minimized. By analyzing the FK pose of the chain, for example, in Figure 2.3.6, we can determine a value that describes how bent Bone 0 is with respect to the rest of the chain. Figure 2.3.6. Calculating the bone angle and the maximum FK and IK bone angles.

We can use the law of cosines to calculate a value that defines the angle of Bone 0 if the remaining chain were to be laid out in a straight line. The remaining chain length, the distance to the FK chain tip, and the bone length are used to calculate the max bone angle. This angle is referred to as the max FK bone angle. Comparing this max FK bone angle to the actual FK bone angle gives us a value that defines our FK bone angle relative to the distance to the FK chain tip. Equation 2

The bone angle fraction is defined relative to the initial shape of the chain and is a correlation between the bone‘s orientation and the rest of the chain pose. Conversely, the remaining bone length can also be used to calculate the maximum possible angle that the bone can assume in IK. This maximum IK bone angle is multiplied by the bone angle fraction value to determine the new IK bone angle. Four-Bone IK Solver Figure 2.3.7 illustrates that with chains comprising three or more bones, the bones before the last two bones are all solved using the method described previously. Considering only the bone length, the remaining bone lengths, the distance to the FK chain tip, and the distance to the IK goal, a bone angle can be calculated. Once the IK bone angle has been calculated for Bone 0, then we simply continue down the chain, and Bone 0 can be solved in exactly the same way. Bone 2 can then be solved using trigonometry, and Bone 3 is simply aligned with the target. Figure 2.3.7. Applying the method to a four-bone chain.

N-Bone IK Solver An N-bone chain consists of three categories of problems. Bones 0 to N–3 are all solved using the method described previously of calculating maximum IK and FK bone angles to derive an IK bone angle. Bone N–2 can be solved using trigonometry, and Bone N–1 is simply aligned with the goal.

Function SolveIKChain( chain ) begin calculate chain target alignment for each bone in chain begin apply chain target alignment to bone if bone is last bone aim bone at target else if bone is second last use trigonometry to calculate bone angle else begin determine FK bone angle determine maximum FK bone angle determine maximum IK bone angle

IK bone angle = ( FK bone angle / maximum FK bone angle ) * maximum IK bone angle end end end

Handling Extreme Deformation In some cases, the remaining chain length is greater than the bone length plus the distance to the chain tip or IK goal. In these cases, the technique described previously of using trigonometry to calculate the maximum FK and IK angles cannot generate a maximum angle greater than π. Furthermore, during animation there may be visual artifacts if the max IK bone angles hit this limit. As shown in Figure 2.3.8, if the distance between the bone and the chain tip is greater than the remaining chain length, then trigonometry is used as described previously. Once the distance to the IK goal is less than the remaining chain length, the maximum IK and FK bone angle values can be calculated. Figure 2.3.8. Handling chains with extreme curvature.

The following pseudocode describes how to calculate the maximum FK bone angle for a long chain.

if( distToFkChainTip > remainingChainLength ) {

use trigonometry to calculate maximum bone angle } else { maxFkBoneAngle = acos( ( boneLength/2 ) / remainingChainLength ) maxFkBoneAngle += ( remainingChainLength distToFkChainTip ) / boneLength; }

Effectively, the remaining chain length is applied in an arc around the one position, allowing us to define maximum angles greater than π. This technique gives a much greater range of angles for the maximum FK/IK bone angle values, while maintaining the important limits when the chain is extended.

Additional Details We have only discussed the basic case in this gem. Here are some additional details that may be helpful for specific cases. IK/FK Blending In many cases, you will want to turn off IK because it may not be applicable. Simply disabling IK evaluation will cause a visual pop in the pose of your character. A better approach is to interpolate your IK pose back to your original FK pose before disabling IK. To do this, you need to keep an IK pose buffer separate from your FK pose buffer. Animated Joint Offsets So far in this gem, we have only considered chains with static local offsets along the local Xaxis. This solver can readily be extended to support animated joint offsets. The local position offset of a joint can be used as the bone length vector of the parent bone, as shown in Figure 2.3.9. The FK bone angle and max FK/IK bone angles are all calculated using this vector and its length, as described in this gem. This ensures that the bones pivot in the same position relative to their parent bone and that local position offsets are still applied in parent space. Because we are required to know the total length of the chain before solving, the entire chain‘s length needs to be measured prior to solving for IK. Figure 2.3.9. Animated joint offsets.

Extension Limit One concern with applying inverse kinematics to a chain in a game is that often the IK solvers will generate artifacts in the motion of the limb when the limb reaches the limit of its extension. The term hyperextension refers to the visual pop that happens when a chain reaches the limits of its reach. Fixing this issue is a minor addition to the IK solver algorithm. In the code samples provided with the book, extension dampening has been implemented to show how this can be achieved.

Future Work Our algorithm will not address all cases that you will come across. In the future, we plan to extend the system in several ways. Sliding Joints This IK solver will not currently calculate new local joint position offsets but only calculates orientation changes for each bone in the hierarchy. For example, it will not extend a hydraulic joint to reach the goal. Sliding joints could be implemented using a similar technique as that used to calculate angles. By defining the FK bone‘s local position relative to a slide vector, a new local offset position could be defined by comparing the distance to the FK chain tip and the actual distance to the IK goal. Joint Limits Aside from limiting the overall extension of the chain, it may be required that certain bones do not rotate beyond a certain limit or that they are not free to rotate on any plane. Because this chain is evaluated from top to bottom, it is possible to start imposing limits on angles. A simple way to do this is to limit the angle generated for each bone. In this way, we can limit the angle of a bone with respect to its parent. DOF Constraints Creature limbs are usually made up of a ball joint followed by a collection of hinge joints. In this gem, however, no twisting is ever applied to the chain, meaning that the solver only modifies the pose of the root bone on two axes, and every other joint rotates on only one axis. The axis used to modify the bone‘s orientation is assumed to be perpendicular to the bone vector and the bone-to-goal vector. Chains with limited degrees of freedom are not supported in the algorithm presented in this gem. If an axis of motion were defined for a particular bone, then solving would still be feasible by calculating the maximum bone angles projected onto the plane defined by this axis. General-Purpose Programming on Graphics Processors The algorithm presented in this gem does not employ any recursion, or major branching, and has a fixed cost per evaluation. These features make it an ideal candidate for implementations using some of the newer SIMT architectures, such as CUDA or OpenCL. Perhaps, as in the case of shaders, specific versions of the solver would be generated for categories of chains and the loops unrolled. A two-, three-, and four-bone version could be generated and the loops unrolled at compile time. Depending on the bone count, a chain would be solved in a batch with all other limbs of the same structure. This would avoid costly synchronization points, which would slow all chain evaluations to the cost of the longest chain.

Conclusion In this gem, I have presented a new method for solving inverse kinematics on chains of bones that is simple, fast, and accurate. This is achieved by calculating bone angles directly using values derived from the initial pose of the chain and the position of the IK goal. This method minimizes the modification of the original chain‘s pose, while ensuring the goal is always reached if within range. There are many possibilities for further development to expand this concept beyond a simple chain solver.

References [Forsyth04] Forsyth, Tom. ―How to Walk.‖ 2004. Game Tech. n.d. . [Lander98] Lander, Jeff. ―Oh My God, I Inverted Kine!‖ Game Developer. (September 1998): 9–14. [Melax06] Melax, Stan. ―The Shortest Arc Quaternion.‖ Game Programming Gems. Boston: Charles River Media, 2006. 214–218.

2.4. Particle Swarm Optimization for Game Programming Dario L. Sancho-Pradel [email protected] This gem presents the foundations of Particle Swarm Optimization (PSO), a simple yet powerful meta-heuristic optimization technique that can be applied to complex non-linear problems, even in the absence of a precise analytical formulation of the system. As a result, PSO is used in a variety of engineering applications, such as artificial neural network training, mechanical design, and telecommunications. I have also applied this technique to robotic systems. PSO is related to other population-based search strategies, such as Genetic Algorithms (GA), and can solve similar problems. However, PSO works natively with real numbers and tends to be more efficient than GA, often reaching a near-optimal solution in fewer function evaluations [Hassan05]. Next-generation consoles are pushing the boundaries of realism and complexity in game development. Often, games contain large sets of parameters that drive or influence the behavior of various systems, such as physics, animation, or AI. More often than not, the final values of many of these parameters are the result of a manual selection process based on experience and trial and error. Under certain basic conditions, PSO can be a valuable tool for better tuning these parameters in an automated fashion.

A Few Words on Optimization Optimization problems are typically modeled as finding the maximum or minimum value of a set of m-dimensional real functions, whose variables are normally subject to constraints. Maximizing a function f is equivalent to minimizing the function –f and vice versa. Using vector notation, the optimization problem is reduced to finding the roots (solutions) of the vector function: Equation 1

where, for the general case of p non-linear equations in m variables: F(X) = [f1 (X),..., fp (X)]T, X = x1,..., xm]T and 0 = [0,...,0]T In some cases, Equation (1) may be solved analytically, which means that the optimum point (or set of points) can be calculated exactly. In most real applications, however, this is not possible, and numerical methods are applied to find an approximate solution. Numerical Methods Numerical methods are algorithms that return a numerical value that in most cases represents an approximation to the exact solution of a problem. Classical optimization techniques have been successful at solving many optimization problems common in industry and science. However, there are optimization scenarios that classical approaches cannot solve in a reasonable amount of time. Combinatorial optimization problems—in other words, problems with a discrete set of solutions from which we want to find the optimal one—and NP-hard problems in general are good examples. For those cases, approximate solutions can be obtained relatively quickly using meta-heuristic approaches (that is, high-level strategies used for guiding different heuristics in search problems). No Free Lunch (NFL) Theorem for Optimization Optimizing can be regarded as a search problem where the selected optimization method represents the particular style or mechanism of executing the search. The NFL theorem [Wolpert97] states that for finite spaces and algorithms that do not resample points, the performance of all search (optimization) algorithms averaged over all possible objective functions is the same. In practice, however, we do observe that some algorithms perform, on average, significantly better than others over certain objective functions. This is because the objective functions considered in most real optimization scenarios have a structure that is far from random. As a result, algorithms that exploit such a structure will perform, on average, better than ―uninformed‖ search strategies (for example, a linear sequential search). The PSO search strategy is based on an exchange of information between members of a swarm of candidate solutions and operates under the assumption that ―good solutions‖ are close together in the search space. If this is not the case, PSO will not perform better than a random search.

The PSO Paradigm and Its Canonical Formulation PSO is a stochastic, population-based computer algorithm modeled on swarm intelligence. The PSO paradigm was originally developed by Dr. Eberhart and Dr. Kennedy [Kennedy95], an electrical engineer and a social psychologist, respectively, who had the idea of applying social behavior to continuous non-linear optimization problems. They reasoned that social behavior is so ubiquitous in the animal kingdom because it optimizes results. Social behavior is the result of an exchange of information within the members of the society. The original PSO algorithm evolved from a bird flocking simulator into an optimization tool motivated by the hypothesis that social sharing of information among members of the society offers an evolutionary advantage. In order to implement and exploit that flow of information, each member was provided with a small amount of memory and a mechanism to exchange some knowledge with its social network (in other words, its neighbors). Although different improvements and variations have been proposed since the

initial PSO algorithm was presented, most of them resemble closely the original formulation. A significant exception is the Quantum PSO (QPSO) algorithm [Sun04], which for reasons of space will not be covered here. Canonical Equations of Motion The way PSO works is by ―flying‖ a swarm of collision-free particles over the search space (also called the problem space). Each particle represents a candidate solution of the optimization problem, and each element (dimension) of the particle represents a parameter to be optimized. The movement of each particle in the swarm is dictated by the equations of motion that the particular flavor of the PSO algorithm defines. In its canonical form, these equations can be expressed as: Equation 2

where x(t) and V(t) represent respectively the particle‘s position and velocity at time t. By choosing Δt = 1, t becomes the iteration step. Note that x(t) and V(t) are m-dimensional vectors. At t = 0, the position of each particle is selected from a uniform random distribution, i.e. xj (0) = U(xj min, xj max), j = 1,2,...m, where xj min, xj max are the search space boundaries for the j-th dimension. V(0) can either be randomly initialized or set equal to the zero vector. Velocity Update As Equation (2) shows, the search process is driven by the velocity update, based on: Cognitive information. Experience gained during the particle‘s search, expressed as the best location ever visited by the particle xCognitive_Best. Socially exchanged information. The best location found so far by any member of the particle‘s social network (xSocial_Best). The calculation of xSocial_Best depends on the chosen network‘s topology. Figure 2.4.1 shows some common topologies. In a fully connected network (for example, Star), xSocial_Best represents the best location found by any particle in the swarm, hence called global best (gBest). If the swarm is not fully connected xSocial_Best becomes the best position found by any particle in its local neighborhood, hence called local best (IBest). Using IBest tends to provide more accurate results, whereas algorithms running gBest execute faster [Engelbrecht02]. The neighborhood is typically defined based on the particles‘ indices, although spatial information could also be used. Allowing neighborhood overlapping helps the information exchange across the swarm. Figure 2.4.1. Three classical social network structures: (a) Star topology (fully connected), (b) Ring topology, (c) Cluster topology.

Inertia. Provides certain continuity to the motion of the particle. Adding inertia to the velocity update helps to decrease the change in momentum between two consecutive iterations and provides a means of controlling the balance between the particle‘s exploration and exploitation. An exploratory strategy tends to direct the particle toward unexplored areas of the problem space. Exploitation refers to the movement of a particle around previously explored areas, resulting in a finer-grain search on these areas. Clearly, low inertia will favor exploitation, whereas higher values will result in a more exploratory behavior. In general, the PSO strategy is to use the cognitive and social best positions as attractors in the search space of the particle. It achieves that by defining the particle‘s velocity vectors as:

where ω∊(0,1). r1 and r2 are random m-dimensional vectors taken from a uniform random distribution, in other words, ri = {rij ∊ U(0,1)}, j = 1,2,...,m, i=1,2. The values of the two acceleration coefficients are normally chosen to be the same (typically cCognitive = cSocial ∊ (0,2.05]). It is important to notice that here the operator ⊗ denotes a per-component multiplication of two vectors, and therefore its result is another vector. Finally, the equations of motion for the basic canonical PSO system can be expressed as: Equation 3

Algorithm 1 details the basic canonical PSO formulation implementing the gBest strategy, and Figure 2.4.2 illustrates graphically the position and velocity update of one particle based on the equations of motion presented in Equation (3).

Algorithm 1: Canonical PSO (<<eq32.pdf>> version) 1: // Create a swarm of N randomly distributed particles 2: FOR EACH particle(P) i=1,...,N do 3: FOR EACH dimension j=1,..,M do 4: P[i].X[j] = Xmin[j]+rand(0,1)*(Xmax[j]Xmin[j]);

5: P[i].V[j] = rand(0,1)*Vmax[j]*sign(rand(0,1)0.5);// or =0; 6: END 7: END 8: g_Best = P[0]; 9: numIterations = 1; 10: // Iterative optimisation process 11: REPEAT 12: FOR EACH particle(P) i=1,...,N do 13: IF Eval(P[i].X) BETTER_THAN Eval(g_Best.X) 14: g_Best = P[i]; 15: END 16: IF Eval(P[i].X) BETTER_THAN Eval(P[i].x_best) 17: P[i].x_best = P[i].X; 18: END 19: END 20: // Apply_Equations_of_Motion 21: FOR EACH particle i=1,...,N do 22: FOR EACH dimension j=1,..,M do 23: V_Inertia = w*P[i].V[j]; 24: V_social = c1*rand(0,1)*(g_Best.X[j] P[i].X[j]); 25: V_cognitive = c2*rand(0,1)*(P[i].x_best[j] P[i].X[j]); 26: 27: P[i].V[j] = V_Inertia + V_social + V_cognitive; 28: Clamp(P[i].V[j]); // optional 29: P[i].X[j] = P[i].X[j] + P[i].V[j]; 30: Clamp(P[i].X[j]); // or any other strategy 31: END 32: END 33: numIterations++; 34: UNTIL (m_numIterations > MAX_NUM_ITER || GOOD_ENOUGH(Eval(g_Best)))

Figure 2.4.2. Illustration of the velocity and position updates of a particle during the optimization of a simple 2D parabolic function f(x,y). (Left) Geometrical illustration showing contour lines of f(x,y). (Right) Sketch of f(x,y) illustrating the particle’s relevant information and labels.

The position update may take the particle outside of the predefined boundary. In fact, it has been proven that in high-dimensional swarms, most of the particles will leave the search space after the first iteration [Helwig08]. Some strategies to constrain the particles to the search space include: Moving the particle to its closest boundary and setting its velocity to zero. Assuming a cyclic problem space. For instance, for ε < | x j max – x j min |, if x j (t) = x j max + ε, then it is recalculated as x j (t) = x j min + ε . Allowing the particle to leave the boundary, omitting its evaluation if the objective function is not defined at the new coordinates. Reinitializing all components whose values are outside of the search space. Evaluating the Particles: The Objective Function The objective function, also referred to as the fitness function or cost function, does not need to be the actual mathematical representation of the system to be optimized, but a measure of the quality of any solution found. Therefore, each call to Eval(P[i].X) in Algorithm 1 requires the simulation of the system using the parameters encoded in P[i]. During the simulation, various indicators related to the performance of the solution are recorded and combined in a function to provide a numerical measure of performance. This function is the objective function. Imagine that we want to optimize the parameters that define the way an NPC performs a certain task, such as handling a car. Most likely, we do not have a dynamic model of the car and its controllers, so we cannot directly optimize the non-linear system of equations that defines the problem at hand. Nevertheless, by defining an objective function that penalizes, for instance, the time the NPC spends outside the training path and the time required to drive a predetermined distance, we are effectively giving the particles valid feedback to evaluate their quality, and therefore a means of improving in later iterations.

Add-Ons to the Classical Formulation Beyond the basic formulation, we can add a few improvements to deal with specific cases that may come up in a particular use case. Velocity Clamping

The formulation described so far may generate particles with increasingly large oscillations around a potential optimum due to an ―explosion‖ in the velocity values. This issue was addressed in [Kennedy01] by the introduction of a positive velocity clamping parameter Vmax, which modifies the velocity update as follows:

where V j culated (t + 1) is the j-th component of the velocity vector calculated using Equation (3). The selection of Vmax is problem dependent. Typically, each of its components is initialized within the size of the search space boundaries, in other words, V j max = κ · | x j j max – x min |, κ ∊ (0,1). Initially, we want κ to be close to 1 in order to favor exploration. During the optimization process, Vmax can be periodically adjusted by: Setting Vmax to the dynamic range of the particle. Using linearly or exponentially decaying Vmax in order to progressively move toward exploitation (fine-grained searches) of the area around the best particle(s). Increasing or decreasing Vmax after α iterations without improvement in the best found solution. Variable Inertia Weight ω(t) The inertia coefficient has a clear influence in the particle‘s search strategy. Large values of ω favor exploring new areas, while small values favor the exploitation of known areas. Initially, we would like the algorithm to favor exploration, and as we approach the vicinity of the optimum, we could decrease the value of ω in order to perform finer-grain searches. There are many ways to define a decreasing ω(t) . A common one is a linear interpolation between the desired values of ωInitial and ωFinal :

where t is the current iteration and T represents the number of iterations required to reach ωFinal, keeping ω(t) = ω,Final for t > T. Another strategy is to select T = MAX_NUM_ITER/N, resetting ω(t) and applying again the interpolation every N iterations. Constriction Coefficient Based on the dynamic analysis of a simplified and deterministic PSO model, Kennedy and Clerc [Clerc02] reformulated the velocity update in terms of the constriction coefficient χ. They also analyzed the convergence properties of the algorithm, obtaining a set of theoretical optimal values for the PSO parameters to control both convergence and the velocity explosion. A simple constriction model for velocity update can be expressed as: V (t + 1) = χ · [V(t) + φ1 · (X 1) = X(t) + V(t + 1)

Cognitive_Best

– X (t)) + φ

2

· (X

where X is a diagonal matrix, whose elements are calculated as:

Social_Best

– X(t))] X (t +

and: φ = φ1 + φ2, φ > 4, x

jj

∊ (0,1), κ ∊ (0,1)

Low values of κ encourage exploitation, whereas high values result in a more exploratory system. Typical values are χjj = 0.7298 and θ1 = θ2 = 2.05. Notice that this model can be easily converted into the original inertia-based formulation. Extended Full Model for Velocity Update The velocity update could include both the global best and the local best particle: V (t + 1) = ω · V(t) + Θ (IBest; – X(t))

1

⊗ (X

Cognitive._Best

– X(t)) + Θ

2

⊗(g

Best

– X(t)) + Θ

3

⊗

where typically Θ i ∊ U(0,1). Adding Constraints In an optimization scenario, the constraints define feasible subspaces within the search space of the unconstrained problem. In the original PSO algorithm, each dimension is bounded individually. (See Line 4 of Algorithm 1.) This translates into an axes-aligned hyper-rectangular search space. Therefore, constraints such as g (x) ≤ b cannot be generally defined. As a result, neither a simple circular search space (x2 + y2 ≤ radius 2), nor a combination of constraints (for example, a rectangular search space with a rectangular hole inside), is supported. Different approaches have been proposed to extend the constraint-handling capabilities of evolutionary algorithms. (For a review, see [Michalewicz96] and [Mezura09].) A simple approach presented in [Hu02] uses the canonical PSO algorithm and introduces two modifications: All particles are initialized inside the feasible region. After evaluating the population, only particles within the feasible region can update the values of xCognitive_Best and xSocial_Best. Integer Optimization Although PSO deals natively with optimization problems of continuous variables, it can also be modified to cope with discrete and mixed-parameter problems. The simplest way to do that is to use the same equations of motion, but round the results of the integer components (at least for the position update) to the nearest integer. In some cases this can result in invalid particles (for example, duplicated elements in a particle are generally not allowed in permutation problems), and particles need to be ―fixed‖ after every position update. The fixing mechanism is problem dependent, but it may consist of making all the elements of a particle different, while keeping their values within a certain range. Maintaining Diversity A common problem in numerical optimization is premature convergence—in other words, getting trapped around a local optimum. In stochastic methods, the diversity of the population has a major effect on the convergence of the problem; hence, diversity loss is commonly regarded as a threat to the optimization process. Some ways of favoring diversity are:

Alternating periodically PSO with other search methods. Every α iterations, reinitializing (randomizing) the location of a random number of particles in the swarm. Every β stalls, reinitializing the system. Including mutation and crossover operators and applying them to the position of a small subset of particles in the swarm. Running multiple independent swarms, allowing an occasional exchange of information (for example, gBest) and/or particles between swarms. Furthermore, each swarm could use different search strategies (for example, some could use IBest, some others could use gBest, some could use parameters that favor exploration, some others could use more exploitative ones, and so on). It is important to notice that the selected networking strategy also has an impact on the diversity of the population. Distributed topologies, such as Ring or Cluster, generate a number of local attractors and tend to maintain higher levels of diversity, leading to good solutions more often than other centralized approaches.

A Note on Randomness PSO is a stochastic optimization technique that initializes its particles by placing them randomly in the search space. In order to do so, pseudo-random values are drawn from a uniform distribution. The reason is that in the absence of any prior knowledge about the location of the optimum, it is best to distribute the particles randomly, covering as much of the problem space as possible. However, as Figure 2.4.3(a) shows, the result may be less uniform than desired. The spatial uniformity of the randomized particles can be improved by using Halton point sets, which are uniformly distributed and stochastic-looking sampling patterns, generated by a deterministic formula at low computational cost [Wong97]. Figure 2.4.3. One instance of two sets of 25 particles initialized in a twodimensional space using (a) a uniform random distribution and (b) a Halton sequence.

Case Study 1: Typical Benchmarking Functions

There are sets of functions that are commonly used for benchmarking optimization algorithms. These functions tend to have areas with a very low gradient (for example, Rosenbrock‘s saddle) and/or numerous local optima (for example, Rastrigin‘s function). Figure 2.4.4 illustrates the 2D versions of three of them. The accompanying CD includes the multidimensional version of a larger set of benchmarking functions together with a PSO algorithm implementation. You are encouraged to experiment with the different PSO parameters, observing the variation in convergence speed and accuracy. More information about different benchmarking functions can be found in [Onwubolu04]. Figure 2.4.4. 2D versions of some common benchmarking functions.

Case Study 2: Optimization of Physical Parameters for In-Game Vehicle Simulation To obtain realistic vehicle responses in a game, simplified physical models of their key components are generated. These models contain a number of physical parameters that define the way the vehicle reacts in many situations. Those parameters are often obtained based on experience and a trial-and-error process of selection. Such a manual process tends to be inevitably slow, and as a result, little of the search space is often covered. It is clear that there is room for improvement, and PSO is a good candidate for automating the search process. The only prerequisite is to be able to estimate numerically the quality of each candidate solution. The following example illustrates the general principle. We want to optimize the performance of a car, whose simplified model is shown in Figure 2.4.5. The model includes friction between the tires and the ground, the suspension, and the car‘s mass distribution. Based on the model‘s parameters, the particle‘s components can be defined as:

P.X[0] = P.X[2] = (Rear); P.X[4] = (Front); P.X[6] = Mass 2;

Forward Friction Coeff.; P.X[1] = Side Friction Coeff.; Spring Constant (Rear); P.X[3] = Damper Constant Spring Constant (Front); P.X[5] = Damper Constant Mass 1; P.X[7] = Mass 1 location (along Z); P.X[8] =

Figure 2.4.5. A simplified physics model of a car.

As an example of dimensionality reduction, the value and location of the mass M3 and the location of the mass M2 have been fixed. They can be included at any moment in the optimization by extending the dimensionality of the particle. Most importantly, we need to define an objective function capable of quantifying the response of the vehicle while following a path along a network of roads. The objective function to be minimized could include the following variables: distance to goal on a crash, total distance driven without being on the marked spline, accumulated distance from the car‘s center of mass to the marked spline (shown as d in Figure 2.4.6 (b)), accumulated heading deviation with respect to the selected path, oscillations around the Z-axis, and tilt around the Z-axis on curves (c.f. Figure 2.4.6 (a)). The function can also include big penalties for crashing and in cases where only two wheels are in contact with the road. The final objective function could be a quadratic combination of each of these variables ri.

Figure 2.4.6. Illustration of some of the parameters included in the objective function. (a) Tilt around the Z-axis during a curve. (b) Distance to the path’s splines and heading error.

The normalization constant ηi ensures that all the variables have the same weight (typically between 0 and 1), regardless of their particular ranges. The relative importance of ri, with respect to the rest of the variables, is represented by the coefficient ki, which can also be used for scaling the objective function. Squaring ri is useful in cases where only the magnitude of ri is relevant, ensuring that the objective function does not decrease based on the sign of the variables. As Algorithm 1 showed, each iteration requires the evaluation of the whole population (in other words, N particles). Therefore, it would be convenient to evaluate N vehicles simultaneously following identical paths.

Case Study 3: Physics-Based Animation of Mechanical Systems Physics-based animation can improve the gaming experience by adding physical realism and believability to the game. Cloth and hair procedural animation, ragdolls, and general collisions between rigid bodies are some examples featured in many titles. Extending this concept to complex mechanical systems, such as robotic NPCs and vehicles, can result in more realistic behaviors that are in accordance with our experience of how mechanical systems move. Figure 2.4.7 shows a hexapod NPC robot exhibiting a tripod gait, which is a gait pattern often used by insects and other animals due to its performance and stability. Gait patterns can be defined by a series of parameters (for example, maximum joint angles, feet trajectory, stance phase offset, stance duty factor, step duration) whose values will depend on the physical characteristics of the robot, the environment, the pattern configuration, the task (for example, turn, go forward, climb, and so on), and the task qualifiers (for example, fast, safe, and so on). PSO can be used for finding good values for these parameters. Each gait could be optimized individually by using specialized objective functions and running single-gait simulations. Next, gait transitions could be worked out. Once the optimization is finished, the result would be a hexapod robotic NPC with newly acquired locomotion skills. Figure 2.4.7. Snapshot of a hexapod NPC robot using a tripod gait. The legs shown in white are lifted, while the darkened ones are in stance.

The design of the robot can be taken one step further. The configuration of the legs could be optimized by adding the length of their segments and the number of joints into the optimization‘s parameters list. Ultimately, PSO could be used as a tool for evolving different types of robots by including the number of legs and their position in body coordinates as optimization parameters. Once the robot design is finished and the gait parameters optimized, the gait controller of the robot would send the appropriate sequence of actions to the different joints, and the physics engine would do the rest. Incidentally, the controller of each joint (typically some

flavor of proportional-integral-derivative controller, or PID) could also be optimized using PSO. Let‘s see one final example. We want to find the parameters that generate the fastest walking gait for the particular robotic NPC illustrated in Figure 2.4.8.Figure 2.4.8 (b) shows the feet trajectory selected for this gait. This half-ellipse trajectory is defined by two parameters, namely the step height (h) and the step size (s). Another important parameter is the duration of the step (Ts). To improve stability, the torso of the robot is initially tilted sideways to shift its center of mass toward the supporting leg (refer to Figure 2.4.8 (a)). This can be parameterized by a maximum tilt angle (α) and the percentage of step time that the body is tilted toward the leg in stance (Tt) before shifting the weight to the other side. Assuming a 50-percent duty cycle for each leg, the structure of the particles could be P.X[]= {h,s,Ts,α,Tt}. Figure 2.4.8. Illustration of some basic parameters in a humanoid-like robot. (a) Weight-shifting angle for improving stability, (b) half-ellipse gate parameters, (c) the different joint angles in a crouching action.

The objective function could be designed as in Case Study 2, taking into account the walking time, the distance to the goal before falling, the accumulative heading error, the accumulative position error with respect to the given path, and the stability of the gait measure in terms of oscillations around various axes. It is worth noticing that the gait parameters obtained are subject to external factors, such as the friction between the robot‘s feet and the ground. In principle, it could be possible to find a continuous mapping between the walking parameters and different types of terrains by feeding the friction data and the PSO solutions to a neural network or a fuzzy inference system. In that way, the robotic NPC could try to adapt its gait to optimize the walking performance when the type of terrain changes. Gaits are not the only movements that could be optimized. Learning other actions, such as crouching (refer to Figure 2.4.8 (c)) or climbing stairs would follow the same principles.

Conclusion The aim of meta-heuristic optimization algorithms is to provide approximated solutions to complex non-linear optimization problems that more traditional approaches have difficulties

addressing. These algorithms are not guaranteed to converge at a good solution, but they are designed to find good approximations to the global optimum with high probability. Population-based algorithms are sensitive to the diversity of their individuals and to their configuration parameters. PSO relies on a small set of intuitive parameters, such as the number of particles, the maximum number of iterations, and the topology of the social network. Often, the mapping between an optimization problem and the function that quantifies the quality of a solution is not unique. In these cases, the quality of the selected objective function has a significant impact on the convergence of the algorithm. This gem has shown a number of variations proposed around the canonical PSO algorithm. The simplicity of the PSO paradigm makes its extension a relatively simple task. Games are becoming extremely complex and highly parameterized software products. In many cases, the response of different systems is driven, or at least influenced, by sets of parameters stored in configuration files. PSO could be a useful tool for optimizing some of these parameters in an automated fashion. AI, physics, and animation are examples of systems that could benefit from this optimization technique.

References [Clerc02] Clerc, M. and J. Kennedy. ―The Particle Swarm-Explosion, Stability and Convergence in a Multidimensional Complex Space.‖ IEEE Transactions on Evolutionary Computation 6 (2002): 58–73. [Engelbrecht02] Engelbrecht, A.P. Computational Intelligence: An Introduction. Wiley, 2002. [Hassan05] Hassan, R., B. Cohanim, and O. de Weck. ―A Comparison of Particle Swarm Optimization and the Genetic Algorithm.‖ Proceedings of 46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference. 2005. [Helwig08] Helwig, S. and R. Wanka. ―Theoretical Analysis of Initial Particle Swarm Behavior.‖ Proceedings of the 10th International Conference on Parallel Problem Solving from Nature (Sept 2008): 889–898. [Hu02] Hu, X. and R. Eberhart. ―Solving Constrained Nonlinear Optimization Problems with Particle Swarm Optimization.‖ 6th World Multiconference on Systemics, Cybernetics and Informatics (SCI 2002): 203–206. [Kennedy95] Kennedy, J. and R. Eberhart. ―Particle Swarm Optimization.‖ Proceeding of IEEE International Conference on Neural Networks 4 (Dec. 1995): 1942–1948. [Kennedy01] Kennedy, J., R.C. Eberhart, and Y. Shi. Swarm Intelligence. Morgan Kaufmann Publishers, 2001. [Mezura09] Mezura-Montes, E. Constraint-Handling in Evolutionary Optimization. Springer, 2009. [Michalewicz96] Michalewicz, Z. and M. Schoenauer. ―Evolutionary Algorithms for Constrained Parameter Optimization Problems.‖ Evolutionary Computation 4 (1996): 1–32. [Onwubolu04] Onwubolu, G. and B. Babu. New Optimization Techniques in Engineering. Springer, 2004. [Sun04] Sun, J., B. Feng, and Wenbo Xu. ―Particle Swarm Optimization with Particles Having Quantum Behavior.‖ Proc. Cong. Evolutionary Computation 1 (June 2004): 325–331.

[Wolpert97] Wolpert, D.H. and W.G. Macready. ―No Free Lunch Theorems for Search.‖ IEEE Transactions on Evolutionary Computation 1.1 (April 1997): 67–82. [Wong97] Wong, Tien-Tsin, Wai-Shing Luk, and Pheng-Ann Heng. ―Sampling with Hammersley and Halton Points.‖ Journal of Graphics Tools 2.2 (1997): 9–24.

2.5. Improved Numerical Integration with Analytical Techniques Eric Brown [email protected] There is a fairly standard recipe for integrating the equation of motion in the context of a game physics engine. Usually the integration technique is based on the Symplectic Euler stepping equations. These equations are fed an acceleration, which is accumulated over the current time step. Such integration methods are useful when the exact nature of the forces acting on an object is unknown. In a video game, the forces that are acting on an object at any given moment are not known beforehand. Therefore, such a numerical technique is very appropriate. However, though we may not know beforehand the exact nature of the forces that act on an object, we usually know the exact nature of forces that are currently acting on an object. If this were not so, we would not be able to provide the stepping equations with the current acceleration of the body. If it were possible to leverage our knowledge of these current forces, then we might expect to decrease the error of the integration dramatically. This gem proposes such a method. This method allows for the separation of numerical integration from analytic integration. The numerical integration steps the state of the body forward in time, based on the previous state. The analytic integration takes into account the effect of acceleration acting over the course of the current time step. This gem describes in detail the differences and implications of the integration techniques to aid the physics developer in understanding design choices for position, velocity, and acceleration updates in physics simulation. As we build up to the introduction of this method, we will first discuss a heuristic model for classifying errors of integration techniques.

Classifying Errors Often, the method for classifying errors in integration techniques is to label them as first order, second order, and so on. Methods that are first order have an upper bound error on the order of the time step taken to the first power. Methods that are second order have an upper bound error on the order of the time step to the second power. Taking a small number to a large power makes the small number smaller. Thus, higher-order methods yield more accuracy. Error can also be classified in terms of how well an integrator conserves energy. Integrators might add or remove energy from the system. Some integrators can conserve energy on average. For instance, the semi-implicit, or Symplectic Euler, method is a first-order method, but it conserves energy on average. If an integrator adds energy to the system, the system can become unstable and diverge, especially at higher time steps. The accuracy of a method can affect its stability, but it does not determine it, as shown by the Symplectic Euler method. More often than not, it is the stability of a method that we desire, more than the accuracy.

In this gem we will be taking a different approach to classifying error. This approach is based on the fact that the stepping equations usually assume that the acceleration is constant over the course of a time step. The kinematics of constant acceleration are a problem that can be solved easily and exactly. Comparing the kinematic equations of constant acceleration with the results of a numerical method provides qualitative insight into sources of error in the method. When derivatives are discretized, it is done by means of a finite difference. Such a finite difference of positions implies that the velocity is constant over the course of the time step. In order to introduce a non-constant velocity, you must explicitly introduce an equation involving acceleration. Similarly, the only way to deal with non-constant acceleration is to explicitly introduce an equation involving the derivative of acceleration. Since many numerical methods do not involve any such equation, we are safe in making the comparison with the kinematic equations for constant acceleration, at least over the course of a single time step.

Kinematics of Constant Acceleration We know the exact form of the trajectory of particles that are subject to a constant acceleration.

We can compare this set of equations to the results of common numerical methods in order to gain a qualitative idea about the error in the method. Consider the standard Euler method: vn+1 = vn + an Δ t xn+1 = xn + vn Δ t This set of equations can be transformed into a set that more closely resembles the kinematic equations by inserting the velocity equation into the position equation: vn+1 = vn + an Δ t xn+1 = xn + vn-1 Δ t + an-1 Δ t2 The appearance of vn-1 in the position equation is due to the fact that we must insert vn and must therefore re-index the velocity equation. The differences in form of this equation to the kinematic equations can be considered as qualitatively representative of the error of the method. We may perform this same procedure with the Symplectic Euler method: vn+1 = vn + an Δ t xn+1 = xn + vn+1 Δ t Insert the velocity equation into the position equation to transform to the kinematic form:

vn+1 = vn + an Δ t xn+1 = xn + vn Δ t + an Δ t2 This equation is a much closer match in form to the kinematic equations. We can use this resemblance to justify the fact that the Symplectic Euler method must in some way be better than the standard Euler method.

The Kinematic Integrator If we are trying to find an integration method that when converted to a kinematic form is identical to the kinematic equations, why not just use the kinematic equations as the integration method?

If we do this, then we are guaranteed to get trajectories that are exact, within the assumption that acceleration is constant over the course of the time step. However, the accelerations we are usually interested in modeling are not constant over the course of the time step. We must use a value for the acceleration that encapsulates the fact that the acceleration is changing. We could use the acceleration averaged over the time step, , as the constant acceleration value. By inserting the average acceleration into the kinematic equations, we achieve a method that we will refer to as the average acceleration method. In order to calculate this average exactly, we must analytically integrate the acceleration over the time step, which in many instances can be done easily. The average acceleration method therefore represents a blend between numerical and analytic integration. We are numerically integrating the current position and velocity from the previous position and velocity, but we are analytically integrating accelerations that are acting during the current time step. Of course, calculating the average acceleration exactly requires that we know how to integrate the particular force in question. Luckily, most forces that are applied in game physics are analytic models that are easily integrated. Calculating the average acceleration from an analytic model of a force is usually just as easy as calculating the acceleration at an instant of time. If the average acceleration is calculated analytically, then the velocity portion of the kinematic equations produces exact results. However, the position portion would require a double integral in order to achieve an exact result. If the forces that we are dealing with follow simple analytic models, then calculating a double integral is usually just as easy as calculating a single integral. We will generalize the idea of the average acceleration method in order to introduce the kinematic integrator. The kinematic integrator is a set of stepping equations that allow for exact analytic calculation of both the velocity integral and the position integral. vn+1 = vn + dv

xn+1 = xn + vn Δ t + dx The exact method uses the following definitions for dv and dx:

If we are using the average acceleration method, then we define dv and dx as:

In the case of constant acceleration, it is very easy to perform both the single and the double integral. The integral contributions of a constant acceleration are given as:

If there are multiple forces acting on a body, we can express the integral contributions as a sum of contributions that are due to each force. dv = dv1 + dv2 + … dx = dx1 + dx2 + … Thus for the kinematic integrator we accumulate dv‘s and dx‘s rather than accelerations. All forces acting on a body provide contributions to dv and dx. The amount that is contributed is dependent on the nature of the force and can usually be calculated exactly. If all forces acting on a body are integrable, then every contribution is exact. The kinematic integrator can be used to perform the Symplectic Euler method with the following integral contributions: dv = an Δ t dx = an Δ t2 This method is useful if the acceleration is not integrable (or if we are too lazy to calculate the integrals). These contributions are not going to be exact, but they will at least conserve energy, which will maintain stability.

The kinematic integrator does not represent a specific integration method, but rather the ability to isolate the portions of the stepping equations that actually require integration. Since the method used to evaluate the integral contributions is not explicitly specified, we have a degree of freedom in choosing which method might be best for a particular force. For instance, if there is a contribution from a constant gravitational force, then we can easily use the exact method. We will see that for some forces we will want to use the average acceleration method. Or we could use the Symplectic Euler method if the force in question is too complicated to integrate or if we are performing a first pass on the implementation of a particular force.

Integral Contributions Due to a Spring Force Probably one of the most common forces, next to a constant uniform force, is a spring force. The spring force is proportional to the displacement of the spring from equilibrium. This results in the following equation of motion:

The solution of this equation of motion is analytically solvable and is given by: x – l = A cos (ωt) + B sin(ωt) v = –A sin(ωt) + ωB cos(ωt) Using this exact trajectory, we can determine the integral contributions:

where c and s represent the cosine and sine, respectively, of ω(t). We could represent this calculation as a matrix operation.

The components of this matrix depend on the size of the time step Δt, the mass of the body m, the strength of the spring k, and the equilibrium position of the spring l. The components of the matrix can be cached and reused until any of these parameters change. In many instances, these parameters do not change for the lifetime of the spring. Thus, the calculation of the integral contributions of a spring is relatively trivial—in other words, six multiplies and four adds. We can consider that the spring that we have been discussing is anchored to an infinitely rigid object at the origin, since we have only taken into account the action of the spring on a single body, and the spring coordinates are the same as the body coordinates. It is only

slightly more complicated to calculate the integral contributions due to a spring that connects two movable bodies.

Multiple Forces Before discussing the integral contributions of other possible forces, we need to discuss what happens when multiple forces act on a body. If all of the forces acting on a body depend only on time, then the result of accumulating exact integral contributions will be exact. But consider the case where at least one of the forces depends on the position of the body, such as the spring force. The integral contribution of the spring takes into account the position of the body at intermediate values as the spring acts over the course of the time interval. However, the calculation is not aware of intermediate changes in position that are due to other forces. The result is a very slight numerical error in the resulting trajectory of the particle. As an example, consider two springs that are acting on the same body. For simplicity, both springs are attached to an infinitely rigid body at the origin, and the rest length of both springs is zero. The springs have spring constants k1 and k2. The acceleration becomes: a = –(ω1)2 x – (ω2)2x If we are to handle these forces separately, then we would exactly calculate the integral contributions of two springs, with frequencies ω1 and ω2, and accumulate the results. The springs can also be combined into a single force.

In this case, only a single integral contribution would be calculated. It might be surprising to discover that these two methods produce different results. The second method is exact, while the first contains a slight amount of numerical error. This seems to imply that: ∫(a1 + a2)dt ≠ ∫ a1dt + ∫ a2dt which is usually not true. However, in the current circumstance, if the acceleration is a function of the position, which in turn is a function of the acceleration, then the integrals have feedback. The effect of this feedback is that we cannot separate the sum into independent integrals, since the independent integrals will not receive feedback from each other. If the acceleration only depends on time, there is no feedback, and the integrals can safely be separated. Because of this, you may want to separate out forces that depend only on time and accumulate their integral contributions as a group. The integral contributions of these forces can be safely added together without introducing numerical error. Unfortunately, most of the forces that are applicable to game physics depend on the position of the body. Though the error due to multiple forces is incredibly small, it represents a tiny violation of energy conservation. If this violation persists for long enough, then the trajectory can

eventually diverge. The error due to multiple forces is initially much less than the error in the Symplectic Euler method, but it can eventually grow until it is out of control. Since we can integrate exactly the system with two springs, we can use this example to gauge the error present in the different methods for calculating the integral contributions. Results of this error calculation are contained in the figures. The first figure contains the results of integrating one spring; the second figure represents errors due to integrating two springs. The integration takes place over one half of the period of the two-spring system. The Symplectic Euler method as well as the pulse method (which will be introduced later) both conserve energy on average. Neither the average acceleration method nor the exact method conserve energy over long intervals of time. It takes quite a while, but the exact method will eventually diverge. The average acceleration method will eventually dampen out. For this reason, the average acceleration method is preferred if the integration is going to take place over a long interval. Figure 2.5.1. The relative errors in the case of one spring and two springs.

Integral Contributions of a Pulse One solution to the problem due to multiple forces is to approximate the action of the force as a pulse. A pulse is a force that acts very briefly over the course of the time step. A perfect pulse acts instantaneously. Pulses acting at the beginning of the time step do not depend on intermediate values of position or velocity; therefore, pulses are immune to the multiple-force problems. Consider a rectangular pulse. The area under the curve of a rectangular pulse is dv = adt, where dt is the width of the pulse, and a is the magnitude of the pulse. It is possible to shrink the width of the pulse while maintaining the same area. To do this, we must, of course, increase the magnitude of the pulse. If we allow the pulse width to go to zero, the height diverges to infinity. However, the product of the width and the height is still equal to dv. In the case where dv = 1, there is a special name for this infinite pulse. It is called a delta function, and is denoted as δ(t). The delta function is zero at all values of t, except at t = 0. At t = 0, the delta function is equal to infinity. Integrating the delta function is very easy to do.

for all values of a and b where a < 0 < b. If the interval between a and b does not contain zero, then the result of the integral is zero. To represent the force as a pulse, we define the acceleration as: a = dvδ(t) Here, dv is the area under the curve of this instantaneous pulse, as previously described. Integrating this acceleration is easy to do: ∫ adt = dv Integrating the position contribution is now just integrating a constant.

We can choose dv as the amount of acceleration we want to pack into the pulse. If we want to pack all of the acceleration due to an arbitrary force into the pulse, then we use the average value of the acceleration . This produces stepping equations of the form: In a more familiar form, the stepping equations for the pulse would appear as:

In a more familiar form, the stepping equations for the pulse would appear as:

If you make the assumption that the acceleration is constant over the time interval, then you can replace with an. Doing this, you will arrive back at the stepping equations for the standard Symplectic Euler method. This gives meaningful insight into the Symplectic Euler method. It is a method that assumes that the acceleration is constant over the course of the time step and delivers all of the acceleration in an instantaneous pulse at the beginning of the time step. Since pulses can be accumulated without introducing error, the Symplectic Euler method can be considered to be an exact solution to forces of an approximated form. Since the method is exact, it intrinsically conserves energy. The error in the method is entirely due to the differences between the actual form of the force and the approximate form of the pulse.

Integral Contributions of Collision Forces One very common thing that needs to be done in a game physics engine is to resolve collisions. To calculate our collision, we will assume that the forces due to the collision act

within some subinterval that fits entirely within our given time interval. The nature of the force does not matter as much as the fact that it operates on a very small timescale and achieves the desired result. For simplicity, we will assume that the object has normal incidence with a plane boundary. We will also assume that the collision with the boundary is going to take place within the time interval. To handle the collision correctly, we should sweep the trajectory of the body between xn and xn+1 and find the exact position and the exact time where the collision begins to take place. If we want to find the projected position at the next time step xn+1, we should wait to apply the collision force until all other forces have been applied, so that the projected next position is as accurate as possible. For the purpose of simplification, we will consider that the trajectory is a straight line that connects the position xn with xn+1. The position of the body at some intermediate value of the time step is given by: Xf = Xn(1–f)+Xn+1(f) where f is the fraction of the time step that represents the moment of impact. The intermediate velocity is simplified as well, to be: Vf = Vn(1–f)+Vn+1(f) With this physically inaccurate yet simplified trajectory, we can perform a sweep of the body against the collision boundary. The result of this sweep will determine the time fraction f. This will tell us the position and velocity (almost) of the body at the moment of impact. To determine the force that is applied in order to resolve the collision, we need to apply a pulse force that reflects the component of velocity vf about the contact plane. This reflection incorporates the interaction of the body with the contact surface; thus, the result should incorporate surface friction. For now, we will assume that there is no friction. To apply the pulse force, we will begin by applying a constant force over a small subinterval of the time step. The subinterval begins at time fΔt and ends at (f + df)Δt. The velocity of the body at the end of the subinterval is given by: vf + df = vf + avdf According to our initial simplification, the velocity is aligned with the normal of the contact plane. Thus, all of the velocity is reflected. We choose av to enact this reflection.

The integral contributions of this force in the limit that df → 0 are given by: dvv – 2vf dxv = 0 Since this force pulse does not contribute anything to the position integral, the application of this force does not change the value of xn+1 from what it would have been. Thus, the value of xn+1 will still violate the collision constraint.

We need to apply a second force pulse, which moves the position of the body out of the collision surface. Assuming that the body has normal incidence to the plane, then the amount the body needs to be pushed is given by Δx = xf – xn+1. Again, we are going to apply a constant force on a subinterval. The acceleration of this force ax is determined by considering the kinematic equations for constant acceleration.

The integral contributions for this acceleration as df → 0 are:

Adding these two sets of contributions together gives the final result for the collision force. This set of contributions can be expressed in terms of the position at the beginning and end of the interval, as well as the parameter f, which represents the moment when the collision first begins to happen.

Integral Contributions of Viscous Forces A viscous force is a force that is related to, and usually opposing, the velocity of an object. F = –KV The equation can be solved for the velocity in terms of the constant k and the mass m. This is done by expressing the acceleration as the derivative of velocity.

This equation can be integrated to get:

The integral contributions of this force can be determined exactly and are given by:

It may very well be sufficient to approximate the exponent function with the first-order Taylor series expansion. Since the force dissipates energy, exactness for the sake of accuracy is not required.

Viscosity can be applied in an anisotropic fashion, meaning that the viscosity constant k can actually be a vector

. The viscosity force then contains the dot product of

with vn.

A simple method of introducing surface contact friction is to generate an anisotropic viscous force in the event that a collision is determined. The viscosity in the direction of the collision surface normal can be tuned separately from the components in the surface plane in order to decouple the restitution of reflected velocity with sliding friction. Of course, this general viscosity force does not accurately model dry friction, where there is a transition from static to kinetic friction. But it is a good place to start.

Integral Contributions of Constraint Forces Many physics engines offer a variety of constraints on objects. We will not calculate the integral contributions of every possible constraint, but rather suggest a general mechanism of determining resolution forces to enforce that constraints are satisfied. In many physics engines, collisions are handled as constraints. The pattern for evaluating the resolution forces for a collision can apply to all constraints. This general pattern is as follows. 1. Approximate the trajectory and find the value of f, which represents the moment when the constraint violation began. 2. Forces are applied as pulses. Using the Jacobian of the constraint equations to determine the direction of the force, the magnitude is calculated in order to bring the velocity vn+1 into compliance with the constraint. 3. If needed, another force pulse is applied in the same direction in order to bring the position vn+1 into compliance. 4. The contributions for both pulses are added together.

Summary When using the kinematic integrator, the state of a body is defined by the position xn and velocity vn at the beginning of the current time step tn. The state of a body is stepped forward in time using the kinematic integration equations: vn+1 = vn + dv xn+1 = xn + vn Δt + dx The quantities dv and dx represent the portions of the stepping equations that can be analytically integrated per force and accumulated. The integral contributions can be calculated using the exact method as:

where the inner integral of dx is indefinite, and the outer integral is a definite integral. The integral contributions for the average acceleration method are:

The integral contributions for a pulse are:

If desired the integral contributions of the Symplectic Euler method are defined as: dv = an Δt dx = an Δt2 These different methods can be mixed, depending on the desired results. When there is more than one force applied that depends on position, then the average acceleration method can be much more accurate than the exact method. The acceleration contributions for the spring with a spring constant k and a natural length l are:

The contribution of a viscous force, with a viscosity constant v, is:

For a collision response force, the integral contributions are defined as:

where f is the fraction of the time step when the collision begins. For a general constraint, the generic framework is as follows: 1. Approximate the trajectory and find the value of f, which represents the moment when the constraint violation began. 2. Forces are applied as pulses. Using the Jacobian of the constraint equations to determine the direction of the force, the magnitude is calculated in order to bring the velocity vn+1 into compliance with the constraint. 3. If needed, another force pulse is applied in the same direction in order to bring the position xn+1 into compliance. 4. The contributions for both pulses are added together.

Conclusion Using a well-chosen mix of numerical integration and analytic integration, it is possible to achieve exact trajectories for some force models. If there are multiple forces applied, error may accumulate, since the analytic integration of individual forces cannot take other forces into account. Using the average acceleration method for calculating the integral

contributions can result in relative errors that are millions of times smaller than the errors from the Symplectic Euler method alone. We have seen that the Symplectic Euler method can be thought of as an exact method, which approximates the force as pulses. The errors in this method are due to this misrepresentation of the force. This gem has demonstrated the fact that the kinematics of simple physical models, which are prevalent in game-related physics, can be leveraged to dramatically reduce and sometimes eliminate the error of integration methods. Going forward, work on this topic might include determining the integral contributions due to the different flavors of translational and rotational constraints. Also, determining whether there is a way to pre-accumulate elements of the acceleration integrands prior to integration would provide a very natural solution to the problems that arise because of multiple forces that depend on position.

2.6. What a Drag: Modeling Realistic Three-Dimensional Air and Fluid Resistance B. Charles Rasco, Ph.D., President, Smarter Than You Software [email protected] Basic physics simulations have been around in games for quite a while. But beyond simple gravity, simple 2D collisions, and not-so-simple 3D collisions, there is a lack of further refinement of object motion. Drag physics is one area that has not been adequately addressed. Most games simulate drag physics with a simple linear model, if they simulate it at all. This gem demonstrates and contrasts two different mathematical models that simulate drag: a linear drag model and a quadratic drag model. The quadratic drag model is more realistic, but both models have applicability to different situations in physical simulations for many different types of games. The gem also defines and explains relevant parameters of drag physics, such as the terminal velocity and the characteristic decay time. The linear three-dimensional drag problem and both the one-dimensional linear and parts of the quadratic drag problems are discussed in [Davis86]. This gem adds a stable implementation of quadratic drag in three dimensions. The integrals used in this article may be found in any calculus book or on the web [Salas82, MathWorld]. Games could use improved drag physics for many things: artillery shells, bullets from smaller guns, golf ball trajectories, car racing games (air resistance is the main force that limits how fast a car can go), boats in water, and space games that involve landing on alien planets. Even casual games could use improved drag physics. The quadratic drag model in this gem was developed to make the iPhone and iPod Touch app Golf Globe by ProGyr. In this game, the user tries to get a golf ball onto a tee inside a snow globe. Originally the game was designed with linear water resistance, but it did not feel quite right. This was the motivation to improve the water resistance to the more realistic, and more complicated, quadratic model. I have a real golf ball in a snow globe and have never been able to get it onto the tee. To be honest, I have never gotten the ball on the tee in the Golf Globe game on the 3D eagle level, but in the game the easy levels are much simpler because the easier levels are two-dimensional.

Physics

This section provides the general mathematics and physics background for both the linear and the quadratic drag models. Three-Dimensional Physics The forces for the linear model are:

And the forces for the quadratic model are given by:

where is the velocity squared, and the unit vector that points in the direction of the velocity.

is

The first term of both equations is the force due to gravity, and in my representation down is the negative z direction. The direction of gravity depends on your coordinate system. The second term of both equations is the drag term. I chose the alpha and beta as different labels for the drag coefficient in order to keep it clear which version of air resistance I am referencing. For both models, if this value is big, there is a lot of fluid resistance, and if it is small, there is little fluid resistance. This parameter should always have positive value and does not actually need to remain constant. It can change if an object changes shape or angle from the direction of the fluid motion. A sail in the wind does not have a constant value, but changes depending on how taut the sail is and how perpendicular it is relative to the wind, among other factors. The velocity in these equations is the velocity of the object in the fluid, so if there is wind, then it would be the velocity of the object minus the velocity of the wind. The linear equation is solvable in a complete analytical closed-form solution. Since the quadratic mixes all of the components of the velocity vector together, it is not solvable in a closed form, but we can solve it numerically. In addition, the quadratic drag model has the advantage of being a better approximation of realistic air and fluid resistance. Interestingly, both are exactly solvable when considered as one-dimensional problems. This will be discussed in more detail later in this gem. Conversion from Three-Dimensional Physics to One-Dimensional Physics To convert the drag problem from three dimensions into one dimension, we need to break the object‘s motion into two components: motion along the velocity (one dimension) and motion perpendicular to the current velocity (two dimensions). The drag force only acts along the current direction of velocity; thus, we can treat it as a one-dimensional problem.

The unit vector in the direction of the velocity is given by

, or if

the velocity is zero it is valid to set . Setting the normal to zero works because if the object is not moving, there will be no fluid resistance. The components of the forces along the direction of motion and perpendicular to the direction of motion are shown in Figure 2.6.1. We can update the velocity along the current direction of motion as well as perpendicular to it once it is decomposed this way. Figure 2.6.1. Breaking three-dimensional space into components along the direction of motion and perpendicular to the current direction of motion.

One-Dimensional Solutions For a one-dimensional solution, we start from the one-dimensional version of Newton‘s Second Law, F = MA, and from the definition of acceleration, a = dv/dt. If the force is a function of the velocity alone, then these two equations can be combined to get time as a function of velocity, as shown here:

The force equations that are relevant to the present matter are F = mg cos(θ)– αv, the linear fluid resistance equation, and F = mg cos(θ)±βv2, the quadratic fluid resistance equation. The cosine in these equations is the cosine of the angle between the current velocity and the vertical. For the linear equation, the negative sign in front of the velocity term means the force always opposes the direction of motion. If the velocity is positive, then the force is in the negative direction. If the velocity is negative, then the force is in the positive direction, which is taken care of automatically by the linear equation. In the quadratic equation, the different sign must be applied based on which way the object is traveling, since squaring the velocity loses the directional information. For the linear equation, we have:

where vt1 = mg cos(θ)/α is the terminal velocity. There is one other variable that describes the motion, and that is the characteristic time, η1 = m/α. The characteristic time is a measure of how quickly (or slowly) the object speeds up or slows down to the terminal velocity, and it shows up in the equations after we integrate. The formula for the quadratic drag model‘s terminal velocity and characteristic time is different than the linear drag model‘s characteristic time and terminal velocity. The linear drag models are labeled with subscript 1s. Another way to calculate the terminal velocity is to set the force equation to

zero and solve for the velocity. This integral is evaluated with a change of variables x = 1 – v/vt1, so that dx = – dv/vt1.

Lastly, we solve this somewhat intimidating equation for vf, which turns out nicely as: Equation 1

If you feel like practicing taking limits, see what happens when the air resistance coefficient goes to zero. The traditional vf = v0 + g cos(θ)t pops up. The first few terms in this expansion are useful if you want to approximate the exact solution in order to avoid the expensive exponential function calls. For the quadratic case, we solve the equations in the same way. We start with the general equation:

From this equation we define the terminal velocity,

, and the

characteristic time is (not obvious from the equation) , which shows up after we integrate. Integrating this equation is straightforward. Although related, the plus version and the minus version are different integrals, and we need to evaluate both. First, let us do the integral when the terminal velocity term (labeled with subscript 2s for the quadratic model) and the velocity square terms have the same sign. This is the case where the force of gravity and the fluid resistance are in the same direction, which happens when the particle is traveling up and both forces are pointing down. To keep track of the overall sign relative to the velocity direction, we introduce the variable, sign, which is either 1 or –1, on the outside of the integral. This is important to get the correct overall sign compared to the direction of the velocity unit vector, . We continue to solve for the time as a function of initial and final velocity to obtain:

Now we solve this equation for the final velocity and obtain: Equation 2

Secondly, we solve for the case where the terminal velocity and velocity squared terms have opposite signs:

There is a relevant connection between the inverse hyperbolic tangent and the natural log of the form:

The absolute value bars from the integral are important in order to get the equation to involve the inverse hyperbolic tangent correctly. We must answer whether the initial and final velocities are bigger than the terminal velocity. In addition, given that the initial velocity is less than the terminal velocity, is the final velocity necessarily less than the terminal velocity? It turns out that it is (see the solution below), and it is acceptable to convert both natural logarithms to inverse hyperbolic tangents. A similar question can be asked if the initial velocity is greater then the terminal velocity. So there are four different cases with which we need to deal: V0 Vt2, V0 = Vt2, and Vt2 = 0. For the first case V0 < Vt2:

and solving for the final velocity: Equation 3a

For the second case V0 > Vt2:

and solving for the final velocity: Equation 3b

For the third case, v0 = vt2, where the object begins traveling at the terminal velocity in the same direction as the force of gravity, we get: Equation 3c

There are several ways to arrive at this result. We can take a limit of both of the first two cases, Equations (3a) and (3b), or if we look at the force in the quadratic force equation, it is zero, hence the object must travel at a constant velocity. And lastly is the case when the terminal velocity is identically zero, Vt2 = 0. It is easiest to start from the original equation and solve like the previous cases.

As previously, we solve this for the final velocity: Equation 3d

All right, this is all the information we need to solve for the linear and quadratic threedimensional cases, and we now transition to solving the three-dimensional linear case.

Solution of the Three-Dimensional Linear Case We could solve the linear three-dimensional case in a similar manner as we solve the quadratic version. However, we choose to treat it as three separate one-dimensional

problems because the results can be used by the AI in your game. The previous equations are modified to suit each of the Cartesian directions x, y, and z. Again, gravity is in the negative z direction. The solutions of the three different velocity equations are: vxf = vx0 exp(–t/η1) vyf = vy0 exp(–t/η1 and vzf = (vz0 – vtl)exp(– t/η1) + vtl with vtl = –mg/α and η1 = m/α. The terminal velocities in the horizontal direction are zero. If we integrate these equations with respect to time, then we get the position as a function of time. x = vx0 η1 (1 – exp(–t/η1) + x0 y = vy0 η1 (1 – exp(–t/η1)) + y0 and z = (vz0 – vtl)η1(1 – exp(–t/η1)) + vtlt + z0 The complete distance equations are useful for AI targeting, AI pathfinding, and calculating targeting info for players, without actually propagating the solutions forward in time step by step. In general, the distance equations are well approximated for most video games with x = x0 + vt, since the time for most games is very small and is much faster than updating with the complete position equations.

Solution of the Three-Dimensional Quadratic Case The three-dimensional quadratic equations are

,

. It is impossible to solve these as three independent equations since they are all coupled by the quadratic term of the fluid resistance. One way to solve this is to break the motion into two directions, along the current velocity and perpendicular to the current velocity, and then update the velocity based on the force calculated. Along the direction of motion, the force is Fparallel = –mgnz – βv2, where nz is the z component of and is equal to negative of the cosine of the angle between the direction of motion and the gravity vector. Notice that the drag term is always negative since it is in the opposite direction as the current velocity. Perpendicular to the direction of motion, the forces are . If the velocity is either straight up or straight down, nz = ±1 and nx = ny, there is no component of force perpendicular to the current velocity. And if the velocity is completely horizontal, the perpendicular force is completely in the negative z direction. If you have a different coordinate system, this equation will look slightly different.

For the motion along the velocity normal, there are three different cases to consider: nz> 0, nz< 0, and nz= 0. If nz > 0, the object is moving upwards, the opposite direction from gravity, and the fluid resistance is in the same direction as gravity. If nz< 0, then the object is moving in the same direction as gravity, and the fluid resistance force is in the opposite direction as gravity. Lastly, if nz= 0, then the object is moving horizontally, and gravity has no effect on horizontal motion.

For nz> 0, the relevant solution is Equation (2) with

,

, and sign = – 1 The velocity is:

For nz < 0, the relevant solution is Equation (3a), (3b), (3c), or (3d) depending on the current velocity relevant to the current terminal velocity with

,

, and sign = – 1. The velocity update is:

or

Notice that the characteristic time for the last instance is undefined, but it is never used in the solution, so there should not be a problem. For the force perpendicular to the initial velocity, the updated velocity is the same as Equation (3) with v0= 0 and its own distinct terminal velocity, , and characteristic time, This leads to:

.

For most cases this can be approximated (since the characteristic time is usually large and the update time is usually small for games) as: Equation 4

This last equation should look familiar. So usually there is no need to calculate the separate terminal velocity or the characteristic time for the perpendicular motion if the mentioned approximations are valid. There is one issue that occurs when a slow-moving object changes from traveling upward to going downward, as shown in Figure 2.6.2. If this case is not handled, then the object appears to go back in the direction it came from, when in fact it would stop traveling in the horizontal direction and travel down along the direction of gravity. Figure 2.6.2. A bad thing that can happen for slow-moving particles and/or large update times.

There are several ways to handle this. The most straightforward approach is to propagate the object for the time it takes to get to the top with Equation (2) and then for the rest of the time propagate with Equation (3a). From Equation (2), we can calculate the time it takes to stop moving in the positive velocity direction. The result is: Equation 5

If the time to the top is less than the total time to propagate, then subtract the time to the top from the total time to propagate. Then use the downward velocity update for the time remaining and use zero as the initial velocity. There might be a slight discontinuity in the object‘s smooth motion on the screen, but usually there is not. If there is too much, then update the position in a more rigorous manner. The last thing to consider is how to update the position. All of these equations are integrable, which is more accurate, but they provide little extra accuracy noticeable in a gaming environment and are computationally expensive to evaluate. For most game purposes, it is accurate enough to use Euler integration to calculate the position Xf = X0 + Vt, where the time, t, in the equation is the time since the last update and is not the global time. For the Golf Globe game on an iPod Touch, the game runs smoothly with no visible numerical issues at 60 frames per second by updating the position using Euler integration.

Pseudocode This is the pseudocode for a function that calculates the final velocity, velocityFinal, given the initial velocity, velocityInitial, of an object by the time, deltaTime. This does not update the position of the object, which should be updated separately but with the same deltaTime.

void updateVelocityWithFluidResistance( const float deltaTime, const ThreeVector& velocityInitial, ThreeVector& velocityFinal ) { Calculate Velocity Normal or zero it. Calculate terminal velocity and characteristic time along velocity normal. For objects going up use Equation (2), but first check to make sure the velocity does not reach the top Equation (5). For objects going down use either Equation (3a), (3b), (3c), or (3d) as is appropriate to the initial velocity at the beginning of the frame. Update the velocity perpendicular to the initial velocity with Equation (4). }

Comparison of Linear versus Quadratic Solutions There are benefits to using either the linear version or the quadratic version. The computational advantages lie with the linear solution. The quadratic computational cost is hardly prohibitive, especially if there are not too many objects updated with the quadratic model. And the quadratic model looks and feels much nicer, especially in a game where the user is getting feedback from movement due to handheld motion, such as from the iPod Touch or another handheld device. There are several advantages to the linear solution: There are exact velocity and position equations for all time, which is useful for AI and targeting; it is faster computationally; and the exponent in the solution is easily expanded, making it only minutely more computationally intense than updates with gravity only. One disadvantage to the linear solution is that it is not as realistic, especially when compared directly to the quadratic case. There are some advantages to using the quadratic solution: It is more realistic and it is not hugely more computationally intense (as long as there are not too many objects). There are several disadvantages to the quadratic solution: It is computationally more intense, especially if there are many objects; there is no analytic solution for all time, thus it is not as easy to use for AI and player targeting; and its approximations are more difficult and are more numerically touchy to the input drag parameters. There is a possible way to balance the two approaches if there are too many objects to update with the quadratic drag model. Use the linear model for the vast majority of these objects and then use the quadratic for the more important, or highly visible, objects. For both the linear and the quadratic solutions, it is a good approximation to use Euler integration to update the object position. It will save big on computational cost if there are many objects, especially for the quadratic case.

Conclusion The concept of drag is simple: Slow things down in the opposite direction they are moving. However, it is difficult to implement in a numerically accurate and stable manner. The simple and mathematically accurate linear drag model as presented here would be an improvement to many current games. The mathematically and physically accurate quadratic model is an additional improvement. There are many areas of simulation in games that would visibly benefit from using the mathematically and physically accurate quadratic drag model instead of the linear drag model: boats, sailboats (both with respect to the water and the wind in the sails), parachutes, car racing, snow-globe games, golf ball trajectories, craft flying through atmospheres, and artillery shells. Along the lines of this gem, there are several areas that would be interesting to research further. Physically based drag coefficients depend on all sorts of factors, including temperature, density, type of fluid, and the shape of the object. For spacecraft entering an atmosphere, several of these things are relevant and can make for some interesting gameplay while flying into land or chasing AIs or other players in a multiplayer game.

References [Davis86] Davis, A. Douglas. Classical Mechanics. Academic Press, Inc., 1986. [MathWorld] ―Wolfram Math World.‖ n.d. . [Salas82] Salas, S. L. and Einar Hille. Calculus: One and Several Variables with Analytic Geometry: Part 1. John Wiley and Sons, 1982.

2.7. Application of Quasi-Fluid Dynamics for Arbitrary Closed Meshes Krzysztof Mieloszyk, Gdansk University of Technology [email protected] Physics in real-time simulations as well as in computer games has been gaining importance. The game producers have observed that the advanced realism of the graphics is not enough to keep players content and that the virtual world created in a game should follow the rules of physics in order to behave more realistically. Until now, such realism has been mainly reflected by illustrating the interactions between the material objects. However, the phenomenon of reciprocity between such media as liquid or gas and more complex objects is also of high importance. With increasing processor power, the growing number of physics simulation methods that were initially developed purely for scientific reasons are being utilized in the entertainment market. However, simulation methods commonly used in games tend to be relatively simple in the computational sense or very focused on a specific case. An attempt to adapt them to other game variants is often impossible or requires an application of complicated parameters with values that are difficult to acquire. Hence, some more universal methods that allow for more easily modeled game physics are valuable and attractive tools for hobbyists and creators of later game extensions.

Requirements for the Mesh Representing the Object Commonly, the medium is a physical factor that in various ways affects the complete surface of the object. Hence, for the purpose of fluid dynamics simulation, every shape needs to be represented as a closed body with all its faces pointing outward and fully covering its outside surface. The inside should be looked upon as full; it must not have any empty spaces or ―bubbles.‖ Every edge should directly contact exactly two surfaces, while the entire object mesh must fulfill the following condition: Equation 1

where V= vertices, F= faces, and E= edges. Because the preliminary interaction value is calculated on points, it is convenient to organize the mesh in a list of points in the form of coordinates and a list of triangles represented as three sequential indices of its vertices. Additionally, every triangle should have a normal vector of its surface, as well as the surface value. In some cases, depending on the computational method used, the position of the triangle center can also be useful. In this article, we will sometimes refer to vertices as points and triangle meshes as faces.

Physical Rudiments Using some physical basics of fluid mechanics (referring among others to Pascal‘s Law), pressure is defined as a quotient of force F, which exercises perpendicularly to the surface sector limiting the object given, and the surface S of this sector [Bilkowski83]. Equation 2

The total pressure affecting the body surface in a given medium point can be represented by a sum of two components: static and dynamic pressure. Equation 3

In order to define the static pressure that is needed to create buoyancy in a selected medium, we need the medium density ρ[kg/m3], gravitational field intensity g [m/s2], and the distance to the medium surface h. (In the case of air, the altitude is needed.) Equation 4

Dynamic pressure originates from the kinetic energy pressure of the medium, which results from its movement and can be determined by knowing the medium density ρ as well as the fluid velocity V [m/s] at a selected position (applicable for subsonic speed). Equation 5

Equation (5) can be extended with a nondimensional pressure coefficient cp, whose value can be calculated as a cosine of the angle between the normal vector of the surface sector and the vector of the actual velocity in a selected point (see Figure 2.7.1). This can also be represented as a scalar product of the normal surface and the normalized velocity vector. Figure 2.7.1. Normal vector of the surface and normal vector of velocity and the angle between them.

Equation 6

As an effect, we obtain a general equation for dynamic pressure exerted on a segment of the surface shown in Equation (7). The method presented in this gem solves the computation of dynamic pressure based on velocity and assumes that we use a rigid body. The use of physical models based on joints or springs is also possible. Equation 7

Pre-Computation Based on a Triangle Mesh The whole issue of calculating the pressure distribution on a mesh comes down to computing the values on the mesh vertices (or center of triangles) taking into consideration the immersion, velocity, surface area, and normal of each of the triangles that form the object. This allows us to calculate the force of the pressure that influences the physical model. Additional problems arise in the case of the object being placed in two different media simultaneously. Here the division plane separates the object faces into those faces that are totally immersed in only one medium and faces that are located in both of them. The factors requiring serious consideration are characterized in the following section. Calculating the Distance to Medium Boundary For calculations of the static pressure affecting the object, it is necessary to define the distance h between the selected point and the medium boundary (for example, a water surface). For this, the scalar product can be applied, giving us the following Equation (8), schematically illustrated in Figure 2.7.2. Equation 8

Figure 2.7.2. Defining the distance of the point from the surface.

Assuming that the normal vector of the surface is pointing upward, the negative value of the obtained distance means that the point is located under the ―water‖ surface, and the positive is above its surface. In cases when the boundary is located at the virtual world ―sea level,‖ we know in which of the two media the point is located. To reduce the possibility of a calculation error, point Psurface (through which the surface dividing the media is crossing) should correspond to the projection of the object‘s center onto this surface, in accordance with its normal vector. Calculating the Point Velocity Because the objects simulated in the virtual world are frequently moving, one of the most important factors needing to be improved in their dynamics is the influence of the surrounding medium. Therefore, it is required to include the velocities of each of the object mesh points (Equation (9)) as a sum of the mass center‘s linear velocity Vobj and the object‘s angle velocity (Figure 2.7.3). The velocity Vp obtained this way represents the global velocity for selected point P. To calculate the dynamic pressure, the local velocity vector, which is equal to the negative global point velocity, is required [Padfield96]. Figure 2.7.3. Velocity of the arbitrary point belonging to the moving object.

We represent this mathematically as follows: Equation 9

Morphing Shape Objects The simulated object is not always a rigid body, because some objects move by changing their body shape. A fish or a bird can be used as an example here because various body fragments of those animals move with different velocities in relation to other parts. For such objects, when calculating the point velocity, it is necessary to take the so-called morphing effect into consideration. The simplest method here is to find the change between the old and new local positions of the mesh points (see Figure 2.7.4), followed by the velocity computation, keeping the time difference Dt between the morphing animation frames in mind. As a result and an extension to Equation (9), we obtain Equation (10), which is presented below. Equation 10

Figure 2.7.4. Moving vertex position in morphing mesh.

Triangle Clipping Because there is a situation in which the object triangle can be located in both media simultaneously, it is necessary to determine the borderline and find those parts that are completely submerged in only one of the media. To achieve this, the triangle should be divided into three triangles calculated by cutting points of the edges that cross the medium border. By applying Equation (11), we obtain PAB, PAC, and dividing the triangle ABC, we create a triangle A PAB PAC, located in one medium, and two arbitrary triangles located in the other medium (for example, B C PAB and C PAC PAB), as presented in Figure 2.7.5. Figure 2.7.5. Assigning the cutting points on the surface by edge AB.

It is important to keep in mind that the newly created triangles must have the same normal vectors as the primary triangle. Even though each of the new triangles has the same normal, it is still necessary to recalculate the current surface area of each of those triangles. It might be very useful to apply the cross-product property, described in Equation (12), for the exemplary triangle ABC. This surface area can then be used to calculate the pressure force. Equation 11

Equation 12

Calculation of the Pressure on the Triangle Surface

To calculate the pressure force exerted on the arbitrary triangle of the body, the pressure at each of the triangle vertices should be taken into consideration. Hence, we need to calculate the distance of each vertex from the medium surface for static pressure (Equation (6)) and the vertex local velocity, taking the angle between the velocity and the triangle surface normal into account, for dynamic pressure (Equation (7)). In the next step, the average complete pressure of the triangle vertices is calculated. However, it is useful to know that some simplifications are possible for cases when any mesh vertex has the velocity calculated from ωx r that is considerably smaller than the object linear velocity Vobj or when ω is close to zero. As a result of this, the pressure gradient distribution is almost linear, and the calculation of the complete pressure exerted on the triangle surface is reduced to obtaining the pressure in the triangle center (based on h and Vp). Thus, the pressure force is a product of a triangle normal, its surface area, and pressure (Equation (2)). For models based on a rigid body, calculation of the complete force and the torque comes down to summing the pressure forces from each surface fragment, as well summing all torques as pressure force vector cross-products and the forces at the center of the triangles.

Simple Correction of Dynamic Pressure The method for computing the static pressure presented in this gem is based solely on simple physical rules. However, the effect of dynamic pressure is one of the most complicated problems of the mechanics, for which no simple modeling method exists. The computational method used here is highly simplified and based on a heuristic parameter cp. This is sufficient for the purpose of physical simulation, which visually resembles the correct one; however, obtained values strongly diverge from the real values, resulting in too low of lift and too high of drag values. Hence, when calculating a force vector obtained from dynamic pressure, it is beneficial to introduce a correction in accordance with the velocity vector at a selected point. As in Equation (13), the factors used here are determined empirically and should be interpreted as recommended, not required. The corrected dynamic pressure force vector of the selected triangle has to be summed with the static pressure force vector occurring on its surface. The results obtained in that way approximate the results of the real object (for example, an airplane) for which the dynamic pressure is most important. Equation 13

Conclusion The simplified method of computing the flow around an arbitrary closed body presented here allows us to easily obtain approximate values of forces surrounding a medium. This is possible thanks to simulating some fundamental physical phenomena, such as static and dynamic pressure. Figure 2.7.6 shows screenshots from an application that interactively simulates real time using the algorithm presented here. This clearly shows a potential possibility for use in simulating the behavior of objects with fairly complicated shapes. The control here is carried out by changing the interaction between the medium and the object obtained, by morphing the shape of the object‘s surface, or by moving or rotating the object elements. This allows us to create controlled planes, ships, airships, submarines, hydrofoils,

seaplanes, swimming surfers, or drifting objects, as well as objects that undergo morphing, such as birds or fish. All of this is simulated based on the object mesh only. Figure 2.7.6. Examples of applications: drifting object (top), static object (middle), morphing object (bottom). (Grayscale marks the pressure distribution in the middle and bottom images.)

The simplifications used in the algorithm bring some inaccuracies. The main problem here is the lack of consideration for the reciprocal interferential influences for each face on the final pressure values. This is a consequence of analyzing each face separately, as if it were moving in the flow consistent with its own velocity. In reality, the velocity of the flow at a given fragment of the body surface depends on the velocity over the adjacent surface fragments [Wendt96, Ferziger96]. Dependence of the value on the pressure coefficient parameter cp (as angle cosine) is a generalization. In fact, this issue would require some more specialized characteristics. Another drawback of the material presented here is that the medium viscosity has not been taken into consideration, which also adds to inaccuracies in the results. These computation versus performance tradeoffs need to be evaluated when comparing real-time methods and those that may be used in fluid dynamics engineering and design. Implementations of the equations in this article motivated by specific examples are available on the CD-ROM.

References [Bilkowski83] Bilkowski, J. and J. Trylski. Physics. Warsaw: PWN, 1983.

[Ferziger96] Ferziger, Joel H. and Milovan Peri. Computational Methods for Fluid Dynamics. Berlin: Springer-Verlag, 1996. [Padfield96] Padfield, Gareth D. Helicopter Flight Dynamic. Oxford: Blackwell Science Limited, 1996. [Wendt96] Wendt, John F. Computational Fluid Dynamics. Berlin: Springer-Verlag 1996.

2.8. Approximate Convex Decomposition for Real-Time Collision Detection Khaled Mamou [email protected] Collision detection is essential for realistic physical interactions in video games and physically based modeling. To ensure real-time interactivity with the player, video game developers usually approximate the 3D models, such as game characters and static objects, with a set of simple convex shapes, such as ellipsoids, capsules, or convex hulls. While adequate for some simple interactions, these basic shapes provide poor approximations for concave surfaces and generate false collision detections (see Figure 2.8.1). Figure 2.8.1. Convex hull versus approximate convex decomposition.

In this gem, we present a simple and efficient approach to decomposing a 3D mesh into a set of nearly convex surfaces. This decomposition is used to compute a faithful approximation of the original 3D mesh, particularly adapted to collision detection. First, we introduce the approximate convex decomposition problem. Next, our proposed segmentation technique and performance characteristics will be evaluated. Finally, we conclude with some areas of future work.

Approximate Convex Decomposition Let S be a triangular mesh of connectivity k and geometry γ. Intuitively, S is a piecewise linear surface composed of a set of triangles stitched along their edges. The connectivity k is represented as a simplical complex describing the topology of the mesh. k is composed of a set of vertices X = {v1,v2,...,vv} ∊ IN (IN denotes the set of positive integers and V the number of vertices of k), together with a set of non-empty subsets of X, called simplices and verifying the following conditions: Each vertex v ∊ X is a simplex of k, and Each subset of a simplex of k is also a simplex of k. Ad-simplex is defined as a simplex composed of d + 1 elements of X. The 0-simplices correspond to the set of vertices of k, the 1-simplices to the set of its edges, and the 2-

simplices to the set of its triangles, denoted as θ = {t1,t2, ...,tT}. (T represents the number of triangles.) The geometry γ specifies the shape of the surface by associating 3D positions and usually surface normals to the vertices of k. A surface S is convex if it is a subset of the boundary of its convex hull (in other words, the minimal convex volume containing S). Computing an exact convex decomposition of an arbitrary surface S consists of partitioning it into a minimal set of convex sub-surfaces. Chazelle, et al. prove that computing such decomposition is an NP-hard problem and evaluate different heuristics to resolve it [Chazelle95]. Lien, et al. claim that the exact convex decomposition algorithms are impractical since they produce a high number of clusters, as shown in Figure 2.8.2 [Lien04]. To provide a tractable solution, they propose to relax the exact convexity constraint and consider instead the problem of computing an approximate convex decomposition (ACD) of S. Here, for a fixed parameter ε>0, the goal is to determine a partition Π = {π1,π2, ...,πK} of θ with a minimal number of clusters K and verify that each cluster has concavity lower than ε. Figure 2.8.2. Exact convex decomposition versus approximate convex decomposition.

The ACD problem has been addressed in a number of recent publications [Lien04, Lien08, Kraevoy07, Attene08]. Attene, et al. apply a hierarchical segmentation approach to a tetrahedral mesh generated from S [Attene08]. The tetrahedralization process exploited by Attene, et al. is, in practice, hard to compute and introduces extra computational complexity. Other methods avoid this limitation by considering the original 3D mesh directly [Lien04, Lien08, Kraevoy07]. Kraevoy, et al. introduce an incremental Lloyd-type segmentation technique exploiting a concavity-based seed placement strategy [Kraevoy07]. Here, the concavity measure is defined as the area weighted average of the distances from the clusters to their corresponding convex hulls. Lien, et al. claim that such concavity measure does not efficiently capture the important features of the surface [Lien04, Lien08]. They propose instead to compute the maximal distance between the mesh vertices and the clusters‘ convex hulls. Their divide-and-conquer approach iteratively divides the mesh until the concavity of each sub-part is lower than the threshold ζ, as shown in Figure 2.8.3. Here, at each step i, the vertex

with the highest concavity is selected, and the cluster to which

it belongs is divided into two sub-clusters by considering a bisection plane incident to

.

Figure 2.8.3. The divide-and-conquer ACD approach introduced in [Lien04] and [Lien08].

The main limitation of this approach is related to the choice of the ―best‖ cut plane, which requires a sophisticated analysis of the model‘s features. A public implementation of [Lien04] is provided in [Ratcliff06]. Here, the sophisticated feature analysis procedure is replaced by a simple cut plane selection strategy that splits each cluster according to its longest direction. This suboptimal choice generates over-segmentations, which are optimized by applying a post-processing procedure aiming at aggregating the maximal number of clusters while ensuring the maximal concavity constraint. As illustrated in Figure 2.8.7, the ACDs produced by Ratcliff are, in practice, suboptimal, since the aggregation procedure is applied to clusters generated only by plane-based bisections. To overcome the aforementioned limitations, we introduce, in the next section, a simple and efficient hierarchical ACD approach for 3D meshes.

Hierarchical Approximate Convex Decomposition Our proposed hierarchical approximate convex decomposition (HACD) proceeds as follows. First, the dual graph of the mesh is computed. (See the upcoming ―Dual Graph‖ section.) Then, its vertices are iteratively clustered by successively applying topological decimation operations, while minimizing a cost function related to the concavity and the aspect ratio of the produced segmentation clusters. Finally, by approximating each cluster with the boundary of its convex hull [Preparata77], a faithful approximation of the original mesh is computed. This surface approximation is piecewise convex and has a low number of triangles (when compared to T), which makes it particularly well adapted for collision detection. Let‘s first recall the definition of the dual graph associated with a 3D mesh. Dual Graph The dual graph S* associated with the mesh S is defined as follows. Each vertex of S* corresponds to a triangle of S. Two vertices of S* are neighbors (in other words, connected by an edge of the dual graph) if and only if their corresponding triangles in S share an edge. Figure 2.8.4 illustrates an example of a dual graph for a simple 3D mesh. Figure 2.8.4. Example of a dual graph.

Decimation Operator Once the dual graph is computed, the algorithm starts the decimation stage, which consists of successively applying half-edge collapse operations to S*. Each half-edge collapse operation applied to an edge (v,w), denoted hecol (v,w), merges the two vertices v and w, as illustrated in Figure 2.8.5. The vertex w is deleted, and all its incident edges are connected to v. Figure 2.8.5. Half-edge collapse decimation operation.

Let A(v) be the list of the ancestors of the vertex v. Initially, A(v) is empty. At each operation hecol (v,w) applied to the vertex v, the list A(v) is updated as follows: Equation 1

Simplification Strategy

The decimation process described in the previous section is guided by a cost function describing the concavity and the aspect ratio of the surface S(v,w) resulting from the unification of the vertices v and w and their ancestors [Garland01]: Equation 2

As in [Garland01], we define the aspect ratio Eshape(V,W) of the surface S (v,w) as follows: Equation 3

where ρ(S(v,w)) and ζ(S(v,w)) represent the perimeter and the area of S (v,w), respectively. The cost EShape(v,w) was introduced in order to favor the generation of compact clusters. In the case of a disk, the cost EShape equals one. The more irregular a surface, the higher its aspect ratio cost. Inspired by [Lien08], we define the concavity C(v,w) of S(v,w), as follows (see Figure 2.8.6): Figure 2.8.6. Concavity measure for a 3D mesh.

Equation 4

where P(M) represents the projection of the point M on the convex hull CH(u,w) of S(v,w), with respect to the half ray with origin M and direction normal to the surface S(v,w) at M. The global decimation cost E(v,w) associated with the edge (v,w) is given by: Equation 5

where D is a normalization factor equal to the diagonal of the bounding box of S, and α is a parameter controlling the contribution of the shape factor EShape(v,w) with respect to the concavity cost. (See the upcoming ―Choice of the Parameter α‖ section.) At each step of the decimation process, the hecol (v,w) operation with the lowest decimation cost is applied, and a new partition Π(i)= {π1(i),π2(i), ...,πK(i) } is computed as follows: Equation 6

where represents the vertices of the dual graph S* obtained after i halfedge collapse operations. This process is iterated until all the edges of S* generating clusters with concavities lower than ω are decimated. Choice of the Parameter α The clusters detected during the early stages of the algorithm are composed of a low number of adjacent triangles with a concavity almost equal to zero. Therefore, the decimation cost E is dominated by the aspect ratio related cost EShape, which favors the generation of compact surfaces. This behavior is progressively inverted during the decimation process, since the clusters become more and more concave. To ensure that the cost (α.EShape) has no influence on the choice of the later decimation operations, we have set the parameter α as follows: Equation 7

This choice guarantees, for disk-shaped clusters, that the cost (α.EShape) is 10 times lower

than the concavity-related cost

.

Experimental Results To validate our approach, we have first compared its segmentation results to those of [Ratcliff06], which provides a simplified version of the original algorithm described in [Lien04] and [Lien08]. In Figure 2.8.7, we compare the ACDs generated by our approach to those obtained by using Ratcliff‘s method. Here, the accuracy of the generated piecewise convex approximations is objectively evaluated by using the root mean squares (RMS) and Hausdorff errors [Aspert02]. Let‘s recall that the RMS error measures the mean distance from the original mesh S to its piecewise convex approximation S′. It is defined as follows: Equation 8

Figure 2.8.7. Comparative evaluation: (a,d,g,j,m,p) original meshes, (b,e,h,k,n,q) piecewise convex approximations generated by [Ratcliff06], and (c,f,i,l,o,r) piecewise convex approximations generated by the proposed HACD technique.

where D is the diagonal of the bounding box of S, α(S) is its area, and d(p,S′) is the distance from a point p∊S to S′. The distance d(p,S′) is given by: Equation 9

The Hausdorff error, denoted H, measures the maximal distance from S to S′ and is defined as follows: Equation 10

The reported results show that the proposed HACD technique provides significantly (that is, from 20 percent to 80 percent) lower RMS and H errors, while detecting a lower number of clusters. Figures 2.8.7(j) through 2.8.7(r) clearly show the limitations of the plane-based bisection strategy and the aggregation post-processing procedure of Ratcliff, which generates over-segmentations and poor ACDs. Color Plate 8 presents the segmentation results and the approximate convex decompositions generated by our approach for different 3D meshes. For all the models, the generated segmentations ensure a concavity lower than ε and guarantee that the maximal distance from S to S′ is lower than 3 percent of D. Therefore, the generated piecewise convex approximations provide faithful approximations of the original meshes with a small number of clusters. Moreover, our technique successfully detects the convex parts and the anatomical structure of the analyzed 3D models. For all of the models shown in Color Plate 8, the piecewise convex approximations were computed by considering an approximation of the clusters‘ convex hulls with a maximum of 32 vertices for each. The number of triangles composing the obtained convex surfaces is lower than 8 percent of T. Furthermore, the piecewise convexity property makes the generated approximations particularly well suited to collision detection.

Conclusion We have presented a hierarchical segmentation approach for approximate convex decomposition of 3D meshes. The generated segmentations are exploited to construct faithful approximations of the original mesh by a set of convex surfaces. This new representation is particularly well suited for collision detection. We have shown that our proposed technique efficiently decomposes a concave 3D model into a small set of nearly convex surfaces while automatically detecting its anatomical structure. This property makes the proposed HACD technique an ideal candidate for skeleton extraction and pattern recognition applications.

References [Attene08] Attene, M., et al. ―Hierarchical Convex Approximation of 3D Shapes for Fast Region Selection.‖ Computer Graphics Forum 27.5 (2008): 503–522. [Chazelle95] Chazelle, B., et al. ―Strategies for Polyhedral Surface Decomposition: An Experimental Study.‖ Symposium on Computational Geometry (1995): 297–305. [Garland01] Garland, M., et al. ―Hierarchical Face Clustering on Polygonal Surfaces.‖ Symposium on Interactive 3D Graphics (2001): 49–58.

[Hoppe96] Hoppe, H. ―Progressive Meshes.‖ International Conference on Computer Graphics and Interactive Techniques (1996): 99–108. [Kraevoy07] Kraevoy, V., et al. ―Model Composition from Interchangeable Components.‖ Pacific Conference on Computer Graphics and Applications (2008): 129–138. [Lien04] Lien, J.M., et al. ―Approximate Convex Decomposition.‖ Symposium on Computational Geometry (2004): 457–458. [Lien08] Lien, J.M., et al. ―Approximate Convex Decomposition of Polyhedra and Its Applications.‖ Computer Aided Geometric Design (2008): 503–522. [Aspert02] Aspert, N., et al. ―MESH: Measuring Error Between Surfaces Using the Hausdorff Distance.‖ IEEE International Conference on Multimedia and Expo 1 (2002): 705–708. [Preparata77] Preparata, F. P., et al. ―Convex Hulls of Finite Sets of Points in Two and Three Dimensions.‖ ACM Communication 29.2 (1977): 87–93. [Ratcliff06] Ratcliff, J. ―Approximate Convex Decomposition.‖ John Ratcliff‘s Code Suppository. April 2006. Bolgspot.com. n.d. .

Section 3: AI Introduction AI Level of Detail for Really Large Worlds A Pattern-Based Approach to Modular AI for Games Automated Navigation Mesh Generation Using Advanced Growth-Based Techniques A Practical Spatial Architecture for Animal and Agent Navigation Applying Control Theory to Game AI and Physics Adaptive Tactic Selection in First-Person Shooter (FPS) Games Embracing Chaos Theory: Generating Apparent Unpredictability through Deterministic Systems Needs-Based AI A Framework for Emotional Digital Actors Scalable Dialog Authoring Graph-Based Data Mining for Player Trace Analysis in MMORPGs

Introduction Borut Pfeifer

[email protected] With recent advances in graphics and animation, AI is one of the areas of game programming with the most potential for growth. The techniques in this section address some of the most troublesome and encouraging future areas for game AI development. Advances in AI architecture, believable decision-making, more detailed character simulation, and player modeling all offer the possibility of creating new or improved aspects of gameplay, as well as enhancing our ability to actually build that gameplay more quickly. AI architecture is crucial for effective development. A poor architecture can bring the development of new AI gameplay on a project to a halt. A good one can empower new features and new experiences. Cyril Brom, Tomáš Poch, and Ondřej Šerý discuss creating worlds with high numbers of NPCs in their gem, ―AI Level of Detail for Really Large Worlds,‖ managing multiple simulation levels for each area of the game world. Kevin Dill writes about using patterns in AI decision-making code in his gem ―A Pattern-Based Approach to Modular AI for Games.‖ The more effective an AI programmer is in creating reusable and scalable code, the more time there is to iterate on gameplay. AI movement and pathfinding are always the earliest difficult problems an AI programmer has to solve on a project. ―Automated Navigation Mesh Generation Using Advanced GrowthBased Techniques,‖ by D. Hunter Hale and G. Michael Youngblood, details their research on new methods of creating navigation meshes using space-filling algorithms. Michael Ramsey covers the pathfinding used for a wide variety of animals and movement types in World of Zoo in his gem ―A Practical Spatial Architecture for Animal and Agent Navigation.‖ Brian Pickrell sheds some light on the often-overlooked area of control theory and how it can be applied to an agent‘s steering in ―Applying Control Theory to Game AI and Physics.‖ Decision-making is the aspect of AI that affects players most directly. The challenge is finding meaningful decisions characters can make to appear intelligent, while also being understandable to the player. It is much easier to create decision-making that allows for an NPC‘s success than it is to balance understandable behavior with the complexity and depth required to appear intelligent. Thomas Hartley and Quasim Mehdi describe a method to allow NPCs to adapt better to players‘ combat behavior over time with their gem ―Adaptive Tactic Selection in First-Person Shooter (FPS) Games.‖ Dave Mark discusses how to use complexity in building AI decision-making models in ―Embracing Chaos Theory: Generating Apparent Unpredictability through Deterministic Systems.‖ More detailed character simulation can also help create that depth and believability. Rob Zubek breaks down an AI decision-making approach used in games such as the Sims series in his gem, ―Needs-Based AI.‖ Phil Carlisle takes a look at how we can easily add emotional modeling to a behavior tree–based architecture in his gem, ―A Framework for Emotional Digital Actors.‖ Baylor Wetzel‘s gem, ―Scalable Dialog Authoring,‖ confronts the difficult problem of creating dialog for many NPCs by abstracting concepts, such as a cultural group an NPC belongs to, and describes how to empower designers to author more variety in dialog with these abstractions. Player modeling is a burgeoning area of game AI. It can be used to help improve a player‘s experience, such as in Left 4 Dead. In the gem ―Graph-Based Data Mining for Player Trace Analysis in MMORPGs,‖ Nikhil Ketkar and G. Michael Youngblood write about their work modeling players in massively multiplayer games (MMOs). They‘ve used MMO player data to determine models to detect gold farmers and bots as well to determine effective locations for in-game advertising. Such models and processes can be used to improve a wide variety of aspects of the player experience. As game AI programmers, we are now in the spotlight to create the next level of innovative game experiences. People are no longer impressed by the same fancy normal maps and motion-captured animation; they want their characters to be more believable and more

entertaining in how they act. It‘s up us to make this a reality, and I hope the gems in this section will help give you ideas to solve some of these problems.

3.1. AI Level of Detail for Really Large Worlds Cyril Brom, Charles University in Prague [email protected] Tomáš Poch [email protected] Ondřej Šerý [email protected] One challenge for games featuring large worlds with many non-player characters (NPCs) is to find a good balance between the consumption of computational resources and simulation believability. On the one hand, the cost of simulation of a whole world, including all the NPCs, is enormous. On the other hand, if one simulates just the proximity of a player, one asks for plausibility troubles. For instance, when a player is supposed to return to a once left area, NPCs and objects in this area may not be in a believable state—the area has not changed since it was left, people have not moved, the ice cream left on the table has not melted, and so on. Additionally, this approach cannot handle NPCs that can move freely around the world. To compromise between these two extremes, a number of level-of-detail AI techniques (LOD AI) have been invented. While LOD AI for a traffic simulation [Chenney01] and for enemies in an RPG [Brockington02] and an action game [Grinke04] has already been written about, less is known about how to vary simulation detail for general NPCs. This gem presents a LOD AI technique tailored for simulations of large worlds featuring hundreds of commonplace NPCs with relatively complex behavior. These NPCs can perform tasks that include manipulation with several objects and require a variety of everyday behaviors, such as harvesting, merchandising, or watering a garden (as opposed to pure walking and fighting). The NPCs are interactive, and they can move around the world as the story dictates. The technique is gradual, which means that it allows for several levels of detail (LOD) based on the distance from the player or important places (Figure 3.1.1). The technique also considers the fact that whole locations containing objects and NPCs may cease to exist when the LOD decreases and need to be re-created when it increases again. Figure 3.1.1. Three types of level-of-detail AI techniques. Places are projected on the X-axis.

Gradual LOD Example Think of a classic fantasy RPG setting: a medieval county with many villages. There is a pub in one of them, where miners go to have some fun after their shift. A player can go there as well; she can leave and return any time. You want to avoid the situation in which she would realize that the miners were not being simulated properly when she was at the other end of the village, but you want to save the resources when she is out. You may define the following LODs ranging from the full simulation to almost no simulation. Detail 5: Full simulation—used when the player is in the pub or in its proximity. Detail 4: Every room from the pub is abstracted to a single point. The tables in the saloon are organized just in a list or an abstract graph; exact positions become unimportant. Miners (and other NPCs) are still sitting at the tables, but they are not drinking properly; they just empty the whole glass of beer in one go, say, every 20 to 40 minutes. The barman brings new beer, but now he is not walking as he would with LOD 5. Instead, he ―jumps‖ from table to table. The beer still has to be paid for. A miner can jump to another table as well, or to a next room, or he can leave the pub. LOD 4 will be used typically if the player is, say, 100 to 300 meters from the pub. Detail 3: The glasses, the tables, and the pub‘s rooms cease to exist. The whole pub is abstracted to a single point. Yet the miners can still be there (we don‘t know where exactly, but we need not care, as will be detailed later). The beer level in the barrel will decrease a bit every half an hour or so based on the number of miners in the pub. The miners can leave the pub or enter. LOD 3 will be used if the player is at the other end of the village. Notice that when she is approaching the pub, the detail first jumps to 4 and only then to 5. Additionally, when it jumps to 4 from 3, the miners must be assigned to tables, and glasses must be generated properly (for example, at the tables). Detail 2: The whole village is abstracted to a single point. The barman is not simulated because his story, as specified by designers, never leads him out of the bar, while the miners still are; they may go from a village (meaning the pub or home) to the mine (in other words, to work). This detail is used when the player is not in the village but is in its proximity. Detail 1: The village and its surroundings are abstracted to a single point; no village‘s inhabitants are simulated, but a foreigner may pay a visit. You can imagine that the foreigner is a story-important persona. His importance can demand that the area he is located in has an LOD of at least 4. Thus, as he moves, he redefines LOD similarly to the player. Alternatively, some events may be considered

important. When miners—in the pub, at LOD 3—start a brawl, the brawl can level up the detail to 5 no matter where the user is. Throughout, we will assume that LOD increases or decreases just by 1; larger changes can be done by repetition. We will assume one player in the simulation, but the technique can be used for multiplayer games as well. The LOD technique introduced in this game has been implemented as a part of a general simulator of 2D virtual words, including the aforementioned example (with some modifications). The simulator and the example are included on the CD.

Graphical LOD versus LOD AI Conceptually, it is often useful to think about a game‘s control mechanisms in terms of a two-layered architecture. While AI is the higher layer, physics and graphics are the lower layer. (See [Chenney01] for more on this point.) This is actually a simplified view, nevertheless useful for explanatory purposes. According to this metaphor, the player‘s direct experience is provided by the lower animation layer, which is only influenced by the higher layer. The LOD AI technique is only related to the higher layer. Since the position of the player‘s avatar forces the maximum LOD AI in correctly designed worlds, only the areas (or their parts) simulated at the maximum detail may be visualized. The animation layer takes the outcome of the AI layer as an abstract prescription of what to show. The animation layer operates with several graphical levels of detail, and it may add additional complexity above the finest LOD AI detail. In our example, LOD AI Detail 5 takes care of whether a miner will drink a beer or whether he will go to the waypoint on the right or on the left, but the movement of his hand or walking smoothly will be dealt with by the animation engine.

Simulation at the Full Detail This gem concerns itself only with the AI layer. Assume to start that our goal is to simulate the whole world at the full AI LOD and that we need not care about animation. A good way to think about what happens in the AI layer is in terms of a discrete-event simulation paradigm. According to this view, time is represented by a chronological sequence of simulation events, which are ordered in an event list. These simulation events are abstract entities that mark time instants in which the state of the game is modified. Additionally, every simulation event can generate a new simulation event to the event list or remove an existing event. Technically, every simulation event is associated with a piece of code. In a nutshell, after initialization, the whole system works in the following cycle: 1. Take the first simulation event from the event list and remove it from the list. 2. Process this event; that is, run the code associated with the event. As a part of this: a. Change the state variables of some entities. b. Insert new simulation events to the event list at appropriate places. c. Remove some simulation events from the event list. When this paradigm is used, it is important to distinguish between real time, which is the time the user experiences, and simulation time, which is the time of the simulated world as represented by the event list. One processing cycle happens, by definition, in zero simulation time (though it cannot happen in zero real time). For real-time games, these two times must of course be synchronized. For more on discrete-event simulations and event handling, see [Channey01, Harvey02, Wiki09].

The simulation paradigm per se says nothing about how the simulation events relate to what really happens in the game. There are designers who specify this; they must create simulation events around those changes in the state of the game that the AI layer should be aware of. In a sense, the simulation events present the AI layer‘s window into the virtual world. One class of simulation events is story-important events (for example, the dungeon‘s gate will open every day at midnight). These events typically will be pre-scheduled in the event list from the beginning or hooked into the event list by a trigger in run time (for example, from the moment the player enters the village, the neighboring dungeon will open every day at midnight). A different class of events is due to slow changes of objects‘ states (for example, increasing the rust level of a sword every week by one). But the most important class is due to changes caused by atomic actions of NPCs or the player; these actions must be represented by simulation events in the event list. Because these actions are indivisible from the standpoint of the AI layer, it suffices to represent each of them by a start event and an end event. Typically, these two events will not be present in the event list simultaneously. During processing of a start event, first, other parts of the game are notified that the action starts, and second, it is estimated how long the action will last, and its end event is hooked to the event list at the appropriate place (Figure 3.1.2). When this time comes and the end event is processed, states of some objects are changed, and the NPC‘s AI controller is invoked to decide what the next action of this NPC is, hooking the start event of that action to the event list. In fact, because start events tend to be hooked at the beginning of the event list, it is often possible to skip their generation and to hook the respective end events directly. Note that this mechanism allows you to have atomic actions with various durations. Figure 3.1.2. Atomic action of sipping a beer is represented in the event list by a start event and an end event.

Classic discrete-event simulations typically work only with simulation events hooked into the event list. However, for games, asynchronous events are also needed. That is, sometimes another part of the game can generate an event that has to be propagated to the AI layer and processed by it immediately. For instance, the collision detection may recognize that someone has nudged the sipping person with an elbow. This means that the atomic action of sipping a beer cannot be finished. Thus, the AI layer must delete its end event and generate the start event of spilling action instead. Another issue is that the time of actions‘ ends (thus, the time of end events) is only estimated by the AI layer. This is fine when we simulate a part of the world that is not visualized: An estimate becomes the actual duration provided the action is not interrupted by an asynchronous event. However, for visualized parts, duration of some atomic actions will be determined by the animation engine. End events of these actions need to be synchronized with the actual end of the action.

Toward LOD AI: Hierarchical Behavior The question of this gem is how to make the mechanism of start events and end events cheaper when the full detail is not required. Our solution will capitalize on the fact that behavior of NPCs can be represented hierarchically. This means, more or less, that the NPCs are conceived as having a list of behaviors that are decomposed to sub-behaviors, which are further refined until some atomic actions are reached. Figure 3.1.3 shows how this applies to a drinking miner. Figure 3.1.3. Hierarchical representation of a drinking miner’s behavior. Atomic actions are in gray. Other nodes represent tasks. Note that one detail can be assigned to more levels.

Conceptually, this follows the Belief-Desire-Intention architecture [Bratman87]. This will most likely be implemented with behavior trees or hierarchical FSMs—for example [Fu&Houlette04, Isla05, Champandard08]. In academic literature, the decomposition of high-level behaviors to sub-behaviors is often more complicated. There, the distinction between goals and tasks is often made. While goals represent what shall be achieved, tasks represent how to achieve this. Every goal can be accomplished by several tasks, and every task can be achieved by adopting some subgoals. Importantly, an NPC needs to perform only one task to achieve a goal, provided there is no failure, but it must fulfill all sub-goals, or most of them, to solve a task. Consequently, the behavioral decomposition is given by an AND-OR tree (AND levels for goals, OR levels for tasks). The CD examples feature such AND-OR trees; however, the LOD technique works well with any behavioral decomposition; the distinction between goals and tasks is unimportant for it. Thus, for explanatory purposes, we assume throughout that we have just ordinary sub-behaviors, and we call them tasks.

Whatever the exact representation is, the key is that a) the hierarchy can be constructed so that its highest level represents abstract behavior to which lower levels add more detail, and b) each level can be made executable. Thus: 1. In the design phase, construct the hierarchy in this way and manually assign LODs to its levels (refer to Figure 3.1.3). 2. During execution, determine what level of the behavioral hierarchy is the lowest that should be executed. This is the level of the LOD that corresponds to the detail of the simulation at the place where the NPCs are located (see Figure 3.1.4). Figure 3.1.4. ―Drinking at a table‖ is executed as atomic. Atomic tasks are in gray.

3. Execute this task as atomic, approximating what would happen when the simulation is run at a finer detail. Unfortunately, these three points brings many problems. We will address them in turn. What Does It Mean to Execute a Task as Atomic? Atomic execution means two things. First, every task to be executed atomically must be represented in the event list by a start event and an end event, similar to atomic actions. For instance, assume that there is LOD 4 in the pub. The corresponding behavioral level contains the task of drinking at a table. The event list will contain the end event of this act but not of atomic actions of sipping a beer or chatting. Second, for every task, designers must specify a) its result, and b) its estimated duration. When a task is simulated in full detail, its particular runs can have different durations. Often, the exact duration of a run will be influenced by the animation layer (for example, an NPC walks from one room to another in a different amount of time based on the positions of objects the NPC has to avoid). Fortunately, with lower LODs, you need not worry about exactly how long the task would have lasted had the simulation run in the full detail. All you need is a plausible estimate of that time. Based on this estimate, you place the task‘s end event in the event list. Note that you may generate this estimate from a probabilistic distribution given by designers.

When should the result (in other words, Point A) be revealed? There are three possibilities: 1) at the beginning of the task (that is, when the start event is processed); 2) at the end (in other words, when the end event is processed); or 3) during the task. For the drinking miner, (1) would mean drinking the glass at once and then doing nothing for 20 to 40 minutes, while (2) would be doing nothing for 20 to 40 minutes and then drinking the glass at once. Variant (3) is not consistent, and (1) changes the world sooner than it is known that the task is successfully finished. Thus, we recommend using (2). However, designing the scenario at different levels presents extra work, and you have to consider in which situations these shortcuts are really needed. In extreme, you program the whole simulation from scratch at each LOD, though programming more abstract levels is much simpler than the full detail. What to Do When the Detail Increases Sadly, the detail can elevate in the middle of a task‘s atomic execution—in other words, between its start event and end event. For instance, the player may enter the pub while the miners are drinking at a table atomically. We call such situation a task expansion and the part of the task that has already been performed at the lower detail a task stub (see Figure 3.1.5). When this happens, you need to: 1. Remove the end event of the expanded task from the event list. 2. Compute the partial effect of the task stub, if needed. 3. Start the simulation at the higher detail. Figure 3.1.5. LOD has increased from 4 (up) to 5 (bottom), creating a stub from drinking atomically. The original end event must be removed.

The partial effect of a task stub should approximate the outcome of the stub‘s detailed simulation (in other words, as if the task was running from the beginning until the moment of expansion at the higher detail). You again need the designer‘s specification and the code for determining the partial effect, which is extra work. Actually, it is necessary to perform Point 2 only if the user may notice the discrepancy; otherwise, it is sufficient to start execution of the expanded task at the lower detail from its beginning, pretending that the task stub has never happened. In other words, you specify that the partial effect is nothing. In our drinking example, this would mean that all the drinkers would start with a full glass at the time of the LOD increase, but this may be fine if

you do not visualize how much beer is in the glasses. However, consider watering a garden: Here, the player may find it strange that the gardener is always just starting with the first garden bed at the time the player enters the garden. It may be necessary to decide that some fraction of the garden has already been watered at the time of expansion. When you need to compute the partial effect of a task stub, what are the options? Often, it is desirable to avoid simulation of the task stub at the higher detail. This would give you an exact result, but at the cost of too much overhead. Instead, you need to consider which state variables the task changes predictably (see Figure 3.1.6, left and middle) and which it does not (see Figure 3.1.6, right). In the former case, the partial effect can be determined by a simple formula. In the latter case, a more sophisticated ad hoc mechanism has to be created. Figure 3.1.6. Although the final outcome of three tasks being executed atomically is the same (Point A), their details differ. At the finer LOD, the state variable may change in a predictable way (left, middle) or in an unpredictable way (right). The arrows denote the LOD increase.

For instance, think of the LOD increase from Detail 3 to 4 in the pub. On one hand, you can easily figure out how much beer has been drunk based on the number of miners in the pub. On the other hand, you also need to assign the miners to the tables. To do the latter, the designer has to come up with an ad hoc mechanism. What to Do When the Detail Decreases Let us have a location in which detail n should be decreased to n-1. We say that tasks (or atomic actions) corresponding to detail n are being pruned away, whereas tasks corresponding to detail n-1 are being shrunk—that is, starting to be executed as atomic. Since the LOD decrease must apply to a whole area, as detailed later, more tasks may need to be pruned away—for example, drinking tasks of all individual miners in the pub. These tasks may not end at the same moment. Thus, they should be stopped using the mechanism of partial effect described previously. Only then can you execute shrinking tasks as atomic. It is important to stop the tasks being pruned away at one instant; otherwise, the area will end up in an inconsistent state, with some tasks being simulated at detail n and others at n-1. An important point of any LOD AI is that, by definition, some information is lost during a LOD decrease. This brings two problems. First, the partial effect of tasks and actions being pruned away should be partly forgotten but also partly exploited by the subsequent simulation at LOD n-1. Second, when the LOD increases again, the missing information should be reconstructed. This is similar to lossy compression. The second problem was partly treated earlier (assigning miners to tables), and we will also return to it later. The first problem is exemplified now in Figure 3.1.7, showing a barman walking to a table while LOD goes from 5 to 4. In this example, we are concerned with the barman‘s position, but similar reasoning applies for any state of an object.

Figure 3.1.7. LOD decrease. ―Steps‖ are being pruned away at t2, while ―go-tonext-object‖ task is being shrunk into the atomic ―jump.‖

Recall that the saloon is a single point at LOD 4. Figure 3.1.7 shows that when the simulation runs on the LOD 4, the walking barman is engaged in the task ―go-to-nextobject,‖ which makes the barman ―jump‖ from one object to another. These objects are tables or the bar. If the saloon is not too oblong, we may further assume unitary distances (at LOD 4) between all pairs of these objects. At LOD 5, ―go-to-next-object‖ breaks down into a sequence of ―step-to‖ atomic actions. Now, consider the LOD decrease from 5 to 4 at time t2 —that is, around the middle of the ―go-to-next-object‖ from Object X to the table. Should the barman start his ―jump‖ from Object X or Place Y, and how long should it take? Because Place Y does not exist at LOD 4, it seems that the best option is to let the barman start his ―jump‖ from X. However, since you cannot roll back what has already happened, the barman would walk from X to Y twice, and a clever user may notice this should the detail increase soon (say, at time t3, when the barman is expected to be around Z). The second option is to say that the barman is somewhere on the way and compute how long the rest of the ―jump‖ (in other words, from Y to the table) will last. This is more accurate, but it creates an extra work. Additionally, this works well only when the object‘s state is updated in a predictable way at the finer detail (refer to Figure 3.1.6, left and middle). The first method actually works acceptably if you manage to avoid an early LOD increase. The longer the period between a LOD decrease and the subsequent LOD increase, the more inconsistencies due to a simplified determination of the initial state of the simulation at LOD n-1 are disguised. A simple mechanism for avoiding early LOD increase will be shown later.

Space Representation So far, we have spoken about how to simplify behavior given a LOD, but we also need to know the value of the LOD and its radius, and we need these values for all parts of the virtual world. Perhaps the simplest way to get this information is to represent the world hierarchically, as you can examine on the CD example. The spatial hierarchy keeps information about children, parents, and neighbors of every location except for the leaves and the root, which lack children or has only them, respectively. For simplification, assume now that the number of LODs equals the number of spatial levels. (This is not a strict requirement.) Now, a membrane metaphor can be used to describe which LOD is where. Imagine an elastic membrane cutting through the spatial hierarchy (see Figure 3.1.8), touching some locations. We say that every location or an atomic place that is at the membrane at a particular instant is simulated as an abstract point. No location ―below‖ the membrane exists. Every NPC is simulated at the LOD equal to the level on which the membrane is touching the area in which that NPC is located; spatial LOD determines behavioral LOD. Figure 3.1.8. LOD membrane.

Formally, the membrane is nothing more than a list of locations that are simulated as abstract points at a particular moment. The membrane can be reshaped in every time step. For the purposes of coherence, we enforce the following shaping rules: 1. If a location or an atomic place X is at the membrane, every location or atomic place with the same parent as X is at the membrane, too. 2. If a location Y is above the membrane, every location with the same parent as Y is above or at least at the membrane, too. For example, if an atomic place from the saloon is at the membrane, all the saloon‘s atomic places will be at the membrane, too (Rule 1). This ensures that when the simulation runs in the full detail somewhere in the saloon, it will run in the full detail everywhere here. Because the saloon itself is above the membrane in this case, all rooms from the pub must be simulated at least as abstract points (Rule 2). This ensures that almost always at least something happens in the locations near the center of attention.

Because LODs of two adjacent locations can differ (that is, when they do not have the same parent), the visibility of a player should be limited to the location she is in. If this is not possible, such as in open spaces, another shaping rule must be settled. 3. The detail must be maximum in all locations the player can see at a given instant. Limitations of Rules 1 through 3 Rules 1 through 3 work fine, but you should know that there is an exception in which they do not guarantee that something happens near the center of attention. The problem is that two neighboring locations may have different parents, thus their LODs can differ by more than 1. Think of a world with two kingdoms. There are two houses there next to each other, but the border goes right between them. Even though the first house is simulated in full detail, the second kingdom may still have just LOD 1; the houses do not have a common parent. To deal with this, a mechanism of influences is needed. This would allow you to specify another shaping rule based on the influence a location has on its all neighboring locations. The trouble with this mechanism is that it may cause cascade effects during membrane reshaping, increasing the overhead. You can read more about this topic in the documentation on the CD. Pathfinding So far, we have been silent on pathfinding. With the spatial hierarchy, it is apparent that a hierarchical refinement of A* can be used easily for simplifying pathfinding at a lower LOD.

Positions of Objects and NPCs We know that the pub is abstracted to a single point at LOD 3. Assume there is one miner and one barman there. What happens with their positions during LOD changes? Because the pub‘s rooms do not exist at LOD 3, both the barman and the miner are, so to speak, spread out in the whole pub as two uncollapsed quantum particles; they are at the membrane. Assume further that the LOD decreases to 2. While this lifts the miner a level up, spreading him out in the village, the barman becomes non-simulated because his story never leads him out of the pub (see Figure 3.1.9). This means that the barman ―fell through‖ the membrane; his traces have been lost for the rest of the simulation. Figure 3.1.9. LOD increase. Some objects are ―lifted up,‖ while others are not.

The problem is that we need to narrow down object and NPC positions after the detail elevates. When the LOD goes from 2 to 3, we know that the barman has to be collapsed into the pub, but this is not the case of the miner. We do not know where to generate him; he can be collapsed into the pub or into his house or on the street, and so on. A similar problem arises for the LOD increase from Detail 3 to 4. We now introduce a basic mechanism dealing with generation of positions of objects. The real case is actually more complex than this mechanism. Following, we will extend it and comment on how to use it for NPCs. 1. Objects in every location should be divided into two groups—those that are locationnative, for objects owned by the location, and those that are location-foreign, for objects owned by other locations. (A glass is location-native in the pub, as opposed to a mine truck.) An object can be location-native in more locations. (A glass is location-native in the bar as well as in the kitchen.) 2. Every location from the nth level of the spatial hierarchy should have a method implementing where to generate location-native objects if the detail elevates from n to n+1 (for example, during the Detail 3 to 4 transition, the pub ―knows‖ where to generate the tables and glasses). 3. When LOD decreases in a particular location, the detailed positional information for all the location-foreign objects having been ―lifted up‖ is memorized, but not for the location-native objects having been ―lifted up‖ (due to Point 2). 4. When location-native objects ―fall through‖ the membrane, only their total number in the area where they ―fall though‖ is remembered (for example, after the 4-to-3 transition, it will be remembered that there are, say, 27 beer glasses in the pub). For location-foreign objects, the exact positions are remembered. (After the same transition, the watering can will become non-simulated, but two memory records will be kept: for the 5-to-4 and 4-to-3 transitions.) 5. When LOD increases, location-foreign objects are generated based on their stored positional information. In some rare situations, this information may not be available; this will be discussed in a moment. The idea behind this is that during the design phase, one can specify relatively easily which objects are native in which locations and implement placing algorithms appropriately. Storage of Positional Information

When a location-foreign object is simulated, it can hold its own positional information. The records about location-foreign non-simulated objects‘ positions and about numbers of location-native non-simulated objects can be kept by parent locations. (For example, the pub ―knows‖ that it contains 27 glasses.) Often, when the parent location ceases to exist, the player is so far away that the records held by this location can be happily forgotten. Nevertheless, should a piece of information survive destruction of the location to which it is attached, you can do the following. In the design phase, specify the information level denoting the LOD at which this particular kind of information can be forgotten. At run time, after the location ―falls through‖ the membrane, attach your record to the upper location (in our case, the village) provided the new LOD is still higher than or equal to the information level of this record. (Mind the direction: A higher LOD is lower in the hierarchy!) General Information and Expirations The aforementioned mechanism can be generalized in two ways. First, one can store not only positional information, but any state information. Assume a player has broken a window in the pub. A clever player would not return soon, but if she does, she may expect the window to still be broken. The record about the broken window can be stored by the pub. Similarly, sometimes it may be beneficial to record the exact positional information for a location-native object. If the player moves a table a bit, and the pub‘s LOD goes to and fro, the basic aforementioned mechanism would generate the table incorrectly, on the original place. The extra record will remedy this problem. After a while, this record may become useless; sooner or later, someone may repair the window and move the table back. If you need this feature, you can basically delete a record after a specified amount of time using a mechanism of expirations. Initialization and Traveling Objects There are two situations in which it is necessary to initialize objects‘ positions. The first one occurs after the simulation starts. Additionally, sometimes it is necessary to initialize objects‘ positions when the objects move between locations. The first situation is trivial: You must have an implicit initialization procedure. Let us now elaborate on the second case. Assume the pub is simulated at LOD 4, meaning all the pub‘s rooms are single points. A miner puts down a watering can in the saloon, where the can is location-foreign. Now, the LOD elevates to 5. Where do you place the can? Sometimes, the procedure initializing positions at the beginning of the simulation can help. More often, you would find useful a general placing mechanism that groups objects based on their typical places of occurrence into floor-objects, table-objects, and so on, and that generates these objects randomly within constraints of the objects‘ categories. Consider now that LOD decreases later from 5 to 4, and the miner picks up the can in the pub and takes it out. If this happens, do not forget to delete the record about the can‘s detailed position. NPCs When the detail increases, we have to refine a) positions of NPCs, and b) tasks they are engaged in. For (a), we can treat NPCs as objects. For (b), the task expansion mechanism described earlier should be used.

Reshaping the Membrane

The final question is how to shape the membrane. A simple solution, adopted by the example on the CD, is to assign an existence level and a view level to every object. If the detail decreases below the existence level, the object ceases to exist; otherwise, it is to be simulated. The view level then determines the detail required by the object after it starts to be simulated. Hence, the view level is always equal to or higher than the existence level. For many common objects, these levels will be equal, but not for story-important objects or NPCs. Every user‘s avatar will have the existence level equal to 1 and the view level equal to the maximum. Assume now a traveler with his view level set to 4 and his existence level to 2. He will not exist until the detail is at least 2, but when it elevates to this value, the traveler will demand that in the location to which he has been generated, the detail goes further to 4. Note that according to the shaping rules, this further determines increasing detail in some of the neighboring locations. Because this may create new objects demanding additional LOD increases, a cascade of changes may be triggered. The algorithm for doing this consistently is detailed in [Šerý06]. Sometimes, you may want to increase the detail even if no important object is around; for instance, think of a brawl being simulated in a village far away that is expected to grow into a local uprising. You may need to show the behavior of those NPCs to the player in a cut scene. To do this, you can simply put an invisible object into the simulation with the appropriate view level to ensure the player will see the AI behavior. What Is the Radius of LODs? Using this technique, all users and important objects/NPCs tend to automatically create a ―simulation crater‖ around them due to the spatial hierarchy and the shaping rules. As they move, the detail elevates (typically) by one in locations at the edge of the crater, avoiding abrupt jumps of LOD. LODs are not specified in terms of metrical distances, but in terms of number of locations between the object/NPC and the edge of the crater. This has two advantages. First, the LOD does not change all the time, as would be the case with pure metrical distances, helping to reduce the overhead. Further, the overhead is spread in time: If you go from 1 to 5, you do this in more steps. Figure 3.1.10 demonstrates that there is indeed a qualitative difference in overhead between the 4-to-5 increase (left) and 3-to-5 (right) in the pub in our CD example. Figure 3.1.10. Processor consumption during the LOD increase.

The X-axis represents the time in the game. The LOD increases at 10:15 p.m.; this time is arbitrary. This figure shows data from three particular runs. Note that the data for the 3to-4 increase resembles the data for the 4-to-5 increase.

Second, by the time a person arrives at a location, that location typically has been simulated for a while, disguising inconsistencies caused due to lower LODs.

When to Decrease the Detail There is a problem with reshaping the LOD membrane when an object moves between two locations repeatedly. The LOD may start to oscillate, increasing the resources‘ consumption. You can do either of the following two things. First, you can have a larger crater determining when to decrease the LOD. This larger crater would embrace the smaller crater that enforces the LOD increase. Second, you can use, as we did, a garbage mechanism. With the garbage mechanism, you do not decrease LOD until the resources are needed for something else. This mimics the larger crater automatically. However, because the cleanup causes overhead, it is better to use the garbage mechanism a bit sooner than the resources are actually needed, letting the mechanism work over several time steps.

Creating the Structure of the World In practice, it rarely makes sense to have more LODs than levels of the spatial hierarchy. In the spatial hierarchy, you should keep a reasonable number of sublocations for every parent—neither 1 nor 50. This is not a strict rule, but if you violate this principle too often, the hierarchy should be reconstructed. Even though higher numbers may make sense logically (a building with many small rooms), with respect to the technique presented here, this would increase the overhead for a LOD change. Recall that LODs also have to be assigned to the behavioral hierarchy. You should do this consistently, meaning the degree of abstraction of two tasks from a particular level should be similar. For instance, if LOD 3 is assigned to watering a garden, it should also be assigned to cooking, but not peeling potatoes. For technical purposes, you may need more levels in behavioral hierarchy than LODs, as demonstrated on the CD example. Another aspect to keep in mind is that after assigning LODs to tasks, objects required for these tasks must have their existence level set accordingly. Assume watering a garden has LOD 3, and the garden is an abstract point at LOD 3. If it is required by designers that watering a garden has to run with a watering can at this LOD, meaning the gardener has to pick up the can before he comes to the garden, the existence level of the can must be 3 or less. Watering a garden would then mean something like, ―Stay there holding the can for a specified amount of time, after which the garden will become watered at one instant.‖ Of course, designers might also specify that the gardener is able to water the garden without the can at LOD 3; the task‘s outcome would be the same, only the can would not be required. In the latter case, you would need to generate the can after the LOD increase. Note that the technique does not allow for two existence levels for one object kind—for example, it is not possible that a knife for fighting has existence level 3, while a knife for cooking has 5. You must define either one object kind with the lower number or two different object kinds with two different existence levels.

Source Code Summary The application included on the CD is a simulator of virtual worlds with LOD AI (in other words, it is not a game). The examples discussed in this chapter are based on the implemented demo world, which features five LODs and three kinds of NPCs: miners, barmen, and singers. The following behaviors have been implemented: walking home from a pub or a mine and vice versa, leisure time behavior (for pubs), and working behavior (for mines). The code is in Java, and the documentation that is included details the LOD technique further.

Conclusion Level-of-detail techniques allow for compromising between the consumption of computational resources and simulation plausibility. However, every LOD technique presents extra work for designers and programmers. Thus, it should be contemplated carefully which LOD approach is needed and whether it is needed at all. The technique introduced in this gem fits well for large worlds with many NPCs with everyday behavior. It capitalizes on the fact that both space and behavior of NPCs can be, and often already are, represented hierarchically. The different number of LODs helps not only with simulation plausibility, but also with keeping the overhead under control during LOD changes. The technique is generic, which means that for specific domains, special-purpose mechanisms can outperform it—for example, for fighting behavior [Brockington02, Grinke04]. Even in complex worlds, any special-purpose mechanism can be augmented with some or all of this technique if needed.

Acknowledgements This technique and the simulator on the CD were developed as a part of research projects 1ET100300517 of the Program Information Society and MSM0021620838 of the Ministry of Education of the Czech Republic. We want to thank several students who participated in the simulator development: Martin Juhász, Jan Kubr, Jiří Kulhánek, Pavel Šafrata, Zdeněk Šulc, Jiří Vorba, and Petr Zíta.

References [Bratman87] Bratman, Michael E. Intention, Plans, and Practical Reason. Harvard University Press, 1987. [Brockington02] Brockington, Mark. ―Level-Of-Detail AI for a Large Role-Playing Game.‖ AI Game Programming Wisdom I. Boston: Charles River Media, 2002. 419–425. [Champandard08] Champandard, Alex J. ―Getting Started with Decision Making and Control Systems.‖ AI Game Programming Wisdom IV. Boston: Charles River Media, 2008. 257–264. [Chenney01] Chenney, Stephen. ―Simulation Level-Of-Detail.‖ 2001. University of Wisconsin. n.d. . [Fu04] Fu, Dan, and Ryan Houlette. ―The Ultimate Guide to FSMs in Games.‖ AI Game Programming Wisdom II. Boston: Charles River Media, 2004. 283–302. [Grinke04] Grinke, Sebastian. ―Minimizing Agent Processing in ‗Conflict: Desert Storm.‘‖ AI Game Programming Wisdom II. Boston: Charles River Media, 2004. 373–378. [Harvey02] Harvey, Michael, and Carl Marshall. ―Scheduling Game Events.‖ Game Programming Gems 3. Boston: Charles River Media, 2002. 5–14.

[Isla05] Isla, Damian. ―Handling Complexity in Halo 2.‖ 3 Nov. 2005. Gamasutra. n.d. . [Šerý06] Šerý, Ondřej, et al. ―Level-Of-Detail in Behaviour of Virtual Humans.‖ Proceedings of SOFSEM 2006: Theory and Practice of Computer Science 3831 (2006): 565–574. [Wiki09] Wikipedia, The Free Encyclopedia. ―Discrete Event Simulation.‖ 2009. Wikipedia. n.d. .

3.2. A Pattern-Based Approach to Modular AI for Games Kevin Dill, Boston University [email protected] A great deal of time and effort is spent developing the AI for the average modern game. Years ago, AI was often an afterthought for a single gameplay programmer, but these days most game projects employ at least one dedicated AI specialist, and entire AI teams are becoming increasingly common. At the same time, more and more developers are coming to realize that, even in multiplayer games, AI is not only a critical component for providing fun gameplay, but it is also essential if we are going to continue to increase the sense of realism and believability that were previously the domain of physics and rendering. It does no good to have a brilliantly rendered game with true-to-life physics if your characters feel like cardboard cutouts or zombie robots. Given this increase in team size, the increasing prominence of AI in the success or failure of a game, and the inevitable balancing and feature creep that occur toward the end of every project, it behooves us to search for AI techniques that enable fast implementation, shared conventions between team members, and easy modification. Toward that end, this gem describes methods for applying patterns to our AI in such a way as to allow it to be built, tuned, and extended in a modular fashion. The key insight that drives the entirety of this work is that the decisions made by the AI can typically be broken down into much smaller considerations, individual tests used in combination to make a single decision. The same considerations can apply to many different decisions. Furthermore, the evaluation of those considerations can be performed independent of the larger decision, and then the results can be combined as necessary into a final decision. Thus, we can implement the logic for each consideration once, test it extensively, and then reuse that logic throughout our AI. None of the core principles described here are new. They can be found throughout software engineering and academic AI in a variety of forms. However, all too often we game programmers rush into building code, trying to solve our specific problem of the day and get the product out the door, without taking a step back and thinking about how to improve those systems. With some thought and organization, it could be easier to change, extend, and even reuse bits and pieces in future projects. Taking the time to do that would pay off both in the short run, making life easier (and thus improving the final result) for the current title, and in the long run, as more AI code is carried forward from game to game.

A Real-World Example: Apartment Shopping Let‘s begin with a real-world example that illustrates the general ideas behind this work. Imagine that you have just taken a new job as an AI engineer at Middle of Nowhere Games, and you are in the process of searching for an apartment in some faraway city. You might

visit a variety of candidates, write down a list of the advantages and disadvantages of each, and then use that list to guide your final decision. For an apartment in a large complex near a busy shopping area, for example, your list might look something like this: 606 Automobile Way, Apt 316 Pros

Cons

Close to work

Great view…of a used car lot

Easy highway access

No off-street parking

Convenient shopping district Highway noise

Another apartment, located in the attic of a kindly old lady‘s country house, might have a wholly different list: 10-B Placid Avenue Pros

Cons

Low rent

45-minute commute

Nearby woods, biking trails

No shopping nearby

Electricity and water included Thin walls, landlady downstairs

These lists clearly reflect the decision-maker‘s personal taste—in fact, it seems likely that if two people were to make lists for the same apartment, their lists would have little in common. It is not the actual decision being made that‘s important, but rather the process being used to arrive at that decision. That is to say, given a large decision (Where should I live for the next several years of my life?), this process breaks that decision down into a number of independent considerations, each of which can be evaluated in isolation. Only after each consideration has been properly evaluated do we tackle the larger decision. There is a reasonably finite set of common considerations (such as the rent, ease of commute, size of the apartment, aesthetics of the apartment and surrounding environment, and so forth) that would commonly be taken into account by apartment shoppers. From an AI point of view, if we can encode those considerations, we can then share the logic for them from actor to actor, and in some cases even from decision to decision. As an example of the latter advantage, imagine that we wanted to select a location for a picnic. Several of the considerations used for apartments—such as overall cost, the aesthetics of the surrounding environment, and the length of the drive to get there—would also be used when picking a picnic spot. Again, if it were an NPC making this decision, then cleverly designed code could be shared in a modular way and could perhaps even be

configured by a designer once the initial work of implementing the overall architecture and individual considerations was complete. Another advantage of this type of approach is that it supports extensibility. For example, imagine that after visiting several apartments, we came across one that had a hot tub and pool or a tennis court. Previously, we had not even considered the availability of these features. However, we can now add this consideration to our list of pros and cons without disturbing the remainder of our logic. At this point we have made a number of grandiose claims—hopefully sufficient to pique the reader‘s interest—but we clearly have some practical problems as well. What is described above is a wholly human approach to decision-making, not easily replicated in code. Our next step, then, should be to see whether we can apply a similar approach to making relatively simple decisions, such as those that rely on a single yes or no answer.

Boolean Decisions Many common architectures rely on simple Boolean logic at their core. For example, from a functional point of view, finite state machines have a Boolean decision-maker attached to each transition, determining whether to take that transition given the current situation. Behavior trees navigate the tree through a series of Boolean decisions (take this branch or don‘t take this branch) until they arrive at an action they want to take. Rule-based architectures consist of a series of ―rules,‖ each of which is a Boolean decision stating whether or not to execute the associated action. And so forth. Constructing Decisions To build a pattern-based architecture for our AI, we first need to define a shared interface for our considerations and then decide how to combine the results of their evaluation into a final decision. For Boolean decisions, consider the following interface:

class IConsiderationBoolean { public: IConsiderationBoolean() virtual ~IConsiderationBoolean()

{} {}

// Evaluate this consideration bool Evaluate(const DecisionContext& context) = 0; // Load the data that controls our decisions void LoadData(const DataNode& node) = 0; }

Every consideration will inherit from this interface and therefore will be required to specify the Evaluate() and LoadData() methods.

Evaluate() takes a DecisionContext as its only argument. The context contains whatever information might be needed to make a decision. For example, it might include the current game time, a pointer to the actor being controlled, a pointer to the game world, and so on. Alternatively, it might simply be a pointer to the actor‘s knowledge base, where beliefs about the world are stored. Regardless, Evaluate() processes the state of the world (as contained in the context) and then returns true if execution is approved or false otherwise.

The other mandatory function is LoadData(). The data being loaded specifies how the decision should be made. We will go into more detail on this issue later in this gem. Games are rife with examples of considerations that can be encoded in this way. For example, we might have a health consideration that will only allow an action to be taken if a character‘s hit points are in a specified range. A time-of-day consideration might only allow an action to take place during the day (or at night or between noon and 1:00 p.m.). A cooldown consideration could prevent an action from being tak