GPU-Based Interactive Visualization Techniques

Mathematics and Visualization Series Editors Gerald Farin Hans-Christian Hege David Hoffman Christopher R. Johnson Konra...

Author: Daniel Weiskopf

88 downloads 1548 Views 8MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

Mathematics and Visualization Series Editors Gerald Farin Hans-Christian Hege David Hoffman Christopher R. Johnson Konrad Polthier Martin Rumpf

Daniel Weiskopf

GPU-Based Interactive Visualization Techniques With 112 Figures, 42 in Color and 11 Tables

ABC

Daniel Weiskopf School of Computing Science Simon Fraser University Burnaby, BC V5A 1S6, Canada E-Mail: [email protected]

Library of Congress Control Number: 2006931796 Mathematics Subject Classification: 68U05, 68W10, 76M27 ISBN-10 3-540-33262-6 Springer Berlin Heidelberg New York ISBN-13 978-3-540-332626 Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com c Springer-Verlag Berlin Heidelberg 2006 The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. A EX macro package Typesetting by the author and SPi using a Springer LT Cover picture by Daniel Weiskopf Cover design: design & production GmbH, Heidelberg

Printed on acid-free paper

SPIN: 11730132

46/SPi/3100

543210

F¨ ur Elisabeth und Gerhard

Preface

Graphics processing units (GPUs) have been revolutionizing the way computer graphics and visualization are practiced. Driven by the computer-games industry and its demand for eﬃcient hardware support for 3D graphics, GPUs have dramatically increased in performance and functionality within only a few years. Although graphics hardware is primarily designed for the fast rendering of 3D scenes, it can also be used for other types of computations. In fact, GPUs have evolved to programmable processors that can facilitate applications beyond traditional real-time 3D rendering. This book addresses scientiﬁc visualization as one application area that signiﬁcantly beneﬁts from the use of GPUs. In general, scientiﬁc visualization has become an important tool for visual analysis in many scientiﬁc, engineering, and medical disciplines. For example, scientists and engineers regularly use visualization to interpret simulations of air or water ﬂow in computational ﬂuid dynamics. Another example is the medical imaging of 3D CT (computer tomography) or MRI (magnetic resonance imaging) scans. The interactive exploration of data sets is becoming increasingly more important with the growing amount and complexity of those data sets: the user – as an expert in his or her ﬁeld – uses visualization as a tool to investigate the data and extract insight from it. This book focuses on eﬃcient visualization techniques, which are the prerequisite for interactive exploration. High performance is primarily achieved by algorithms speciﬁcally designed for GPUs and their special features. Other aspects discussed in this work include parallelization on cluster computers with several GPUs, adaptive rendering methods, multi-resolution models, and non-photorealistic rendering techniques for visualization. This book also addresses the eﬀectiveness of visualization methods, which can be improved by taking into account perceptual aspects and user interaction. Covering both the theoretical foundations and practical implementations of algorithms, this text provides a basis to understand and reproduce modern GPU-based visualization approaches.

VIII

Preface

This work constitutes my Habilitationsschrift, written at the University of Stuttgart. Being a research thesis, this Habilitationsschrift aims at describing visualization methods on a scientiﬁc level. Therefore, the intended audience includes researchers and students of computer science who are interested in interactive visualization methods. This book may serve as a starting point to delve into the current research in GPU-based visualization. It may also serve as reading material for a course that covers scientiﬁc visualization on an advanced undergraduate or a graduate level. This work also includes a discussion of practical issues such as how algorithms are mapped to GPU programs or performance characteristics of implementations. Therefore, practitioners and software developers might also ﬁnd this book interesting. The reader is expected to have a basic background in scientiﬁc visualization, but not in GPU-based visualization methods. In addition, some familiarity with GPU programming is recommended. Although this book does not cover these background topics in detail, it nevertheless contains some introductory material on the basics of visualization and GPU programming. In particular, a wealth of references is provided to guide the reader to background reading.

How to Read This Book This book is structured in a way that it can be read from cover to cover. However, you may also pick out some passages that are most interesting to you, and you may read through the book in non-sequential order. This section gives some hints on what topics are covered in which chapter and which parts of the book are built on other parts. It is recommended reading the introductory Chap. 1 because it describes basic concepts used to organize the book. In particular, the abstract visualization pipeline is discussed along with a classiﬁcation scheme for visualization methods (Sect. 1.1). Throughout this book, visualization techniques are related to the three main stages of the visualization pipeline, namely ﬁltering, mapping, and rendering. This chapter also covers fundamentals of GPUs (in Sect. 1.2) that can be skipped if you are already familiar with GPU programming. In addition, Sect. 1.3 presents methods and goals of this work. The main part of the book is organized in three large chapters on 3D scalar ﬁeld visualization (Chap. 2), vector ﬁeld visualization (Chap. 3), and perception-oriented and non-photorealistic rendering (Chap. 4). To a large extent, each of these chapters represents a portion of the book that can be read independently of the other main chapters. However, a few interdependencies are present as outlined below. Chapter 2 addresses methods for direct volume visualization of 3D scalar ﬁelds. It is recommended reading the basics of volume rendering as laid out in Sects. 2.1–2.3. Sections 2.1 and 2.2 describe the underlying optical model and the volume rendering pipeline. Section 2.3 discusses basic volume rendering

Preface

IX

methods, focusing on real-time GPU rendering. In particular, texture-based volume rendering (also called texture slicing) is described because it is the method of choice in this work. A multi-bricking approach for 3D texturebased rendering is introduced in Sect. 2.4. This approach maintains an approximately constant rendering performance for real-time applications. Sections 2.5 and 2.6 discuss advanced topics that could be read independently. Section 2.5 focuses on a number of new techniques for volume clipping that allow for complex clipping geometries. Clipping plays an important role in improving the perception of a 3D data set because it enables the user to explore otherwise hidden internal parts of the data set. Here, object-space and image-space clipping methods are compared, pre-integrated volume clipping is described, and issues of consistent volume shading are discussed. The visualization of very large, time-dependent volume data is addressed in Sect. 2.6. One element of this large-data approach is parallelization on a cluster computer with commodity-of-the-shelf GPUs. Another element is wavelet compression in combination with adaptive rendering. Chapter 2 concludes with a brief summary of the described volume rendering techniques. Chapter 3 discusses techniques for vector ﬁeld visualization, with the focus on texture-based methods. Section 3.1 is recommended as basis for this chapter because it presents the fundamental concept of particle tracing. Section 3.2 provides an overview and a classiﬁcation of vector ﬁeld visualization methods. This section contains an extensive list of references, serving as a good starting point to delve into state-of-the-art vector ﬁeld visualization. Section 3.3 continues with a more detailed discussion of texture-based vector ﬁeld visualization. It describes semi-Lagrangian texture advection and shows how advection can be used for dense, noise-based vector ﬁeld visualization and sparse, dye-based visualization alike. In part, this section relies on volume rendering techniques that are described in Chap. 2 (especially Sects. 2.1– 2.3). Section 3.3 is recommended as basis for the following sections of this chapter. These subsequent sections cover advanced topics and can be read independently of each other. A novel level-set advection scheme is introduced in Sect. 3.4 to overcome numerical diﬀusion that is inherent to semi-Lagrangian dye advection. While the methods discussed so far work in Cartesian 2D and 3D space, Sect. 3.5 addresses vector ﬁeld visualization on curved surfaces, for example, on the boundary surface of an automobile model enclosed by wind ﬂow. A hybrid object-space and image-space method is introduced to achieve an eﬃcient, yet accurate dense ﬂow representation. Section 3.6 describes a generic framework for the visualization of time-dependent vector ﬁelds. This framework comprises all relevant previous visualization methods and allows us to compare them on a mathematical basis. In addition, the ﬂexibility of the framework leads to the development of novel visualization approaches. The ﬁnal section of Chap. 3 summarizes the presented ﬂow visualization methods. Perception-oriented rendering and non-photorealistic rendering are discussed in Chap. 4. This chapter focuses on the third stage of the visualization pipeline – the rendering stage. It is recommended reading the brief

X

Preface

discussion of previous work in Sect. 4.1. Sections 4.2–4.5 can be read independently of each other because they cover diﬀerent aspects of perceptionoriented and non-photorealistic rendering. Section 4.2 addresses the inﬂuence of color on the visual perception of moving patterns. Based on an extensive review of psychophysics and psychology literature, a set of design guidelines is derived for eﬀective animated visualization. These guidelines are especially useful for texture-based ﬂow visualization; therefore, some background reading in Chap. 3 is recommended. Section 4.3 improves depth-perception by utilizing perception-oriented color cues. In particular, depth-dependent intensity and saturation modiﬁcations are employed. Section 4.4 introduces nonphotorealistic rendering methods that improve the perception of spatial structures in complex tone-shaded illustrations. A view-dependent transparency model is proposed in Sect. 4.4.2, whereas alternative cutaway methods are described in Sect. 4.4.3. These tone-shaded illustrations are tightly connected to non-photorealistic volume rendering and volume clipping. Therefore, some background reading in Chap. 2 is recommended. Non-photorealistic halftoning approaches are explicated in Sect. 4.5; here, the focal point is frame-to-frame coherent halftoning and a generic GPU-based concept for G-buﬀer operations. Frame-to-frame coherent halftoning relies on texture advection as a basic technique. Therefore, background material from Sect. 3.3 is useful for understanding Sect. 4.5. Chapter 4 ends with a summary of presented rendering methods. Following the main part outlined above, Chap. 5 concludes this book. This chapter classiﬁes the visualization methods discussed in this book and puts them in context. The appendix contains lists of ﬁgures, tables, and color plates, as well as a bibliography, an index, and color plates.

Acknowledgments I conducted the research for this book while I was with the Institute for Visualization and Interactive Systems (VIS) at the University of Stuttgart. I would like to thank all VIS members for the great time I had in Stuttgart and for many fruitful discussions. In particular, I thank Thomas Ertl for lots of support and advise; Joachim Diepstraten, Mike Eißele, and Sabine IserhardtBauer for having been great roommates; and Matthias Hopf, Guido Reina, Ulrike Ritzmann, and Simon “Grbic” Stegmaier for their help. I especially thank the following VIS members for successful collaborations on research that has become part of this work: Joachim Diepstraten [88, 89], Mike Eißele [108, 109], Klaus Engel [469, 470], Matthias Hopf [473, 478, 479], Stefan R¨ ottger [356], Martin Rotard [352], Tobias Schafhitzel [481], Magnus Strengert and Marcelo Magall´ on [402], as well as Manfred Weiler [484]. In addition, I had the pleasure to work with many VIS members on topics that, although not directly part of this book, have inﬂuenced my research in general: Ralf Botchen [27, 468], Joachim Diepstraten [87, 90], Mike

Preface

XI

Eißele [107, 216], Thomas Klein [216], Guido Reina [351], Dirc Rose [351], Martin Rotard [353], Tobias Schafhitzel [374, 480], Frederik Schramm [483], and Simon Stegmaier [107, 351]. Furthermore, I would like to thank the following collaborators outside VIS: Gordon Erlebacher [115, 471, 472, 473, 483], Stefan Guthe [356, 402], Bob Laramee and Helwig Hauser [240, 242], as well as Frits Post [240]. Finally, I thank Manfred Weiler for proof-reading Chaps. 1 and 2, and Nikolaus Weiskopf for help and discussions on various topics. Financial support from the following funding organizations has made my research possible: the “Landesstiftung Baden-W¨ urttemberg” for support within the program “Elitef¨ orderprogramm f¨ ur Postdoktoranden”; the “Deutsche Forschungsgemeinschaft” (DFG) for funding the projects D4 and D8 within SFB 382 and the project C5 within SFB 627; and the University of Stuttgart for ﬁnancing the project “i-modanim” within the program “self-study online”. I also would like to thank the BMW group and Roger Crawﬁs for providing data sets, Gerik Scheuermann for making available the image for Fig. 3.1 (d), and Martin Schulz for providing the image for Fig. 3.3. The following publishers have kindly provided the permission to use material for this book from papers (co-)authored by myself: A K Peters, Ltd. [475]; Elsevier/Academic Press [471]; the Eurographics Association [88, 89, 356, 402, 464, 481]; IEEE [108, 465, 469, 470, 472, 484]; UNION Agency/WSCG [476]; and the publisher of the SimVis 2004 proceedings (SCS Publishing House) [109]. I thank Thomas Ertl, Eduard Gr¨ oller, Kurt Rothermel, and Hans-Peter Seidel for reviewing my Habilitationsschrift. I also thank the Faculty 5 (Computer Science, Electrical Engineering, and Information Technology) at the University of Stuttgart for a very timely organization and pleasant handling of the Habilitation procedure. I am indebted to all people at Springer who supported this book project. In particular, I thank Martin Peters (Executive Editor, Mathematics, Computational Science, and Engineering) and the editors of the book series Mathematics and Visualization, Gerald Farin, Hans-Christian Hege, David Hoﬀman, Christopher R. Johnson, Konrad Polthier, and Martin Rumpf. I also thank Ute McCrory for organizing the publishing process. Very special thanks to Bettina A. Salzer for thorough proof-reading. This work would not have been possible without her help, patience, and love. Finally, I thank my parents Elisabeth and Gerhard for all their support.

Vancouver, April 2006

Daniel Weiskopf

Contents

1

2

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Visualization Pipeline and Classiﬁcation of Visualization Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 GPU Rendering Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 APIs and Eﬀect Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Methods and Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visualization of 3D Scalar Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Optical Model for Volume Rendering . . . . . . . . . . . . . . . . . . . . . . . 2.2 Volume Rendering Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Volume Rendering Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 3D Texture Slicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 2D Texture Slicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Ray Casting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.4 Shear-Warp Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.5 Splatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.6 Cell Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.7 Pre-Integrated Volume Rendering . . . . . . . . . . . . . . . . . . . 2.3.8 Variants and Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Maintaining Constant Frame Rates in 3D Texture-Based Volume Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Volume Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Depth-Based Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 Clipping Based on Volumetric Textures . . . . . . . . . . . . . . 2.5.3 Clipping and Pre-Integration . . . . . . . . . . . . . . . . . . . . . . . . 2.5.4 Clipping and Volume Illumination . . . . . . . . . . . . . . . . . . . 2.5.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Hierarchical Volume Visualization on GPU Clusters . . . . . . . . . . 2.6.1 Previous and Related Approaches to Large-Data Volume Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 4 5 8 8 11 13 15 17 18 20 21 25 26 27 30 32 33 38 39 47 52 55 63 65 66

XIV

Contents

2.6.2 Distributed Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.3 Accelerated Compositing Scheme . . . . . . . . . . . . . . . . . . . . 2.6.4 Hierarchical Compression and Adaptive Rendering . . . . . 2.6.5 Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

67 68 69 73 77

Vector Field Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.1 Basics of Particle Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.1.1 Time-Dependent Vector Fields . . . . . . . . . . . . . . . . . . . . . . 82 3.1.2 Lagrangian Particle Tracing . . . . . . . . . . . . . . . . . . . . . . . . 84 3.1.3 Eulerian Particle Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.2 Classiﬁcation and Overview of Flow Visualization Methods . . . 85 3.2.1 Point-Based Direct Flow Visualization . . . . . . . . . . . . . . . 86 3.2.2 Sparse Representations for Particle-Tracing Techniques . 87 3.2.3 Dense Representations for Particle-Tracing Methods . . . 92 3.2.4 Feature-Based Visualization Approaches . . . . . . . . . . . . . . 95 3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3.1 Semi-Lagrangian Transport Mechanism . . . . . . . . . . . . . . 98 3.3.2 GPU Implementation of Semi-Lagrangian Advection . . . 99 3.3.3 Lagrangian-Eulerian Advection . . . . . . . . . . . . . . . . . . . . . . 105 3.3.4 Particle Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 3.3.5 Visual Mapping and Rendering . . . . . . . . . . . . . . . . . . . . . . 109 3.4 Dye Advection Based on Level-Sets . . . . . . . . . . . . . . . . . . . . . . . . 116 3.4.1 Distance Field as Level-Set . . . . . . . . . . . . . . . . . . . . . . . . . 118 3.4.2 Reinitialization of the Level-Set . . . . . . . . . . . . . . . . . . . . . 119 3.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 3.4.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 3.4.5 Implementation and Results . . . . . . . . . . . . . . . . . . . . . . . . 126 3.5 Flow Visualization on Curved Surfaces . . . . . . . . . . . . . . . . . . . . . 128 3.5.1 Lagrangian Particle Tracing on Surfaces . . . . . . . . . . . . . . 129 3.5.2 Visual Mapping and Noise Injection . . . . . . . . . . . . . . . . . 134 3.5.3 Eﬀective Object and Flow Rendering . . . . . . . . . . . . . . . . 136 3.5.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 3.6 Spacetime Framework for Time-Dependent Vector Fields . . . . . 140 3.6.1 Continuous Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 3.6.2 Texture-Based Discretization . . . . . . . . . . . . . . . . . . . . . . . 143 3.6.3 Visualization Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 3.6.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 3.6.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Contents

XV

4

Perception-Oriented and Non-Photorealistic Rendering . . . . 161 4.1 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 4.2 Color and the Perception of Motion . . . . . . . . . . . . . . . . . . . . . . . . 163 4.2.1 Psychophysical and Physiological Research . . . . . . . . . . . 164 4.2.2 Design Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4.2.3 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 4.2.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 4.3 Color-Based Depth-Cueing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 4.3.1 Linear Transformations and Weighted Sum of Colors . . . 177 4.3.2 Appropriate Parameters for the Generic Depth-Cueing Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 4.3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 4.3.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 4.4 Non-Photorealistic Rendering of Depth Structures via Continuous Tone-Shading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 4.4.1 Traditional Visualization of Semi-Transparency in Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 4.4.2 Semi-Transparency in Computer-Generated Illustrations 194 4.4.3 Overview of Cutaway Illustrations . . . . . . . . . . . . . . . . . . . 201 4.4.4 Cutout Drawings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 4.4.5 Breakaway Illustrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 4.4.6 Implementation and Performance of Surface Rendering . 213 4.4.7 Extension to Volume Illustrations . . . . . . . . . . . . . . . . . . . 215 4.5 Non-Photorealistic Halftoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 4.5.1 The G-Buﬀer Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 4.5.2 Frame-to-Frame Coherent Halftoning . . . . . . . . . . . . . . . . 220 4.5.3 The G2 -Buﬀer Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 229 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

5

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Color Plates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

1 Introduction

Visualization has become an essential part in today’s engineering, research, and business workﬂows. The typical amount of data that originates from numerical simulations or sensor measurements is very large and, thus, visualization is indispensable for understanding this data. The ever increasing performance of supercomputers and resolution of scanning devices is a challenge that requires us to keep on developing improved visualization methods. This book addresses this challenge in manifold ways. Both the eﬃciency and eﬀectiveness of visualization methods are improved. High eﬃciency is achieved by using appropriate algorithms and data structures. This work speciﬁcally considers the usage of the fast graphics processing units (GPUs) of modern graphics hardware to obtain high performance – the performance gain can be as much as two to three orders of magnitude compared to a CPU (central processing unit) implementation. The eﬀectiveness of visualization methods is improved by taking into account perceptual aspects and an appropriate representation of the underlying data model. Eﬃciency and eﬀectiveness are not completely separated but interlinked: a highly eﬃcient visualization method is the prerequisite for an interactive application, which, in return, leads to signiﬁcantly improved eﬀectiveness. For example, an interactive exploration may promote spatial perception by motion parallax or result in a better understanding of the large parameter space that controls data generation and visualization. Particularly interesting is real-time visualization for time-dependent data. Increased eﬃciency can lead to a qualitatively improved eﬀectiveness because it might make the diﬀerence between a visualization that allows for eﬀective user interaction and a noninteractive ineﬃcient implementation. The use of GPUs for high-performance visualization is motivated by several aspects. First, GPUs typically have more transistors than CPUs. For example, an NVIDIA GeForce 6800 Ultra GPU comprises 222 million transistors, whereas an Intel Pentium 4 EE (Extreme Edition) CPU contains 29 million transistors for the processor core and 149 million transistors for the L2 (level 2) cache [419]. Second, the speed of GPUs and their number of transis-

2

1 Introduction

tors increase faster than described by Moore’s law for CPUs. Third, graphics hardware has wide data paths to its memory, and eﬃcient bilinear and trilinear reconstruction of texture data is available. Fourth, the programming model for GPUs is rather restricted and results in highly parallel execution and deep pipelining. Finally, a transfer of visualization results to the graphics board for ﬁnal display is superﬂuous for GPU-based implementations.

1.1 Visualization Pipeline and Classiﬁcation of Visualization Methods An abstract visualization pipeline is used throughout this book to structure the large variety of visualization methods. This pipeline approximately follows the description by Haber and McNabb [149] – up to some slight modiﬁcations. Figure 1.1 sketches the elements of the visualization pipeline. Input to the visualization pipeline is acquired from a data source, such as a numerical simulation, a measurement of physical data, or a database. This raw data is transformed by the ﬁltering stage of the visualization pipeline into abstract visualization data. Typical ﬁltering operations are denoising by convolution with a ﬁlter kernel or data enhancement by segmentation. The next stage is visualization mapping, which constructs a renderable representation from visualization data. The renderable representation has extension in space and time, and it contains attribute ﬁelds that may comprise geometry, color, transparency, reﬂectance, and surface texture. A typical example for a

databases

simulation

sensors

raw data

visualization pipeline

filtering visualization data

mapping renderable representation

rendering displayable image

interaction

Fig. 1.1. Abstract visualization pipeline

data acquisition

1.1 Visualization Pipeline and Classiﬁcation of Visualization Methods

3

mapping operation is the application of a color table that assigns colors to input data values. Finally, the rendering stage generates a displayable image from the renderable representation. Typical rendering operations include view transformations, scene illumination, and shading. For some visualization algorithms, it may be diﬃcult to ﬁnd a clear and unique assignment of algorithmic elements to the ﬁltering, mapping, or rendering stages. GPU-based methods, in particular, often require a reordering of the conceptual visualization pipeline to achieve an eﬃcient implementation. Examples for such a reordering are given for corresponding visualization methods when their details are discussed later in this work. Nevertheless, the concept of the visualization pipeline has proven to be useful in classifying and comparing diﬀerent visualization methods. For example, this scheme has been successfully used as a basis for teaching visualization courses (see the article by Rotard, Weiskopf, and Ertl [352]). Therefore, all visualization techniques of this book are analyzed by taking into account the concept of the visualization pipeline. As pointed out in Fig. 1.1, a user may potentially interact with all three stages of the visualization pipeline. Interaction plays a crucial role throughout this work because the large parameter space that controls data acquisition and visualization can only be explored interactively. As a prerequisite, visualization techniques need to be eﬃcient to allow for real-time application. In addition to the structure of the visualization pipeline, a more detailed taxonomy is useful for classifying visualization methods. Following Bergeron and Grinstein [17] and Brodlie [31], the dimensionality of independent variables (on the domain) and the dimensionality of dependent variables (i.e., data type) are important categories. Therefore, visualization methods are ordered according to their data type, starting with scalar data in Chap. 2. Vector ﬁelds, which are more complex than scalar ﬁelds, are discussed in Chap. 3. In addition, the dimensionality of the domain plays an important role in substructuring this chapter on vector ﬁeld visualization. Chapters 2 and 3 cover all steps of the visualization pipeline for the speciﬁc cases of scalar and vector ﬁelds, whereas Chap. 4 focuses on the rendering stage. This chapter discusses perception-oriented and non-photorealistic rendering methods that could be applied to scalar ﬁeld and vector ﬁeld visualization, but also to a much wider range of visualization applications. A third category that can be used to classify visualization methods is the type of data structure that holds the data set. The data structure is important from an algorithmic point of view and therefore inﬂuences the design of visualization algorithms. Throughout this book, visualization techniques for diﬀerent data structures are discussed. Usually, data is represented by a set of data points located in the domain space. The data structure forms a grid if a connectivity between data points is established. In contrast, scattered data does not have such a connectivity. We can distinguish two classes of grids: structured and unstructured grids. Structured grids have an implicitly given connectivity, whereas unstructured grids need their connectivity explic-

4

1 Introduction

itly stored. Examples of structured grids are Cartesian, uniform, rectilinear, or curvilinear grids. Cartesian, uniform, and rectilinear grids have all their cells aligned with the coordinate axes of the domain. A Cartesian grid has the same overall grid spacing regardless of direction. A uniform grid has an equidistant spacing along each of the directions, although the spacing is independently chosen for the diﬀerent directions. A rectilinear grid has varying cell sizes. A curvilinear grid exhibits the most ﬂexibility by allowing for deformed cells, provided the topological structure is still regular. A typical example of an unstructured grid is a grid consisting of simplical cells, i.e., a tetrahedral grid in the 3D case. More details on these basic grid types and other, more advanced grid structures are given in the book by Schroeder et al. [382].

1.2 GPU Rendering Pipeline This book discusses eﬃcient visualization algorithms for GPUs. The basic properties of GPUs, their programming model, and the underlying rendering pipeline are outlined in this section. The stages of the GPU rendering pipeline are only brieﬂy reviewed; detailed background information on the traditional ﬁxed-function pipeline can be found in a textbook by Foley et al. [122] or the OpenGL programming guide [496]. Additional information on using programmable graphics hardware is given in Cg manuals [118, 312] and documentations of the OpenGL shading language [207] and Direct3D [288], which is part of DirectX. Figure 1.2 sketches the GPU rendering pipeline. A 3D graphics application communicates with graphics hardware via a 3D API (application programming interface). Typically, either OpenGL or Direct3D are used as 3D API. 3D scenes are usually described by a boundary representation (BRep) formed by a collection of primitives. In most cases, primitives are triangles deﬁned by vertices. Vertex information consists of geometric position and further attributes such as normal vector, texture coordinates, color, or opacity. Primitives are sent from CPU to GPU as a stream of vertices, which are then transformed by the vertex processor. In the traditional ﬁxed-function graphics pipeline, vertex processing accomplishes transform and lighting operations. A vertex of the input geometry is transformed from its original object coordinates into subsequent world, eye, and normalized clip coordinates by multiplication with respective matrices for model, view, and normalization transformations. Matrix formulations of these aﬃne or projective transformations are adequate because all computations are based on homogeneous coordinates. Lighting is usually based on the Blinn-Phong model and performed with respect to eye coordinates. GPUs allow the programmer to replace these ﬁxed-function vertex operations by a ﬂexible vertex program (OpenGL lingo) or vertex shader (Direct3D lingo). Primitives are converted into fragments after vertex processing. Today’s GPUs exclusively process triangles at these stages of the pipeline; i.e., prim-

1.2 GPU Rendering Pipeline

5

3D Application

3D API CPU − GPU Boundary

GPU Front End

Primitive Assembly

Vertex Processor

Rasterization & Interpolation

Raster Operations

Frame Buffer

Fragment Processor

Fig. 1.2. GPU graphics pipeline (inspired by [312])

itive and triangle are synonymous in this context. During primitive assembly and rasterization the following operations are executed. Individual vertices of the vertex pipeline are organized into primitives, and primitives are clipped against the viewing frustum. At the transition from the vertex pipeline to the fragment pipeline, rasterization employs scanline conversion to generate the fragments covered by an input triangle. At the same time, vertex attributes are interpolated for fragments that lie inside the input triangle. Position coordinates are transformed from normalized clip space into normalized device coordinates by a homogeneous division by the w clip coordinate. Also, the viewport transformation to window coordinates is applied. Primitive assembly, rasterization, and interpolation are ﬁxed and cannot be programmed by the user. The subsequent fragment processor modiﬁes fragment attributes and applies textures. Texture lookup can make use of eﬃcient built-in bilinear or trilinear texture interpolation. Analogously to the vertex processor, fragment processing can either be used as a traditional ﬁxed-function element or freely speciﬁed in the form of a fragment program (OpenGL lingo) or pixel shader (Direct3D lingo). The ﬁnal raster operations combine the current fragment with pixel information given at the same position in the frame buﬀer: ﬁrst, the alpha, stencil, and depth tests (the latter is also called z test) are executed; then, if a fragment passes all tests, it may be directly written into the frame buﬀer or composited with frame buﬀer information by blending. 1.2.1 Programming Model GPU-based visualization methods utilize the programmability of the vertex processor and fragment processor stages. Typical operations for the vertex processor comprise the calculation of quantities that can later be interpolated

6

1 Introduction

linearly via scanline conversion. For example, texture coordinates can be generated to attach a 3D vector ﬁeld stored in a 3D texture to a surface that cuts through the vector ﬁeld. Much more important, however, is the fragment processor because it allows us to compute information on a per-fragment level. Moreover, an eﬃcient access to texture data is only possible at this stage. Therefore, the visualization methods of this book make extensive use of pixel shaders. Fragment processing can be considered as a streaming model of computation [38]. The streaming model extends the SIMD (single instruction, multiple data) concept that was used as abstraction of the OpenGL architecture [324], where each rendering pass implements a SIMD instruction that performs a basic arithmetic operation and updates the frame buﬀer atomically. The stream programming model represents computational locality that is not present in the SIMD model. The key elements are streams and kernels. A stream consists of a set of data elements for which similar computations are executed. A streaming processor applies a computing kernel to all elements of an input stream and writes the results into an output stream. Dally et al. [70] and Buck et al. [38] discuss how the stream model supports programs with high arithmetic and computational intensity (i.e., a large ratio of arithmetic operations to memory bandwidth). Figure 1.3 depicts the basic execution model of a GPU that applies to fragment and vertex processing alike. A graphical input element (e.g., a vertex or a fragment) is transmitted through read-only input registers into the shader program. The result of shader execution is written to output registers. The shader program has read and write access to temporary registers, and read-only access to textures and constants. Therefore, a shader can be regarded as a stream kernel that is applied to stream data held in input registers or textures. Please note that vertex programs used to have no access to texture; however, with Shader Model 3 compliant GPUs (e.g., NVIDIA GeForce 6 and GeForce 7 series), texture access is also available in vertex programs. Although the streaming model slightly restricts the broader functionality of GPUs, it is well-suited for most visualization methods of this book. Eﬃcient visualization techniques thrive on a good mapping to the streaming model. For example, such a mapping has to achieve a uniform kernel structure with a high degree of parallelism, no branching, and little or no conditional expressions. A regular data access to textures is highly advantageous to utilize optimized data paths to internal memory. If possible, eﬃcient built-in bilinear or trilinear interpolation should be exploited. The stream output often needs to be re-used as input to a subsequent stream computation. In this case, the output register is directly transferred into a texture by using the eﬃcient render-to-texture functionality of GPUs. A number of speciﬁc GPU properties should be taken into account. First, the accuracy of GPUs is usually limited and might vary along the rendering pipeline. For example, color channels in the frame buﬀer or in textures have a typical resolution of eight bits, whereas fragment processing may take place

1.2 GPU Rendering Pipeline

7

Input Registers

Textures

Shader Program

Constants

Temp Registers

Output Registers

Fig. 1.3. Programming model for current GPUs (after [38]). A shader program processes a single input element and writes the result to an output register

at higher precision. Even ﬂoating-point accuracy within textures and fragment processing, which is provided by modern GPUs, is not comparable to double-precision numbers, which are available on CPUs. Second, the number of instructions might be limited. Therefore, the algorithms have to be designed in a way that allows a concise implementation – sometimes at the cost of accuracy – or a less eﬃcient multi-pass rendering approach has to be pursued. Similarly, the number of indirection steps in fragment processing (i.e., dependent texture lookups) may be restricted. Third, an eﬃcient abortion of a fragment program (texkill command) is not possible on many GPUs. Therefore, the early z test is often (mis)used to emulate such a program abortion. Fourth, GPUs can simultaneously apply numerical operations to four diﬀerent color channels (red, green, blue, alpha) of a register, which can be used for an eﬃcient implementation. Fifth, a read-back of information from GPU to main memory should be avoided because the bandwidth from GPU to CPU can be signiﬁcantly smaller than the bandwidth from CPU to GPU. Finally, a simultaneous read and write access to textures is not speciﬁed; e.g., an in-place replacement of texel information is not feasible. This problem can be overcome by a ping-pong rendering scheme: two copies of a texture are held, one for the read access to “old” information (regular texture read) and the other one for writing “new” information (with a render-to-texture operation); after each computational step, both textures are swapped and their roles are exchanged, i.e., a ring buﬀer with two elements is employed. Although GPUs have a potential to increase visualization performance by more than one order of magnitude, an inappropriate implementation or algorithm could decrease the speed of a GPU version signiﬁcantly, compared to a reference CPU version. Therefore, the above issues and restrictions have to be carefully taken into account. Speciﬁc problems and their solutions are discussed throughout this work in the context of respective visualization algorithms.

8

1 Introduction

1.2.2 APIs and Eﬀect Files Another issue is the choice of API for GPU programming. High-level scene graph APIs, such as OpenInventor, OpenGL Performer, OpenSceneGraph, or OpenSG, are not considered in this book because they anyway employ lowlevel APIs for GPU conﬁguration and actual rendering. Therefore, only the widespread low-level APIs OpenGL and Direct3D are used in this work. GPU programs can be developed on assembler level or with high-level shading languages. Typical assembly-like programming environments are ARB vertex programs and ARB fragment programs for OpenGL and vertex shader and pixel shader programs for Direct3D. The advantage of assembler-level shaders is the direct control of shader instructions, which allows for handcoded optimization. The disadvantages are a time-consuming and error-prone shader development and program codes that are diﬃcult to read and maintain. These problems are overcome by high-level shading languages, such as the OpenGL shading language [207], Direct3D’s HLSL [288], or NVIDIA’s Cg [118, 312]. These high-level shading languages for GPUs are all very similarly structured, oriented along the C programming style, and strongly inﬂuenced by the Renderman shading language [152]. Old GPU programming concepts such as NVIDIA’s register combiners or texture shaders [210] do not play a role in this book because they are vendor-speciﬁc and inﬂexible. However, references to older publications based on such GPU programming models are sometimes included in the text. In addition to actual vertex and pixel shaders, states of the graphics pipeline strongly inﬂuence rendering results, i.e., GPU programming is more than shader development. Traditionally, diﬀerent parts of the program source code (for example, in the C++ code) are responsible for setting graphics states. Eﬀect ﬁles (FX ﬁles) were introduced in Direct3D 8 and further enhanced in Direct3D 9 [288] to simplify GPU programming. They encapsulate the whole setup and programming of the graphics pipeline within a so-called eﬀect. An eﬀect is described by a character string or an external text ﬁle and thus this concept is mainly referred to as eﬀect ﬁle. The source code responsible for the GPU programming is decoupled from the main program code, which deals with CPU programming. In this way, a better maintainability and readability of the source code for both parts is achieved.

1.3 Methods and Goals The goal of this book is the development of methods that improve both the eﬃciency and eﬀectiveness of visualization. Eﬃciency mainly concerns technical aspects of algorithms, data structures, and implementations, whereas eﬀectiveness is strongly aﬀected by perceptual issues related to the human observer and domain knowledge for a speciﬁc application.

1.3 Methods and Goals

9

A wide spectrum of subgoals requires an equally broad and interdisciplinary set of methods. Typical methods from practical computer science are concerned with the use of eﬃcient algorithms and memory-friendly data structures with fast access. Due to the restricted programming model of GPUs, speciﬁc algorithms that are eﬃciently supported by GPUs are developed. Algorithms are typically validated by running tests on their implementation, along with performance measurements. Actual implementation on GPUs is indispensable to verify these algorithms because GPU programming is subject to various subtle pitfalls. Moreover, visualization methods are discussed in the context of the generic visualization pipeline to show diﬀerences and analogies. Parallelization on distributed-memory architectures, such as cluster computers, is another computer-science approach used in this book. Mathematical methods are equally important to achieve eﬃcient visualization techniques. For example, appropriate numerical methods have to be employed for diﬀerential equations and level-sets, wavelets are used for multiresolution modeling and compression, and reconstruction and ﬁlter theory plays a signiﬁcant role in analyzing convolution-based vector ﬁeld visualization. The last major aspect takes into account human perception and cognition because visualization techniques have to target comprehension by a human observer. This book thrives on knowledge from ﬁelds like psychophysics, psychology, and neurophysiology to achieve perception-oriented rendering. Literature in these ﬁelds is analyzed to ﬁnd formalized methods that can be transferred into a computer program. Likewise, well-established experience from artists, designers, and illustrators is condensed into perception-guided and non-photorealistic rendering algorithms. The generic goals of eﬃciency and eﬀectiveness are pursued for speciﬁc but widely useful applications: the visualization of 3D scalar ﬁelds and vector ﬁelds in 2D and 3D. Interactivity plays a crucial role in all applications because it greatly enhances their eﬀectiveness. For example, interactivity allows a user to explore the large parameter space of visualization and data generation, it promotes spatial perception by motion parallax, and naturally represents temporal changes of time-dependent data. Real-time capable visualization is a requirement achieved by eﬃcient GPU techniques that accelerate the complete visualization pipeline.

2 Visualization of 3D Scalar Fields

Volume rendering is a widely used technique in many ﬁelds of application, ranging from direct volume visualization of scalar ﬁelds for engineering and sciences to medical imaging and ﬁnally to the realistic rendering of clouds or other gaseous phenomena in visual simulations and virtual environments. In recent years, texture-based volume rendering on consumer graphics hardware has become a popular approach for direct volume visualization. The performance and functionality of GPUs have been increasing rapidly, while their prices have been kept attractive even for low-cost PCs. Volume rendering targets the visualization of 3D scalar ﬁelds. A timedependent scalar ﬁeld, in general, is a map s : M × I −→ R , where M is an m-dimensional manifold with boundary and I ⊂ R is an open interval of real numbers. An element t ∈ I serves as a description for time, x ∈ M is a position in space, and the result of s yields the associated scalar value. For the more speciﬁc case of volume rendering, data is given on a ﬂat 3D manifold. A corresponding 3D time-dependent scalar ﬁeld is described by s : Ω × I −→ R ,

(x, t) −→ s(x, t) .

This scalar ﬁeld is deﬁned on the 3D Euclidean space Ω ⊂ R3 and depends on time t ∈ I. For a stationary 3D scalar ﬁeld, the time-dependency is obsolete: s : Ω −→ R ,

x −→ s(x) .

This chapter focuses on the visualization of scalar ﬁelds without explicit timedependency: either the input data set is already stationary or just a single time step of a time-dependent data set is considered at a time. The basic idea of direct volume visualization is to simultaneously show all important aspects of a data set in a single image. Semi-transparent rendering has to be applied to represent a full 3D data set on a 2D image plane.

12

2 Visualization of 3D Scalar Fields

Section 2.1 describes the prevalent optical model used for semi-transparent volume rendering. Essentially, viewing rays starting at the camera are traversed through the data set. Color contributions are accumulated while stepping along a viewing ray. This image synthesis process can be structured via the volume rendering pipeline, which consists of the stages data traversal, interpolation, gradient computation, classiﬁcation, shading, and compositing (see Sect. 2.2). The volume rendering pipeline can beneﬁt from direct GPU support. Section 2.3 describes basic volume rendering methods, focusing on real-time GPU rendering. To a large extent, this chapter addresses the visualization of data sets given on uniform grids. This type of grid has a natural representation via 3D textures or stacks of 2D textures. Both the scalar data set and its gradients can be held in these textures. An advantage of a GPU is that interpolation on textures is eﬃcient due to built-in hardware support. Section 2.3 also discusses alternative methods that support unstructured grids, especially tetrahedral grids. Furthermore, Sect. 2.3 shows how data traversal can be implemented using graphics hardware and how classiﬁcation, shading, and compositing can be handled by fragment processing. The stages of the volume rendering pipeline can be related to the steps of the more abstract visualization pipeline. The classiﬁcation stage can be identiﬁed as the mapping step, whereas shading and compositing can be considered as rendering step. Direct volume visualization exempliﬁes the fact that the visualization pipeline should be regarded as a conceptional pipeline. The actual implementation shows that a direct combination of the three steps may be more eﬃcient than a clear separation with a subsequent execution of the steps of the visualization pipeline. For volume rendering, the order of operations is signiﬁcantly modiﬁed – the roles of the outer processing of the visualization pipeline (the ﬁltering, mapping, and rendering steps) and the inner loop (stepping along a viewing ray) are exchanged. This chapter particularly focuses on two aspects of direct volume visualization. The ﬁrst one is volume clipping, which is discussed in detail in Sect. 2.5. Clipping provides a means of selecting regions of the data set on a purely geometric basis and plays a decisive role in understanding 3D volumetric data sets because it allows the user to cut away selected parts of the volume based on the position of voxels in the data set. Clipping reduces the domain of the data set to be visualized; therefore, clipping can be regarded as an example of a selection process (with respect to the domain), which is part of the ﬁltering step of the visualization pipeline. From an alternative point of view, one may associate clipping with the mapping step (in the form of a space-dependent mapping to opacities) or the rendering step (as a clipping of the renderable geometry). Clipping shows that the assignment to the diﬀerent steps of the visualization pipeline is not always unique. In this chapter, however, clipping is considered as part of the ﬁltering step. The second important aspect covered in this chapter is the eﬃcient rendering of large volume data sets. Section 2.6 addresses large-data visualization by applying compression techniques, adap-

2.1 Optical Model for Volume Rendering

13

tive rendering, and parallel processing on GPU clusters. A complementary multi-bricking approach leads to constant frame rates for volume rendering with 3D textures, independent of the viewing positions (see Sect. 2.4). Both volume clipping and large-data visualization rely on eﬃcient texture-based volume rendering as a basic technique (see Sect. 2.3).

2.1 Optical Model for Volume Rendering Direct volume visualization can be derived from optical models for volume rendering and volume shading. Here, a brief overview of a widely used approach is given. A detailed presentation of the underlying physical and mathematical models can be found in the review articles by Hege et al. [160] or Max [275]. This chapter follows the terminology used by Max. By ignoring scattering, the generic equation of transfer for light can be reduced to the simpler emission-absorption model or density-emitter model [362]. The respective volume rendering equation can be formulated as the diﬀerential equation dI(t) = g(t) − τ (t)I(t) . (2.1) dt The amount of radiative energy is described by radiance I(t), and its derivative with respect to the length parameter t is taken along the direction of light ﬂow. The source term g(t) describes the emission of light from the participating medium (and later, through reﬂection as well). The extinction coeﬃcient τ (t) deﬁnes the rate that light is attenuated by the medium. The optical properties are determined by a transfer function that assigns a value for the extinction coeﬃcient and the source term to each sample point of the data set. Radiance and optical properties are written as scalars, which is appropriate for luminance-based gray-scale images. A wavelength-dependent behavior or multiple wavelength bands (for example, red, green, and blue) can be taken into account by computing these quantities for each wavelength separately. The volume rendering equation (2.1) can be integrated to compute the radiance that leaves the volume and ﬁnally arrives at the eye point. Integrating from the initial point t = t0 to the end point t = D along the light direction results in D D D − τ (t ) dt − τ (t ) dt + g(t) e t dt . I(D) = I0 e t0 t0

The background light that enters the volume at the position t = t0 is denoted by I0 . The radiance that leaves the volume at t = D is described by I(D). With the deﬁnition of transparency, −

T (t) = e

D t

τ (t ) dt

,

14

2 Visualization of 3D Scalar Fields

one obtains the volume rendering integral D I(D) = I0 T (t0 ) +

g(t)T (t) dt .

(2.2)

t0

So far, the optical model is restricted to emission and absorption. The visual realism of volume rendering can be substantially increased by considering scattering of light. A simple and widely used volume illumination model implements single scattering by assuming that light from an external light source unimpededly reaches the scattering location. Light attenuation is neglected on the way from the light source to the scattering point, which is similar to neglecting shadowing in surface graphics. The result of single scattering can be computed similarly to local illumination models known from surface rendering, such as the Phong or the Blinn-Phong models. The gradient of the scalar ﬁeld is used as the normal vector in local illumination because the gradient is identical to the normal vector of an isosurface. In this way, volume illumination essentially imitates the eﬀect of lighting isosurfaces. The source term of the volume rendering integral (2.2) can be extended to include local illumination: g(x, ω) = E(x) + S(x, ω) . (2.3) Here, the source term is given with respect to position x, which directly corresponds to the length parameter t along the light ray. The non-directional emissivity E(x) is identical to the source term in the pure emission-absorption model. The additional scattering term S(x, ω) depends on the position x, the direction of the reﬂected light, ω, and the properties of the light source. The integral in (2.2) is typically approximated by a Riemann sum over n equidistant segments of length ∆x = (D − t0 )/n. The approximation yields I(D) ≈ I0

n i=1

where

ti +

n i=1

gi

n

ti ,

(2.4)

j=i+1

ti = e−τ (i∆x+t0 )∆x

(2.5)

is the transparency of the ith segment and gi = g(i∆x + t0 )∆x

(2.6)

is the source term for the ith segment. Note that both the discretized transparency ti and source term gi depend on the segment length ∆x. Opacities αi = 1 − ti are often used instead of transparencies ti . When the RGB color model is applied, the source term gi is described by its RGB values. The emissive RGB part is denoted Ci . Therefore, source and opacity terms can be combined in the form of an RGBA contribution. These terms have to be adapted if the sampling distance ∆x is changed.

2.2 Volume Rendering Pipeline

15

Although the emission-absorption model is predominant in real-time volume graphics, it should be pointed out that there exist more advanced rendering algorithms that strive for a higher degree of realism by including further elements of global illumination. Background information on the physical models for advanced global illumination can be found in the physics literature, for example, in a book by Chrandrasekhar [53]. Light transport can be regarded as a special case of a generic transport mechanism described by the Boltzmann equation. Mathematical details of the Boltzmann equation are discussed, for example, by Duderstadt and Martin [100] or Case and Zweifel [47]. Transport theory can be directly applied to volume visualization, as shown by Krueger [230, 231], and to image synthesis in general, as shown by Arvo and Kirk [4]. There are many previous papers on advanced global illumination specifically designed for participating volumetric media. Due to their complexity, most of these global illumination algorithms are implemented on CPUs and do not facilitate interactive rendering. Some examples are volume photon mapping [186, 95], the zonal method for radiosity [360], the use of spherical harmonics [199], or a method for scattering in layered surfaces [151]. Global illumination and physics-based light transport for computer graphics in general are presented in both breadth and detail in textbooks by Pharr and Humphreys [333] and Dutr´e et al. [103]. The overview by Cerezo et al. [51] focuses on state-of-the-art rendering techniques for participating media.

2.2 Volume Rendering Pipeline The evaluation of the emission-absorption model for a given data set can be split into several subsequent stages of the volume rendering pipeline. This section reviews the volume rendering pipeline only brieﬂy; a more detailed description is given by Pﬁster [330]. The following stages are commonly found in volume rendering techniques: (a) data traversal, (b) interpolation, (c) gradient computation, (d) classiﬁcation, (e) shading, and (f) compositing. During data traversal, resampling positions are chosen throughout the volume. Some kind of interpolation scheme is applied at these positions to reconstruct the data set at locations that diﬀer from grid points. Typical ﬁlters are nearest-neighbor interpolation or variations of linear ﬁltering. Trilinear interpolation is most common for uniform grids and is also used in most of the methods presented in this book. The gradient of a discretized volumetric data set is typically approximated by using discrete gradient ﬁlters. Many gradient ﬁlters are extensions of 2D edge ﬁlters from image processing, e.g., the Sobel operator. Central diﬀerences are a popular way of computing the partial derivatives for the gradient because they require only a small number of numerical operations. Classiﬁcation maps properties of the data set to optical properties for the volume rendering integral (2.2), i.e., it represents the relationship between

16

2 Visualization of 3D Scalar Fields

data and the source term g and the extinction coeﬃcient τ . The classiﬁcation typically assigns the discretized optical properties Ci and αi , which are combined as RGBA values. Pre-classiﬁcation ﬁrst maps grid points of the data set to optical properties and afterwards applies the interpolation scheme to the RGBA values. Post-classiﬁcation, in contrast, ﬁrst evaluates the interpolation scheme for the input data set and then assigns the corresponding optical properties. The mapping to optical properties is represented by a transfer function. The transfer function typically depends on the scalar value s, which is given at position x by the term s(x). This leads to a slightly modiﬁed description of the source term via g˜: g(x) = g˜(s(x)) . Similarly, the extinction coeﬃcient can be written as τ˜ according to τ (x) = τ˜(s(x)) . An analogous description can be applied for the RGBA transfer function used in the discrete approach. Additional parameters are often included in transfer functions, e.g., gradient magnitude or even higher-order derivatives [212, 217] in multi-dimensional transfer functions. Volume shading can be incorporated into transfer functions by adding an illumination term, as shown in (2.3). The design of useful transfer functions, in general, is a crucial aspect of volume visualization that attracts a lot of attention [331]. The discretized volume rendering integral (2.4) can be iteratively computed by compositing. Two types of compositing schemes are most common: front-to-back and back-to-front compositing. Front-to-back compositing is used for stepping along viewing rays from the eye point into the volume. The front-to-back iteration equations are Cdst ← Cdst + (1 − αdst )Csrc , αdst ← αdst + (1 − αdst )αsrc . Color is accumulated in Cdst and opacity is accumulated in αdst . The subscript dst is an abbreviation for “destination”, denoting a variable that can be modiﬁed by an update operation. The transfer function assigns the color contribution Csrc and opacity αsrc as a representation of the optical properties at the current location on the ray. The subscript src is an abbreviation for “source”. The same terminology of “source” and “destination” is used by OpenGL in the context of frame-buﬀer blending. In fact, the front-to-back iteration equation can be implemented by blending, as discussed in Sect. 2.3. The values Cdst and αdst are initialized with zero before ray traversal. The iteration ends once the ray has left the volume. Then, background light can be added, corresponding to the term I0 in (2.2) and (2.4). Finally, Cdst contains the color that is transported to the eye.

2.3 Volume Rendering Approaches

17

Back-to-front compositing is an alternative scheme. Here, a viewing ray is traversed from the backside of the volume to the eye. The corresponding iteration equation is Cdst ← (1 − αsrc )Cdst + Csrc . Again, color is accumulated in Cdst , and Csrc and opacity αsrc are assigned by the transfer function. Background light is used to initialize Cdst . Note that opacity does not need to be accumulated to compute the ﬁnal color. In addition to these compositing schemes, a number of alternative approaches might be used. For example, maximum intensity projection (MIP) is useful in some medical imaging applications. Sometimes, the simpliﬁed emission-only or absorption-only models are applied, which only consider the emission term or the absorption term, respectively. The MIP, emission-only, and absorption-only schemes share the advantage of being order-independent: their compositing equations are commutative and therefore do not require any particular traversal order. Thus, they do not need any kind of spatial sorting. In contrast, the emission-absorption model is order-dependent and requires spatial sorting, e.g., according to a front-to-back or back-to-front scheme. In what follows, this book focuses on the compositing schemes for the emissionabsorption model. Although the volume rendering pipeline provides the usual way of describing the structure of a volume rendering algorithm, the stages of this pipeline can be identiﬁed with steps of the generic visualization pipeline: classiﬁcation corresponds to the mapping step, whereas shading and compositing implement the rendering step. The other stages (data traversal, interpolation, and gradient computation) deal with internal data handling and serve as input for the mapping and rendering steps. Since these data handling stages do not provide any non-trivial ﬁltering element, they are not considered as part of the ﬁlter step of the visualization pipeline, but are rather included in the respective mapping and rendering steps.

2.3 Volume Rendering Approaches Several ways of implementing the generic volume rendering pipeline are discussed in this section. Volume rendering methods primarily diﬀer in the way the volume data set is traversed. In particular, methods can be classiﬁed either as image-order approaches or as object-order approaches. Object-order methods traverse the 3D volume in its object space and project the volumetric information onto the image plane. Here, this book focuses on 2D and 3D texture slicing, which are popular object-order approaches directly supported by graphics hardware (see Sects. 2.3.1 and 2.3.2). In contrast, image-order methods use the 2D image plane as starting point for data traversal: the image is scanned and the result of volume rendering is

18

2 Visualization of 3D Scalar Fields

computed for each pixel on the image plane. Ray casting (Sect. 2.3.3) is the most prominent example of image-order volume rendering. There is a large body of previous work on CPU-based ray casting, but this section primarily focuses on the recent developments in GPU-based ray casting. The subsequent sections cover alternative object-order methods: shearwarp factorization in Sect. 2.3.4, splatting in Sect. 2.3.5, and cell projection in Sect. 2.3.6. Furthermore, pre-integration is discussed in Sect. 2.3.7 as a means of improving most of the previously mentioned rendering methods. Finally, Sect. 2.3.8 brieﬂy presents some variations and extensions of volume rendering methods. In general, this book focuses on visualization methods for Cartesian or uniform grids. Therefore, volume rendering techniques for these types of grids play a dominant role in this chapter. Nevertheless, alternative methods for other grid structures, especially tetrahedral grids, are also discussed. The reader is referred to the following detailed descriptions for information on supported grid types. 2.3.1 3D Texture Slicing Texture slicing is a prominent example of an object-order approach for GPU volume rendering. The speciﬁc example of 3D texture slicing – also called 3D texture-based volume rendering – makes use of image-aligned slices [1, 41, 68]. These view-aligned slices can be considered as 2D proxy geometry for the actual 3D volume that has to be rendered. The data set itself is stored in a 3D texture that represents volume data given on a Cartesian or uniform grid. An advantage of 3D texture slicing is its good support by graphics hardware, which leads to eﬃcient volume rendering. The stages of the volume rendering pipeline are now described in the context of 3D texture slicing. For data traversal, view-aligned planar polygons are rendered. Rasterization generates fragments on the image-aligned slices, producing sampling positions throughout the volume. Figure 2.1 illustrates the layout of the viewaligned slices. The view-dependent slice polygons are generated by the CPU and clipped against the bounding box of the volume. Sampling locations in the volume are addressed by 3D texture coordinates that are attached to the vertices of the slice polygons. Trilinear interpolation is a built-in feature of 3D texture mapping on GPUs, i.e., the interpolation scheme is directly and eﬃciently supported. Gradients can be pre-computed for each grid point of the data set and stored along with the scalar data. For example, the RGB channels can be used for the gradient and the alpha channel for the scalar value [485]. Data sets with pre-computed gradients need four times the memory of the original scalar data, which can be a problem because texture memory is a limited resource on most, if not all, GPUs. Alternatively, gradients can be computed on-theﬂy during rasterization. Here, only scalar data has to be stored in texture

2.3 Volume Rendering Approaches

19

eye

image plane virtual rays

Fig. 2.1. 3D texture slicing. View-aligned slices are rendered, leading to sampling positions marked by dots. The dashed rays indicate onto which pixels the volume samples are projected

memory, but the evaluation of the gradient needs more numerical operations and texture accesses than the approach of pre-computed gradients. Post-classiﬁcation is implemented by a transfer function in the form of a color table represented by a texture. During rasterization, the interpolated scalar value is interpreted as the texture coordinate for a dependent texture lookup in the transfer function. The dimensionality of the transfer function determines the dimensionality of the dependent texture, i.e., multi-dimensional transfer functions with up to three diﬀerent parameters are directly supported. Pre-classiﬁcation can be implemented by paletted textures, where the index that maps to the color palette is stored in the data texture and the color palette itself represents a 1D transfer function. The shading stage is wellsupported by programmable GPUs because typical illumination models such as the Blinn-Phong model can be evaluated on a per-fragment level if the gradient information is available. Even on older graphics hardware, however, volume shading is possible. For example, Westermann and Ertl [485] introduced ambient and diﬀuse illumination for texture-based isosurfaces. This approach could be extended to semi-transparent volume rendering [284, 285]. Compositing is typically implemented by alpha blending with the frame buﬀer. Both front-to-back and back-to-front strategies with their respective compositing equations are supported by alpha blending. Accumulated opacities have to be stored for front-to-back compositing, i.e., an RGBA frame buﬀer is required. Current graphics hardware is often restricted to 8-bit resolution for the RGBA channels of the frame buﬀer. Quantization artifacts can therefore occur when opacity is close to zero; see Bitter et al. [22] for a thorough discussion of quantization eﬀects. Texture formats with 16-bit or 32-bit resolution can be used to overcome this accuracy problem [110]; here, compositing is performed by ping-pong rendering with these high-resolution textures.

20

2 Visualization of 3D Scalar Fields

2.3.2 2D Texture Slicing 2D texture slicing is an object-order approach tightly related to 3D texture slicing. The main diﬀerence is that 2D slicing employs object-aligned slices instead of view-aligned slices. The scalar data set is stored in stacks of 2D textures that ﬁll the entire bounding box of the volume. Three stacks of textures are required, one for each of the main coordinate axes. Since 2D slicing is closely related to 3D slicing, only the diﬀerences between both approaches are mentioned here. Data traversal is based on rendering slice polygons. Here, however, the polygons are parallel to one of the faces of the cube-shaped volume data set. Figure 2.2 illustrates the layout of those axis-aligned slices. One of the three stacks of 2D textures is chosen as input for the rendering process. The stack direction is determined by minimizing the angle between viewing direction and processing axis. Similarly to data traversal, the interpolation stage is aﬀected by changing from 3D to 2D slicing. Only bilinear interpolation is applied as the object-aligned slice polygons have a one-to-one correspondence to respective 2D textures of the data set. Due to more coherent memory access and fewer numerical operations, bilinear interpolation is faster than trilinear interpolation, i.e., 2D slicing is usually faster than 3D slicing. However, trilinear interpolation is possible on additional slices [340] by applying multi-texturing. Similarly, additional copies for three texture stacks can be avoided at the cost of decreased rendering performance [340]. All other stages (gradient computation, classiﬁcation, shading, compositing) are analogous to 3D slicing. In conclusion, 3D texture slicing oﬀers some advantages compared to the alternative of using stacks of 2D textures. First, only one third of the texture memory is required for the 3D texture method; the 3D approach just holds a single instance of the volume, whereas the 2D approach has to store the complete volume for each of the three main axes. Second, view-aligned slicing through a 3D texture avoids the artifacts that occur when texture stacks are switched. Third, the intrinsic trilinear interpolation in 3D textures directly

eye

image plane virtual rays

Fig. 2.2. 2D texture slicing. Axis-aligned slices are rendered, leading to sampling positions marked by dots. The dashed rays indicate onto which pixels the volume samples are projected

2.3 Volume Rendering Approaches

21

allows for an arbitrary number of slices with an appropriate resampling on these slices, i.e., the quality of the rendering can be easily adjusted by adapting the slice distance. The main advantage of 2D slicing is its support by almost any graphics hardware and its higher rendering performance. Most of the rendering techniques in later parts of this book are compatible with 2D and 3D slicing alike. All other cases are speciﬁcally noted. 2.3.3 Ray Casting Ray casting is an image-order approach for volume rendering. The starting point is the image plane, which is scanned in a pixel-by-pixel fashion. Typically, a viewing ray is traversed from the eye point through each pixel into the volume. For improved image quality, supersampling on the image plane leads to more than a single ray per pixel. While a ray is being traversed, the volume rendering integral is evaluated for that ray. The natural compositing order is front-to-back, i.e., from the eye into the volume. Figure 2.3 illustrates the principle of ray casting. Ray casting can be regarded as the volume-rendering analogue of traditional ray tracing for surface-based geometry according to Whitted [489]. One important diﬀerence is that ray casting does not spawn secondary rays like ray tracing does. Ray casting is often used for data given on uniform or Cartesian grids. Here, it is common practice to employ an equidistant sampling along the viewing rays, as shown in Fig. 2.3. Equidistant sampling leads to the Riemann sum approximation (2.4) of the volume rendering integral. An interpolation ﬁlter is applied during sampling because the sampling positions usually diﬀer from grid locations. Trilinear interpolation is the most popular reconstruction ﬁlter for uniform grids. The ray casting algorithm can be described by the pseudo code from Fig. 2.4. Data traversal is split in two parts. In the ﬁrst part – ray setup – the initial sample position along the ray is determined by computing the

eye

image plane rays

Fig. 2.3. Ray casting idea. Rays are traversed from the eye into the volume. The volume is sampled at positions along the rays (indicated by dots) to compute the volume rendering integral

22

2 Visualization of 3D Scalar Fields Ray setup (entry position and ray direction) Traversal loop (while ray in volume) Interpolate data value at current position Perform classification and compositing Propagate position along ray End loop Fig. 2.4. Ray casting: pseudo code

ﬁrst intersection between the viewing ray and the data volume. Ray setup also calculates the other parameters of the viewing ray, e.g., in the form of a root point at the eye and the viewing direction. The second part of data traversal is the actual traversal loop. Here, positions are sampled along the ray by iteratively advancing them along the ray direction. In addition, it has to be checked whether the sample position is still inside the volume. Ray traversal has to be stopped once the ray exits the volume, i.e., a mechanism for ray termination has to be included. In addition to data traversal, Fig. 2.4 explicitly includes three other stages of the generic volume rendering pipeline from Sect. 2.2: interpolation, classiﬁcation, and compositing. The stages gradient computation and shading are optional and can be added as additional operations within the traversal loop. Ray casting can use the same approaches for classiﬁcation, gradient computation, and shading as used for texture slicing (see Sect. 2.3.1). The implementation of compositing is described below. Ray casting is a very well-known approach for CPU-based volume rendering. It has been used early on, dating back to the 1980s, as it can be seen in the discussion by Levoy [245]. For an overview of the large body of earlier research on CPU-based ray casting, the reader is referred to the survey chapter by Kaufman and Mueller [201]. Whereas CPU ray casting is a well-established approach, its GPU counterpart is rather new: GPU ray casting for the visualization of uniform grids was introduced by R¨ ottger, Guthe, Weiskopf, Ertl, and Straßer [356], followed by a variant by Kr¨ uger and Westermann [232]. In general, GPU ray casting requires advanced fragment shader functionality that has only recently become available. The basic idea is to map the processing steps from Fig. 2.4 to a fragment program. GPUs that support loops and branching in fragment programs are most suitable for GPU ray casting. For example, Shader Model 3 compliant GPUs such as the NVIDIA GeForce 6 and GeForce 7 series or the ATI X1900 series provide this kind of shader functionality. The implementation by Stegmaier et al. [398] follows such a direct mapping of Fig. 2.4 to a fragment program. Their implementation just needs a single render pass: fragments are generated by rendering the front faces of the volume; the code from Fig. 2.4 is implemented as an assembler-level fragment program. An important element of the fragment program is the branching

2.3 Volume Rendering Approaches

23

instruction that realizes the ray-termination mechanism. A 3D texture serves as data structure that holds the data set, and graphics hardware supports trilinear interpolation within that 3D texture. Front-to-back compositing is implemented by updating the accumulated color and opacity values held in a ﬂoating-point fragment-program variable (register). Therefore, no blending with frame buﬀer contents is required, and compositing is always performed with high ﬂoating-point quality. The source code of the implementation can be downloaded from Stegmaier et al.’s web page (see [398]). The earlier approaches [232, 356] are designed for Shader Model 2 GPUs, without relying on the availability of branching and loops in fragment programs. This lack of functionality is overcome by multi-pass rendering that essentially splits the ray setup and the iterations of the traversal loop in several render passes. Ray termination makes use of the early z test (see [232] for details) or occlusion query in combination with the early z test (see [356] for details). Compositing is implemented by blending with accumulated values stored in the frame buﬀer. In general, ray casting has the advantage of directly supporting a variety of acceleration techniques. The overview by Kaufman and Mueller [201] provides references to CPU-based acceleration methods. Many of these approaches can be mapped to GPU ray casting. A prominent example is early ray termination: ray traversal can be stopped once a high level of opacity is accumulated in front-to-back volume rendering because contributions to the volume rendering integral from further away regions are negligible. The GPU implementation of early ray termination essentially requires an additional test in the ray-termination mechanism to check the current value of the accumulated opacity. For example, this extension can be added to the branching operation in Shader Model 3 GPUs [398]. Alternatively, the early z test can be used to eﬃciently break out of fragment processing in multi-pass ray casting [232, 356]. Another example is empty-space skipping (also called empty-space leaping), which eﬃciently passes over completely transparent regions. Two diﬀerent GPU approaches for empty-space skipping are proposed by Kr¨ uger and Westermann [232] and Stegmaier et al. [398]. Adaptive sampling can be regarded as a generalization of empty-space skipping, modifying the sampling rate along a viewing ray according to the properties of the data set or the transfer function. For example, a larger step size can be used in almost uniform data regions, which subsumes empty-space skipping as a special case. R¨ottger, Guthe, Weiskopf, Ertl, and Straßer [356] discuss a GPU approach for adaptive sampling. GPU ray casting for uniform grids and 3D texture slicing rely on the same scalar ﬁeld representation: a 3D texture. In addition, both techniques apply a fragment-oriented point sampling of the data set to evaluate the volume rendering integral. Therefore, almost all of the rendering techniques in later parts of this book that are designed for 3D slicing can also be used with GPU ray casting. Ray casting and 3D slicing can be regarded as basic volume rendering techniques in this book.

24

2 Visualization of 3D Scalar Fields

eye

image plane rays

Fig. 2.5. Ray casting for tetrahedral grids. Rays are traversed from the eye into the volume. The volume is sampled at the intersections of rays and cell boundaries (indicated by dots)

So far, ray casting has only be considered for uniform grids. However, unstructured grids can be useful in many simulation applications because their resolution can be adapted to the required accuracy of the numerical computation. For example, computational ﬂuid dynamics (CFD) is often based on unstructured grids. In general, an unstructured grid may consist of different types of cells such as tetrahedra, hexahedra, pyramids, or prisms. By decomposing complex cells into tetrahedra, an unstructured grid can always be transformed to a tetrahedral grid. Therefore, a tetrahedral grid is the most important example of an unstructured grid. The above ray casting algorithm needs only slight modiﬁcations to facilitate tetrahedral grids. Garrity [133] describes a corresponding CPU approach, illustrated in Fig. 2.5. A viewing ray is traversed in a front-to-back fashion from cell to cell, following the faces between adjacent cells. Again, ray setup determines the ﬁrst intersection with the data set in order to initialize the viewing ray. In other words, the main diﬀerence between ray casting for uniform grids and tetrahedral grids is the choice of sampling positions along rays. Weiler et al. [459] were the ﬁrst to propose tetrahedral grid volume rendering on GPUs, adopting Garrity’s algorithm. Designed for Shader Model 2 GPUs, Weiler et al.’s implementation performs the ray traversal in multiple render passes. A fragment program is used to compute intersections between rays and tetrahedral cells in order to ﬁnd sampling positions. In each render pass, the color and opacity contribution of the current cell is computed analogously to the GPU-based view-independent cell projection by Weiler et al. [460]. The basic idea is to pre-compute the volume rendering integral for the ray segment within a cell and to use this information during volume rendering. Section 2.3.7 discusses this pre-integration approach for tetrahedral cells [357] in more detail. One issue of the method by Weiler et al. [459] is its restriction to convex meshes. Their method does not support a re-entry of a viewing ray into the volume. This problem can be overcome by convexiﬁcation: a non-convex

2.3 Volume Rendering Approaches

25

mesh is transformed into a convex mesh by ﬁlling the empty space between the boundary of the mesh and its convex hull with imaginary cells. For example, the convexiﬁcation approach by Williams [490] can be employed. An alternative approach [18, 461] applies a technique similar to depth-peeling [116] to facilitate non-convex meshes. Depth-peeling decomposes the tetrahedral mesh into several depth layers each of which can be processed by the original algorithm for convex meshes. Weiler et al. [461] also include the compression of tetrahedral strips to accommodate large data sets in GPU memory. More background information can be found in the survey article by Silva et al. [394], which discusses GPU methods for tetrahedral meshes in general. 2.3.4 Shear-Warp Factorization Shear-warp volume rendering [97, 235] is another object-space approach, tightly related to 2D texture slicing. The order of volume traversal is identical: an axis-aligned stack of slices is processed in a slice-by-slice order. The main diﬀerence is that slices are not directly projected onto the ﬁnal image plane, but onto the base plane, which is an intermediate image plane aligned with two of the main axes of the data set. The simplest case involves an orthographic camera. Here, the volume grid is sheared before the projection so that the projection can be performed along a direction perpendicular to the base plane. In this way, the projection of a slice can be reduced to a resampling of a 2D image. All slices are combined via compositing (in front-to-back or backto-front order), similarly to 2D texture slicing. In the ﬁnal step, the image of the base plane is warped to the ﬁnal image plane. The warp operation is performed only once per frame. The name of this volume rendering approach is derived from a factorization of the view transformation in a shearing part (applied to the 3D volume) and a subsequent warp part (applied to the base plane). In summary, shear-warp rendering and 2D texture slicing diﬀer in the order of executing compositing and warping: 2D texture slicing warps each slice and performs compositing on the ﬁnal image, whereas the shear-warp approach ﬁrst applies the compositing scheme and only warps the intermediate image. An advantage of shear-warp volume rendering is its high eﬃciency provided by an optimized CPU implementation. As described by Lacroute and Levoy [235], run-length encoding of the data set and early ray termination can be incorporated. It should be noted that the implementation by Lacroute and Levoy also contains some aspects of an image-space method because early ray termination is evaluated based on image-space information. Unfortunately, the optimizations for shear-warp rendering lead to non-uniform patterns in accessing memory, which is unfavorable for a GPU implementation. Another issue is that perspective projection makes the rendering algorithm more complex and introduces additional computations, as discussed by Schulze et al. [386]. Therefore, shear-warp factorization has not yet played a role in GPU volume rendering.

26

2 Visualization of 3D Scalar Fields

2.3.5 Splatting Splatting is an alternative object-order approach: it traverses the data set in 3D object space and projects volumetric elements, e.g., cells or voxels, onto the image plane. A key element is to project 3D reconstruction kernels onto the 2D image plane. Westover [488] uses the term splat to describe this ﬂattening of the kernel on the image plane. Footprint is an alternative term for the projected image of a kernel. For splatting to work, the data set has to be expressed in terms of 3D kernels. The scalar ﬁeld needs to be written as si h(x − xi ) , s(x) = i

where h is the 3D kernel function, xi are center points for the kernel, and si are corresponding weights. The index i serves as label for copies of the 3D kernel. The basic objective of volume rendering is the eﬃcient evaluation of the volume rendering integral. Let us start with a slightly simpliﬁed integration of s along ray positions x(t) parameterized by t: si h(x(t) − xi ) dt = si h(x(t) − xi ) dt . s(x(t)) dt = i

i

This equation allows us to separate the data representation si from the kernel representation h. The integration along a ray reduces the dimensionality of the 3D kernel function to its 2D projection. The footprint of the 3D kernel can be expressed as footprint(u, v) = h(x(t)) dt , where the 2D coordinates (u, v) are computed by projection along the rays: (u, v) = project(x) . Note that, for a single ray, all positions x(t) are projected onto the same image location (u, v). By using the footprint function, the overall integration can be rewritten as s(x(t)) dt = si footprint(project(x(t) − xi )) . (2.7) i

Therefore, the integral is the sum of translated copies (translation due to xi ) of the footprint function, weighted by the data values si . The idea of splatting is to pre-compute the footprint function and rely on the pre-computed images during rendering. Graphics hardware can be used for volume rendering by accessing the footprint stored in a 2D texture. Typically, spherically symmetric kernels are used because their footprint function is

2.3 Volume Rendering Approaches

27

view-independent, which greatly reduces the amount of pre-computation. The actual volume rendering integral is slightly more complex than the prototypical integral in (2.7). However, the basic idea of pre-computing the footprint of the integration works in the same way as before. More background information on footprint computation can be found in the survey chapter [64]. In general, splatting is quite ﬂexible in the order of traversing the volume, i.e., in choosing the positions xi . Often, splatting is applied to traverse a uniform grid in a voxel-by-voxel order. It can also be used for scattered data and is especially eﬃcient for sparse data. The only requirement is a correct spatial sorting in order to apply back-to-front or front-to-back compositing. An issue with splatting occurs when 3D kernels overlap in 3D space. Here, a separation of the footprint computation and the summation similar to (2.7) is no longer valid, leading to color bleeding and popping artifacts. Image-aligned sheet-based splatting by Mueller and Crawﬁs [294] oﬀers a solution for this problem. Their approach uses image-aligned sheets (i.e., slicing slabs) to clip the 3D kernels outside a thin slab. In this way, the overlap issue is overcome, however, at the cost of more compositing steps and several footprint sections per cell. Other improvements of the original splatting technique include blur reduction described by Mueller et al. [295], an anti-aliasing method proposed by Swan II et al. [407], and perspectively accurate splatting discussed by Zwicker et al. [511]. EWA (elliptical weighted average) reconstruction can be used for anti-aliasing in splatting, as shown by Zwicker et al. [510]. Chen et al. [56] describe a GPU implementation of EWA splatting to improve the rendering speed. Other approaches to accelerate splatting include opacity-based culling according to Mueller et al. [296], post-convolution described by Neophytou and Mueller [304], and hierarchical splatting investigated by Hopf et al. [171]. Vega-Higuera et al. [443] describe interactive GPU-based splatting in the context of a medical application. 2.3.6 Cell Projection Cell projection is an object-order approach that traverses the cells of a grid and projects them onto the image plane. In general, cell projection is suitable for unstructured grids that may consist of diﬀerent types of cells. A most prominent approach to cell projection is the projected-tetrahedra algorithm by Shirley and Tuchman [393]. This algorithm works for an important case of unstructured grids: grids consisting of tetrahedra, i.e., simplicial cells. A key element of the algorithm is the observation that the projection of a tetrahedron onto the image plane can be decomposed into a collection of triangles. Figure 2.6 illustrates the three classes of non-degenerate cases that can occur. A tetrahedron is either decomposed into three triangles after projection (class 1a and class 1b) or into four triangles (class 2). Triangles can be eﬃciently rendered by using graphics hardware – even without the need for GPU shaders. Compositing is supported in the form of frame-buﬀer blending.

28

2 Visualization of 3D Scalar Fields

cell configuration

triangle decomposition

class 1a

class 1b

class 2

Fig. 2.6. Classiﬁcation of non-degenerate cases for the projected-tetrahedra algorithm: cell conﬁguration (top) and decomposition into projected triangles (bottom)

The projected-tetrahedra algorithm can also be applied to general unstructured grids. In this case, a pre-processing step needs to split complex cells into a collection of tetrahedra. For example, hexahedral cells can be decomposed into ﬁve or six tetrahedra. Similarly, pyramid and prism cells are split into two or three tetrahedra. Working on a purely tetrahedral grid, the projected-tetrahedra algorithm consists of the following steps: 1. The cells are sorted according to their spatial structure in back-to-front or front-to-back order. 2. Each tetrahedron is classiﬁed according to its projected proﬁle (see Fig. 2.6). The projection is decomposed into a number of triangles. 3. Color and opacity is assigned to the vertices of the triangles, based on an evaluation of the volume rendering integral. 4. The triangles are rendered and blended. One of the issues of cell projection is an eﬃcient spatial sorting for step 1 of the algorithm. Spatial sorting is necessary for non-commutative blending, for example, for the emission-absorption model of volume rendering. Depth sorting is view-dependent and needs to be recomputed for each frame when the camera parameters are changing. Eﬃcient sorting algorithms exist for unstructured grids [58, 280, 490]. In particular, there are speciﬁc algorithms for non-convex grids (i.e., grids with non-convex hull) [60, 399] and for cyclic grids (i.e., grids with cycles in the depth structure of their cells) [226]. All these algorithms are designed for execution on the CPU. In fact, a traditional implementation of the projected-tetrahedra algorithm uses a hybrid CPU/graphics hardware approach: steps 1–3 of the above algorithm are performed on the CPU, whereas the actual rendering and blending (step 4) is performed by the graphics hardware. A more recent approach by Callahan et al. [45] uses a GPU for parts of the sorting process: the CPU performs only an approximate spatial sorting of

2.3 Volume Rendering Approaches

29

the tetrahedral grid, followed by an accurate ﬁnal sorting by GPU fragment operations. The initial sorting on the CPU is fast because it does not need to be completely accurate. Other improvements of the original projected-tetrahedra algorithm address the evaluation of the volume rendering integral in step 3, as well as the interpolation of the opacities and colors within the projected triangles. The original algorithm by Shirley and Tuchman uses a linear interpolation of color and opacity, which may introduce errors. Stein et al. [399] overcome this problem for the case of linear transfer functions. To compute the interpolated opacity, they use a 2D texture parameterized by the thickness of the projected tetrahedron and the averaged absorption coeﬃcient. Color is still interpolated linearly. Pre-integration by R¨ ottger et al. [357] is capable of accurately evaluating the volume rendering integral within projected triangles. The idea is to compute the integral in a pre-processing step and to use this information during actual rendering. Section 2.3.7 describes pre-integrated volume rendering in general. Another subtle problem occurs for perspective projection: linear interpolation on the image plane (for the parameters of pre-integration) is inconsistent with a linear interpolation of scalar values and the thickness parameter in 3D object space. However, linear (i.e., barycentric) interpolation is the typical interpolation scheme in a tetrahedron. Kraus et al. [229] show that this problem can be overcome by perspectively correct interpolation. One of the main issues of the projected-tetrahedra algorithm and its variants is that the decomposition into triangles is view-dependent. Therefore, the geometric description of the projected triangles needs to be transferred to graphics hardware for each frame, leading to a possible bottleneck. Ideally, we would like to use a rendering primitive that represents a volumetric tetrahedron, not a planar triangle. Then, we could use a static geometric description of the tetrahedral grid (provided the data set is static) – without the need for repeated transfer of data to graphics hardware. Unfortunately, GPUs have no native support for tetrahedral primitives. This problem can be overcome by realizing a view-independent scan conversion of tetrahedra by means of GPU programs. For example, Weiler and co-workers [458, 460] describe a GPU approach that allows us to render a tetrahedron without an explicit decomposition into triangles. The basic idea is to adopt GPU ray casting (see Sect. 2.3.3) for a single tetrahedron, whose implementation primarily relies on fragment programs. Wylie et al. [497] discuss an alternative approach for a GPU-based projected-tetrahedra algorithm based on vertex processing. Although view-independent scan conversion works well for displaying tetrahedral cells, it still requires spatial sorting (step 1 of the projected-tetrahedra algorithm) when a non-commutative compositing scheme is applied. For example, for the emission-absorption model a spatial sorting with a corresponding transfer of grid data to the GPU is required. Therefore, view-independent cell rendering is best suited for order-independent compositing, such as emissiononly or absorption-only models.

30

2 Visualization of 3D Scalar Fields

More information and further references on cell projection in general can be found in the survey articles by Kaufman and Mueller [201] or Silva et al. [394]. 2.3.7 Pre-Integrated Volume Rendering This section brieﬂy describes pre-integrated volume rendering, following the detailed discussion by Kraus and Ertl [228]. The discrete version (2.4) of the volume rendering integral (2.2) converges to the true result when the sampling distance ∆x → 0. The sampling theorem requires the sampling rate to be higher than the Nyquist frequency. It is important to note that the volume rendering integral computes the contributions of the scalar data set after the transfer function has been applied. Therefore, the Nyquist frequency depends on the properties of both the actual data set and the transfer function. Non-linear features of the transfer function may increase the required sampling rate signiﬁcantly because this sampling rate depends on the characteristic of the scalar data set and the transfer function [225, 385]. Therefore, it is not suﬃcient to sample the volume rendering integral with the Nyquist frequency of the scalar data set. Pre-integrated volume rendering improves the approximation of the volume rendering integral to overcome the aforementioned issues. Pre-integrated classiﬁcation assumes a piecewise linear, continuous reconstruction of the scalar ﬁeld along a viewing ray. With this assumption, the contribution of a ray segment (which shows a linear behavior of its scalar value) can be precomputed in a preprocessing step according to an accurate numerical solution of the volume rendering integral (2.2). For example, the discrete Riemann sum of this rendering integral (2.4) can be evaluated with a very small sampling distance ∆x to obtain an accurate solution. The pre-computed contributions of a ray segment depend on three parameters: the scalar value at the beginning of the segment, the scalar value at the end point of the segment, and the length of the segment. Therefore, the pre-integrated values can be stored in a 3D array with respect to these three parameters. During runtime, the contribution to the current segment is obtained from this 3D lookup-table. The opacities and colors of several subsequent segments along a viewing ray are blended to a ﬁnal color according to one of the compositing schemes from Sect. 2.1 – either a front-to-back or a back-to-front scheme is possible. In the literature, the idea of pre-integration was ﬁrst used for cell projection. Under the assumption that the color is constant in a cell, pre-integration is related to the shading of polyhedral cells according to Max et al. [280]. For linear transfer functions, Stein et al. [399] use a 2D lookup-table parameterized by the thickness of the projected tetrahedron and the averaged absorption coeﬃcient. Full pre-integration is introduced by R¨ ottger et al. [357] in the context of the projected-tetrahedra algorithm. Their observation is that barycentric interpolation is the natural interpolation scheme for tetrahedra. Since barycentric interpolation is linear, the linear behavior that is assumed

2.3 Volume Rendering Approaches

31

L

sf

sb

slab

Fig. 2.7. Slab-based representation for pre-integrated texture-based volume rendering. The scalar data values at the entry and the exit point of the viewing ray are denoted by s f and s b , respectively. The thickness of the slab is denoted by L

along a viewing ray during pre-processing is compatible with the interpolant for the 3D data set, i.e., no error is introduced by the assumption of a piecewise linear interpolation along light rays. Pre-integration can also be used for a view-independent projection of tetrahedra [458, 460]. A strength of the concept of pre-integration is its applicability to other volume rendering methods. For example, slice-based rendering for uniform grids can be enhanced by pre-integration [111]. Both 2D and 3D slicing can be combined with pre-integration: the original point-sampling on slices is replaced by a slab-based representation that assumes a linear interpolation along a viewing ray crossing a slab. Figure 2.7 illustrates this slab-oriented point of view. A slab is constructed from the two adjacent slices, i.e., two slices have to be considered at a time (instead of only one slice in slice-based rendering). The scalar values at the entry face, sf , and the exit face of a slab, sb , serve as parameters for the lookup in the pre-integration table. When a ﬁxed slab thickness L is assumed, the pre-integration table depends only on the two scalar values. The lookup in this 2D pre-integration table can be implemented by means of a dependent texture lookup within a fragment program [111]. It should be noted that the assumption of a piecewise linear behavior along a viewing ray is not compatible with the trilinear interpolation in uniform grids. However, the error introduced by this approximation is typically negligible – at least it tends to be much smaller than the approximation error caused by point-sampling with a Riemann sum computation along viewing rays. Pre-integration can also be incorporated in other volume rendering techniques: shear-warp volume rendering according to Schulze et al. [385], CPUbased ray casting according to Knittel [222], and special-purpose hardware for ray casting according to Meißner et al. [286]. R¨ ottger, Guthe, Weiskopf, Ertl, and Straßer [356] use pre-integration in the context of GPU-based ray casting in uniform grids. Analogously, pre-integration can be used for GPUbased ray casting in tetrahedral grids, as shown by Weiler et al. [459]. For the projected-tetrahedra algorithm, Kraus et al. [229] propose a perspectively correct interpolation of texture coordinates for the lookup in the pre-integration

32

2 Visualization of 3D Scalar Fields

table. They also present an approach for a more accurate representation of the pre-integration table. 2.3.8 Variants and Extensions Frequency domain volume rendering – also called Fourier volume rendering – is an approach that diﬀers signiﬁcantly from the previous ways of performing volume rendering. Frequency domain volume rendering, introduced by Levoy [246] and Malzbender [268], makes use of the projection slice theorem and computations in the frequency domain. The key observation is that projections in the spatial domain correspond to a slicing in the frequency domain. The extraction of one slice in the frequency domain and the subsequent inverse Fourier transformation to the image plane lead to a computational complexity O(m2 log m), where m is a measure for the size of a slice in one dimension. Therefore, frequency domain volume rendering has a lower computational complexity than other volume rendering methods: the rendering algorithms discussed earlier tend to be O(n3 ), where n is the size of the data set in one dimension. A problem of frequency domain volume rendering is its limitation to x-ray images (i.e., an absorption-only model). Totsuka and Levoy [420] address the quality issues of frequency domain volume rendering by including depth cues and shading calculations. Illumination models are also discussed by Entezari et al. [113]. While the above papers are exclusively based on CPU implementations, Viola et al. [450] describe how frequency domain volume rendering can be mapped to an eﬃcient GPU implementation. Another line of research targets volume illumination models beyond the emission-absorption model (see Sect. 2.1). A survey of advanced rendering techniques for participating media is presented by Cerezo et al. [51]. Unfortunately, a general model of global illumination for participating media leads to complex numerical computations that typically prohibit interactive rendering. In fact, most of these rendering methods are implemented on CPUs and limited to non-interactive rendering. However, restricted models of global illumination can be employed for real-time volumetric rendering on GPUs. A ﬁrst, yet important, step toward increased optical realism is the support for shadowing. For example, Kniss et al. [218] include shadow computations in 3D texture slicing by using a common slicing direction for simultaneously accumulating the weights for shadowing and the color and opacity contributions for the ﬁnal image. In this way, only little computational and memory costs are added to the costs of emission-absorption rendering. Zhang and Crawﬁs [504, 505] independently developed a similar shadowing method for splatting. One step further toward optical realism is to add translucent light transport within a volume, taking into account multiple scattering. Typically, the scattering model needs to be restricted to allow for real-time rendering. Kniss et al. [220, 221] follow this approach in their GPU implementation of multiple scattering based on texture slicing. In important application of global

2.4 Maintaining Constant Frame Rates in Texture-Based Volume Rendering

33

illumination is GPU-based cloud rendering, as discussed by Riley et al. [345] and Schpok et al. [381]. In a related paper, Riley et al. [346] describe the GPU rendering of atmospheric phenomena. Magnor et al. [267] propose a multi-resolution model to approximate multiple scattering in astronomical reﬂection nebulae and demonstrate how their model leads to an eﬃcient GPU implementation. Finally, Hegeman et al. [162] introduce a lighting model for inhomogeneous media that can be mapped to programmable graphics hardware.

2.4 Maintaining Constant Frame Rates in 3D Texture-Based Volume Rendering Texture-based volume rendering is one of the most popular approaches for interactive volume visualization due to its direct and eﬃcient support by graphics hardware. Therefore, the remainder to this book focuses on texture slicing as basic technique. This section, in particular, addresses the issue of eﬃcient rendering by means of 3D texture-based volume visualization. 3D texture slicing typically relies on a trilinear interpolation of the input data set. Since this interpolation scheme needs access to a neighborhood of eight voxels, 3D texture slicing is associated with high bandwidth requirements for data transfer from texture memory. Usually, the texture cache is optimized for fetch operations in 2D textures, which are important for most applications. On the other hand, diﬀerent hardware architectures use diﬀerent memory layouts for 3D textures. In one approach, the volume is essentially built from a collection of 2D textures in a slice-by-slice fashion. Here, a good caching strategy is only available within a 2D slice, but not along the stacking axis. Therefore, the cache behavior greatly depends on the order of accessing the 3D texture and the performance of volume rendering can become viewdependent. This characteristic is inappropriate for real-time applications that have to maintain constant frame rates. In a second approach, a memory layout is chosen that achieves a comparable cache coherence along all directions. Here, the volume is not constructed slice-by-slice, but in a more complex, staggered manner. Both memory layouts can be found in today’s GPUs since there are good reasons for both approaches. For example, the latter architecture shows a uniform performance behavior for volume rendering and therefore is well-suited for real-time applications like virtual environments. On the other hand, the slice-by-slice approach allows the application to update a volume slice very eﬃciently (because of its direct mapping to the memory); with the superbuﬀer extension [327] or the frame-buﬀer object extension [390], even a direct rendering into a volume slice is possible. With the increasing popularity of general-purpose computations on GPUs (GPGPU [142]), this ability will become even more important in the future.

2 Visualization of 3D Scalar Fields

IMAGE: WEISKOPF, WEILER, ERTL [484]

c 2004 IEEE

34

Fig. 2.8. Partitioning a volume data set into 43 bricks

This section describes a modiﬁcation of 3D texture-based volume rendering that maintains approximately constant frame rates on any GPU architecture. This approach was proposed by Weiskopf, Weiler, and Ertl [484], and the following discussion is based on their work.1 The idea of their technique is to split the volume into smaller sub-volumes, i.e., bricks [145]. These bricks can be oriented in diﬀerent directions; thus, the varying performance for diﬀerent viewing directions is balanced out within a single image. Multi-Oriented Bricking The volume can be partitioned into smaller bricks to overcome the potential view dependency in texture-based volume rendering. Figure 2.8 shows an example in which the volume data set (that is visualized as a sphere) is split into sub-volumes, 43 in number. The borders of the bricks are marked by white lines. These bricks can be oriented in diﬀerent directions. Figure 2.9 illustrates (in the simpliﬁed analogue of a 2D drawing) that the number of bricks which are oriented along the viewing direction can be kept constant. Therefore, the number of elements that can be rendered at high speed is independent of the viewing direction. Note that the “fast” bricks may change. For example, in the left image of Fig. 2.9, the bricks in light gray lead to higher rendering speed, whereas, in the right image, the bricks in dark gray are faster to render. This idea works as long as the whole volume is in the visible view frustum and roughly the same number of fragments are associated with each brick. These two assumptions are only partly fulﬁlled in real applications. Here, the uniformity of rendering speed is the better the ﬁner the granularity of the 1

c 2004 IEEE. Reprinted, with permission, from Weiskopf, Weiler, and Ertl [484]

2.4 Maintaining Constant Frame Rates in Texture-Based Volume Rendering

Eye

35

Eye IMAGE: WEISKOPF, WEILER, ERTL [484]

c 2004 IEEE

Fig. 2.9. For any viewing direction, two bricks are oriented along the viewing direction and the other two bricks are roughly perpendicular. The lines inside the bricks indicate the orientation of internal slices for the corresponding 3D texture

bricking is chosen, i.e., the smaller the sub-volumes become. On the other hand, ﬁner granularity leads to a larger number of slicing polygons to be computed and rendered because each slice through the complete volume has to be partitioned into sub-slices for the diﬀerent bricks. Moreover, a brick should have a one-texel-wide overlap with neighboring bricks [145]. Therefore, additional texture memory is needed, of which the consumption increases when the number of bricks is increased. Note, however, that the sizes of the additional bricks can be chosen independently of the other bricks’ sizes and thus only little memory is wasted by the restriction to power-of-two textures. In summary, the granularity should be optimized by taking into account the above aspects. The optimal number of bricks is highly dependent on the application and the characteristics of user navigation. From experience, bricking into 23 or 43 sub-blocks leads to reasonable results, as long as the viewing parameters are not too extreme. Corresponding performance measurements can be found in the following section. Bricking is frequently used in volume rendering already. For example, large data sets that do not ﬁt into texture memory at once can only be handled by bricking [219, 237, 451]. The same approach is also taken in Sect. 2.6 for large-data volume visualization on a GPU cluster computer. Moreover, bricking overcomes the waste of texture memory that occurs when data sets of arbitrary size have to be extended by empty space to meet the restriction to power-of-two textures. Therefore, bricking is already available in many volume rendering applications (e.g., in OpenGL Volumizer [19] or the volume node of OpenSG [316]) and multi-oriented bricking causes only minimal additional implementation eﬀorts. Thus, multi-oriented bricking is a practical and easy-to-handle solution that, in particular, improves the usage of volume rendering in real-time sensitive applications such as virtual reality and visual simulations. Typical implementations of virtual environments freeze the overall frame rate to the lowest achievable frame rate, which would be un-

36

2 Visualization of 3D Scalar Fields

acceptable if the ratio between maximum and minimum rendering speeds is large. Performance Comparison This section shows a performance analysis for the bricking approach and compares the bricking method with the original non-bricking rendering. Two typical representatives for the two types of memory layout are used: an ATI Radeon 9800 Pro as an example of a GPU with slice-oriented layout and an NVIDIA GeForce FX 5950 Ultra as an example of a graphics board with an optimized memory structure for volume rendering. The size of the test data set is 2563 and shows a spherically symmetric behavior with a linear dependency on the distance from the center point. Figure 2.8 depicts a volume visualization of this data set. Pre-classiﬁcation is used with a two-component 3D texture that stores pre-computed luminance and alpha values. The slicing distance is kept constant, i.e., the number of slices may change with the viewing direction. Just the ﬁxed-function vertex and fragment pipeline is used. In this way, the test scenario focuses on the 3D texture access speed. The viewport has a size of 8002 pixels in all performance tests. Figure 2.10 compares the rendering speed for the two architectures, based on a single brick (i.e., standard view-aligned volume slicing). The rendering speeds are shown in fps (frames per second) along the vertical axis. The horizontal axis describes the viewing direction in degrees. The angles denote a rotation of the volume about the y axis, which forms a vertical line on the viewing plane. The z axis is assumed to be parallel to the stacking axis for building the 3D texture from slices. For an angle of zero, the viewer looks along the z axis of the texture. Figure 2.10 demonstrates that the rendering performance of the slice-oriented architecture heavily depends on the viewing angle. In this case, the minimum and maximum frame rates form a ratio of 1:4.8. In contrast, the other hardware architecture maintains a much more uniform frame rate; there is only a 12 percent diﬀerence between minimum and maximum frame rates. Figure 2.11 shows that multi-oriented bricking achieves a much more even rendering performance than the standard approach for the Radeon 9800 Pro GPU. Here, bricking is employed with a total number of 43 sub-volumes. The directions of the bricks are alternated between the x, y, and z axes, based on the index of the bricks. The ratio between minimum and maximum frame rates is reduced to 1:1.2. Figure 2.12 shows the same comparison between 43 bricking and standard volume rendering for the FX 5950 Ultra GPU. The performance numbers indicate that the bricking approach is a little bit slower, which is mainly caused by the additional work for the increased number of slice polygons. These measurements illustrate how much performance penalty is associated with bricking: the overall rendering speed is reduced by less than ﬁve percent.

2.4 Maintaining Constant Frame Rates in Texture-Based Volume Rendering

37

Single-brick 3D texture-based rendering 50

FX 5950 Ultra Radeon 9800 Pro

45 40 35 fps

30 25 20 15 10 5 0 0

50

100

150

200

250

300

350

rotation angle (in degrees)

Fig. 2.10. Single-brick volume rendering on FX 5950 Ultra and Radeon 9800 Pro Single-brick vs multi-brick rendering (Radeon 9800 Pro) 50

Single-brick rendering Rendering with 4*4*4 bricks

45 40 35 fps

30 25 20 15 10 5 0 0

50

100

150

200

250

300

350

rotation angle (in degrees)

Fig. 2.11. Comparing volume rendering with single brick and 43 bricks on Radeon 9800 Pro Single-brick vs multi-brick rendering (FX 5950 Ultra) 50

Single-brick rendering Rendering with 4*4*4 bricks

45 40 35 fps

30 25 20 15 10 5 0 0

50

100

150

200

250

300

350

rotation angle (in degrees)

Fig. 2.12. Comparing volume rendering with single brick and 43 bricks on FX 5950 Ultra c 2004 IEEE ALL IMAGES: WEISKOPF, WEILER, ERTL [484]

38

2 Visualization of 3D Scalar Fields

2.5 Volume Clipping

IMAGE: WEISKOPF, ENGEL, ERTL [469]

c 2002 IEEE

Volume clipping plays a decisive role in understanding 3D volumetric data sets because it allows the user to cut away selected parts of the volume based on the position of voxels in the data set. Clipping is an important example of a ﬁltering method in the volume rendering pipeline because it may be the only way to uncover important, otherwise hidden details of a data set. This geometric approach can be regarded as complementary to the speciﬁcation of transfer functions that are based on the data values and their derivatives only. Therefore, a combination of data-driven transfer functions and geometryguided clipping can be used to achieve eﬀective volume visualizations. Although clipping is frequently used in volume rendering applications, only a straightforward approach is usually adopted – with one or multiple clip planes serving as clip geometry. For example, Van Gelder and Kim [436] use clip planes to specify the boundaries of the data set in 3D texture-based volume rendering. Although some clip geometries can be approximated by multiple clip planes, many useful and important geometries are not supported: cutting a cube-shaped opening into a volume (Fig. 2.13) is a prominent example. To achieve more ﬂexibility in volume clipping, a number of more advanced clipping techniques are described in this section. They can handle complex clip geometries and are designed to be used in the context of GPU-based volume rendering. Although the presented clipping techniques are discussed in combination with texture slicing, most of them could be adapted for GPU ray casting as well. Two basic approaches to volume clipping, as proposed by

Fig. 2.13. Volume clipping in an MR data set

2.5 Volume Clipping

39

Weiskopf, Engel, and Ertl [469, 470], are discussed in the beginning. In the ﬁrst approach (Sect. 2.5.1), a clip object is represented by a tessellated boundary surface. The basic idea is to store the depth structure of the clip geometry in 2D textures whose texels have a one-to-one correspondence to pixels on the viewing plane. The depth structure of the clip object can be used by fragment operations to clip away unwanted parts of the volume. This depthbased approach is particularly well-suited for producing high-quality images because clipping is performed with per-pixel accuracy. In the second approach (Sect. 2.5.2), a clip object is voxelized and represented by an additional volume data set. Clip regions are speciﬁed by marking corresponding voxels in this volume. The description of both approaches is based on Weiskopf, Engel, and Ertl [470].2 These approaches were originally designed for slice-based volume rendering. The extension to pre-integrated volume rendering needs a couple of modiﬁcations that were introduced by R¨ ottger, Guthe, Weiskopf, Ertl, and Straßer [356]. Besides the core clipping techniques for volume visualization, issues related to volume illumination are speciﬁcally considered. Lighting introduces further information on the spatial structure and orientation of features in the volume data set and can thus facilitate a better understanding of the original data. The combination of volume shading and volume clipping, however, introduces issues of how illumination needs to be computed in the vicinity of the clipping object. On the one hand, the orientation of the clipping surface should be represented. One the other hand, properties of the scalar ﬁeld should still inﬂuence the appearance of the clipping surface. Weiskopf, Engel, and Ertl [470] proposed an optical model that takes into account a consistent shading of the clipping surface. This model is described in Sect. 2.5.4. In addition to applications in scientiﬁc visualization, volume clipping can play an important role in volume illustrations. As detailed later in Sect. 4.4.7, volume illustration is an example of non-photorealistic rendering (NPR) and it has the goal of emphasizing important elements of an image, while deemphasizing or removing unimportant elements. Clipping is an extreme, yet very useful basis technique for such a volume illustration because it allows us to completely remove unimportant regions. Section 4.4.7 also discusses the generalization to the rendering of deformed volumes and it describes how deformation can be used for illustrative rendering. Similarly, volume clipping is related to volume sculpting (see, e.g., Wang and Kaufman [453] or Perry and Frisken [329]), although the typical focal point of sculpting is modeling, not rendering. 2.5.1 Depth-Based Clipping This section focuses on rendering algorithms for volume clipping that exploit the depth structure of the clip geometry. In all depth-based approaches, it is 2

c 2003 IEEE. Reprinted, with permission, from Weiskopf, Engel, and Ertl [470]

40

2 Visualization of 3D Scalar Fields ray eye

image plane

object

depth structure inside

z value IMAGE: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

Fig. 2.14. Depth structure of a clip geometry for a single pixel in the frame buﬀer

assumed that clip objects are deﬁned by boundary surface representations. Graphics hardware expects surfaces to be tessellated – usually in the form of triangular meshes. Figure 2.14 illustrates the depth structure of a typical clip object. In what follows, the underlying 3D scenario is reduced to just a 1D geometry along a single light ray that originates from the eye point and reaches into the scene. From this point of view, only the corresponding single pixel on the image plane is taken into account. Therefore, the following descriptions can be mapped to operations on those fragments that correspond to the respective position of the pixel in the frame buﬀer. Let us start with a generic algorithm for clip objects of arbitrary topology and geometry. First, the depth structure of the clip geometry is built for the current pixel. This structure stores the depth values for each intersection between eye ray and object. In this way, the 1D depth values are partitioned into intervals whose bounds are given by the boundaries of the objects. In addition, the intervals are classiﬁed either as inside or as outside of the clip object. The classiﬁcation is determined by the clip geometry and can be based on the orientation of the tessellated surface or on the direction of normal vectors associated with the clip geometry. For a closed, non-selﬁntersecting clip surface, the classiﬁcation alters repeatedly from outside to inside and vice versa. The user can choose to clip away either the inside or outside of the clip object. In this way, the intervals are uniquely speciﬁed as visible or invisible. Second, the rendering of each fragment of a volume slice has to check to which class of interval the fragment belongs. Based on the visibility, the fragment is blended into the frame buﬀer or clipped away. As mentioned before, intervals are either marked as visible or invisible. The term volume probing is used for a scenario where the volume is clipped away outside the clip geometry and the volume inside the clip object remains unchanged. Conversely, a volume cutting approach inverts the role of the visibility property. Here, only the volume outside the clip object remains visible.

2.5 Volume Clipping

41

The following issues have to be considered for an actual implementation of the above algorithm. What kind of data structure is appropriate to store the depth structure? How is the depth structure generated from clip geometry? How is the visibility property of a fragment evaluated? To be more speciﬁc, the question is how the above algorithm can be eﬃciently implemented on graphics hardware. Since the depth structure is stored on a per-pixel basis, a pixel-oriented mechanism has to be considered, such as a texture on the graphics board. Depth-Based Clipping against a Single Boundary A straightforward approach exploits the depth buﬀer as storage medium for interval boundaries. A major disadvantage of this approach is the fact that only one depth value per pixel can be stored. Therefore, just a single boundary can be implemented, which greatly reduces the ﬁelds of application. The implementation of this approach begins with rendering a simple clip geometry to the depth buﬀer, i.e., the depth structure is generated from the clip geometry by standard scan conversion of the graphics hardware. Then, writing to the depth buﬀer is disabled and depth testing is enabled. Finally, the slices of the volume data set are rendered and blended into the frame buﬀer. Here, the depth test implements the evaluation of the visibility property of a fragment. The user can choose to clip away the volume in front or behind the geometry, depending on the logical operator for the depth test. Two-Pass Rendering for Convex Volume Cutting Volume cutting with a convex and closed clip object can be implemented by two-pass rendering based on depth tests. A convex and closed object implies that the number of boundaries along the depth structure is not larger than two because, regardless of the viewing parameter, an eye ray cannot intersect the object more than twice. Concave clip geometries can beneﬁt from this approach as well: a concave clip geometry can often be approximated by two boundaries without introducing inappropriate artifacts. The algorithm is illustrated in Fig. 2.15. First, the depth values zback for the second boundary are determined by rendering the back faces of the clip geometry into the depth buﬀer. The depth buﬀer contains the initial value for the far plane of the viewing frustum in regions where the clip object is not apparent. Then, volume rendering is applied with the depth test set to “greater” and without modifying the depth buﬀer. In this way, the volume behind the clip object is drawn (cf. the gray portion in Fig. 2.15). The second rendering pass is responsible for drawing the volume in front of the clip object and in regions not covered by the clip object (i.e., the hatched portion in Fig. 2.15). First, the depth buﬀer is cleared and then ﬁlled with the depth values zfront for the ﬁrst boundary by rendering the front faces of the clip geometry. Afterwards, volume rendering is accomplished with the depth test set to “less” and without modifying the depth buﬀer.

42

2 Visualization of 3D Scalar Fields rendered in second pass

eye

convex clip object

rendered in first pass

rectangular volume IMAGE: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

Fig. 2.15. Illustration of depth-based two-pass rendering for a convex clip geometry

Single-Pass Rendering for Convex Volume Probing Based on Depth Clipping A volume probing approach for a convex clip object is now considered. The corresponding maximum number of two boundaries along the depth structure can be implemented by single-pass rendering. The basic algorithm for depth-based volume probing is as follows. First, the depth values zfront for the ﬁrst boundary are determined by rendering the front faces of the clip geometry into the depth buﬀer. Second, the contents of the depth buﬀer are stored in a texture and the fragment shader is conﬁgured to shift the depth values of all fragments in the following rendering passes by −zfront . Third, the depth buﬀer is cleared and the backside of the clip geometry is rendered into the depth buﬀer (with depth shift enabled) to build the second boundary. Finally, slices through the volume data set are rendered and blended into the frame buﬀer, without modifying the depth buﬀer. However, depth shift and depth testing have to be enabled. The following discussion describes in more detail how the depth of a fragment determines its visibility. The visibility depends on both depth clipping and depth testing. The decision whether a fragment passes depth clipping and depth testing can be expressed by a Boolean function df (zf ), where zf is the depth of the fragment. Depth values are assumed to be described with respect to normalized window coordinates, i.e., valid depth values lie in [0, 1]. In general, the value of df (zf ) can formally be written as a logical expression, df (zf ) = dclip (zf ) ∧ (zf op zb ) ,

(2.8)

with dclip (zf ) = (zf ≥ 0) ∧ (zf ≤ 1) . In this notation, ∧ symbolizes a logical “and”. The function dclip (zf ) describes depth clipping against the bounds 0 and 1 of the view frustum. The binary

2.5 Volume Clipping

43

ray eye convex clip object

removed by clipping test

removed by depth test

visible

z

front

z

back IMAGE: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

Fig. 2.16. Illustration of depth-based volume probing for a convex clip geometry

logical operator op is used for depth testing against zb , the current entry in the depth buﬀer. Figure 2.16 illustrates how the clipping algorithm operates on the depth structure. A shift by −zfront is applied to all depth values. By rendering the back faces of the clip object with the same shift, entries in the depth buﬀer are initialized to zback − zfront , where zback is the unmodiﬁed depth of the back face’s fragment. During the ﬁnal rendering of the volume slices, the logical operator for the depth test is set to “≤”. Therefore, (2.8) can be written as df (zf ) = ((zf − zfront ≥ 0) ∧ (zf − zfront ≤ 1)) ∧(zf − zfront ≤ zback − zfront ) = dprobe (zf ) ∧ (zf ≤ 1 + zfront ) , with dprobe (zf ) = (zf ≥ zfront ) ∧ (zf ≤ zback ) .

(2.9)

The term dprobe (zf ) exactly represents the required logical operation for displaying the volume only in the interval [zfront , zback ]. The last term, (zf ≤ 1 + zfront ), implies just a clipping against a modiﬁed far clipping plane. A shift of depth values by −zfront can be eﬃciently accomplished on a per-fragment basis by modern graphics hardware. The implementation on NVIDIA GeForce 3/4 is described in detail by Weiskopf, Engel, and Ertl [469]. A high-resolution texture is used to store the respective depth values zfront and a fragment program is employed to replace the z value of a fragment by z − zfront . The details of the vertex and fragment programs are also presented in [469]. Figure 2.17 illustrates clipping in a medical MR data set. Image (a) shows the data set without clipping. Depth-based volume probing is applied in image (b). This picture demonstrates that depth-based volume clipping allows for geometries with a large number of triangles – such as the Stanford bunny.

2 Visualization of 3D Scalar Fields

IMAGE: WEISKOPF, ENGEL, ERTL [469]

c 2002 IEEE

44

(a)

(b)

Fig. 2.17. Depth-based volume probing for an MR data set: (a) shows the data set without clipping; (b) illustrates depth-based volume probing with the Stanford bunny as clip object

Single-Pass Rendering for Convex Volume Cutting Based on Depth Clipping The previous clipping mechanism can also be used to invert the role of the visibility property. Volume cutting replaces the above volume probing approach, i.e., only the volume outside the clip object remains visible. Once again, a maximum number of two boundaries along the depth structure is assumed. By inverting (2.9) for volume probing, one obtains dcutting (zf ) = (zf ≤ zfront ) ∨ (zf ≥ zback ) ,

(2.10)

for volume cutting, with “∨” representing the logical “or” operator. The cutting function dcutting (zf ) is almost identical to the negation of the probing function, ¬ dprobe (zf ) – except for the fact that zfront and zback satisfy the Boolean functions for both volume probing and volume cutting. Unfortunately, logical operations for depth clipping and depth testing have to be of the form of (2.8), which is not compatible with (2.10). The solution to this problem is based on the following considerations. First, a simpliﬁed scenario is discussed. The expression (x ≥ −1) ∧ (x ≤ 1) can be rewritten as (f (x) ≥ −1) ∧ (f (x) ≤ 1) by introducing an identity function f (x) = x. The valid values x are determined by the intersection of f (x) with the horizontal lines at ±1. Now a similar situation is considered in which the function f (x) is replaced by another function g(x) = 1/x. In this case, valid values of the logical expression (g(x) ≥ −1) ∧ (g(x) ≤ 1) also satisfy (x ≤ −1) ∨ (x ≥ 1). Therefore, a mapping between the form of (2.10) and the required form of (2.8) can be achieved. As a matter of fact, there are other possible choices of mapping functions to achieve a similar inversion of z clipping. The advantage of a hyperbolic mapping is its availability in the fragment operations unit even on older graphics boards.

2.5 Volume Clipping

45

1

z scale 0 z mid

z back

z

z front valid z values IMAGE: WEISKOPF, ENGEL, ERTL [469]

c 2002 IEEE

Fig. 2.18. Schematic illustration of the depth structure for volume cutting via 1/z mapping

The adaptation of this idea to graphics hardware needs to take into account that window coordinates lie in [0, 1] and not in [−1, 1]. Moreover, the vertical asymptote of the hyperbola must not meet the origin, z = 0, but has to be located at the midpoint of the clip geometry. The midpoint is obtained by zmid = (zfront +zback )/2. In addition, the actual distance between the front and the back boundaries of the clip geometry is taken into account by the scaling factor zscale = (zback − zfront )/2. Figure 2.18 illustrates how the quantities of the clip object can be used for volume cutting via 1/z mapping. The correct function has to be 1 1 zscale (2.11) + , g(z) = 2 z − zmid 2 in order to produce the graph of Fig. 2.18. The algorithm for depth-based volume cutting can now be formulated as follows. First, the depth values of the front faces of the clip geometry are rendered into the depth buﬀer. Then, these values zfront are transferred to main memory. Analogously, the depth values of the back faces, zback , are determined and copied to main memory. Based on zfront and zback , the CPU computes zmid and zscale and stores them as ﬁrst and second components of a high-resolution texture. A fragment program is enabled to implement the mapping according to (2.11) by employing the above mentioned texture to modify the depth value of each fragment. Finally, slices through the volume data set are rendered and blended into the frame buﬀer. In contrast to volume probing described in the previous section, the depth buﬀer does not contain any information on the clip geometry and depth testing is not required. Volume cutting is only based on the data from the high-resolution texture and on view frustum clipping.

46

2 Visualization of 3D Scalar Fields

Once again, the paper by Weiskopf, Engel, and Ertl [469] contains details of the implementation on GeForce 3/4, including both the fragment program and the vertex program that issues texture coordinates on a per-vertex basis. Convex Volume Clipping Based on Shadow Test The two previous clipping techniques are based on high-resolution textures, transformation of depth values, and view frustum clipping. An alternative, similarly structured approach exploits shadow mapping and the shadow test to achieve comparable results. A related approach is taken by Everitt [116] for spatial sorting in surface-based rendering. Compared to the algorithm from the previous two sections, shadow-based clipping introduces the following modiﬁcations. First, the contents of the z buﬀer is transferred to a depth texture that can store values for a subsequent shadow test with high accuracy. Second, the fragment operations now perform a comparison between the depth of a fragment and the depth structure stored in the depth texture by means of the shadow test. In typical applications of shadow mapping, the positions of the virtual light source and the camera are diﬀerent. Here, however, the virtual light for shadow testing is identical with the camera. The test result is stored in the alpha component of the fragment. Finally, the fragment may be removed by the alpha test, depending on the result of the preceding depth comparison. With a Shader Model 2 compliant GPU, a related alternative implementation is feasible. Here, the depth structure is stored in ﬂoating-point textures. The fragment program has to include a conditional removal of the fragment by a texkill instruction, depending on the comparison between the fragment’s depth value and the stored depth structure. Multi-Pass Rendering for Concave Clip Objects So far, only convex clip objects can be correctly handled. In order to allow for concave objects, the management of the depth structure has to be extended to consider a larger number of possible intersections between the clipping geometry and an eye ray. The following multi-pass rendering technique is proposed for concave clipping. First, the depth structure of the clip object is analyzed and neighboring pairs of z values in this depth structure are combined. These pairs are organized in a way which ensures that each pair represents one visible segment of the viewing ray, i.e., the volume remains visible within the interval bounded by such a pair of z values. Note that, in the case of volume cutting, this organization generates single, unpaired depth entries at the front and back parts of the depth structure. The depth-peeling algorithm [116] can be used to determine the depth structure from a triangle mesh. Each pair of z values is stored in separate textures, i.e., n pairs of textures are required if n describes the maximum number of depth pairs in the depth structure. Note that this process is performed only once per frame.

2.5 Volume Clipping

47

Second, the actual multi-pass rendering of the volume is accomplished. For each pair of z values from the depth structure, the corresponding region of the volume is rendered. Since each depth pair represents a local volume probing against two boundaries, rendering is based on the previously introduced techniques for volume probing. To put it another way, the complete volume is reconstructed in a layer-by-layer fashion by processing the depth structure pair-by-pair. The depth structure is processed in the same direction as the direction of slice compositing. For example, a back-to-front compositing implies that the depth pairs are also processed back-to-front. The case of volume cutting – with single depth values at the front and back of the depth structure – can be handled by clipping only at a single boundary. 2.5.2 Clipping Based on Volumetric Textures In contrast to the depth-based clipping algorithms from the previous section, volumetric clipping approaches model the visibility information by means of a second volume texture of which the voxels provide the visibility information for voxels of the “real” volume. Here, the clip geometry is voxelized and stored as a binary volume. This clipping texture is either a 3D texture or a stack of 2D textures. It is initialized with one for all voxels inside the clip geometry and with zero for all voxels outside. Thus, arbitrary voxelized clip objects are possible. Clipping techniques based on a volumetric description of clipping objects are related to so-called tagged or attributed volumes. Volume tagging is particularly useful in medical applications that provide segmentation data. For example, Hastreiter et al. [153] modify texture-based volume rendering with pre-classiﬁcation by applying diﬀerent transfer functions to diﬀerent regions of the segmented data set. In their approach, volume tags are spatially ﬁxed and cannot be modiﬁed interactively. Tiede et al. [418] use a similar approach for the visualization of attributed volumes by ray casting. Pﬂesser et al. [332] extend this ray casting approach to incorporate arbitrarily shaped cut surfaces. Konrad-Verse et al. [223] use a deformable cutting plane to specify and modify virtual resections in a medical application. Two-level volume rendering by Hauser et al. [154] is related to the visualization of attributed volumes because it combines diﬀerent volume rendering techniques for diﬀerent subsets of a 3D data set. In a publication on texture-based volume visualization with OpenGL Volumizer [19], a clipping mechanism based on a volumetric description of clipping objects is described. This implementation requires two-pass rendering and models visibility by means of a stencil test on each slice. In the volumetric clipping approach by Weiskopf, Engel, and Ertl [469, 470], the clipping texture and the volume data set are combined by multiplying their texture entries. In this way, voxels to be clipped are multiplied by zero. During actual rendering, a texture slice of the data set and a slice of the clip texture are simultaneously mapped onto the same slice polygon and combined

48

2 Visualization of 3D Scalar Fields

using a per-component multiplication. Fragments lying outside the clip object have an alpha value of zero and can be discarded by a subsequent alpha test. The clip object can be scaled and moved by adapting the texture coordinates of the clipping texture. Additionally, when using 3D textures, a rotation of the clip object is possible. To be more precise, any aﬃne transformation of the 3D texture-based clip geometry can be represented by a corresponding transformation of texture coordinates. Switching between volume probing and volume cutting is easily achieved by applying a per-fragment invert mapping to the clipping texture. In this way, the original texture value is replaced by (1 − value). Since all these operations only require the change of texture coordinates or a re-conﬁguration of per-fragment operations, all above transformations of the clip geometry can be performed very eﬃciently. Only a complete change of the clip geometry’s shape requires a revoxelization and a reload of the clipping texture. The above technique is based on a binary representation and therefore requires a nearest-neighbor sampling of the clipping texture. If a bilinear (for a stack of 2D textures) or a trilinear (for a 3D texture) interpolation was applied, intermediate values between zero and one would result from a texture fetch in the clipping texture. Therefore, a clearly deﬁned surface of the clip geometry would be replaced by a gradual and rather diﬀuse transition between visible and clipped parts of the volume, which is inappropriate for most applications. However, a missing interpolation within the clipping texture introduces quite visible, jaggy artifacts similar to those in aliased line drawings. To overcome this problem, the following modiﬁcation of the original approach can be used. The voxelized clipping texture is no longer used as a binary data set but interpreted as a distance volume: each voxel stores the (signed) Euclidean distance to the closest point on the clip object. When a 3D texture format with ﬁxed-point numbers (from the interval [0, 1]) is used, the distance is mapped to this range. In this way, the isosurface for the isovalue 0.5 represents the surface of the clip object. During rendering, a trilinear interpolation in the 3D clipping texture is applied (a stack of 2D textures is not considered here because of the missing interpolation between slices). Based on a comparison between the isovalue 0.5 and the value from the clipping texture, either the RGBA value from the data set (for a visible fragment) or zero (for an invisible fragment) is used as ﬁnal fragment color. In this way, a clearly deﬁned surface without jaggy artifacts is introduced at the isovalue 0.5. In Fig. 2.19, a cube-shaped voxelized clip object is used for volume cutting within the data set of an engine block. Trilinear interpolation is applied in image (a), nearest-neighbor sampling in (b). Due to trilinear interpolation, the corners and edges of the cubic clip object are smoothed out unless the faces of the cube are identical with voxel boundaries in the clip volume. Therefore, the original clip geometry cannot be reproduced in (a). An example of a signiﬁcant diﬀerence between both methods is marked by red arrows. For this speciﬁc example of an axis-aligned cube, nearest-neighbor sampling gives acceptable

2.5 Volume Clipping

(a)

49

(b) IMAGES: WEISKOPF, ENGEL, ERTL [469]

c 2002 IEEE

Fig. 2.19. Comparison for a voxelized cube-shaped clip object: (a) trilinear interpolation; (b) nearest-neighbor sampling (see also Color Plate 1)

results. However, a generic clip object that contains both smooth regions and sharp features can be more accurately handled by depth-based clipping. The main problem of volumetric clipping is that the resolution of the voxel representation may not be suﬃcient to capture all details of the clipping surface. This issue could be addressed by locally increasing the voxel resolution, for example, by means of an adaptively sampled distance ﬁeld [127]. However, an eﬃcient implementation of adaptively sampled distance ﬁelds on GPUs has not yet been done and should be addressed in future work. Another solution is the use of a procedural description of the distance volume that represents the clipping surface. For example, simple shapes such as spheres or cylinders could be compactly described by a procedural shader. Advantages of a procedural representation are a highly accurate clipping model and savings in required texture memory. The main disadvantage is the restriction to a small class of supported shapes. Therefore, the remainder of this section focuses on a voxel representation of the clipping geometry. Fortunately, due to the availability of multiple texturing units, usually no or only little performance penalty is introduced for the actual rasterization of a voxelized clip object. However, performance might be aﬀected by additional memory access for the clipping texture. An important advantage of this method is the fact that any kind of clip geometry can be speciﬁed – without any limitation with respect to topology or geometry. The main disadvantages of the approach are additional texture memory requirements for the clipping volume, the need for a voxelization of the clip geometry, and possible artifacts due to a limited resolution of the clipping texture. Moreover, the clipping surface can only be represented as an isosur-

2 Visualization of 3D Scalar Fields

IMAGES: WEISKOPF, ENGEL, ERTL [469]

c 2002 IEEE

50

(a)

(b)

Fig. 2.20. Volumetric clipping in a medical MR data set, with a voxelized cylindrical clip object in (a) and a complex clip object in (b)

face in a piecewise trilinearly interpolated scalar ﬁeld, which makes it diﬃcult to reproduce clip objects with sharp edges and creases. Figure 2.20 shows volumetric clipping in a medical MR data set. Image (a) uses a voxelized cylindrical clip object. The clipped part of the volume is moved away from the remaining parts of the volume. Figure 2.20 (b) demonstrates that volumetric clipping is capable of processing complex clip geometries and topologies. Figure 2.21 compares the diﬀerent volume clipping methods of this section and the previous section. The underlying test data set is of size 1283 and shows a piecewise linear change of scalar values along the vertical axis, which are mapped to yellow and blue colors. The boundary of the cubic volume has a diﬀerent scalar value that is mapped to a brownish–orange color. Lighting is disabled and texture-based rendering with pre-classiﬁcation based on paletted textures is used. The ﬁrst three images in the top row of Fig. 2.21 are produced by volume probing, the images in the second row are produced by volume cutting. Images (a), (b), (e), and (f) show clipping based on a volumetric texture. The clipping texture has the same resolution as the data set. In (a) and (e), nearest-neighbor sampling is applied; in (b) and (f), a trilinear interpolation within the distance ﬁeld is used. Artifacts are clearly visible for nearest-neighbor sampling but not for trilinear interpolation. Images (c) and (g) demonstrate clipping as presented in Sect. 2.5.1 (single-pass depth-based clipping). Depth-based clipping achieves high-quality visualization because clipping is performed with per-pixel accuracy. For image (h), only a single clip surface is implemented according to the depth-based approach that makes use of the z buﬀer only. This technique erroneously removes parts of the volume data set around the closest vertical edge. In an interactive application the introduced errors are quite disturbing; the spatial structure is hard to grasp

Fig. 2.21. Comparison of volume clipping techniques (see also Color Plate 2)

2.5 Volume Clipping 51

c 2002 IEEE IMAGES: WEISKOPF, ENGEL, ERTL [469]

52

2 Visualization of 3D Scalar Fields

because clipping properties become view-dependent in this simple approach. Finally, image (d) shows the clip geometry in the form of an opaque sphere. 2.5.3 Clipping and Pre-Integration The aforementioned clipping approaches are based on a slice-by-slice representation of the volume data set. Therefore, the visibility computation can be reduced to checking whether a fragment on a slice is visible or not. Preintegrated volume rendering is a modiﬁcation of this slice-by-slice rendering (see Sect. 2.3.7) and makes use of slabs between two neighboring slices, i.e., the basic building blocks have a ﬁnite thickness. The extension of volume clipping in the context of pre-integrated volume rendering was introduced by R¨ ottger, Guthe, Weiskopf, Ertl, and Straßer [356] and the following discussion is based on their paper.3 To combine preintegration with volumetric clipping, two steps are necessary: (a) a modiﬁcation of the scalar data values within a clipped slab, and (b) an adaptation of the length of a ray segment. In what follows, a volumetric description of the clip geometry is used via a signed distance ﬁeld (as in Sect. 2.5.2). Once again, the clipping surface is assumed to be located at isovalue 0.5. Let df and db be the distance values of the clip volume at the entry and exit points of the ray segment (i.e., of the slab). If both distance values are below or above 0.5, the complete slab is either invisible or visible and no special treatment is necessary. Considering the case df < 0.5 and db > 0.5 (as in Fig. 2.22), only the dark gray part of the volume has to be rendered. In this case, the front scalar data value sf has to be changed to the data value sf at entry point into the clipped region. Then, a lookup into the pre-integration table is performed with the parameters (sf , sb ) rather than with (sf , sb ). In general, the parameter sf is obtained by

[0.5 − df ] µ= , sf = (1 − µ)sf + µ sb . db − df The brackets denote clamping to the range [0, 1]. Similarly, the parameter sb is replaced by sb according to

[0.5 − db ] ν =1− , sb = (1 − ν)sf + ν sb . df − db If both clip values are below 0.5, the slab is completely invisible and the scalar values do not matter. The second issue is the reduced length L of the clipped ray segment. In general, the numerical integration depends on the parameters sf , sb , and L. The volume rendering integral (2.2) can be used to obtain the contribution 3

c The Eurographics Association 2003. Reprinted, with permission, from R¨ ottger, Guthe, Weiskopf, Ertl, and Straßer [356]

2.5 Volume Clipping

53

L

s ′f

sf

sb

df

db

slab L′

Fig. 2.22. Using slabs instead of slices for pre-integrated volume clipping. The scalar data values at the entry point and the exit point of the viewing ray are denoted by s f and s b , respectively. The corresponding distance values from the clip texture are d f and d b . The thickness of the slab is denoted by L; L is the length of the visible part of the slab. The dark gray region remains after volumetric clipping

for a ray segment. The scalar ﬁeld sL (t) is assumed to be linear between the entry point and exit point. The respective mathematical expressions are: sL (t) = sb + L C(sf , sb , L) =

t (sf − sb ) L L −

g˜(sL (t)) e

τ˜(sL (q)) dq

dt

t

0 −

T (sf , sb , L) = e

L

τ˜(sL (t)) dt

0

α = 1−T . C denotes the emission (as RGB color) and T the transparency of the segment. The reduced ray segment length L can be taken into account in the following way. The pre-integration table is still based on a constant ray segment length L. The visible fraction of the slab is denoted by b = L /L. Then, the transparency T of the clipped ray segment is the pre-integrated transparency T (associated with the original segment length L) raised to the bth power because L L τ˜(sL (t )) dt = b τ˜(sL (t)) dt , 0 −b

T = e

0

L 0

⎛ τ˜(sL (t)) dt

−

⎜ = ⎝e

L 0

⎞b τ˜(sL (t)) dt

⎟ b ⎠ =T .

2 Visualization of 3D Scalar Fields

¨ IMAGES: ROTTGER, GUTHE, WEISKOPF, ERTL, STRASSER [356] c The Eurographics Association 2003

54

(a)

(b)

¨ IMAGES: ROTTGER, GUTHE, WEISKOPF, ERTL, STRASSER [356] c The Eurographics Association 2003

Fig. 2.23. Comparison of volumetric clipping approaches: (a) slice-based, (b) slab-based. See also Color Plate 3

(a)

(b)

Fig. 2.24. Rendering of a bonsai without (a) and with (b) pre-integrated volumetric clipping (see also Color Plate 4)

A ﬁrst-order approximation is suﬃcient if slabs are reasonably thin. Furthermore, if self-attenuation is neglected, the emission of the clipped ray segment is given by C = bC. Instead of calculating the factors for the adjustment of scalar values, emission, and opacity in the fragment program, the factors for all combinations of the clip values df and db can be pre-computed and stored in a 2D dependent texture. The content of the RGBA channels of the dependent texture corre-

2.5 Volume Clipping

55

sponds to the respective adjustment factors. Within a fragment program, the scalar values can be adjusted by a single linear interpolation. A comparison of the rendering quality between the slab-based approach and the slice-based method from previous sections is depicted in Fig. 2.23. A sphere has been cut out of the Bucky Ball data set so that the holes of the carbon rings are visible as green spots. Both images have been rendered with only 32 slices or slabs. While the slices are clearly visible on the left, the slab-oriented method reproduces the clipped volume accurately, i.e., it is accurate along the viewing direction. As another example, Fig. 2.24 shows a bonsai [358] of size 2563 without and with pre-integrated volumetric clipping. 2.5.4 Clipping and Volume Illumination Volume illumination extends volume rendering by adding illumination terms. It is based on gradients of the scalar ﬁeld and works very well if clipping is not applied. The combination of volume shading and volume clipping, however, introduces issues of how illumination has to be computed in the vicinity of the clip object. Here, illumination should not only be based on properties of the scalar ﬁeld, but should rather represent the orientation of the clipping surface itself. On the other hand, the scalar ﬁeld should still inﬂuence the appearance of the clipping surface, for example, with respect to color and transparency. Figure 2.25 (b) demonstrates how lighting the volume and the clipping surface improves the perception of the spatial structure of the clip object. For comparison, Fig. 2.25 (a) shows the same data set without speciﬁc lighting on the clipping surface; it is diﬃcult to recognize the spherical cutting surface because of the inappropriate lighting conditions. This section discusses ways to combine illuminated clipping surfaces with shaded volumes in an appropriate way. As shown by Weiskopf, Engel, and Ertl [470], the depth-based and volumetric clipping techniques from Sects. 2.5.1 and 2.5.2 can be extended to take into account a consistent shading of the clipping surface. The following discussion is based on the paper by Weiskopf, Engel, and Ertl [470].4 An Optical Model for Clipping in Illuminated Volumes For a consistent combination of volume clipping and volume shading, the following requirements should be met. First, the illumination of the clipping surface should allow the viewer to recognize the orientation and shape of this surface. Second, the optical properties at the clipping surface should be compatible with the optical properties of the scalar data ﬁeld at the same position to avoid discontinuities in the visual appearance. Third, volume shading in regions distant from clipping surfaces should not be aﬀected by the clipping algorithm. Fourth, the generated images should be independent of the chosen sampling rate (e.g., slice distance) – up to improvements of accuracy by 4

c 2003 IEEE. Reprinted, with permission, from Weiskopf, Engel, and Ertl [470]

2 Visualization of 3D Scalar Fields

IMAGES: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

56

(a)

(b)

Fig. 2.25. Combining volume clipping and volume shading. Depth-based volume cutting with a spherical clip object is applied to a technical engine data set. (a) No speciﬁc lighting is applied at the cutting surface; (b) reveals lighting (e.g., a highlight) on the cutting surface by combining surface-based and volumetric shading. See also Color Plate 5

evaluating the volume rendering integral on a ﬁner level. In particular, from a mathematical point of view, the image of a clipped volume has to converge to a reasonable limit value for an inﬁnitely small sampling distance. Let us consider a ﬁrst approach to combining clipping and volume shading. The idea is to apply unmodiﬁed clipping techniques (e.g., from Sects. 2.5.1 and 2.5.2) to the shaded volume. Illumination is based on the gradient of the scalar ﬁeld and on the optical properties assigned by the transfer function. Afterwards, the shaded clipping surface is layered on top of the volume, applying illumination based on the normal vector of the surface. The optical properties and thus the material properties of the clipping surface are deﬁned by the same transfer function and scalar values as for volume shading. Furthermore, the same local illumination model is applied as for volume shading. The surface of the clip object can be pictured as a “skin” surrounding the actual volume, as illustrated in Fig. 2.26 (a). This approach meets the ﬁrst three requirements for a successful combination of clipping and volume shading. First, the lighting computation at the “skin” is based on the normal vector on the clipping surface. Therefore, the orientation of the cut can be recognized. Second, the appearance of the “skin” and the neighboring volume is compatible because identical material proper-

2.5 Volume Clipping

clip object

57

clip object

infinitesimally thin boundary

boundary of finite thickness

volume data set

volume data set

(a)

(b) IMAGES: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

Fig. 2.26. Combining volume shading for a volumetric data set with surface-based shading on the boundary of the clip geometry: (a) shows an inﬁnitesimally thin “skin” around the clip object; (b) illustrates the extension of the surrounding layer to ﬁnite thickness

ties are applied. Third, volume shading away from the clipping surface is not aﬀected at all. Unfortunately, this ﬁrst approach is not invariant if the sampling rate ∆x changes in the evaluation of the discretized version (see p. 14), I(D) ≈ I0

n

ti +

i=1

n i=1

n

gi

ti ,

(2.4)

j=i+1

of the continuous volume rendering integral (see p. 14), D I(D) = I0 T (s0 ) +

g(s)T (s) ds .

(2.2)

s0

In discretized form, the source term (see p. 14) gi = g(i∆x + s0 )∆x ,

(2.6)

converges to zero for ∆x → 0, i.e., the color contribution of an inﬁnitely thin 2D surface is zero. Similarly, the transparency (see p. 14) ti = e−τ (i∆x+s0 )∆x

(2.5)

converges to one for ∆x → 0 unless τ is inﬁnite. (The special case of completely opaque, surface-like regions in the volume – where transparency is zero and τ and g are delta distributions – are automatically handled by the “skin” and the following approach alike.) The underlying problem is that a continuous volumetric description of a 3D volume is combined with an inﬁnitely thin 2D hypersurface. In the original, continuous representation by the volume

58

2 Visualization of 3D Scalar Fields

rendering integral (2.2), the terms g and τ have no contribution if their support is of zero length. This problem can be overcome by modifying the ﬁrst approach in a way that the “skin” is broadened to a ﬁnite thickness. The volume is covered by a thick layer of which the illumination is governed by the normal vectors on the clipping surface. The inﬂuence of the orientation of the surface reaches into the volume for some distance, which could be pictured as “impregnating” the shaded volume with lighting information from the surrounding clipping surface. Figures 2.26 (a) and (b) compare both the original “skin” approach and the modiﬁed “impregnation” approach. The “impregnation” reaches only into the remaining volume; it may not change the geometric clipping (i.e, the visibility of voxels) but only the illumination in the layer region. The transition between clipping surface and interior volume is introduced into the volume shading model by extending the source term to g(x, ω) = E(x) + w(x)Ssrf (x, ω) + (1 − w(x)) Svol (x, ω) , with the weight function w(x), the surface-based illumination term Ssrf (x, ω), and the volumetric scattering term Svol (x, ω). From construction, the thickness of the layer is limited and therefore the support of the weight function w(x) has to be compact (i.e., closed and bounded). Moreover, the function values w(x) have to be from [0, 1] to ensure an aﬃne and convex combination of the two illumination terms. When the model is restricted to a uniform layer thickness, one important parameter – the “impregnation” thickness – is introduced. It is assumed that the thickness parameter is speciﬁed by the user. Another choice concerns the functional behavior of the weight itself. A linear behavior with respect to the distance from the clipping surface (i.e., a linear ramp), a step function, or a more sophisticated function could be used. A continuous or even smooth weight function has the advantage that illumination shows a continuous or smooth transition between the “impregnation” layer and the remaining parts of the volume. One further issue is the evaluation of the surface-related illumination part, Ssrf (x, ω), at positions that are distant from the clipping surface. In particular, what is the normal vector in those regions? The deﬁnition of a surface-based normal vector in the “impregnation” layer makes use of the concept of parallel transport. For each point in the layer, the closest point on the clipping surface is determined. The normal vector is evaluated at that point on the surface and then transported to the respective location within the layer. The material parameters can be directly computed from the optical properties of the scalar data set at any point in the layer. The “impregnation” approach meets all four requirements for an appropriate combination of clipping and volume shading. By introducing a ﬁnite boundary region, the contribution of the illuminated clipping surface to the volume rendering integral (2.2) is ﬁnite. The discretized version of the integral (2.4) converges to this contribution for successive reﬁnements of the sampling rate. Furthermore, volume shading in regions distant from clipping

2.5 Volume Clipping

59

surfaces is not aﬀected by clipping because of the compact support of the weight function. The ﬁrst two requirements are met due to the same arguments as for the “skin” approach. Combining Clipping and Volume Shading in an Object-Space Approach The above mathematical discussion of the “impregnation” approach leads to the following algorithm for combining clipping and volume shading. All considerations are based on object space. It is assumed that the clipping object is deﬁned as the isosurface for the isovalue zero on a signed distance volume (analogously to volumetric clipping in Sect. 2.5.2). Furthermore, the distance ﬁeld should be based on the Euclidean norm in 3D Euclidean space and the clipping surface should be non-selﬁntersecting and smooth. With these assumptions, a direct mapping of the above mathematical formulation can be accomplished. The generic weight function w(x) can be expressed in terms of distance volume vdistance (x) and an alternative weight function w(d) ˜ according to w(x) = w ˜ (vdistance (x)) .

(2.12)

The weight function w(d) ˜ has support [0, dmax ], where dmax describes the thickness of the “impregnation” layer. Function values w(d) ˜ are from [0, 1]. By the above construction, w(x) models a uniform thickness of the layer. Surface-based illumination also makes use of the distance volume. Not only is the signed distance stored but the gradient of the distance ﬁeld as well. The gradient is identical to the normal vector of a respective isosurface, i.e., the gradient represents the normal vector on the clipping surface. Furthermore, even in a small neighborhood (ε neighborhood) around the smooth clipping surface, the gradient reﬂects the normal vector that could be obtained by parallel transport from the isosurface. This speciﬁc property of a distance volume is caused by the fact that the gradient represents the direction to or from the closest point on the isosurface. This behavior is illustrated in Fig. 2.27. The maximum size of the neighborhood depends on the curvature of the clipping surface; the neighborhood shrinks with increasing curvature. If the “impregnation” layer is contained in these local neighborhoods, the surface-shading term Ssrf (x, ω) can be computed inside the layer by using the gradient of the distance ﬁeld and the optical properties of the scalar ﬁeld in that region. Therefore, a quite generic formulation is available to combine clipping and volume shading. This object-space approach can be incorporated in a wide class of volume rendering algorithms, such as ray casting or splatting. This section, however, is focused on texture-based rendering. Thus, it is demonstrated how the previously described depth-based and volumetric clipping techniques can be extended to allow for consistent volume and surface shading.

60

2 Visualization of 3D Scalar Fields respective gradient point inside layer

clipping surface

point on isosurface with respective gradient IMAGE: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

Fig. 2.27. Gradients in distance ﬁeld. The normal vector on an isosurface – on the clipping surface – is identical with the gradient of the distance ﬁeld. The gradient can also be used to compute the parallel-transported normal vector inside the layer

Volumetric Clipping and Volume Shading Volumetric clipping from Sect. 2.5.2 already models the clipping object by means of an isosurface through a distance ﬁeld. Accordingly, the above formulation can be directly mapped to volumetric clipping. The following minor changes need to be included in the original clipping algorithm. First, the texture representations of the volume data set and the distance ﬁeld are extended to include both scalar values and gradients. For example, an RGBA texture format can be employed to store the scalar value in alpha and the gradient in RGB. The gradient ﬁelds serve as a basis to evaluate the local illumination model. Second, a fragment program is used to evaluate local illumination for the original data set and the distance ﬁeld. The material properties for both volumes are based on the entries in the data set and the transfer function. Since lighting is not precomputed, the user may change lighting parameters, such as material properties or color and position of the light source, without redeﬁning the RGBA textures for the volume data set and the distance ﬁeld. Third, the weight function w(d) ˜ is evaluated per fragment to implement (2.12). The distance d is obtained from the distance ﬁeld. The thickness of the “impregnation” layer can be adapted by scaling d. A linear ramp behavior for w(d) ˜ can be realized by a linear interpolation with subsequent clamping at 0 and 1; a step function is implemented by a comparison operation. More sophisticated weighting functions need respective numerical operations or a dependent texture lookup in a lookup-table. The ﬁnal function value w(d) ˜ serves as weight for a linear interpolation between the lighting computation for the original data set and the distance ﬁeld. All other parts of the original clipping algorithm are not aﬀected. In particular, actual clipping is still based on a comparison between the entry in the distance ﬁeld and the isovalue of the clip geometry. Moreover, the number of

2.5 Volume Clipping

61

texture fetches is not increased – two textures are still suﬃcient for the data set and the distance ﬁeld. Figure 2.29 demonstrates the consistent combination of clipping and volume shading for an orbital data set. Figure 2.29 (a) shows the original data set, and Fig. 2.29 (d) is based on volumetric clipping. (Depth-based clipping, as shown in Figs. 2.29 (b) and (c), is explained in the following subsection.) Depth-Based Clipping and Volume Shading The ideas from the above object-space approach need some modiﬁcations before they can be applied to depth-based clipping. The main problem is that a measure for the distance from the clipping surface is not available. An approximation of the basic optical clipping model that does not require any distance information is proposed. An “impregnation” layer of ﬁnite thickness has been introduced to take into account that transparency ti and source term gi in (2.5) and (2.6) depend on slice distance. In particular, an inﬁnitely thin layer will have no eﬀect on the ﬁnal image when the sampling rate is very high. This observation leads to the following modiﬁed approach. The clipping boundary is not modeled as a thick layer but as a 2D hypersurface. Shading inside the volume is based on transparency values and source terms that are adapted to the slicing distance ∆x. Surface-based shading on the clip geometry, however, yields modiﬁed contributions to transparency tsrf = e−τ (ssrf )∆xsrf , and source term gsrf = g(ssrf )∆srf , where ssrf describes the location of the clipping surface and ∆srf represents the thickness of the original “impregnation” layer, which is not dependent on the sampling rate. Once again, the optical properties are deﬁned by the volume data set and the transfer function. To put it another way, geometrically the clip boundary is treated as 2D surface, whereas the contribution to the rendering integral comes from a thick virtual layer. This approach results in the following approximations of the original distance-based model. First, the terms g(ssrf ) and τ (ssrf ) are evaluated only at the clipping surface and thus surface-based transparency and source term are constant inside the “impregnation” layer. Second, the weight function w(d) ˜ is implicitly set to a step function. Third, the volume rendering integral has double contributions from the “impregnation” layer – both from the above terms tsrf and gsrf , and erroneously from the slice-based sampling of the volume data set. Fourth, the layer is rendered as a surface of inﬁnitesimal thickness. The eﬀects of these approximations depend on the layer thickness, rate of change of scalar values around the clipping surface, and opacity. In typical applications, the approximations are not noticeable to the user because

62

2 Visualization of 3D Scalar Fields

the layer thickness tends to be small and, more importantly, the opacity is quite high in regions that have strong illumination. A clipping surface of high opacity essentially hides parts of the volume that are further away. On the other hand, the following beneﬁts are introduced by surface-based shading. First, surface shading oﬀers higher visual quality than volume shading because of a better representation of normal vectors. Second, the normal vector is always well-deﬁned because no parallel transport is employed. The normal vector ﬁeld on the clipping surface does not even need to be continuous and thus clipping geometries with sharp edges and creases are possible. For the actual implementation, the following modiﬁcations are introduced into the original depth-based clipping techniques from Sect. 2.5.1. Essentially, pure slice-based volume rendering is extended to hybrid volume and surface rendering. The volume and the surface parts are interleaved according to the depth structure of the clip geometry, and so is the order of surface and volume rendering. Accordingly, multi-pass rendering approaches have to be used; single-pass rendering for convex volume cutting is not suitable because the surface part is contained within the volume part and cannot be separated. All other techniques from Sect. 2.5.1, however, are prepared to alternatingly render regions of the clipped volume and the surrounding clipping surfaces. Volume rendering and volume clipping are adopted from Sect. 2.5.1, with volume shading being added. Surface rendering is based on the tessellated representation of the clip object. Material properties are obtained from the 3D scalar data set and the transfer function, i.e., texture coordinates are assigned to the vertices of the surface in order to access the 3D texture of the data set. The normal vector is computed from the surface model. Fragment operations compute the modiﬁed source and transparency terms to obtain the contribution of the clipping surface to the image. Finally, this contribution is combined with the volume parts by blending into the frame buﬀer. Figures 2.28 and 2.29 demonstrate the consistent combination of clipping and volume shading. Depth-based clipping via shadow testing is applied to a medical CT data set in Fig. 2.28. A specular highlight on the clipping surface allows the viewer to recognize surface orientation. Here, the transfer function is chosen to render the head almost opaque, i.e., the image resembles surface-based graphics. In contrast, the visualization of the orbital data set in Fig. 2.29 reveals both transparent and opaque structures. Figure 2.29 (a) shows the original data set. Figures 2.29 (b) and (c) are based on depthbased clipping by means of shadow testing, whereas Fig. 2.29 (d) is based on volumetric clipping. This comparison demonstrates that the quality of the specular highlight is very high for surface-based shading in images (b) and (c). In contrast, the highlight in image (d) is based on the gradients of the clip volume and shows artifacts caused by an inaccurate reconstruction of the gradients and by quantization. In Fig. 2.29 (b), an extended version of surface shading is applied to speciﬁcally mark the clipping surface for a fast recognition of its shape, position, and orientation. An additional ambient lighting term makes the clipping object visible as a dark-gray surface.

63

IMAGE: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

2.5 Volume Clipping

Fig. 2.28. Depth-based clipping in an illuminated CT data set (see also Color Plate 6)

2.5.5 Implementation An example implementation of slice-based volume clipping (Sects. 2.5.1 and 2.5.2) is based on C++, OpenGL, and GLUT. The implementation makes use of NVIDIA’s OpenGL extensions [210] for GeForce 3/4. The performance measurements in Table 2.1 were conducted on a Windows XP PC with Athlon XP 1800+ CPU and NVIDIA GeForce 4 Ti 4600 GPU with 128 MB texture memory. The size of the data set was 2563 and the viewport sizes were 5122 or 10242 . A slicing approach for a 3D texture was chosen, pre-classiﬁcation by paletted textures was applied, and volume shading was disabled. The performance numbers show that clipping based on a voxelized geometry causes only modest additional rendering costs. The performance of depthbased clipping depends on the chosen implementation. On GeForce 3/4, the shadow-testing approach is faster than the approach based on view frustum clipping. The main reason is the larger number of texture stages required by depth clipping. Because of the rather restricted functionality of the GeForce 3/4, a shift of depth values needs three texture stages. On the other hand, a lookup in a depth texture combined with a shadow test can be handled by a

2 Visualization of 3D Scalar Fields

(a)

(b)

(c)

(d)

IMAGES: WEISKOPF, ENGEL, ERTL [470]

c 2003 IEEE

64

Fig. 2.29. Clipping in an illuminated orbital data set: (a) without clipping; (b) and (c) depth-based clipping; (d) volumetric clipping. The clipping surface is additionally marked in dark gray in (b). See also Color Plate 7

2.6 Hierarchical Volume Visualization on GPU Clusters

65

Table 2.1. Performance measurements for volume clipping techniques in frames per second (NVIDIA GeForce 4 Ti 4600, 128 MB) [470] Viewport size

5122

10242

No clipping Voxelized clip object Depth-based probing (depth clipping) Depth-based cutting (depth clipping) Depth-based probing (shadow test) Depth-based cutting (shadow test)

28.0 15.3 8.5 7.1 18.5 12.2

8.8 5.2 2.5 2.2 5.6 3.4

single texture stage. Another reason is that data can be directly copied from the z buﬀer into a depth texture, whereas the high-resolution texture for the depth-clipping approach can only be ﬁlled by data from main memory. Moreover, the early z test can improve the performance of shadow-based probing. Depth-based cutting by means of shadow testing is slower than shadow-based probing because a second depth texture has to be used. For volume shading, the above relative performance behavior with respect to diﬀerent clipping approaches remains roughly the same; however, the overall performance is reduced due to additional texture and fragment operations for post-classiﬁcation and illumination. An example implementation of pre-integrated clipping (Sect. 2.5.3) is based on C++ and DirectX 9.0. A Shader Model 2 compliant GPU is required. The prototype was implemented and tested on an ATI Radeon 9700 GPU. The main functionality is included in a fragment program that provides data access and interpolation, pre-integrated classiﬁcation, and modiﬁcations for clipping. The complete source code (assembler-like pixel shader code) can be found in [356].

2.6 Hierarchical Volume Visualization on GPU Clusters While the previous sections focus on improvements for single-GPU volume rendering, this section addresses the issue of large-data visualization on GPU clusters. Large-data visualization is becoming more and more important because, for example, the increasing resolution of medical CT scanners leads to increasing sizes of scalar data sets, which can be in the range of gigabytes. Even more challenging is the visualization of time-dependent CFD simulation data that can comprise several gigabytes for a single time step and several hundred or thousand time steps. Parallel visualization can be used to address the issues of large data processing in two ways: both the available memory and the visualization performance are scaled by the number of nodes in a cluster computer.

66

2 Visualization of 3D Scalar Fields

The idea is to combine the “traditional” beneﬁts of parallel computing with the high performance oﬀered by GPU-based techniques. The following approach is based on hierarchical wavelet compression, which is adapted to the distributed-memory architecture of a cluster computer to increase the eﬀective size of volumes that can be handled. This visualization method was proposed by Strengert, Magall´ on, Weiskopf, Guthe, and Ertl [402] and the following discussion is based on their paper.5 Here, the wavelet hierarchy is also used for an adaptive, texture-based volume rendering approach for a PC cluster. An advanced compositing scheme takes into account the footprints of volume bricks to minimize costs for reading from frame buﬀer, network communication, and blending. Finally, performance numbers for diﬀerent combinations of parameters are included to clarify performance and scaling characteristics. 2.6.1 Previous and Related Approaches to Large-Data Volume Visualization Previous and related techniques to large-data volume rendering are brieﬂy reviewed in this section. The ﬁrst class of techniques considers a single GPU only. Bricking (see also Sect. 2.4) is a simple method to handle large data sets that do not ﬁt into texture memory at once [219, 237, 451]. Bricking, however, overcomes the limitations of texture memory only at cost of lower speed due to the permanent transfer of data over the AGP or PCI Express bus. Adaptive techniques aim at actually reducing the texture memory needed. They subdivide the volume to represent volumes of interest at high resolution and other parts of the volume at lower resolution. For example, an octree structure can be traversed in a top-to-down manner to ﬁnd the ﬁrst nodes that deliver a suﬃciently high quality [237]. Special care has to be taken at the boundary of sub-blocks to avoid artifacts at the transition zone between diﬀerent resolutions [462]. In a related approach, a variable-resolution texture model with an octree structure is utilized to render large volume data [24]. Packing techniques use sub-blocks of diﬀerent sizes and pack non-empty blocks to save memory for empty sub-regions. In the context of GPU-based volume rendering, Kraus and Ertl [227] implemented this idea by means of fragment programs with texture indirections (dependent textures). A related method by Li and Kaufman [248] uses a vertex program to realize volume packing. Vector quantization is another well-known compression technique that can be applied in GPU-based volume visualization. Here, the volume is partitioned into sub-blocks of same size. Each sub-block holds one index pointing into a code book that represents the actual data. Therefore, the data set can be decompressed during volume rendering by applying a dependent texture lookup in the code book [379]. Another volume compression technique is used by Rosa et al. [350], who developed a system for the visualization of time-varying volume data from 5

c The Eurographics Association 2004. Reprinted, with permission, from Strengert, Magall´ on, Weiskopf, Guthe, and Ertl [402]

2.6 Hierarchical Volume Visualization on GPU Clusters

67

thermal ﬂow simulations. The system is based on the work of Lum et al. [263], who apply quantization and lossy compression to texture data by means of a discrete cosine transformation and then store the result as indexed texture. A texture represented in this way can be decoded in graphics hardware by changing the texture palette. Binotto et al. [21] have recently presented a system that uses a hierarchical representation, but is oriented toward the compression of time-dependent, highly sparse and temporally coherent data sets. Their algorithm uses fragment programs to perform the decompression of the data sets. Finally, wavelet compression can be used to compress large data sets and build a hierarchy of diﬀerent resolutions. Guthe and co-workers [147, 148] show how a wavelet hierarchy can be constructed in a preprocessing step and then used during adaptive volume rendering on a single GPU. Their work is the basis for the volume compression scheme of this chapter – a more detailed description of the wavelet algorithm can be found in Sect. 2.6.4. Another line of research focuses on parallelization to display a large amount of data. In general, intermediate results of distributed rendering have to be composited into a single ﬁnal image. Parallel rendering approaches can be classiﬁed as sort-ﬁrst, sort-middle, or sort-last. In the ﬁrst category, graphical primitives (e.g., triangles) are sorted and then distributed to the diﬀerent render nodes. A sort-middle method distributes data after geometric transformations and lighting, but before rasterization. Sort-last methods apply sorting after rasterization. WireGL [173] and Chromium [174] are generic systems that enable distributed rendering for existing OpenGL applications. To speed up the per-pixel compositing in sort-last approaches, special-purpose hardware can be used, such as Sepia [292], Lightning-2 [400], or Metabuﬀer [23]. Speciﬁcally targeting volume rendering, Garcia et al. [130] use a combination of object-space partitioning and image-space partitioning while Magall´ on et al. [266] employ object-space partitioning with a sort-last approach. Another implementation of volume rendering builds upon the Sepia-2 architecture [257]. In the related ﬁeld of indirect volume visualization, Zhang et al. [507] describe parallel isosurface extraction and rendering with the Metabuﬀer system. 2.6.2 Distributed Visualization In this section, a sort-last strategy [293] is used to distribute the visualization process in a cluster environment. With increasing size of the input data set, this sorting scheme is favorable, since the input data becomes larger than the compositing data and a static partitioning in object space avoids communication regarding the scalar ﬁeld during runtime. The basic structure of the implementation is based on the aforementioned parallel volume rendering system by Magall´ on et al. [266], which originally did not support adaptive and hierarchical representations. During a preprocessing step object-based partitioning is performed to split the input data set into multiple, identically sized sub-volumes, depending on

68

2 Visualization of 3D Scalar Fields

the number of nodes in the cluster conﬁguration. To overcome possible memory limitations in connection with large data sets, this step is executed using the same set of nodes as the subsequent render process. The render loop is entered when all sub-volumes are created and transferred to their corresponding nodes. Rendering can be split into two consecutive tasks. The ﬁrst task is to render each brick separately on its corresponding node. An intermediate image is generated by texture-based direct volume visualization with screenaligned slices (see Sect. 2.3.1). By adapting the model-view matrix for each node, it is ensured that each sub-volume is rendered at its correct position in image space. Since the partitioning is performed in object space, the rendering process of diﬀerent nodes can produce intermediate images that overlap in image space. The second task blends these images and takes into account that multiple nodes can contribute to a single pixel in the ﬁnal image. The distributed images are depth sorted and processed through a compositing step based on alpha blending. To this end, each node reads back its frame buﬀer, including the alpha channel, and sends it to other nodes. To take advantage of all nodes for the computationally expensive alpha blending, direct send is used as compositing scheme [305]. Each intermediate result is horizontally cut into a number of stripes that match the total number of nodes. All these regions are sorted and transferred between the nodes in a way that each node receives all stripes of a speciﬁc area in the image space. Then, each node computes an identically sized part of the ﬁnal image. The alpha blending of the intermediate images is completely performed on the CPU. Although the GPU is highly specialized for this task, the additional costs for loading all stripes into texture memory and reading back the information after blending would lead to a lower overall performance. Instead, an optimized MMX [326] code is used to determine the result of the blend function for all four color channels of a pixel simultaneously (details are given in [402]). This approach can also handle time-dependent scalar ﬁelds without major changes. During the bricking process a static partitioning scheme is used for all time steps, i.e., each sub-volume contains the complete temporal sequence for the corresponding part of the input volume. To synchronize all nodes the information regarding the current time step is broadcast to the render nodes. 2.6.3 Accelerated Compositing Scheme The overall performance of the distributed visualization system may be limited by three factors (besides actual volume rendering): the process of reading back the results from the frame buﬀer, the data transfer between nodes, and the compositing step. In the following, these issues are addressed by minimizing the amount of image data to be processed. The key observation is that the image footprint of a sub-volume usually covers only a fraction of the intermediate image. For the scaling behavior, it is important to note that the relative size of the footprint shrinks with an increasing number of nodes. For

2.6 Hierarchical Volume Visualization on GPU Clusters

69

simplicity, an upper bound for the footprint is determined by computing the axis-aligned bounding box of the projected sub-volume in image space. Since the time needed to read back a rectangular region from the frame buﬀer depends almost linearly on the amount of data, reducing the size of the region leads to a performance increase of this part of the rendering process. Similarly, the communication speed also beneﬁts from the reduction of image data. The compositing step is accelerated by avoiding unnecessary alpha blending operations for image regions outside the footprints. Similarly to SLIC (Scheduled Linear Image Compositing) [401], a line-based compositing scheme is employed. For each line, the span containing already blended data is tracked. Since the images are blended in the depth-sorted order of their corresponding volume blocks and all blocks together represent the convex shape of the non-partitioned volume, the tracked region always forms one segment instead of multiple separated spans. If a projected volume face is parallel to the image plane, depth sorting results in an ambiguous ordering that may break this property. In this case, the topology is considered to ensure the connectivity of the marked span. With this information the new image data of the next compositing step can be separated into a maximum number of three segments. Two segments contain pixels that map into the region outside the marked span. These pixels need no further processing and can be copied into the resulting image. The remaining segment maps into an area where already other color information resides and alpha blending has to be performed. An example of this procedure is given in Fig. 2.30. After one iteration, the size of the span containing data needs to be updated and the next image stripe can be processed. In doing so, only a minimal amount of blending operations for a given volume partitioning is carried out. 2.6.4 Hierarchical Compression and Adaptive Rendering Even with distributed rendering techniques the size of a data set can exceed the combined system memory of a cluster conﬁguration and even a brick of the data set may be larger than one single node can handle. Another challenge is to further improve the rendering speed. The memory issue is addressed by using a hierarchical compression technique, and the performance issue by adaptive rendering. Single-GPU Wavelet Compression A single-GPU visualization approach is adopted that utilizes compression for large data sets [148]. The idea is to transform the input data set into a compressed hierarchical representation in a preprocessing step. An octree structure is created with the help of wavelet transformations. The input data set is split into cubes of size 153 , which serve as starting point for recursive preprocessing. Eight cubes that share one corner are transformed at a time using linearly interpolating spline wavelets. The resulting low-pass ﬁltered portion

70

2 Visualization of 3D Scalar Fields

´ WEISKOPF, GUTHE, ERTL [402] IMAGE: STRENGERT, MAGALLON,

c The Eurographics Association 2004

Fig. 2.30. Depth-sorted blending of footprints of four volume blocks. The regions with and without the need for blending are marked for each compositing step

is a combined representation of the eight input cubes with half the resolution of the original data. The size of this portion is 153 voxels again. The wavelet coeﬃcients representing high frequencies replace the original data of the eight input blocks. After all cubes of the original data set are transformed, the next iteration starts with the newly created low-pass ﬁltered cubes as input. The recursion stops as soon as the whole volume is represented through one single cube. This cube forms the root node of the hierarchical data structure and is the representation with the lowest quality. Except for the root node, all other nodes hold only high-pass ﬁltered data, which is compressed through an arithmetic encoder [147]. Although it is possible to increase the compression ratio by thresholding, a lossless compression is used for best visualization results in this chapter. During rendering an adaptive decompression scheme is used that depends on the viewing position and the data set itself. Starting at the root node of the hierarchical data structure, a priority queue determines which parts of the volume are decompressed next. Depending on the ratio between the resolution of a volume block and the actual display resolution, regions closer to the viewer are more likely decompressed than others. Additionally, an error criterion describing the diﬀerence between two representations of varying quality is used to identify regions that can be rendered in low quality without

2.6 Hierarchical Volume Visualization on GPU Clusters

´ WEISKOPF, GUTHE, ERTL [402] IMAGE: STRENGERT, MAGALLON,

71

c The Eurographics Association 2004

Fig. 2.31. Texture interpolation at a sub-volume border for a 1D case. Texels on one side of the border (white cells) are ﬁlled with previously transferred information of the neighboring node. Errors arise if the quality level of the neighboring node is unknown and hence a wrong level is chosen. For the incorrect case, border information of level 0 is used for interpolation although the rendering of the neighboring node is performed on level 1

noticeable artifacts. After the quality classiﬁcation is completed, the required blocks are decompressed by the CPU and then transferred to the graphics board’s texture memory for rendering. Depending on the reconstructed quality level of a block, the number of slices used for rendering is determined. With increasing reconstruction quality the number of slices increases as well, delivering higher quality for areas closer to the viewer. Additionally, a cache strategy is used to avoid the expensive decompression step for recently processed blocks. Unnecessary texture transfers are avoided by tracking already loaded textures. Extension to Parallel Rendering In a distributed visualization system, this approach leads to a problem concerning correct texture interpolation between sub-volumes rendered on different nodes. A typical solution is to create sub-volumes with an overlap of one voxel. With multi-resolution rendering techniques it is necessary to know not only the border voxels of the original data set but also the data value at the border of all other used quality levels [462]. This information can be determined in the preprocessing step. After creating the sub-volumes and constructing the hierarchical data structure, each node transfers the border information of all quality levels to its appropriate neighbors. But even with this information available on each node, a correct texture interpolation cannot be generated easily. As illustrated in Fig. 2.31, the remaining problem is to determine the quality level used for rendering a neighboring node in order to choose the correct border information. Since communication between nodes suﬀers from network latency, requesting information from a neighboring node is not suitable. Another approach is to compute the quality classiﬁcation on each node for an expanded area. Unfortunately, this is also impractical because the quality classiﬁcation is dependent on the volume data.

72

2 Visualization of 3D Scalar Fields

´ WEISKOPF, GUTHE, ERTL [402] IMAGE: STRENGERT, MAGALLON,

c The Eurographics Association 2004

Fig. 2.32. In the left part of the image, the volume is rendered using diﬀerent quality levels for each of the two sub-volume blocks. Assuming identical classiﬁcation for interpolation leads to visible artifacts as seen in the left magniﬁed area. For comparison, the right image is rendered with identical classiﬁcation for the subvolume blocks

Instead, an approximate solution is proposed which assumes that there are no changes in quality classiﬁcation at the border of the sub-volumes. With this approach errors only occur if diﬀerent qualities are used on each side of a subvolume border (see the example visualization in Fig. 2.32). Due to the similar position of adjacent parts of the sub-volumes it is likely that both regions are classiﬁed with the same quality. Experimental data that shows the error rates for the approximate solution is given in Table 2.2 for both the nonweighted number of transitions (i.e., just the number of transitions) and the area-weighted ratio (i.e., transitions multiplied by the attached area). The measurement was performed while rendering the Visible Human male data set on 16 rendering nodes (Fig. 2.34). In this conﬁguration, a total number of 185, 212 cube transitions is present in the whole data set. Only 723 cube borders show transitions that lead to an interpolation error, i.e., less than one percent of the total amount of transitions is aﬀected. For a correct solution of the interpolation problem, another approach is proposed that separates the rendering process and the computation of quality classiﬁcation. In each frame, an adaptive classiﬁcation is determined, but the associated rendering is delayed by one frame. Therefore, the information about the currently used quality levels can be transferred to neighboring nodes at the same time when intermediate images are distributed for compositing. Since at this time communication between all nodes must be performed anyway, the additional data can be appended to the image data. Having the transferred data available, the rendering process can produce a properly interpolated visualization during the next frame. The downside is that the latency between user interaction and the system’s reaction is increased by one frame. To avoid this, a hybrid technique that exploits both described approaches is possible. While the viewing parameters are changed, the approximate solution is used to generate an image without increased latency times during user interac-

2.6 Hierarchical Volume Visualization on GPU Clusters

73

Table 2.2. Portion of changes in quality classiﬁcation at block faces [402] non-weighted area-weighted total volume same quality diﬀerent quality

89.8% 10.2%

81.7% 18.3%

sub-volume borders only same quality 91.2% diﬀerent quality 8.8%

83.0% 17.0%

borders compared to total volume same quality 99.6% 99.1% diﬀerent quality 0.4% 0.9%

tion. As soon as the camera parameters are kept constant, a correct image is rendered based on the quality classiﬁcation that is transferred from the previous rendering step. Thus, a fast user interaction is combined with a correct sub-volume interpolation for the static case. 2.6.5 Implementation and Results The implementation is based on C++ and OpenGL. Volume rendering adopts post-classiﬁcation, which is realized either through NVIDIA’s register combiners or alternatively through an ARB fragment program. MPI is used for all communication between nodes. Two diﬀerent cluster environments were used for the following evaluation. The ﬁrst one is a 16-node PC cluster. Each of the nodes runs a dual-CPU conﬁguration with two AMD 1.6 GHz Athlon CPUs, 2 GB of system memory, and NVIDIA GeForce 4 Ti 4600 (128 MB) graphics boards. The interconnecting network is a Myrinet 1.28 GBit/s switched LAN providing low latency times. Linux is used as operating system, the SCore MPI implementation drives the communication [323]. The second environment consists of standard PCs with a Gigabit Ethernet interconnection. Each node has an Intel Pentium 4 2.8 GHz CPU and 4 GB system memory. The installed graphics boards are a mixture of NVIDIA GeForce 4 Ti 4200 and GeForce 4 Ti 4600, both with 128 MB memory. Running Linux, the MPI implementation LAM/MPI is used for node management and communication [238]. Three diﬀerent large-scale data sets are used to evaluate the performance of the implemented visualization system. If not stated otherwise, all measurements were performed on the cluster interconnected through Myrinet. The ﬁrst data set is an artiﬁcial scalar ﬁeld of size 10243 , showing a radial distance volume that is distorted by Perlin noise [328] (Fig. 2.33). The second data set is derived from the anatomical RGB cryosections of the Visible Human male data set [415]. The slices are reduced to 8 bit per voxel and cropped to exclude

2 Visualization of 3D Scalar Fields

Fig. 2.33. Radial distance volume combined with Perlin noise. A high-frequency transfer function is applied. See also Color Plate 8

´ IMAGES: STRENGERT, MAGALLON, WEISKOPF, GUTHE, ERTL [402] c The Eurographics Association 2004

74

Fig. 2.34. Parallel volume rendering of anatomical cryosections through the Visible Human male data set. The whole body is rendered with a total of 16 nodes. This image shows the result of four render nodes that display the upper part of the body. See also Color Plate 9

2.6 Hierarchical Volume Visualization on GPU Clusters

´ WEISKOPF, GUTHE, ERTL [402] IMAGES: STRENGERT, MAGALLON,

75

c The Eurographics Association 2004

Fig. 2.35. Visualization of the time-dependent CFD simulation. From top to bottom, timesteps 0, 45, and 89 are shown. See also Color Plate 10

external data like gray-scale cards and ﬁdual markers. The obtained data set has a resolution of 2048 × 1024 × 1878 voxels (Fig. 2.34). The third data set is a time-dependent CFD simulation of a ﬂow ﬁeld with increasing turbulence. The sequence contains a total of 89 time bins, each sized 2563 (Fig. 2.35). The Visible Human male data set can be visualized on a 10242 viewport using 16 nodes with 5 frames per second. The quality classiﬁcation is set to use the original resolution for most regions. Due to the uniform characteristics of the surroundings, these areas are displayed in a lower resolution without any noticeable artifacts. With a viewport of half size in each dimension, the obtained frame rate increases to 8 frames per second. Conﬁgurations with between two and 16 render nodes were measured to investigate the scaling behavior of the visualization system. The gigacube that contains the perturbed radial distance ﬁeld (Fig. 2.33) is used for all these tests. Results are shown in Fig. 2.36. For a 16-node conﬁguration, the data set can be rendered in 174 ms, which corresponds to a refresh rate of 5.7 Hz.

76

2 Visualization of 3D Scalar Fields 350

time in ms

300

250

200

150 2

4

6

8

10

12

14

16

Number of nodes

Fig. 2.36. Scalability of the visualization system with the number of rendering nodes 450

full resolution 1/2 resolution 1/4 resolution

400

time in ms

350 300 250 200 150 100 50 0

10

20

30

40 50 timestep

60

70

80

´ WEISKOPF, GUTHE, ERTL [402] BOTH IMAGES: STRENGERT, MAGALLON,

Fig. 2.37. Fig. 2.35

90

c The Eurographics Association 2004

Rendering performance for the time-dependent data set shown in

Figure 2.37 shows the results for the time-dependent data set. The test was performed using three diﬀerent quality levels. In case of the original (highest) quality, the required time clearly increases toward the end of the sequence. The reason for this behavior is found in the characteristics of the data set, which becomes more and more turbulent over time, leading to a higher amount of blocks that have to be decompressed. Furthermore, the cache becomes invalid with a progress in time and all blocks have to be decompressed starting at the root node. Therefore, rendering is rather slow for time-dependent data sets compared to the static ones. Using the second cluster environment with

2.7 Summary

77

its 8 nodes, only 2 frames per second are achieved for rendering the distance volume. Due to the similar conﬁguration of each node this gap is solely caused by the Gigabit Ethernet in comparison to Myrinet. The limiting factor for this type of application is the latency rather than the bandwidth limitation of the used network. The Myrinet conﬁguration clearly outperforms a conventional Gigabit Ethernet regarding latency times although both network technologies provide comparable bandwidth.

2.7 Summary This chapter has focused on eﬃcient GPU-based volume rendering. It has been demonstrated that all parts of the volume rendering pipeline can beneﬁt from the usage of GPUs. Traditionally, graphics hardware is used for eﬃcient interpolation (in 2D and 3D textures), classiﬁcation (by dependent textures for transfer functions), shading (by fragment operations), and compositing (by alpha blending). Real-time volume rendering is very important because it is a prerequisite for interactive volume visualization. Interaction is crucial for a user-centric exploration process in which the large parameter space for rendering and, possibly, the generation of the underlying data set (e.g., simulation or measurement parameters) has to be investigated. In particular, this chapter has provided details on two speciﬁc aspects of direct volume visualization: ﬁrst, volume clipping and, second, large-data visualization by applying compression techniques, adaptive rendering, and parallel rendering on GPU clusters. In this work, volume clipping is considered as an example of the ﬁltering step of the visualization pipeline because it provides a reduction of the input data by restricting the domain of the scalar ﬁeld. Clipping facilitates the selection of regions of the data set on a purely geometric basis and, therefore, is an important interaction element for a user-controlled examination. A free choice of the shape of the clip geometry plays a substantial role – in combination with the support for real-time rendering. Two alternative GPU approaches for eﬃcient volume clipping have been presented in Sect. 2.5: an image-space approach in which a depth structure is represented in 2D textures (Sect. 2.5.1) and an object-space approach that holds a voxelized clip geometry in a 3D texture (Sect. 2.5.2). When volume clipping and illumination have to be combined seamlessly, a modiﬁcation of the assignment of color and opacity values has to be introduced. Section 2.5.4 has shown how the corresponding extended optical model for volume rendering can be incorporated into both the image-space method and the object-space method. Moreover, it has been demonstrated that the advanced clipping techniques of this chapter can be used in combination with texture slicing and pre-integrated volume rendering alike (Sect. 2.5.3). Experiments with the two basic clipping approaches of this chapter indicate that each method has speciﬁc advantages and disadvantages. The

78

2 Visualization of 3D Scalar Fields

best-suited technique has to be chosen by the type of application and the requirements stated by the user. The main advantage of the volumetric clipping approach is that it provides an explicit control over each voxel of the clip volume. Therefore, arbitrarily complex objects can be used as clip geometry. The main disadvantage of the volumetric clipping approach is a possibly inaccurate reproduction of clip geometry. Nearest-neighbor sampling of the clipping texture results in visible, jaggy artifacts and a poor image quality. Trilinear interpolation, on the other hand, does not allow for arbitrary sharp geometric features of the clip object. For volume shading, the image quality can suﬀer further because of the sometimes inaccurate reconstruction of normal vectors from an interpolated gradient ﬁeld. Moreover, the voxelization approach prohibits changing the shape of a clip object in real time. The additional texture memory requirements for the clipping volume are another problem. An advantage of depth-based volume clipping is the high quality of the resulting images since clipping is performed with per-pixel accuracy. High image quality is even more noticeable for clipping in shaded volumes because the illumination of the clipping surface is derived from surface-based information and not from a volumetric gradient ﬁeld. Aliasing problems related to texture mapping cannot occur because there is a one-to-one mapping between pixels on the image plane and texels representing the clip geometry. Arbitrary changes of the clip geometry cause no additional costs because no voxelization is required. Depth-based volume clipping methods beneﬁt from the fact that the clip geometry has to be processed only once per frame. This is an important diﬀerence to the method by Westermann and Ertl [485], which needs to update the values in the stencil buﬀer for each slice by rendering the whole clip geometry. Their approach does not scale well for complex clip geometries or large volume data sets with many slices. The main problem of depth-based volume clipping is the costly handling of concave and complex clip geometries by means of multi-pass rendering. Therefore, depth-based clipping provides best performance for geometries with little depth complexity. Large-data visualization has been addressed by using a distributed rendering system for texture-based volume rendering (Sect. 2.6). By adapting a hierarchical wavelet compression technique to a cluster environment, the eﬀective size of volume data that can be handled is further improved. The adaptive decompression and rendering scheme results in a reduction of rendering costs, depending on the viewing position and the characteristics of the data set. The problem of texture interpolation at brick borders in connection with multi-resolution rendering has been addressed and diﬀerent solutions have been provided. Parts of the rendering process that are crucial to the system’s performance beneﬁt from the applied reduction of the processed region in image space, especially with increasing numbers of rendering nodes. The achieved performance is often restricted by the capabilities of the interconnection between rendering nodes and the computation of blending operations during the compositing step. With a viewport of size 10242 , this upper bound is approximately 11 frames per second for the tested cluster

2.7 Summary

79

conﬁguration with Myrinet network. To increase this upper limit, an exact calculation of the footprints instead of using a bounding box could be helpful. In the same way, the remaining unnecessary blending operations could be avoided and communication costs could be further reduced. In the case of time-dependent data sets, the performance is additionally bound by the decompression step because caching of decompressed blocks cannot be used in this context. Finally, a multi-bricking approach has been proposed to achieve constant frame rates for volume rendering with 3D textures, independent of the viewing positions (Sect. 2.4). Multi-bricking is complementary to the aforementioned distributed rendering of compressed data sets and can be combined with existing rendering methods.

3 Vector Field Visualization

Vector ﬁeld visualization is another important and classical topic in scientiﬁc visualization that has been the subject of active research for many years. Typically, data originates from numerical simulations – such as those of computational ﬂuid dynamics – or from measurements, and needs to be analyzed by means of visualization to gain an understanding of the ﬂow. With the rapid increase of computational power for simulations, the demand for more advanced visualization methods has grown. This chapter focuses on a recent class of vector ﬁeld visualization techniques: interactive texture-based methods. Texture-based techniques are well equipped to provide a dense representation of the vector ﬁeld – they essentially cover the complete domain by visual patterns that show the underlying vector data. The visual patterns are constructed from particle tracing, which computes the motion of massless marker particles along the vector ﬁeld. A dense representation has the advantage that the seed-positioning problem is automatically solved because particle traces are computed everywhere and, therefore, important features cannot be missed. All particle-tracing approaches for vector ﬁeld visualization focus on the mapping step of the generic visualization pipeline. Here, the mapping step comprises the numerical aspect of particle tracing, as well as the actual mapping of particle traces to renderable representations. Texture-based vector ﬁeld visualization greatly beneﬁts from an implementation on GPUs because a large number of texels has to be processed by uniformly structured numerical operations. Only through the advent of programmable graphics hardware, interactive dense vector ﬁeld visualization has become possible. Consequently, one important topic of this chapter is to address the eﬃciency of texture-based vector ﬁeld visualization. The mapping to intuitively understandable visual representations, however, plays an equally important role because the user – as a human being – plays a crucial role in the complete visualization process. Section 3.3, for example, explicitly discusses both the numerical eﬃciency and the eﬀective visual mapping for semi-Lagrangian texture advection.

82

3 Vector Field Visualization

Section 3.4 speciﬁcally addresses dye advection and solves the numerical diﬀusion problems that are inherent to traditional semi-Lagrangian advection schemes by using a level-set description. While the other parts of this chapter are restricted to Cartesian 2D or 3D domains, Sect. 3.5 extends texture-based vector ﬁeld visualization to curved surfaces. Finally, Sect. 3.6 presents a novel description of texture-based vector ﬁeld visualization that allows us to represent the various existing techniques within a uniﬁed framework. On the one hand, this uniﬁcation facilitates a better understanding of the relationship between the diﬀerent techniques, including their speciﬁc advantages and disadvantages. On the other hand, the ﬂexibility of the framework allows us to come up with novel visualization approaches by just modifying parameters of the framework. As outlined before, the vector ﬁeld visualization approaches of this chapter focus on the mapping step of the visualization pipeline. The ﬁltering step is not discussed here, but could be easily added just before the mapping step. For example, denoising a measured vector ﬁeld data set could be a useful ﬁlter module. Finally, the rendering step is rather simple for 2D vector ﬁelds because there is a natural mapping of a 2D texture-based vector ﬁeld representation to the 2D screen. Rendering is more sophisticated for 3D vector ﬁelds; here, volume rendering techniques – as introduced in Chap. 2 and discussed in Sect. 3.3.5 in detail – are applied.

3.1 Basics of Particle Tracing The notion of a vector ﬁeld can be deﬁned mathematically by making use of concepts from diﬀerential geometry and the theory of diﬀerential equations. More detailed background information on these topics can be found in textbooks on diﬀerential topology and geometry [167, 168, 239, 289]. Although this mathematical approach might seem quite abstract for many applications, it has the advantage of being a ﬂexible and generic description that is applicable to a wide range of problems. The following discussion is partly based on the work by Weiskopf and Erlebacher [471].1 3.1.1 Time-Dependent Vector Fields The deﬁnition of a vector ﬁeld is given ﬁrst. Let M be a smooth m-manifold with boundary, N be an nD submanifold with boundary (N ⊂ M ), and I ⊂ R be an open interval of real numbers. A map u : N × I −→ T M is a time-dependent vector ﬁeld provided that 1

c 2005, with permission from Reprinted from Weiskopf and Erlebacher [471], Elsevier

3.1 Basics of Particle Tracing

83

u(x, t) ∈ Tx M . An element t ∈ I serves as a description for time; x ∈ N is a position in space. T M is a tangent bundle – the collection of all tangent vectors, along with the information of the point of tangency. Finally, Tx M is the tangent space associated with x. The vector ﬁeld maps a position in space and time, (x, t), to a tangent vector located at the same reference point x. For a tangential time-dependent vector ﬁeld, the mapping remains in the tangent bundle T N and therefore does not contain a normal component, i.e., u : N × I −→ T N . For a non-tangential vector ﬁeld, a related tangential vector ﬁeld can be computed by projection from Tx M to Tx N , i.e., by removing the normal parts from the vectors. Integral curves are directly related to vector ﬁelds. Let u : N × I → T N be a continuous (tangential) vector ﬁeld. Let x0 be a point in N and J ⊂ I be an open interval that contains t0 . The C 1 map ξx0 ,t0 : J −→ N with

dξx0 ,t0 (t) = u (ξx0 ,t0 (t), t) dt is an integral curve for the vector ﬁeld with initial condition x = x0 at t = t0 . The subscripts in the notation of ξx0 ,t0 denote this initial condition. These integral curves are usually referred to as pathlines, especially in the context of ﬂow visualization. If u satisﬁes the Lipschitz condition, the above diﬀerential equation for ξx0 ,t0 has a unique solution. In all practical applications of ﬂow visualization, the data is given on a manifold of two or three dimensions. To investigate a vector ﬁeld on an arbitrary curved surface, the above formalism is necessary and, for example, the issue of describing the surface by charts has to be addressed. An alternative solution for vector ﬁelds on surfaces is described in Sect. 3.5. Very often, however, N is just a Euclidean space. This allows us to use a simpler form of the tangential vector ﬁeld given by ξx0 ,t0 (t0 ) = x0

and

u : Ω × I −→ Rn ,

(x, t) −→ u(x, t) .

The vector ﬁeld is deﬁned on the nD Euclidean space Ω ⊂ Rn and depends on time t ∈ I. Here, the reference point x is no longer explicitly attached to the tangent vector. In general, boldface letters are used to denote vectors in Euclidean space. In this modiﬁed notation, the integral curve is determined by the ordinary diﬀerential equation dxpath (t; x0 , t0 ) = u(xpath (t; x0 , t0 ), t) . dt

(3.1)

84

3 Vector Field Visualization

It is assumed that the initial condition xpath (t0 ; x0 , t0 ) = x0 is given at time t0 , i.e., all integral curves are labeled by their initial conditions (x0 , t0 ) and parameterized by t. By construction, the tangent to the pathline at position x and time t is precisely the velocity u(x, t). In more general terms, the notation x(t; x0 , t0 ) is used to describe any curve parameterized by t that contains the point x0 for t = t0 . 3.1.2 Lagrangian Particle Tracing Integral curves play an important role in visualizing the associated vector ﬁeld, and in understanding the underlying physics of the ﬂow. There exist two important additional types of characteristic curves: streamlines and streaklines. Pathlines, streamlines, and streaklines are identical in steady ﬂows. When the vector ﬁeld depends explicitly on time, these curves are distinct from one another. In the special case of a steady ﬂow, a streamline is deﬁned as a solution to dxstream (t; x0 , t0 ) = u(xstream (t; x0 , t0 )) , dt

(3.2)

which is identical to a pathline. In the general case of an unsteady ﬂow, a streamline is deﬁned as the particle trace through the instantaneous vector ﬁeld at a ﬁxed time τ . In other words, the vector ﬁeld is made artiﬁcially steady by virtually “freezing” it for one time. Accordingly, a streamline is generally deﬁned as the solution to dxstream (t; x0 , t0 ) = u(xstream (t; x0 , t0 ), τ ) . dt Here, t and t0 are just parameters along the curve and do not have the meaning of physical time, in contrast to the physical time τ . A third type of curve is produced by dye released into the ﬂow. If dye is released continuously into a ﬂow from a ﬁxed point x0 , it traces out a streakline. For example, smoke emanating from a chimney follows a streakline. It is instructive to derive integrated equations for pathlines, streamlines, and streaklines. The solution to the ordinary diﬀerential equation (3.1) is obtained by formal integration, which gives the pathline t u (xpath (s; x0 , t0 ), s) ds . (3.3) xpath (t; x0 , t0 ) = x0 + t0

Similarly, streamlines at time τ can be computed by t xstream (t; x0 , t0 ) = x0 + u (xstream (s; x0 , t0 ), τ ) ds .

(3.4)

t0

To obtain the snapshot of a streakline at time t, particles are released from x0 at times s ∈ [tmin , t] and their positions are evaluated at time t:

3.2 Classiﬁcation and Overview of Flow Visualization Methods

85

xstreak (s; x0 , t) = xpath (t; x0 , s) . The streakline is parameterized by s, and tmin is the ﬁrst time that particles are released. 3.1.3 Eulerian Particle Tracing The above approach to particle tracing adopts a Lagrangian point of view. Here, each single particle can be identiﬁed individually – labeled in some sense – and the properties of each particle are computed over time t. In the case of particle tracing, the properties – such as colors – can be subsumed by a function φ which does not change during time for a speciﬁc particle, i.e., the property function is time-independent: φLagrangian (x0 , t) = φLagrangian (x0 ). From an Eulerian point of view, the property φ is a function of position x and time t: the observer is located at a ﬁxed position x and measures changes of φ caused by diﬀerent particles that cross this position. As the property does not change for a speciﬁc particle, φ(x, t) = φ(x0 , t0 ) in the Eulerian approach if the spacetime positions (x, t) and (x0 , t0 ) belong to the pathline of the same particle, cf. (3.3). For a continuously diﬀerentiable property function, the advection equation can thus be obtained: ∂φ(x, t) + u(x, t) · ∇φ(x, t) = 0 . ∂t

(3.5)

In contrast to the Lagrangian approach, line-like structures are not directly available in an Eulerian description. Nevertheless, line-like patterns can be introduced by successive blending of Eulerian property ﬁelds or by convolution, respectively. Details are given in Sect. 3.3.4.

3.2 Classiﬁcation and Overview of Flow Visualization Methods There exist many diﬀerent vector ﬁeld visualization techniques, which can be distinguished according to their properties with respect to a number of categories. The following classiﬁcation should be rather considered as a collection of important issues than a complete taxonomy. In this overview section, the primary classiﬁcation scheme for vector ﬁeld visualization distinguishes techniques according to the relationship between a vector ﬁeld and its associated visual representation. Point-based direct visualization approaches take into account the vector ﬁeld at a point and possibly its neighborhood to obtain a visual representation. The vector ﬁeld is directly

86

3 Vector Field Visualization

mapped to graphical primitives in the sense that no sophisticated intermediate processing of data is performed. Another class is based on characteristic curves obtained by particle tracing. The third class thoroughly preprocesses data to identify important features, which then serve as a basis for the actual visualization. Another type of property is the density of representation: the domain can be sparsely or densely covered by visualization objects. Density is particularly useful for subclassing particle-tracing approaches. Related to density is the distinction between local and global methods. A global technique essentially shows the complete ﬂow, whereas important features of the ﬂow can be missed by a local technique. A visualization method is also inﬂuenced by the structure of the data. The dimensionality of the manifold on which the vector ﬁeld is deﬁned plays an important role. For example, strategies that work well in 2D might be much less useful in 3D because of perceptual issues: the recognition of orientation and spatial position of graphical primitives is more diﬃcult, and important primitives could be hidden by others. Dimensionality also aﬀects performance; a 3D technique has to process substantially more data. If visualization is restricted to slices or more general hypersurfaces of a 3D ﬂow, the projection of vectors onto the tangent spaces of the hypersurfaces has to be considered. Moreover, a distinction has to be made between time-dependent and time-independent data. A steady ﬂow usually is much less demanding since frame-to-frame coherence is easy to achieve and streamlines, streaklines, and pathlines are identical. Finally, the type of grid has to be taken into account. Data can be provided, for example, on uniform, rectilinear, curvilinear, or unstructured grids. The grid type mainly aﬀects the visualization algorithms with respect to data storage and access mechanisms or interpolation schemes. Ideally, the ﬁnal visual representation does not depend on the underlying grid. In what follows, the techniques are roughly ordered according to their classiﬁcation into: point-based direct methods, sparse representations for particle-tracing techniques, dense representations based on particle tracing, and feature-based approaches. The other types of properties are discussed along with the description of the individual methods. This classiﬁcation approach and the corresponding review of ﬂow visualization methods can also be found in a book chapter by Weiskopf and Erlebacher [471]. 3.2.1 Point-Based Direct Flow Visualization The traditional technique of arrow plots is a well-known example of direct ﬂow visualization based on glyphs. Small arrows are drawn at discrete grid points, showing the direction of the ﬂow and serving as local probes for the velocity ﬁeld; see Fig. 3.1 (a). In the closely related hedgehog approach, the ﬂow is visualized by directed line segments of which the lengths represent velocity magnitude. To avoid a possible distracting pattern for a uniform sampling by arrows or hedgehogs, their positions can be slightly perturbed by random

3.2 Classiﬁcation and Overview of Flow Visualization Methods

87

variations [96]. Arrow plots can be directly applied to time-dependent vector ﬁelds by letting the arrows adapt to the velocity ﬁeld for the current time. For 3D representations, the following issues have to be considered. The position and orientation of an arrow is diﬃcult to understand due to the projection onto the 2D image plane, and an arrow might occlude other arrows in the background. The problem of clutter can be addressed by highlighting arrows with orientations in a range speciﬁed by the user [26], or by selectively seeding the arrows. Illumination and shadows serve to improve spatial perception; for example, shadowing can be applied to hedgehog visualizations on 2D slices of a 3D ﬂow [215]. More complex glyphs [80] can be used to provide additional information on the ﬂow, as illustrated in Fig. 3.2. In addition to the actual velocity, information on the Jacobian of the velocity ﬁeld is revealed. The Jacobian is presented in an intuitive way by decomposing the Jacobian matrix into meaningful components and by mapping them to icons based on easily understandable metaphors. Typical data encoded into glyphs comprises velocity, acceleration, curvature, local rotation, shear, or convergence. Glyphs can also be used to represent information on the uncertainty of the vector ﬁeld data [493]. Glyphbased uncertainty visualization is also covered in [252, 320], which additionally discuss uncertainty representations for other visualization styles. Another strategy is to map ﬂow properties to a single value and apply techniques known from the visualization of scalar data. Typically, the magnitude of the velocity or one of the velocity components are used. For 2D ﬂow visualization, a mapping to color or to isolines (contour lines) is often applied. Volume visualization techniques have to be employed in the case of 3D data. Direct volume rendering, which avoids occlusion problems by selective use of semi-transparency, can be applied to single-component data derived from vector ﬁelds [106, 434]; recent developments are speciﬁcally designed for time-dependent data [59, 135, 314]. 3.2.2 Sparse Representations for Particle-Tracing Techniques Another class of visualization methods is based on the characteristic lines obtained by particle tracing. Among these are the aforementioned pathlines, streamlines, and streaklines. In addition, time lines, constructed from particles released at the same earlier time from diﬀerent points along a curve, can be used. All these lines are quite intuitive because they represent some kind of transport along the ﬂow. This section describes sparse representations, in which the spatial domain is not densely covered. A traditional particle-based approach computes characteristic curves and draws them as thin lines. Since many researchers handle time-independent vector ﬁelds, the notion of streamlines is used frequently. The visualization concepts can often be generalized to pathlines, streaklines, or time lines, even if not explicitly mentioned. Streamlines just serve as a role model for the

3 Vector Field Visualization

IMAGES A–C: WEISKOPF, ERLEBACHER [471]

c 2005 Elsevier

88

(a)

(b)

(c)

(d)

Fig. 3.1. Comparison of visualization techniques applied to the same 2D ﬂow: (a) arrow plot, (b) streamlets, (c) LIC, (d) topology-based (image (d) courtesy of Gerik Scheuermann)

other characteristic lines. Particles traced for a very short time generate short streamlines or streamlets. Streamlines and streamlets can be used in 2D space, on 2D hypersurfaces of an underlying 3D ﬂow, and for 3D ﬂows. Hypersurfaces typically are sectional slices through the volume, or they are curved surfaces such as boundaries or other characteristic surfaces. It is important to note that the use of particle traces for vector ﬁelds projected onto slices may be misleading, even within a steady ﬂow: a streamline on a slice may depict a closed loop, even though no particle would ever traverse the loop. The problem is caused by the fact that

89

IMAGE: WEISKOPF, ERLEBACHER [471]

c 2005 Elsevier

3.2 Classiﬁcation and Overview of Flow Visualization Methods

Fig. 3.2. Glyph-based 3D ﬂow visualization, combined with illuminated streamlines (see also Color Plate 11)

ﬂow components orthogonal to the slice are neglected during ﬂow integration. For 3D ﬂows, perceptual problems might arise due to distortions resulting from the projection onto the image plane. Moreover, issues of occlusion and clutter have to be considered. An appropriate solution is to ﬁnd selective seed positions for particle traces that still show the important features of the ﬂow, but do not overcrowd the volume; for example, a thread of streamlets along characteristic structures of 3D ﬂow can be used [254]. The method of illuminated streamlines [508], based on illumination in diverse co-dimensions [6], improves the perception of those lines. In addition, depth information is increased and the problem of occlusion is addressed by making the streamlines partially transparent. An example is shown in Fig. 3.2. In 2D, particle traces are usually represented by thin lines, although the width of a line is sometimes modiﬁed to represent further information. Figure 3.1 (b) shows an example of a collection of streamlets. In 3D applications, however, the additional spatial dimension allows more information to be encoded into the graphical representation by using geometric objects of ﬁnite extent perpendicular to the particle trace. Examples of such an extension of streamlines in 3D are streamribbons and streamtubes. A streamribbon is the area swept out by a deformable line segment along a streamline. The strip-like shape of a streamribbon displays the rotational behavior of a 3D ﬂow. Figure 3.3 shows a visualization of a 3D ﬂuid simulation combining streamribbons, streamlines, arrows, and color coding [384]. An iconic streamtube [431] is a thick tube-shaped streamline of which the radial extent shows

90

3 Vector Field Visualization

Fig. 3.3. Combination of streamlines, streamribbons, arrows, and color coding for a 3D ﬂow (courtesy of the BMW Group and Martin Schulz). See also Color Plate 12

the expansion of the ﬂow. As a further extension of streamtubes, dash tubes [129] provide animated, opacity-mapped tubes. Stream polygons [383] trace an arbitrary polygonal cross section along a streamline and thus are closely related to streamtubes and streamribbons. The properties of the polygons – such as size, shape, or orientation – reﬂect properties of the vector ﬁeld, including strain, displacement, and rotation. Streamballs [30] use their radii to visualize divergence and acceleration in a ﬂow. Instead of spheres, other geometric objects such as tetrahedra [409] may be used. Another extension of streamlines is provided by stream surfaces, which are everywhere tangent to the vector ﬁeld. A stream surface can be modeled by an implicit surface [439] or approximated by explicitly connecting a set of streamlines along time lines. Stream surfaces present challenges related to occlusion, visual complexity, and interpretation, which can be addressed by choosing an appropriate placement and orientation based on principal stream surfaces [44] or by user interaction [172]. Ray casting can be used to render several stream surfaces at diﬀerent depths [128]. Stream arrows [255] cut out arrow-shaped portions from a stream surface and thus provide additional information on the ﬂow,

3.2 Classiﬁcation and Overview of Flow Visualization Methods

91

such as ﬂow direction and convergence or divergence. Stream surfaces can also be computed and visualized via surface particles [438], which are subject to less occlusion than a full-bodied surface. The generalization of the concept of particles released from single discrete points (for streamlines or streaklines) or from several points on a 1D line (for stream surfaces) leads to ﬂow volumes [277]. A ﬂow volume is a region of a 3D ﬂow domain traced out by a 2D patch over time. The resulting volume can be visualized by volume rendering techniques. Since any ﬂow volume can be (at least approximately) represented by a collection of tetrahedral cells, volume rendering techniques for unstructured grids can be applied, such as GPU-accelerated cell projection [357, 393] (also see Sect. 2.3.6). Flow volumes can be extended to unsteady ﬂows [14], yielding the analogue of streaklines. Finally, time surfaces extend time lines to surfaces that are built from particles released from a 2D patch. The evolution of time surfaces can be handled by a level-set approach [486]. A fundamental issue of all particle-based techniques is an appropriate choice of initial conditions – seed point positioning – in order to catch all relevant features of the ﬂow. Two main strategies can be identiﬁed: interactive or automatic placement of seed points. The interactive approach leaves the problem to the user and, in this sense, simpliﬁes the problem from an algorithmic point of view. Nevertheless, the visualization system should be designed to support the user in identifying appropriate seed points. For example, the virtual wind tunnel [37] is an early virtual reality implementation of a ﬂow visualization system where particles can be interactively released by the user. A useful approach for the automatic placement of seed points is to construct a uniform distribution of streamlines, which can be achieved for 2D vector ﬁelds [193, 430] or for surfaces within curvilinear grids of a 3D ﬂow [269]. The idea behind a uniform distribution of streamlines is that such a distribution very likely will not miss important features of the ﬂow. Therefore, this approach can be regarded as a step toward a completely dense representation, which is discussed in the following section. Equally spaced streamlines can be extended to multi-resolution hierarchies that support an interactive change of streamline density while zooming in and out of the vector ﬁeld [196]. Moreover, with this technique, the density of streamlines can be determined by properties of the ﬂow, such as velocity magnitude or vorticity. Evenly spaced streamlines for an unsteady ﬂow can be realized by correlating instantaneous streamline visualizations for subsequent time steps [195]. Seeding strategies may also be based on vector ﬁeld topology; for example, ﬂow structures in the vicinity of critical points can be visualized by appropriately setting the initial conditions for particle tracing [445]. Since all particle-tracing techniques are based on solving the diﬀerential equation for particle transport, issues of numerical accuracy and speed must be addressed. Diﬀerent numerical techniques can be applied for the initial value problem of ordinary diﬀerential equations. In many applications, ex-

92

3 Vector Field Visualization

plicit integration schemes are used, such as non-adaptive or adaptive RungeKutta methods. The required accuracy for particle tracing depends on the visualization technique; for example, ﬁrst-order Euler integration might be acceptable for streamlets but not for longer streamlines. A comparison of diﬀerent integration schemes [410] helps to judge the trade-oﬀ between computation time and accuracy. Besides the actual integration scheme, the grid on which the vector ﬁeld is given is very important for choosing a particletracing technique. Point location and interpolation heavily depend on the grid and therefore aﬀect the speed and accuracy of particle tracing. Both aspects are detailed in [307], along with a comparison between C-space (computational space) and P-space (physical space) approaches. The numerics of particle tracing is discussed, for example, for the tetrahedral decomposition of curvilinear grids [206], especially for the decomposition of distorted cells [363], for unstructured grids [431], for analytical solutions in piecewise linearly interpolated tetrahedral grids [308], for stiﬀ diﬀerential equations originating from shear ﬂows [410], and for sparse grids [412]. 3.2.3 Dense Representations for Particle-Tracing Methods Another class of visualization approaches is based on a representation of the ﬂow by a dense coverage through structures determined by particle tracing. Typically, dense representations are built upon texture-based techniques, which provide images of high spatial resolution. A detailed description of texture-based and dense ﬂow visualization can be found in a survey paper by Sanna et al. [373], in a state-of-the-art report by Laramee, Hauser, Doleisch, Vrolijk, Post, and Weiskopf [240], and in an overview paper by Weiskopf and Scheuermann [482]. The distinction between dense and sparse techniques should not be taken too rigidly because both classes of techniques are closely related by the fact that they form visual structures based on particle tracing. Therefore, dense representations also lead to the same intuitive understanding of the ﬂow. Often, a transition between both classes is possible [444]; for example, texturebased techniques with only few distinct visual elements might resemble a collection of few streamlines and, on the other hand, evenly spaced streamline seeding can be used with a high density of lines. An early texture-synthesis technique for vector ﬁeld visualization is spot noise [437], which produces a texture by generating a set of spots on the spatial domain. Each spot represents a particle moving over a short period of time and results in a streak in ﬂow direction at the position of the spot. Enhanced spot noise [81] improves the visualization of highly curved vector ﬁelds by adapting the shape of the spots to the local velocity ﬁeld. Spot noise can also be applied on boundaries and surfaces [74, 413]. A divide-and-conquer strategy makes possible an implementation of spot noise for interactive environments [73]. As an example application, spot noise was applied to the visualization of turbulent ﬂow [75].

3.2 Classiﬁcation and Overview of Flow Visualization Methods

93

Line integral convolution (LIC) [43] is a widely used technique for the dense representation of streamlines in steady vector ﬁelds. It serves as a role model for the development of the visualization methods in this chapter. A LIC example is shown in Fig. 3.1 (c). LIC takes a vector ﬁeld and a white noise texture as input. The noise texture is locally smoothed along streamlines by convolution with a ﬁlter kernel. This ﬁltering leads to high correlation between the gray-scale values of neighboring pixels along streamlines, and to little or no correlation perpendicular to streamlines. The contrast and quality of LIC images can be improved by postprocessing techniques, such as histogram equalization, high-pass ﬁltering, or a second pass of LIC [313]. The basic LIC technique does not show orientation and magnitude of the velocity ﬁeld, an issue that is addressed by variants of LIC. Periodic motion ﬁlters can be used to animate the ﬂow visualization, and a kernel phase shift can be applied to reveal the direction and magnitude of the vector ﬁeld [43, 123]. Oriented Line Integral Convolution (OLIC) [457] exploits the existence of distinguishable, separated blobs in a sparse texture and smears these blobs into the direction of the local velocity ﬁeld by convolution with an asymmetric ﬁlter kernel to show the orientation of the ﬂow. In another approach, orientation is visualized by combining animation and adding dye advection [391]. Multi-frequency noise for LIC [214] visualizes the magnitude of the velocity by adapting the spatial frequency of noise. Other visualization techniques achieve LIC-like images by applying methods that are only indirectly related to line integral convolution. For example, fur-like textures [208] can be utilized by specifying the orientation, length, density, and color of fur ﬁlaments according to the vector ﬁeld. Another method employs texture generation according to a reaction diﬀusion model [370], which can be also implemented on GPUs [371]. The integrate and draw [347] approach deposits random gray-scale values along streamlines. Pseudo LIC (PLIC) [444] is a compromise between LIC and sparse particle-based representations and therefore allows for a gradual change between dense and sparse visualizations. PLIC uses LIC to generate a template texture in a preprocessing step. For the actual visualization, the template is mapped onto thin or thick streamlines, thus ﬁlling the domain with LIC-like structures. The idea of thick streamlines can be extended to an error-controlled hierarchical method for a GPU-accelerated level-of-detail approach [25]. LIC can be extended to non-uniform grids and curved surfaces, for example, to curvilinear grids [123], and to 2D unstructured or triangular meshes [12, 271, 411]. Multi-granularity noise as input for LIC [270] compensates for the non-isometric mapping from texture space to the cells of a curvilinear grid that diﬀer in size. The projection of the normal component of the vector ﬁeld needs to be considered for LIC-type visualizations on hypersurfaces [375]. Unsteady Flow LIC (UFLIC) [392] or its accelerated version [250] incorporate time into the convolution to visualize unsteady ﬂow. The issue of temporal coherence is addressed by successively updating the convolution results over time. In [123], a visualization of time-dependent ﬂows on curvilinear surfaces

94

3 Vector Field Visualization

is presented. Dynamic LIC (DLIC) [405] is another extension of LIC, which allows for time-dependent vector ﬁelds, such as electric ﬁelds. A LIC-like image of an unsteady ﬂow can also be generated by an adaptive visualization method using streaklines, where the seeding of streaklines is controlled by the vorticity [372]. Since LIC has to perform a line integral convolution for each element of a high-resolution texture, computational costs are an issue. One solution to this problem utilizes the coherence along streamlines to speed up the visualization process [161, 396]. Parallel implementations are another way of dealing with high computational costs [42, 509]. Finally, implementations based on graphics hardware can enhance the performance of LIC [163]. From a conceptional point of view, an extension of LIC to 3D is straightforward. The convolution along streamlines is performed within a volume; the resulting gray-scale volume can be represented by volume visualization techniques, such as texture-based volume rendering. However, computational costs are even higher than in 2D and therefore interactive implementations of the ﬁltering process are hard to achieve. Even more importantly, possibly severe occlusion issues have to be considered: in a dense representation, there is a good chance of hiding important features behind other particle lines. A combination of interactive clipping and user intervention is one possible solution [341]. Alternatively, 3D LIC volumes can be represented by selectively emphasizing important regions of interest in the ﬂow, enhancing depth perception, and improving orientation perception [177, 178, 179]. The main focus of this chapter is a closely related class of dense representations that is based on texture advection. The basic idea is to represent a dense collection of particles in a texture that is transported according to the motion of particles [276, 279]. For example, the Lagrangian coordinates for texture transport can be computed by a numerical scheme for convection equations [15]. The motion map [194] is an application of the texture-advection concept for animating 2D steady ﬂows. The motion map contains a dense representation of the ﬂow as well as the information required for animation. Lagrangian-Eulerian advection (LEA) [191] is a scheme for visualizing unsteady ﬂows by integrating particle positions (i.e., the Lagrangian part) and advecting the color of particles based on a texture representation (i.e., the Eulerian aspect). LEA can be extended to visualizing vertical motion in a 3D ﬂow by means of time surfaces [143]. Image-based ﬂow visualization (IBFV) [440] is a recently developed variant of 2D texture advection. Not only is the (noise) texture transported along the ﬂow, but additionally a second texture is blended into the advected texture at each time step. IBFV is a ﬂexible tool that can imitate a wide variety of visualization styles. Another approach to the transport of a dense set of particles is based on nonlinear diﬀusion [91]. An initial noise image is smoothed along integral lines of a steady ﬂow by diﬀusion, whereas the image is sharpened in the orthogonal direction. Nonlinear diﬀusion can be extended to the multi-scale visualization of transport in time-dependent ﬂows [40].

3.2 Classiﬁcation and Overview of Flow Visualization Methods

95

Finally, some 3D ﬂow visualization techniques adopt the idea of splatting, originally developed for volume rendering (see Sect. 2.3.5). Even though some vector splatting techniques do not rely on particle tracing, they are included in this section because their visual appearance resembles dense curve-like structures. Anisotropic “scratches” can be modeled onto texture splats that are oriented along the ﬂow to show the direction of the vector ﬁeld [62]. Line bundles [278] use the splatting analogy to draw each data point with a precomputed set of rendered line segments. These semi-transparent line bundles are composited together in a back-to-front order to achieve an anisotropic volume rendering result. Crawﬁs et al. [63] detail the splatting approach in the context of volume rendering techniques for ﬂow visualization. For timedependent ﬂows, the animation of a large number of texture-mapped particles along pathlines can be used [146]. For all splatting approaches, the density of representation depends on the number of splats. 3.2.4 Feature-Based Visualization Approaches The visualization concepts discussed so far operate directly on the vector ﬁeld. Therefore, it is the task of the user to identify the important features of the ﬂow from such a visualization. Feature-based visualization approaches seek to compute a more abstract representation that already contains the important properties in a condensed form and suppresses superﬂuous information. In other words, an appropriate ﬁltering process is chosen to reduce the amount of visual data presented to the user. Examples of more abstract data are ﬂow topology based on critical points, other ﬂow features such as vortices and shock waves, or aggregated ﬂow data via clustering. After features are computed, the actual visual representation has to be considered. Diﬀerent features have diﬀerent attributes; to emphasize special attributes for each type of feature, suitable representations must be used. Glyphs or icons can be employed for vortices or for critical points and other topological features. Examples are ellipses or ellipsoids that encode the rotation speed and other attributes of vortices. A comprehensive presentation of feature extraction and visualization techniques can be found in a survey article by Post et al. [334]. Topology-based 2D vector ﬁeld visualization [165] aims to display only the essential information of the ﬁeld. The qualitative structure of a vector ﬁeld can be globally represented by portraying its topology. The ﬁeld’s critical points and separatrices determine the nature of the ﬂow. From a diagram of the topology, the complete ﬂow can be inferred. Figure 3.1 (d) shows an example of a topology-based visualization. From a numerical point of view, the interpolation scheme is crucial for identifying critical points. Extended versions of topology-based representations make use of higher-order singularities [376] and C 1 -continuous interpolation schemes [377]. Another extension is based on the detection of closed streamlines, a global property that is not detected by the aforementioned algorithms [492]. Topology-based visualization can also

96

3 Vector Field Visualization

be extended to time-dependent vector ﬁelds by topology tracking [425]. The original topology-based techniques work well for data sets with a small number of critical points. Turbulent ﬂows computed on a high-resolution grid, however, may show a large number of critical points, leading to an overloaded visual representation. This issue is addressed by topology simpliﬁcation techniques [76, 77, 78, 423, 424, 495], which remove some of the critical points and leave only the important features. For the visualization of 3D topology, suitable visual representations need to be used. For example, streamlines that are traced from appropriate positions close to critical points connect to other critical points or the boundary to display the topology, whereas glyphs can be used to visualize various classes of critical points [136]. Topology can also serve as a means for determining the similarity between two diﬀerent vector ﬁelds [11, 243]. Vector ﬁeld clustering is another way of reducing the amount of visualization data. Vectors of the original high-resolution ﬁeld are combined into fewer vectors that approximately represent the vector ﬁeld at a coarser resolution, leading to a visualization of aggregated data. An important issue is to establish appropriate error measures to control the way vectors are combined into clusters [253, 413]. The extension of vector ﬁeld clustering to several levels of clustering leads to hierarchical representations [132, 131, 159]. In related approaches, segmentation [91], multi-scale visualization [40], or topology-preserving smoothing based on level-set techniques [486] reduce the complexity of the displayed vector ﬁelds. An important class of feature detection algorithms is based on the identiﬁcation of vortices and their respective vortex cores. Vortices are useful for identifying signiﬁcant features in a ﬂuid ﬂow. One way of classifying vortex detection techniques is the distinction between point-based, local techniques, which directly operate on the vector data set, and geometry-based, global techniques, which examine the properties of characteristic lines around vortices [366]. Local techniques build a measure for vortices from physical quantities of a ﬂuid ﬂow. An overview on these techniques is given in [7, 354]; more recent contributions are presented in [13, 189, 205, 355]. A mathematical framework [325] makes it possible to unify several vortex detection methods. Point-based methods are rather simple to compute, but are more likely to miss some vortices. For example, weak vortices, which have a slow rotational component compared to the velocity of the core, are hard to detect by these techniques. Geometry-based, global methods [188, 364, 365] are usually associated with higher computational costs, but allow a more robust detection or veriﬁcation of vortices. Shock waves are another important feature of a ﬂuid ﬂow because they can increase drag and even cause structural failure. Shock waves are characterized by discontinuities in physical quantities, such as pressure, density, or velocity. Therefore, shock detection algorithms are related to edge detection methods known from image processing. A comparison of diﬀerent techniques for shock extraction and visualization is available [265]. Flow separation and

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

97

attachment, which occur when a ﬂow abruptly moves away from, or returns to, a solid body, are another interesting feature of a ﬂuid ﬂow. Attachment and separation lines on surfaces in a 3D ﬂow can be automatically extracted by means of phase plane analysis [204].

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains The remaining parts of this chapter focus on texture-based techniques for ﬂow visualization. The discussion begins with the display of vector ﬁelds by means of semi-Lagrangian texture advection. Semi-Lagrangian texture advection can be separated into two diﬀerent parts: ﬁrst, a transport mechanism and, second, a method to introduce visual patterns. The semi-Lagrangian approach is based on an Eulerian description of “particles”, which are represented indirectly via a property ﬁeld φ that is a function of position x and time t (see Sect. 3.1.3). The observer is located at a ﬁxed position x and measures changes of φ caused by the fact that diﬀerent particles cross that position. The ﬁrst aspect of texture advection deals with the transport mechanism, which implements the propagation of particles according to (3.5). Slightly diﬀerent numerical schemes to solve this equation are discussed in Sects. 3.3.1 and 3.3.3. In contrast to the Lagrangian, particle-based approach, line-like structures are not directly available in this Eulerian description. Therefore, an additional visual processing is needed to introduce visual patterns that resemble characteristic curves of the vector ﬁeld. These visual patterns are usually introduced by means of convolution that modiﬁes the spatial frequency of the image along characteristic curves – a higher coherence is present along these lines than perpendicular to these lines. A clear distinction between transport mechanism and visual mapping is often missing in the previous literature. This book, in contrast, makes a distinction between the two fundamental aspects of texture-based ﬂow visualization. One beneﬁt of this point of view is that diﬀerent methods for transport and visual mapping can be assembled in new, and possibly improved, combinations. Moreover, this point of view allows for a better analysis of the contributions and properties of existing visualization techniques. Dye advection and dense, noise-based representation are the two most dominant metaphors for texture-based ﬂow visualization. They diﬀer in their mapping to visual properties, but can be realized by the same underlying transport mechanism(s). For example, either a semi-Lagrangian approach (as described in this section) or a completely Lagrangian approach (see Sect. 3.6) can be used for both dye and noise transport. The advection of dye and noise textures allows for diﬀerent but equally useful results. Both approaches are often combined within the same visualization system. Dye is released at userspeciﬁed positions and therefore supports an interactive and user-centric visual exploration process. At the same time, a dense representation can provide

98

3 Vector Field Visualization

additional information on the overall ﬂow behavior and serve as a background context in which the user can focus on details of the dye-based visualization. 3.3.1 Semi-Lagrangian Transport Mechanism The in-depth discussion of semi-Lagrangian texture advection begins with the underlying transport mechanism. Here, particles or marker “objects” (such as dye) are modeled as massless material that is perfectly advected along the input vector ﬁeld. From an Eulerian point of view (see Sect. 3.1.3), particle properties are stored in a ﬁeld φ(x, t) of which the evolution is governed by ∂φ(x, t) + v(x, t) · ∇φ(x, t) = 0 , ∂t

(3.5)

where v can be identiﬁed with u in the notation of this equation on p. 85. This partial diﬀerential equation could be discretized and solved by diﬀerent Eulerian techniques [317]. Instead, a semi-Lagrangian approach is often used because it is well-suited for such a convection process and leads to a stable evolution even for large step sizes [397]. The advection is performed along characteristic lines: typically, pathlines xpath (t; x0 , t0 ) or streamlines xstream (t; x0 , t0 ) are considered. Therefore, (3.1) or (3.2) can be used to compute the transport in a semi-Lagrangian manner (see, e.g., [397, 190]). One possible method is based on backward integration: φ(x(t0 ), t0 ) = φ(x(t0 − ∆t; x0 , t0 ), t0 − ∆t) .

(3.6)

Starting from the current time step t0 , an integration backward in time provides the position along the characteristic curve (e.g., pathline or streamline) at the previous time step, x(t0 − ∆t; x0 , t0 ). The property ﬁeld is evaluated at this previous position to access the value that is transported to the current position. This method can be considered as a semi-Lagrangian approach because the property is represented on a texture in an Eulerian way, whereas the temporal evolution relies on a Lagrangian integration of particle paths. Similarly to (3.6), an integration forward in time leads to a forward mapping in which property values are actively scattered from their current location to a new position. In an analytic description, backward and forward mapping are equivalent. Slight diﬀerences may appear when a numerical approximation for pathline or streamline integration is employed. To tackle the advection problem on a computer, the particle-tracing equations (3.1) or (3.2) have to be solved numerically and the property function φ has to be discretized. A simple method to solve an ordinary diﬀerential equation is the ﬁrst-order explicit Euler integration, which yields x(t0 − ∆t; x0 , t0 ) = x(t0 ; x0 , t0 ) − ∆t v(x(t0 ; x0 , t0 ), t0 ) for a pathline. An analogous expression describes the Euler integration for a streamline. Furthermore, the property φ is discretized on a Cartesian or

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

99

uniform grid, both spatially and temporally. Bilinear (in 2D) or trilinear (in 3D) interpolation is applied to reconstruct the property ﬁeld at locations that are diﬀerent from grid points. Combining a discrete grid for φ with Euler integration for particle positions, the following numerical scheme is obtained to propagate a solution from time (t0 − ∆t) to t0 : φt (x) = φt−∆t (x − ∆t vt (x)) .

(3.7)

In this slightly changed notation, the superscripts in φt and vt denote the time dependency of the property ﬁeld and vector ﬁeld, respectively. 3.3.2 GPU Implementation of Semi-Lagrangian Advection A GPU implementation has to address the problem of how data structures and operations working on that data can be mapped to graphics hardware. The uniform grid that stores the property ﬁeld can be directly represented by 2D or 3D textures, depending on the dimensionality of the domain. In purely GPU-based implementations, the velocity ﬁeld is also held in a 2D or 3D texture. For hybrid CPU and GPU-based implementations, the vector ﬁeld is stored in main memory and used for advection computations by the CPU. In general, a single advection step computes the particle paths for the corresponding time step and transports the property ﬁeld accordingly. Often, an explicit ﬁrst-order Euler integration is used for particle tracing, which is either computed by the CPU (for a hybrid approach) or by the GPU. The transport of the property texture is based on an appropriate speciﬁcation of texture coordinates, which can be done either on a per-vertex basis (for hybrid CPU / GPU approaches) or on a per-fragment basis (i.e., purely GPU-based). 2D Semi-Lagrangian Advection Three diﬀerent approaches can be used for semi-Lagrangian advection with planar 2D vector ﬁelds. The ﬁrst approach applies per-fragment operations: a fragment program determines the previous position (x − ∆t vt (x)), based on the vector ﬁeld vt that stores the vx and vy components in a two-component texture format. The coordinates of the previous position are interpreted as texture coordinates for a dependent texture lookup in φt−∆t . This GPU-based method allows for an eﬃcient update of the property texture in a single rendering pass; this technique was ﬁrst proposed by Weiskopf, Hopf, and Ertl [479] and later described in more detail by Weiskopf and Hopf [478]. Figure 3.4 shows the DirectX pixel shader code for this advection method. A single quadrilateral spanning the spatial domain is rendered to generate the fragments for the pixel shader. The texture coordinates t0 and t1 are chosen to cover the domain by the textures. The dependent texture lookup (shader instruction texbem) uses the previously fetched vector ﬁeld entry v, scales this vector by a constant value (a parameter that is speciﬁed outside the pixel

100

3 Vector Field Visualization ps.1.1 // Pixel shader V1.1 tex t0 // Load vector field data v texbem t1,t0 // Dependent lookup in tex φ with // shifted texture coordinates mov r0, t1; // Result of advection Fig. 3.4. DirectX pixel shader code for 2D backward advection

shader and that takes into account the step size −∆t), and adds the texture coordinates t0 in order to obtain the coordinates for accessing the property texture φt−∆t . An animated visualization is built from iterative executions of the above advection step. In all computations, the property ﬁeld is accessed for only two time steps: the current time step (with a write access) and the previous time step (with a read access). Therefore, it is suﬃcient to provide two 2D textures to hold these two time steps. The texture φt−∆t is used as an input texture, whereas the texture φt is modiﬁed by directly rendering into it via the renderto-texture functionality. After each single advection computation, the roles of the two textures are exchanged, according to a ping-pong rendering scheme. The second approach also employs a backward advection. However, this approach computes the advection on a per-vertex basis, as described by Van Wijk [441]. The domain of the property ﬁeld is covered by a mesh, with velocities assigned at its vertices. A CPU program calculates the previous positions (x − ∆t v(x)) based on the vector ﬁeld held in main memory. These previous positions are used as per-vertex texture coordinates for a lookup in φt−∆t . The domain is rasterized by rendering the mesh. The third approach uses a per-vertex advection forward in time [440]. A forward Euler integration (x+∆t v(x)) provides the positions for the following time step. These new coordinates are used as the vertex coordinates for a distorted triangle mesh, i.e., vertex coordinates are modiﬁed instead of texture coordinates. All three approaches employ a texture lookup in the property ﬁeld that is based on bilinear interpolation. An interpolation scheme is required because pathline or streamline integration usually results in texture or vertex coordinates that do not have a one-to-one correspondence between pixels on the image plane and grid points of the property ﬁeld φt−∆t . 3D Semi-Lagrangian Advection The extension to 3D vector ﬁelds requires the following modiﬁcations, as proposed in a paper by Weiskopf and Ertl [476]. First, the property ﬁeld and the vector ﬁeld are represented by 3D textures. The transport of the property ﬁeld along one time step can be realized by a dependent texture lookup that is analogous to the per-fragment approach for 2D ﬁelds; the only diﬀerence

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

101

load flow field into GPU memory for i = 1 to max slice render quad for slice i, with dependent texture lookup update slice i in new property field end for Fig. 3.5. Pseudo code for 3D advection [476]

is that 2D texture coordinates are replaced by 3D texture coordinates for a lookup in a 3D texture. The second modiﬁcation takes into account that GPUs can only render 2D images, not directly a completely 3D output. Therefore, the property ﬁeld for a subsequent time step is built in a slice-by-slice manner. Figure 3.5 shows the pseudo code for a single time step of the advection process. Each slice of the property ﬁeld is updated by rendering a quadrilateral that represents this 2D subset of the full 3D domain. The quadrilateral is rasterized by the GPU to generate the fragments. The dependent-texture lookup can be realized by a fragment program that computes the modiﬁed texture coordinates according to the Euler integration of the ﬂow ﬁeld. Figure 3.6 shows the corresponding OpenGL ARB fragment program. Note that

!!ARBfp1.0 PARAM stepSize = program.local[0]; PARAM biasExtend = 2.0, -1.0, 0.0, 0.0; ATTRIB iTexCoord = fragment.texcoord[0]; OUTPUT oColor = result.color; # Temporary variables (registers) TEMP velocity; TEMP oldPos; # Fetch flow field TXP velocity, iTexCoord, texture[0], 3D; # Mapping to positive and negative values MAD velocity, velocity, biasExtend.x, biasExtend.y; # Compute previous position (Euler integration) MAD oldPos, velocity, stepSize, iTexCoord; # Dependent texture lookup: # Advected property value from previous time step TEX oColor, oldPos, texture[1], 3D; END Fig. 3.6. ARB fragment program for 3D texture advection [476]

102

3 Vector Field Visualization

the components of the vector ﬁeld texture are stored as unsigned ﬁxed-point numbers and thus a bias/extend transformation is applied to achieve positive and negative values. A related, but slower approach that requires multiple render passes per slice on an Octane2 with pixel textures was described by Weiskopf, Hopf, and Ertl [479]. The above advection method is completely formulated in 3D and therefore takes into account any ﬂow ﬁeld. In contrast, 3D image-based ﬂow visualization (3D IBFV) [414] uses a restricted model with 2D textures and with gathering of values along the z axis, and thus is restricted to a limited class of ﬂows: the z component of the vector ﬁeld has to be very small so that the absolute value of ∆tvz is not larger than the slice distance along the z axis. Moreover, the 3D texture-based approach makes use of built-in trilinear interpolation and thus allows for diﬀerent resolutions of particle and vector ﬁelds, whereas 3D IBFV supports only a bilinear resampling on 2D slices. Finally, 3D IBFV requires multiple render passes to update a single slice of the property ﬁeld. 3D Advection Based on 2D Texture Memory A potential problem of the above 3D texture advection is connected with the slice-by-slice update of the 3D texture for the property ﬁeld. An update of a 3D texture often is possible only by transferring data to and from main memory. For example, Direct3D does not provide a mechanism to directly modify 3D textures from other data on the GPU. Although OpenGL allows us to update a slice of a 3D texture by glCopyTexSubImage3D, the speed of such an update can vary between GPU architectures because of diﬀerent internal memory layouts of 3D textures (see the related discussion on read access for 3D textures in Sect. 2.4). Even though the OpenGL superbuﬀer extension [327] or the frame-buﬀer object extension [390] allow us to render directly into a 3D texture, the fundamental issues of the speed of updating 3D textures and the extent of the vendor’s support for those extensions may remain. Therefore, Weiskopf, Schafhitzel, and Ertl [481] proposed an alternative approach that is based on 2D texture memory instead of 3D texture memory. The following discussion is based on their paper.2 2D textures have several advantages: they are available on any GPU, they provide good caching mechanisms and eﬃcient bilinear resampling, and they support an extremely fast write access by the render-to-texture functionality. For a 2D texture-based implementation of 3D advection, logical memory and physical memory have to be distinguished. Logical memory is identical as for 3D texture advection – it is organized in the form of a uniform 3D grid. Physical memory is a 2D uniform grid represented by a 2D texture. The coordinates for addressing the logical memory are denoted by x = (x, y, z) and the coordinates for physical memory by u = (u, v). A slice of constant value 2

c The Eurographics Association 2005. Reprinted, with permission, from Weiskopf, Schafhitzel, and Ertl [481]

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains lookup table Φ0

0

1

2

. . . . . . . . .

i tex

Φ

logical memory

103 z (u,v) i tex

i tex

. . .

physical memory IMAGE: WEISKOPF, SCHAFHITZEL, ERTL [481]

c The Eurographics Association 2005

Fig. 3.7. Mapping between logical 3D memory and physical 2D memory by means of a lookup table

z in logical memory corresponds to a tile in physical memory, as illustrated in Fig. 3.7. Diﬀerent tiles are positioned in physical memory with a row-ﬁrst order. Since the maximum size of a 2D texture may be limited, several “large” 2D textures may be used to provide the necessary memory. These 2D textures are labeled by the integer-valued index itex . Since all numerical operations of 3D advection are conceptually computed in logical 3D space, an eﬃcient mapping from logical to physical memory is needed. This mapping is described by the function Φ : (x, y, z) −→ ((u, v); itex ) . The 2D coordinates can be separated into the coordinates for the origin of a tile, u0 , and the local coordinates within the tile, ulocal : Φ : (x, y, z) −→ (u0 + ulocal ; itex ) ,

(3.8)

u0 = Φ0,u (z), itex = Φ0,itex (z), ulocal = (sx x, sy y) .

(3.9)

with The function Φ0 maps the logical z value to the origin of a tile in physical memory and is independent of the x and y coordinates. The map Φ0 can be represented by a lookup-table (see Fig. 3.7), which can be eﬃciently implemented by a dependent texture lookup. Conversely, the local tile coordinates are essentially identical to (x, y) – up to scalings (sx , sy ) that take into account the diﬀerent relative sizes of texels in logical and physical memory. If more than one “large” 2D texture is used, multiple texture lookups in these physical textures may be necessary. However, multiple 2D textures are only required in tiles that are close to the boundary between two physical textures because the maximum diﬀerence vector for the backward lookup in (3.7) is bounded by ∆tvmax , where vmax is the maximum velocity of the data set. Diﬀerent fragment programs are used for boundary tiles and internal tiles to reduce the number of texture samplers for the advection in internal regions.

104

3 Vector Field Visualization

Table 3.1. Performance for steady vs. unsteady 3D advection based on 2D textures (in frames per second) [481] Radeon X800 Domain size Steady Unsteady

128

3

114.2 51.2

GeForce 6800

3

256

1283

2563

13.0 9.9

96.4 39.0

16.7 6.7

Trilinear interpolation in logical space is implemented by two bilinear interpolations and a subsequent linear interpolation in physical space. Bilinear interpolation within a tile is directly supported by built-in 2D texture interpolation. A one-texel-wide border is added around each tile to avoid any erroneous inﬂuence by a neighboring tile during bilinear interpolation. The subsequent linear interpolation takes the bilinearly interpolated values from the two closest tiles along the z axis as input. This linear interpolation is implemented within a fragment program. While trilinear interpolation by the above mapping scheme is necessary for the read access in (3.6) or (3.7), write access is more regularly structured. First, write access does not need any interpolation because it is restricted to grid points (i.e., single texels). Second, the backward lookup for (3.6) or (3.7) allows us to ﬁll logical memory in a slice-by-slice manner and, thus, physical memory in a tile-by-tile fashion. A single tile can be ﬁlled by rendering a quadrilateral into the physical 2D texture if the viewport is restricted to the corresponding subregion of physical memory. Table 3.1 shows performance measurements for the implementation of 3D advection with 2D textures on ATI Radeon X800 Platinum Edition (256 MB) and NVIDIA GeForce 6800 Ultra (256 MB) GPUs. This implementation is based on C++ and DirectX 9.0. All shader programs are formulated with high-level shading language (HLSL). The size of the property ﬁeld is given in Table 3.1. A vector ﬁeld of size 1283 (tornado from Fig. 3.11) is used. The measurements indicate that advection speed is roughly proportional to the number of texels. The table also compares the visualization performance for unsteady and steady ﬂow. For unsteady ﬂow, a new 3D texture for the vector ﬁeld is transferred for each frame. Although the rendering speed is reduced for unsteady ﬂow, the overall performance still facilitates interactive visualization for property ﬁelds up to 2563 . Table 3.2 compares the 3D advection method based on 2D textures (implemented in DirectX) with the advection technique realized by 3D textures (implemented in OpenGL with glCopyTexSubImage3D). These performance measurements were conducted on an ATI Radeon X800 GPU. Table 3.2 shows that the 2D texture-based method outperforms the 3D texture-based method, i.e., the beneﬁt of fast read and write access to 2D textures outweighs the ad-

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

105

Table 3.2. Comparison between the 3D advection method based on 2D textures and the advection method based on 3D textures on a Radeon X800 GPU (in frames per second) [481] 2D texture Domain size Advection only

128

3

114.2

3D texture

3

256

1283

2563

13.0

44.4

9.7

ditional operations for the mapping of texture coordinates from logical 3D space to physical 2D space. 3.3.3 Lagrangian-Eulerian Advection Lagrangian-Eulerian advection (LEA) [191] addresses issues introduced by bilinear interpolation within the backward lookup in semi-Lagrangian integration (3.6). Especially for a noise signal stored in the property ﬁeld, this bilinear resampling leads to a rapid decrease of contrast and high spatial frequencies: the property ﬁeld is increasingly blurred. The following description of LEA speciﬁcally aims at improving the advection of white noise ﬁelds. The basic idea is to use nearest-neighbor interpolation for resampling, along with a mechanism that corrects errors introduced by nearest-neighbor interpolation. Once again, all information concerning particles or property values is stored in 2D arrays or textures, i.e., the discussion is restricted to planar 2D ﬂow. A coordinate system is assumed in which the grid points of textures are located at integer-valued locations (i, j). LEA uses an additional 2D ﬁeld c to store the coordinates of particles within a texture cell. Note that the two-component ﬁeld c(i, j) describes only the fractional part of the coordinates – actual coordinates with respect to the grid are x = (i, j) + c(i, j). Stated diﬀerently, each texture cell contains a single virtual particle that can be moved within the cell and even be transported into diﬀerent cells by advection. Based on the array of fractional coordinates, a backward integration step yields the coordinates x at the previous time step: (3.10) x = (i, j) + c(i, j) − ∆t v(i, j) . To maintain a high contrast in the advected noise and to avoid artiﬁcial diﬀusion, φ is computed by a nearest-neighbor sampling of the property ﬁeld from the previous time step, based on the coordinates x for the previous time step. A correction step is introduced to counteract a duplication eﬀect that occurs during the computation of (3.10). Eﬀectively, if particles in neighboring cells of φ retrieve their property value from within the same cell of φ, this value will be duplicated in the corresponding cells of φ. This eﬀect is undesirable since lower noise frequency reduces the spatial resolution of features that

106

3 Vector Field Visualization

Table 3.3. Performance measurements for Lagrangian-Eulerian advection in frames per second (ATI Radeon 8500 GPU vs. Athlon 1.2 GHz CPU) [473] Noise Advection Particle Size Flow Size GPU (steady) GPU (unsteady) CPU-based

Noise, Dye & Mask

2

1024 2562

2

1024 10242

10242 2562

10242 10242

24.8 24.3 0.8

23.8 19.4 0.8

20.5 20.3 NA

19.9 16.7 NA

can be represented. This duplication eﬀect is caused by the discrete nature of texture sampling and is further reinforced in regions where the ﬂow has a strong positive divergence. To break the undesirable formation of uniform blocks and to maintain a high frequency random noise, a user-speciﬁed percentage of noise is injected into φ. Random cells are chosen in φ and their value is inverted. From experience, a ﬁxed percentage of two to three percent of randomly inverted cells provides adequate results over a wide range of ﬂows. Finally, the array c of fractional coordinates is re-initialized to prepare a new collection of particles to be integrated backward in time for the next iteration. Nearest-neighbor sampling implies that a property value can only change if it originates from a diﬀerent cell. If fractional coordinates were neglected in the next iteration, subcell displacements would be ignored and the ﬂow would be frozen where the velocity magnitude or the integration step size is too small. Nearest-neighbor sampling without fractional coordinate tracking causes the ﬂow to be partitioned into distinct regions within which the integer displacement vector is constant. Therefore, the array c is set to the fractional part of the newly computed coordinates. Since the original CPU-based implementation of LEA allows for the interactive visualization of only small datasets (in the range of 1282 ), a much faster GPU-based approach was proposed by Weiskopf, Erlebacher, Hopf, and Ertl [473]. This method makes use of eﬃcient single-pass noise advection on an ATI Radeon 8500 GPU, which has DirectX pixel shader 1.4 support. A similar implementation is feasible with any of the more recent GPUs, like ATI Radeon 9700, NVIDIA GeForce FX, or newer. Table 3.3 compares performance measurements for the GPU-based implementation (ATI Radeon 8500, Athlon 1.2 GHz CPU, Windows 2000) with those for a CPU-based implementation on the same machine. Included are the numbers for mere noise advection (single-pass advection) and for advection of noise and dye images (two advection passes) with subsequent velocity masking (see Sect. 3.3.5 for a discussion of velocity masking). A ﬁnal rendering to a 10242 viewport is included in all tests. The performance measurements indicate that the download of the 2D vector ﬁeld data from main memory

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

107

to texture memory via AGP bus (line “GPU (unsteady)”) takes only a small portion of the overall rendering time even for large data sets. For many applications, the ﬂow ﬁeld is medium-sized and leads to typical overall frame rates of 15–25 fps. The cost of the CPU implementation is dominated by the advection of the noise texture and is quite independent of the size of the ﬂow ﬁeld data; a transfer of ﬂow data to texture memory is not necessary. 3.3.4 Particle Injection So far, only basic advection mechanisms have been discussed. However, a useful visualization needs – besides the computation of particle traces – a mapping of the particle traces to a graphical representation. In the model of this section, the mapping is essentially restricted to an appropriate injection of property values. The basic idea of the IBFV approach is adopted and further extended. IBFV introduces new property values at each time step, described by an injection texture I. The structure of the injection mechanism of 2D IBFV [440] is illustrated in Fig. 3.8. The pure advection mechanism is typically based on semi-Lagrangian advection (Sects. 3.3.1 and 3.3.2) or LEA (Sect. 3.3.3). Original 2D IBFV is based on an aﬃne combination of values from the previously advected texture φ(t − ∆t) and the injection texture I(t) to yield the new property texture φ(t): φ(t) = (1 − α)φ(t − ∆t) + αI(t) .

(3.11)

The variable α denotes a scalar blending parameter that is constant for the complete visualization domain. A repeated application of this alpha blending results in an exponential decay of the injected noise over time. For a timeindependent injection texture and a steady vector ﬁeld, this alpha blending resembles the result of LIC with an exponential ﬁlter kernel. In the context of 3D texture advection, 3D IBFV [414] introduces a slightly extended injection scheme that allows for space-variant noise injection weights H(t): φ(t) = (1 − H(t))φ(t − ∆t) + H(t)I(t) . (3.12) The structure and contents of the injection texture determines the appearance of the ﬁnal visualization. A dense, LIC-like result is achieved by specifying a

advection for the next time step

advected texture

injection texture blending

IMAGE: WEISKOPF, SCHAFHITZEL, ERTL [481]

c The Eurographics Association 2005

Fig. 3.8. Basic structure of the injection mechanism of IBFV: new property values are introduced from an injection texture by blending

108

3 Vector Field Visualization

noise texture as injection texture. The spatial frequency of the noise corresponds to the spatial frequency of the visualization output. The injection texture typically contains ﬁltered noise to avoid aliasing artifacts (see the discussion by Van Wijk [440]). Moreover, the injection may be time-variant to introduce time-dependent visualizations even for steady ﬂows. As already proposed for IBFV [440], noise is used that is periodically switched on and oﬀ with slow-in and slow-out behavior. As long as the periodicity of this time dependency is the same for the complete domain, a single, time-independent texture that holds both the noise values and the random phases is suﬃcient. For a speciﬁc time step, the actual noise injection value is obtained from a lookup table that describes the temporal behavior of the slow-in and slow-out. The relative phase from the noise injection texture is added to the current global time to yield the local time of that noise. This time – modulo the temporal periodicity – serves as the basis for the aforementioned table lookup. The advantage of this approach is that a single noise-injection texture is suﬃcient to represent a time-dependent noise injection. Besides this noise-based dense representation, dye advection is a powerful and intuitive visualization metaphor. Virtual dye is released at user-speciﬁed positions and therefore supports an interactive and user-centric visual exploration process. Texture-based advection is well-suited for dye visualization because of its built-in support for transporting extended regions of dye. The visualization performance is independent of the number of cells covered by dye. Therefore, a user interface is possible that allows the user to control dye injection by virtually painting into the ﬂow, as shown by Weiskopf, Erlebacher, Hopf, and Ertl [473]. This interaction approach is more diﬃcult to implement for particle-based Lagrangian tracking. To maintain a dense and uniform coverage of the domain in regions of divergent or convergent ﬂow, data structures have to be implemented that allow for dynamic insertion and deletion of particles in the Lagrangian approach [392]. Alternatively, a large number of particles can be traced to achieve a minimum density of particles in divergent ﬂow areas, which decreases the eﬃciency of the implementation [79]. The combination of a dense representation by noise injection and a userguided exploration by injecting dye at isolated locations is very powerful because it combines both an overall view and a detailed visualization of speciﬁc features. To achieve this goal, the paper by Weiskopf and Ertl [476] generalizes the aforementioned injection schemes to allow for a uniﬁed description of both noise and dye advection. The extensions are: ﬁrst, the restriction to an aﬃne combination of the advected value and of the newly injected value is suspended and replaced by a generic combination of both; second, several materials can be advected and blended independently. The extended blending equation is given by φ(t) = W (t) ◦ φ(t − ∆t) + V (t) ◦ I(t) ,

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

109

where the two, possibly space-variant, weights W (t) and V (t) need not add up to one. The symbol “◦” denotes a component-wise multiplication of two vector quantities, i.e., W (t), V (t), φ(t), and I(t) must have the same number of components. In this approach, the diﬀerent components of each texel in the property ﬁeld describe the density of diﬀerent materials that are transported along the ﬂow (rather than color or gray-scale values). The advantages of this extended blending scheme are: ﬁrst, diﬀerent materials are blended independently from each other and may therefore have diﬀerent lengths for exponential decay; second, material can be added on top of existing material (e.g., additional dye), which is impossible with an aﬃne combination as in (3.11) and (3.12). A uniﬁed description of both dye and noise advection is supported by the extended blending. Typically, dye is faded out only very slowly or not at all. Since new dye has to be injected at seed points, both W (t) and V (t) need to be one or close to one, and therefore the sum is larger than one. A possible saturation of material can be controlled by clamping. Conversely to dye visualization, the streaklines generated by noise advection tend to be much shorter. Furthermore, the overall brightness of noise-based representations should be independent of blending weights. Therefore, noise advection typically makes use of an aﬃne combination of advected and newly injected material. Dye injection is represented by an injection texture as well. The shape of the dye emitter is stored in a voxelized form. A random phase is not needed here; dye injection is either switched on or oﬀ, depending on user interaction. 3.3.5 Visual Mapping and Rendering In the aforementioned blending schemes, the property ﬁelds do not need to directly contain color values, but could represent densities of diﬀerent materials, which are coded into the RGBA channels of the property texture (i.e., a maximum number of four materials is possible with a single texture). A separate transfer function is applied to each material to obtain the corresponding color values. A scenario is assumed with a ﬁxed assignment of materials, so that a collection of diﬀerent 1D textures can be used to represent the transfer functions. The diﬀerent transfer functions are evaluated on a per-fragment level and the results are combined to a single color by summation. From a visualization point of view, it is very eﬀective to interactively choose whether and how diﬀerent materials are displayed. For example, the noise part could be rendered very faintly to give an overall context and, at the same time, the dye part could be emphasized by bright colors to focus on this detail. In another scenario, dye could be completely removed and noise could be rendered more prominently. It is important that these diﬀerent visualization approaches can be interactively changed in order to adapt the appearance of the visualization to the users’ needs. This goal is achieved by modifying the respective transfer functions.

110

3 Vector Field Visualization

To further reduce the visual complexity and to highlight important features of the ﬂow, additional scalar quantities can be mapped to the color and opacity values in the ﬁnal rendering. Scalar quantities can either be derived from the vector ﬁeld itself (e.g., velocity magnitude) or from other parameters of the data set (e.g., pressure or temperature in a ﬂuid ﬂow). For example, the velocity magnitude can be used to emphasize interesting regions of high ﬂow magnitude and fade out parts with low speeds (i.e., velocity masking). Rendering is trivial for 2D ﬂow visualization: the planar 2D property ﬁeld is transformed into an RGB image by applying the aforementioned transfer functions, and the resulting image is directly displayed to the user. Figure 3.9 shows an example visualization of a 2D unsteady ﬂow ﬁeld by means of 2D texture advection. The transport of the noise texture is implemented by LEA (Sect. 3.3.3); the data set is produced by the interaction of a planar shock with a longitudinal vortex (in axisymmetric geometry) [114]. The images show a single time step from a time series of an unsteady ﬂow simulation. The complete data set has 200 time steps and a spatial resolution of 256 × 151. In Fig. 3.9 (a), only noise advection is employed. The noise blending factor is α = 0.05. Uninteresting regions of the ﬂow are faded out by reducing their brightness via velocity masking. Therefore, one clearly sees the primary shock, secondary shock, and the slip lines [176]. Figures 3.9 (b) and (c) additionally include the advection of colored dye which was released into the ﬂow by the user. In Fig. 3.9 (c), velocity masking is applied to enhance important parts of the ﬂow. In Fig. 3.9 (b), no velocity masking is applied. Direct volume visualization is used to render the results of 3D texture advection because it allows the user to view a volume data set at diﬀerent depth positions simultaneously by means of semi-transparency. For texture advection based on 3D textures, volume rendering makes use of 3D texture slicing with image-aligned slices, as described in Sect. 2.3.1. Alternatively, GPU ray casting could be employed. A 3D texture holds the diﬀerent material densities in diﬀerent color channels, and the aforementioned transfer functions are applied on-the-ﬂy in the form of dependent-texture lookups while a fragment of a slice or along a ray is processed. For 3D advection based on 2D textures, volume rendering with 2D slicing is employed (see Sect. 2.3.2). Volume rendering accesses tile after tile in frontto-back order by rendering object-aligned quadrilaterals. Texture coordinates are issued to address a tile in the physical memory of the “large” 2D texture. Otherwise, rendering is the same as for the 3D texture-based approach. In contrast to traditional 2D texture slicing, which holds three copies of a volume data set (one for each of the main axes), volume rendering of advection data maintains only a single copy of the property ﬁeld. The stacking direction is changed on-the-ﬂy during advection if the viewing angle becomes too large with respect to the stacking axis to avoid holes in the ﬁnal display. Figure 3.10 illustrates a reordering of the stacking direction from y to x axis. A tile in the new stacking order is rendered in a stripe-by-stripe fashion, according to

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

111

(a)

IMAGES: WEISKOPF, ERLEBACHER, HOPF, ERTL [473] c 2002 Greiner, Niemann, Ertl, Girod, Seidel

(b)

(c) Fig. 3.9. Visualization of the interaction of a planar shock with a longitudinal vortex by 2D texture advection. The transport mechanism makes use of LEA. (a) shows a dense, noise-based visualization with a velocity mapping that modiﬁes the grayscale image according to the magnitude of the velocity to enhance the important structures in the ﬂow. (b) and (c) show the combination of noise (gray-scale) and dye (colored) advection – in (b) without velocity mapping and in (c) with velocity mapping. See also Color Plate 13

112

3 Vector Field Visualization y ordering

x ordering

physical memory

y logical memory

x IMAGE: WEISKOPF, SCHAFHITZEL, ERTL [481]

c The Eurographics Association 2005

Fig. 3.10. Reordering of stack direction. A tile in the new stacking order is rendered in a stripe-by-stripe fashion, according to portions of tiles from the old stacking order

portions of tiles from the old stacking order. The reordering process takes into account that the number of tiles, their sizes, and their positions may change. Figure 3.11 shows results of advection with 3D textures. The underlying data set represents the behavior of a tornado (the data set courtesy of Roger Crawﬁs). The size of the ﬂow ﬁeld is 1283 , the particle textures are 2563 , and the noise injection texture is 1283 . Figures 3.11 (a)–(f) compare diﬀerent visualization styles for the same viewpoint. Images (a)–(d) are rendered with velocity masking, i.e., only regions of high velocity magnitude are displayed. Image (a) employs a dense noise injection of two materials (bluish and green) and (b) a sparse noise injection. Due to the asymmetric, exponential ﬁlter kernel (based on iterative alpha blending), image (b) resembles oriented LIC [457]. In (c), additional red dye is injected at a user-speciﬁed position. The seed point is visualized by the intersection of three orangecolored, axis-oriented, thin tubes. In this way, the user can easily identify the spatial position of dye injection. In (d), diﬀerent lengths for blue and green noise material demonstrate that diﬀerent blending weights can be used for each material. Image (e) shows a dense visualization without masking. In this dense representation, interior parts of the ﬂow are completely occluded by advected material in the front. Therefore, clipping approaches are required to view these inner regions, e.g., with a slanted clipping plane as in Fig. 3.11 (f). The 3D texture advection in Fig. 3.11 demonstrates that perception challenges have to be addressed in dense 3D representations. Perceptual issues are much more important for 3D ﬂow visualization than for the 2D case because the fundamental problems of occlusion, clutter, and depth perception need to be addressed [178]. One basic approach is to reduce the visual complexity by removing unimportant elements of the 3D representation and, in this way, emphasizing the remaining, more important elements. As one important example of this approach, transfer functions can be used to selectively specify opacity. “Flow features” in their broadest sense are

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

(b)

(c)

(d)

(e)

(f)

IMAGES: WEISKOPF, ERTL [476]

c 2004 UNION Agency/WSCG

(a)

113

Fig. 3.11. Visualization of a tornado data set by 3D texture advection with velocity masking: (a) dense noise injection, (b) sparse noise injection, (c) additional red dye, (d) diﬀerent lengths for blue (dark) and green (bright) material. (e) shows a dense visualization without masking; (f ) the same with a slanted clipping plane. See also Color Plate 14

3 Vector Field Visualization

IMAGE: WEISKOPF, SCHAFHITZEL, ERTL [481]

c The Eurographics Association 2005

114

Fig. 3.12. Benard ﬂow with λ2 vortex visualization. See also Color Plate 15

mapped to a single importance value that is used as measure for opacity. A “ﬂow feature” can be as simple as the velocity magnitude in Figs. 3.11 (a)–(d). Other opacity measures could be based on traditional features such as vorticity or shock fronts (see Sect. 3.2.4). Figure 3.12 illustrates vortex detection by means of the λ2 criterion [187]. Vortex regions are rendered as opaque red or yellow material (see also Color Plate 15), whereas the surrounding ﬂow regions are still visible as semi-transparent material with blue or green color. Features are, in general, derived from the vector ﬁeld and possibly from additional attributes (such as temperature or pressure). The actual deﬁnition of a feature heavily depends on the application context and domain knowledge. Therefore, the expertise of a user could also be used to specify a feature, e.g., by means of user input via a feature-deﬁnition language [93]. The advection and rendering methods of this section are prepared to include this feature-based approach because they can immediately incorporate any of these features by adapting the transfer function. Volume clipping pushes the idea of selective rendering to its extremes – unimportant elements are completely removed (or rendered transparent). Therefore, clipping techniques from Sect. 2.5 can be adopted to ﬁlter the ﬂow representation. For example, the clip geometry could be adapted to curved boundaries of the problem domain in ﬂow simulations or the clip object could be aligned to the surface of a car body in automotive CFD applications to explore interesting features of a ﬂow close to such boundaries. Therefore, volume clipping can play a crucial role in investigating dense 3D representations; see, e.g., the discussion by Rezk-Salama et al. [341]. When clipping is applied to an otherwise completely opaque volume visualization, the rendering is reduced to a ﬂow visualization on curved surfaces deﬁned by the clip geometry. Vector ﬁeld visualization on curved surfaces is called a 2.5D approach because

3.3 Semi-Lagrangian Noise and Dye Advection on Cartesian Domains

115

it is a compromise between visualization on a 2D plane and in full 3D space. An eﬃcient implementation of the 2.5D approach is discussed in Sect. 3.5. Finally, the perception of shape, orientation, and depth could be promoted by perception-guided rendering techniques. Some of these techniques are discussed in Chap. 4. A simple, yet very eﬀective approach is to include volumetric illumination in the display of the property ﬁeld. As discussed in Sect. 2.1, local illumination needs gradients, which serve as normal vectors. One issue is that the property ﬁeld continuously changes due to advection. Therefore, gradients have to recomputed after each advection step. Weiskopf, Schafhitzel, and Ertl [481] use central diﬀerences to compute gradients after each advection and blending iteration and before volume rendering. Central diﬀerences provide an acceptable quality at moderate computational costs. In particular, they access the 3D uniform grid in a regular pattern and need only directly neighboring grid points along the three main axes. Therefore, gradient computation can be implemented as a fragment program that processes the texture representation of the property ﬁeld. Once the gradient is available, any gradient-based volume shading method can be applied. Figure 3.13 compares a few shading techniques. The data set is the same tornado as in Fig. 3.11. Figure 3.13 (a) illustrates the emissionabsorption model, rendered with high opacities. Figure 3.13 (b) shows an example of volume illumination by the Phong model with diﬀuse and specular components, which are added on top of the emissive part present in Fig. 3.13 (a). Figure 3.13 (c) demonstrates cool/warm shading, which encodes the orientation of streamlines by warm or cool colors, respectively [137, 105]. An advantage of cool/warm shading is that orientation is represented by (almost) isoluminant colors. Therefore, brightness can still be used to visualize another attribute or property. In general, both cool/warm shading and Phong illumination provide a better impression of shape than the emissionabsorption model. More information on the eﬀective use of color and illumination in animated visualizations is discussed in Sect. 4.2, which focuses on perception-oriented rendering. An example of a more advanced rendering approach is halo rendering, which is eﬀective in visualizing relative depth between curves [178]: curves behind a closer streamline are partly hidden by dark halos. In Fig. 3.13 (d), halos are rendered as thick silhouette lines. Silhouettes are identiﬁed by examining the dot product of gradient and viewing directions because a silhouette is associated with a gradient perpendicular to the viewing vector. In addition, limb darkening through high opacities in the transfer function [164] also contributes to the halo eﬀect. More details on volume illumination for GPU-based 3D texture advection can be found in a paper by Weiskopf, Schafhitzel, and Ertl [481]. The work by Interrante and Grosch [178] discusses perception-oriented rendering for 3D texture-based ﬂow visualization in general.

3 Vector Field Visualization

(a)

(b)

(c)

(d)

c The Eurographics Association 2005 IMAGES: WEISKOPF, SCHAFHITZEL, ERTL [481]

116

Fig. 3.13. Volume rendering of 3D advection applied to a tornado data set: (a) emission-absorption model only, (b) with Phong illumination added, (c) cool/warm shading, and (d) halos. See also Color Plate 16

3.4 Dye Advection Based on Level-Sets Semi-Lagrangian dye advection is subject to numerical diﬀusion that may lead to an inappropriate blurring of the visualization, especially when short integration step sizes are employed. Figure 3.14 shows an example of numerical diﬀusion within a 2D circular ﬂow. The underlying problem is the continuous resampling with a tensor product of linear ﬁlters (i.e., bilinear interpolation in 2D and trilinear interpolation in 3D). Figure 3.15 illustrates how linear interpolation leads to artiﬁcial diﬀusion in a simple 1D example. The top image shows the property values p for the time step t. The dye is represented

117

IMAGE: WEISKOPF [464]

c The Eurographics Association 2004

3.4 Dye Advection Based on Level-Sets

Fig. 3.14. Numerical diﬀusion in a circular ﬂow (see also Color Plate 17)

by large values p, the background by vanishing p. The bottom image shows the property values for the following time step (t + ∆t). A uniform vector ﬁeld is applied to transport the dye to the right. In this example, the step size ∆t and the velocity v are chosen in a way that the dye is shifted by one half of the grid spacing ∆x, i.e., ∆tv = ∆x/2. Ideally, the proﬁle for the values p should not be changed by such a translation. However, the access to positions in between grid points leads to a dilution of dye by linear interpolation. As a result, the width of the interface between dye and background is increased from one grid spacing to two grid spacings. Subsequent advection steps will continuously increase the size of this interface layer. This blurring eﬀect is not related to physical diﬀusion that may occur in experimental ﬂow visualization. In fact, the smearing is only caused by the advection algorithm and depends on many diﬀerent parameters, such as step size, velocity, grid resolution, and the orientation with respect to the main axes of the texture. Furthermore, the blurring prevents users from clearly identifying streaklines or other isolated visual patterns. In summary, this artiﬁcial diﬀusion should be avoided for an eﬀective vector ﬁeld visualization. The use of a level-set approach was proposed by Weiskopf [464] to overcome the above mentioned problems and to provide a dye advection method that is able to accurately track the boundary between dye and background. The following discussion is based on the paper by Weiskopf [464].3 In general, level-set techniques were adapted to other computer graphics applications before, e.g., the interface between ﬂuid and air can be modeled by level-sets in a water simulation [112, 124]. In addition, tracking of interfaces 3

c The Eurographics Association 2004. Reprinted, with permission, from Weiskopf [464]

118

3 Vector Field Visualization

p(t )

x p(t+∆t)

∆t v

∆x

x

IMAGE: WEISKOPF [464]

c The Eurographics Association 2004

Fig. 3.15. Numerical diﬀusion caused by linear interpolation. The top image shows the property values p for the time step t; the bottom image shows the property values for the following time step (t + ∆t)

plays an important role in computational ﬂuid dynamics and computational physics. Here, only a brief list of related publications in this ﬁeld can be given. One class of numerical methods makes use of a Lagrangian tracking of particles, as, e.g., described in [433]. Volume-of-ﬂuid (VOF) techniques [169] model the fraction of a material in each computational cell to reconstruct the material interface; examples are the SLIC method (Simple Line Interface Calculation) [309], the PLIC technique (Piecewise Linear Interface Construction) [502], or a cubic-polynomial interpolation to represent the VOF fraction [499]. Another approach is the ghost ﬂuid method [117], in which a level-set function is used to track the motion of a material interface. 3.4.1 Distance Field as Level-Set In the level-set approach of this section, the interface between dye and background is represented as an implicit surface within a signed distance function. A signed distance function φ(x) describes the distance of the point x to the surface. The interface is the implicit surface for φ(x) = 0. It is assumed that φ(x) is negative inside the dye region and positive on the outside. Dye is modeled as a massless marker material that is perfectly advected along the input vector ﬁeld. For such a convection scenario, the evolution of the level-set is governed by ∂φ(x, t) + v(x, t) · ∇φ(x, t) = 0 , (3.5) ∂t

3.4 Dye Advection Based on Level-Sets

119

which was already used for the generic description of advection in an Eulerian context. The vector v can be identiﬁed with u in the original notation of above equation on p. 85. The same semi-Lagrangian approach is used as in Sect. 3.3, leading to the evolution equation φ(x(t0 ), t0 ) = φ(x(t0 − ∆t), t0 − ∆t) .

(3.13)

Besides the transport of dye, the injection of additional dye has to be modeled in this level-set approach. Newly injected dye is represented by a signed distance ﬁeld φinject . After each integration step (3.13), the new dye and the “old”, transported dye φtransp are combined according to φinject (x) if φinject (x) < φtransp (x) φ(x) = . φtransp (x) otherwise In this way, the signed distance is modiﬁed by the injection ﬁeld at positions that are closer to the new dye region, compared to the original distance from the interface. The distance ﬁeld itself cannot be directly used for the ﬁnal visualization. In fact, dye is drawn where the signed distance is negative and the background remains visible where the distance is positive. Figures 3.16 (c) and (d) demonstrate level-set dye advection. Dye is injected in the lower left part of the domain and transported by a uniform vector ﬁeld that points toward the upper right of the domain. Image (c) visualizes the distance ﬁeld: bright regions correspond to negative distances, whereas dark regions correspond to positive distances. This illustration uses a nonlinear gray-scale mapping to cover all relevant distance values. Figure 3.16 (d) shows the ﬁnal visualization