High Performance Computing in Science and Engineering ' 07: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2007

High Performance Computing in Science and Engineering ’07 Wolfgang E. Nagel · Dietmar Kröner · Michael Resch Editors ...

Author: Wolfgang E. Nagel

10 downloads 1079 Views 18MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

High Performance Computing in Science and Engineering ’07

Wolfgang E. Nagel · Dietmar Kröner · Michael Resch

Editors

High Performance Computing in Science and Engineering ’07 Transactions of the High Performance Computing Center Stuttgart (HLRS) 2007

123

Wolfgang E. Nagel Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH) Technische Universität Dresden Willers-Bau, A-Flügel Zellescher Weg 12 01069 Dresden, Germany [email protected]

Dietmar Kröner Abteilung für Angewandte Mathematik Universität Freiburg Hermann-Herder-Str. 10 79104 Freiburg, Germany [email protected]

Michael Resch Höchstleistungsrechenzentrum Stuttgart (HLRS) Universität Stuttgart Nobelstraße 19 70569 Stuttgart, Germany [email protected]

Front cover figure: Streamlines of the mean flow and turbulent kinetic energy contours obtained from a simulation of separated flow over an axisymmetric hill, Institute for Hydromechanics, University of Karlsruhe

ISBN 978-3-540-74738-3

e-ISBN 978-3-540-74739-0

DOI 10.1007/978-3-540-74739-0 Library of Congress Control Number: 2007939457 Mathematics Subject Classification (2000): 65Cxx, 65C99, 68U20 © 2008 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: by the editors using a Springer TEX macro package Production and data conversion: LE-TEX Jelonek, Schmidt & Vöckler GbR, Leipzig, Germany Cover design: WMXDesign, Heidelberg Printed on acid-free paper 987654321 springer.com

Preface

Again, the last year has been very successful for high performance computing in Baden-Württemberg and beyond. The NEC SX-8 vector supercomputer is highly utilized and has been very successfully used by many scientiﬁc projects. In February 2007, the new “Landes-Höchstleistungsrechner HP XC4000” has been inaugurated at the SSC in Karlsruhe. This massively parallel high performance system complex, built from hundreds of Intel Itanium processors and more than three thousand AMD Opteron cores, has helped in doing additional science on a very high level. Nevertheless, the time has already come to start thinking about the next follow-up system in future. The discussions and plans on all scientiﬁc, advisory, and political levels to realize an even larger “European Supercomputer” in Germany, where the hardware costs alone will be around 200 Million Euro, are getting closer to realization. There are many good reasons to invest in such a program because – beyond the infrastructure – such a scientiﬁc research tool will attract the best brains to tackle the problems related to the software and methodology challenges. As part of the strategy, the three national supercomputing centres HLRS (Stuttgart), NIC/JSC (Jülich) and LRZ (Munich) have formed the Gauss-Centre for Supercomputing (GCS) as a new virtual organization enabled by an agreement between the Federal Ministry of Education and Research (BMBF) and the state ministries for research of Baden-Württemberg, Bayern, and Nordrhein-Westfalen from July 2006. Already today, the GCS provides the most powerful high-performance computing infrastructure in Europe. Moreover, it is expected that in the next few months – following the proposal of the German HPC community, guided by Professor Andreas Reuter (EML) – the reshape of the High Performance Computing in Germany will proceed to form the German HPC “Gauss Alliance”, with the goal to improve and establish competitiveness for the coming years. Beyond the stabilization and strengthening of the existing German infrastructures – including the necessary hardware at a worldwide competitive level – a major software research and support program to enable Computational Science and Engineering on the

VI

Preface

required level of expertise and performance – which means: running Petascale applications on more than 100,000 processors – is promised by the BMBF. It is expected that for the next years 20 Million Euro will be spend – on a yearly basis – for projects to develop scalable algorithms, methods and tools in many areas to support massively parallel systems. As we all know, we do not only need competitive hardware but also excellent software and methods to approach – and solve – the most demanding problems in science and engineering. To achieve this challenging goal every three-year project supported by that program will need to integrate excellent research groups at the universities with colleagues from the competence network of HPC centers in Germany. The success of this approach is of utmost importance for our community and also will strongly inﬂuence the development of new technologies and industrial products; beyond that, this will ﬁnally determine if Germany will be an accepted partner among the leading technology and research nations. The role of national supercomputing centers like HLRS can never be limited to the pure operation and services of hardware systems. Research in methods and tools is strongly necessary to be able to support users in exploiting the full potential of supercomputing systems. HLRS has emphasized its leading role in this ﬁeld over the last years. Most recently, as part of the German national elite research program HLRS and its research partners at the University of Stuttgart were awarded a funding of 30 Million Euro for the next ﬁve years for a cluster of excellence in Simulation Technology. With this success, the University of Stuttgart and HLRS will further strengthen their national and international position as centers for high performance computing and simulation technology. The goal of the cluster of excellence in simulation technology is to go from isolated approaches (numerical kernels, methods, and tools) to an integrated system science. Research areas will include molecular and particle simulation, advanced mechanics of multi-scale and multi-ﬁeld problems, systems analysis and inverse problems, numerical and computational mathematics, integrated data management and interactive visualization, hybrid high-performance computing systems and simulation software engineering and an integrative platform of reﬂection and evaluation. Since 1996, HLRS is supporting the scientiﬁc community as part of its ofﬁcial mission. Like in the years before, the major results of the last 12 months were reported at the Tenth Results and Review Workshop on High Performance Computing in Science and Engineering, which was held October 4–5, 2007 at Stuttgart University. This volume contains the written versions of the research work presented. The papers have been selected from all projects running at HLRS and at SSC Karlsruhe during the one year period beginning October 2006. Overall, 43 papers have been chosen from Physics, Solid State Physics, Computational Fluid Dynamics, Chemistry, and other topics. The largest number of contributions, as in many other years, came from CFD with 17 papers. Although such a small collection cannot represent a large area in total, the selected papers demonstrate the state of the art in high perfor-

Preface

VII

mance computing in Germany. The authors were encouraged to emphasize computational techniques used in solving the problems examined. This often forgotten aspect was the major focus of this volume, nevertheless this should not disregard the importance of the newly computed scientiﬁc results for the speciﬁc disciplines. We gratefully acknowledge the continued support of the Land BadenWürttemberg in promoting and supporting high performance computing. Grateful acknowledgement is also due to the Deutsche Forschungsgemeinschaft (DFG): many projects processed on the machines of HLRS and SSC could not have been carried out without the support of the DFG. Also, we thank the Springer Verlag for publishing this volume and thus helping to position the local activities into an international frame. We hope that this series of publications is contributing to the global promotion of high performance scientiﬁc computing. Stuttgart, Oktober 2007

Wolfgang E. Nagel Dietmar Kröner Michael Resch

Contents

Physics R. Speith . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

The SuperN-Project: Current Progress in Modelling Core Collapse Supernovae A. Marek, K. Kifonidis, H.-T. Janka, and B. Müller . . . . . . . . . . . . . . . . .

3

Toward Conquering the Parameter Space of Gravitational Wave Signals from Black Hole Coalescence B. Brügmann, J. Gonzalez, M. Hannam, S. Husa, U. Sperhake, and I. Christadler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Massless Four-Loop Integrals and the Total Cross Section in e+ e− Annihilation J.H. Kühn, M. Steinhauser, and M. Tentyukov . . . . . . . . . . . . . . . . . . . . . . 33 Structural Transitions in Colloidal Suspensions M. Hecht and J. Harting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Solid State Physics W. Hanke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Simulations of Strongly Correlated Quantum Systems out of Equilibrium S.R. Manmana, K. Rodriguez, S. Wessel, and A. Muramatsu . . . . . . . . . . 71 Computer Simulations of Soft Matter- and Nano-Systems K. Franzrahe, J. Neder, M. Dreher, P. Henseler, W. Quester, C. Schieback, F. Bürzle, D. Mutter, M. Schach, T. Sorg, and P. Nielaba 83 Signal Transport in and Conductance of Correlated Nanostructures P. Schmitteckert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

X

Contents

Ab initio Simulations of PbTe-CdTe Nanostructures R. Leitsmann and F. Bechstedt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 The Basic Structure of Ti-Si-N Superhard Nanocomposite Coatings: Ab Initio Studies X. Liu, B. Gottwald, C. Wang, Y. Jia, and E. Westkaemper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Chemistry C. van Wüllen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Shared Memory Parallelization of the Multi-Conﬁguration Time-Dependent Hartree Method and Application to the Dynamics and Spectroscopy of the Protonated Water-Dimer M. Brill, O. Vendrell, F. Gatti, and H.-D. Meyer . . . . . . . . . . . . . . . . . . . . 141 Green Chemistry from Supercomputers: Car–Parrinello Simulations of Emim-Chloroaluminates Ionic Liquids B. Kirchner and A.P. Seitsonen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 DFT Modelling of Oxygen Adsorption on CoCr Surfaces J. Zimmermann and L. Colombi Ciacchi . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Comparison of the Incorporation of Watson-Crick Complementary and Mismatched Nucleotides Catalyzed by DNA Polymerase I T.E. Exner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Reacting Flows D. Kröner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Assumed PDF Modeling of Turbulence Chemistry Interaction in Scramjet Combustors M. Kindler, P. Gerlinger, and M. Aigner . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Simulations of Premixed Swirling Flames Using a Hybrid Finite-Volume/Transported PDF Approach S. Lipp and U. Maas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 Computations of Premixed Turbulent Flames M. Lecanu, K. Mehravaran, J. Fröhlich, H. Bockhorn, and D. Thévenin 229 Ignition of Droplets in a Laminar Convective Environment R. Stauch and U. Maas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Contents

XI

Computational Fluid Dynamics S. Wagner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Laminar-Turbulent Transition in a Laminar Separation Bubble: Inﬂuence of Disturbance Amplitude on Bubble Size and Bursting O. Marxen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Direct Numerical Simulation on the Inﬂuence of the Nozzle Design for Water Sheets Emerged at Moderate Reynolds Numbers W. Sander and B. Weigand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 DNS of Heat Transfer from a Flat Plate Aﬀected by Free-Stream Fluctuations J.G. Wissink and W. Rodi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Direct Numerical Simulation of Turbulent Flow Over Dimples – Code Optimization for NEC SX-8 plus Flow Results M. Breuer, P. Lammers, T. Zeiser, G. Hager, and G. Wellein . . . . . . . . . 303 Direct Numerical Simulation of a Serrated Nozzle End for Jet-Noise Reduction A. Babucke, M. Kloker, and U. Rist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Direct Numerical Simulation of a Round Jet into a Crossﬂow – Analysis and Required Resources J.A. Denev, J. Fröhlich, and H. Bockhorn . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Transport of Heavy Spherical Particles in Horizontal Channel Flow M. Uhlmann and J. Fröhlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Analysis of Turbulent Structures in a Czochralski System Using DNS and LES Predictions A. Raufeisen, M. Breuer, T. Botsch, and A. Delgado . . . . . . . . . . . . . . . . . 371 Aeroacoustic Prediction of Jet and Slat Noise E. Gröschel, D. König, S. Koh, W. Schröder, and M. Meinke . . . . . . . . . 387 Investigation of the Turbulent Flow Separation from an Axisymmetric Hill M. García-Villalba and W. Rodi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 Large Eddy Simulation (LES) with Moving Meshes on a Rapid Compression Machine: Part 2: Numerical Investigations Using Euler–Lagrange-Technique F. Magagnato, M. Gabi, T. Heidenreich, A. Velji, and U. Spicher . . . . . . 419 Numerical Characterization of the Non-Reacting Flow in a Swirled Gasturbine Model Combustor A. Widenhorn, B. Noll, M. Stöhr, and M. Aigner . . . . . . . . . . . . . . . . . . . . 431

XII

Contents

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers for Two-Phase Flow Simulations D. Gaudlitz and N.A. Adams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 Lattice Boltzmann Simulations of Microemulsions and Binary Immiscible Fluids Under Shear J. Harting and G. Giupponi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 Numerical Investigation of Hypersonic Intake Flows M. Krause and J. Ballmann . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471 Trimmed Simulation of a Complete Helicopter Conﬁguration Using Fluid-Structure Coupling M. Dietz, M. Kessler, and E. Krämer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 FEAST: Development of HPC Technologies for FEM Applications C. Becker, S. Buijssen, and S. Turek . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503 Transport and Climate C. Kottmeier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517 Global Long-Term MIPAS Processing M. Kiefer, U. Grabowski, and H. Fischer . . . . . . . . . . . . . . . . . . . . . . . . . . . 519 Modelling the Regional Climate of Southwest Germany: Sensitivity to Simulation Setup C. Meissner and G. Schaedler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 OpenMP Parallelization of the METRAS Meteorology Model: Application to the America’s Cup W. Augustin, V. Heuveline, G. Meschkat, K.H. Schlünzen, and G. Schroeder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 Structural Mechanics P. Wriggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561 Adaptive Analysis of Bifurcation Points of Shell Structures E. Ewert and K. Schweizerhof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 Miscellaneous Topics W. Schröder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 Molecular Modeling of Hydrogen Bonding Fluids: Formic Acid and Ethanol + R227ea T. Schnabel, B. Eckl, Y.-L. Huang, J. Vrabec, and H. Hasse . . . . . . . . . . . 573 Modeling Elastic and Poroelastic Wave Propagation in Complex Geological Structures F. Wenzlau, T. Xia, and T.M. Müller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587

Contents

XIII

Whole-Mantle Convection, Continent Generation, and Preservation of Geochemical Heterogeneity U. Walzer, R. Hendel, and J. Baumgardner . . . . . . . . . . . . . . . . . . . . . . . . . 603 Numerical Simulation of Human Radiation Heat Transfer Using a Mathematical Model of Human Physiology and Computational Fluid Dynamics (CFD) R. Yousaf, D. Fiala, and A. Wagner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647 Parallel Finite Element Methods with Weighted Linear B-Splines K. Höllig, J. Hörner, and M. Pfeil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667

The SuperN-Project: Current Progress in Modelling Core Collapse Supernovae A. Marek, K. Kifonidis, H.-Th. Janka, and B. M¨ uller Max-Planck-Institut f¨ ur Astrophysik, Karl-Schwarzschild-Straße 1, Postfach 1317, D-85741 Garching bei M¨ unchen, Germany [email protected] Summary. We give an overview of the problems and the current status of our twodimensional (core collapse) supernova modelling, and discuss the system of equations and the algorithm for its solution that are employed in our code. In particular we report our recent progress, and focus on the ongoing calculations that are performed on the NEC SX-8 at the HLRS Stuttgart. Especially, we will argue that it might be possible that neutrino-driven supernova explosions set in at much later times than previously considered. This, of course, enhances the need of a code that can make eﬃcient use of the multi-node capability of the NEC SX-8 for long-time simulations of the postbounce evolution of collapsing stellar cores.

1 Introduction A star more massive than about 8 solar masses ends its live in a cataclysmic explosion, a supernova. Its quiescent evolution comes to an end, when the pressure in its inner layers is no longer able to balance the inward pull of gravity. Throughout its life, the star sustained this balance by generating energy through a sequence of nuclear fusion reactions, forming increasingly heavier elements in its core. However, when the core consists mainly of irongroup nuclei, central energy generation ceases. The fusion reactions producing iron-group nuclei relocate to the core’s surface, and their “ashes” continuously increase the core’s mass. Similar to a white dwarf, such a core is stabilised against gravity by the pressure of its degenerate gas of electrons. However, to remain stable, its mass must stay smaller than the Chandrasekhar limit. When the core grows larger than this limit, it collapses to a neutron star, and a huge amount (∼ 1053 erg) of gravitational binding energy is set free. Most (∼ 99%) of this energy is radiated away in neutrinos, but a small fraction is transferred to the outer stellar layers and drives the violent mass ejection which disrupts the star in a supernova. Despite 40 years of research, the details of how this energy transfer happens and how the explosion is initiated are still not well understood. Observational

4

A. Marek et al.

evidence about the physical processes deep inside the collapsing star is sparse and almost exclusively indirect. The only direct observational access is via measurements of neutrinos or gravitational waves. To obtain insight into the events in the core, one must therefore heavily rely on sophisticated numerical simulations. The enormous amount of computer power required for this purpose has led to the use of several, often questionable, approximations and numerous ambiguous results in the past. Fortunately, however, the development of numerical tools and computational resources has meanwhile advanced to a point, where it is becoming possible to perform multi-dimensional simulations with unprecedented accuracy. Therefore there is hope that the physical processes which are essential for the explosion can finally be unravelled. An understanding of the explosion mechanism is required to answer many important questions of nuclear, gravitational, and astro-physics like the following: • How do the explosion energy, the explosion timescale, and the mass of the compact remnant depend on the progenitor’s mass? Is the explosion mechanism the same for all progenitors? For which stars are black holes left behind as compact remnants instead of neutron stars? • What is the role of the – poorly known – equation of state (EoS) for the proto neutron star? Do softer or stiﬀer EoSs favour the explosion of a core collapse supernova? • What is the role of rotation during the explosion? How rapidly do newly formed neutron stars rotate? • How do neutron stars receive their natal kicks? Are they accelerated by asymmetric mass ejection and/or anisotropic neutrino emission? • What are the generic properties of the neutrino emission and of the gravitational wave signal that are produced during stellar core collapse and explosion? Up to which distances could these signals be measured with operating or planned detectors on earth and in space? And what can one learn about supernova dynamics from a future measurement of such signals in case of a Galactic supernova?

2 Numerical Models 2.1 History and Constraints According to theory, a shock wave is launched at the moment of “core bounce” when the neutron star begins to emerge from the collapsing stellar iron core. There is general agreement, supported by all “modern” numerical simulations, that this shock is unable to propagate directly into the stellar mantle and envelope, because it looses too much energy in dissociating iron into free nucleons while it moves through the outer core. The “prompt” shock ultimately stalls. Thus the currently favoured theoretical paradigm needs to exploit the fact that a huge energy reservoir is present in the form of neutrinos, which are

Simulations of Supernovae

5

abundantly emitted from the hot, nascent neutron star. The absorption of electron neutrinos and antineutrinos by free nucleons in the post shock layer is thought to reenergize the shock, and lead to the supernova explosion. Detailed spherically symmetric hydrodynamic models, which recently include a very accurate treatment of the time-dependent, multi-flavour, multifrequency neutrino transport based on a numerical solution of the Boltzmann transport equation [1, 2, 3], reveal that this “delayed, neutrino-driven mechanism” does not work as simply as originally envisioned. Although in principle able to trigger the explosion (e.g., [4, 5, 6]), neutrino energy transfer to the postshock matter turned out to be too weak. For inverting the infall of the stellar core and initiating powerful mass ejection, an increase of the eﬃciency of neutrino energy deposition is needed. A number of physical phenomena have been pointed out that can enhance neutrino energy deposition behind the stalled supernova shock. They are all linked to the fact that the real world is multi-dimensional instead of spherically symmetric (or one-dimensional; 1D) as assumed in the work cited above: (1) Convective instabilities in the neutrino-heated layer between the neutron star and the supernova shock develop to violent convective overturn [7]. This convective overturn is helpful for the explosion, mainly because (a) neutrino-heated matter rises and increases the pressure behind the shock, thus pushing the shock further out, and (b) cool matter is able to penetrate closer to the neutron star where it can absorb neutrino energy more eﬃciently. Both eﬀects allow multi-dimensional models to explode easier than spherically symmetric ones [8, 9, 10]. (2) Recent work [11, 12, 13, 14] has demonstrated that the stalled supernova shock is also subject to a second non-radial low-mode instability, called SASI, which can grow to a dipolar, global deformation of the shock [14, 15]. (3) Convective energy transport inside the nascent neutron star [16, 17, 18, 19] might enhance the energy transport to the neutrinosphere and could thus boost the neutrino luminosities. This would in turn increase the neutrinoheating behind the shock. This list of multi-dimensional phenomena awaits more detailed exploration in multi-dimensional simulations. Until recently, such simulations have been performed with only a grossly simplified treatment of the involved microphysics, in particular of the neutrino transport and neutrino-matter interactions. At best, grey (i.e., single energy) flux-limited diﬀusion schemes were employed. All published successful simulations of supernova explosions by the convectivly aided neutrino-heating mechanism in two [8, 9, 20] and three dimensions [21, 22] used such a radical approximation of the neutrino transport. Since, however, the role of the neutrinos is crucial for the problem, and because previous experience shows that the outcome of simulations is indeed very sensitive to the employed transport approximations, studies of the explosion mechanism require the best available description of the neutrino physics.

6

A. Marek et al.

This implies that one has to solve the Boltzmann transport equation for neutrinos. 2.2 Recent Calculations and the Need for TFlop Simulations We have recently advanced to a new level of accuracy for supernova simulations by generalising the VERTEX code, a Boltzmann solver for neutrino transport, from spherical symmetry [23] to multi-dimensional applications [24, 25]. The corresponding mathematical model, and in particular our method for tackling the integro-diﬀerential transport problem in multidimensions, will be summarised in Sect. 3. Results of a set of simulations with our code in 1D and 2D for progenitor stars with diﬀerent masses have recently been published by [25, 26], and with respect to the expected gravitational-wave signals from rotating and convective supernova cores by [27]. The recent progress in supernova modelling was summarised and set in perspective in a conference article by [24]. Our collection of simulations has helped us to identify a number of eﬀects which have brought our two-dimensional models close to the threshold of explosion. This makes us optimistic that the solution of the long-standing problem of how massive stars explode may be in reach. In particular, we have recognised the following aspects as advantageous: • The details of the stellar progenitor (i.e. the mass of the iron core and its radius–density relation) have substantial influence on the supernova evolution. Especially, we found explosions of stellar models with low-mass (i.e. small) iron cores [26, 28], whereas more massive stars resist the explosion more persistent [25]. Thus detailed studies with diﬀerent progenitor models are necessary. • Stellar rotation, even at a moderate level, supports the expansion of the stalled shock by centrifugal forces and instigates overturn motion in the neutrino-heated postshock matter by meridional circulation flows in addition to convective instabilities. All these eﬀects are potentially important, and some (or even all of them) may represent crucial ingredients for a successful supernova simulation. So far no multi-dimensional calculations have been performed, in which two or more of these items have been taken into account simultaneously, and thus their mutual interaction awaits to be investigated. It should also be kept in mind that our knowledge of supernova microphysics, and especially the EoS of neutron star matter, is still incomplete, which implies major uncertainties for supernova modelling. Unfortunately, the impact of diﬀerent descriptions for this input physics has so far not been satisfactorily explored with respect to the neutrino-heating mechanism and the long-time behaviour of the supernova shock, in particular in multi-dimensional models. However, first

Simulations of Supernovae

7

multi-dimensional simulations of core collapse supernovae with diﬀerent nuclear EoSs [19, 29] show a strong dependence of the supernova evolution on the EoS. From this it is clear that rather extensive parameter studies using multidimensional simulations are required to identify the physical processes which are essential for the explosion. Since on a dedicated machine performing at a sustained speed of about 30 GFlops already a single 2D simulation has a turn-around time of more than half a year, these parameter studies are not possible without TFlop simulations.

3 The Mathematical Model The non-linear system of partial diﬀerential equations which is solved in our code consists of the following components: • • • • •

The Euler equations of hydrodynamics, supplemented by advection equations for the electron fraction and the chemical composition of the fluid, and formulated in spherical coordinates; the Poisson equation for calculating the gravitational source terms which enter the Euler equations, including corrections for general relativistic effects; the Boltzmann transport equation which determines the (non-equilibrium) distribution function of the neutrinos; the emission, absorption, and scattering rates of neutrinos, which are required for the solution of the Boltzmann equation; the equation of state of the stellar fluid, which provides the closure relation between the variables entering the Euler equations, i.e. density, momentum, energy, electron fraction, composition, and pressure.

In what follows we will briefly summarise the neutrino transport algorithms. For a more complete description of the entire code we refer the reader to [25], and the references therein. 3.1 “Ray-by-Ray Plus” Variable Eddington Factor Solution of the Neutrino Transport Problem The crucial quantity required to determine the source terms for the energy, momentum, and electron fraction of the fluid owing to its interaction with the neutrinos is the neutrino distribution function in phase space, f (r, ϑ, φ, ǫ, Θ, Φ, t). Equivalently, the neutrino intensity I = c/(2πc)3 · ǫ3 f may be used. Both are seven-dimensional functions, as they describe, at every point in space (r, ϑ, φ), the distribution of neutrinos propagating with energy ǫ into the direction (Θ, Φ) at time t (Fig. 1). The evolution of I (or f ) in time is governed by the Boltzmann equation, and solving this equation is, in general, a six-dimensional problem (as time

8

A. Marek et al. Fig. 1. Illustration of the phase space coordinates (see the main text)

is usually not counted as a separate dimension). A solution of this equation by direct discretisation (using an SN scheme) would require computational resources in the PetaFlop range. Although there are attempts by at least one group in the United States to follow such an approach, we feel that, with the currently available computational resources, it is mandatory to reduce the dimensionality of the problem. Actually this should be possible, since the source terms entering the hydrodynamic equations are integrals of I over momentum space (i.e. over ǫ, Θ, and Φ), and thus only a fraction of the information contained in I is truly required to compute the dynamics of the flow. It makes therefore sense to consider angular moments of I, and to solve evolution equations for these moments, instead of dealing with the Boltzmann equation directly. The 0th to 3rd order moments are defined as 1 J, H, K, L, . . . (r, ϑ, φ, ǫ, t) = I(r, ϑ, φ, ǫ, Θ, Φ, t) n0,1,2,3,... dΩ (1) 4π where dΩ = sin Θ dΘ dΦ, n = (cos Θ, sin Θ cos Φ, sin Θ sin Φ), and exponentiation represents repeated application of the dyadic product. Note that the moments are tensors of the required rank. This leaves us with a four-dimensional problem. So far no approximations have been made. In order to reduce the size of the problem even further, one needs to resort to assumptions on its symmetry. At this point, one usually employs azimuthal symmetry for the stellar matter distribution, i.e. any dependence on the azimuth angle φ is ignored, which implies that the hydrodynamics of the problem can be treated in two dimensions. It also implies I(r, ϑ, ǫ, Θ, Φ) = I(r, ϑ, ǫ, Θ, −Φ). If, in addition, it is assumed that I is even independent of Φ, then each of the angular moments of I becomes a scalar, which depends on two spatial dimensions, and one dimension in momentum space: J, H, K, L = J, H, K, L(r, ϑ, ǫ, t). Thus we have reduced the problem to three dimensions in total.

Simulations of Supernovae

9

The System of Equations With the aforementioned assumptions it can be shown [25], that in order to compute the source terms for the energy and electron fraction of the fluid, the following two transport equations need to be solved: „ « ∂(sin ϑβϑ ) 1 ∂(r 2 βr ) 1 + J +J r 2 ∂r r sin ϑ ∂ϑ j j „ «ﬀ ﬀ 2 1 ∂(r H) βr ∂H ∂ ǫ ∂βr ∂ 1 ∂(sin ϑβϑ ) βr + 2 ǫJ + − H − + r ∂r c ∂t ∂ǫ c ∂t ∂ǫ r 2r sin ϑ ∂ϑ j „ «ﬀ „ « ∂(sin ϑβϑ ) ∂(sin ϑβϑ ) ∂βr βr βr 1 1 ∂ − − + ǫK +J − ∂ǫ ∂r r 2r sin ϑ ∂ϑ r 2r sin ϑ ∂ϑ „ « ∂βr βr 1 2 ∂βr ∂(sin ϑβϑ ) +K + − − H = C (0) , (2) ∂r r 2r sin ϑ ∂ϑ c ∂t „

∂ βϑ ∂ 1 ∂ + βr + c ∂t ∂r r ∂ϑ

„

∂ βϑ ∂ 1 ∂ + βr + c ∂t ∂r r ∂ϑ

«

« ∂(sin ϑβϑ ) 1 ∂(r 2 βr ) 1 + r 2 ∂r r sin ϑ ∂ϑ « j „ ﬀ ∂K βr ∂K 3K − J ∂ ǫ ∂βr ∂βr + + + +H − K ∂r r ∂r c ∂t ∂ǫ c ∂t j „ «ﬀ ∂(sin ϑβϑ ) ∂βr βr 1 ∂ − − ǫL − ∂ǫ ∂r r 2r sin ϑ ∂ϑ j „ «ﬀ ∂ 1 ∂βr ∂(sin ϑβϑ ) 1 βr − ǫH + + (J + K) = C (1) . ∂ǫ r 2r sin ϑ ∂ϑ c ∂t «

H +H

„

(3)

These are evolution equations for the neutrino energy density, J, and the neutrino flux, H, and follow from the zeroth and first moment equations of the comoving frame (Boltzmann) transport equation in the Newtonian, O(v/c) approximation. The quantities C (0) and C (1) are source terms that result from the collision term of the Boltzmann equation, while βr = vr /c and βϑ = vϑ /c, where vr and vϑ are the components of the hydrodynamic velocity, and c is the speed of light. The functional dependences βr = βr (r, ϑ, t), J = J(r, ϑ, ǫ, t), etc. are suppressed in the notation. This system includes four unknown moments (J, H, K, L) but only two equations, and thus needs to be supplemented by two more relations. This is done by substituting K = fK · J and L = fL · J, where fK and fL are the variable Eddington factors, which for the moment may be regarded as being known, but in our case is indeed determined from a separate simplified (“model”) Boltzmann equation. A finite volume discretisation of (2–3) is suﬃcient to guarantee exact conservation of the total neutrino energy. However, and as described in detail in [23], it is not suﬃcient to guarantee also exact conservation of the neutrino number. To achieve this, we discretise and solve a set of two additional equations. With J = J/ǫ, H = H/ǫ, K = K/ǫ, and L = L/ǫ, this set of equations reads

10

A. Marek et al.

„ « ∂(sin ϑβϑ ) 1 1 ∂(r 2 βr ) + J +J r 2 ∂r r sin ϑ ∂ϑ ﬀ j j „ «ﬀ 2 ∂(sin ϑβϑ ) βr βr ∂H ∂ ǫ ∂βr ∂ 1 1 ∂(r H) + − H − + ǫJ + 2 r ∂r c ∂t ∂ǫ c ∂t ∂ǫ r 2r sin ϑ ∂ϑ j „ «ﬀ ∂ ∂(sin ϑβϑ ) βr 1 1 ∂βr ∂βr − − − H = C (0) , (4) ǫK + ∂ǫ ∂r r 2r sin ϑ ∂ϑ c ∂t „

1 ∂ ∂ βϑ ∂ + βr + c ∂t ∂r r ∂ϑ

«

« ∂(sin ϑβϑ ) 1 ∂(r 2 βr ) 1 + r 2 ∂r r sin ϑ ∂ϑ « j „ ﬀ ∂K 3K − J ∂ ǫ ∂βr ∂βr βr ∂K + + + +H − K ∂r r ∂r c ∂t ∂ǫ c ∂t j „ «ﬀ ∂(sin ϑβϑ ) ∂βr βr 1 ∂ − − ǫL − ∂ǫ ∂r r 2r sin ϑ ∂ϑ j „ «ﬀ „ « ∂ 1 βr 1 ∂(sin ϑβϑ ) ∂(sin ϑβϑ ) βr ∂βr − ǫH −L + − − ∂ǫ r 2r sin ϑ ∂ϑ ∂r r 2r sin ϑ ∂ϑ « „ ∂(sin ϑβϑ ) 1 1 ∂βr βr + J = C (1) . (5) + −H r 2r sin ϑ ∂ϑ c ∂t „

∂ βϑ ∂ 1 ∂ + βr + c ∂t ∂r r ∂ϑ

«

H+H

„

The moment equations (2–5) are very similar to the O(v/c) equations in spherical symmetry which were solved in the 1D simulations of [23] (see Eqs. 7, 8, 30, and 31 of the latter work). This similarity has allowed us to reuse a good fraction of the one-dimensional version of VERTEX, for coding the multidimensional algorithm. The additional terms necessary for this purpose have been set in boldface above. Finally, the changes of the energy, e, and electron fraction, Ye , required for the hydrodynamics are given by the following two equations 4π ∞ de =− Cν(0) (ǫ), (6) dǫ dt ρ 0 ν∈(νe ,¯ νe ,... ) 4π mB ∞ (0) dYe (0) =− dǫ Cνe (ǫ) − Cν¯e (ǫ) (7) dt ρ 0 (for the momentum source terms due to neutrinos see [25]). Here mB is the baryon mass, and the sum in (6) runs over all neutrino types. The full system consisting of (2–7) is stiﬀ, and thus requires an appropriate discretisation scheme for its stable solution. Method of Solution In order to discretise (2–7), the spatial domain [0, rmax ] × [ϑmin , ϑmax ] is covered by Nr radial, and Nϑ angular zones, where ϑmin = 0 and ϑmax = π correspond to the north and south poles, respectively, of the spherical grid. (In general, we allow for grids with diﬀerent radial resolutions in the neutrino transport and hydrodynamic parts of the code. The number of radial zones

Simulations of Supernovae

11

for the hydrodynamics will be denoted by Nrhyd .) The number of bins used in energy space is Nǫ and the number of neutrino types taken into account is Nν . The equations are solved in two operator-split steps corresponding to a lateral and a radial sweep. In the first step, we treat the boldface terms in the respectively first lines of (2–5), which describe the lateral advection of the neutrinos with the stellar fluid, and thus couple the angular moments of the neutrino distribution of neighbouring angular zones. For this purpose we consider the equation 1 ∂(sin ϑ βϑ Ξ) 1 ∂Ξ + = 0, c ∂t r sin ϑ ∂ϑ

(8)

where Ξ represents one of the moments J, H, J , or H. Although it has been suppressed in the above notation, an equation of this form has to be solved for each radius, for each energy bin, and for each type of neutrino. An explicit upwind scheme is used for this purpose. In the second step, the radial sweep is performed. Several points need to be noted here: •

•

•

terms in boldface not yet taken into account in the lateral sweep, need to be included into the discretisation scheme of the radial sweep. This can be done in a straightforward way since these remaining terms do not include derivatives of the transport variables (J, H) or (J , H). They only depend on the hydrodynamic velocity vϑ , which is a constant scalar field for the transport problem. the right hand sides (source terms) of the equations and the coupling in energy space have to be accounted for. The coupling in energy is non-local, since the source terms of (2–5) stem from the Boltzmann equation, which is an integro-diﬀerential equation and couples all the energy bins the discretisation scheme for the radial sweep is implicit in time. Explicit schemes would require very small time steps to cope with the stiﬀness of the source terms in the optically thick regime, and the small CFL time step dictated by neutrino propagation with the speed of light in the optically thin regime. Still, even with an implicit scheme 105 time steps are required per simulation. This makes the calculations expensive.

Once the equations for the radial sweep have been discretized in radius and energy, the resulting solver is applied ray-by-ray for each angle ϑ and for each type of neutrino, i.e. for constant ϑ, Nν two-dimensional problems need to be solved. The discretisation itself is done using a second order accurate scheme with backward diﬀerencing in time according to [23]. This leads to a non-linear system of algebraic equations, which is solved by Newton-Raphson iteration with explicit construction and inversion of the corresponding Jacobian matrix.

12

A. Marek et al.

4 Recent Results and Ongoing Work We make use of the computer resources available to us at the HLRS to address some of the important questions in SN theory (see Sect. 1) with 2Dsimulations. At the HLRS, we typically run our code on one node of the NEC SX-8 (8 processors, OpenMP-parallised) of the SX-8 with 98.3% of vector operations and 22000 MFLOPS per second. In the following we present some of our results from these simulations that are currently performed at the HLRS. For the neutrino interaction rates we use the full set as described in [30] and general relativistic eﬀects are taken into account according to [31]. 4.1 The Importance of Hydrodynamic Instabilities Inside the Supernova Core Buras et al. [26] recently reported two simulations of an 11.2 M⊙ stellar progenitor model with two diﬀerent setups: on the one hand they calculated a 90◦ wedge centred around the equatorial plane with periodic boundary conditions (from here on called model s11.2-wedge). On the other hand a full 180◦ model (from north to south pole) with reflecting boundary conditions was calculated (from here on this model is called s11.2-180). Interestingly, the latter model showed an explosion whereas the former one failed to explode. In order to investigate whether the diﬀerent wedge size or the diﬀerent boundary conditions caused this qualitative diﬀerence of the simulations, we have performed simulations for two additional models: the first one, model s11.2-90, employed a 90◦ wedge from the pole to the equator with reflecting boundary conditions. The second model, s11.2-wedge-refl, was chosen with a setup as model s11.2-wedge, however, reflecting boundary conditions were used. The basic setup of all four models is sketched in Fig. 2. We found that both models containing the polar axis developed explosions and that the wedge models did not explode independent of the boundary

Fig. 2. Schematic overview over the setups chosen for model s11.2-180 (a), s11.2-90 (b), s11.2-wedge-reﬂ (c), and s11.2-wedge (d). Blue arrows indicate the use of reﬂecting boundary conditions, whereas red arrows indicate the use of periodic boundary conditions. The sketches are oriented such that the equatorial plane is represented by the horizontal line

Simulations of Supernovae

13

condition, although all models show a similar strength of neutrino heating. A more detailed analysis [19] showed that in the exploding models a strong non-radial hydrodynamical instability of the accretion shock, the so-called SASI [13], developed, which was suppressed in the non-exploding (wedge) models. A projection of the shock-positions onto spherical harmonics Yl,m , rshock (θ, t) =

∞

al,0 (t)Yl,0 (θ) ,

(9)

l=0

where m is set to zero due to the adopted axial symmetry of our 2D calculations, clearly shows the presence of this time and angle dependent shock instability, see Fig. 3.

a

b

Fig. 3. Time-dependent coeﬃcients of the expansion of the shock position into spherical harmonics (cf. (9)) Note that the the coeﬃcients are normalised to the amplitude of the l = 0 mode. a: The coeﬃcients corresponding to the l = 1 mode. b: The coeﬃcients corresponding to the l = 2 mode

From this, together with (semi) analytical arguments given in [12, 13] (see also Sect. 2.1), we conclude that the SASI can play a crucial role for the development of an explosion. Indeed up to now all multi-dimensional explosion models of the Garching supernova group showed such kind if instabilities gaining strength before the onset of the explosion (see also paragraph 4.2). 4.2 Eﬀects of Rotation As we have already described in our last report [29], we have started a full 180◦ simulation of a 15 M⊙ progenitor star including rotation. This still ongoing simulation, model L&S-rot, – it was partly performed at the SX-8 of the HLRS – is now the longest (tpb > 600 ms) multidimensional Boltzmann neutrino transport simulation worldwide. As we argued in our last report, the reason for pushing this simulation to such late times is that rotation and angular momentum become more and more important at later times as matter has

14

A. Marek et al.

fallen from larger radii to the shock position. During all the simulated supernova evolution we find the presence of the already mentioned SASI low-mode deformation of the accretion layer and shock (see previous Sect. 4.1), and at a time of roughly 500 ms after the shock formation we observe the onset of an explosion (see Fig. 4). However, it is yet not clear whether the presence of rotation is crucial for the explosion of this 15 M⊙ model, or whether this

Fig. 4. The shock position (solid white line) at the north pole (upper panel ) and south pole (lower panel ) of the rotating 15 M⊙ model as function of postbounce time. Colour coded is the entropy of the stellar ﬂuid

Fig. 5. The ratio of the advection timescale to the heating timescale for the rotating model L&S-rot and the non-rotating model L&S-2D. Also shown is model L&S-rot90 which is identical to model L&S-rot except for the computational domain that does not extend from pole to pole but from the north pole to the equator (see also Sect. 4.1). The advection timescale is the characteristic timescale that matter stays inside the heating region before it is advected to the proto-neutron star. The heating timescale is the typical timescale that matter needs to be exposed to neutrino heating for observing enough energy to become gravitationally unbound

Simulations of Supernovae

15

model would also explode without rotation. Since the comparison of the rotating and a corresponding non-rotating model reveals qualitatively the same behaviour, see e.g. Fig. 5, it is absolutely necessary to evolve both models to a time of more than 500 ms after the shock formation in order to answer this question. Although the importance of rotation in this particular model is not yet fully understood, this calculation nevertheless shows that the convectivly or SASI supported neutrino-heating mechanism is viable to produce explosions for massive progenitors (M > 10 M⊙ ) much later than previously considered.

5 Conclusions and Outlook We continued to simulate well resolved 2D models of core collapse supernovae with detailed neutrino transport at the HLRS. We found that non-radial hydrodynamic instabilities support the onset of supernova explosions and for a 15 M⊙ progenitor model we obtained a supernova explosion at a time of roughly 600 ms after the shock formation. These results indicate that supernova explosions might develop at much later times than previously thought, which has to be investigated in more detail in the future. Especially the nonrotating model has to be evolved to a time of roughly 700 ms after the shock formation, in order to clarify the importance of rotation in the exploding model. All in all, this means longer evolution times per model and thus emphasises the need for very eﬃcient numerical tools. With support by the staﬀ of the HLRS, our code is currently optimised for multi-node usage on the NEC SX-8 which will allow us to evolve our supernova models faster to longer evolution times than it is possible at the moment. This will be crucial for investigating the importance of the stellar rotation as well as the possibility of supernova explosions at late times (t > 0.5 s). Acknowledgements Support from the SFB 375 “Astroparticle Physics”, SFB/TR7 “Gravitationswellenastronomie”, and SFB/TR27 “Neutrinos and Beyond” of the Deutsche Forschungsgemeinschaft, and computer time at the HLRS and the Rechenzentrum Garching are acknowledged. We also thank M. Galle and R. Fischer for performing the benchmarks on the NEC machines. We thank especially K. Benkert for further optimising our code for the SX-8 architecture. The simulations were performed on the national super computer NEXC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number SuperN/12758.

16

A. Marek et al.

References 1. Rampp, M., Janka, H.T.: Spherically Symmetric Simulation with Boltzmann Neutrino Transport of Core Collapse and Postbounce Evolution of a 15 M⊙ Star. Astrophys. J. 539 (2000) L33–L36 2. Mezzacappa, A., Liebend¨ orfer, M., Messer, O.E., Hix, W.R., Thielemann, F., Bruenn, S.W.: Simulation of the Spherically Symmetric Stellar Core Collapse, Bounce, and Postbounce Evolution of a Star of 13 Solar Masses with Boltzmann Neutrino Transport, and Its Implications for the Supernova Mechanism. Phys. Rev. Letters 86 (2001) 1935–1938 3. Liebend¨ orfer, M., Mezzacappa, A., Thielemann, F., Messer, O.E., Hix, W.R., Bruenn, S.W.: Probing the gravitational well: No supernova explosion in spherical symmetry with general relativistic Boltzmann neutrino transport. Phys. Rev. D 63 (2001) 103004–+ 4. Bethe, H.A.: Supernova mechanisms. Reviews of Modern Physics 62 (1990) 801–866 5. Burrows, A., Goshy, J.: A Theory of Supernova Explosions. Astrophys. J. 416 (1993) L75 6. Janka, H.T.: Conditions for shock revival by neutrino heating in core-collapse supernovae. Astron. Astrophys. 368 (2001) 527–560 7. Herant, M., Benz, W., Colgate, S.: Postcollapse hydrodynamics of SN 1987A – Two-dimensional simulations of the early evolution. Astrophys. J. 395 (1992) 642–653 8. Herant, M., Benz, W., Hix, W.R., Fryer, C.L., Colgate, S.A.: Inside the supernova: A powerful convective engine. Astrophys. J. 435 (1994) 339 9. Burrows, A., Hayes, J., Fryxell, B.A.: On the nature of core-collapse supernova explosions. Astrophys. J. 450 (1995) 830 10. Janka, H.T., M¨ uller, E.: Neutrino heating, convection, and the mechanism of Type-II supernova explosions. Astron. Astrophys. 306 (1996) 167–+ 11. Thompson, C.: Accretional Heating of Asymmetric Supernova Cores. Astrophys. J. 534 (2000) 915–933 12. Foglizzo, T.: Non-radial instabilities of isothermal Bondi accretion with a shock: Vortical-acoustic cycle vs. post-shock acceleration. Astron. Astrophys. 392 (2002) 353–368 13. Blondin, J.M., Mezzacappa, A., DeMarino, C.: Stability of Standing Accretion Shocks, with an Eye toward Core-Collapse Supernovae. Astrophys. J. 584 (2003) 971–980 14. Scheck, L., Plewa, T., Janka, H.T., Kifonidis, K., M¨ uller, E.: Pulsar Recoil by Large-Scale Anisotropies in Supernova Explosions. Phys. Rev. Letters 92 (2004) 011103–+ 15. Scheck, L.: Multidimensional simulations of core collapse supernovae. PhD thesis, Technische Universit¨ at M¨ unchen (2006) 16. Keil, W., Janka, H.T., Mueller, E.: Ledoux Convection in Protoneutron Stars – A Clue to Supernova Nucleosynthesis? Astrophys. J. 473 (1996) L111 17. Burrows, A., Lattimer, J.M.: The birth of neutron stars. Astrophys. J. 307 (1986) 178–196 18. Pons, J.A., Reddy, S., Prakash, M., Lattimer, J.M., Miralles, J.A.: Evolution of Proto-Neutron Stars. Astrophys. J. 513 (1999) 780–804

Simulations of Supernovae

17

19. Marek, A.: Multi-dimensional simulations of core collapse supernovae with different equations of state for hot proto-neutron stars. PhD thesis, Technische Universit¨ at M¨ unchen (2007) 20. Fryer, C.L., Heger, A.: Core-Collapse Simulations of Rotating Stars. Astrophys. J. 541 (2000) 1033–1050 21. Fryer, C.L., Warren, M.S.: Modeling Core-Collapse Supernovae in Three Dimensions. Astrophys. J. 574 (2002) L65–L68 22. Fryer, C.L., Warren, M.S.: The Collapse of Rotating Massive Stars in Three Dimensions. Astrophys. J. 601 (2004) 391–404 23. Rampp, M., Janka, H.T.: Radiation hydrodynamics with neutrinos. Variable Eddington factor method for core-collapse supernova simulations. Astron. Astrophys. 396 (2002) 361–392 24. Janka, H.T., Buras, R., Kifonidis, K., Marek, A., Rampp, M.: Core-Collapse Supernovae at the Threshold. In Marcaide, J.M., Weiler, K.W., eds.: Supernovae, Procs. of the IAU Coll. 192, Berlin, Springer (2004) 25. Buras, R., Rampp, M., Janka, H.T., Kifonidis, K.: Two-dimensional hydrodynamic core-collapse supernova simulations with spectral neutrino transport. I. Numerical method and results for a 15 M⊙ star. Astron. Astrophys. 447 (2006) 1049–1092 26. Buras, R., Janka, H.T., Rampp, M., Kifonidis, K.: Two–dimensional hydrodynamic core–collapse supernova simulations with spectral neutrino transport. II. Models for diﬀerent progenitor stars. Astron. Astrophys. 457 (2006) 281–308 27. M¨ uller, E., Rampp, M., Buras, R., Janka, H.T., Shoemaker, D.H.: Toward Gravitational Wave Signals from Realistic Core-Collapse Supernova Models. Astrophys. J. 603 (2004) 221–230 28. Kitaura, F.S., Janka, H.T., Hillebrandt, W.: Explosions of O–Ne–Mg cores, the Crab supernova, and subluminous type II–P supernovae. Astron. Astrophys. 450 (2006) 345–350 29. Marek, A., Kifonidis, K., Janka, H.T., M¨ uller, B.: The supern-project: Understanding core collapse supernovae. In Nagel, W.E., J¨ager, W., Resch, M., eds.: High Performance Computing in Science and Engineering 06, Berlin, Springer (2006) 30. Marek, A., Janka, H.T., Buras, R., Liebend¨ orfer, M., Rampp, M.: On ion-ion correlation eﬀects during stellar core collapse. Astron. Astrophys. 443 (2005) 201–210 31. Marek, A., Dimmelmeier, H., Janka, H.T., M¨ uller, E., Buras, R.: Exploring the relativistic regime with Newtonian hydrodynamics: an improved eﬀective gravitational potential for supernova simulations. Astron. Astrophys. 445 (2006) 273–289

Toward Conquering the Parameter Space of Gravitational Wave Signals from Black Hole Coalescence Bernd Br¨ ugmann1 , Jose Gonzalez1 , Mark Hannam1 , Sascha Husa1 , Ulrich Sperhake1 , and Iris Christadler2 1

2

Theoretisch Physikalisches Institut, Universit¨at Jena, Max-Wien-Platz 1, 07743 Jena, Germany [email protected] Leibniz Rechenzentrum der Bayerischen Akademie der Wissenschaften, Boltzmannstraße 1, 85748 Garching, Germany

1 Introduction We present a status report on our second year of work on the numerical simulation of gravitational wave signals from coalescing black holes at HLRS. We report in particular on a significant number of important results we have obtained, and on progress with optimizing our algorithms and code. Gravitational waves and their sources are described by the Einstein equations, which underly the theory of general relativity (see e.g., our previous report [23], or the overview article [31]). A large international eﬀort is currently underway to detect gravitational waves and start a new field of astrophysical research, gravitational wave astronomy. Within Germany, this research is conducted within the SFB/TR 7 “Gravitational Wave Astronomy”. In our work we solve the Einstein equations numerically, mainly with finite-diﬀerence methods. Essentially, the solution process has two parts: first, appropriate initial data have to be constructed by solving elliptic constraint equations, which in some sense generalize the divergence constraints of electromagnetism. Second, these data are evolved by solving a system of hyperbolic equations. The hyperbolic character of the evolution equations expresses the fact that in general relativity physical information propagates at the speed of light. The particular type of gravitational wave source that we study is the inspiral and coalescence of black-hole binaries. The eﬃciency of gravitational wave sources is related to their compactness: A weak field calculation yields the loss of energy of a system as

20

B. Br¨ ugmann et al.

2 G d3 Qij dE =− 5 dt 5c i,j dt3

(1)

where Qij = ̺(xi xj − 31 δij r2 )d3 x is the mass quadrupole moment. The radiation power thus scales with the sixth power of the frequency of the system. Due to the weakness of gravity, expressed in the factor 5cG5 , only systems of astrophysical dimensions – large masses moving at a significant fraction of the speed of light – generate significant amounts of gravitational radiation. Since black holes are the most compact objects we can consider, black-hole binaries are a particularly eﬃcient and important source for current and future gravitational wave detectors. Gravitational waves have so far been detected only indirectly by comparing the tightening of the orbit of the Hulse-Taylor binary pulsar with the theoretical prediction from energy loss due to gravitational wave emission [41, 54]. However, a growing network of gravitational-wave detectors such as LIGO [1, 43], GEO [29, 33] and VIRGO [56] is currently taking science data and direct detection is forseen for the next few years. In order to actually extract information about the sources from observations, accurate signal templates are needed for various types of sources. Producing templates for gravitational-wave data analysis will require large parameter studies, and correspondingly large computational resources: The eventual goal of our simulations is to map the parameter space of gravitational wave signals from black hole coalescence. Our simulations typically start with initial data that correspond to the astrophysically most relevant case of quasicircular inspiral, which essentially means that the eccentricity is very small. The parameter space is then essentially given by the mass ratio and individual spins, as well as the initial orientation of the spins. The latter determines in particular the spin orientation at merger time, which has a strong influence on the gravitational wave signal. In order to produce “complete” waveforms, which contain large numbers of gravitational wave cycles from the inspiral phase, as well as the merger and ringdown phases, it is necessary to start the numerical simulations in the regime where Post-Newtonian analytical calculations are valid. These describe very accurately the waveforms of the early inspiral process, but break down for small separations of the black holes. This “matching” of analytical and numerical results requires large initial blackhole separations and large integration times. Improving current estimates for the validity of the Post-Newtonian approximation is an integral part of the research, and will eventually determine the computational cost. Crucial technical problems in the field at this point relate to the eﬃciency of the simulations, and to the establishment of a “data analysis pipeline”, connecting analytical calculations of the early inspiral phase with numerical simulations and gravitational-wave searches in actual detector data. Since our last report, we have obtained a large number of important results:

Black Hole Coalescence

•

• •

• •

•

•

21

Through careful analysis of simulations of a single black hole, we described for the first time the geometrical behavior of numerical solutions in the now-standard “moving puncture” method of evolving black-hole spacetimes [38, 39]. We have presented the most detailed study of the two proposed versions of the moving-puncture method as applied to equal-mass binaries [21]. All moving-puncture simulations to date have used a particular type of initial data, Bowen-York puncture data [19, 20]. These data make the physically unrealistic assumption of conformal flatness. We have made significant progress in removing this assumption for binaries made up of spinning black holes [37]. An independent code, LEAN, has been developed, and used to perform a comparison of head-on collision simulations using puncture and excision initial data [53]. Astrophysicists have for many years been waiting for numerical simulations to accurately calculate the recoil when two black holes of unequal mass collide. In the largest parameter study to date of black-hole binaries requiring approximately 150000 CPU hours, we have calculated the maximum recoil of unequal-mass nonspinning binaries to be 175.2 ± 11 km/s [35], and performed a detailed analysis of the gravitational waves emitted during unequal-mass binary mergers [17]. Far more spectacularly, we have found that extremely large recoils are possible for spinning black holes, and estimated the kick from one configuration as 2500 km/s, large enough to eject the remnant black hole even from a giant elliptical galaxy [34]. We have begun work on establishing a data-analysis pipeline to provide our numerical waveforms to the gravitational-wave data-analysis community [2, 3]. This work may increase the chances of gravitational-wave detection by at least a factor of ten.

Further research in all of these areas is still underway; there are a wealth of areas of explore. In this paper we will provide further details of these results, and present performance tuning results and a summary of code optimization over the last twelve months.

2 Status of the Field The numerical solution of the full Einstein equations represents a very complex problem, and for two black holes the spacetime singularities that are encountered in the interior of black holes pose an additional challenge. In order to obtain accurate results both the use of mesh refinement techniques and a good choice of coordinate gauge are essential. Together with the complicated structure of the equations – a typical code has between ten and several dozen evolution variables, and, when expanded, the right hand sides of the

22

B. Br¨ ugmann et al.

evolution equations have thousands of terms – this yields a computationally very complex and mathematically very subtle problem. For a long time, typical runs had been severely limited by the achievable evolution time before the simulations become too inaccurate or before the computer code became unstable, and there were serious doubts whether numerical relativity techniques could produce gravitational-wave templates, at least in the near future. This picture has drastically changed ever since the first simulations of a complete black hole orbit were obtained in early 2004 [24]. In spring of 2005 Pretorius [49] presented the first simulation lasting for several orbits, using adaptive mesh refinement, second-order finite diﬀerencing, a sophisticated method to excise the singular interior of the black hole from the grid, and an implicit evolution algorithm. An alternative to the “excision” method of treating black holes is to “fill” the black hole with a topological defect in the form of an interior spacelike asymptotic end, the “puncture” [5, 20], and to freeze the evolution of the asymptotic region through a judicious choice of coordinate gauge [6, 7, 10, 22]. The latter approach, combined with a setup where the topological defect is allowed to move across the grid (“moving puncture” approach [11, 28]) has lead to a giant leap forward in the field [12, 13, 26, 27, 39, 40, 53], taking the first orbit simulations of black holes [24, 49] to more than ten orbits and allowing accurate wave extraction. It is this approach of moving punctures that we and most groups follow. There are now ten groups capable of performing black hole binary evolutions, with Jena being one of five groups leading the field. Recent work in the field has shifted focus from methodological work to studies of astrophysical relevance. A particular focus of the last few months has been the on the so-called recoil or rocket eﬀect due to “beamed” emission of gravitational radiation [15, 18, 48]. By momentum conservation, radiation of energy in a preferred direction corresponds to a loss of linear momentum and the black hole that results from the merger thus recoils from the centerof-mass frame with speeds of up to several thousand km/s. The velocity of this “kick” depends on the configuration of the system (e.g. the mass ratios and spins) and details of the merger dynamics, but not on the total mass (velocity is dimensionless in geometric units). From an astrophysical point of view, the recoil eﬀect is particularly interesting for massive black holes with masses > 105 M⊙ , which exist at the center of many galaxies and have a substantial impact on the structure and formation of their host galaxies, as is demonstrated by the correlation of the black-hole mass with the bulge mass, luminosity and velocity dispersion [32, 45, 46, 47]. The largest recoil eﬀects have so far been found [34] for a particularly simple configuration: equal mass black holes with (initially) anti-aligned spins in the orbital plane. Such large kicks are on the order of 1% of the speed of light, and larger than the escape velocity of about 2000 km/s of giant elliptical galaxies. In order to predict accurate gravitational wave signals, it is necessary to perform simulations over many orbits and connect such numerical simula-

Black Hole Coalescence

23

tions to post-Newtonian analytical calculations. It turns out that in such long simulations phase errors of the orbits are very hard to control, and accurate simulations with second or fourth order accurate methods are prohibitively expensive. As an example for long evolutions we show results from three simulations with initial coordinate separations D = 10, 12, 14, corresponding to somewhat more than 5, 9 and 14 orbits. Orbital tracks are shown in Fig. 1, and gravitational wave signals in Fig. 2. The signals have been shifted in time to reach the maximum at t = 0, which in particular compensates for the large numerical errors in phase. In order to overcome phase inaccuracies in long

Fig. 1. Coordinate tracks of the puncture locations for inspirals with initial coordinate separations D = 10, 12, 14, corresponding to a bit more than 5, 9 and 14 orbits

Fig. 2. Gravitational wave signals from the simulations shown in Fig. 1, shifted in time to reach the maximum at t = 0

24

B. Br¨ ugmann et al.

evolutions, spectral methods have been suggested and significant progress has been made by the Caltech-Cornell group [50]. In our group we have recently obtained excellent results with 6th order accurate finite diﬀerencing methods as described below. We are currently in the process of fine-tuning our grid setup for 6th order evolutions for diﬀerent types of initial data (adjustments are required for black holes with spins or unequal mass evolutions).

3 Description of the Method 3.1 Continuum Problem Writing the Einstein equations as an initial-value problem yields a coupled system of second diﬀerential order elliptic constraints and hyperbolic evolution equations that preserve the constraints. In the free evolution approach, which is most common in the field, the constraints are only solved initially, and later only the hyperbolic equations are used to construct the solution. There is a large freedom in writing the Einstein equations as a system of partial diﬀerential equations, and much research has gone into finding optimal choices. We currently employ the most popular choice, the so-called BSSN system [4, 7, 14, 36, 52]. 3.2 Numerical Methods for the Evolution Problem Discretization in space is performed with standard fourth-order accurate stencils, although we are currently working on a transition to making sixth order accurate methods our standard choice. Symmetric stencils are used with the exception of the advection terms associated with the shift vector, where we use lop-sided upwind stencils, see e.g. [57] for the fourth order case. For the sixth order case we find that several choices for the advection term stencils yield stable evolutions, but the lop-sided upwind stencil which is closest to the symmetric case yields by far the best accuracy, i.e. we use 2f (x − 2h) − 24f (x − h) − 35f (x) + 80f (x + h) 60h 1 d7 f (x) 6 −30f (x + 2h) + 8f (x + 3h) − f (x + 4h) − + h . 60h 105 dx7 Alternative asymmetric choices would be f ′ (x) =

−10f (x − h) − 77f (x) + 150f (x + h) − 100f (x + 2h) 60h 1 d7 f (x) 6 50f (x + 3h) − 15f (x + 4h) + 2f (x + 5h) + + h , 60h 42 dx7 −147f (x) + 360f (x + h) − 450f (x + 2h) + 400f (x + 3h) f ′ (x) = 60h −225f (x + 4h) + 72f (x + 5h) − 10f (x + 6h) 1 d7 f (x) 6 − + h 60h 7 dx7

f ′ (x) =

Black Hole Coalescence

25

for the stencils that deviate more from the symmetric choice. We can see that the first choice has the smallest leading error term. The symmetric stencil 7 1 d f (x) 6 has an even smaller error term 140 dx7 h , but does not show equally robust results, as is common for solving advection equations. For non-advection derivative terms we again use the standard symmetric stencil. Time integration is performed by standard Runge-Kutta type methods, in particular 3rd and 4th order Runge-Kutta and second-order accurate three-step iterative Crank-Nicholson integrators as described in [25], where Courant limits and stability properties are discussed for the types of equations used here. Mesh refinement techniques are utilized to resolve the diﬀerent scales of the problem, and to follow the motion of the black holes. The relevant spatial scales of the binary black-hole problem are the scales of the holes, their orbital motion, the typical wave lengths of the ring-down of the individual and merged black holes, the typical wavelength of the merger waveform and the asymptotic falloﬀ of the fields. All of these scales can be estimated from the initial data and vary relatively slowly with time. It is thus very eﬃcient to essentially use a fixed mesh refinement strategy, with inner level refinement boxes following the motion of the black holes. Typically we use about 10 refinement levels (refining the grid spacing by factors of 2), roughly half of which follow the movement of the black holes. The numerical algorithm is a modification of the standard Berger-Oliger algorithm [16] as described in [21]. 3.3 Numerical Methods for Solving the Constraints Since the Einstein equations constitute a constrained system, initial data for the evolution equations has to satisfy the constraints. The binary black hole initial data used in our simulations is so-called puncture initial data, which requires the solution of one scalar non-linear elliptic equation [20]. In the BAM code such data can be obtained with a parallel geometric multigrid solver, which however is using second order finite diﬀerencing. During the last funding period we have implemented an interface to the special purpose pseudo-spectral collocation code described in [9] for the BAM code. Such data was already available to LEAN as a Cactus module. In BAM, the AMR data is obtained by barycentric interpolation, typically with 8th order polynomials for both the 4th and 6th order finite diﬀerencing methods. The eﬃciency of the spectral solver is suﬃcient to solve the initial data problem on a single processor, which is what we currently do. 3.4 Code Structure We have implemented our methods in two independent codes, which we use for cross-checking of results and performance, in order to achieve reliable results from accurate and eﬃcient simulations. The LEAN code is based on the Cactus infrastructure [8, 55], which is a community code and thus not fully under our control. The BAM [22, 24]

26

B. Br¨ ugmann et al.

code has been developed by our group and is designed to solve partial diﬀerential equations on structured meshes, in particular a coupled system of (typically hyperbolic) evolution equations and elliptic equations. The complexity of the equations is addressed by using a Mathematica package integrated into the code, which produces C-code from high-level problem descriptions in Mathematica notebooks. The BAM code is organized as a “framework”, similar in spirit to the Cactus code [8, 55], but dropping much of its complexity. The computational domain is decomposed into cubes, following standard domain-decomposition algorithms, and is parallelized with MPI. Our mesh refinement algorithm is based on the standard Berger-Oliger algorithm, but with additional buﬀer zones, along the lines of [42, 51]. 3.5 Computational Cost Since adaptive meshes are being used, the memory requirement is not easily expressed in terms of Cartesian box sizes. A typical large run has a cubical box as its coarsest level with about 1503 grid points. Typical refinements use 10 levels with 1 or 2 boxes per level (depending on whether the black holes fit into one box or require two individual boxes). Depending on the refinement criterion and in particular when two boxes are merged into one larger box, the memory requirement is not constant and may exceed 1503 by up to a factor of 4. About 100–160 grid functions are required in double precision. Extrapolating the memory requirement from the smaller runs that we performed so far, we estimate that not more than 200 GByte of RAM are needed. For 4 GByte of RAM per core on the LRZ Altix system, memory usage is therefore not a limiting factor. However, production runs typically require on the order of 105 time steps on the finest level, and hence the number of floating point operations required per grid point per time step is a more stringent issue than memory. The aim is to perform simulations of black hole binaries at large separation, and the larger the separation, the longer it takes the black holes to complete one orbit. Typical large runs take from about 3 days to up to a week using checkpointing, not counting the time spent in re-queuing simulations. Each such run results in one data point in the parameter space of black hole binaries. The number of data points in the space of initial data needed to provide useful information for gravitational wave data analysis is very much an open question right now, which we hope to address with our research.

4 Status Report of Optimization Work 4.1 General Remarks Our BAM and LEAN codes have so far been run on Linux clusters based on AMD Opteron, and Intel architectures, and we find rather consistent perfor-

Black Hole Coalescence

27

mance results for the systems we use: our in-house clusters doppler (24 AMD Opteron nodes with Infiniband network) and kepler (35 Intel Woodcrest nodes with Infiniband network), the HLRS cluster strider and the Itanium-based Altix cluster at LRZ. In summer 2005 the Jena group applied for and was granted support by the LRZ in its porting and optimization initiative for the new Altix system. We find that optimizations performed on the Altix system carry over to our in-house clusters and strider at HLRS. Apart from standard timing tests we have used various tools: VampirTrace has been used successfully to identify communication overheads, in particular in analysis routines that compute quantities like the gravitational wave content in our simulations. For large numbers of processors, these routines have become the performance bottleneck. These calculations require computing integrals over spheres, where the collocation points are not aligned with the grid used for evolving the configuration, consequently the required interpolation operations are rather expensive in communication. Recent optimizations have removed this bottleneck, but systematic scaling results for larger number of processors are not yet available due to end of our allocation at HLRS and the downtime of HLRB2 at LRZ caused by a system-upgrade. Various of our analysis tools require interpolation of tensor fields from Cartesian grids onto parameterized surfaces, say for wave extraction and apparent horizon finding. It was found that the scaling of the wave extraction routine in BAM was poor due to the interpolator, and this part of the code took tens of percent for typical production runs on 32 and more processors. We optimized the MPI implementation of BAM’s point-wise interpolator so that wave extraction dropped to less than one percent of the overall run time. Significant gains in eﬃciency have been obtained by replacing fourth order finite diﬀerencing by sixth order finite diﬀerencing as summarized in Sec. 3. This result is nontrivial for our application for several reasons: One of the problems one faces is that higher order methods generally require a larger number of buﬀer zones for our adaptive mesh refinement algorithm, and some experiments are required to to test diﬀerent choices in practice. Furthermore, the moving puncture method deals with non-smooth features inside of the black hole horizons, and the AEI and Brownsville groups have previously reported failures to obtain stable and accurate moving puncture evolutions with sixth order methods [30, 44]. An important issue that we could identify, and which is an equally serious problem in alternative numerical relativity codes against which we benchmarked, is the overflow of registers due to the complexity of the Einstein equations. We have started to analyze the memory access pattern of the equations, and were able to perform some promising experiments, but significant further work will be required to rewrite the equations in an optimal way.

28

B. Br¨ ugmann et al.

4.2 Memory Optimizations After development of the new moving puncture methods and wave extraction tools had been completed in BAM, we optimized the memory usage of the current production code. After analyzing the use of temporary memory, it was possible to reduce the peak memory footprint counted in 3d AMR grid variables from about 280 variables to 175 variables. Part of this improvement was due to a more memory eﬃcient implementation of the Runge-Kutta time evolution scheme. 4.3 Scaling Our most extensive scaling tests have been performed with the second order accurate version of our code, still with non-moving boxes of refinement. In such cases we typically reach 60–90% of scaling on up to 128 processors, which has been tested on diﬀerent machines. With the new fourth-order code we have so far only tested scaling on up to 32 processors. We get 77% scaling on the Cray Opteron cluster Strider going from 4 to 32 processors, and increasing the problem size by a factor close to 8. The 32-processor run corresponds to the initial 10M of runtime of a typical production run with 14.4 GByte memory usage (i.e. approximately 450 MByte per processor). Already our initial scaling results on the LRZ Altix system have been very promising. In Fig. 3 we show results for the performance of the original BAM code (second order code with fixed mesh refinement). As reference point we compare to a SUN cluster (LION-XO at Penn State University) with 80 dual Opteron SunFire v20z nodes, 2.4 GHz processors, 4 GB RAM per processor, with Infiniband network, and using the gcc compiler. The timings on the first Altix test system were obtained prior to any optimizations. The test system had 64 Itanium processors at 1.6 GHz and 4 GB RAM per CPU. The code reached 4 to 8% of peak performance or 260 to 530 MFlops on the Altix. These numbers have been improved upon in subsequent optimizations, but since the new numbers are not final yet, and also since there has been a major upgrade to the BAM code very recently, we do not want to quote premature benchmarks. However, note that even these early results show very promising network performance, see the lower panel in Fig. 3. The run labelled 5 × 80 requires 3.7 GB memory, run 5 × 128 requires 15 GB memory. For these problem sizes the Altix system clearly scales better than the Opteron/Infiniband cluster. This is important since the new moving-box/AMR algorithm of BAM and LEAN requires even more communication than the previous version of the code reported on here. We do not have recent scaling results for strider, since our grant has expired – we are currently in the process of preparing a grant application for the NEC vector machines at HLRS.

Black Hole Coalescence

29

Fig. 3. Benchmarking the BAM code on the initial LRZ Altix system and an Opteron/Inﬁniband cluster, LION-XO. The upper panel shows the run times in seconds for diﬀerent problem sizes, while the lower panel shows performance relative to perfect scaling

5 Conclusions and Plans for Future Work We have presented a summary of recent work of the Jena numerical relativity group on simulating the coalescence of black hole binaries in full general relativity. In the last year we have obtained astrophysically relevant results: In the largest numerical parameter study to date of black-hole binaries, we have calculated the recoil of nonspinning binaries for mass ratios from 1:1 to 1:4 [35], and performed a detailed analysis of the gravitational waves emitted during unequal-mass binary mergers [17]. We have also found that extremely

30

B. Br¨ ugmann et al.

large recoils are possible for spinning black holes, and estimated the kick from one configuration as 2500 km/s, large enough to eject the remnant black hole even from a giant elliptical galaxy [34]. We have also made significant progress with developing our methods and further optimizing our code. Large parameter studies forseen for the near future include the case of large mass ratios on the order of 1:10, and general spin configurations. While our initial HLRS allocation for strider has recently expired, we are currently in the process of porting our code the the NEC vector processor architecture, and prepare a grant application for the NEC SX-8.

References 1. A. A. Abramovici, W. Althouse, R. P. Drever, Y. Gursel, S. Kawamura, F. Raab, D. Shoemaker, L. Sievers, R. Spero, K. S. Thorne, R. Vogt, R. Weiss, S. Whitcomb, and M. Zuker. Ligo: The laser interferometer gravitational-wave observatory. Science, 256:325–333, 1992. 2. P. Ajith and et. al. in preparation. 3. P. Ajith and et. al. Phenomenological template family for black hole coalescence waveforms. In Proceedings of the GWDAW11 conference, Potsdam, 18–21 December 2006. in preparation. 4. Miguel Alcubierre, Gabrielle Allen, Bernd Br¨ ugmann, Edward Seidel, and WaiMo Suen. Towards an understanding of the stability properties of the 3+1 evolution equations in general relativity. Phys. Rev. D, 62:124011, 2000. 5. Miguel Alcubierre, W. Benger, B. Br¨ ugmann, G. Lanfermann, L. Nerger, E. Seidel, and R. Takahashi. 3D Grazing Collision of Two Black Holes. Phys. Rev. Lett., 87:271103, 2001. 6. Miguel Alcubierre and Bernd Br¨ ugmann. Simple excision of a black hole in 3+1 numerical relativity. Phys. Rev. D, 63:104006, 2001. 7. Miguel Alcubierre, Bernd Br¨ ugmann, Peter Diener, Michael Koppitz, Denis Pollney, Edward Seidel, and Ryoji Takahashi. Gauge conditions for long-term numerical black hole evolutions without excision. Phys. Rev. D, 67:084023, 2003. 8. G. Allen, T. Goodale, J. Mass´ o, and E. Seidel. The cactus computational toolkit and using distributed computing to collide neutron stars. In Proceedings of Eighth IEEE International Symposium on High Performance Distributed Computing, HPDC-8, Redondo Beach, 1999. IEEE Press, 1999. 9. Marcus Ansorg, Bernd Br¨ ugmann, and Wolfgang Tichy. A single-domain spectral method for black hole puncture data. Phys. Rev. D, 70:064011, 2004. 10. John Baker, Bernd Br¨ ugmann, Manuela Campanelli, Carlos O. Lousto, and Ryoji Takahashi. Plunge waveforms from inspiralling binary black holes. Phys. Rev. Lett., 87:121103, 2001. 11. John G. Baker, Joan Centrella, Dae-Il Choi, Michael Koppitz, and James van Meter. Gravitational wave extraction from an inspiraling conﬁguration of merging black holes. 2005. 12. John G. Baker, Joan Centrella, Dae-Il Choi, Michael Koppitz, and James van Meter. Binary black hole merger dynamics and waveforms. 2006. Unpublished, gr-qc/0602026.

Black Hole Coalescence

31

13. John G. Baker, Joan Centrella, Dae-Il Choi, Michael Koppitz, James van Meter, and M. Coleman Miller. Getting a kick out of numerical relativity. 2006. Unpublished, astro-ph/0603204. 14. Thomas W. Baumgarte and Stuart L. Shapiro. On the numerical integration of Einstein’s ﬁeld equations. Phys. Rev. D, 59:024007, 1999. 15. J. D. Bekenstein. Gravitational-Radiation Recoil and Runaway Black Holes. Astrophys. J., 183:657–664, 1973. 16. Marsha J. Berger and Joseph Oliger. Adaptive mesh reﬁnement for hyperbolic partial diﬀerential equations. J. Comput. Phys., 53:484–512, 1984. 17. Emanuele Berti et al. Inspiral, merger and ringdown of unequal mass black hole binaries: A multipolar analysis. 2007. 18. W. B. Bonnor and M. A. Rotenberg. Transport of momentum by gravitational waves: the linear approximation. Proc. R. Soc. Lond. A., 265:109–116, 1961. 19. Jeﬀrey M. Bowen and James W. York. Time-asymmetric initial data for black holes and black hole collisions. Phys. Rev. D, 21(8):2047–2056, 1980. 20. S. Brandt and B. Br¨ ugmann. A simple construction of initial data for multiple black holes. Phys. Rev. Lett., 78(19):3606–3609, 1997. 21. Bernd Bruegmann et al. Calibration of moving puncture simulations. 2006. 22. B. Br¨ ugmann. Binary black hole mergers in 3D numerical relativity. Int. J. Mod. Phys. D, 8:85, 1999. 23. B. Br¨ ugmann, G. Gonzalez, M. Hannam, S. Husa, P. Marronetti, U. Sperhake, and W. Tichy. Gravitational wave signals from simulations of black hole dynamics. In W.E. Nagel, W. J¨ ager, and M. Resch, editors, High Performance Computing in Science and Engineering ’06, pages 3–17. Springer, 2006. 24. Bernd Br¨ ugmann, Wolfgang Tichy, and Nina Jansen. Numerical simulation of orbiting black holes. Phys. Rev. Lett., 92:211101, 2004. 25. Gioel Calabrese, Ian Hinder, and Sascha Husa. Numerical stability for ﬁnite diﬀerence approximations of Einstein’s equations. 2005. 26. M. Campanelli, C. O. Lousto, and Y. Zlochower. Gravitational radiation from spinning-black-hole binaries: The orbital hang up. 2006. 27. Manuela Campanelli, C. O. Lousto, and Y. Zlochower. The last orbit of binary black holes. Phys. Rev. D, 73:061501(R), 2006. 28. Manuela Campanelli, Carlos O. Lousto, Pedro Marronetti, and Yosef Zlochower. Accurate evolutions of orbiting black-hole binaries without excision. Phys. Rev. Letter, 96:111101, 2006. 29. K. Danzmann. The geo project: a long baseline laser interferometer for the detection of gravitational waves. Lecture Notes in Physics, 410:184–209, 1992. 30. Peter Diener. private communication, 2007. ´ ´ Flanagan and Scott A. Hughes. Measuring gravitational waves from 31. Eanna E. binary black hole coalescence: I. signal to noise for inspiral, merger, and ringdown. Phys. Rev. D, 57:4535, 1998. 32. K. Gebhardt, R. Bender, G. Bower, A. Dressler, S. M. Faber, A. V. Filippenko, R. Green, C. Grillmair, L. C. Ho, J. Kormendy, T. R. Lauer, J. Magorrian, J. Pinkney, and S. Richstone, D. Tremaine. A Relationship between Nuclear Black Hole Mass and Galaxy Velocity Dispersion. Astrophys. J, 539:L13–L19, 2000. astro-ph/0006289. 33. GEO600 - http://www.geo600.uni-hannover.de/. 34. J. A. Gonzalez, M. D. Hannam, U. Sperhake, B. Brugmann, and S. Husa. Supermassive kicks for spinning black holes. 2007.

32

B. Br¨ ugmann et al.

35. Jose A. Gonzalez, Ulrich Sperhake, Bernd Bruegmann, Mark Hannam, and Sascha Husa. Total recoil: the maximum kick from nonspinning black-hole binary inspiral. Phys. Rev. Lett., 98:091101, 2007. 36. Carsten Gundlach and Jose M. Martin-Garcia. Hyperbolicity of second-order in space systems of evolution equations. 2005. 37. Mark Hannam, Sascha Husa, Bernd Bruegmann, Jose A. Gonzalez, and Ulrich Sperhake. Beyond the bowen-york extrinsic curvature for spinning black holes. 2006. ´ Murchadha, Bernd Br¨ 38. Mark Hannam, Sascha Husa, Niall O ugmann, Jos´e A. Gonz´ alez, and Ulrich Sperhake. Where do moving punctures go? Journal of Physics: Conference series, 2007. in press. ´ Mur39. Mark Hannam, Sascha Husa, Denis Pollney, Bernd Brugmann, and Niall O chadha. Geometry and regularity of moving punctures. 2006. 40. Frank Herrmann, Deirdre Shoemaker, and Pablo Laguna. Unequal-mass binary black hole inspirals. 2006. 41. R.N. Hulse and J.H. Taylor. Astrophys. J., 195:L51–L53, 1975. 42. Luis Lehner, Steven L. Liebling, and Oscar Reula. Amr, stability and higher accuracy. Class. Quant. Grav., 23:S421–S446, 2006. 43. LIGO - http://www.ligo.caltech.edu/. 44. Carlos Lousto. private communication, 2006. 45. J. Magorrian, S. Tremaine, D. Richstone, R. Bender, G. Bower, A. Dressler, S. M. Faber, K. Gebhardt, R. Green, C. Grillmair, J. Kormendy, and T. Lauer. The Demography of Massive Dark Objects in Galaxy Centers. Astron. J, 115:2285–2305, 1998. astro-ph/9708072. 46. R. J. McLure and J. S. Dunlop. On the black hole-bulge mass relation in active ind inactive galaxies. MNRAS, 331:795–804, 2002. astro-ph/0108417. 47. D. Merritt and L. Ferrarese. Black hole demographics from the m• -σ relation. MNRAS, 320:L30–L34, 2001. astro-ph/0009076. 48. A. Peres. Classical Radiation Recoil. Phys. Rev., 128:2471–2475, 1962. 49. Frans Pretorius. Evolution of binary black hole spacetimes. Phys. Rev. Lett., 95:121101, 2005. 50. Mark A. Scheel, Harald P. Pfeiﬀer, Lee Lindblom, Lawrence E. Kidder, Oliver Rinne, and Saul A. Teukolsky. Solving Einstein’s equations with dual coordinate frames. Phys. Rev. D, 74:104006, 2006. 51. Erik Schnetter, Scott H. Hawley, and Ian Hawke. Evolutions in 3D numerical relativity using ﬁxed mesh reﬁnement. Class. Quantum Grav., 21(6):1465–1488, 21 March 2004. 52. Masaru Shibata and Takashi Nakamura. Evolution of three-dimensional gravitational waves: Harmonic slicing case. Phys. Rev. D, 52:5428, 1995. 53. Ulrich Sperhake. Binary black-hole evolutions of excision and puncture data. 2006. 54. J. H. Taylor and J. M. Weisberg. A new test of general relativity: Gravitational radiation and the binary pulsar PSR 1913+16. Astrophys. J., 253:908–920, 1982. 55. Cactus Computational Toolkit. http://www.cactuscode.org. 56. VIRGO - http://www.virgo.infn.it/. 57. Y. Zlochower, J. G. Baker, M. Campanelli, and C. O. Lousto. Accurate black hole evolutions by fourth-order numerical relativity. Phys. Rev. D, 72:024021, 2005.

Massless Four-Loop Integrals and the Total Cross Section in e+ e− Annihilation J. H. K¨ uhn, M. Steinhauser, and M. Tentyukov Institut f¨ ur Theoretische Teilchenphysik, Universit¨at Karlsruhe, 76128 Karlsruhe, Germany

This is the report for the project ParFORM for the period June 2006 to June 2007.

1 Aim of the Project The main purpose of particle physics is the explanation of the fundamental mechanism for the interaction of the elementary particles. On the experimental side the investigations take mainly place at the big accelerators at CERN (Geneva) or FERMILAB (Chicago). On the other hand it it essential to develop theoretical models which describe the fundamental interactions and which, of course, have to be confronted with the experiment. Currently there is a well-established theory, the so-called Standard Model, which has been verified by experimental studies to a very high precision. However, there are strong hints that the Standard Model is not the final theory describing the fundamental interaction of the elementary particles. E.g., it can not describe the large amount dark matter present in the universe and it can not explain the observed huge excess of matter over anti-matter. There are basically two routes which explore deviations from the Standard Model and which provide hints for so-called “Grand Unified Theories” (GUTs): one either performs experiments where the colliding particles have very high energy or one extracts physical observables to very high precision at lower energies and confronts them with precise theoretical calculations. Both ways are currently followed. The main emphasis of this project is centered on the second option. In particular, the corresponding calculations are important in the precise determination of the fundamental parameters of the Standard Model like coupling constants or quark masses. The basic tool in order to perform precise calculations is perturbation theory which is based on the expansion in a small parameter, in our case the coupling constants between the various particles. In particle physics perturbation theory is organized in a very intuitive way: all mathemematical expressions

34

J.H. K¨ uhn, M. Steinhauser, M. Tentyukov

which have to be computed can be visualized in terms of so-called Feynman diagrams where each particle is represented by a line and interactions between particles by vertices. The main diﬃculty in practical applications is the occurrence of closed loops at higher order in perturbation theory. This project deals at the forefront of what is currently possible and considers Feynman diagrams up to five loops. (Examples will be given below.) The basic object which is considered in this project is the photon two-point function, Π(q 2 ), which is related to the correlator of two vector currents, j μ , through1

2 2 (1) −q gμν + qμ qν Π(q ) = i dx eiqx 0|T jμ (x)jν† (0)|0 .

Some sample Feynman diagrams up to five-loop order are shown in Fig. 1. The external current j μ is represented by a blob, the quark lines by the straight lines with an arrow and the gluons by the curly lines. Currently a closed analytic calculation taking into account the masses of the quarks, mq , is only possible up to two loops. At three-loop order a semi-numerical method has been developed based on expansions in m2q /q 2 and q 2 /m2q which leads to accurate approximations of the three-loop result for Π(q 2 ). The four-loop result for the massive and the five-loop one for the massless correlator are under investigation in this project.

Fig. 1. Sample diagrams contributing to Π(q 2 )

The imaginary part of the quantity Π(q 2 ) is directly related to the physical cross section, σ, for the production of hadrons in e+ e− annihilation, respectively, its normalized version leads to the famous R-ratio given by R(s) =

σ(e+ e− → hadrons) = 12π Im Π(q 2 = s + iǫ) , + − + − σ(e e → μ μ )

(2)

√ where s is the considered center-of-mass energy. R(s) is one of the most important and clean places for precise tests of Quantum Chromodynamics (QCD) and the Standard Model. This quantity together with the related semileptonic τ lepton decay rate provide us with invaluable information about 1

For the notation commonly used in high-energy physics we refer to the standard textbooks like, e.g., [1].

Massive and Massless Four-Loop Integrals

35

the numerical value of the strong coupling constant αs as well as its running from the τ lepton mass to the one of the Z boson. There is also a significant amount of purely theoretical interest to higher order contributions to this quantity related, e.g., to renormalons. Diagrammatically the imaginary part is represented by those Feynman diagrams which are obtained by cutting (i.e. separating the left and the right blob by chopping individual lines) the diagrams of Fig. 1 in all possible ways. The main purpose is the computation of quantum corrections induced by the strong interaction to R(s) which can be cast in the form R(s) =

αs i

i≥0

π

δR(i) + quark mass eﬀects .

(3)

For massless quarks the corrections up to i = 2 are known since more than 30 years and δR(3) has been computed beginning of the nineties (for comprehensive reviews, see [2, 3]). Currently, the term δR(4) is only partially known and considered in the current project. It originates from the imaginary part of five-loop diagrams for Π(q 2 ). This fact is used for the practical calculations. Actually, a sophisticated method has been developed which allows to reduce the computation of the imaginary part of five-loop diagrams to real parts of four-loop diagrams. Thus, in order to get δR(4) for massless quarks one has to consider four-loop integrals, which are actually of the same type as the ones needed for the calculation of the four-loop approximation of Π(q 2 ) (in the massless case). There are a variety of important physical applications of Π(q 2 ) and its imaginary part R(s). E.g., the quantity R(s) can be used for the determination of αs by confronting (3) with √ experimental results. In this context it is particularly promising to consider s = MZ , where MZ is the mass of the Z boson, since in the past very precise data has been collected at the CERN Large Electron Positron (LEP) Collider. Furthermore, it is planned that an International Linear Collider could significantly improve the experimental data. Thus, αs could be extracted with an uncertainty below 1% – once δR(4) is available. The technique developed for the quantity R(s) can immediately be applied to the decay rate of the τ lepton, which can be used for the determination of the strange quark mass [4]. Furthermore, if one replaces in the above discussion the vector by a scalar current, there are applications in connection to the Higgs boson. More details will be presented below. From the above discussion it becomes clear that the main aim of this project is the computation of massless propagator-type diagrams to four-loop order within perturbative quantum field theory. The most important software tools necessary for our calculations are described in the next Section and the physical applications are discussed in Sects. 3 and 4.

36

J.H. K¨ uhn, M. Steinhauser, M. Tentyukov

2 Parallel Computer Algebra The workhorse for most of the computations performed in this project is the computer algebra program ParFORM [5]. ParFORM is the parallel version of the Computer Algebra System FORM [6] which – in contrast to Mathematica or Maple – is designed for the manipulation of huge expressions ranging up to several tera byte. The latter is very crucial for for multi-loop calculations. Thus it is essential to have a fast access to the hard disk where the temporary expressions are stored. There are essentially two implementations of ParFORM: one is based on MPI (“Message Passing Interface”) which is quite good for systems that have processors with separated memory, and the other one is specially adopted to Symmetric-Multi-Processor (SMP) architectures with shared memory implemented using the NUMA (“Non-Uniform-Memory-Access”) technology. The comparison of the speed-up curves can be seen in Fig. 2. In the recent years ParFORM has been used on many diﬀerent platforms ranging from SMP machines with up to 32 processors to simple PC-clusters connected by a Gigabit ethernet. In Fig. 3 the results are summarized and compared with the speed-up obtained at the XC6000 cluster. It should be stressed that, although the MPI version of ParFORM has to be used on the XC6000 cluster the same speed-up as for the faster SMP version (cf. Fig. 2) has been obtained. The run-time of our problems varies from a few days or weeks up to about two months. Due to the very structure of FORM and due to the organization of the calculation it is not possible to set check points. Since the maximum CPU time at the Landesh¨ ochstleistungsrechner is limited to about seven days it is only possible to submit small and medium-sized jobs. In Fig. 4 the performance of ParFORM is shown for a typical job where up to 60 processors have been used. For the XC6000 a good scaling behaviour is observed up to about 16 processors. Above approximately 24 processors the saturation region starts and only a marginal gain is observed once 60 processors are employed. The situation is much worse for the new Lan-

Fig. 2. Speed-up as a function of the used processors. The full (red ) curve is based on the MPI and the dotted (blue) curve on the shared-memory (SM) version of ParFORM. The data point have been obtained on a 32-processor Itanium computer

Massive and Massless Four-Loop Integrals

37

Fig. 3. Speed-up curves for various computer clusters with diﬀerent interconnections

Fig. 4. CPU-time and speedup curve for a typical job on the XC6000 and XC4000

desh¨ ochstleistungsrechner (XC4000). Beyond about ten processors the system is very unstable and is thus less attractive for our applications. It seems that the interconnection of the individual nodes is much worse than for the XC6000 cluster – at least for our applications. Very recently a diﬀerent concept for the parallelization of FORM has been developed. The basic idea is the use of POSIX2 threads in order to realize the communication between the various processors of a shared memory machine. The main application is thus centered around multi-core machines with two, four or eight cores. First tests of TFORM [7] were quite successful and comparable speed-up to ParFORM could be achieved. In future we will continue to further development of TFORM. Due to the hardware structure of the Landesh¨ ochstleistungsrechner this is very promising. It is particularly tempting to combine TFORM and ParFORM in order to reach an optimal speed-up. A further program, Crusher, which has been developed in our group, implements the so-called Laporta-algorithm [8, 9]. In high-energy physics the Laporta-algorithm is a widespread tool which is used to reduce the huge num2

“a Portable Operating System Interface for uniX”

38

J.H. K¨ uhn, M. Steinhauser, M. Tentyukov Fig. 5. Speed-up curve for the program TCrusher. The solid (red ) and dotted (blue) curves corresponds to computers with two and four dual core opteron CPUs, respectively

ber of Feynman integrals occuring in the calculation of physical quantity to a small set of basis integrals. The reduction is essentially based on a particular implementation of the Gauss elimination algorithm which is applied to a system of equation obtained from the original Feynman diagrams. The main problem in the practical implementation of the Laporta-algorithm is that the number of equations ranges typically up to several millions and thus an eﬀective method is mandatory. Crusher is written in C++ and uses the computer algebra program Fermat [10] for the manipulation of the coeﬃcients of the sought integrals which are rational functions in the space-time, mass and momentum of the space-time and mass parameters of the problem. Very recently a thread-based version of Crusher has been developed, TCrusher. In Fig. 5 the speed-up curves obtained on a four- and eight-core opteron computer, are shown as a function of the number of used threads. The speed-up curves have been generated for a relatively small problem with a runtime of approximately one hour. They show the speed-up for the main and most time consuming part of the program, namely the solving of the equations. A speed-up of three can be reached with four threads. For problems with a higher complexity an even larger speed-up can be reached.

3 Massless Four-Loop Integrals: σ(e+ e− → hadrons) For most of the compute-jobs connected to the massless four-loop integrals we have used about eight processors which leads to the total amount of about 50 processors assuming six jobs in the batch queue. For some jobs it turned out that it is advantageous to use more processors up to a maximum of 32. Almost all results connected to this sub-project were obtained on the XC6000 Itanium nodes. As mentioned in the Introduction R(s) is currently known to order α3s and the corresponding theoretical uncertainty in the value of αs (MZ ) is around 3% which is the same as the experimental one. In future the experimental errors

Massive and Massless Four-Loop Integrals

39

will be decreased and thus it is necessary to compute the O(α4s ) term in order to reduce also the uncertainties in the theoretical prediction. The calculation is highly non-trivial and requires a lot of preparation work to be done which is necessary to fine tune the programs and to accumulate experience. The preparation work has been under way during last years. A few related projects have been successfully completed and their results have been published [11, 12, 13, 14]. Thus, the theoretical possibility of the complete calculation has been demonstrated. As mentioned in Sect. 1 the order α4s contribution to R(s) is related to the absorptive part of the five-loop vector current correlator, whose calculation eventually boils down to finding a host of four-loop propagator-type integrals (p-integrals). In order to cope with the problem the special package, BAICER, has been created. This is a FORM package capable to analytically compute p-integrals up to four loops. Since the conventional method [15] to treat such integrals is not applicable (and is commonly considered as not being possible for the foreseeable future), BAICER uses a completely diﬀerent approach [16, 17, 18]. Namely, the package computes coeﬃcients in decomposition of a given p-integral into the irreducible ones. The coeﬃcients are known to be rational functions of the space-time dimension D and are computed as expansion over 1/D as D → ∞. From the knowledge of suﬃciently many terms in the expansion one can reconstruct their exact form. The terms in the 1/D expansion are expressed in terms of simple Gaussian integrals. Starting from four loops very large number of the latter are necessary to calculate. For example, the first non-trivial five-loop QCD result obtained with BAICER – the α3s n2f contribution to R(s) in QCD [19] – involves several billions of Gaussian integrals. During 2006 and 2007 the following problems have been computed with BAICER on the basis of our local SGI multi-processor computer and the XC6000 in the Rechenzentrum. 1. The O(α5s ) correction to the H → gg partial width of the Standard Model Higgs boson with intermediate mass MH < 2Mt [20]. Its knowledge is useful because with around 20% the O(α4s ) correction is sizeable. The new four-loop correction increases the total Higgs-boson hadronic width by a small amount of order 1 per mille and stabilizes significantly the residual scale dependence. 2. The four-loop anomalous dimensions of the tensor current and the nonsinglet twist-two operator. In the second case we do not only compute the anomalous dimension but also the four-loop matrix element of the (zeromomentum) insertion of the operator between two quark states. Such information is useful for lattice calculations within so-called regularizationindependent renormalization schemes [21, 22]. The knowledge of the anomalous dimension of the tensor quark current is important because it occurs in the matching between currents in QCD eﬀective quark theory [23]. It also appears during the extraction of one of

40

J.H. K¨ uhn, M. Steinhauser, M. Tentyukov

the least known CKM parameters, |Vub |, from the corresponding leptonic and semileptoinc decays of B-mesons (see, e.g., [24, 25]). The results for specific (and phenomenologically most interesting) value for the number of the active quark flavours, nf = 3, have been published in [26]. Meanwhile we have finished calculations for generic nf ; the corresponding publication in preparation. 3. The most extensive calculation performed on the XC6000 is directly related to R(s); namely the evaluation of the β-function for the so-called Quenched3 QED (QQED) in the five loop approximation. In fact, the quantity can be considered as an important gauge independent contribution to R(s) at order α4s (namely the one proportional to the colour structure CF4 ). The QQED β-function is also a fascinating theoretical object by its own right: (i) It is scheme independent in all orders. (ii) Its coeﬃcients are simple rational numbers at one, two, three and four loops: (4/3, 4, −2, −46). (iii) There is a belief that this characteristic reflects some deep not yet understood property of the quantum field theory which should be valid in all orders (see, e.g., [27, 28]). The result of our calculation for the five-loop term of the QQED β-function unexpectedly contains an irrational contribution proportional to ζ(3). At the moment we are checking our calculations and simultaneously we extend them to include all other colour structures necessary for R(s).

4 Massive Vacuum Integrals: Π(q 2 ) to Four Loops As mentioned in the Introduction, it is also one of the aims of the project to compute four-loop corrections to the polarization function Π(q 2 ) including massive quarks. An immediate application is discussed in [29] where very precise values for the charm and bottom quark mass have been extracted. Furthermore, after taking the imaginary part, one would obtain the full quark mass dependence of R(s) to order α3s . The main challenge in the computation of Π(q 2 ) is due the occurrence of two mass scales: the quark mass and the external momentum. After writing down the mathematical expression for the four-loop Feynman diagrams contributing to Π(q 2 ) one has millions of diﬀerent integrals. As mentioned in Sect. 1, the standard approach is to reduce them to a small set of basis integrals with the help of the Laporta-algorithm which we implemented in the parallel program TCrusher. The status of this sub-project is as follows: TCrusher has successfully been applied to the three-loop diagrams. The reduction to master integrals took 3

i.e., diagrams with closed fermion loops are not considered

Massive and Massless Four-Loop Integrals

41

about 48 CPU hours where 4 processors have been used. Once the result is expressed in terms of basis integrals the latter have to be computed. A promising approach is based on an expansion for small external momentum. For illustration we present the results for the coeﬃcient of the 30th expansion term of Π(q 2 ) in limit of small external momentum:

30 αs q2 2 − 0.88385CA CF + 6.0034CF2 Π(q )|3 loops = . . . + π 4m2Q − 0.092427CF TF + 0.52365CF nl TF

+ ... ,

(4)

(nl = nf − 1) which is in agreement with the literature [30]. At this point is should be stressed that in the approach where in a first step an expansion for small q is done and afterwards the reduction to the basis integrals is performed it is hardly possible to reach the tenth expansion term [31]. Currently the reduction of the four-loop diagrams is performed on the cluster XC4000. On average 64 processors are used. The average CPU time needed for the reduction of a typical diagram amounts to 100 days. Acknowledgements The computations presented in this contribution were performed on the Landesh¨ ochstleistungsrechner XC4000 and XC6000.

References 1. F. J. Yndurain, “Relativistic quantum mechanics and introduction to ﬁeld theory,” Berlin, Germany: Springer (1996) 332 p. (Text and monographs in physics) 2. K. G. Chetyrkin, J. H. Kuhn and A. Kwiatkowski, “QCD corrections to the e+ e- cross-section and the Z boson decay rate: Concepts and results,” Phys. Rept. 277 (1996) 189. 3. R. V. Harlander and M. Steinhauser, “rhad: A program for the evaluation of the hadronic R-ratio in the perturbative regime of QCD,” Comput. Phys. Commun. 153 (2003) 244 [arXiv:hep-ph/0212294]. 4. K. G. Chetyrkin and A. Khodjamirian, “Strange quark mass from pseudoscalar sum rule with O(alpha(s)**4) accuracy,” Eur. Phys. J. C 46 (2006) 721 [arXiv:hep-ph/0512295]. 5. M. Tentyukov, D. Fliegner, M. Frank, A. Onischenko, A. Retey, H. M. Staudenmaier and J. A. M. Vermaseren, “ParFORM: Parallel Version of the Symbolic Manipulation Program FORM,” arXiv:cs.sc/0407066; M. Tentyukov, H. M. Staudenmaier and J. A. M. Vermaseren, “ParFORM: Recent development,” Nucl. Instrum. Meth. A 559 (2006) 224. H. M. Staudenmaier, M. Steinhauser, M. Tentyukov, J. A. M. Vermaseren, “ParFORM,” Computeralgebra Rundbriefe 39 (2006) 19. See also http://www-ttp.physik.uni-karlsruhe.de/∼parform.

42

J.H. K¨ uhn, M. Steinhauser, M. Tentyukov

6. FORM version 3.0 is described in: J. A. M. Vermaseren, “New features of FORM,” arXiv:math-ph/0010025; for recent developments, see also: M. Tentyukov and J. A. M. Vermaseren, “Extension of the functionality of the symbolic program FORM by external software,” arXiv:cs.sc/0604052; FORM can be obtained from the distribution site at http://www.nikhef.nl/∼form. 7. M. Tentyukov and J. A. M. Vermaseren, “The multithreaded version of FORM,” arXiv:hep-ph/0702279. 8. S. Laporta and E. Remiddi, “The analytical value of the electron (g-2) at order alpha 3 in QED,” Phys. Lett. B 379 (1996) 283 [arXiv:hep-ph/9602417]. 9. S. Laporta, “High-precision calculation of multi-loop Feynman integrals by diﬀerence equations,” Int. J. Mod. Phys. A 15, 5087 (2000) [arXiv:hepph/0102033]. 10. R. H. Lewis, Fermat’s User Guide, http://www.bway.net/˜lewis. 11. P. A. Baikov, K. G. Chetyrkin and J. H. Kuhn, “Five-loop vacuum polarization in pQCD: O(alpha(s)**4 N(f)**2) results,” Nucl. Phys. Proc. Suppl. 116 (2003) 78. 12. P. A. Baikov, K. G. Chetyrkin and J. H. Kuhn, “Vacuum polarization in pQCD: First complete O(alpha(s)**4) result,” Nucl. Phys. Proc. Suppl. 135 (2004) 243. 13. P. A. Baikov, K. G. Chetyrkin and J. H. Kuhn, “Strange quark mass from tau lepton decays with O(alpha(s**3)) accuracy,” Phys. Rev. Lett. 95 (2005) 012003 [arXiv:hep-ph/0412350]. 14. P. A. Baikov, K. G. Chetyrkin and J. H. Kuhn, “Scalar correlator at O(alpha(s)**4), Higgs decay into b-quarks and bounds on the light quark masses,” Phys. Rev. Lett. 96 (2006) 012003 [arXiv:hep-ph/0511063]. 15. K. G. Chetyrkin and F. V. Tkachov, “Integration By Parts: The Algorithm To Calculate Beta Functions In 4 Loops,” Nucl. Phys. B 192 (1981) 159. 16. P. A. Baikov, “Explicit solutions of the multi-loop integral recurrence relations and its application,” Nucl. Instrum. Meth. A 389 (1997) 347 [arXiv:hepph/9611449]. 17. P.A. Baikov, “Explicit solutions of the three loop vacuum integral recurrence relations,” Phys. Lett. B 385 (1996) 404 [arXiv:hep-ph/9603267]. 18. P. A. Baikov, “The criterion of irreducibility of multi-loop Feynman integrals,” Phys. Lett. B 474 (2000) 385 [arXiv:hep-ph/9912421]. 19. P. A. Baikov, K. G. Chetyrkin and J. H. Kuhn, “The cross section of e+ eannihilation into hadrons of order alpha(s)**4 n(f)**2 in perturbative QCD,” Phys. Rev. Lett. 88 (2002) 012001 [arXiv:hep-ph/0108197]. 20. P. A. Baikov and K. G. Chetyrkin, “Higgs decay into hadrons to order alpha(s)**5,” Phys. Rev. Lett. 97 (2006) 061803 [arXiv:hep-ph/0604194]. 21. G. Martinelli, C. Pittori, C. T. Sachrajda, M. Testa and A. Vladikas, “A General Method For Nonperturbative Renormalization Of Lattice Operators,” Nucl. Phys. B 445 (1995) 81 [arXiv:hep-lat/9411010]. 22. M. Gockeler et al., “Nonperturbative renormalisation of composite operators in lattice QCD,” Nucl. Phys. B 544 (1999) 699 [arXiv:hep-lat/9807044]. 23. D. J. Broadhurst and A. G. Grozin, “Matching QCD And Hqet Heavy - Light Currents At Two Loops And Beyond,” Phys. Rev. D 52 (1995) 4082 [arXiv:hepph/9410240]. 24. V. M. Braun, T. Burch, C. Gattringer, M. Gockeler, G. Lacagnina, S. Schaefer and A. Schafer, “A lattice calculation of vector meson couplings to the vector

Massive and Massless Four-Loop Integrals

25.

26. 27. 28. 29. 30. 31.

43

and tensor currents using chirally improved fermions,” Phys. Rev. D 68 (2003) 054501 [arXiv:hep-lat/0306006]. D. Becirevic, V. Lubicz, F. Mescia and C. Tarantino, “Coupling of the light vector meson to the vector and to the tensor current,” JHEP 0305 (2003) 007 [arXiv:hep-lat/0301020]. P. A. Baikov and K. G. Chetyrkin, “New four loop results in QCD,” Nucl. Phys. Proc. Suppl. 160 (2006) 76. D. J. Broadhurst, “Four-loop Dyson-Schwinger-Johnson anatomy,” Phys. Lett. B 466 (1999) 319 [arXiv:hep-ph/9909336]. A. Connes and D. Kreimer, “Renormalization in quantum ﬁeld theory and the Riemann-Hilbert problem,” JHEP 9909 (1999) 024 [arXiv:hep-th/9909126]. J. H. Kuhn, M. Steinhauser and C. Sturm, “Heavy quark masses from sum rules in four-loop approximation,” arXiv:hep-ph/0702103. R. Boughezal, M. Czakon and T. Schutzmeier, “Four-loop tadpoles: Applications in QCD,” Nucl. Phys. Proc. Suppl. 160 (2006) 160 [arXiv:hep-ph/0607141]. K. G. Chetyrkin, J. H. Kuhn and M. Steinhauser, “Heavy quark current correlators to O(alpha(s)**2),” Nucl. Phys. B 505 (1997) 40 [arXiv:hep-ph/9705254].

Structural Transitions in Colloidal Suspensions Martin Hecht and Jens Harting Institut für Computerphysik, Pfaﬀenwaldring 27, 70569 Stuttgart, Germany

Summary. In suspensions of colloidal particles diﬀerent types of interactions are in a subtle interplay. In this report we are interested in sub-micro meter sized Al2 O3 particles which are suspended in water. Their interactions can be adjusted by tuning the pH-value and the salt concentration. In this manner diﬀerent microscopic structures can be obtained. Industrial processes for the production of ceramics can be optimized by taking advantage of speciﬁc changes of the microscopic structure. To investigate the inﬂuences of the pH-value and the salt concentration on the microscopic structure and the properties of the suspension, we have developed a coupled Stochastic Rotation Dynamics (SRD) and Molecular Dynamics (MD) simulation code. The code has been parallelized using MPI. We utilize the pair correlation function and the structure factor to analyze the structure of the suspension. The results are summarized in a stability diagram. For selected conditions we study the process of cluster formation in large scale simulations of dilute suspensions. Key words: Stochastic Rotation Dynamics; Molecular Dynamics; colloids; clustering

1 Introduction Colloid science is a very fascinating research ﬁeld, gaining more and more importance in the last years. It closely connects physics, chemistry, material science, biology, and several branches of engineering technology. According to its key role in modern science a considerable amount of research has been performed to describe colloidal suspensions from a theoretical point of view and by simulations [16, 28, 29, 41, 47, 49] as well as to understand the particleparticle interactions [3, 11, 12, 15, 51, 52], the phase behavior [10, 23, 32, 50], the relevant processes on the microscale and their inﬂuence on macroscopic parameters [13, 40, 54]. Colloidal suspensions are in fact complicated systems, since depending on the particle sizes, materials, and concentrations, diﬀerent interactions are of relevance and often several of them are in a subtle interplay: electrostatic repulsion, depletion forces, van der Waals attraction,

46

M. Hecht, J. Harting

hydrodynamic interaction, Brownian motion, and gravity are the most important inﬂuences. The properties of the suspension strongly depend on the balance of the microscopic forces between the particles. Especially for industrial processes, where one needs to optimize certain material properties a detailed understanding of the relevant inﬂuences is needed. The stability of diﬀerent microstructures and especially the clustering process are key properties which are of interest. In our work we investigate these properties, focusing on Al2 O3 particles suspended in water. This is a widely used material in ceramics [37]. We have developed a simulation code for a Brownian suspension [20] and have adjusted the simulation parameters so that the simulation corresponds quantitatively to a real suspension such that experimental data can be compared directly. The diﬀusion coeﬃcient, sedimentation velocity [20], and the viscosity of the suspension can be reproduced [17]. We also have tested the inﬂuence of polydispersity and found that its inﬂuence on the results is small. It is much more important to choose the correct mean size of the particles [17]. For Al2 O3 suspensions attractive van der Waals forces are important for the behavior of this material. Electrostatic repulsion of the charged particles counteracts the attraction and can prevent clustering depending on the particle surface charge. In [17] we have presented how one can relate parameters of DLVO potentials [11, 52] with experimental conditions. In the experiment one can control the pH-value and the salt concentration. The latter can be expressed by the ionic strength I, which is an eﬀective concentration of all ions present in the solution. Both, the pH-value and the ionic strength, inﬂuence the charge of the colloidal particles. We have shown that for not too strongly attractive forces one can obtain reasonable quantitative agreement with experimental results. Three regimes can be identiﬁed and plotted in a stability diagram [17], which we want to investigate here in more detail: A clustered regime, in which particles aggregate to clusters, a ﬂuid-like and stable stable suspension and a repulsive region, for which the microstructure is similar to the ones known from glassy systems. From our previous work we know that our model works well, even quantitatively, in the suspended regime of the stability diagram and close to the borders between the diﬀerent microstructures. Here we extend our investigations to diﬀerent pH-values, deeper in the clustered regime, and to the repulsive structure. We expect to gain insight to the microscopic structure on a qualitative level. On these grounds we have explored the stability diagram of Al2 O3 suspensions. The particles are uncharged close to the so called “isoelectric point” at pH = 8.7. There, for all ionic strengths the particles form clusters. For lower pH-values particles can be stabilized in solution by the electrostatic repulsion due to the charge the particles carry in this case. For low pH-values, low salt concentrations, and high volume fractions a repulsive structure can be found. In the following section we shortly describe our simulation method. After that we discuss the properties which can be found in our suspensions and how

Structural Transitions in Colloidal Suspensions

47

diﬀerent regimes of the stability diagram are distinguished. In the section thereafter we describe our simulation setup. Then, we present our results and discuss the criteria we apply to characterize the microstructures. We utilize the pair correlation function and the structure factor to characterize the clustering behavior. Both of them in principle contain the same information, but we concentrate on certain peaks in either of them. Each peak in the correlation function and in the structure factor corresponds to a certain length scale and we chose either the correlation function or the structure factor, depending on which of the two quantities is more suitable under numerical criterions to observe on a given length scale. To characterize the repulsive region we evaluate the mean squared displacement (MSD), which shows a plateau, if the particle motion consists of diﬀerent processes acting on well separated time scales. Finally, the results are summarized in a stability diagram for our Al2 O3 -suspension. It shows the behavior of the suspension in an intuitive way and helps to design industrial processes using this material. After that, we turn to dilute suspensions of only 5% volume fraction and study cluster growth at low shear rates in these suspensions. Finally, we shortly summarize our results. The results which we present in this report have been accepted for publication in [18, 19].

2 Simulation Method Our simulation method is described in detail in [17, 20] and consists of two parts: a Molecular Dynamics (MD) code, which treats the colloidal particles, and a Stochastic Rotation Dynamics (SRD) simulation for the ﬂuid solvent. In the MD part we include eﬀective electrostatic interactions and van der Waals attraction, known as DLVO potentials [11, 52], a lubrication force and Hertzian contact forces. DLVO potentials are composed of two terms, the ﬁrst one being an exponentially screened Coulomb potential due to the surface charge of the suspended particles 2 d2 zeζ 2 + κd 4kB T VCoul = πεr ε0 · tanh exp(−κ[r − d]), (1) × 1 + κd ze 4kB T r

where d denotes the particle diameter, r the distance between the particle centers, e the elementary charge, T the temperature, kB the Boltzmann constant, and z is the valency of the ions of added salt. ε0 is the permittivity of the vacuum, εr = 81 the relative dielectric constant of the solvent, κ the inverse Debye length deﬁned by κ2 = 8πℓB I, with ionic strength I and Bjerrum length ℓB = 7 . The eﬀective surface potential ζ can be related to the pH-value of the solvent with a 2pK charge regulation model [17]. The Coulomb term competes with the attractive van der Waals interaction (AH = 4.76 · 10−20 J is the Hamaker constant) [21] 2 d2 AH d2 r − d2 VVdW = − . (2) + 2 +2 ln 12 r2 − d2 r r2

48

M. Hecht, J. Harting

The attractive contribution VVdW is responsible for the cluster formation we observe. However, depending on the pH-value and the ionic strength, it may be overcompensated by the electrostatic repulsion. When particles get in contact, the potential has a minimum. However, (2) diverges due to the limitations of DLVO theory. We cut oﬀ the DLVO potentials and model the minimum by a parabola as described in [20]. The particle contacts are modeled as Hertzian contacts and for non-touching particles. Below the resolution of the SRD algorithm short range hydrodynamics is corrected by a lubrication force, which we apply within the MD framework, as we have explained in [17, 20]. For the integration of translational motion of the colloidal particles we utilize a velocity Verlet algorithm [5]. For the simulation of a ﬂuid solvent, many diﬀerent simulation methods have been proposed: Stokesian Dynamics (SD) [6, 7, 41], Accelerated Stokesian Dynamics (ASD) [45, 46], pair drag simulations [47], Brownian Dynamics (BD) [21, 22], Lattice Boltzmann (LB) [27, 28, 29, 30], and Stochastic Rotation Dynamics (SRD) [20, 26, 38]. These mesoscopic ﬂuid simulation methods have in common that they make certain approximations to reduce the computational eﬀort. Some of them include thermal noise intrinsically, or it can be included consistently. They scale diﬀerently with the number of embedded particles and the complexity of the algorithm diﬀers largely. We apply the Stochastic Rotation Dynamics method (SRD) introduced by Malevanets and Kapral [33, 34]. It intrinsically contains ﬂuctuations, is easy to implement, and has been shown to be well suitable for simulations of colloidal and polymer suspensions [4, 17, 20, 26, 38, 42, 53] and recently for star-polymers in shear ﬂow [44]. The method is also known as “Real-coded Lattice Gas” [26] or as “multi-particle-collision dynamics” (MPCD) [43]. It is based on so-called ﬂuid particles with continuous positions and velocities. A streaming step and an interaction step are performed alternately. In the streaming step, each particle i is moved according to ri (t + τ ) = ri (t) + τ vi (t),

(3)

where ri (t) denotes the position of the particle i at time t and τ is the time step. In the interaction step the ﬂuid particles are sorted into cubic cells of a regular lattice and only the particles within the same cell interact among each other according to an artiﬁcial interaction. The interaction step is designed to exchange momentum among the particles, but at the same time to conserve total energy and total momentum within each cell, and to be very simple, i.e., computationally cheap: each cell j is treated independently. Nj (t′ ) First, the mean velocity uj (t′ ) = Nj1(t′ ) i=1 vi (t) is calculated. Nj (t′ ) is the number of ﬂuid particles contained in cell j at time t′ = t + τ . Then, the velocities of each ﬂuid particle in cell j are rotated according to vi (t + τ ) = uj (t′ ) + Ωj (t′ ) · [vi (t) − uj (t′ )].

(4)

Ωj (t ) is a rotation matrix, which is independently chosen at random for each time step and each cell. We use rotations about one of the coordinate axes ′

Structural Transitions in Colloidal Suspensions

49

by an angle ±α, with α ﬁxed. The coordinate axis as well as the sign of the rotation are chosen at random, resulting in 6 possible rotation matrices. To remove anomalies introduced by the regular grid, one can either choose a mean free path of the order of the cell size or shift the whole grid by a random vector once per SRD time step as proposed by Ihle and Kroll [24, 25]. Three diﬀerent methods to couple the SRD and the MD simulation have been introduced in the literature. Inoue et al. proposed a way to implement no slip boundary conditions on the particle surface [26]. Padding and Louis very recently came up with full slip boundaries, where the ﬂuid particles interact via Lennard-Jones potentials with the colloidal particles [39]. Falck et al. [14] have developed a “more coarse grained” method which we use for our simulations and which we descibe shortly in the following. To couple the colloidal particles to the ﬂuid, the colloidal particles are sorted into the SRD cells and their velocities are included in the rotation step. One has to use the mass of each particle –colloidal or ﬂuid particle –as a weight factor when calculating the mean velocity ′

Nj (t ) 1 uj (t ) = vi (t)mi , Mj (t′ ) i=1 ′

(5)

Nj (t′ )

with

′

Mj (t ) =

mi ,

(6)

i=1

where we sum over all colloidal and ﬂuid particles in the cell, so that Nj (t′ ) is the total number of both particles, ﬂuid plus colloidal ones. mk is the mass of the particle with index i and Mj (t′ ) gives the total mass contained in cell j at time t′ = t + τ . To some of our simulations we apply shear. This is realized by explicitly setting the mean velocity uj to the shear velocity in the cells close to the border of the system. Both, colloidal and ﬂuid particles, are involved in this additional step. A thermostat is applied to remove the energy introduced to the system by the shear force. We have described the simulation method in more detail in [17, 20]. A single simulation run as presented in these papers took between one and seven days on a 3GHz Pentium CPU. However, for strongly clustering systems we easily end up with only a single cluster inside the simulation volume. In order to be able to gather statistics on cluster growth and formation, as well as to minimize ﬁnite size eﬀects, we parallelized our code. While MD codes have been parallelized by many groups, only few parallel implementations of a coupled MD and SRD program exist. This is in contrast to the number of parallel implementations of other mesoscopic simulation methods like for example the lattice Boltzmann method. A possible explanation is that SRD is a more recent and so far not as widely used algorithm causing the parallelization to be a more challenging task. We utilize the Message Pasing Interface (MPI) to create a C++ code based on domain decomposition for both involved simulation methods. In

50

M. Hecht, J. Harting

the MD code the position of neighbouring particles is needed to compute the interactions. Since the intractions have a limited range, and a linked cell algorithm is already used in the serial code, we apply linked cells here as well. Particle positions at the border of the domain of each processor are communicated to the neighbouring processors for the calculation of the forces. Then, the propagation step is performed and particle positions are updated, whereby the particles crossing a domain boundary are transferred from one processor to the other one. Since (in principle), ﬂuid particles can travel arbitrary large distances in one time step, one either has to limit the distance they can move, or one needs all-to-all communication between the processors. Even though the mean free path in our systems is small enough to limit communication to nearest neighbours only, the current version of our code tries to be as general as possible and allows ﬂuid particles to move to any possible position in the total simulation volume within a single timestep. First, we determine locally which ﬂuid particles have to be sent to which destination CPU and collect all particles to be sent to the same destination into a single MPI message. If no particles are to be sent, a zero dummy message is transmitted. On the receiving side, MPI_Probe with the MPI_ANY_SOURCE option is utilized to determine the sender’s rank and the number of particles to be accomodated. Now, MPI_Recv can be used to actually receive the message. All processors send and receive in arbitrary order, thus waiting times are kept at a minimum allowing a very eﬃcient communication. The standard MPI all-to-all communication procedure should be less eﬃcient since the size of every message would be given by the size of the largest message. However, we still do ﬁnd a substantial communication overhead from our benchmark tests of the scalability of the code. Due to this overhead, we are currently limited to 32 CPUs. In order to achieve Gallilean invariance, a random shift of the SRD lattice is performed for every rotation step [24, 25]. Since the domains managed by each CPU do not move, this would include the borders between the processors to cross SRD cells, which is undesirable. Therefore, we keep the position of the lattice ﬁxed and shift the ﬂuid particle positions before sorting them into the cells instead. After the rotation step they are shifted back.

3 Background We examine the microstructures obtained in our simulations for diﬀerent conditions. We vary the pH-value and the ionic strength I. The shear rate γ˙ as an external inﬂuence is varied as well. We classify the microstructures in three categories: suspended, clustered, and repulsive. In the suspended case, the particles can move freely in the ﬂuid and do not form stable clusters. In the clustered regime the particles form clusters due to attractive van der Waals forces. These clusters can be teared apart if shear is applied. In some of our simulations the clusters are very weakly connected and at small shear rates

Structural Transitions in Colloidal Suspensions

51

they are not only broken up into smaller pieces, but they dissolve to freely moving individual particles. In this case, we assign the microstructure to the suspended region, although in complete absence of the shear ﬂow clusters are formed. At the borders between the diﬀerent regimes in fact no sharp transitions can be observed. The DLVO forces rather steadily increase and compete with the hydrodynamic interactions. Accordingly, in experiments one cannot observe a sudden solidiﬁcation, but a steadily increasing viscosity when leaving the suspended regime [17]. Similarly as for attractive forces, repulsive interactions can restrict the mobility of the particles. If this happens, the mean squared displacement of the particles shows a pronounced plateau, as it can be found in glassy systems. However, we speak of a “repulsive structure”, because the change of the viscosity is not as strong as in glasses, where it often changes by many oders of magnitude, when the glass transition is approached. In addition, to claim a system shows a glassy behavior would require to investigate the temperature dependence of a typical time (e.g. particle diﬀusion time) and to show its divergence as the glass temperature is approached. This is diﬃcult to do in the framework of our simulation model [20] and therefore we prefer to speak about a “repulsive structure” which might be identiﬁed as a colloidal glass in future work. Here we would like to emphasize the analysis of the microstructure for diﬀerent conditions. Our aim is to reproduce a so-called stability diagram by simulations. The stability diagram depicts the respective microstructure depending on the pH-value and the ionic strength I. We apply diﬀerent numerical tools to analyze the microstructure in our simulations and ﬁnally arrive at a stability diagram shown in Fig. 7, which summarizes the results which we present in the following sections.

4 Simulation Setup In this study the colloidal particles are represented by three dimensional spheres of d = 0.37 µm in diameter. This is the mean diameter of the particles used in the experiments to which we refer in [17]. We have simulated a small volume, 24 d = 8.88 µm long in x-direction, which is the shear direction, and 12 d = 4.44 µm long in y- and z-direction. We have varied the volume fraction between Φ = 10 % (660 particles) and Φ = 40 % (2640 particles). Most of the simulations were performed at Φ = 35 % (2310 particles). To study low volume fractions Φ = 5 % we have enlarged the simulation volume to 24 d = 8.88 µm in each direction and we have further scaled up the system in each dimesion by a factor of 2 or 4, resulting in a cube of 48d = 17.76µm. For selected pH-values and ionic strengths we have studied the cluster growth of dilute suspensions at low shear rates (γ˙ = 20/s). We use periodic boundaries in x- and y-direction and closed boundaries in z-direction [17]. Shear is applied in x-direction by moving small zones of

52

M. Hecht, J. Harting

particles and ﬂuid close to the wall with a given shear velocity. The xy-plane is our shear plane. For simulations without shear, to achieve the best comparability, we use the same boundary conditions and just set the shear rate to γ˙ = 0. In addition we have performed simulations with two diﬀerent shear rates: with γ˙ = 100/s and with γ˙ = 500/s.

5 Results and Discussion First, we focus on simulations without shear, where one can predict intuitively, what should happen. Qualitatively the results are similar to our earlier work [20], but the quantitative relation between the pH-value and the potentials is new. The relation was presented in [17], but here we apply it to diﬀerent cases and we focus more on the characterization of the microstructure. However, given the particle particle interaction potentials, the microstructure in equilibrium can be predicted easily, at least on a qualitative level. But, the matter changes and gets more sophisticated, when shear is applied and an interplay between shear ﬂow and particle particle interactions becomes responsible for the resulting microstructure. At the end of this section we move on to dilute suspensions and study the growth of clusters at low shear rates. 5.1 Correlation Function For constant ionic strength I = 3 mmol/l the local microstructure can be examined using the correlation function. Depending on the pH-value the behavior of the system changes from a repulsive structure around pH = 4 to a stable suspension around pH = 6 towards a clustered region if the pH-value is further increased, until the isoelectric point is reached at pH = 8.7. There clustering occurs in any case, independent on the ionic strength. This can be seen in the structure of the correlation function V g(r) = 2 δ(r − rij ) , (7) N i j =i

(see [5] p. 55), where V is the volume, N the number of particles and rij the distance of two particles i and j. At pH = 4 electrostatic repulsion prevents clustering: Particles are suspended, and there is no ﬁxed long range ordering in the system. The correlation function (Fig. 1) shows a maximum at a typical nearest neighbor distance slightly above dr = 1 with d denoting the particle diameter, then in the layer of next neighbors small correlations can be found (at dr = 2). For larger distances the correlation function is rather constant. When the pH-value is increased, the surface charge is lower, which at ﬁrst causes the particles to approach each other more closely. The maximum of the correlation function is shifted to smaller distances (see Fig. 1, note that the

Structural Transitions in Colloidal Suspensions

53

curves are shifted vertically in the plot by a factor of 3 for better visibility.). Then, van der Waals attraction becomes more important and clustering begins. One can see this in the correlation function where a sharp structure at particle distances between 1.5 and 2 particle diameters occurs. There is a nearest neighbor peak, and more complicated structures at larger distances, which we have assigned to typical particle conﬁgurations for small distances [20]. In a solid like cluster the position of the next neighbor is ﬁxed more sharply than in the suspension, consequently the nearest neighbor peak becomes sharper, and its height is increased. Close to the isoelectric point (pH = 8.7) the barrier between primary and secondary minimum disappears. The particles, once clustered, cannot rearrange anymore, and therefore the correlations to the next neighbors become less sharp again (compare the cases of pH = 8.7 and pH = 7.7 in Fig. 1 at the positions denoted by the arrows). Instead of varying the pH-value, one can also vary the ionic strength to achieve similar eﬀects. Increasing the ionic strength, experimentally speaking “adding salt” decreases the screening length 1/κ and therefore the attractive forces become more important: the particles start to form clusters. The eﬀects described up to here can be observed with or without shear qualitatively in an analogous manner. If the suspension is sheared clustering occurs at higher pH-values and the peaks found in the correlation function are slightly broadened, because the relative particle positions are less ﬁxed. But a new feature appears, if a stable suspension of not too high volume fraction is sheared. Induced by the shear particles arrange themselves in layers. Regular nearest neighbor distances in the shear plane cause the correlation function to

Fig. 1. Dependence of the particle correlation function on the pH value, I = 3 mmol, γ˙ = 0/s Φ = 35%. The plots for four diﬀerent pH-values are shifted against each other for better visibility by a factor of 3. For pH = 4 the particles are not clustered. Hence the structure at dr = 2 is less sharp than in the other three curves of the plot and the nearest neighbor peak (at dr = 1) is broad. For pH = 6.5 slight clustering starts, the structures become sharper. For pH = 7.7 strong cluster formation is reﬂected in very sharp structures. For pH = 8.5 electrostatic repulsion nearly disappears so that no barrier between primary and secondary minimum exists anymore. The particles cannot rearrange anymore, and therefore the structures labeled by the arrows become smoothened compared to the case of pH = 7.7 (source: [19])

54

M. Hecht, J. Harting

Fig. 2. Nearest neighbor peak (primary and secondary minimum of the potential) of the correlation function I = 3 mmol/l, Φ = 35 %: For low pH-values clustering is prevented by the electrostatic repulsion. For high pH-values the particles form clusters, which is reﬂected by an increased nearest neighbor peak. First, shear prevents clustering, then depending on the shear rate, cluster formation takes place. Low shear rates even support cluster formation at high pH-values (source: [19])

become more structured even for large distances. The long range structure of the pair correlation function appears after a transient time the particles need to arrange themselves in the layered structure. Shear induced layer formation has been found in both, experiments [1, 2] and simulations [8, 9, 36] We have integrated over the nearest neighbor peaks, both, the peaks of the primary and the secondary minimum, and plotted the integral versus pH-value in Fig. 2. We have chosen I = 3 mmol/l and Φ = 35 % and three diﬀerent shear rates: γ˙ = 0, 100 and 500/s. We have integrated the correlation function for r < 1.215 d, where for all pH-values the potential in the secondary minimum has a value of − 21 kB T . In other words, we have captured the primary and the secondary minimum of the potential for this plot. For low pH-values clustering (in the secondary minimum) is only possible for low shear rates. For high shear rates, the hydrodynamic forces do not allow the formation of stable clusters. For rising pH-values the clustering increases, ﬁrst for the un-sheared suspension, at higher pH-values for low shear rates (γ˙ = 100/s) and ﬁnally for high shear rates (γ˙ = 500/s). Remarkably, for pH > 7.5 the curve for γ˙ = 100/s shows stronger cluster formation than the other ones. Particles are brought together by the shear ﬂow, so that compared to the case of no shear, the clustering process is supported here. On the other hand, the shear stress may not be too strong, because otherwise the clustering process is limited by the shear ﬂow again (for γ˙ = 500/s the clustering is less pronounced than for γ˙ = 100/s). 5.2 Structure Factor The pair correlation function can be used to characterize the local order of the microstructure on the length scale of the particle size. However, to do the characterization on the length scale of the system size, we use the structure

Structural Transitions in Colloidal Suspensions

55

factor deﬁned by S(k) =

N 1 exp(ik · rlm ), N

(8)

l,m=1

where N is the number of particles, and rlm is the vector from particle l to particle m. i denotes the imaginary unit here. The structure factor is related to the pair correlation function in real space by a three dimensional Fourier transform. In principle the structure factor contains the same information as the pair correlation function. However, due to numerical reasons and our implementation of shear boundary conditions it is easier to observe the longrange structure in the structure factor than in the pair correlation function. In Fig. 3 we have plotted several typical structure factors of our simulations. For these plots the pH-value is ﬁxed to pH = 6. The cases a) and b) are sheared with γ˙ = 500/s at an ionic strength of I = 0.3 mmol/l. In case a) the volume fraction Φ = 20% is relatively low. Therefore the particles can arrange themselves in layers parallel to the shear plane, which move relatively independently in the shear ﬂow. They have a certain distance ﬁxed in space and time. This can be seen in a sharp peak at a dimensionless k-vector of k = 5.2, which corresponds to a distance of 1.2 particle diameters. In fact, this is exactly the distance between two neighboring layers, as one can easily verify by counting the layers in a snapshot of the system (Fig. 4a)). The particles in the layers do not have a ﬁxed distance and therefore no 2nd-order-peak can be observed.

Fig. 3. Structure factor for some selected examples, with pH = 6 ﬁxed for all plots: γ˙ = 500/s, I = 0.3 mmol/l: a) Φ = 20% and b) Φ = 35% , γ˙ = 0, I = 25 mmol/l: c) Φ = 40% and d ) Φ = 10% . The curves are shifted vertically for better visibility. In case a) ten layers can be identiﬁed in the system, resulting in the strong peak close to 5. But, since the particles in the layers can still move freely, there is no 2nd-order-peak. In case b) layers are formed, but particles are moving from one layer to the other, disturbing the ﬂow. As a result the nearest neighbor peak is much broader. Due to the structure in the layers, a 2nd-order-peak appears. In case c) the interaction is strongly attractive, hence the particles approach each other and the nearest neighbor peak is shifted to higher k-vectors. In case d ) the volume fraction is much less. The slope of the low-k-peak is much ﬂatter, which depicts that the cluster is fractal (source: [19])

56

M. Hecht, J. Harting

For case b) the volume fraction is increased to Φ = 35%. The particle layers are packed more densely and therefore the interactions between one layer and the neighboring one become relevant. Particles jump from one layer to the other, which disturbs the ﬂow and therefore the distance between the layers is not ﬁxed anymore. The sharp peak on top of the nearest neighbor peak disappears. Instead of that, in each layer a regular hexagonal order appears and therefore the 2nd-order-peak is much more pronounced. In case c) the ionic strength is increased to I = 25 mmol/l. The interparticle potentials are attractive enough that aggregation takes place. In this simulation we did not apply shear, therefore one ﬁnds only one big cluster in the system (compare Fig. 4c)). In the cluster the particles are packed more densely and consistently the nearest neighbor peak in the structure factor is shifted to larger k-vectors. The volume fraction is Φ = 40% in this case. In case d) the volume fraction is decreased to Φ = 10%. The particles still form clusters, but their mobility is not high enough to create one compact cluster. The system has a fractal structure (see Fig. 4d)). This can be seen in the structure factor as well: The slope for low k-vectors is ﬂatter in this case compared to cases a)–c). A ﬂatter slope of the low-k-peak is typical for structure factors of fractal objects. The fractal dimension of the cluster extracted from the slope of the low-k-peak is 2.5. In experiments this relation is often used to determine the fractal dimension of a sample: Lattuada et al. [31] have evaluated the fractal dimension of agglomerates of latex particles from the slope of the structure factor. McCarthy et al. [35] give an introduction to scattering intensities at fractal objects, without mentioning the structure factor, but their arguments refer to the contribution of the structure factor on the scattering intensity. The underlying mechanism which is responsible for these structures is cluster cluster aggregation [48]. In Fig. 5 we show the dependence of the low-k-peak of the structure factor on the pH-value. Here we have integrated over dimensionless k-vectors smaller than 3 which means, we have captured structures larger than twice a particle diameter. A large integral over the low-k-peak is due to a large inhomogeneity in the system. In one part of the system particles are present and in the other part not. In other words, we observe the process of cluster formation on a length scale of the system size. Without shear, particles cluster in the secondary minimum for all pH-values. If the system is slightly sheared (γ˙ = 100/s) clustering is suppressed for low pH-values. Starting at pH = 6 cluster formation starts and is even supported by the shear ﬂow for pH-values larger than 7.5. For large shear rates (γ˙ = 500/s) cluster formation is suppressed by the shear ﬂow. By analyzing the low-k-peak of the structure factor one observes on the length scale of the system size. The same behavior of the system can be seen by analyzing the pair correlation function, as we have already shown in Fig. 2. In that case one analyzes the number of nearest neighbors, that means, one observes the length scale of a particle diameter. Nevertheless, both graphs show the same behavior of the system, i.e., we have

Structural Transitions in Colloidal Suspensions

57

Fig. 4. Snapshots of the systems analyzed in Fig. 3: In case a) one can see the layers resulting in the sharp peak in the structure factor. In case b) the layers are packed closer due to the higher volume fraction. Collisions between particles of neighboring layers happen more frequently. In case c) one big cluster is formed. The particles are packed densely. In case d) the fractal nature of the system can be seen directly (source: [19])

Fig. 5. low-k-peak for diﬀerent pH-values and diﬀerent shear rates. The ionic strength I is kept constant at I = 3 mmol/l and the volume fraction is always Φ = 35%. For γ˙ = 0/s the particles tend to cluster in the secondary minimum of the potential. This clustering can easily be broken up, if shear is applied. If the pH-value is increased, shear cannot prevent cluster formation anymore. At low shear rates (γ˙ = 100/s) clustering is even enhanced, since the particles are brought closer to each other by the shear ﬂow (source: [19])

58

M. Hecht, J. Harting

a consistent picture of the cluster formation process on the length scale of the nearest neighbors and on the length scale of the system size. Thus we have conﬁrmed that the cluster formation process is not limited to length scales smaller than our system size. This is reﬂected especially by the transition between pH = 7–8 and its shear rate dependence in the plots in Fig. 5 and Fig. 2. There is a strong similarity of the two plots, which are obtained by two evaluation methods referring to two diﬀerent length scales. This conﬁrms that the plots do not only reﬂect how clusters are formed on the respective length scale, but that the clustering process is a phenomenon which can be observed on any length scale by applying a suitable method to characterize it. 5.3 Repulsive Regime To characterize the repulsive regime, we evaluate the mean squared displacement for the particles. In Fig. 6 we plot the mean squared displacement for diﬀerent ionic strengths. The pH-value is kept constant at pH = 6 and the volume fraction is Φ = 35% for this plot. Three diﬀerent regimes can be identiﬁed. For very short times, the ballistic regime: particles move on short distances without a notable inﬂuence by their neighbors. The distances are in the order of some percent of the particle diameter and the times are a few SRD time steps. For larger times the particles interact with their neighbors and therefore their mobility is limited due to collisions with the neighbors. This is reﬂected in the mean squared displacement by a plateau of reduced slope, which is the more pronounced the more the mobility of the particles is restricted. For even larger time scales collective motion starts, i.e., clusters or

Fig. 6. Mean squared displacement at pH = 6 for diﬀerent ionic strengths, without shear. One can see a ballistic regime for short times, a central plateau and a collective long time movement which can be a movement of a whole cluster or cage escape events of single particles. Depending on the ionic strength, the central plateau is more or less pronounced. A comparison of the plateau for diﬀerent simulations can be used to decide, if a certain state belongs to the repulsive region of the stability diagram. A state well in the liquid microstructure should be used as a reference for the comparison (source: [19])

Structural Transitions in Colloidal Suspensions

59

groups of particles move, or single particles can escape from a cage formed by its neighbors. Depending on the ionic strength diﬀerent eﬀects are important and thus the shape of the curve is diﬀerent. For large ionic strengths the particles form clusters and these clusters may drift or rotate in the system. Then the collective motion is more dominant and the mean squared displacement grows faster than in single particle diﬀusion. The mean squared displacement does not show a plateau, then. But in the repulsive regime, the neighbors limit the motion of the particles, and the slope of the plateau is ﬂatter, i.e., the plateau is even more pronounced, compared to the suspended case. In the repulsive regime the particles tend to arrange themselves in layers when shear is applied [17] and long range correlations can be found in un-sheared systems [20]. 5.4 Stability Diagram The results of the investigations presented up to here can be summarized in a stability diagram for our Al2 O3 -suspension (Fig. 7). Three diﬀerent microstructures can be identiﬁed: a repulsive structure, a suspended region and a clustered region. In contrast to our previous work [17, 20], we have explored the parameter space more in the repulsive regime and deeper in the clustered region. We use the mean squared displacement, the correlation function, and the structure factor, to decide to which of the three microstructures a certain point in the stability diagram belongs. However, the borders between the re-

Fig. 7. Stability diagram (plotted for Φ = 35% and without shear): depicting three regions: a clustered region (filled circles), a suspended regime (open squares), and a repulsive structure (filled squares). In the clustered region particles aggregate which leads to inhomogeneity in the system. In the suspended regime, the particles are distributed homogeneously in the system and they can move freely. In the repulsive regime the mobility of the particles is restricted by electrostatic repulsion exerted by their neighbors. As a result they arrange in a local order which maximizes nearest neighbor distances. The borders between the regimes are not sharp. They depend on the shear rate and on the volume fraction. Therefore we have indicated the crossover regions by the shaded patterns. The lines are guides to the eye (source: [19])

60

M. Hecht, J. Harting

gions are not sharp and they depend on the shear rate. We have indicated the crossover regions by the shaded patterns in the stability diagram. If the volume fraction is decreased, the region of the repulsive structure becomes smaller. To decide if a state is in the suspended region or in the repulsive one of the stability diagram, we have compared the plots of the mean squared displacement for the simulations without shear. If the plateau was pronounced there, we have counted the state among the repulsive regime. As a second criterion one can compare the pair correlation function. If there are long range correlations even though the system is not sheared, then the microstructure is the repulsive one. Finally, the shear force can be used to localize the border to the repulsive regime. For a given shear rate and a ﬁxed volume fraction, the shear force depends on the particle interactions. If the shear force increases compared to a state well in the suspended regime, the motion of the particles is blocked by the electrostatic interaction in the repulsive regime. Thus, to decide, if a state belongs to the clustered or to the suspended regime, we ﬁrst study the snapshots of the system. If we see no clusters there, the clustered regime can be excluded. But, if we see clusters, we also consult numerical quantities like the density of clusters and the rate of cluster growth into account. Both, the density and the time for cluster growth are indications for the stability of the clusters. If they grow slowly and their density is low, we count the state to the suspended regime. The stability diagram obtained by these criteria is consistent with the results of the simulations with shear ﬂow, shown in Fig. 2 and Fig. 5. Especially, the increased cluster formation for I = 3 mmol/l starting between pH = 7–8 is reﬂected in an increased nearest neighbor peak in Fig. 2, and low-k-peak in Fig. 5 respectively, and in a border between suspended and clustered regime in Fig. 7. The repulsive structure for pH = 4 and I = 3 mmol/l can not be recognized in Fig. 2 and Fig. 5, but in a pronounced layer formation.

6 Dilute Suspensions To study the process of cluster growth we simulate dilute suspensions of Φ = 5% only. To see cluster-cluster aggregation we scale up the simulation volume to 17.76 µm3 containing 10560 MD particles and 1.3 · 107 ﬂuid particles. Due to the computational demands of the ﬂuid solver, a single simulation of 5 s real time requires about 5000 CPU hours on 32 CPUs of a state of the art supercomputer. We use a cluster counting algorithm [18] to detect clusters of particles in the suspension and to evaluate their size. In Fig. 8 we present the time dependence of the mean cluster size (a) and of the number of clusters in the system (b). We ﬁnd that both observables can be ﬁtted by a power law of the form A · (t + B)C , where A, B, C are ﬁtting parameters. The lines in the ﬁgure correspond to the ﬁt and the symbols to the simulation data. It would

Structural Transitions in Colloidal Suspensions

61

Fig. 8. The time dependence of the mean cluster size is plotted for diﬀerent simulation parameters (a). Fig. b) depicts the time dependence of the number of clusters found in the system. Each curve is shifted vertically by a factor of two for better visibility. While the symbols correspond to simulation data, lines are given by a power law ﬁt (source: [18])

be of great interest to investigate if a general scaling behavior can be observed depending on the volume concentration, the ionic strength and the pH-value. However, for this a detailed investigation of the parameter space would be needed which will be the focus of a future work.

7 Summary We have utilized our new parallel simulation code to model colloidal particles in shear ﬂow and investigated how the clustering process due to attractive DLVO potentials is aﬀected by the hydrodynamic forces. We ﬁnd a consistent behavior on diﬀerent length scales. The nearest neighbor peak of the pair correlation function has been used to observe the direct neighborhood of the particles and the low-k-peak of the structure factor to keep track of the length scales up to the system size. In both cases a suppression of the cluster formation by the shear ﬂow can be seen at low pH-values. For large pH-values low shear rates even support the clustering process. In contrast, for high shear rates it suppresses the cluster formation. We have evaluated the mean squared displacement to characterize the repulsive regime. To summarize the results we have drawn the stability diagram as given in Fig. 7. To our knowledge this stability diagram for Al2 O3 suspensions is reproduced quantitatively for the ﬁrst time from simulations. It helps to predict the behavior of a real suspension. Our ﬁndings on the cluster formation process suggest that soft stirring can enhance the cluster formation in industrial processing of this material. Further investigations can be carried out on the fractal dimension and its dependence on the experimental conditions. The low-k-peak of the structure factor can be used for that. We have followed up this point by simulating dilute suspensions for selected conditions. We have found that

62

M. Hecht, J. Harting

the time dependence of the mean cluster size and the number of clusters in the system can be well described by power laws. Acknowledgements We thank H.J. Herrmann for valuable collaboration and his support. The High Performance Computing Center Stuttgart, the Scientiﬁc Supercomputing Center Karlsruhe and the Neumann Institute for Computing in Jülich are highly acknowledged for providing the computing time and the technical support needed for our research. This work was partly ﬁnanced by the German Research Foundation (DFG) within the project DFG-FOR 371 “Peloide”.

References 1. B. J. Ackerson, C. G. DeKruif, N. J. Wagner, and W. B. Russel. Comparison of small shear ﬂow rate-small wave vector static structure factor data with theory. J. Chem. Phys., 90(6):3250, 1989. 2. B. J. Ackerson and P. N. Pusey. Shear-induced order in suspensions of hard spheres. Phys. Rev. Lett., 61(8):1033, 1988. 3. S. Alexander, P. M. Chaikin, P. Grant, G. J. Morales, P. Pincus, and D. Hone. Charge renormalization, osmotic pressure, and bulk modulus of colloidal crystals: Theory. J. Chem. Phys., 80(11):5776–81, 1984. 4. I. Ali, D. Marenduzzo, and J. M. Yeomans. Dynamics of polymer packaging. J. Chem. Phys., 121:8635–8641, Nov. 2004. 5. M. P. Allen and D. J. Tildesley. Computer simulation of liquids. Oxford Science Publications. Clarendon Press, Oxford, 1987. 6. J. F. Brady. The rheological behavior of concentrated colloidal suspensions. J. Chem. Phys., 99(1):567–81, 1993. 7. J. F. Brady and G. Bossis. Stokesian dynamics. Ann. Rev. Fluid Mech., 20:111– 57, 1988. 8. A. A. Catherall, J. R. Melrose, and R. C. Ball. Shear thickening and order– disorder eﬀects in concentrated colloids at high shear rates. Journal of Rheology, 44(1):1–25, 2000. 9. I. Cohen, T. G. Mason, and D. A. Weitz. Shear-induced conﬁgurations of conﬁned colloidal suspensions. Phys. Rev. Lett., 93(4):046001, 2004. 10. A. de Candia, E. del Gado, A. Fierro, N. Sator, and A. Coniglio. Colloidal gelation, percolation and structural arrest. Physica A, 358:239–248, 2005. 11. B. V. Derjaguin and L. D. Landau. Theory of the stability of strongly charged lyophobic sols and of the adhesion of strongly charged particles in solutions of electrolytes. Acta Phsicochimica USSR, 14:633, 1941. 12. J. Dobnikar, Y. Chen, R. Rzehak, and H. H. von Grünberg. Many-body interactions in colloidal suspensions. J. Phys.: Condens. Matter, 15:S263, 2003. 13. J. Dobnikar, R. Rzehak, and H. H. von Grünberg. Eﬀect of many-body interactions on the solid-liquid phase behavior of charge-stabilized colloidal suspensions. Europhys. Lett., 61(5):695–701, 2003.

Structural Transitions in Colloidal Suspensions

63

14. E. Falck, J. M. Lahtinen, I. Vattulainen, and T. Ala-Nissila. Inﬂuence of hydrodynamics on many-particle diﬀusion in 2d colloidal suspensions. Eur. Phys. J. E, 13:267–275, 2004. 15. M. J. Grimson and M. Silbert. A self-consistent theory of the eﬀective interactions in charge-stabilized colloidal dispersions. Macromol. Phys., 74(2):397–404, 1991. 16. L. Harnau and S. Dietrich. Depletion potential in colloidal mixtures of hard spheres and paltelets. Phys. Rev. E, 69:051501, 2004. 17. M. Hecht, J. Harting, M. Bier, J. Reinshagen, and H. J. Herrmann. Shear viscosity of clay-like colloids in computer simulations and experiments. Phys. Rev. E, 74:021403, 2006. 18. M. Hecht, J. Harting, and H. J. Herrmann. Formation and growth of clusters in colloidal suspensions. Int. J. Mod. Phys. C, 2007. in print. 19. M. Hecht, J. Harting, and H. J. Herrmann. A stability diagram for dense suspensions of model colloidal al2o3-particles in shear ﬂow. arXiv:cond-mat/0606455, 2006. Accepted for publication in Phys. Rev. E. 20. M. Hecht, J. Harting, T. Ihle, and H. J. Herrmann. Simulation of claylike colloids. Phys. Rev. E, 72:011408, jul 2005. 21. M. Hütter. Brownian Dynamics Simulation of Stable and of Coagulating Colloids in Aqueous Suspension. PhD thesis, Swiss Federal Institute of Technology Zurich, 1999. 22. M. Hütter. Local structure evolution in particle network formation studied by brownian dynamics simulation. J. Colloid Interface Sci., 231:337–350, 2000. 23. A.-P. Hynninen, M. Dijkstra, and R. van Roij. Eﬀect of three-body interactions on the phase behavior of charge-stabilized colloidal suspensions. Phys. Rev. E, 69:061407, 2004. 24. T. Ihle and D. M. Kroll. Stochastic rotation dynamics I: Formalism, galilean invariance, green-kubo relations. Phys. Rev. E, 67(6):066705, 2003. 25. T. Ihle and D. M. Kroll. Stochastic rotation dynamics II: Transport coeﬃcients, numerics, long time tails. Phys. Rev. E, 67(6):066706, 2003. 26. Y. Inoue, Y. Chen, and H. Ohashi. Development of a simulation model for solid objects suspended in a ﬂuctuating ﬂuid. J. Stat. Phys., 107(1):85–100, 2002. 27. A. Komnik, J. Harting, and H. J. Herrmann. Transport phenomena and structuring in shear ﬂow of suspensions near solid walls. Journal of Statistical Mechanics: theory and experiment, P12003, 2004. 28. A. J. C. Ladd. Numerical simulations of particulate suspensions via a discretized boltzmann equation. part 1. theoretical foundation. J. Fluid Mech., 271:285–309, 1994. 29. A. J. C. Ladd. Numerical simulations of particulate suspensions via a discretized boltzmann equation. part 2. numerical results. J. Fluid Mech., 271:311–339, 1994. 30. A. J. C. Ladd and R. Verberg. Lattice-boltzmann simulations of particle-ﬂuid suspensions. J. Stat. Phys., 104(5):1191, 2001. 31. M. Lattuada, H. Wu, and M. Morbidelli. Estimation of fractal dimension of colloidal gels in the presence of multiple scattering. Phys. Rev. E, 64:061404, 2001. 32. Y. Levin, T. Trizac, and L. Bocquet. On the ﬂuid-ﬂuid phase separation in charged-stabilized colloidal suspensions. J. Phys.: Condens. Matter, 15:S3523, 2003.

64

M. Hecht, J. Harting

33. A. Malevanets and R. Kapral. Mesoscopic model for solvent dynamics. J. Chem. Phys., 110:8605, 1999. 34. A. Malevanets and R. Kapral. Solute molecular dynamics in a mesoscale solvent. J. Chem. Phys., 112:7260, 2000. 35. D. W. McCarthy, J. E. Mark, and D. W. Schaefer. Synnthesis, structure, and properdies of hybrid organic-inorganic composites based on polysiloxanes. i. poly(dimathylsiloxane) elastomers containing silica. J. Polym. Sci. B, 36(7):1167, 1998. 36. J. R. Melrose and D. M. Heyes. Simulations of electrorheological and particle mixture suspensions: Agglomerate and layer structures. J. Chem. Phys., 98(7):5873–5886, 1993. 37. R. Oberacker, J. Reinshagen, H. von Both, and M. J. Hoﬀmann. Ceramic slurries with bimodal particle size distributions: Rheology, suspension structure and behaviour during pressure ﬁltration. In N. C. S. Hirano, G.L. Messing, editor, Ceramic Processing Science VI, volume 112, pages 179–184. American Ceramic Society, Westerville, OH (USA), 2001. ISBN 1574981048. 38. J. T. Padding and A. A. Louis. Hydrodynamic and brownian ﬂuctuations in sedimenting suspensions. Phys. Rev. Lett., 93:220601, 2004. 39. J. T. Padding and A. A. Louis. Hydrodynamic interactions and brownian forces in colloidal suspensions: Coarse-graining over time and length-scales. Phys. Rev. E, 74:031402, 2006. 40. T. Palberg, W. Mönch, F. Bitzer, R. Piazza, and T. Bellini. Freezing transition for colloids with adjustable charge: A test of charge renormalization. Phys. Rev. Lett., 74:4555, 1995. 41. T. N. Phung, J. F. Brady, and G. Bossis. Stokesian dynamics simulation of brownian suspensions. J. Fluid Mech., 313:181–207, 1996. 42. M. Ripoll, K. Mussawisade, R. G. Winkler, and G. Gompper. Low-reynoldsnumber hydrodynamics of complex ﬂuids by multi-particle-collision dynamics. Europhys. Lett., 68:106–12, 2004. 43. M. Ripoll, K. Mussawisade, R. G. Winkler, and G. Gompper. Dynamic regimes of ﬂuids simulated by multiparticle-collision dynamics. Phys. Rev. E, 72:016701, 2005. 44. M. Ripoll, R. G. Winkler, and G. Gompper. Star polymers in shear ﬂow. Phys. Rev. Lett., 96:188302, 2006. 45. A. Sierou and J. F. Brady. Accelerated stokesian dynamics simulations. J. Fluid Mech., 448:115, 2001. 46. A. Sierou and J. F. Brady. Shear-induced self-diﬀusion in non-colloidal suspensions. J. Fluid Mech., 506:285, 2004. 47. L. E. Silbert, J. R. Melrose, and R. C. Ball. Colloidal microdynamics: Pairdrag simulations of model-concentrated aggregated systems. Phys. Rev. E, 56(6):7067–7077, December 1997. 48. K. G. Soga, J. R. Melrose, and R. C. Ball. Continuum percolation and depletion ﬂocculation. J. Chem. Phys., 108(14):6026–6032, 1998. 49. K. G. Soga, J. R. Melrose, and R. C. Ball. Metastable states and the kinetics of colloid phase separation. J. Chem. Phys., 110(4):2280–2288, 1999. 50. V. Trappe, V. Prasad, L. Cipelletti, P. N. Segre, and D. A. Weitz. Jamming phase diagram for attractive particles. Nature, 411(3):772–774, 2001. 51. R. van Roij and J.-P. Hansen. Van der waals-like instability in suspensions of mutually repelling charged colloids. Phys. Rev. Lett., 79(16):3082–85, 1997.

Structural Transitions in Colloidal Suspensions

65

52. E. J. W. Vervey and J. T. G. Overbeek. Theory of the Stability of Lyophobic Colloids. Elsevier, Amsterdam, 1948. 53. R. G. Winkler, K. Mussawisade, M. Ripoll, and G. Gompper. Rod-like colloids and polymers in shear ﬂow: a multi-particle-collision dynamics study. J. Phys.: Condens. Matter, 16(38):S3941–54, 2004. 54. R. Yamamoto, K. Kim, Y. Nakayama, K. Miyazaki, and D. R. Reichman. On the role of hydrodynamic interactions in colloidal gelation. arXiv:condmat/0604404, 2006.

Simulations of Strongly Correlated Quantum Systems out of Equilibrium S.R. Manmana1,2,3 , K. Rodriguez1 , S. Wessel1 , and A. Muramatsu1 1

2 3

Institut f¨ ur Theoretische Physik III, Universit¨ at Stuttgart, Pfaﬀenwaldring 57, D-70550 Stuttgart, Germany Fachbereich Physik, Philipps Universit¨ at Marburg, D-35032 Marburg, Germany ´ Institute of Theoretical Physics, Ecole Polytechnique F´ed´erale de Lausanne, CH-1015 Lausanne, Switzerland

Summary. We use the recently developed adaptive time-dependent density matrix renormalization group method as well as advanced quantum Monte Carlo simulations to analyze the properties of strongly correlated quantum systems out of equilibrium. In particular, we consider the evolution of one-dimensional spinless fermions, after a quantum quench from the metallic into the insulating interaction regime. We ﬁnd that following the quench the systems relaxes after a characteristic time-scale, set by its kinetic energy, to a non-thermal state. We show, how the system’s properties in this quasi-stationary regime can be characterized based upon appropriately generalized Gibbs ensembles. Furthermore, we consider the emergence of coherence with bosons starting from a Mott-insulator, that lead to an alternative concept for an atom-laser.

1 Introduction Recently, interesting experimental results were obtained for quantum manybody systems far from equilibrium, realized with ultra-cold atom gases on optical lattices. Examples are the collapse and revival of a Bose-Einstein condensate (BEC) [1], the realization of a quantum version of Newton’s cradle [2], and the quenching of a ferromagnetic spinor BEC [3], where the time evolution of a quantum system after a sudden change of a control parameter can be accurately monitored. While these systems have no significant energy exchange with a heat bath, they consist of a large number of interacting degrees of freedom. Applications of the ergodic hypothesis then suggests that after sufficiently long times, the time averages of observables might become equal to thermal averages [4, 5, 6]. However, experiments on one-dimensional strongly interacting bosons did not exhibit such thermalization, a fact that was ascribed to the integrability of this system [2]. Subsequent theoretical works found that one-dimensional integrable bosonic systems (hard-core bosons on a lattice and Luttinger liquids) show relaxation to states that can be described

72

S.R. Manmana et al.

by a generalized Gibbs ensemble, which accounts for the full set of constants of motion [7, 8]. Here, we present results from numerical simulations of such quench situations for both integrable and non-integrable systems of spinless fermions. Furthermore, based on a recent theoretical finding made in our group [9], we study the conditions for an optimal operation of an atom-laser. The exact numerical simulations performed with large systems (103 –104 sites) with a large number of particles (∼ 300–500 particles) have shown that starting with a state, where particles are essentially localized on the sites of an underlying lattice (Mott-insulator), maximal coherence is reached during the free expansion short after their release. The original numerical study was performed in one dimension and with bosons with an infinitely strong repulsion (hard-core bosons or Tonks-Girardeau gas). In the present work we present results for a finite strength of the repulsion. In the following, we present results from large-scale numerical studies of out-of-equilibrium properties of strongly correlated quantum many-body systems. Our research has become possible due to important recent progress on calculating the time evolution of strongly correlated quantum many body systems via the adaptive time-dependent density matrix renormalization group method (t-DMRG) [10, 11, 12, 13, 14]. Some of us have previously developed such methods [14], and here we apply it to the above mentioned physical situations. The rest of this report is organized as follows: In the next section, we provide a basic description of the t-DMRG method. Then, we discuss results on the time-evolution of spinless fermion systems after a quantum quench [15] in Sec. 3 and the physics of the atom-laser in Sec. 4. Finally, we give a summary in Sec. 5. In addition to the work detailed below, our research in the last year focused also on the following topics: (i) emerging phases and quantum phase transitions of ultra-cold atom gases on frustrated optical lattices [16, 17, 18], (ii) the eﬀect of randomness on ultra-cold atom systems and quantum magnets [19, 20, 21, 22, 23], (iii) supersolid phases of ultra-cold fermion [24], (iv) weakly coupled quantum spin chains [25]. Here, we will however focus on our non-equilibrium studies. We start by reviewing the t-DMRG method.

2 Adaptive Time-Dependent DMRG Method The basic idea of the density-matrix renormalization group method is to represent one or more pure states of a finite system approximately by dividing the system in two and retaining only the m most highly weighted eigenstates of the reduced density matrix of the partial system. In combination with the numerical renormalization group approach (NRG) developed by Wilson [26] and the superblock algorithms developed by White and Noack [27], this leads to a very powerful and eﬃcient tool for the investigation of one-dimensional

Simulations of Strongly Correlated Quantum Systems out of Equilibrium

73

strongly correlated quantum systems on a lattice. We refer to recent reviews, e.g., Refs. [28, 29, 30], for a detailed description. The main diﬃculty in calculating the time evolution using the DMRG is that the restricted basis determined at the beginning of the time evolution is not able, in general, to represent the state well at later times [12] because it covers a subspace of the total Hilbert space which is not appropriate to properly represent the state at the next time step. Since both the Hamiltonian and the wave function | ψ(τ ) at time τ are represented in an incomplete basis, the result for the next time step | ψ(τ + Δτ ) will have additional errors because the reduced basis is not an optimum representation for this state. In order to minimize these errors, it is necessary to form a density matrix whose m most important eigenvectors are “optimal” for the representation of the state | ψ(τ ) , as well as for | ψ(τ + Δτ ) in the reduced Hilbert space. The most straightforward approach is to mix all time steps | ψ(τi ) into the density matrix [12, 13]. However, this can be extremely costly computationally. A more eﬃcient way is to adapt the density matrix at each time step. An approach for adaptive time evolution based on the Trotter-Suzuki [31] decomposition of the time-evolution operator was developed in Refs. [10, 11, 32]. The idea is to split up the time-evolution operator in local time-evolution operators Ul acting only on the bond l. For lattice Hamiltonians containing only terms connecting nearest-neighbor sites, this is easily obtained using the Trotter-Suzuki decomposition, which in second order is given by e−iΔτ H ≈ e−iΔτ Heven /2 e−iΔτ Hodd e−iΔτ Heven /2 .

(1)

Here Heven and Hodd is the part of the Hamiltonian containing terms on even and odd bonds, respectively. Since each bond term Hl within Heven or Hodd commutes, e−iΔτ H can then be factorized into terms acting on individual bonds. In the DMRG procedure usually two sites are treated exactly, i.e., the entire Hilbert space of the two sites is included. The Trotter variant of the t-DMRG exploits this feature by applying Ul = e−iΔτ Hl at the bond given by the two “exact” sites. In this way, the time-evolution operator has no further approximations other than the error introduced by the Trotter decomposition. In particular, the error introduced by the cutoﬀ is avoided. The wave function of the lattice is then updated by performing one complete sweep over the lattice and applying Ul at the “dividing bond”. In this way, only one wave function must be retained and it is possible to work with the density matrix for a pure state. However, the method is restricted to systems with local or nearest-neighbor terms in the Hamiltonian. A more general basis adaption scheme aims at adapting the density matrix basis by approximating the density matrix for a time interval [33], ρΔτ =

τ +Δτ τ

| ψ(τ ′ ) ψ(τ ′ ) | dτ ′ .

74

S.R. Manmana et al.

The integral is approximated by adding a few intermediate time steps within the time interval [τ, τ + Δτ ]. In Ref. [33], the intermediate time steps are obtained using a Runge-Kutta integration scheme and using 4 to 10 intermediate time steps. Here we instead obtain the intermediate time steps using a Krylovspace approximation for the time-evolution operator [13, 14, 34]. This can be done easily because the Hamiltonian of the system is usually constructed anyway in the DMRG scheme. Within the restricted basis, the Lanczos iteration can then be performed, leading to the desired intermediate time steps. With this approach, it is, in principle, possible to treat more general Hamiltonians, as long as they can be treated accurately using the DMRG. For the systems investigated by us, both variants of the t-DMRG have been used.

3 Spinless Fermions Following a Quantum Quench In the following, we consider a system of spinless fermions on a finite onedimensional lattice, described by the Hamiltonian † cj+1 cj + h.c. + V nj nj+1 , (2) H = −th j

j

with a nearest-neighbor hopping amplitude th and a nearest-neighbor interac(†) tion strength V at half-filling. The ci annihilate (create) fermions on lattice site i, ni = c†i ci , and we take = 1. We measure energies in units of th , and, accordingly, time. The model is integrable via the Bethe ansatz [35]. For weak interactions, V < Vc = 2 the equilibrium state of the system is in a metallic Luttinger liquid (LL) phase, whereas for V > Vc it forms a charge-densitywave (CDW) insulator, with a quantum critical point located at V = Vc . In the following, we consider open chains of up to L = 100 sites pushed out of equilibrium by suddenly quenching the strength of V from an initial value V (t = 0) = V0 to a diﬀerent value V (t > 0) = V . The state of the system at t = 0 was taken as the ground state of the system for V0 . We also studied the eﬀect of adding a next-nearest-neighbor repulsion V2 j nj nj+2 to the model, which makes it non-integrable. To characterize the system, we measure the momentum distribution function (MDF) nk (t) =

L 1 ik(l−m) † e cl cm (t), L

(3)

l,m=1

i.e., the Fourier transform of the one-particle density matrix, ρlm = c†l cm . This observable thus contains global information about the state of the system based on correlations over all length scales. In Fig. 1, we show results for the time evolution of the MDF after a quench for two diﬀerent final values of the interaction strength: In Fig. 1a, we fix V = Vc = 2, and in Fig. 1a, V = 5. To illustrate the time evolution of

Simulations of Strongly Correlated Quantum Systems out of Equilibrium

75

Fig. 1. Time-averaged MDF for quenching (a) from V0 = 0.5 to V = 2 and (b) from V0 = 1.5 to V = 5 for L = 50 sites. The time averages of two independent initial states with the same energy are compared to each other and to the thermal expectation value. Insets: nπ vs. time t; the horizontal line is the ﬁnite T value

the system, we show in the insets of both panels the evolution of the MDF nk (t) at a specific value of k = π. Let us first discuss the case V = 2. In our calculations, we considered two diﬀerent initial conditions: V0 = 0.5, and V0 = 3.57. The precise value of V0 > Vc has been chosen such that the expectation value of the energy H is the same for both initial states. While the former value of V0 corresponds to the LL regime, the latter one corresponds to the CDW insulating regime. We find that in both cases the MDF relaxes after an initial dephasing time of order t−1 h , and then shows only small residual fluctuations. Taking the time average of the MDF inside this regime, we obtain the MDF shown in the main parts of Fig. 1a. We find, that although the initial states are very diﬀerent (LL or CDW, respectively), both relax to states with almost identical MDFs. One possibility is that the system relaxes to a thermal state, determined by the constant value of the energy H after the quench. In order to check for this possibility, we performed also quantum Monte Carlo (QMC) simulations [36] in order to obtain the MDF

76

S.R. Manmana et al.

for a thermal ensemble, with the temperature T fixed such as to reproduce the value of the energy H. A comparison to the finite temperature MDF shows that the quenched systems, while being close to a thermal state, clearly do not exhibit thermal equilibrium behavior. Turning to the case of Fig. 1b, we find very similar behavior, but with somewhat large deviations of the final MDFs to the corresponding thermal values. We find even larger deviations for a quench resulting in large values of |V0 − V |, such as shown in Fig. 2 for the case of V0 = 0.5, and V = 10. We find that the system never relaxed to a thermal state – however, still a quasi-stationary state results for long times, which exhibits only very little dependence on the initial state, as seen in Fig. 1. Turning on a finite value of V2 , we arrived at the same findings also for this non-integrable case [15]. Furthermore, our conclusions did also not change, when we considered larger system sizes [15]. In general, we thus find that such systems after a quench relax to non-thermal quasi-stationary states, which become very similar when diﬀerent initial configurations share the same energy. In order to characterize such quasi-stationary, non-thermal states, we propose to consider a generalized ensemble, in which the higher moments of the Hamiltonian H, which are constants of motion, are taken into account as constraints. Such an ensemble generalizes the unusual thermal density matrix ̺β , which is uniquely fixed by the single constraint Hβ = H. In a generalized Gibbs ensemble, the density matrix is determined by maximizing the entropy while accounting for all constraints on the system [37]. In terms of the operators On , which form the set of observables whose expectation values remain constant in time, the appropriate density matrix is [37] ̺ = exp − (4) λn On . n

Fig. 2. Time-averaged momentum distribution nk for V = 10. Inset: nπ vs. time t; the horizontal line is the ﬁnite T value

Simulations of Strongly Correlated Quantum Systems out of Equilibrium

77

In this expression, the parameters λn are fixed by the conditions that Tr (̺ On ) = On , were O0 = 1 relates to normalization. For any closed quantum system, all higher moments of the Hamiltonian can be used in the above expression, thus taking On = H n . Furthermore, for a finite system, dim(H) powers of H fully determine ̺ [15]. We find that taking successively more such constraints into account, the quasistationary properties of the system, such as the MDF are indeed better accounted for by the generalized Gibbs ensemble [15]. This is shown for a specific case with 16 sites in Fig. 3, where the energy distribution in the quasi-stationary state is compared to the energy distribution of generalized Gibbs ensembles with the number of constraints increasing from 1 to 3.

ˆ n on the energy distribution function P (E) Fig. 3. Inﬂuence of the constraints H

4 Atom Laser We concentrate here on the evolution out of equilibrium of a one-dimensional Bose gas with strong interactions such that its initial state is a Mott-insulator. It was previously shown that in the hard-core limit, an initial Fock state develops quasi-long range correlations, when the bosons evolve freely on a lattice [9]. In particular, the momentum distribution function n(k) develops sharp peaks at momenta k = ±π/2a, where a is the lattice constant. However, the experimental investigation is hampered by quite stringent requirements for the realization of hard-core bosons.. We therefore consider here the case of finite interactions, modeled by the one-dimensional Hubbard model U † bi bi+1 + h.c. + ni (ni − 1) , (5) H = −t 2 i i

78

S.R. Manmana et al.

where b†i and bi are bosonic creation and annihilation operators, respectively, and ni = b†i bi is the density operator. The hard-core limit corresponds to U → ∞. The value at which the Mott-insulator appears was estimated as Uc /t ∼ 3.5 in one dimension and for a commensurate density n = 1 [38]. Hence, all the cases considered here correspond to U > Uc . As in the hard-core case, we start with bosons in a Mott-insulating state spread over several lattice sites and monitor the free expansion on a lattice. The time evolution of the system is obtained with t-DMRG, as discussed in Sec. 2. In the cases considered here we have Nb = 20 bosons and systems with length L = 60. Due to memory limitations, it is not possible to allow for all the possible occupations of a given site. In general the maximal number of bosons per site needed to have an accurate description of the system increases as U decreases. In all the cases treated here a maximum of three bosons per site was enough. Even at the lowest value studied here (U/t = 6), no appreciable diﬀerence was observed for a cutoﬀ of 3 and 4 bosons per site. Since in the course of the free expansion the system becomes more dilute, the limitation in the number of bosons per site is less serious at later times. The evolutions shown are all limited to times shorter than the one in which the matter wave reaches the boundary of the system. A detailed analysis for 6 ≤ U/t ≤ 40 is presented in Fig. 4. Figure 4(a) shows n(k) around k = π/2a with the data points from t-DMRG denoted by symbols with spline interpolations between them. There it is clearly seen that the maximum of n(k) is displaced to lower values as U decreases. At higher values of U (U = 40t), the results of the hard-core limit are recovered, while lowering the value of the interaction strength shifts the maximum of the momentum distribution function to lower values of k. The spline inter-

Fig. 4. n(k) at time τ = 4.5/t for diﬀerent values of U/t. The symbols corresponds to t-DMRG results on a lattice with L = 60, while the lines in the respective colors are spline interpolations

Simulations of Strongly Correlated Quantum Systems out of Equilibrium

79

polation allows for a better determination of the maxima occurring in n(k), since a denser set of k-points corresponds to having a much longer lattice in a physical realization. On the other hand, the actual set of k-points in the t-DMRG simulation corresponding to L = 60 is dense enough to allow for a smooth interpolation without introducing artefacts due to the spline procedure. Figure 4(b) shows the location of the maxima of n(k) as a function of U in units of 2a/π, giving a guide for a fine tuning of the wavelength of the matter wave by means of the interaction strength. The results in this section have shown that the main features found in hardcore bosons [9], namely the emergence of a coherent matter wave from a Mottinsulator, persist under more general conditions. Besides the determination of the wavelength of the matter wave by the underlying lattice, where the expansion takes place, a finer tuning is possible by regulating the interaction strength of the bosons, i.e. in the case of an optical lattice, not only by the wavelength of the corresponding laser beam but also by its intensity. A more extended account of the results reported in this section can be found in Ref. [39].

5 Summary We have reported on our recent numerical studies of out-of-equilibrium properties of strongly correlated quantum systems. From our analysis of interacting spinless fermions on one-dimensional lattices, we found that such systems, following a quantum quench, evolve towards non-thermal quasi-stationary states. A generalized Gibbs ensemble, taking into account the higher moments of the Hamiltonian as constraints, describes this quasi-stationary state. For the future, we plan a detailed analysis of evolution of local properties of the system following the quench, and the relevance of the excitation of the system for the relaxation process [40]. The simulation of the free expansion of strongly interacting bosons out of a Mott-insulator has shown that new control possibilities appear by tuning the interaction among the particles. Therefore, new perspectives open to produce in this way an atom-laser with tunable wavelength. Acknowledgements We wish to thank R.M. Noack and M. Rigol for their collaboration, and HLRS-Stuttgart (Project CorrSys) and NIC-J¨ ulich for the allocation of computer time. We acknowledge financial support by the DFG programs SFB 382 and SFB/TR 21, and by the Landesstiftung Baden-W¨ urttemberg (program Atomoptik).

80

S.R. Manmana et al.

References 1. M. Greiner, O. Mandel, T. W. H¨ansch, and I. Bloch, Nature 419, 51 (2002). 2. T. Kinoshita, T. Wenger, and D. S. Weiss, Nature 440, 900 (2006). 3. L. E. Sadler, J. M. Higbie, S. R. Leslie, M. Vengalattore, and D. M. StamperKurn, Nature 443, 312 (2006). 4. K. Sengupta, S. Powell, and S. Sachdev, Phys. Rev. A 69, 053616 (2004). 5. J. Berges, S. Borsanyi, and C. Wetterich, Phys. Rev. Lett. 93, 142002 (2004). 6. C. Kollath, A. L¨ auchli, and E. Altman, Phys. Rev. Lett. 98, 180601 (2007). 7. M. Rigol, V. Dunjko, V. Yurovsky, and M. Olshanii, Phys. Rev. Lett. 98, 050405 (2007). 8. M. A. Cazalilla, Phys. Rev. Lett. 97, 156403 (2006). 9. M. Rigol and A. Muramatsu, Phys. Rev. Lett. 93, 230404 (2004). 10. A. J. Daley, C. Kollath, U. Schollw¨ ock, and G. Vidal, J. Stat. Mech.: Theor. Exp., P04005 (2004). 11. S. R. White and A. E. Feiguin, Phys. Rev. Lett. 93, 076401 (2004). 12. H. G. Luo, T. Xiang, and X. Q. Wang, Phys. Rev. Lett. 91, 049701 (2003). 13. P. Schmitteckert, Phys. Rev. B 70, 121302(R) (2004). 14. S. R. Manmana, A. Muramatsu, and R. M. Noack, AIP Conf. Proc. 789, 269 (2005). 15. S.R. Manmana, S. Wessel, R.M. Noack, and A. Muramatsu, Phys. Rev. Lett. 98, 210405 (2007). 16. S. Wessel and M. Troyer, Phys. Rev. Lett. 95, 127205 (2005). 17. S. V. Isakov, S. Wessel, R. G. Melko, K. Sengupta, and Yong Baek Kim, Phys. Rev. Lett. 97, 147202 (2006). 18. S. Wessel, Phys. Rev. B 75, 174301 (2007). 19. A. Jagannathan, A. Szallas, S. Wessel, and M. Duneau, Phys. Rev. B 75, 212407 (2007). 20. A. Jagannathan, R. Moessner, and S. Wessel, Phys. Rev. B 71, 184410 (2006). 21. O. Gygi, H. G. Katzgraber, M. Troyer, S. Wessel, and G. G. Batrouni, Phys. Rev. A 73, 063606 (2006). 22. H. Gimperlein, S. Wessel, J. Schmiedmayer, and L. Santos, Appl. Phys. B 82, 217 (2006). 23. N. Laﬂorencie, S. Wessel, A. L¨ auchli, and H. Rieger, Phys. Rev. B 73, 060403(R) (2006). 24. F. Karim Pour, M. Rigol, S. Wessel, and A. Muramatsu, Phys. Rev. B bf 75, 161104(R) (2007). 25. B. Pedrini, J. L. Gavilano, H. R. Ott, S. M. Kazakov, J. Karpinski and S. Wessel, Eur. Phys. J. B 55, 219 (2007). 26. K. G. Wilson, Rev. Mod. Phys. 47, 773 (1975). 27. S. R. White and R. M. Noack, Phys. Rev. Lett. 68, 3487 (1992). 28. R. M. Noack and S. R. Manmana, AIP Conf. Proc. 789, 93 (2005); also available as cond-mat/0510321. 29. I. Peschel, X. Wang, M. Kaulke, and K. Hallberg (editors), Density Matrix Renormalization - A New Numerical Method in Physics (Springer Verlag, Berlin, 1999). 30. U. Schollw¨ ock, Rev. Mod. Phys. 77, 259 (2005). 31. M. Suzuki, Prog. Theor. Phys. 56, 1454 (1976). 32. G. Vidal, Phys. Rev. Lett. 93, 040502 (2004).

Simulations of Strongly Correlated Quantum Systems out of Equilibrium 33. 34. 35. 36. 37.

81

A. E. Feiguin and S. R. White, Phys. Rev. B 72, 020404(R) (2005). M. Hochbruck and C. Lubich, SIAM J. Numer. Anal. 34, 1911 (1997). C. N. Yang and C. P. Yang, Phys. Rev. 150, 321 (1966). A. W. Sandvik, Phys. Rev. B 59, R14157 (1999). R. Balian, From Microphysics to Macrophysics: Methods and Applications of Statistical Physics, Texts and Monographs in Physics, Springer, 1991. 38. T. K¨ uhner, S. White, and H. Monien, Phys. Rev. B 61, 12474 (2000). 39. K. Rodriguez, S.R. Manmana, M. Rigol, R.M. Noack, and A. Muramatsu, New J. Phys. 8. 169 (2006). 40. S. R. Manmana, S. Wessel, R. M. Noack, and A. Muramatsu, in preparation.

Computer Simulations of Soft Matterand Nano-Systems K. Franzrahe, J. Neder, M. Dreher, P. Henseler, W. Quester, C. Schieback, F. B¨ urzle, D. Mutter, M. Schach, T. Sorg, and P. Nielaba Physics Department, University of Konstanz, 78457 Konstanz, Germany [email protected] Summary. Soft matter systems have been investigated by Monte Carlo and Brownian Dynamics simulations. In particular the behaviour of two dimensional binary hard disk mixtures in external periodic potentials has been studied as well as the transport of colloids in micro-channels and the features of proteins in lipid bilayers. Ni nanocontacts have been analyzed by Molecular Dynamics simulations with respect to their conductance and structural properties under stretching, and the eﬀect of temperature, composition and system size on the structural properties of Nix Fe1−x alloys has been studied. The properties of Si clusters in external ﬁelds have been computed by density functional methods, and the static and dynamic properties of model magnetic systems by the Landau-Lifshitz-Gilbert equation. In the next sections we give an overview on our recent results.

1 Two-Dimensional Colloidal Systems in Periodic External Fields In monolayers on crystalline surfaces one can observe an intricate competition between eﬀects due to the interaction of components within the layer and those with the underlying substrate. Such complicated experimental systems can be modelled by two-dimensional colloidal systems. The interactions within the monolayer can be altered by changing the interaction potential of the colloids, while the shape and strength of the substrate potential can be modelled by external light fields. The advantage of the model system is, that via laser scanning microscopy direct access to the particle configurations is given. In this way it is possible to gain insight in the relative importance of the various possible physical processes that occur. From the theoretical point of view, even the relatively simple combination of a monodisperse system in a one-dimensional, spatially periodic light field shows a highly non-trivial phase behaviour as the amplitude of the external field is raised: Laser Induced Freezing (LIF) and Laser Induced Melting (LIM).

84

K. Franzrahe et al.

Both phase transition could be observed in experiments: LIF first by A. Chowdhury et. al. [1] and LIM by Wei et. al. [2]. With the help of extensive Monte Carlo Computer Simulations the complete phase diagram was calculated by W. Strepp et. al. [3, 4, 5, 6, 7]. The choice of the wavelength of the periodic light field, i.e. the commensurability ratio determines what structural phases can possibly occur [8, 9]. As monolayers are often not necessarily monodisperse, one question to tackle is: what happens if we expose a binary mixture to an external light field? In order to determine the influence of the size and mixing ratio of the components on the phase behaviour in comparison to the monodisperse case as directly as possible a two-dimensional binary hard disk mixture was analyzed [10, 11]. Defining the binary mixture that is to be studied, we set the mixing ratio to xA = xB = 50% and the diameter ratio to dB /dA = 0.414. The external potential is spatially periodic in the x-direction with V (r) = V0 sin (K · r) The wave vector is K = 4π a (1, 0), where a is the lattice constant of the square lattice. The corresponding wavelength of the external field was chosen to be commensurate to the square lattice, which yields the highest packing fraction for the given mixture. The commensurability ratio, i.e. the ratio of the wave vector to the corresponding parallel reciprocal lattice vector is therefore p = 2. The potential landscape is schematically depicted in Fig. 1. Simulations in the N V T -Ensemble were carried out. In order to facilitate equilibration a cluster move by L. Lue and L.V. Woodcock [12] and additional trough moves were used, besides the standard Metropolis algorithm. Periodic boundary conditions were employed in all simulations.

Fig. 1. Schematic illustration of the external light ﬁeld V (r)

Computer Simulations of Soft Matter- and Nano-Systems

85

In these studies one has to distinguish three cases: (I) only the smaller component interacts with the external field (II) both components interact with the external field (III) only the larger component interacts with the external field 1.1 Laser Induced Freezing Scenarios A comparison of the LIF scenarios for case I and II shows that the underlying ordering mechanisms in the two cases are very diﬀerent. At low amplitudes of the external light field we observe a laser induced de-mixing, i.e. a coexistence of a monodisperse lattice of the larger component with a small component enriched binary fluid. This de-mixing does not occur in the field free case. Figure 2 shows a snapshot of the simulation for case I at V0 = 0.6 and an overlay of all configurations of the simulation (the movement of the centre of mass was subtracted), where black points denote the positions of the larger components and red points those of the smaller components. A droplet of a monodisperse triangular lattice with a random orientation in a binary fluid is formed as the smaller component tries to align itself with the minima of the external field. In case II, where both components interact with the external field, we also observe a laser induced phase separation. At even lower amplitudes of the external light field a monodisperse rhombic lattice coexists with a binary fluid. As can be seen in Fig. 3 only two distinct orientations of the monodisperse rhombic lattice are possible. This is due to the fact, that the phase separation is driven by the attempt of the larger component itself to minimize the energy of the system by aligning with the external field. As the wavelength of the external field is incommensurate to a monodisperse triangular lattice, the rhombic lattice forms instead. At higher field amplitudes laser induced freezing into the commensurate square lattice occurs. If only the smaller component interacts with the external field (case I) the square lattice coexists with a 50% binary fluid for V0 ≥ 1.7

Fig. 2. Case I: a) snapshot at V0 = 0.6 and ̺∗ = 1.71; b) overlay of all conﬁgurations (black : positions of large component; red : small component)

86

K. Franzrahe et al.

(see Fig. 4). The coexisting square lattice is nearly defect free. In contrast to this we observe even at low amplitudes V0 at ̺∗ = 1.71 a strong competition of square lattice and rhombic domains in case II, where both components interact with the field. The existence of competing domains leads to the formation of defects and grain boundaries in the laser induced freezing of the system into the square lattice (see Fig. 5). We observe no phase coexistence in this case, but a gradual healing of the defects as the amplitude of the external field is raised.

Fig. 3. Case II: a) snapshot at V0 = 0.3 and ̺∗ = 1.71; b) overlay of all conﬁgurations (black : positions of large component; red : small component)

Fig. 4. Case I: a) snapshot at V0 = 2.7 and ̺∗ = 1.71; b) overlay of all conﬁgurations (black : positions of large component; red : small component)

Fig. 5. Case II: a) snapshot at V0 = 2.7 and ̺∗ = 1.71; b) overlay of all conﬁgurations (black : positions of large component; red : small component)

Computer Simulations of Soft Matter- and Nano-Systems

87

1.2 Melting in the Presence of the External Field In the monodisperse LIM scenario melting at high external field amplitudes occurs via a decoupling of the particle fluctuations in adjacent potential minima. This melting scenario is geometrically blocked in the analyzed binary mixture due to the chosen combination of diameter ratio and wavelength of the external field. Instead we observe a decoupled melting of the sub-lattices. For the case when both components interact with the external light field, we took a commensurate path through phase space, by keeping the potential amplitude constant and lowering the overall density of the system. We observe a melting of the sub-lattice of the smaller component perpendicular to the potential minima at intermediate densities, while the sub-lattice of the larger component persists. Only at low densities do we observe a modulated binary fluid. The melting of the sub-lattice of the smaller component occurs via trough hopping, as can be seen in the two dimensional pair correlation functions in Fig. 6. At ̺∗ = 1.79 the pair correlation function of the smaller components gBB (r) shows the signatures of the square lattice: four-fold rotational symmetry and only every second minimum of the external field shows a peak (which is elongated along the y-direction due to the presence of the external field (V0 = 5.0)). In the hopping regime at ̺∗ = 1.69 gBB (r) exhibits peaks in all minima, while gAA (r) still shows the characteristics of the square lattice, with strong fluctuations along the potential minima.

Fig. 6. Case II at V0 = 5.0: a) two dimensional pair correlation function for the sub-lattice of the smaller component at ̺∗ = 1.79; b) the same at ̺∗ = 1.69; c) two dimensional pair correlation function for the sub-lattice of the larger component at ̺∗ = 1.69

2 Transport of Colloids in Micro-Channels We conducted Brownian dynamics (BD) simulations of a two-dimensional microchannel [13] setup in order to investigate the flow behavior of the colloidal particles within the channel systematically for various parameter values of

88

K. Franzrahe et al.

constant driving force, overall particle density, and channel width. The pair interaction V (r) = (μ0 /4π)M 2 /r3 (M is the dipole moment) is purely repulsive and can be characterized by the dimensionless interaction strength Γ = μ0 M 2 ρ2/3 /(4πkB T ). Particles are confined to the channel by hard walls in y-direction and at x = 0 (channel entrance). These walls are realized both as ideal elastic hard walls and as proposed in [14], where a particle crossing the wall is moved back along the line perpendicular to the wall until contact. Both realizations result in the same flow behavior. Also we performed simulations with the particles at the wall kept fixed. The channel end is realized as an open boundary. To keep the overall number density in the channel fixed, every time a particle leaves the end of the channel a new particle is inserted at a random position (avoiding particle overlaps) within the first 10% of the channel, acting as a reservoir. A cutoﬀ of 10σ was used along with a Verlet next neighbor list [15]. Checks of particle overlaps are included in the simulation, but for all ordered systems we never found two overlapping particles. A typical snapshot from the experiment [13] of the particles moving along the channel is shown in Fig. 7(a). Similar snapshots we get from simulations [13] with co-moving (Fig. 7(b)) and fixed boundary particles (Fig. 7(c)), i.e., the velocity is kept to zero for the particles at the channel wall. In most regions of the channel the particles are placed in a quasi-crystalline order. This behavior is due to the strength of the particle interactions caused by the external magnetic field (high Γ -values), which leads to quasi-crystalline behavior in unbounded systems as well. The formation of this order naturally gives rise to the formation of layers in the motion of the particles along the channel. A similar layering phenomenon has been observed in channels under

Fig. 7. (a) Video microscopy snapshot of colloidal particles moving along the lithographically deﬁned channel. (b) Simulation snapshots for a channel (692 × 60 µm, Γ ≈ 2.5) with ideal hard walls (573.3 × 45 µm, Γ = 115), (c) the same as in (b) with the particles at the walls (marked green) kept ﬁxed (573.3 × 45 µm, Γ = 902). The blue rectangles mark the layer transition region

Computer Simulations of Soft Matter- and Nano-Systems

89

equilibrium condition [16]. Additionally to this layer formation we observe, both in experiment and in simulation, a decrease of the number of layers in the direction of motion. In between both regions therefore a region exists in which the particles cannot be well-ordered. This region is called the layerreduction zone. In Fig. 7 these regions have been marked. The reduction of the number of layers originates from a density gradient along the channel. The local particle density inside the channel is shown in Fig. 8(a) and (b) together with the particle separations in x- and y-directions. In the experiment (Fig. 8(a)) the density decreases monotonically along the direction of the motion of the particles by about 20%. The average density in the channel shows fluctuations on the order of 10% as a function of time. The total increase in density, however, is less than 3% during the total time of the experiment. We therefore argue that the density gradient is formed in a quasi-static situation. This argument is confirmed by results of BD simulations (Fig. 8(b)), where the corresponding decrease of the particle density is observed.

Fig. 8. Local lattice constants dx and dy and local particle density (a) in the experiment and (b) in the BD simulation. The results are obtained for the systems of Fig. 7(a) and (b) respectively. (c) Potential energies per particle of diﬀerent layer conﬁgurations as a function of the particle density. The dots mark the perfect triangular lattices for 5, 6 and 7 layers. Also shown are parts of the conﬁgurations with 7 and 6 layers at the intersection point. (d) Plots of the layer order parameter for the conﬁguration snapshot of Fig. 7(b)

90

K. Franzrahe et al.

3 Proteins in Lipid Bilayers Lipid bilayers and incorporated proteins form biological membranes. These barriers define the inside and the outside of a cell and are indispensable for live [17]. Usually the microscopic surface tension of membranes is small or vanishes altogether [18]. One aim of our work [19] is to study the eﬀect of an applied tension to a model bilayer. Does this tension cause a change in the behavior of incorporated model proteins, e.g. lead to an increasing lipid mediated attraction or repulsion between two proteins? In our lipid model [20] the molecules are represented by chains made of one head bead and six tail beads. Neighboring beads at a distance r within the molecule interact via a FENE potential and the angle θ between subsequent bonds in the lipid gives rise to a stiﬀness potential. Tail beads, which are not direct next neighbors, interact via a truncated and shifted Lennard-Jones potential. Head-head, head-tail, head-solvent and tail-solvent interactions are modelled by the repulsive part of the Lennard-Jones potential. The proteins in our system are modelled as cylinders and their axes are parallel to the z-axis of the unsheared system. Their interaction with each other, the head, the tail and the solvent beads is modelled by Lennard-Jones potentials, depending on the eﬀective distance in the xy-plane rxy − rp,ef f , potein-bead dxy = , rxy − 2rp,ef f , potein-protein where rp,ef f = rp − σp /2. For the tail beads an additional attractive potential Vpt (r) = U (dxy )W (z) is added. rxy and z are measured from the center C of the protein. Following [21], we put the tension into eﬀect by an additional energy term −γA to the Hamiltonian of the system, where A is the projected area of the bilayer onto the xy-plane. Thus, we are performing Monte Carlo simulations with the eﬀective Hamiltonian Hef f = H + P V − γA − N kB T ln(V /V0 ) . H is the interaction energy, V the volume of the simulation box, V0 an arbitrary reference volume and N the total number of beads (cf. [22]). We calculated the distribution function of the protein-protein distance r via umbrella sampling (cf., e.g., [23]). The range of interest was subdivided into about 20 smaller windows, which had to have overlapping regions with the neighboring windows. By assuming a constant g(r) for suﬃciently large r the overlap of the histograms for smaller distances can be matched to the value of the preceding histogram by multiplication of a suitably chosen factor. By taking the negative logarithm of g(r) one obtains w(r), which is proportional to the eﬀective pair potential. The diagram in Fig. (9) shows w(r) = − ln(g(r)) vs. r at an increasing tension γ on the bilayer. The temperature and the pressure were set to T = 1.3

Computer Simulations of Soft Matter- and Nano-Systems

91

Fig. 9. Eﬀective protein-protein pair potential for increasing γ

Fig. 10. Left: Snapshot of a double bilayer conﬁguration; Right: Tether consisting of 4,800 lipids

and P = 2.0, which corresponds to the fluid Lα phase of the bilayer in the tensionless state (cf. [24]). The system consisted of approximately 800 lipids, 6,000 solvent particles and 2 proteins. At the chosen parameter set with ǫpt = 1.0, σp = 1.0 the additional tension does not aﬀect the position of the maxima and minima in the eﬀective potential between the proteins, but leads to ”damped” oscillations in this function. Further investigations of how an increasing protein-tail interaction ǫpt in a bilayer under tension aﬀects the eﬀective potential between proteins are in progress. For one of the curves depicted approximately 2000 h of CPUtime on a Intel Xeon 2.8 GHz are required. Other points of interest in our project are the examination of membrane multi layers (10) and tethers (10) within and without an applied tension, respectively. These configurations require high computing capacities due to the system size. The shown tether, e.g., consists of 4800 lipids and more than 90,000 solvent particles. The program has been parallelized and technical details about that can be found in [22].

92

K. Franzrahe et al.

4 Theoretical Analysis of the Conductance and Structural Properties of Ni Nanocontacts During the last years a lot of attention has been devoted to the analysis of contacts of magnetic materials. In these nanowires the spin degeneracy is lifted, which can potentially lead to interesting spin-related phenomena in the transport properties. Here we address the issue of the conductance quantization and the spin polarization of the current of Ni contacts. We have combined classical molecular dynamics simulations of the breaking of nanocontacts with conductance calculations based on a tight-binding model. [25, 26, 27] For Ni we have applied our method to a Hamiltonian with spin-dependent matrix elements. [28] In Fig. 11 we show the evolution of the conductance during the formation of a Ni dimer structure, which is the most common geometry in the last stages of the breaking process.In addition to the evolution of the conductance and transmission eigenchannels for both spin components separately, we have plotted the MCS radius, strain force, spin polarization of the current, and contact configurations. The spin polarization P , shown in the inset of the lower panel, is defined as G↑ − G↓ × 100%, (1) G↑ + G↓ where Gσ is the conductance of the spin component σ Here, spin up (σ =↑) means majority spins and spin down (σ =↓) minority spins. Notice that in the last stages of the stretching the conductance is dominated by a single channel for the majority spins, while for the minority spin there are still up to four open channels. In the final stages [see regions with three or one open channel(s) for G↑ ] the conductance for the majority spin lies below 1.2e2 /h, while for the minority spin it is close to 2e2 /h, adding up to a conductance of around 1.2–1.6G0. For ferromagnetic Ni, we have shown that the contacts behave as a mixture of a noble metal (such as Ag) and a transition metal (such as Pt). While the 4s orbitals play the main role for the transport of the majority-spin electrons, the conduction of the minority-spin electrons is controlled by the partially occupied 3d orbitals. This follows from the position of the Fermi energy, which lies in the s band for the majority spins and in the d bands for the minority spins. Our results indicate the absence of any conductance quantization, and show how the current polarization evolves from negative values in thick contacts to even positive values in the tunneling regime after rupture of the contact. P =

5 Nano Shape Memory Alloys Shape Memory Alloys (SMAs) show interesting macroscopic structural deformation eﬀects [30], examples are Ni-Ti, Ni-Al, Ni-Fe alloys. Essential is

Computer Simulations of Soft Matter- and Nano-Systems

93

Fig. 11. Formation of a dimer conﬁguration for Ni (4.2 K, [001] direction). The upper panel shows the strain force as a function of the elongation of the contact. In the lower two panels the conductance Gσ , the MCS (minimum cross-section) radius, and the channel transmissions (colored ) are displayed for the respective spin component σ. Vertical lines separate regions with diﬀerent numbers of open channels ranging from 7 to 1 and 18 to 4, respectively [29]. An inset shows the evolution of the spin polarization. Above and below these graphs snapshots of the stretching process are shown. The inner structure of the starting conﬁguration has the length of about 2.29 nm

a transformation between the high-temperature Austenite structure and the low-temperature Martensite structure. An interesting questions in the context of nano-science is, to which microscopic length scales these eﬀects are still present, and how they can be controlled by temperature or external fields. Based on the experience with MD simulations for Ni nano-contacts, described in the previous section, we performed MD simulations [31] for a selected Nix Fe1−x alloy in the NVT and NpT ensembles for diﬀerent system sizes and compositions using embedded atom potentials [32, 33]. The particular case with x = 0.2 was analyzed in detail, since it permits a comparison with experimental data [34] and previous simulations [33].

94

K. Franzrahe et al.

After an investigation of the ground state energy for BCC- or FCC- lattice structures, the Austenite transition was studied. By computation of the pair correlation function the transition temperature was detected. The pair correlation function for x = 0.2 indicates a FCC/HCP-structure at high temperature and a BCC structure at low temperature. Configurations at high and low temperature are shown in Fig. 12. We found a strong eﬀect of the composition on the transition temperature: for a 20% increase of Ni concentration a 50% reduction of transition temperature resulted, see Fig. 13. These results are in agreement with experiments [34] and previous simulations [33] at corresponding values of x. The Austenite transition temperature decreases strongly by about 200 K for a reduction of particle number N from about 6000 to about 1000 particles, see Fig. 13.

Fig. 12. Conﬁgurations for the Ni0.2 Fe0.8 system at two temperatures. Left: Conﬁgurations at T = 100 K and T = 800 K. Right: cut in xy-plane, colors indicate position in z-direction

Fig. 13. Left: Austenite transition temperature [31] for various Ni concentrations, Right: Austenite transition temperature [31] for various system sizes

6 DFT-Investigations of Sin Clusters in External Fields We computed the properties of selected Sin clusters approaching each other and the eﬀect of external fields and surfaces [35, 36, 37] by DFT methods [38]. One example shown here is the pproach of Si-atoms to a graphite surface. The resulting binding energy as function of distance over certain sites [35] is

Computer Simulations of Soft Matter- and Nano-Systems

95

shown in Fig. 14. It follows that the site above the C-C bonds is energetically preferred. Another example [37] is the eﬀect of an external parabolic potential, V (x, y, z) = k 2 y 2 on the potential energy of two Si4 clusters as a function of distance, s. Fig. 14. Two nearest atoms of the two clusters are fixed at a distance R, the other atoms are allowed to move freely. For increasing k values the potential energy develops a fusion barrier at a distance of about 3 ˚ A.

Fig. 14. Left: Binding energy of Si atoms as function of distance over certain sites of a graphite surface Right: Potential energy for two Si4 clusters as function of distance in an external parabolic potential

7 Model Magnetic Systems By use of the Heisenberg Hamiltonian and the Landau-Lifshitz-Gilbert equation several interesting insights into the behavior of domain walls in confined geometry at finite temperature have been computed [39, 40, 41]. The dynamics of the spin reversal process in systems with moments attached to caps of (colloidal) spheres has been computed [41] for diﬀerent system sizes and external magnetic fields. Fig. 15 shows the hysterese eﬀect of external magnetic fields at angle ϑ relative to the x-axis.

Fig. 15. Left: Sketch of the geometry; Right: Hysterese for diﬀerent angles ϑ relative to the x-axis. The caps have a diameter of 48 nm and a height of 12 nm

96

K. Franzrahe et al.

References 1. A. Chowdhury, B.J. Ackerson, N.A. Clark, Phys. Rev. Lett. 55, 833 (1985) 2. Q.-H. Wei,C. Bechinger, D. Rudhardt, P. Leiderer, Phys. Rev. Lett. 81, 2602 (1998) 3. W. Strepp, S. Sengupta, P. Nielaba, Phys. Rev. E63, 046106 (2001). 4. W. Strepp, S. Sengupta, P. Nielaba, Phys. Rev. E66, 056109 (2002) 5. W. Strepp, S. Sengupta, M. Lohrer, P. Nielaba, Comput. Phys. Commun. 147, 370-373 (2002) 6. W. Strepp, S. Sengupta, M. Lohrer, P. Nielaba, Math. and Comput. in Simul. 62, 519 (2003). 7. P. Nielaba, W. Strepp, S. Sengupta, in Computer Simulations in Condensed Matter: From Materials to Chemical Biology, edited by M. Ferrario, G. Ciccotti, K. Binder, Springer, Berlin, pp.163 (2006). 8. Chr. Kircher, Diplomarbeit, U. Konstanz (2004). 9. F. B¨ urzle, Diplomarbeit, U. Konstanz (2006). 10. K. Franzrahe, Dissertation, U. Konstanz (in preparation). 11. K. Franzrahe, P. Henseler, A. Ricci, W. Strepp, S. Sengupta, M. Dreher, Chr. Kircher, M. Lohrer, W. Quester, K. Binder, P. Nielaba, Comp. Phys. Commun. 169, 197 (2005). 12. L. Lue, L.V. Woodcock, Mol. Phys.96, pp. 1435 (1999) 13. M. K¨ oppl, P. Henseler, A. Erbe, P. Nielaba, and P. Leiderer. Phys. Rev. Lett. 97, 208302 (2006). 14. D. M. Heyes and J. R. Melrose. J. Non-Newtonian Fluid Mech. 46, 1 (1993). 15. M. P. Allen and D. J. Tildesley. Computer Simulation of Liquids (Oxford Science Publications, 1987). 16. R.Haghgooie, C.Li, P.Doyle, Langmuir22, 3601 (2006). 17. J. M. Berg, J. L. Tymoczko and L. Stryer, Biochemistry, W. H. Freeman and Company (2002) 18. D. Nelson, T. Piran and S. Weiberg, Statistical Mechanics of Membranes and Surfaces, World Scientiﬁc (2004) 19. J. Neder, Dissertation, U. Konstanz (in preparation). 20. O. Lenz and F. Schmid, J. Mol. Liq. 117, 147 (2005) 21. Y. Zhang, S. E. Feller, B. R. Brooks and R. W. Pastor, J. Chem. Phys. 103 (23), 10252 (1995) 22. F. Schmid, et al.,http://arxiv.org/pdf/physics/0608226 (2006) 23. P. Virnau and M. M¨ uller, J. Chem. Phys. 120, 10925 (2004) 24. O. Lenz and F. Schmid, Phys. Rev. Lett. 98, 058104 (2007) 25. M. Dreher, Dissertation, U. Konstanz (in preparation). 26. F. Pauly, M. Dreher, J.K. Viljas, M. H¨ afner, J.C. Cuevas, and P. Nielaba, Phys. Rev. B 74, 235106 (2006). 27. M. Dreher, F. Pauly, J. Heurich, J.C. Cuevas, E. Scheer, and P. Nielaba, Phys. Rev. B 72, 075435 (2005). 28. See parameters ni ferro par at http://cst-www.nrl.navy.mil/bind/ni.html; M.I. Haftel, N. Bernstein, M.J. Mehl, and D.A. Papaconstantopoulos, Phys. Rev. B 70, 125419 (2004). 29. For the division of a conductance trace into regions with diﬀerent numbers of open conductance channels we use a criterion of Tnσ < 0.01 to consider the nth channel to be closed. This division is only approximate: Due to ﬂuctuations

Computer Simulations of Soft Matter- and Nano-Systems

30.

31. 32. 33. 34. 35. 36.

37. 38. 39. 40. 41.

97

in the geometry, a channel transmission may fall temporarily below 0.01 but reenter later on. In other cases many channels are present and, in order not to overload our pictures, we need to combine several channel-closing events into one. I. Takeuchi et al., Nature Mater. 2, 180 (2003); Z. Nishiyama, Martensitic Transformations, Academic Press (1978); E.Q.B. Winzek, T. Sterzl Stellelemente und Mikroaktoren aus Formged¨ achtnislegierungen, VDE (2001). T. Sorg; Diplomarbeit, U. Konstanz (2007). M.S. Daw, M.I. Baskes; Phys. Rev. B 29, 6443 (1984); S.M. Foiles, M.I. Baskes, M.S. Daw; Phys. Rev. B 33, 7983 (1986). R. Meyer, P. Entel; Phys. Rev. B 57, 5140 (1998). M. Acet, T. Schneider, E.F. Wassermann; J. Phys. IV C2, 123 (1995). W. Quester, Dissertation, U. Konstanz (in preparation). F. von Gynz-Rekowski, W. Quester, R. Dietsche, Dong Chan Lim, N. Bertram, T. Fischer, G. Gantef¨ or, M. Schach, P. Nielaba, Young Dok Kim, Eur. Phys. J. D, in press. M. Schach, Diplomarbeit, U. Konstanz (2007). CPMD. Copyright IBM Corp 1990–2001, Copyright MPI f¨ ur Festk¨ orperforschung Stuttgart 1997–2004. http://www.cpmd.org/. C. Schieback, Dissertation, U. Konstanz (in preparation). C. Schieback, M. Kl¨ aui, U. Nowak, U. R¨ udiger, P. Nielaba, Eur. J. B, in press. D. Mutter, Diplomarbeit, U. Konstanz (2007).

Signal Transport in and Conductance of Correlated Nanostructures Peter Schmitteckert Institut für Theorie der Kondensierten Materie, Wolfgang-Gaede-Straße 1, D-76128 Karlsruhe, Germany [email protected] Summary. Transport properties of strongly interacting quantum systems are a major challenge in todays condensed matter theory. While much is known for transport properties of non-interacting electrons, based on the Landauer Büttiker formalism, the non equilibrium properties of interacting fermions are an open problem. Due to the vast improvements in experimental techniques there is an increasing theoretical interest in one-dimensional quantum systems. Since in low dimension the screening of electrons is reduced the eﬀective interaction gets increased and can drive the electron systems into new phases beyond the standard description of a Fermi liquid, e.g. into a Luttinger liquid. Formally the conductance of a quantum device attached to leads is given by the Meir Wingreen formula. Besides the special case of proportional coupling, the Meir Wingreen can only be treated within perturbative approaches. The density matrix renormalization group method [2, 3, 4] is a well established method to treat onedimensional interacting quantum systems. In this project we apply the real time evolution within the density matrix renormalization group method (RT-DMRG) to simulate the signal transport in onedimensional, interacting quantum systems, and the conductance of interacting nanostructures attached to onedimensional, non-interacting leads. In addition we calculate the conductance from the current-current and current-density correlations functions as a comparison to the real time evolution scheme and as a tool as itself, as it allows for a higher energy resolution as compared to the real time approach. In this project we developed a DMRG code applying Posix threads to parallelize the code which is described in detail in [8]. While the DMRG is an approximative scheme, it has a systematic parameter, namely the number of states kept per block, to increase the accuracy of the calculation. In Sect. 1 we show that this code allows us to perform systematic studies of the accuracy of transport problems. The major problem that arouse during our previous work [5, 6, 7, 8] lies in the large ressources needed to perform the actual simulation. In Sect. 2 we show that we have now reformulated the Kubo approach which allows us to obtain a much higher energy resolution and that we could get rid of numerical instabilities.

100

P. Schmitteckert

1 Spin Charge Separation The spin charge separation of a single electron excitation is a prominent example of interaction eﬀects in onedimensional electron systems. The ﬁrst numerical observation was performed with an exact diagonalization approach by Karen Hallberg et al. [9] for a 16 site system. Kollath et al. [10] reported a simulation on a 72 site system with hard wall boundary conditions and 56 electrons. In [8] we showed that with our code it is possible to study spin charge separation within the frame work of RT-DMRG for a 2/3 ﬁlled 33 site Hubbard chain with periodic boundary conditions (PBC). The advantage of PBC lies is the absence of Friedel oscillations from the boundary. It turned out that for accurate results we should at least use of the order of 2000 states per block, which is considerably more than applied in [10]. Here we compare the results of Kollath et al. [10] (KSZ) who employed an adaptive RT-DMRG scheme combined with a Trotter decomposition [11, 12] with results obtained from our code [8] where we combine the adaptive scheme with a Krylov based matrix exponential [7]. The system is a 72 site Hubbard model with an on site interaction of U = 4.0. The perturbation was created by applying a Gaussian perturbation to the potential of the up-electrons in the same way as described in [7]. In Fig. 1 (2) we plot the relative accuracy of the electron density n(x) (and its spin component S z (x) = (n↑ (x) − n↓ (x))/2) at time step t = 0.1. It shows that KSZ and a 300 state calculation already have a relative accuracy which exceeds 10−3 for the electron density, while for the spin component one has to go up to 750 to achieve an accuracy of the order of 10−3 . The relative accuracy for the spin component is much harder as it can get close to zero.

Fig. 1. Comparison of the electron density n(x) of KSZ and our calculations keeping 300, 500, and 750 states per block, M = 72 sites, N↑ = N↓ = 28 with a reference calculation keeping 5000 states per block at time step t = 0.1

DMRG

101

Fig. 2. Comparison of S z (x) of KSZ and our calculations keeping 300, 500, and 750 states per block with a reference calculation keeping 5000 states per block at time step t = 0.1

Fig. 3. Comparison of the density n(x) and S z (x) of KSZ and our calculations keeping 750 states per block with a reference calculation keeping 3000 states per block at time step t = 1.6 after an initial run keeping 5000 states up to t = 0.1

After performing an initial calculation keeping 5000 states per block up to time t = 0.1, we continued the time evolution with 3000 states per block up to t = 12. In Fig. 3 we compare our results with KSZ and a calculation keeping 750 states per block. It shows that keeping 750 states per block we can still obtain an accuracy below 1% for the density and the spin component, while KSZ achieve and accuracy of 1% only for the density, while the spin component goes above an error of 10%. Finally we compare the results of KSZ with our reference calculation keeping 3000 states per block at time time step t = 11.6. While KSZ are able to achieve an accuracy of 1% for the absolute numbers, the spin component shows

102

P. Schmitteckert

a relative deviation larger than a few hundred percent. While we have to be careful whether our results can be trusted at t = 11.6 to serve as an accuracy benchmark, the calculation should be much more accurate than the one performed by KSZ. In summary we have shown that one has to be very careful when employing the real time extensions to the DMRG. However, DMRG allows for a systematic check of the results which is a very important property in a ﬁeld where no other benchmarks are available.

Fig. 4. Comparison of the density n(x) and S z (x) of KSZ with a reference calculation keeping 3000 states per block at time step t = 11.6 after an initial run keeping 5000 states up to t = 0.1. For the S z (x) component we plot the relative and the absolute diﬀerence

2 Linear Response with Momentum Leads Linear response calculations within DMRG [6] provide a method to calculate the conductance of a nanostructure attached to leads. As itis based on the 2 the exact Kubo formula for the linear conductance g ≡ eh J˜ /VSD it is valid for arbitrary interaction. In the DC limit the conductance can be expressed in terms of two diﬀerent correlators, e2 ˆ 4πiη ˆ ψ0 , ψ0 Jnj N 2 2 ˆ h (H0 − E0 ) + η 2 ˆ 0 − E0 ) e 8πη(H ψ0 Jˆn1 = 2 Jˆn2 ψ0 , 2 2 h ˆ (H0 − E0 ) + η

g Jj N = − gJJ

(1) (2)

where the positions nj are in principle arbitrary. However, the positions n1 and n2 should be placed close to the nanostructure to minimize ﬁnite size

DMRG

103

eﬀects. Bohr, Wölﬂe and Schmitteckert [6] had to introduce exponentially reduced hopping terms close to the boundary of the leads which had been described in real space to minimize ﬁnite size eﬀects, which in return leads to ill-conditioned linear systems. In order to solve these equations, they had to employ scaling sweeps to switch on the damping in the leads gradually. While the method proofed to be a valuable tool it turned out that it is getting too expensive to study more interesting systems. Recently we have developed a new scheme [13] based on leads described in momentum space to overcome the diﬃculties we encountered in [6], for details see also [8]. While it is generally accepted that DMRG does not work well in a momentum space description due to the large amount of couplings intersecting the artiﬁcial cut of the system into two parts within DMRG, our transport calculation are performed with non interacting leads. Therefore the number of links intersecting the DMRG splitting of the system is vastly reduced. In order to be able to describe processes on diﬀerent energy scales we ﬁrst couple our nanostructure to a few sites in real space to capture local, i.e. high energy, physics. Then we employ a logarithmic discretization of the momentum leads to cover a large energy range and ﬁnally we use a linear discretization of the low energy scale in order to describe low energy transport properties accurately. We would like to note that these additional sites on a linear discretization close to the Fermi edge are beyond a NRG like description. While they are not needed for a qualitative description, they enable us to get very accurate results even close to the resonant tunneling regime. The reason for that lies in the nature of transport properties, where the η in the correlation function plays a much more important role than for equilibrium properties. It does not only provide a smoothing of the poles, it has to create excitations which then can actually lead to transport. The models considered in this work are the interacting resonant level model (IRLM) and the natural extension of this model to linear chains, deﬁned by the Hamiltonians †

HRS = μg cˆ†j cˆj − tj cˆj cˆj−1 + h.c. j∈S

j,j−1∈SE

+

j,j−1∈SE

HMS =

ˆj − Vj n

ǫk cˆ†k cˆk ,

k∈L,R

HT = −tk

1 1 n ˆ j−1 − , 2 2

k∈L

cˆ†k cˆ1 +

(3) (4)

k∈R

cˆ†k cˆME

+ h.c.,

(5)

where cˆ†ℓ and cˆℓ (ˆ c†k and cˆk ) are the spinless fermionic creation and annihilation operators at site ℓ (momentum k), n ˆ ℓ = cˆ†ℓ cˆℓ . HRS , HMS , and HT denote real space, momentum space, and tunneling between real- and momentum space

104

P. Schmitteckert

Hamiltonians respectively. The symbols S and SE denote the nanostructure and the extended nanostructure (the full real space chain) respectively. The indices 1 and ME denote the ﬁrst and last site in SE . The general setup and the speciﬁc values of the hopping matrix elements tj and the interactions Vj are indicated in Fig. 5, and note speciﬁcally the interactions on the contact links, γV . The momentum dependent coupling tk is chosen to represent an inﬁnite onedimensional tight-binding chain if a cosine band ǫk = −2t cos(k) is chosen. All energies are measured in units of t = 1. In Fig. 6 we show the linear conductance versus gate potential for a contact hopping of t′ = 0.01 and interaction on the contacts ranging from zero to 25. The calculations have been performed with 130 sites in total, ME = 10 real space sites, and 120 momentum space sites. Due to the symmetry of the band we used a discretization that is symmetric around ǫF = 0, and applied an identical discretization scheme to both leads. To represent the ‘large’ energy span in the band we used 20 logarithmically scaled sites, and thereafter used 10 linearly spaced sites to represent the low energy scale correctly. In the DMRG calculations presented we used at least 1300 states per block and 10 ﬁnite lattice sweeps. The data demonstrates a strong increase of the resonance width due to interaction up to a factor of ten. The increase of the resonance width due to interaction on the contact is in contrast to the reduction of conductance due to interaction on nanostructures, see [6]. Once interaction is larger than the Fermi velocity the resonance width gets strongly reduced. The results also shows that we can now resolve resonance width of the order 10−5 . We would like to note that this scheme is not restricted to single impurity models and that it also works for extended nanostructures. The implementation of this new scheme was only ﬁnished recently and we are currently extending it to include the spin degree of freedom. In detail we

Fig. 5. Schematics of the leads coupled to the nanostructure

DMRG

105

Fig. 6. Linear conductance versus gate potential for the interacting resonant level model for t′ = 0.01 and a interaction on the contacts ranging from zero to 25. To each set of DMRG data a Lorentzian of half width 2w has been added as a guide to the eye. The leads are described with a cosine band between ±2 such that the Fermi velocity is vF = 2. In contrast to intradot interaction the interaction on the contacts enhances the conductance and shows a non monotonic behavior versus contact interaction

study the single impurity Anderson model attached to polarized, ferromagnetic leads.

3 Further Projects The code developed within this project has also been used in [14] to study quantum phase transition with entanglement entropy and in [15] to study onedimensional fermions in a harmonic trap with an attractive on site interaction. In [16] we used the DMRG to extract the exact corresponding functionals of a lattice Density Functional Theory and compared the conductance calculations within DMRG and DFT. Acknowledgements We would like to thank Corinna Kollath for interesting discussions and for providing us with the raw data of [10]. This work proﬁted from the parallelization

106

P. Schmitteckert

performed within the project 710 of the Landesstiftung Baden-Württemberg. The reformulation of the momentum leads was performed together with Dan Bohr within the HPC-EUROPA project RII3-CT-2003-506079. Most of the calculations have been performed at the XC1 and XC2 of the SSC Karlsruhe under the grant number RT-DMRG.

References 1. 2. 3. 4.

5. 6. 7. 8.

9. 10. 11. 12. 13. 14. 15. 16.

Y. Meir and N. S. Wingreen, Phys. Rev. Lett. 68, 2512 (1992). S. R. White, Phys. Rev. Lett. 69, 2863 (1992). S. R. White, Phys. Rev. B 48, 10345 (1993). Density Matrix Renormalization – A New Numerical Method in Physics, edited by I. Peschel, X. Wang, M.Kaulke, and K. Hallberg (Springer, Berlin, 1999); Reinhard M. Noack and Salvatore R. Manmana, Diagonalization- and Numerical Renormalization-Group-Based Methods for Interacting Quantum Systems, AIP Conf. Proc. 789, 93-163 (2005). Günter Schneider and Peter Schmitteckert: Conductance in strongly correlated 1D systems: Real-Time Dynamics in DMRG, condmat-0601389. Dan Bohr, Peter Schmitteckert, Peter Wölﬂe: DMRG evaluation of the Kubo formula – Conductance of strongly interacting quantum systems, Europhys. Lett., 73 (2), 246 (2006). Peter Schmitteckert: Nonequilibrium electron transport using the density matrix renormalization group, Phys. Rev. B 70, 121302 (2004). Günter Schneider and Peter Schmitteckert: Signal transport and finite bias conductance in and through correlated nanostructures p. 113 -126 in W.E. Nagel, W. Jäger M. Resch (Eds.), “High Performance computing in Science and Engineering ’06”, Springer Verlag Berlin Heidelberg 2007, ISBN 978-3-540-36165-7. E. A. Jagla, K. Hallberg, C. A. Balseiro, Phys. Rev. B 47, 5849 (1993). C. Kollath, U. Schollwoeck, W. Zwerger, Phys. Rev. Lett. 95, 176401 (2005). S. R. White and A. E. Feiguin, Phys. Rev. Lett. 93, 076401 (2004). A. J. Daley, C. Kollath, U. Schollwöck, and G. Vidal, J. Stat. Mech.: Theor. Exp. P04005 (2004). Dan Bohr and Peter Schmitteckert: Strong enhancement of transport by interaction on contact links, Phys. Rev. B 75 241103(R) (2007). Rafael A. Molina and Peter Schmitteckert, Numerical estimation of critical parameters using the bond entropy, Phys. Rev. B 75, 235104 (2007). Rafael A. Molina, Jorge Dukelsky, and Peter Schmitteckert, Commensurability eﬀects for fermionic atoms trapped in 1D optical lattices, arXiv:0707.3209, accepted by Phys. Rev. Lett. (2007). Peter Schmiteckert and Ferdinand Evers, Exact ground state density functional theory for impurity models coupled to external reservoirs and transport calculations, arXiv:0706.4253.

Ab initio Simulations of PbTe-CdTe Nanostructures R. Leitsmann and F. Bechstedt European Theoretical Spectroscopy Facility (ETSF) and Institut für Festkörpertheorie und -optik Friedrich-Schiller-Universität Jena Max-Wien-Platz 1, 07743 Jena, Germany [email protected] Summary. We present electronic band structures for the (110), and (100) PbTe/ CdTe interfaces. The first principles calculations are based on large supercells containing a large number of atoms, which have to be treated fully quantum mechanically. The treatment of free standing nanodots is conceptual more diﬃcult. For the nearly ionic IV-VI semiconductor nanodots we introduce a novel passivation scheme to model the dot-vacuum interfaces. First results for the electronic structure of PbTe nanodots embedded in a CdTe matrix are presented.

1 Introduction Nanostructuring of semiconductors is the modern way of developing devices for electronic, optoelectronic and sensoric applications. The huge eﬀorts made towards matter manipulation on the nanometer scale are motivated by the fact that desirable properties can be generated by modifying the spatial quantum conﬁnement of electrons and holes, for instance, by changing the system dimension and shape. Very recently the formation of PbTe quantum dots in a crystalline CdTe host matrix has been demonstrated by Heiss et al. [1]. They observed a very high mid infrared luminescence yield, which makes this system a promising candidate for future applications in, e.g. mid infrared quantumdot lasers or medical diagnostic devices. From the knowledge of PbTe/CdTe interface energies for diﬀerent orientations a rhombo-cubo-octahedron equilibrium crystallite shape (ECS) of PbTe nanodots embedded in a CdTe matrix could be predicted [2]. As can be seen in Fig. 1 this prediction is in excellent agreement with experimental observations. The two tellurides possess a face-centered cubic (fcc) translational symmetry with almost identical lattice constants a0 , but diﬀer fundamentally in their bonding conﬁgurations. PbTe consists of group-IV and group-VI atoms with strong ionic bonds. Each atom has six nearest neighbors resulting in the

108

R. Leitsmann, F. Bechstedt

Fig. 1. (a) Schematic picture of the theoretically predicted rohmbo-cubo-octahedral ECS of embedded PbTe nanodots. (b) Experimentally observed HRXTEM image (private communication – F. Schäﬄer, W. Heiss University of Linz) of a small PbTe nanodot embedded in CdTe matrix with indications of the symmetry-allowed lattice planes and interface orientations

rocksalt (rs) structure. The more covalent II-VI compound CdTe has a zincblende (zb) crystal structure, and fourfold coordinated atoms. Despite the almost complete absence of lattice-constant mismatch, the lattice-type mismatch leads to a large miscibility gap [3]. Besides the interface geometry, the diﬀerent arrangements of cations and anions in bulk zb and rs may also lead to a polarity mismatch at the interface. For instance, a (100) face of rs is non-polar, and thus the cleavage plane, whereas in zb the [100] orientation leads to polar, cation- or anion-terminated faces. The only non-polar zb face is the {110} cleavage plane, which is also electrostatically neutral in rs. In the present study we focus on the electronic properties of the embedded PbTe nanodots as well as on the electronic properties of PbTe/CdTe interfaces. In this way we hope to understand the occurrence of the observed high mid infrared luminescence yield.

2 Computational Method 2.1 Kohn-Sham Energy Functional and PAW Method Our calculations are based on the commonly used density-functional theory (DFT) [4]. Its popularity arises from the fact that it is not only of academic interest but also applicable to real-life systems. According to the HohenbergKohn theorems the total energy of a system of N interacting electrons in an external potential vext (r) is an unique functional of the electron density n(r). The ground-state energy E0 can be obtained from the solution of a variational problem. To simplify the calculation of this many electron problem one maps it onto a system of N non-interacting electrons, which has the same groundstate density and can be represented by a set of one-particle equations, the Kohn-Sham equations [5]:

Ab initio Simulations of PbTe-CdTe Nanostructures

2 2 ∇ + vext (r) + vH [n](r) + vxc [n](r) ψnk (r) = εnk ψnk (r), − 2m

109

(1)

where vH [n](r) is the classical Hartree potential. The exchange-correlation potential deﬁned as the density variation of the XC energy functional δExc /δn(r) = vxc [n](r) will be calculated by using the local-density approximation (LDA) [5]. Solving equations (1) self-consistently one ﬁnds the exact ground-state density and thus all physical properties that are functionals of this density. For a numerical solution we have to expand the wavefunctions and potentials into a certain basis set. For systems with periodic boundary conditions like, e.g. bulk crystalline structures or repeated supercells, an expansion into plane waves yields the most eﬃcient numerical algorithms, because in this case the kinetic energy operator is diagonal in real and reciprocal space. Therefore the evaluation of the action of the Hamiltonian is very fast when Fast Fourier Transforms (FFT) are used to transform the wavefunctions to reciprocal space and back. However, representing the rapid oscillations of wavefunctions near the nucleis demands a large number of plane waves. On the other hand in the interstitial region, where the wavefunctions are rather smooth, most of the interesting physical properties are determined. Therefore we employ the Projector Augmented Wave method (PAW) [6] to establish a one-to-one correspondence between the exact, near the nuclei rapidly oscillating wavefunctions ψnk (r) and a set of smooth pseudo-wavefunctions ψ˜nk (r), that match the exact ones outside a certain radius around each nucleus: 1,R 1,R ψ˜nk (r) − ψnk ψnk (r) = ψ˜nk (r) − (r) . (2) R

1,R (r) and the exact one-center waveThe one-center pseudo-wavefunctions ψ˜nk 1,R functions ψnk (r) are represented on radial grids. For the diagonalization of the Kohn-Sham matrix we employ the Residual Minimization Method with Direct Inversion in Iterative Subspace (RMMDIIS) [7] as implemented in the Vienna Ab-initio Simulation Package (VASP) [8, 9]. Parallelization is done using the Message Passing Interface (MPI).

2.2 Computational Cost The Kohn-Sham eigenvalue equations and can be solved very eﬃciently using iterative schemes. The diagonalization can be eﬃciently parallelized, since the Kohn-Sham matrix is diagonal in the index n of the eigenstate (“interband-distribution”) Furthermore, if there are enough nodes available, the diagonalization for the n-th state may be parallelized as well (“intra-banddistribution”). The only limiting factor is the communication overhead required for the redistribution of the wavefunctions between all nodes, which is necessary during the orthogonalization procedure of the eigenstates.

110

R. Leitsmann, F. Bechstedt

Fig. 2. Performance on the NEC SX-8 for two diﬀerent PbTe nanodot systems containing 1000 (red ) and 512 (black ) atoms. Additionally the SpeedUp of the systems in percent are indicated by black circles (512 atoms) and red squares (1000 atoms)

Our calculations were carried out on the NEC SX-8 system. Figure 2 shows the scaling behaviour of our code on this machine for the ﬁrst ionic relaxation step of an embedded PbTe nanodot. The performance for the iteration part is computed as the diﬀerence between a complete run for two ionic steps and a run for only one ionic step.The computation is dominated by complex matrix-matrix multiplication (CGEMM). The sustained iteration performance for both cases exceeds 1 TFLOPS already on 16 nodes NEC SX-8 (Fig. 2). The sustained eﬃciency is between 79 and 50 % [10].

3 Results and Discussion 3.1 PbTe/CdTe interfaces Bulk PbTe and CdTe exhibit nearly identical cubic lattice constants. Therefore PbTe/CdTe interfaces are dominated rather by the lattice-type mismatch than by strain eﬀects. Due to the common fcc Te-sublattice the formation of interface bonds between Te atoms (at the PbTe side of the interface) and the fourfold coordinated Cd atoms (at the CdTe side of the interface) does not require displacements of the participating atoms. However, from the point of view of the p-bonded Te interface atoms at the rs side, the adjacent Cd atoms at the zb side have to be displaced from their (1,1,¯1)a0 /4 positions to (1,1,¯ 1)a0 /2 in order to achieve a sixfold coordination. The interplay between these two tendencies results in the observed atomic displacements at the nonpolar (110) interface, shown in Fig. 3(a) and Table 1. The most prominent eﬀect is a collective shift parallel to the interface of the atoms in [¯100] and

Ab initio Simulations of PbTe-CdTe Nanostructures

111

Fig. 3. Experimentally observed HRXTEM images (private communication – F. Schäﬄer, W. Heiss University Linz) of the {110}-(a),Te-terminated {100}-(b), and Cd-terminated {100}-(c) PbTe/CdTe interfaces. The theoretically predicted atomic positions are indicated by red (Cd), yellow (Te) and green (Pb) cirlcles Table 1. Atomic displacements (in Å) at the interfaces with respect to the ideal starting coordinates for the ﬁrst two interface layers, (1) and (2). In the non-polar case displacements in normal [110] direction (upper line) and parallel to the cubic axis [001] (second line) are listed. In the case of polar interfaces only normal displacements are shown. The interface termination is derived from the zb structure. In all other directions displacements are not observed Interface (110)⊥ (110)

Te1PbTe Te2PbTe Pb1PbTe Pb2PbTe Te1CdTe Te2CdTe Cd1CdTe Cd2CdTe ∓0.15 ∓0.12 ∓0.24 ∓0.02 ∓0.09 ±0.08 ±0.17 ∓0.04 0.25 0.30 0.38 0.35 –0.16 –0.27 –0.24 –0.27

(100)Te−term. ±0.05 ±0.00 ±0.16 ±0.23 ∓0.03 ∓0.05 ∓0.10 ∓0.07 (100)Cd−term. ±0.05 ∓0.18 ∓0.41 ∓0.24 ±0.29 ±0.25 ±0.30 ±0.29

[100] direction on the CdTe and the PbTe side, respectively, which is independent of the cation or anion character. This results in a substantial oﬀset between the Te fcc sublattices on each side of the interface of 0.038 nm [11]. As can be seen in Fig. 3(a) our theoretical predictions for the atomic interface displacements (shown by red, green, and yellow balls representing Cd, Pb, and Pb atoms) are in excellent agreement with the experimentally observed high resolution cross-sectional transmission electron microscopy (HRXTEM) images. The same holds for the (100) interfaces, which diﬀer according to the Cd- or Te-termination of the CdTe(100) half space (see Fig. 3(b) and (c)). In both cases only normal displacements along [100] occur. The distance between the outermost Cd layer of CdTe and the nearest Pb-Te layer of PbTe amounts to almost a0 /4. It is increased to a value of about a0 /2 when the outermost layer of CdTe is a layer of Te atoms. The obtained equilibrium structures at the considered interfaces can be used to calculate the corresponding electronic ground-state properties. In par-

112

R. Leitsmann, F. Bechstedt

ticular we calculate the projected DFT-LDA interface band structures for the (110), the Te-terminated (100), and the Cd-terminated (100) interfaces. In the case of PbTe/CdTe(110) interfaces we are dealing with stoichiometric and hence neutral lattice planes in both crystal structures, rs and zb. The projected interface band structure exhibits no states inside the fundamental gap region (see Fig. 4), which is related to the nearly unperturbed atomic structure at the PbTe/CdTe(110) interface. However, the [100] direction is a polar direction within the zb crystal structure. Due to electrostatic reasons in this case the interface ions are only partially ionized [12] resulting in partially occupied conduction states at the Cd-terminated interface and partially empty valence states at the Te-terminated interface. Therefore the Fermi level is shifted into the conduction or valence bands at the Cd-terminated or Te-terminated (100) interfaces, as can be seen in Fig. 4. The result is a metallic character of both projected interface band structures. A crucial condition for a fast radiative electron-hole-pair recombination, which would explain the obtained high mid infrared luminescence yield in the PbTe nanodots embedded in CdTe matrix is, however, the absence of electronic states inside the fundamental gap region at the nanodot-matrix interfaces. Hence the predicted metallic character of PbTe/CdTe(100) interfaces is inconsistent with the experimental observations at the PbTe nanodot system. On the other hand up to now we have discussed only properties of isolated

Fig. 4. Projected band structures of PbTe/CdTe interfaces. Electronic states around the Fermi level (taken as energy zero) are shown in red

Ab initio Simulations of PbTe-CdTe Nanostructures

113

interfaces between PbTe and CdTe. In a real nanodot system eﬀects beside the properties of the dot-matrix interfaces have to be taken into account, e.g. eﬀects resulting from the conﬁnement of the electron and hole states captured in PbTe nandotots. Therefore it is necessary to ﬁnd a complete ab initio description of the embedded PbTe nanodots. 3.2 PbTe Nanodots Of special interest are the evolution of energetic positions and the localizations of the HOMO (highest occupied “molecular” orbital) and LUMO (lowest unoccupied “molecular” orbital) states of the dot-matrix system. Therefore we have calculated the HOMO-LUMO gap for diﬀerent matrix and nanodot sizes. With an increasing number of matrix layers between the nanodots the HOMO-LUMO gap decreases, as it is shown in Fig. 5. That is an indication for a dot-dot interaction across the matrix region, which decreases with increasing dot-dot distance. As expected we found a decreasing HOMO-LUMO gap with increasing dot size. However, the obtained gap for a Pb92 Te79 nanocrystal of about 0.06 eV is much below the PbTe-bulk gap of 0.19 eV. The large diﬀerence between the Pb38 Te43 and Pb92 Te79 gaps may be the result of the diﬀerent terminations of the nanodots. Unfortunately many other eﬀects, like e.g. diﬀerent dot shapes, dipole ﬁelds (see HOMO and LUMO states in Fig. 5) and so on, have to be considered to obtain a conclusive picture of the behavior of the electronic states in this system. Furthermore the treatment

Fig. 5. HOMO-LUMO gap for diﬀerent dot sizes (red triangles) and diﬀerent dotdot distances, i.e. diﬀerent matrix sizes (green circles). The insets show a schematic stick and ball model of the embedded PbTe nanodots. The dashed black lines mark the dot-matrix interfaces, while the full (empty) HOMO (LUMO) states are shown in green (red ). The observed separation of the HOMO and LUMO states indicate an occurring dipole ﬁeld along the dot diagonal

114

R. Leitsmann, F. Bechstedt

of such nanodot-matrix systems is still a challenge for the modeling and the theoretical/computational implementation. Hence, we decided to investigate the pure conﬁnement eﬀect on the HOMO-LUMO gap ﬁrst. For this purpose we consider free standing PbTe nanodots. To passivate the dangling bonds of the six-fold coordinated Pb and Te atoms at the dot-vacuum interfaces we have tested diﬀerent values for the valencies of the used pseudo-hydrogen atoms. Following the argumentation of Huang et al. [13] the optimal passivation is reached for a passivated PbTe cluster with a maximal HOMO-LUMO gap. In Fig. 6 the structural models as well as the localization of the electronic states near the Fermi level for six diﬀerent passivation schemes are shown. The three models in ¯ 6 1.0 1 Fig. 6(a), including the intuitive Pb38 Te43 H0. 78 H48 model , exhibit a zero gap, i.e. they have only partially occupied HOMO states. The passivation schemes used in Fig. 6(b) exhibit a ﬁnite HOMO-LUMO gap. However, at 0.¯ 3 0.¯ 0.¯ 3 0.¯ the Pb38 Te43 H78 H483 and Pb38 Te43 H78 H486 models we ﬁnd only very narrow gaps of 0.07 and 0.21 eV. The LUMO states are partially localized at the pseudo-hydrogen atoms, i.e. these states are not related to the ﬁnite dot size, but just electronic states introduced by the used pseudo-hydrogen atoms. In ¯ 3 1.¯ 6 contrast the Pb38 Te43 H0. 78 H48 model (with pseudo-hydrogen valencies of 1/3 and 5/3) exhibits a gap of 1.9 eV and HOMO and LUMO states localized in the centre of the dot, but not at the passivating atoms. Therefore this is the most reliable passivation scheme for free standing IV-VI semiconductor nanodots with rocksalt structure.

4 Summary and Outlook We used several structural interface models, which are in excellent agreement with experimental observations to calculate the electronic band structures of PbTe/CdTe interfaces. At the nonpolar (110) interface we found no electronic states inside the fundamental gap region, while we predict a metallic character for both (anion and cation terminated) polar PbTe/CdTe(100) interfaces. To investigate the conﬁnement eﬀect on the HOMO-LUMO gap of PbTe nanodots we have developed a passivation scheme using pseudo-hydrogen atoms for nearly ionic IV-VI semiconductor nanodods. The most reliable valencies of the pseudo-hydrogen atoms are determined to be 1/3 and 5/3 to passivate the six-fold coordinated Te and Pb atoms respectively. In the next steps we plan to give a conclusive picture of all eﬀects inﬂuencing the HOMO-LUMO gap of PbTe nanodots embedded in a CdTe host matrix. Therefore, calculation for nanodots with diﬀerent sizes, separations, and shapes have to carried out. 1

Here we have chosen the valencies of the pseudo-hydrogen atoms according to the valencies of Te and Pb divided by six according to their six-fold coordination, i.e. 1.0 = 6/6 and 0.¯ 6 = 4/6.

Ab initio Simulations of PbTe-CdTe Nanostructures

115

Fig. 6. Structural model (top panel ) and localization of the electronic states near the Fermi level (lower panel ) for pseudo-hydrogen atoms with diﬀerent valencies. Pb, Te, and pseudo-hydrogen atoms are represented by green, yellow, and white balls, respectively. The dots in (a) exhibit a metallic character – the half occupied HOMO states are shown in magenta. Dots in (b) have a semiconducting character – the full (empty) HOMO (LUMO) states are shown in blue (red )

116

R. Leitsmann, F. Bechstedt

Acknowledgements We acknowledge valuable discussions with Prof. F. Schäﬄer (Linz), Prof. F. Heiss (Linz) and colleagues of our group L.E. Ramos, J. Furthmüller and F. Fuchs. The work was ﬁnancially supported through the Fonds zur Förderung der Wissenschaftlichen Forschung (Austria) in the framework of SFB25, Nanostrukturen für Infrarot-Photonik (IR-ON) and the EU NANOQUANTA network of excellence (NMP4-CT-2004-500198). We thank the Höchstleistungsrechenzentrum Stuttgart (HLRS) for granted computer time (grant number xuj12790).

References 1. W. Heiss, H. Groiss, E. Kaufmann, M. Böberl, G. Springholz, F. Schäﬄer, K. Koike, H. Harada, and M. Yano, Appl. Phys. Lett 88, 192109 (2006). 2. R. Leitsmann, L. E. Ramos, and F. Bechstedt, Phys. Rev. B 74, 085309 (2006). 3. V. Leute, N. J. Haarmann, and H. M. Schmidtke, Z. Phys. Chemie 190, 253 (1995). 4. P. Hohenberg and W. Kohn, Phys. Rev. 136, B864 (1964). 5. W. Kohn and L. J. Sham, Phys. Rev. 140, A1133 (1965). 6. P. E. Blöchl, Phys. Rev. B 50, 17953 (1994). 7. P. Pulay, Chem. Phys. Lett. 73, 393 (1980). 8. G. Kresse and J. Furthmüller, Comp. Mat. Sci. 6, 15 (1996). 9. G. Kresse and J. Furthmüller, Phys. Rev. B 54, 11169 (1996). 10. S. Haberhauer, NEC-High Performance Computing Europe GmbH [email protected]. 11. R. Leitsmann, L. E. Ramos, F. Bechstedt, H. Groiss, F. Schäﬄer, W. Heiss, K. Koike, H. Harada, and M. Yano, New J. Phys. 8, 317 (2006). 12. P. W. Tasker, J. Phys. C 12, 4977 (1979). 13. X. Huang, E. Lindgren, and J. R. Chelikowsky, Phys. Rev. B 71, 165328 (2005).

The Basic Structure of Ti-Si-N Superhard Nanocomposite Coatings: Ab Initio Studies Xuejie Liu1,2 , Bernhard Gottwald1 , Changqing Wang3,4 , Yu Jia3 , and Engelbert Westkaemper1 1

2

3

4

IFF, University Stuttgart, 70569 Stuttgart, Germany [email protected] School of Mechanical Engineering, Inner Mongolia University of Science and Technology, Baotou, Inner Mongolia 014010, PR China School of Physics and Engineering, Key Laboratory of Material Physics of National Education Ministry, Zhengzhou, Henan 450052, PR China Department of Civil Engineering, Luoyang Institute of Science and Technology, Luoyang 471023, PR China

Summary. A new interface structure, the cross double tetrahedron Si-4N-4Ti, is reported in this paper. To ﬁnd out the atomic structure of Ti-Si-N superhard nanocomposite coatings, total energy calculations for the diﬀerent conﬁgurations of TiN with Si addition were performed with the ab initio method. The calculation results indicate that (a) there is no interstitial solid solution of Si in the TiN crystallite under the equilibrium condition; (b) the basic structure of the Ti-Si-N composite is the TiN crystallite with the cross double tetrahedron Si-4N-4Ti in the boundary of TiN. The cross double tetrahedron Si-4N-4Ti is an intrinsic mismatch to the B1-NaCl structure of TiN and it ﬁlls the boundary with the covalent combination. The TiN boundary is strengthened and the inter diﬀusion through the boundary is restricted by the interface Si-4N-4Ti so that the hardness and the thermal stability of Ti-Si-N coatings are enhanced. As a ﬁxation element to the rock salt structure, the potential value of the cross double tetrahedron is in the mass production of nanometer patterns or the quantum dots. The formation of the Si-4N-4Ti in the ﬁlm growth process is also studied. The study indicates that Si-4N-4Ti cannot be formed in the island of TiN on the TiN(001) surface. Under some process conditions, Si will stay outside the island of TiN. The investigation of the Si distribution in TiN demonstrates that the congregation of the Si-4N-4Ti structures in TiN will result in an increase in the local strain and a decrease in the cohesive energy of the system. PACS number: 68.35.-p, 68.55.Nq, 71.15.Nc, 81.07.-b

118

X. Liu et al.

1 Introduction The search for and the design of superhard materials are a topic of much interest to scientists because the superhardness has great meaning to science and technical application [1, 2]. In recent years a lot of research interest is attracted to the superhard nanocomposite coatings, e.g., nc-TiN/a-Si3 N4 , nc-W2N/a-Si3 N4 , nc-TiN/a-TiB2 , and so on [3, 4, 5, 6, 7]. Among these superhard composites, the Ti-Si-N nanocomposite coatings is a special one. Its hardness was reported above 100 GPa, reaching the hardness of diamond [8]. Normally its hardness is in the range 30–60 GPa [9]. In addition, it has high thermal stability and high oxidation resistance [10]. Since 1992 a lot of research has been conducted on the Ti-Si-N composite hard coatings. But until now, the basic structure of Ti-Si-N composite is not very clear. The crystallite TiN has been identiﬁed from XRD and SAD patterns [11, 12], whereas no silicon nitride can be found with XRD, SAD, and even with HRTEM [13]. Through the XPS characterization, the binding energy of Si3 N4 has been identiﬁed [8, 11]. It has led to a hypothesis that the structure of Ti-Si-N composite is the nano-crystallite TiN embedded in the amorphous Si3 N4 matrix, expressed as nc-TiN/a-Si3 N4 [14, 15]. However, with this conceptual structure model, it is diﬃcult to analyze the causes of the superhardness and the formation of the nanostructures. An important step could be made by an atomic structure description, especially for the interface structure between TiN and Si3 N4 . The atomic structure description should provide the exact atomistic position relationship or coordinates of the Ti-Si-N structure based on its formation process. So far almost all investigations about the Ti-Si-N nanocomposite were conducted with experiments, only a few studies on the simulation or calculation of the Ti-Si-N composite have been undertaken and reported. For example, Liu and his coworkers used the Monte Carlo method to investigate the crystallite size transition [16]. The atomic structure of Ti-Si-N was not their research interest. Recently, Hao and his coworkers published three papers on the ﬁrst principle studies of the structure and properties of TiN(111)/Six Ny /TiN(111) interfaces and the role of oxygen in the interfaces [17, 18, 19]. A lot of ab initio calculations were performed for 55 possible conﬁgurations of the Ti-Si-N interface in their investigations and many interesting results were obtained. Under the concept of the digital factory [20], we are trying to simulate the Ti-Si-N formation process. Before the simulation, we should ﬁnd out the atomic description of the Ti-Si-N structure in order to obtain the exact proportion of Ti : N : Si. We performed some ab initio calculations to investigate the basic structure of Ti-Si-N coatings. Our research focus was on three basic aspects. (1) Which conﬁguration of TiN with Si addition is the most stable structure? (2) How could the minimum energy structure of Ti-Si-N form on a TiN(001) surface in the ﬁlm growth process? (3) How should the Si atoms be distributed for the low energy structure?

The Basic Structure of Ti-Si-N Superhard Coatings

119

The calculation results show that the basic structure of Ti-Si-N is the crystallite TiN with the cross double tetrahedron Si-4N-4Ti in the boundary of TiN. The Si-4N-4Ti is an intrinsic mismatch to the B1-NaCl structure of TiN. It ﬁlls the boundary with the covalent combination. The study of the Si4N-4Ti formation discloses a necessary condition of the phase segregation. And the investigation of the Si distribution demonstrates that the congregation of the Si-4N-4Ti structures in TiN will result in an increase in the local strain and a decrease in the cohesive energy of the system.

2 Calculations and Results 2.1 Calculation Method Our calculations have been carried out with the VASP code [21, 22, 23] which is based on the density functional theory (DFT). It employs pseudopotentials, a plane-wave basis set and periodic boundary conditions to determine the Kohn-Sham ground state. In our calculations, the ultrasoft pseudopotential of N, Ti, and Si were used to describe the electron and ion inter-action. The Ti 2p electron was considered as valence electron in the calculations dealing with the TiN and the Ti-Si-N structures. The local density was described with the generalized gradient approximation (GGA) based on the Perdew-Wang’91 formulation for exchange correlation [24]. The criteria for terminating the electronic or ionic iterations was an energy diﬀerence of 10−3 or 10−4 eVaccording to the atom numbers of the supercell to balance the calculation accuracy and the calculation cost. To obtain the theoretical equilibrium lattice constant of TiN, a series of total energy calculations for various lattice constants has been performed. The calculated result, 4.262 Å, agrees well with the experimental value of 4.242 Å [25] and is very close to the result obtained by the full-potential linear augmented plane wave (FLAPW) method of 4.26 Å [26]. The TiN coatings have a polycrystalline structure which consists of crystallites and boundaries. A Si atom added in the TiN coating may be in the crystallite, or in the boundary, or in both of them. In following subsections, ﬁrst, whether a Si atom can be added into the crystallite or into the boundary is investigated with VASP calculations. Then, the properties of the Si-4N-4Ti structure and the Ti-Si-N coating are discussed. Finally, the formation of the Si-4N-4Ti structure and the Si distribution in TiN are studied with VASP calculations. 2.2 Si in the TiN Crystallite In this part of researches, a group of calculations were performed to investigate whether a Si atom could be added into the TiN crystallite.

120

X. Liu et al.

In this group of studies, the total energies and the cohesive energies of seven conﬁgurations were calculated. In order to make the energies of the seven conﬁgurations comparable, a supercell of TiN was used in the calculations of all seven conﬁgurations, which consisted of four layers of atoms, 8 N and 8 Ti per layer. For this TiN crystallite model, the lattice constant of the supercell was 4.262 Å in three directions (x, y, z). According to the method of Monkhorst-Pack [27], a grid of {5×5×5} k-points was employed. Three of the seven conﬁgurations had no Si addition: one was a conﬁguration of the perfect TiN crystallite; another one was a TiN conﬁguration in which a Ti atom was missing (TiN-1Ti); and the third one was a TiN conﬁguration in which a N atom was missing (TiN-1N). The other four of the seven conﬁgurations had a Si addition. A Si atom was set at a high symmetry site in the TiN supercell, namely the hollow site (HL) in Fig. 1(a) connected with 4 N and 4 Ti, or the plane hollow site (PHL) in Fig. 1(b) connected with 2 N and 2 Ti, or the Ti position (replacing a Ti atom) in Fig. 1(c), or the N position (replacing a N atom) in Fig. 1(d).

Fig. 1. The conﬁguration of a Si atom in the TiN crystallite: (a) a Si atom is at the hollow site (∼ Si-HL), the figure above is front view and the bottom one is top view; (b) a Si atom is at the plane hollow site (∼ Si-PHL), the figure above is front view and the bottom one is top view; (c) a Si atom occupies the Ti position (∼ SiOCP-Ti); (d) a Si atom occupies the N position (∼ Si-OCP-N). The square stands for Si, the triangle for Ti, and the circle for N

The data in Table 1 are the ab initio calculation results, the total energies and the cohesive energies of the seven conﬁgurations above. In the table, the cohesive energy Ecoh is deﬁned as Ecoh = nEN + mETi + lESi − E0

(1)

E0 is total energy of the conﬁguration concerned, EN , ETi , ESi are the energies of a N atom, a Ti atom, and a Si atom in vacuum, respectively. And n, m and l are the number of N, Ti, Si atoms contained in the supercell investigated. Because the supercells used for the seven conﬁgurations had the same volume, their total energies or cohesive energies could be directly compared with one another. From Table 1 above, an important result can be obtained. If there is no defect in the TiN crystallite, Si addition is possible in the form of ∼ Si-HL or of ∼ Si-PHL. The cohesive energies of the ∼ Si-HL structure and

The Basic Structure of Ti-Si-N Superhard Coatings

121

Table 1. The total energy and the cohesive energy of the conﬁguration investigated. The values are in electron volts (eV). E0 is the total energy; Ecoh is the cohesive energy; and dEcoh is the cohesive energy diﬀerence between the TiN crystallite and the other conﬁgurations. ∼ Si-HL means that a Si atom is at the hollow site in the TiN crystallite supercell; ∼ Si-PHL means that a Si atom at the plane hollow site in the TiN supercell; ∼ Si-OCP-Ti means that a Si atom occupies the Ti position in the TiN crystallite supercell; ∼Si-OCP-N means that a Si atom occupies the N position in the TiN supercell; TiN-1N means that a N atom is missing in the TiN supercell, and TiN-1Ti means that a Ti atom is missing in the TiN supercell TiN TiN-1Ti TiN-1N ∼ Si-HL ∼ Si-PHL ∼ Si-OCP-Ti ∼ Si-OCP-N E0 −629.98 −618.94 −618.98 −627.44 Ecoh 451.13 442.41 443.39 447.81 0.00 −8.72 −7.74 −3.31 dEcoh

−624.55 444.93 −6.20

−623.80 446.49 −4.63

−620.60 444.25 −6.88

that of the ∼ Si-PHL structure are both smaller than that of the TiN structure so that a Si atom cannot be added in the interstice of the TiN crystallite. This means that there is no interstitial solid solution of Si in the TiN crystallite in the equilibrium condition. If a Ti atom is missing in the TiN conﬁguration, i.e. the conﬁguration of TiN-1Ti, it is possible for a Si atom to occupy this vacant site and to form the conﬁguration of ∼ Si-OCP-Ti because the cohesive energy of ∼ Si-OCP-Ti is larger than that of TiN-1Ti. The ∼ Si-OCP-Ti structure is a substitution solid solution of Si in the TiN crystallite. The conﬁgurations of TiN-1N and ∼ Si-OCP-N can be analyzed in the same way.

Fig. 2. The conﬁgurations of a Si atom added in TiN crystalline supercell. (a) A Si atom is at the hollow site in the cell of TiN, noted ∼ Si-HL. A cross double tetrahedron Si-4N-4Ti is in the supercell center. The distances of Ti-N near by the Si-4N-4Ti show that the bond of Si-Ti is compressed. (b) A Si atom occupies the Ti position, noted ∼ Si-OCU-Ti. The Si atom is shown on the red point

122

X. Liu et al. Fig. 3. The cross double tetrahedron Si-4N-4Ti consists of a small tetrahedron Si-4N and a large tetrahedron Si-4Ti with the same Si in the center of the TiN cell. The tetrahedron Si-4N is smaller than the TiN cell and the tetrahedron Si-4Ti is larger than the TiN cell. Therefore, the cross double tetrahedron Si-4N-4Ti is an intrinsic mismatch to the B1 NaCl structure of TiN

Two conﬁgurations of the TiN crystallite with Si addition in Fig. 2 should be noted, in which the Si atom is shown on a red point. One conﬁguration is the structure of ∼ Si-OCP-Ti in Fig. 2(b). When Si occupies the position of Ti, it forms a six fold of Si-N bonds, Si-6N, which is an important factor for the strength of cubic silicon nitride c-Si3 N4 [28]. The other conﬁguration is the structure of ∼ Si-HL in Fig. 2(a). It is a new structure in which the Si atom is connected with 4 N and 4 Ti. This is a cross double tetrahedron, a small tetrahedron of Si and 4 N and a large tetrahedron of Si and 4 Ti with the same Si in the center as shown in Fig. 3. This cross double tetrahedron results in a relatively large distortion of the TiN cell so that the total energy of this conﬁguration is higher and the cohesive energy is smaller than that of the TiN crystallite. The details of this cross double tetrahedron will be discussed below. 2.3 Si in the TiN Boundary In this subsection, it is investigated whether a Si atom can be added into the TiN boundary. Generally speaking, the boundary structure is very complex and it is diﬃcult to create a boundary in the supercell. But it is known that the distances between atoms in the boundary or near the boundary are a bit larger or smaller than that in the crystallite. Therefore, the lattice of the supercell may be changed a bit in one direction to mimic a boundary in calculations. We deﬁned D as the boundary distance between the atom Ti and the atom N which were in the boundary. It should be noted that a boundary is under the non-equilibrium situation. To investigate the system states in a given boundary situation, the cohesive energies of all conﬁgurations should be compared with one another in the same distance D condition. In this subsection, a supercell like in the subsection above was employed which contained four layers of atoms, 8 N and 8 Ti per layer. The k-points setting was also the same as in the subsection above, a grid of {5 × 5 × 5}. But at this time, the supercell was a boundary supercell. The lattice constant of the supercell was 4.262 Å in two directions, while the lattice constant equated {D × 2} Å in the third direction producing a boundary situation. Six groups of calculations were performed.

The Basic Structure of Ti-Si-N Superhard Coatings

123

Table 2. The cohesive energies (eV) of the TiN and of conﬁgurations with Si addition in the boundary condition. D is the distance of Ti-N in the boundary D (Å) TiN ∼ Si-HL ∼ Si-PHL ∼ Si-OCP-Ti ∼ Si-OCP-N 1.900 2.000 2.131 2.350 2.550 2.750

430.19 445.47 451.13 442.92 430.51 419.77

424.25 440.46 447.81 443.74 439.49 435.73

420.30 436.84 444.93 443.19 439.43 434.84

426.86 441.37 446.49 438.31 427.55 423.00

420.08 436.81 444.25 437.88 428.67 423.74

In every group of calculations, the distance D was same. In the six groups, D equated 1.90 Å, 2.00 Å, 2.131 Å, 2.35 Å, 2.55 Å, and 2.75 Å, respectively. And in every group of studies, the cohesive energies of ﬁve conﬁgurations, i.e. TiN, ∼ Si-HL, ∼ Si-PHL, ∼ Si-OCP-Ti, and ∼ Si-OCP-N, were calculated. Because the supercells used in every group had the same volume, the cohesive energies of ﬁve conﬁgurations within the group could be directly compared with one another. Using this model, we studied the inﬂuence of Si addition in the boundary on the cohesive energy. The cohesive energies of ﬁve conﬁgurations in the six groups above are listed in Table 2. The cohesive energies are calculated with equation (1). The results in Table 2 are also presented in Fig. 4 which shows the trends clearly. The curves in Fig. 4 show that the favorable phases of all conﬁgurations are near the distance D = 2.131 which is under the TiN crystalline situation. Under the boundary situations, whether the distance D increases or decreases, the cohesive energies of all conﬁgurations are under

Fig. 4. The cohesive energies (eV) of the ﬁve conﬁgurations (TiN, ∼ Si-HL, ∼ SiPHL, ∼ Si-OCP-Ti, and ∼ Si-OCP-N) in six groups (D = 1.90 Å, 2.00 Å, 2.131 Å, 2.35 Å, 2.55 Å, and 2.75 Å) for the investigation of the system states under the boundary condition. D stands for the distance of Ti-N in the boundary

124

X. Liu et al.

the trend towards reduction. The emphasis here is on the comparison of the cohesive energy within the group, i.e. under the same boundary distance D condition. When the distance D is smaller than 2.35 Å, the cohesive energies of the TiN conﬁgurations is larger than those of other conﬁgurations with Si addition within the relevant groups. Evidently, the Si atom cannot be added into the crystallite of TiN or into the small distance boundary of TiN. Nevertheless, when the distance D is larger than 2.35 Å, the cohesive energy of the TiN boundary conﬁguration decreases rapidly and the cohesive energies of ∼ Si-HL and ∼ Si-PHL are both larger than that of the TiN boundary conﬁguration within the relevant groups. This indicates that the Si atom may move into the TiN boundaries when the distance between the boundary atoms is larger than 2.35 Å. The eﬀect of Si addition in the TiN boundary can be analyzed with Fig. 5. The structure in Fig. 5 is the calculation result with D = 2.35 in the horizontal direction. First we restrict ourselves to considering the Si-4N-4Ti in Fig. 5(a). It is known that the attractive force between the boundary atoms will increase when the boundary distance is large. This attractive force will lead to an increase of the distance between atoms which are near the boundary. Then the stress becomes high in the boundary local region. In this case, the Si atom moves into the boundary, it bonds itself with the boundary atoms Ti and N which belong to diﬀerent crystallites and form the cross double tetrahedron Si-4N-4Ti. This bonding will change the interaction relationship in the local region, weaken the eﬀect of the attractive force between the Ti-N atoms which are in the boundary, and reduce the changes in distance near the boundary. In other words, this bonding relaxes the stress and reduces the strain of crystallite near the boundary. The bond length of Si-Ti is 2.442 Å and the bond length of Si-N is 1.768 Å in Fig. 5(a). Both bond lengths are near the total relaxation state. It means that there is relatively little stress in the

Fig. 5. The conﬁgurations of a Si atom added into the TiN boundary (A change of the lattice constant is in the horizontal direction, a0 = 4.700 , corresponding to D = 2.35 ). (a) the structure of ∼ Si-HL, (b) the structure of ∼ Si-PHL. A obvious relaxation eﬀect is shown. The distances between Ti-N are large in the middle column, but they are relatively small in both side columns

The Basic Structure of Ti-Si-N Superhard Coatings

125

cross double tetrahedron Si-4N-4Ti so that there is little stress in this boundary local region. Clearly, the Si addition in the boundary leads to a stress relaxation and a strain reduction of both the boundary and the crystallite nearby. This should be the reason why the residual stress was very small [12] and the morphology was improved signiﬁcantly [11] in the experiments. A purely nitrogen coordinated Si-4N tetrahedron was suggested to be an interface in the Ti-Si-N composite coating [19]. We calculated the cohesive energies of this Si-4N tetrahedron with the crystallite supercell (no lattice changing) and with the boundary supercell (the lattice changing in one direction), respectively. The calculation results show that in the crystallite model as well as in the boundary model, the cohesive energies of the conﬁgurations with the Si-4N tetrahedron are smaller than those of the conﬁgurations with the Si-4N-4Ti cross double tetrahedron. Therefore, the interface in the Ti-Si-N composite coating is more likely to be in the Si-4N-4Ti form. This cross double tetrahedron Si-4N-4Ti has not yet been identiﬁed through the characterization of Ti-Si-N probes. This ab initio calculation result indicates what and where should be searched in the characterization. There are still three points to note. First, the cohesive energy of the ∼ SiPHL structure is very close to that of the ∼ Si-HL structure when D is larger than 2.35 Å. It shows that the structure of Si-2N-2Ti in Fig. 5(b) is also a favorable form in the TiN boundary. The Si atom can bond not only with 4N-4Ti but also with 2N-2Ti with almost the same level of energy in the boundary. It means that Si can adapt well to the complex boundary situation. In comparison with the Si-4N-4Ti structure, the Si-2N-2Ti structure has relatively small cohesive energy, especially when D = 2.75 . Second, the cohesive energy of the ∼Si-OCP-Ti structure, i.e. the Si-6N structure, is also larger than that of the TiN boundary structure when D is larger than 2.75 Å. This indicates that the Si-6N structure is a possible form in the TiN boundary. The Si atom tends to connect with the N atom in the ﬁlm growth process. Therefore, we state that the Si-6N structure instead of the Si-6Ti structure is a possible form in the TiN boundary. In comparison with the Si-4N-4Ti and the Si-2N2Ti structures, the Si-6N structure has even smaller cohesive energy. Third, our search for the basic structure of Ti-Si-N coatings is under the condition of a clear and neat boundary. The real boundary may be very complex. But generally, the complexity makes the boundary energy increase. If this clear and neat boundary has relatively low energy, our research result is acceptable. To summarize the above calculations and analysis, the basic structure of Ti-Si-N composite coatings is the TiN crystallite with the cross double tetrahedron Si-4N-4Ti in the boundary. The cross double tetrahedron Si-4N-4Ti is a new interface structure. Its properties are discussed in the next subsection.

126

X. Liu et al.

Fig. 6. The electron charge density distribution of Ti-Si-N in D = 2.35 condition: (a) the vertical direction is (001) and the horizontal direction is (100), (b) the vertical direction is (001) and the horizontal direction is (110), (c) the local charge density is parallel to (110) surface, and (d) the charge transfer is parallel to (110) surface (full curve: charge increase and dot-dashed curve: charge decrease)

2.4 The Properties of Si-4N-4Ti and the Properties of Ti-Si-N Coatings From the above description, it is obvious that the cross double tetrahedron Si4N-4Ti is diﬀerent from the B1- NaCl structure of TiN. Its Si-4Ti tetrahedron is larger than the TiN cell and its Si-4N tetrahedron is smaller than the TiN cell. This means the Si-4N-4Ti structure and the TiN structure are of an intrinsic mismatch. This is one characteristic of the Si-4N-4Ti. This intrinsic mismatch, as we will see below, leads to some covalent combinations. Another property of Si-4N-4Ti is the covalent combination. For ease of explanation, we name the atoms connected with Si as Ti1 and N1 and name those not connected with Si as Ti2 and N2. The Si addition leads to a transfer of the electron charge density as shown in Fig. 6(d). The charge density increases in the direction of Si-N1 as well as in the direction of Si-Ti1. The charge density increase between Si and N is relatively large, because the electronegativity diﬀerence between N and Si is larger than that between Si and Ti. The charge density in Fig. 6(b) and in Fig. 6(c) shows that the bond of Si-N1 is a strong covalent bond. The combination between Si and Ti1 is also a covalent bond which is not very strong. Because the bond length of Si-Ti1 is relatively long, the covalent bonds of Si-Ti1 put the Ti1 close to the N2 atoms and make their charge densities overlap as shown in Fig. 6(a). The mix of the charge density increases the covalent constituent of the combination and strengthens the bonding of Ti1 and N2 which belong to the same crystallite. In short, in

The Basic Structure of Ti-Si-N Superhard Coatings

127

the middle of the boundary, there is a group of covalent bonds of Si-N1 and Si-Ti1; on both sides of the boundary, there are the covalent combinations from the charge density mix of Ti1 and N2. The boundary that is full of the covalent combinations has great inﬂuence on the properties of the Ti-Si-N coatings. A direct eﬀect is that the covalent combination with Si-N1 and Si-Ti1 in the TiN boundary will strengthen the boundary, and thus the hardness of the Ti-Si-N coatings is enhanced. With the properties of Si-4N-4Ti, we can analyze the hardness enhancement. On the one hand, the more Si-4N-4Ti are in the boundary, the stronger the boundary is. In case of the same number Si4N4Ti in the unit boundary area, the boundary of the large size crystallite is stronger than that of the small size one with respect to restricting the boundary sliding. On the other hand, the material with small-sized crystallites has fewer dislocation sources in the crystallite and more boundaries to restrict the movement of the dislocation than the material with large-sized crystallites. The material with small-sized crystallites, therefore, is harder than the material with the large-sized crystallites in terms of resisting the deformation and the dislocation movement. Since the Si addition strengthens the boundary, the decrease of the crystallite size could lead to increased hardness. At some crystallite size, the strength of the crystallite and that of the boundary is balanced and the hardness reaches its maximum. If the crystallite size is reduced continually, the boundary becomes relatively weak, the boundary sliding begins to occur, and the total hardness begins to decrease. This is the situation which were shown in the experimental results of the Ti-Si-N coatings [9, 11]. To prove the 100 GPa hardness, further analysis and calculations are needed. Another obvious eﬀect is that the strong covalent combination between Si-N1 and between Ti1-N2 restrict the inter diﬀusion through the boundary. It seems likely that the Si-4N-4Ti locks the boundary region and leads to the absence of Ostwald ripening. Therefore, the Ti-Si-N coatings have very good thermal stability [10]. With the properties of the intrinsic mismatch and the covalent combination as well as the eﬀects of strengthening the boundary and restricting the inter diﬀusion, the cross double tetrahedron works really like a ﬁxation element to the rock salt structure. Most early transition metal nitrides and carbides have the B1-NaCl structure so that the cross double tetrahedron could be used to prepare nanocomposites of the transition metal nitrides and carbides with the addition of Si, Ge, or the like. The early transition metal nitrides and carbides combine the three types of bonding: ionic, covalent, and metallic[29]. They all have good conductivity. When the nanometer-sized (2–10 nm) crystallite of the transition metal nitrides and carbides is surrounded completely by the covalent boundary, their band structure will be changed. And then their electrical properties, optical properties and other properties could be engineered in a controlled way. The Fermi energy EF of TiN is 9.26 eVand the Fermi wavelength is 0.40 nm.

128

X. Liu et al.

If the beat frequency with the lattice is an order of magnitude larger, there should be the quantum eﬀects when the crystallite size is smaller than 4 nm. However, in order to adjust the band structure and hence to tailor the electrical and optical properties, a very precise control of the size and the shape of the nano-crystallite is necessary. Especially for quantum eﬀects, the crystallite size should be smaller than 4 nm, the diﬀerence in size should be very small, and the patterns should be very orderly. This is a big challenge to the fabrication. The cross double tetrahedron as a ﬁxation element will be very helpful in the mass production of this kind of nanostructures, nano-patterns, or quantum dots with the unconventional properties. This is the potential value of this cross double tetrahedron interface. 2.5 Si Adatom and TiN Island on TiN(001) Surface After the cross double tetrahedron Si-4N-4Ti was identiﬁed through above calculations and analysis, the formation of the Si-4N-4Ti structure on the TiN (001) surface was on our investigation agenda. The arrangement of a Si adatom and an island of 2N2Ti which were both on a TiN (001) surface was studied. A slab for this group of calculations was employed which contained ﬁve layers of atoms, containing 8 Ti and 8 N atoms per layer, and about 14.8 Å of vacuum space. The bottom two layers of the ﬁve atom layers were ﬁxed to mimic the conﬁning eﬀect of deeper layers, while the other top three layers were relaxed. The k-points setting was {5 × 5 × 1}. The adsorption energies of ﬁve conﬁgurations were calculated with this slab. One of the ﬁve conﬁgurations was an island of 2N2Ti on the TiN (001) surface. The other four conﬁgurations were a Si adatom and an island of 2N2Ti on the TiN(001) surface. The 2N2Ti island was in the center of the layer on the surface and a Si atom was set at the hollow site in the island [Fig. 7(a), noted Si-in-2N2Ti] or beside N of the island [Fig. 7(b), named Si-by-2N2Ti]. The other two conﬁgurations were 1) Si taking the Ti site of the island and Ti being beside N of the island [Fig. 7(c), Ti-by-2N1Ti1Si]; 2) Si taking the N site of the island and N being beside Ti of the island [Fig. 7(d), N-by-2Ti1N1Si]. The

Fig. 7. The conﬁgurations of a Si atom and a 2N2Ti island on TiN(001) surface: (a) a Si atom in the TiN island, Si-in-2N2Ti; (b) a Si adatom is beside the TiN island, Si-by-2N2Ti; (c) a Ti adatom is beside the island and the Si atom occupies the Ti position of the island, Ti-by-2N1Ti1Si; (d) a N adatom is beside the island and the Si atom occupies the N position of the island, N-by-2Ti1N1Si. The square stands for Si, the triangle for Ti, and the circle for N

The Basic Structure of Ti-Si-N Superhard Coatings

129

Table 3. The total energies Econf and the adsorption energies Ead (eV) of a 2N2Ti island on the TiN(001) surface and of the conﬁguration with a Si atom in the island or by the side of the island

Econf Ead

2N2Ti

Si-by-2N2Ti

Si-in-2N2Ti

−809.24 23.26

−814.52 27.76

−808.17 21.42

Table 4. The adsorption energies (eV) of a Si atom and a 2N2Ti island. Econf is the total energy of the conﬁguration; Ead is the adsorption energy

Econf Ead

Si-by-2N2Ti

Ti-by-2N1Ti1Si

N-by-2Ti1N1Si

−814.52 27.76

−813.80 27.05

−813.18 26.42

four conﬁgurations all had ﬁve adatoms, i.e. a Si adatom, two Ti adatoms, and two N adatoms, on the TiN(001) surface so that the adsorption energies of them could directly be compared with one another. The calculation results, adsorption energies of the ﬁve conﬁgurations above, are listed in Table 3 and the Table 4, where the adsorption energy Ead is deﬁned as Ead = na EN + ma ETi + la ESi + Es − Econf (2)

Econf is the total energy of the conﬁguration concerned; Es is the energy of the relaxed surface; EN , ETi , ESi are the energies of an N atom, a Ti atom, and a Si atom in vacuum, while na , ma , and la are the numbers of N, Ti, Si adatom under investigation. With these adsorption energy calculations, we want to know whether it is possible to build a Si-4N-4Ti structure on the free TiN (001) surface in the full relaxation situation. The adsorption energy of Si-in-2N2Ti is much smaller than that of Si-by-2N2Ti so that the Si adatom cannot move into the island of 2N2Ti and form a Si-4N-4Ti structure in this case. On the contrary, the conﬁguration of the Si adatom by the island of 2N2Ti [Fig. 8(b)] is a relatively stable structure. The cross double tetrahedron in Fig. 8(a) is already in the relaxation state but the TiN cell is, however, under large distortion so that this structure is under greater stress and has smaller adsorption energy than that of the Si outside the island. This fact indicates that the relaxation of the cross double tetrahedron could not reduce the energy of the system. Only if it leads to a stress relaxation and a strain reduction in the cells nearby, can the relaxation of the cross double tetrahedron have a positive eﬀect on the system energy reduction. The data in Table 4 presents a clear picture. According to the adsorption energy, a Si adatom is more likely to take the place beside N of the island when it hits the island of TiN in the formation process. If a Ti adatom comes by, Si may give the place to the Ti adatom to continually decrease the system

130

X. Liu et al.

energy or increase the adsorption energy of the whole cluster. Therefore, it is possible for the Si adatom to be always outside the island of TiN. In a following group of calculations, the transformations from Ti-by2N1Ti1Si to Si-by-2N2Ti [Fig. 9(a)] and from N-by-2Ti-1N-1Si to Si-by-2N2Ti [Fig. 9(b)] were studied in order to obtain the corresponding activation energies and investigate the phase segregation condition. First the diﬀusion path and the relevant activation energy in the case of the transformation from Ti-by-2N1Ti1Si to Si-by-2N2Ti were investigated. The elastic band method [30, 31] was used to calculate the energies of a series of points along the diﬀusion path. The adatom relaxation was limited to a plane perpendicular to the diﬀusion path between the initial site and the ﬁnal site while the surface atom relaxation was unconstrained. The calculation result, the activation energy for the transformation from Ti-by-2N1Ti1Si into Si-by-

Fig. 8. The conﬁgurations of a Si atom and a 2N2Ti island on the TiN(001) surface: (a) a Si atom in the hollow site of the 2N2Ti island; (b) a Si adatom beside N of the 2N2Ti island. The Si addition into the island of 2N2Ti results in a large distortion of the TiN cell

Fig. 9. The conﬁguration transformation from Ti-by-2N1Ti1Si or N-by-2Ti1N1Si into Si-by-2N2Ti: (a) the transformation from Ti-by-2N1Ti1Si into Si-by-2N2Ti i.e. the Ti atom moving to the Si position of the island and the Si atom moving to the site beside the N of the island; (b) the transformation from N-by-2Ti1N1Si into Si-by-2N2Ti i.e. the Si atom moving to the site beside the N of the island and the N atom moving to the Si position of the island. The square stands for Si, the triangle for Ti, and the circle for N

The Basic Structure of Ti-Si-N Superhard Coatings

131

2N2Ti, is 1.43 eV. If the deposition rate was 0.64 nm per second, the time for a monolayer (ML) deposition should be 0.33 s. The maximum mean free time td for the diﬀusion was expected to be one order of magnitude smaller than the time required forming a monolayer ML so that the maximum mean free time td was 0.033 s. The length of the diﬀusion path was 0.30 nm. With the activation energy above, the maximum mean free time td , the diﬀusion path length, and an attempt frequency of 1013 per second, the corresponding temperature for the transformation Ti-by-2N1Ti1Si into Si-by-2N2Ti was calculated. The result is 569.4 K or about 300 ◦C. Then the transformation from N-by-2Ti-1N-1Si to Si-by-2N2Ti was dealt with in the same way. The transformation from N-by-2Ti-1N-1Si to Si-by2N2Ti included two diﬀusion steps. The Si adatom ﬁrst moved to the site beside the N adatom and then the N adatom moved to the place in the island as shown in Fig. 9(b). The relevant activation energies of the two diffusion steps were also calculated with the elastic band method. The results are 0.57 eVand 1.08 eV, respectively. In a similar way, the corresponding temperatures were obtained. They are 244.4 K and 475.6 K or about −30 ◦C and 200 ◦ C, respectively. This means that in this case, if the deposition temperature is higher than 300 ◦ C, Si will stay outside of the island of TiN. It should be noted that it is a necessary condition but not a suﬃcient condition for the phase segregation. The adsorption energy calculations of the conﬁgurations of a Si adatom and a 2N2Ti island indicate that the Si-4N-4Ti structure cannot be formed in the island of TiN on the TiN (001) surface. And the studies of the conﬁguration transition condition show that it is very likely for the adatom Si to stay outside the island of TiN in the ﬁlm growth process, if the above parameters are taken in the deposition process. Therefore, a reasonable deduction is that Si stays outside the island of TiN and diﬀuses around the island. Only when it hits a boundary where the distance is large enough, may Si come into the boundary and form the cross double tetrahedron structure. The conditions for the Si-4N-4Ti formation require further study. 2.6 The Si Distribution The cross double tetrahedron Si-4N-4Ti is responsible for the distortion of the TiN cell so that the Si distribution in TiN could have an inﬂuence on the strain, the stress, and the cohesive energy of the system. When some of the Si4N-4Ti get together or adjoin to one another, the interaction of the distortions could lead to an increase in strain and a decrease in cohesive energy. To conﬁrm this, the cohesive energies of the seven conﬁgurations were calculated with equation (1). The supercell for this group of calculations contained six layers of atoms, containing 18 Ti and 18 N atoms per layer, for which a {3 × 3 × 3} Monkhorst-Pack k-points mesh was used. Two Si atoms were set both at hollow sites. The seven conﬁgurations were shown in Fig. 10: two Si atoms

132

X. Liu et al.

Fig. 10. (a) two Si atoms are at the hollow sites of two cells which are parallel adjoined in the same layer, PA; (b) two Si atoms are at cells which are parallel separated in the same layer, PS; (c) two Si atoms are at the hollow points of two cells which are plane diagonal adjoined in the same layer, DA; (d) two Si atoms are at cells which are plane diagonal separated in the same layer, DS; (e) two Si atoms are at the hollow points of two cells which are spatial diagonal adjoined in diﬀerent layers, SDA; (f ) two Si atoms are at cells which are spatial diagonal separated in diﬀerent layers, SDS; (d) two Si atoms are at the hollow points of two cells which are incline separated in the same layer, IS. The square stands for Si, the triangle for Ti, and the circle for N Table 5. The cohesive energy (eV) of diﬀerent distributions of two Si atoms in the TiN crystalline supercell. The meaning of the abbreviations PA, PS, and so on are shown in Fig. 10. E0 is the total energy and Ecoh is the cohesive energy PA

PS

DA

DS

SDA

SDS

IS

E0 −2118.98 −2120.28 −2119.19 −2119.80 −2117.40 −2120.22 −2120.12 1513.80 1515.10 1514.01 1514.62 1512.23 1515.04 1514.94 Ecoh

were in parallel cells in the same layer – adjoined (PA) [Fig. 10(a)] or separated (PS) [Fig. 10(b)]; two Si atoms were in plane diagonal cells in the same layer – adjoined (DA) [Fig. 10(c)] or separated (DS) [Fig. 10(d)]; two Si atoms were in spatial diagonal cells in diﬀerent layers – adjoined (SDA) [Fig. 10(e)] and separated (SDS) [Fig. 10(f)]; and two Si atoms were in the same layer with the incline separated form (IS) [Fig. 10(g)]. Because the supercells used in this group of calculations had the same volume, the cohesive energies of the seven conﬁgurations above could be directly compared with one another.

The Basic Structure of Ti-Si-N Superhard Coatings

133

Fig. 11. The distribution of two Si atoms in the TiN supercell: (a) two Si atoms in the parallel adjoined cells, (PA); (b) two Si atoms in the parallel separated cell, (PS). The two Si atoms in the adjoin cells result in an increase in strain on the local region

The calculation results, the cohesive energies of the seven conﬁgurations above, are listed in Table 5. We can see that the cohesive energies of Si distributions in TiN in the separated form are all larger than those in the adjoined form. This means that the segregation of the Si atoms in TiN is more stable. The distribution of the parallel-adjoined (PA) form is shown in Fig. 11(a) and the distribution of the parallel-separated (PS) form is shown in Fig. 11(b). It is clear that the distortions of two cells against each other in the PA form leads to a change in distance between the Ti atom and the N atom on the adjoined interface, which are 1.917 Å, 2.491 Å, and 2.304 Å as shown in Fig. 11(a), whereas the structure in PS form is in a relatively low strain state. In experiments with Ti-Si-N coatings, there is an interesting phenomena called “self-hardening”. The hardness of some Ti-Si-N coatings can increase upon annealing [32]. Because the separation of Si-4N-4Ti can lead to a decrease in the local strain and an increase in the cohesive energy of the system, this self-hardening may result from the eﬀect of the separation of the collesive Si atoms. The requirements that the TiN crystallite is completely surrounded by the Si-4N-4Ti and at the same time no Si-4N-4Ti is adjoined with one another impose a very strict demand on the preparation process. It is beyond doubt that the diﬀusion behavior of the adatom Si, Ti, and N during the Ti-Si-N coating formation process is an important topic. We will report our relevant research results in another paper. Although the strain increase introduced by the Si-4N-4Ti congregation in the TiN is not positive to the mechanical property of the composite, the interaction of the distortion from the cross double tetrahedron may be necessary to adjusting the electronic band structure. It paves the way for further research.

134

X. Liu et al.

3 Conclusion In order to ﬁnd out the atomic structure of Ti-Si-N nanocomposite coatings, the ab initio method has been used to perform some group investigations. The conclusions are as follows, 1. Calculations of the diﬀerent conﬁgurations of TiN with Si addition have been performed. The results show that (a) there is no interstitial solid solution of Si in the TiN crystallite in the equilibrium condition; (b) the basic structure of Ti-Si-N composite coatings is the TiN crystallite with the cross double tetrahedron Si-4N-4Ti in the boundary. 2. The formation of the Si-4N-4Ti in the ﬁlm growth process has also been studied. It indicates that Si-4N-4Ti cannot be formed in the island of TiN on the TiN (001) surface. If the deposition rate is less than 3 monolayers per second and the deposition temperature higher than 300 ◦C, the Si adatom may stay outside the island of TiN. Therefore, it is possible for Si to diﬀuse around the island of TiN and form Si-4N-4Ti in the TiN boundary. The condition for the Si-4N-4Ti cross double tetrahedron formation requires further study. 3. The distribution of Si atoms in TiN has been investigated. The cross double tetrahedron Si-4N-4Ti is responsible for the distortion of the TiN cell. When the Si-4N-4Ti cells join one another, the interaction of the distortion increases the local strain and decreases the cohesive energy of the system. On the contrary, the segregation of Si-4N-4Ti structures leads to an increase in cohesive energy of the system. 4. A new interface structure, the cross double tetrahedron Si-4N-4Ti, is an important ﬁnding in this research. The Si-4N-4Ti structure is an intrinsic mismatch to the B1-NaCl structure and it combines the boundary with the covalent bond. The covalent combination in the boundary can alleviate the stress and reduce the energy of both the boundary and the crystallite nearby. The Si-4N-4Ti strengthens the boundary and restricts the inter diﬀusion through the boundary so that the hardness and the thermal stability of Ti-Si-N nano-composite coatings are enhanced. Moreover, the cross double tetrahedron as a ﬁxation element may be very helpful in preparing and fabricating nano-composites of the early transition metal nitrides and carbides with Si addition. It can be used to control the size and the shape of the crystallite for improving not only the mechanical properties, but also other properties such as the electrical, optical, superconductive properties, etc. In fact, through the identiﬁcation of the cross double tetrahedron Si-4N-4Ti, we have found a new structure of, and a new method for, mass production of nanometer patterns or quantum dots. 5. The research ﬁnding reminds us that the digital structure design is very important for nano-composites. Besides the design of the crystallite size and shape, the interface or the combination of the crystallites should be

The Basic Structure of Ti-Si-N Superhard Coatings

135

calculated with atomistic methods. An intrinsic mismatch to, and a combination of, diﬀerent bonding types may be of special beneﬁt to a nanocomposite with unconventional properties. 6. This research also shows that process simulation before experiments is very important to nanostructure fabrication. It could promote an understanding of the basic structure and the formation process. Since the super hardness of Ti-Si-N coatings has been identiﬁed, a lot of research eﬀort has been made to study other similar structures, such as nc-VN/a-Si3 N4 , Ti-B-N [7], nc-MeN/soft phase [6], nc-TiC/a-C [9], or M-X-N [33] in order to search for other nanocomposite structures with even higher hardness. In fact, for all research of this kind, the process simulation should be performed before the experiment. Acknowledgements Liu thanks Professor H.-J. Warnecke and Professor E. Westkämper for their support during his research course in Germany. Thanks should also go to Mrs. M. Korell of Fraunhofer Gesellschaft, Mr. S. Haberhauer of NEC HPCE, Dr. A. Voter, Professor P. Kratzer, Professor G. Henkelman, Dr. D. Music, Dr. W.H. Xie, Ms. X. M Wang, and Mr. G.X. Wu for their support and the helpful discussions that they so graciously had with us. We also thank Dr. Dongying Wu and Dr. M. Schneider for their friendly help on the manuscript revision. This research was supported by the Deutsches Bundesprojekt, under Grant iﬀ12807 through HLRS of the University of Stuttgart, by Inner Mongolia Education and Research Foundation, under Grant NJ06077 through the Education Ministry of Inner Mongolia, PR China, and initially by the project CHN01/203 from the Fraunhofer Gesellschaft (FhG). This work was performed on the NEC SX-8 cluster at the HLRS of the University of Stuttgart.

References [1] [2] [3] [4] [5]

R. Kaner, J. Gilman, and S. Tolbert, Science 308, 1268 (2005) J. Haines, J.M. Leger, and G. Bocquillon, Annu. Rev. Mater. Res. 31, 1 (2001) C. Lu and Y. Mai, J. Mater. Sci. 41, 937 (2006) S. Zhang, D. Sun, Y Fu, and H. Du, Surf. and Coat. Tech., 167(2–3), 113 (2003) R.A. Andrievski, Int. J. of Refractory Metals and Hard Materials, 19(4–6), 447(2001) [6] J. Musil, Surf. and Coat. Tech., 125(1–3), 322 (2000) [7] S. Veprek, J. Vac. Sci. and Tech. A17, 2401 (1999) [8] S. Veprek, A. Niederhofer, K. Moto, T. Bolom, H.-D. Männling, P. Nesladek, G. Dollinger, and A. Bergmaier, Surf. and Coat. Tech. 133–134, 152 (2000) [9] J. Patscheider, MRS Bulletin/March 180, (2003) [10] S Veprek, In: R Riedel (ed), Handbook of Ceramic Hard Materials. Wiley-VCH Weinheim, 104 (2000)

136

X. Liu et al.

[11] S-Z. Li, Y-L. Shi, and H-R. Peng, Plasma Chem. and Plasma Proc., 12(3), 287 (1992) [12] A. Bendavid, P.J. Martin, J. Cairney, M. Hoﬀman, and A.C. Fischer-Cripps, Appl. Phys. A 81, 151–158 (2005) [13] F. Kaufmann, Dissertation an der Universität Stuttgart. Bericht Nr. 140, (2003) [14] S. Christiansen, M Albrecht, H.P. Strunk, and S. Veprek, J. Vac. Sci. and Tech. B16, 19 (1998) [15] S. Veprek and S. Reiprich, Thin Solid Films, 268 (1–2), 64 (1995) [16] Z-J. Liu, C.H. Zhang, and Y.G. Shen, J. of Appl. Phys. 95(2), 758 (2004) [17] S. Hao, B. Delley, S. Veprek and C. Stampﬂ, Phys. Rev. Lett. 97, 086102 (2006) [18] S. Hao, B. Delley, and C. Stampﬂ, Phys. Rev. B 74, 035424 (2006) [19] S. Hao, B. Delley, and C. Stampﬂ, Phys. Rev. B 74, 035402 (2006) [20] E. Westkämper, FtK2006 – Fertigungstechnisches Kolloquium Stuttgarter Impulse. in Stuttgart, 469-490 (2006) [21] G. Kresse and D. Joubert, Phys. Rev. B 59, 1758 (1999) [22] G. Kresse and J. Hafner, Phys. Rev. B 47, 558 (1993); 49, 14251 (1994) [23] G. Kresse and J. Furthmuller, Comput. Mater. Sci. 6, 15 (1996); Phys. Rev. B 54, 11169 (1996) [24] J.P. Perdew, J.A. Chevary, S.H. Vosko, K.A. Jackson, M.R. Pederson, D.J. Singh, and C. Filohais, Phys. Rev. B, 46, 6671 (1992) [25] W. Lengauer, In: R. Riedel (ed), Handbook of Ceramic Hard Materials. WileyVCH Weinheim, 202 (2000) [26] C. Stampﬂ, W. Mannstadt, R. Asahi, and A.J. Freeman, Phys. Rev. B 63, 155106 (2001) [27] H.J. Monkhorst and J.D. Pack, Phys. Rev. B 13, 5188 (1976) [28] A. Zerr, G. Miehe, G. Serghiou, M. Schwarz, E. Kroke, R. Riedel, H. Fueß, P. Kroll, and R. Boehler, Nature. 400, 22 (1999) [29] Hugh O. Pierson, Handbook of Refractory Carbides and Nitrides, Noyes Publications, Westwood, NJ, U.S.A. 174 (1996) [30] G. Mills, H. Jónsson, and G.K. Schenter, Surf. Sci. 324, 305 (1995) [31] G. Henkelman and H. Jónsson, J. Chem. Phys. 113, 9978 (2000) [32] S. Veprek, M.G.J. Veprek-Heijman, P. Karvankova and J. Prochazka, Thin Solid Films, 476 (1), 1 (2005) [33] C.S. Sandu, R. Sanjinés, M. Benkahoul, F. Medjani and F. Lévy, Surf. and Coat. Tech., 201(7), 4083 (2006)

Shared Memory Parallelization of the Multi-configuration Time-dependent Hartree Method and Application to the Dynamics and Spectroscopy of the Protonated Water-dimer Michael Brill1 , Oriol Vendrell1 , Fabien Gatti2 , and Hans-Dieter Meyer1 1

2

Theoretische Chemie, Physikalisch-Chemisches Institut, Universit¨ at Heidelberg, INF 229, D-69120 Heidelberg, Germany [email protected] LDSMS (UMR 536-CNRS), CC 014, Universit´e de Montpellier II, F-34095 Montpellier, Cedex 05, France

1 Introduction For the understanding of physical processes on a molecular scale it is convenient to study the underlying dynamics by wavepacket propagation. Many diﬀerent propagation schemes have been developed in the past, reaching from the numerically exact standard method, that can only treat very small systems to approximate ones like the time dependent Hartree method (TDH). One of those methods, the multi-configuration time-dependent Hartree (MCTDH) method, has been developed in Heidelberg and it has proved its capability to treat large systems ( 9 degrees of freedom) fully quantum mechanically and with high accuracy. From the date of its inception the MCTDH algorithm was programmed and used as a single-processor application. In this report we discuss our recent eﬀorts in the direction of obtaining an eﬃcient, shared-memory parallelized version of the code. We give also full account of the results obtained in our investigations on the Zundel cation (H5 O+ 2 ). This system is a small prototype for the proton-transfer process in water systems. Thanks to the parallelization of the MCTDH code this challenging (15 internal degrees of freedom) and important system can now be fully quantum-dynamically simulated. Specifically, we report on a full quantum simulation of the infrared (IR) absorption spectrum of the protonated water dimer (H5 O+ 2 ) in the spectral range 0–4000 cm−1 . The doublet-peak feature around 1000 cm−1 , which was not understood and subject of debate, is reproduced, assigned and explained. Strong couplings between the proton-transfer and other modes are clearly identified, and their role in the spectrum is clarified. Low-frequency anharmonic torsions are discussed and described in detail for the first time.

142

M. Brill et al.

The report is structured as follows. The MCTDH method [1, 2, 3] is briefly described in Sect. 2. The parallelization of the MCTDH code and algorithm are covered in Sect. 3. Section 4 reports on the results on the dynamics and IR spectrum of the H5 O+ 2 cation and some conclusions and future perspectives are given in Sect. 5.

2 MCTDH: Equations of Motion and Implementation 2.1 Equations of Motion In the MCTDH framework [1, 2, 3] the wavefunction is expanded as: Ψ (Q1 , . . . , Qf , t) =

n1

j1 =1

···

nf

Aj1 ...jf (t)

f

(κ)

ϕjκ (Qκ , t),

(1)

κ=1

jf =1

(κ)

where ϕjκ (Qκ , t) are the so called single particle functions (SPF) that depend on the κ-th degree of freedom and time. The expansion coeﬃcients Aj1 ...jf (t) are time-dependent as well. The SPFs are represented by linear combinations of a primitive basis: (κ)

ϕjκ (Qκ , t) =

Nκ

(κ)

(κ)

ciκ (t)χiκ (Qκ ).

(2)

iκ

Because of the time-dependence of the expansion coeﬃcients and the SPFs the wavefunction is not unique. This problem can be solved by introducing constraints [3]. These constrains can be chosen such that the SPFs remain orthonormal for all times. With aid of the Dirac-Frenkel variational principle, δΨ | H −i∂/∂t |Ψ = 0, the coeﬃcients and the SPFs are variationally optimized and the equations of motion are determined. To simplify the resulting equations some notation has to be established. The composite index J and the configurations ΦJ : AJ = Aj1 ...jf

and ΦJ =

f

(κ)

ϕjκ ,

(3)

κ=1

and the projector: P (κ) =

nκ (κ) (κ) ϕj . ϕj

(4)

j=1

The single hole functions, κ κ (1) (κ−1) (κ+1) (f ) (κ) AJlκ ϕj1 · · · ϕjκ−1 ϕjκ+1 · · · ϕjf = AJlκ ΦJ κ , Ψl = J

J

(5)

Parallelization of the MCTDH Method

143

are used to define the mean fields, (κ) (κ) (κ) Hjl = Ψj H Ψl ,

(6)

and the density matrices,

κ (κ) (κ) (κ) ̺jl = Ψj Ψl = A∗Jjκ AJlκ ,

(7)

J

where Jlκ denotes a composite index J with the κth entry set l and κJ is the sum over the indices excluding the κth degree of freedom. (κ) (κ) With the constraints ϕj |ϕ˙ k = 0 [3], the equations of motion are expressed by the following non-linear, coupled diﬀerential equations: iA˙ J = ΦJ | H |ΦL AL , (8) L

(κ) iϕ˙ j

=

lm

(κ)

−1

(1 − P (κ) )(̺(κ) )jl Hlm ϕ(κ) m .

(9)

2.2 Product Representation of the Hamiltonian To achieve its full numerical power, the MCTDH-algorithm needs a special structure of the used operators. All of them, especially the Hamiltonian, are represented in product structure: ˆ= Ω

S

k=1

ck

f

(κ)

ω ˆk ,

(10)

κ=1

ˆ is an operator in product representation, ck are the expansion coefwhere Ω (κ) ficients and ω ˆ k are the one dimensional operators that only act on the κth degree of freedom. A matrix element then reads: f S ˆ ˜ (κ) (κ) (κ) ωk ϕ˜lκ ϕjκ ˆ ··· ··· A∗j1 ...jf ck A˜l1 ...lf Ψ Ω Ψ = k=1

j1

jf

l1

lf

κ=1

(11) Because of the product structure of the wavefunction and the operator the ˆ Ψ˜ , which would be very elaborate to evalumulti-dimensional integral, Ψ |Ω| ate directly by quadrature, is replaced by a sum of products of one-dimensional integrals. The evaluation of the latter is very fast. This approach is used to evaluate the Hamiltonian matrix elements ΦJ | H |ΦL and the mean (κ) fields Hlm . Kinetic energy operators are in general of product form, but the potential part of the Hamiltonian often is not. To cope with this problem the MCTDH package includes a fitting algorithm called potfit [4, 5] that is used to transfer multidimensional functions into product representation.

144

M. Brill et al.

2.3 The Constant Mean-ﬁeld Integration Scheme The constant mean field (CMF) integration scheme makes use of the fact that the Hamiltonian matrix elements and the product of the inverse density with the mean-field matrices change much slower than the MCTDH coeﬃcients and the SPFs. Hence a wider meshed time discretisation can be chosen for the propagation of these objects, i.e. these quantities are kept constant over the so called update time (1fs, say). By this the coupled equations (8) and (9) are decoupled in f + 1 sets of diﬀerential equations, one set for the expansion coeﬃcients and one set for the SPFs of each degree of freedom. The constant mean-field equations of motion then read: ¯ JL AL (t) iA˙ J (t) = (12) K L

= 1 − P (κ) ⎛

iϕ˙ κj (t)

(13)

(κ)

× ⎝h(κ) ϕj (t) +

nκ

−1

ρ¯(κ)

k,l=1

jk

s r=1

⎞

¯ (κ) h(κ) ϕ(κ) (t)⎠ , H l rkl r

where the KJL are the Hamiltonian matrix elements: KJL = ΦJ | H |ΦL =

s

cr

r=1

f

κ=1

(κ)

(κ) (κ) ϕjκ h(κ) ϕ r lκ

(14)

and the Hrjl are the mean-field matrix elements: (κ)

Hrjl = cr

κ J

A∗Jjκ

l1

(1) (1) (f ) ) (f ) ϕj1 h(1) ··· ϕjf h(f ALκl , (15) r ϕl1 r ϕlf lf

where the “· · · ” does not contain a sum over lκ . The mean-field matrix elements are connected with the mean-field matrix via: (κ)

Hjl =

s r=1

(κ)

Hrjl h(κ) r .

(16)

The bar in (12) and (14) indicates that the corresponding term is held constant over the constant mean-field integration step. The CMF scheme actually used in the calculations is more complicated, because first order errors in the update time have to be removed. For details see [3]. However, the primitive CMF scheme discussed here, is suﬃcient to understand the parallelization strategies to be discussed next.

Parallelization of the MCTDH Method

145

3 Shared Memory Parallelization of MCTDH To increase the eﬃciency of the MCTDH-program the code is parallelized. This work is still in progress. Due to the complicated structure of the diﬀerential equations and the intricate program structure shared memory parallelization is the method of choice. The parallelization is done using POSIX-threads. 3.1 Work Load Between the Main Parts of MCTDH As described above the propagation of the wavefunction consists of three main tasks: • • •

Calculation of the Hamiltonian matrix and the mean-fields: (14, 15, 16), propagation of the MCTDH expansion coeﬃcients: (12), propagation of the single particle functions: (14).

These three parts together take the major part of the computation time (≥ 99%). But unfortunately the work load between these parts varies strongly with the physical system that is treated, for example: •

•

•

H2 +H2 -scattering. In this case more than 98% of the computation time is spend for the computation of the mean-fields and the propagation of the MCTDH coeﬃcients. Only a negligible amount of time is spend for the propagation of the SPFs. Quantum molecular dynamics of C2 H4 . Here the computation time is distributed almost equally among the three tasks (computation of the meanfields: 20.1%, propagation of the coeﬃcients: 41.5%, propagation of the single particle functions: 37.5%, total: 99.1%). Quantum molecular dynamics of H5 O+ 2 . The quantum mechanical investigation of the Zundel-cation is the main topic of the calculations made in Karlsruhe and will be presented in detail below. The computation time is distributed in the following way: mean-field calculation: 34.2%, propagation of the expansion coeﬃcients: 31.8%, propagation of the single particle functions: 33.8%, total: 99.8%.

Whereas the first case is a border case the two others are more common cases. Hence all three parts have to be parallelized. 3.2 Parallelization of the Mean-ﬁeld Calculation and the Propagation of the MCTDH Coeﬃcients The parallelization of the mean-field calculation and of the propagation of the MCTDH coeﬃcients are very similar. Both parts essentially consist of the evaluation of multi dimensional integrals, see (6) and (14). As already discussed, all operators, especially the Hamiltonian, are represented in product structure (10). The product structure of the Hamiltonian

146

M. Brill et al.

and the wavefunction speeds up the calculation of mutli-dimensional integrals, see Sect. 2.2. To evaluate the integrals that are needed for the evaluation of the mean-fields (6) and the propagation of the expansion coeﬃcients (14) a summation over the Hamiltonian terms is performed in each case. To parallelize these integrations the summation over the Hamiltonian terms is distributed to the available processors. 3.3 Parallelization of the Propagation of the Single Particle Functions The parallelization of the propagation of the single particle functions turns out to be more cumbersome. In this part there is no obvious sum to be parallelized as in the two other cases. For a propagation step several objects have to be calculated (see (14)): • The inverse density matrix, • the projection operator, • the product of the mean-field matrices with the SPFs. The latter task, that usually takes more than half of the computation time spend for the single particle function propagation, is already parallelized. Here also a loop over the Hamiltonian terms must be performed, but only over the kinetic part of the Hamiltonian which may be rather short. This loop has been parallelized. However the parallelization of the two other parts may need a rearrangement of the data structure and other changes in the code. Hence the parallelization of the single particle function propagation is not yet completed. 3.4 Results of the Parallelization To get an idea how good the parallelization of the MCTDH-algorithm works the three above mentioned cases (H2 +H2 -scattering and molecular dynamics of ethene and the Zundel-cation) have been performed in parallel. The speedup has been fitted to Amdahl’s law, that gives the ideal speedup of a parallel program with a parallel part of p and a serial part of q (q + p = 1). In this law no overhead eﬀects of the parallelization are included. The H2 +H2 Scattering In Fig. (1) the speedup for the H2 +H2 -scattering case is shown. In this case the parallel part of the program is about p = 0.971, what is close to the 98% that are mentioned above (Sect. 3.1). This means in this case the program is very well parallelized. Since the propagation of the single particle functions doesn’t play a role the main work of MCTDH is done in the mean-field calculation and the propagation of the expansion coeﬃcients. As explained above these

Parallelization of the MCTDH Method

147

Fig. 1. Speedup behaviour of the H2 +H2 -scattering calculation (circles) and the C2 H4 propagation (squares). The circles/squares are the measured values, the solid lines are the according ﬁts to Amdahl‘s law and the dashed line is the ideal (linear) speedup

tasks are parallelized by distributing the summation over the Hamiltonian terms to the diﬀerent processors. In the present case the number of terms is approximately 3500 and the parallel part of the program is dominant. The C2 H4 Molecular Dynamics Calculation Figure (1) shows the speedup in the C2 H4 -case. Now the parallel part of the program is merely p = 0.716 what is much lower than the 99.1% mentioned in Sect. 3.1. In the present case there are two reasons which explain the poorer parallelization. Firstly, the number of Hamiltonian terms is rather low in the present case (∼ = 270 terms) what leads to a small parallel part in the calculation of the mean-fields and the propagation of the expansion coeﬃcients. Secondly the propagation of the single particle functions takes a major part (37.5%). As explained in Sect. 3.3 this is the part of the program that is not fully parallelize yet. Finally, the number of kinetic terms in the Hamiltonian is only 34. The H5 O+ 2 Propagation and Relaxation The Zundel-cation (H5 O+ 2 ) is the molecular system that is in the focus of the calculations that have been performed in Karlsruhe. Figure (2) shows the speedup behaviour for the propagation and relaxation calculations of H5 O+ 2. First the term “relaxation” has to be explained. A relaxation is, in principle, a propagation in imaginary time that leads to eigenenergies and eigenfunctions of the considered molecular system. Excited states are obtained by diagonalizing the Hamiltonian in the set of the thus optimized configurations [6]. Both types of calculation, propagation and relaxation, have been necessary for the investigation of the Zundel-cation. According to the fit the parallel part of the program is about p = 0.912. Due to the high number of Hamiltonian terms, which is about 1600 and hence in the same order of magnitude as in the H2 +H2 case, the mean-field calculation and the propagation of the expansion coeﬃcients is well parallelize. The part of calculation time that is spend for the propagation of the single particle

148

M. Brill et al. Fig. 2. The speedup behaviour in the case of the Zundel-cation for the propagation and the relaxation calculations. The circles indicate the propagation, the squares indicate the relaxation. The solid lines are the corresponding ﬁts, whereas the dashed line shows the ideal (linear) speedup

functions (33.8%) is comparable to the time spend in the C2 H4 case. Because the parallelization of this part is not yet finished the propagation in the case of the Zundel-cation is not as well parallelized as in the H2 +H2 case, but still better than in the C2 H4 -case. Because most routines used for the propagation are reused for the relaxation, this part is parallelized as a byproduct. In the present case the parallel part of the relaxation run is p = 0.884.

4 Dynamics and Infrared Spectrum of the Zundel Cation 4.1 H5 O+ 2 : Introduction and Motivation The dynamics and spectroscopy of water clusters [7, 8, 9] of diﬀerent geometries as well as in bulk water [10] has attracted much research eﬀort, mainly due to the major importance that proton transfer and general properties of water have in central areas of chemistry and biology. Accurate measurements of infrared (IR) spectra of small-size protonated water clusters have become possible in recent years [7, 11, 12, 9]. The protonated water dimer, H5 O+ 2 , the smallest protonated water cluster, has been recently object of intense study. The infrared (IR) spectrum of the system has been measured in the gas phase, either using multiphoton dissociation techniques [7, 11] or measuring the vibrational predissociation spectrum of H5 O+ 2 ·RGn clusters with RG=Ar,Ne [12, 9]. The obtained spectra cannot be consistently assigned in terms of fundamental frequencies and overtones of harmonic vibrational modes due to the large-amplitude anharmonic motions and couplings of the cluster. Hence, more sophisticated theoretical approaches are required. Several theoretical studies have been conducted over the last years in order to understand and assign the IR spectrum of the cation [13, 14, 15, 9, 16, 17]. Recent measurements of the IR predissociation spectrum of the H5 O+ 2 cation in argon-solvate [12] and neon- and argon-solvate [9] conditions present spectra with a simpler structure than the multiphoton IRMPD ones. It is expected that the spectrum of the H5 O+ 2 ·Ne1 complex is close to the linear

Parallelization of the MCTDH Method

149

absorption spectrum of the bare cation [9]. This spectrum features a doublet structure in the region of 1000 cm−1 made of two well-defined absorptions at 928 cm−1 and 1047 cm−1 . This doublet structure has not yet been fully understood, although the highest-energy component has been already assigned to the asymmetric proton-stretch fundamental ([O-H-O ]) [9]. Another doublet structure appears in the experimental spectrum in the region 1800–1900 cm−1 , that would be related to the water bending motions. At the same time, recent classical-dynamics simulations on accurate potential energy surfaces (PES) have shown that the [O-H-O ] motion features large amplitude displacements strongly coupled to other modes of the system. The central-proton displacement would then be involved in most of the lines appearing in the IR spectrum, since this motion relates to the largest changes in the dipole moment of the cation [16, 17]. Despite recent theoretical and experimental studies on this system, little is known about the lowest frequency region, between 0 and 900 cm−1 . Modes vibrating in this frequency range are strongly anharmonic, and thus harmonic-vibrational analysis results are of little value in shedding light on that matter. These low frequency modes may play also an important role in combination with the [O-H-O ] fundamental. Such a possibility has been already suggested [11, 14, 16], but just which modes would participate in such combinations, and how, is still a matter of discussion. In this report we simulate of the IR linear absorption spectrum of the −1 H5 O+ by means of a full2 cation in the range between 0 to 4000 cm dimensional 15D quantum-dynamical treatment, using a set of curvilinear internal coordinates to tackle the anharmonicities and large-amplitude motions of the system. We also report the calculation of some fundamental frequencies and overtones, namely eigenenergies and eigenstates of the vibrational Hamiltonian, which are a key to the interpretation and assignment of diﬀerent spectral features. Our motivation is to illuminate the interpretation of the IR spectral signatures, which is still an open question. At the same time we want to stress that it is now possible, by state-of-the-art quantum-dynamical methods, to solve a 15D floppy and anharmonic molecular system with high accuracy, both in the time-dependent as well as in the time-independent representation. To address the problem, we make use of the potential energy surface (PES) and dipole-moment surfaces recently developed by Bowman and collaborators [15], which constitute the most accurate ab initio surfaces available to date for this system. 4.2 Deﬁnition of the System Hamiltonian Kinetic Energy Operator The Hamiltonian of the system is expressed in a set of polyspherical coordinates based on Jacobi vectors [18]. It is found that only after the introduction of such a curvilinear set of coordinates an adequate treatment of the anharmonic large-amplitude vibrations and torsions of the molecule becomes

150

M. Brill et al.

possible. The kinetic energy operator is exact for J = 0, and the derivation of its lengthy formula (674 terms) will be discussed elsewhere. The correctness of the operator implemented was checked by comparison with data generated by the TNUM program [19]. The internal coordinates used are: the distance between the centers of mass of both water molecules (R), the position of the central proton with respect to the center of mass of the water dimer (x,y,z), the Euler angles defining the relative orientation between the two water molecules (waggings: γa , γb ; rockings: βa , βb ; internal relative rotation: α) and the Jacobi coordinates which account for the particular configuration of each water molecule (r1(a,b) , r2(a,b) , θ(a,b) )) where r1x is the distance between the oxygen atom and the center of mass of the corresponding H2 fragment, r2x is the H–H distance and θx is the angle between these two vectors. These coordinates have the great advantage of leading to a much more decoupled representation of the PES than a normal-mode based Hamiltonian.

Fig. 3. Set of Jacobi vectors in terms of which the kinetic energy of the system is expressed. The set of internal coordinates used corresponds to the length of these vectors and relative angles

Potential Energy Surface As outlined above, the wavefunction is represented by products of SPFs which in turn are represented by discrete variable representation (DVR) grids. The total primitive product-grid consists of 1.3 × 1015 points. This number makes clear that the potential must be represented in a more compact form to make calculations feasible. We choose to represent the PES as follows: the coordinates are divided in five groups, g1 ≡ [x, y, z, α], g2 ≡ [γa , γb ], g3 ≡ [R, βa , βb ], g4 ≡ [r1a , r2a , θa ] and g5 ≡ [r1b , r2b , θb ]. The potential is then expanded as [20]: Vˆ (c) = vˆ(0) +

5

(1)

vˆi (gi ) +

i=1

(3) +ˆ vz,2,3 (z, g2 , g3 )

5 4

(2)

vˆij (gi , gj )

i=1 j=i+1

(17)

Parallelization of the MCTDH Method

151

where c ≡ [g1 , . . . , g5 ]. The vˆ(0) term is the energy at the reference geom(1) etry. The vˆi terms are the intra-group potentials obtained by keeping the (2) coordinates in other groups at the reference geometry, while the vˆij terms account for the group-group correlations. The potential with up to secondorder terms gives already a very reasonable description of the system. The (3) vˆz,2,3 term accounts for three-mode correlations between the displacement of the central proton, the distance between both water molecules and the angular wagging and rocking motions. This PES representation may be sequentially improved in a convergent series by adding more correction terms where coordinates belonging to three or more diﬀerent groups are allowed to vary simultaneously. However, the PES in (17) is found to reproduce the full potential very well, providing a converged zero-point energy of 12376.3 cm−1 , 16 cm−1 below the reported Diﬀusion-Monte-Carlo result [21] on the full potential. 4.3 Calculation of Speciﬁc Vibrational States by Means of Improved Relaxation In Fig. 4 the probability-density projection on the wagging coordinates is shown for the ground vibrational state (g0 ), as well as for one of the two fundamental states (w1a ,w1b ) of the wagging modes, which are degenerate vibrational states with an energy of 106 cm−1 . The energies of the next three wagging-mode states (w2 ,w3 ,w4 ) are, respectively, 232, 374 and 421 cm−1 . w2 , w3 and w4 are all 2-quanta excited states and become degenerate in a harmonic approximation. State w3 is shown in Fig. 4c. State w2 has an energy that nearly doubles the energy of the w1x states, since it has been seen to roughly correspond to one quantum in state w1a and one quantum in state w1b . The strong anharmonicity of the wagging motions can be further appre-

Fig. 4. Probability density of the ground vibrational state (a), ﬁrst (b) and third (c) wagging-mode states projected onto the wagging coordinates γa and γb . Ground vibrational state (d), splitting state (e) and second excited internal-rotation state (f ) projected onto the α coordinate. An extended scale (×10) is used to clarify existence and position of nodes

152

M. Brill et al.

ciated in the progression of w2 , w3 and w4 vibrational-state energies. In addition, the harmonic-analysis energies of the two lowest wagging-fundamentals w1a and w1b are around 300 cm−1 larger than the MCTDH result and do not account for their degeneracy, since harmonic modes are constructed taking as a reference the C2v absolute minimum. The system, however, interconverts between equivalent C2v minima and other stationary points through low-energy barriers (wagging motions and internal rotation), which leads to a highly symmetric ground-state wavefunction. The vibrational levels of H5 O+ 2 can be labeled according to the symmetry group G16 , which is related to the D2d point group but with allowed permutation of the H-atoms within each of the two monomers [22]. The two lowest excited wagging/rocking modes transform according to an E representation within this symmetry group. (−) The first two excited states associated to the internal rotation (g0 , i1 ) (−) have energies of 1 and 160 cm−1 , respectively. Here g0 is the splitting state whose probability density along α is shown in Fig. 4e, while the probability density for i1 is shown in Fig. 4f. The first two fundamentals of the symmetric stretch ([O-O ], R coordinate) have energies of 550 and 1069 cm−1 respectively, while the rocking fundamentals, which are degenerate, have an energy of 456 cm−1 . 4.4 Infrared Spectrum: Calculation and Analysis Figure 5 presents the IR predissociation spectrum of the H5 O+ 2 ·Ne complex [9] and the MCTDH spectrum of H5 O+ in the range 700–1900 cm−1 . The 2 MCTDH spectrum is obtained in the time-dependent picture by Fourier transformation of the autocorrelation of the dipole-operated intial state [23]: E I(E) = 6 c ǫ0 2

∞

exp(i (E + E0 ) t/)

−∞

ˆ t/)|Ψμ,0 dt × Ψμ,0 | exp(−i H

(18)

Fig. 5. Predissociation spectrum of the H5 O+ 2 ·Ne complex [9] (top) and MCTDH (bottom)

Parallelization of the MCTDH Method

153

where E0 is the ground-state energy and |Ψμ,0 ≡ μ ˆ |Ψ0 . The MCTDH spectrum shows a good agreement with the experimental spectrum. The agreement on the doublet structure around 1000 cm−1 is very good, and the position of the doublet at 1700–1800 cm−1 is also in good agreement, despite the relative intensities being larger in MCTDH. The doublet absorption at around 1000 cm−1 deserves a deeper analysis. Due to the high density of states, it was not possible, by means of improved relaxation, to obtain the fully converged states, but reasonably good approximations to the wavefunctions of the low-energy (|Ψdl , 930 cm−1 ) and high energy (|Ψdh , 1021 cm−1 ) eigenstates of the doublet were computed. Even though these wavefunctions contain all the possible information on the two states, their direct analysis becomes complex due to the high dimensionality of such objects. In order to obtain a fundamental understanding of the observed bands, zeroth-order states were constructed by operating with zˆ on the ground state: |Φz = zˆ|Ψ0 N , where N is a normalization constant, ˆ − R0 ) on the third excited wagging state w3 : and by operating with (R ˆ |ΦR,w3 = (R − R0 )|Ψw3 N . The two eigenstates corresponding to the doublet were then projected onto these zeroth-order states. Note that |Φz is characterized by one quantum of excitation in the proton-transfer coordinate whereas |ΦR,w3 by one quantum in [O-O ] and three quanta in the wagging motion. The corresponding overlaps read: |Φz |Ψdl |2 = 0.20, |ΦR,w3 |Ψdl |2 = 0.53 and |Φz |Ψdh |2 = 0.48, |ΦR,w3 |Ψdh |2 = 0.12. One should take into account that these numbers depend on the exact definition of the zeroth-order states, which is not unique. Also, the zeroth-order states do not span the same space as the two eigenstates, so the overlaps do not add up to 1. However, they provide a clear picture of the nature of the doublet: the low-energy band has the largest contribution from the combination of the symmetric stretch and the third excited wagging (see Fig. 4), whereas the second largest is the proton-transfer motion. For the high-energy band the importance of these two contributions is reversed. Thus, the doublet may be regarded as a Fermi resonance between two zero-order states which are characterized by (R, w3 ) and (z) excitations, respectively. The reason why the third wagging excitation plays an important role in the proton-transfer doublet is understood by inspecting Fig. 4c. The probability density of this state has four maxima, each of which corresponds to a planar conformation of H2 O-H+ (H3 O+ character) for one of the waters, and a bend conformation (H2 O character) where a lone-pair H2 O orbital forms a hydrogen bond with the central proton. When the proton oscillates between the two waters, the two conformations exchange their characters accordingly. The simulated spectrum in the range between 0 and 4000 cm−1 is depicted in Fig. 6. The region below 700 cm−1 has not yet been accessed experimentally. Direct absorption of the wagging motions, excited by the perpendicular components of the dipole operator, appears in the range between 100– 200 cm−1 . The doublet starting at 1700 cm−1 is clearly related to bending

154

M. Brill et al.

Fig. 6. Simulated MCTDH spectrum in the range between 0 and 4000 cm−1 . Absorption is given in absolute scale in mega-barns (Mb)

motions of the water molecules, but its exact nature is still to be addressed. The MCTDH spectrum also shows the absorptions of the OH stretchings starting at 3600 cm−1 .

5 Conclusions The parallelization of the MCTDH-program is not yet finished, especially the parallelization of the propagation of the single particle functions must be improved. The performance of the parallelization depends on the physical system that is investigated. In the case of the Zundel-cation the parallelization performs well for both the relaxation and propagation calculations. If eight processors are used the calculation is accelerated by a factor of 4.4 in the case of relaxations and by a factor of 4.7 for the propagations. Using the parallelized version of MCTDH it has been possible to simulate the IR absorption spectrum of the H5 O+ 2 cation by means of fully quantum-dynamical methodology, using an exact kinetic-energy operator and an accurate potential energy surface. The use of curvilinear coordinates was essential because only then the strongly anharmonic large-amplitude motions (waggings, torsion, rockings) could be described conveniently. The reported simulations show a good agreement with the predissociation spectrum of the H5 O+ 2 ·Ne complex. Moreover, they clarify various features of the IR spectrum that remained elusive due to complicated anharmonicities and couplings. Acknowledgements The authors thank Prof. J. Bowman for providing the potential-energy routine, D. Lauvergnat for performing the TNUM calculations and the Scientific Supercomputing Center Karlsruhe for generously providing computer time. M.B. thanks the International Graduiertenkolleg IGK 710 “Complex processes: Modelling, Simulation and Optimization” for a fellowship. M.B. and H.D.M. thank Prof. P. Bastian for helpful discussions on the parallelization of the code. O.V. is grateful to the Alexander von Humboldt Foundation for financial support.

Parallelization of the MCTDH Method

155

References 1. H.-D. Meyer, U. Manthe, and L.S. Cederbaum, Chem. Phys. Lett. 165, 73 (1990). 2. U. Manthe, H.-D. Meyer, and L.S. Cederbaum, J. Chem. Phys. 97, 3199 (1992). 3. M.H. Beck, A. J¨ ackle, G.A. Worth, and H.-D. Meyer, Phys. Rep. 324, 1 (2000). 4. A. J¨ ackle and H.-D. Meyer, J. Chem. Phys. 104, 7974 (1996). 5. A. J¨ ackle and H.-D. Meyer, J. Chem. Phys. 109, 3772 (1998). 6. H.-D. Meyer, F.L. Qu´er´e, C. L´eonard, and F. Gatti, Chem. Phys. 329, 179 (2006). 7. K.R. Asmis et al., Science 299, 1375 (2003). 8. J.M. Headrick et al., Science 308, 1765 (2005). 9. N.I. Hammer et al., J. Chem. Phys. 122, 244301 (2005). 10. D. Marx, M. Tuckerman, J. Hutter, and M. Parrinello, Nature 397, 601 (1999). 11. T.D. Fridgen et al., J. Phys. Chem. A 108, 9008 (2004). 12. J.M. Headrick, J.C. Bopp, and M.A. Johnson, J. Chem. Phys. 121, 11523 (2004). 13. M.V. Vener, O. K¨ uhn, and J. Sauer, J. Chem. Phys. 114, 240 (2001). 14. J. Dai et al., J. Chem. Phys. 119, 6571 (2003). 15. X. Huang, B.J. Braams, and J.M. Bowman, J. Chem. Phys. 122, 044308 (2005). 16. J. Sauer and J. Dobler, Chem. Phys. Chem. 6, 1706 (2005). 17. M. Kaledin, A.L. Kaledin, and J.M. Bowman, J. Phys. Chem. A 110, 2933 (2006). 18. F. Gatti, J. Chem. Phys. 111, 7225 (1999). 19. D. Lauvergnat and A. Nauts, J. Chem. Phys. 116, 8560 (2002). 20. J.M. Bowman, S. Carter, and X. Huang, Int. Rev. Phys. Chem. 22, 533 (2003). 21. A.B. McCoy et al., J. Chem. Phys. 122, 061101 (2005). 22. D.J. Wales, J. Chem. Phys. 110, 10403 (1999). 23. G.G. Balint-Kurti, R.N. Dixon, and C.C. Marston, J. Chem. Soc., Faraday Trans. 86, 1741 (1990).

Green Chemistry from Supercomputers: Car–Parrinello Simulations of Emim-chloroaluminate Ionic Liquids Barbara Kirchner1 and Ari P. Seitsonen2 1

2

Lehrstuhl f¨ ur Theoretische Chemie, Universit¨ at Leipzig, Linn´estraße 2, D-04103 Leipzig, Germany [email protected] IMPMC, CNRS & Universit´e Pierre et Marie Curie, 4, place Jussieu, case 115, F-75252 Paris, France [email protected]

1 Introduction Ionic liquids (IL) or room temperature molten salts are alternatives to “more toxic” liquids [1]. Their solvent properties can be adjusted to the particular problem by combining the right cation with the right anion, which makes them designer liquids. Usually an ionic liquid is formed by an organic cation combined with an inorganic anion [2, 3]. Further discussions on the subject can be found in the following review articles [4–6]. Despite of this continuing interest in ionic liquids their fundamental properties and microscopic behavior are still only poorly understood. Unresolved questions regarding those liquids are still controversially discussed. A large contribution to the understanding of the microscopic aspects can come from the investigation of these liquids by means of theoretical methods [7–22]. In our project AIMD-IL at HLRS/NEC SX-8 we have investigated a prototypical ionic liquid using ab initio molecular dynamics methods, where the interaction between the ions is solved by explicitly treating the electronic structure during the simulation. The amount of computing time is justified by the increased accuracy and reliability compared to simulations employing parameterized, classical potentials. In this summary we will describe the results obtained within our project of a Car–Parrinello simulation of 1-ethyl-3-methylimidazolium ([C2 C1 im]+ , see Fig. 1) chloroaluminates ionic liquids; for a snapshot of the liquid see Fig. 2. Depending on the mole fraction of the AlCl3 to [C2 C1 im]Cl these liquids can behave from acidic to basic. Welton describes the nomenclature of these fascinating liquids in his review article as follows [5]: “Since Cl− is a Lewis base and [Al2 Cl7 ]− and [Al3 Cl10 ]− are both Lewis acids, the Lewis acidity/basicity

158

B. Kirchner, A.P. Seitsonen +

H H C1" N3 H

H2 C2

C4 H4

H H C1' N1

Fig. 1. Lewis structure of 1-ethyl-3methylimidazolium, or [C2 C1 im]+

H C2' H H

C5 H5

Fig. 2. A snapshot from the Car–Parrinello simulation of the “neutral” ionic liquid [C2 C1 im]AlCl4 . Left panel : The system in atomistic resolution. Blue spheres: nitrogen; cyan: carbon; white: hydrogen; silver : aluminium; green: chlorine. Right panel : Center of mass of [C2 C1 im]+ , white spheres, and AlCl− 4 , green spheres

of the ionic liquid may be manipulated by altering its composition. This leads to a nomenclature of the liquids in which compositions with an excess of Cl− (i. e. x(AlCl3 ) < 0.5) are called basic, those with an excess of [Al2 Cl7 ]− (i. e. x(AlCl3 ) > 0.5) are called acidic, and those at the compound formation point (x(AlCl3 ) = 0.5) are called neutral.“ In this report we concentrate on the neutral liquid. In a previous analysis we determined the Al4 Cl− 13 to be the most abundant species in the acidic mixture as a result of the electron deficiency property [23].

2 Method In order to model our liquid we use Car–Parrinello molecular dynamics (CPMD) simulations. The atoms are propagated along the Newtonian trajectories, with forces acting on the ions. These are obtained using density functional theory solved “on the fly”. We shall shortly describe the two main ingredients of this method in the following [24, 25].

Car–Parrinello Simulations of Ionic Liquids

159

2.1 Density Functional Theory Density functional theory (DFT) [26, 27] is nowadays the most-widely used electronic-structure method. DFT combines reasonable accuracy in several diﬀerent chemical environments with minimal computational eﬀort. The most frequently applied form of DFT is the Kohn–Sham method. There one solves the set of equations 1 − ∇2 + VKS [n] (r) ψi (r) = εi ψi (r) 2 2 |ψi (r)| n (r) = i

VKS [n] (r) = Vext ({RI }) + VH (r) + Vxc [n] (r)

Here ψi (r) are the Kohn–Sham orbitals, or the wave functions of the electrons; εi are the Kohn–Sham eigenvalues, n (r) is the electron density (can be interpreted also as the probability of finding an electron at position r) and VKS [n] (r) is the Kohn–Sham potential, consisting of the attractive interaction with the ions in Vext ({RI }), the electron-electron repulsion VH (r) and the so-called exchange-correlation potential Vxc [n] (r). The Kohn–Sham equations are in principle exact. However, whereas the analytic expression for the exchange term is known, it is not the case for the correlation, and even the exact expression for the exchange is too involved to be evaluated in practical calculations for large systems. Thus one is forced to rely on approximations. The mostly used one in first-principles simulations is the generalized gradient approximation, GGA, where one at a given point includes not only the magnitude of the density – like in the local density approximation, LDA – but also its first gradient as an input variable for the approximate exchange correlation functional. In order to solve the Kohn–Sham equations with the aid of computers they have to be discretized using a basis set. A straight-forward choice is to sample the wave functions on a real-space grid at points {r}. Another approach, widely used in condensed phase systems, is the expansion in the plane wave basis set, ψi (r) = ci (G) eiG·r G

Here G are the wave vectors, whose possible values are given by the unit cell of the simulation. One of the advantages of the plane wave basis set is that there is only one parameter controlling the quality of the basis set. This is the so-called cut-oﬀ energy Ecut : All the plane waves within a given radius from the origin, 1 2 |G| < Ecut , 2

160

B. Kirchner, A.P. Seitsonen

are included in the basis set. Typical number of plane wave coeﬃcients in practice is of the order of 105 per electronic orbital. The use of plane waves necessitates a reconsideration of the spiked external potential due to the ions, −Z/r. The standard solution is to use pseudo potentials instead of these hard, very strongly changing functions around the nuclei [28]. This is a well controlled approximation, and reliable pseudo potentials are available for most of the elements in the periodic table. When the plane wave expansion of the wave functions is inserted into the Kohn–Sham equations it becomes obvious that some of the terms are most eﬃciently evaluated in the reciprocal space, whereas other terms are better executed in real space. Thus it is advantageous to use fast Fourier transforms (FFT) to exchange between the two spaces. Because one usually wants to study realistic, three-dimensional models, the FFT in the DFT codes is also three dimensional. This can, however, be considered as three subsequent onedimensional FFT’s with two transpositions between the application of the FFT in the diﬀerent directions. The numerical eﬀort of applying a DFT plane wave code mainly consists of basic linear algebra subprograms (BLAS) and fast Fourier transform (FFT) operations. The previous one generally require quite little communication. However the latter one requires more complicated communication patterns since in larger systems the data on which the FFT is performed needs to be distributed on the processors. Yet the parallellisation is quite straightforward and can yield an eﬃcient implementation, as recently demonstrated in IBM Blue Gene machines [29]; combined with a suitable grouping of the FFT’s one can achieve good scaling up to tens of thousands of processors with the computer code CPMD [30]. Car–Parrinello Method The Car–Parrinello Lagrangean reads as LCP =

1 I

2

˙2+ MI R I

1 μ ψ˙ i ψ˙ i − EKS + constraints 2 i

(1)

where RI is the coordinate of ion I, μ is the fictitious electron mass, the dots denote time derivatives, EKS is the Kohn–Sham total energy of the system and the holonomic constraints keep the Kohn–Sham orbitals orthonormal as required by the Pauli exclusion principle. From the Lagrangean the equations of motions can be derived via Euler-Lagrange equations: ∂EKS ∂RI δ δE KS + {constraints} μψ¨i = − δ ψi | δ ψi |

¨ I (t) = − MI R

(2)

Car–Parrinello Simulations of Ionic Liquids

161

The velocity Verlet is an example of an eﬃcient and accurate algorithm widely used to propagate these equations in time. The electrons can be seen to follow fictitious dynamics in the Car– Parrinello method, i. e. they are not propagated in time physically. However, this is generally not needed, since the electronic structure varies much faster than the ionic one, and the ions see only “an average” of the electronic structure. In the Car–Parrinello method the electrons remain close to the BornOppenheimer surface, thus providing accurate forces on the ions but simultaneously abolishing the need to solve the electronic structure exactly at the Born–Oppenheimer surface. For Born–Oppenheimer simulations there always exists a residual deviation from the minimum due to insuﬃcient convergence in the self-consistency, and thus the ionic forces calculated contain some error. This leads to a drift in the total conserved energy. On the other hand in the Car–Parrinello method one has to make sure that the electrons and ions do not exchange energy, i. e. that they are adiabatically decoupled. Also the time step used to integrate the equations of motion in the Car–Parrinello molecular dynamics has to be 6–10 times shorter than in the Born-Oppenheimer dynamics due to the rapidly oscillating electronic degrees of freedom. In practice the two methods are approximately as fast, and the Car–Parrinello method has a smaller drift in the conserved quantities, but the ionic forces are weakly aﬀected by the small deviation from the Born-Oppenheimer surface. 2.2 Technical Details For the simulations we used density functional theory with the generalized gradient approximation of Perdew, Burke and Ernzerhof, PBE [31] as the exchange-correlation term in the Kohn–Sham equations, and we replaced the action of the core electrons on the valence orbitals with norm-conserving pseudo potentials of Troullier-Martins type [32]; they are the same ones as in [33] for Al and Cl. We expanded the wave functions with plane waves up to the cut-oﬀ energy of 70 Ry. We sampled the Brillouin zone at the Γ point, employing periodic boundary conditions. We performed the simulations in the NVT ensemble, employing a Nos´eHoover thermostat at a target temperature of 300 K and a characteristic frequency of 595 cm−1 , a stretching mode of the AlCl3 molecules. We propagated the velocity Verlet equations of motion with a time step of 5 a.t.u. = 0.121 fs, and the fictitious mass in the Car–Parrinello dynamics for the electrons was 700 a.u. A cubic simulation cell with a edge length of 22.577 ˚ A containing 32 molecules of cations and anions each, equaling to the experimental density of 1.293 g/cm3 . We ran our trajectory employing the Car–Parrinello molecular dynamics for 20 ps which resulted in the simulation of approximately 165300 steps.

162

B. Kirchner, A.P. Seitsonen

2.3 Radial Pair Distribution Functions In order to characterise the ionic structure in our simulation we first consider the radial pair distribution functions. Figure 3 depicts the radial distribution function of the AlCl− 4 anion in our ionic liquid and of AlCl3 in the pure AlCl3 liquid from Ref. [33]. It should be noted that both simulations were carried out at diﬀerent temperature, which results in diﬀerent structured functions. In the case of the neutral [C2 C1 im]AlCl4 ionic liquid there will be hardly a possibility for larger anions to be formed. In contrast to this the pure AlCl3 liquid shows mostly the dimer (45%), but also larger clusters such as trimers (30%), tetramers (10%) and pentamers as well as even larger units (< 10%) [33]. It can be recognised from Fig. 3 that the first Al-Al peak is missing when comparing the pure AlCl3 simulations to the one from the ionic liquid. This is because only monomer units ((AlCl3 )n Cl− with n = 1) exists and these monomers are separated from each other by the cations. The more structured functions of the ionic liquid can be attributed to the lower temperature at which it was simulated. The first Al-Cl peak (black solid line) appears at 222.4 pm while the corresponding peak in the pure AlCl3 simulations occurs already at 214.0 pm. There is no shoulder in the Al-Cl function at the first peak and the second peak occurs at larger distances. The Cl-Cl function presents its first peak at 361.1 pm which is approximately 10 pm earlier than what was observed for the pure AlCl3 liquid. In Fig. 4 we concentrate on the radial distribution functions of the imidazolium protons to the chlorine atoms. For each of the three ring protons we show an individual function in the left panel of Fig. 4. Because the H25 Al-Al (liq) Al-Cl (liq) Cl-Cl (liq) Al-Al (AlCl3)

4

Al-Cl (AlCl3) Cl-Cl (AlCl3)

g(r)

3

2

1

0

2

3

4

5

7 6 RADIUS (100 pm)

8

9

10

Fig. 3. The radial distribution function of the AlCl− 4 anion (bold lines) together with the corresponding function from the pure AlCl3 simulations (dotted lines) of Ref. [33]. Distances are in ˚ A. Black : Al-Al; red : Al-Cl; blue: Cl-Cl

Car–Parrinello Simulations of Ionic Liquids 1.6

1.6

1.2

1.2

0.8

0.8 H2 H4 H5

0.4

0.0

163

0

2

4 6 RADIUS (100 pm)

8

Hethyl,1 (2x) Hethyl,2 (3x)

0.4

10

0.0

Hmethyl

0

2

4 6 RADIUS (100 pm)

8

10

Fig. 4. The radial distribution function of chlorine atoms from AlCl− 4 anion to the A. Left: H2-Cl (black ), H4-Cl (red ), H5-Cl protons from [C2 C1 im]. Distances are in ˚ (blue); Right: Terminal ethyl-H-Cl (black ), α-ethyl-H-Cl (red ), methyl-H-Cl (blue).

Cl function shows the most pronounced peak and the first peak appears at shorter distances than for the functions of H4-Cl and H5-Cl it is clear that this proton is the most popular coordination site for the chlorine atoms. However, the other two protons from the rear show also peaks at slightly larger distances which indicates an involved network instead of individual pairs. It should be noted that from this structural behavior it can not be deduced how long lived the coordination partners are. Considering the protons of the ethyl and the methyl group it is striking that here also small pronounced peaks can be observed. While the ethyl group hydrogen atoms-Cl functions like the H4-Cl functions are least pronounced, the methyl-group-Cl function show a maximum height larger than that of the H4-Cl function. Obviously this functional group is also in touch with the chlorine atoms of the anion. 2.4 Intramolecular Tetrahedral Structure of AlCl− 4 In Table 1 we list some of the characteristic distances of AlCl− 4 from isolated molecule calculation as well as from simulations. For the AlCl− 4 anion we observe a perturbation from the ideal tetrahedral symmetry both in the isolated system and in the liquid. Whereas the shortest and longest Al-Cl distances vary only by 10 pm, the Cl-Cl distances show larger deviations of 30 pm (iso/dyn) to 40 pm (liq). This means that the perturbation is already induced by temperature. However, in the liquid the perturbation from the optimal geometry is somewhat more enhanced. 2.5 Intermolecular Structure: Proton-Cl Coordination We now turn to the intermolecular structure of the hydrogen atom-chlorine atom distances in order to shed more light onto the interesting coordination

164

B. Kirchner, A.P. Seitsonen

Table 1. Geometrical parameters of the isolated and the average AlCl− 4 in the ionic liquid. Distances r are in pm. rmin indicates the shortest while rmax indicates the longest distance. r is the average over all conﬁgurations. The abbreviation “iso/dyn” indicates a dynamic calculation of the isolated anion. “liq” denotes the average values from the simulations of the neutral liquid.

iso/dyn liq

AlCl rmin rmax rmin/max

ClCl rmin rmax rmin/max

218.1 230.2 216.6 229.1

354.4 376.9 344.2 381.8

1.06 1.06

1.06 1.11

behaviour of the imidazolium protons as observed before in the radial distribution function. Therefore we collected the shortest chlorine distance from a particular proton into a histogram which is shown in Fig. 5. On the average the shortest H-Cl distances is 279 pm for H2 and 290 pm for both H4 and H5. A broad range of distances from 200 pm to over 400 pm can be seen. This variety is typical for a weak to a medium ranged hydrogen bond distance. It is obvious from Fig. 5 that the Cl atoms can approach the acidic H2 atom closer than the other two protons H4 and H5. The distribution of H4 and H5 are almost identical, as expected from their similar local geometry in the molecule. 0.8

RELATIVE PROBABILITY

H2 (acidic) H5 (ethyl)

H4 (methyl)

0.6

0.4

0.2

0 2.0

2.5

3.0 3.5 SHORTEST H-Cl DISTANCE [Å]

4.0

4.5

Fig. 5. The distribution of the shortest proton-Cl distance from a particular proton (H2, H4, H5) to any Cl atom

Car–Parrinello Simulations of Ionic Liquids

165

3 Results: Electronic Structure One of the advantages of the Car–Parrinello simulations over the traditional molecular dynamics simulations is that the electronic structure is available on the fly in each step of the simulations. This allows for several ways of analysis of the electronic structure. Electrostatic Potential We begin by considering the electrostatic potential mapped onto the isosurface of the electron density of the two individual ions. From Fig. 6 we obtain insight into the charge distribution according to the particular part of each ion. For the AlCl− 4 we recognize in the blue color (low electro-static potential) the consequence of the negative charge that is associated with this ion. The negative charge is distributed all over the chlorine atoms. Towards the center, consisting of the aluminium atom, we find a decreased negative charge showing as the green color in the left panel of Fig. 6. The opposite is the case for the [C2 C1 im]+ . Here the positive charge leads to the red color (high electrostatic potential) around this ion. Upon closer inspection we find that the ring protons all hold the same red color as most of the molecule. A slight decrease of the charges can be found in the methyl group protons and a stronger decrease (yellow color) can be found at the terminal ethyl protons. This is in accordance with the observation from the radial pair distribution functions and with chemical intuition that these protons are less acidic than the other protons of the cation.

Fig. 6. The electrostatic potential mapped onto the electron density with a surface A3 . The colour scale ranges from −0.1 (blue) to +0.1 (red ) atomic values of 0.067 e− /˚ + units. Left: AlCl− , 4 right: [C2 C1 im]

166

B. Kirchner, A.P. Seitsonen

Fig. 7. The Wannier centers, denoted as red spheres. Top: [C2 C1 im]+ , bottom: AlCl− 4

Wannier Centers We used the maximally localized Wannier functions and their geometric centers, also called Wannier centers, to characterize the distribution of electrons in the condensed phase. An example is shown in Fig. 7, which demonstrates how the Wannier centers can be used to interpret the chemical nature of the bonds, for example the polarity or the alternating single-double bonds (please compare to the scheme in Fig. 1). By observing the average distance of the Wannier center between the carbon atom in the imidazolium ring and the corresponding proton we can see that in the C2-H2 pair the electrons are closer to the carbon than in the C4-H4 and C5-H5 pairs, pointing towards a larger polarity of the C2-H2 pair. Thus the H2 is more positive than the H4 and H5, and can electro-statically attract the negative Cl atoms from the anion molecules towards itself, as was seen in the Sect. 2.5.

4 Computational Performance For the simulation of our system, i. e. 32 [C2 C1 im]AlCl4 pairs, we have to treat 768 atoms and 1216 electronic states in each time step. The amount of atoms is by far larger than in a usual single-molecule static calculation. Therefore the application of GGA-density functional theory is necessary in order to make the simulation computationally tractable. It should be noted that 32 molecules is more or less the lowest limit of a calculation employing

Car–Parrinello Simulations of Ionic Liquids

167

periodic boundary conditions, as smaller sized systems would result in artificial finite-size eﬀects due to interactions with the mirror images. Regarding these circumstances our simulation provides the first real ab initio molecular dynamics simulations of an ionic liquid. Due to the computational constraints previous simulations treated a smaller amount of molecular pairs or only employed simplified models of ionic liquids (for example [C1 C1 im]Cl). For the obvious reasons it was necessary for our calculations to be carried out on a large amount of eﬃcient processors. We used 128 processors on the NEC SX-8. Therefore we were able to carry out our simulations within just two months. The size of our system leads to restart files of 14 GB in size. Before starting the real production runs we measured the scaling of the computing time and computational eﬃciency when changing the system size and/or the number of processors incorporated in a job. The results of these tests are shown in Table 2 and Fig. 8. The smallest system contains 32 (IL-32) pairs. The next system contains 48 (IL-48) and the largest system 64 (IL-64) pairs. We see very good scaling in the computing time still when going from 64 to 128 processors. We did not go beyond 128 processors still, but we estimate a decent or a good scaling in the IL-32 system, or very good scaling in the IL-48 and IL-64 cases. We note that at very large processor counts a diﬀerent parallellisation using OpenMP built in the CPMD code could be tried if the scaling otherwise is no longer satisfactory. A concrete limitation is met if the number of processors is larger than the length of the FFT grid in the first direction; however, further scaling is achieved by applying task groups,-another eﬃcient method inside CPMD; thereby the FFT’s over diﬀerent electronic states are grouped to set of processors, thus overcoming the limitation on the maximum number of processors due to the length of the FFT grid. The task groups can be incorporated particularly well on the NEC SX-8 computer at HLRS due to the large amount of memory at each node, because the task groups increase the memory requirement per node somewhat. We report here Table 2. Scaling of the wall-clock time in seconds per iteration and performance in GFLOPs versus number of processors. IL-x indicates the system size of the ionic liquid

system IL-32 IL-48 IL-64

Time per iteration (s) processors 32 64 128

Performance (GFLOPs) processors 32 64 128

46.0 − −

381 − −

23.5 72.6 159.4

13.2 37.9 83.3

729 784 817

1307 1496 1561

168

B. Kirchner, A.P. Seitsonen 2000

4

TIME SCALING

3

PERFORMANCE (GFLOPS)

il-32 il-48 il-64

2

1

0

32

64 #PROCs

128

il-32 il-48 il-64

1500

1000

500

0

32

64 #PROCs

128

Fig. 8. The scaling of the wall clock time per iteration – left – and numerical performance in GFLOPs – right – plotted against the number of processors. The green, dashed lines denote the ideal scaling and theoretical peak performance, respectively. #P ROCs: gives the number processors used in the calculations

always the best performance over the diﬀerent number of task groups; typically its optimum is at four or eight groups. The numerical performance exceeds one 1012 floating point operations per second (tera-FLOPs or TFLOPs) in all the systems studied at 128 processors. Furthermore, the performance still scales very favorably when going from 64 to 128 processors. Thus from the eﬃciency point of view processor counts exceeding 128 could also be used. However, due to the limited number of processors available, and because we already hit the “magic target” of one TFLOPs we restricted our production to 128 processors. Overall we were more than satisfied with the performance and with the prospect of performing the production calculations for the IL-48 or even IL-64 systems. However, due to the total time of the simulation which is a multiple of the number of molecular dynamics steps, we were forced to choose the IL-32 system for the production job, as otherwise we would not have been able to simulate a trajectory of ≈ 20 ps like we managed to do now. We also want to note that our calculations profit from the computer architecture of the NEC SX-8 at HLRS not only due to the high degree of vectorization and very good single-processor computing power in as evidenced in the high numerical eﬃciency (over 10 GFLOPs/processors; this number also includes the I/O), but also due to the large memory as we could stored some large intermediate results in the memory, thus avoiding the need to recalculate part of the results; this would be unavoidable in a machine with smaller amount of memory per processor. This way almost one third of the FFT’s, and thus of the most demanding all-to-all parallel operations can be avoided,

Car–Parrinello Simulations of Ionic Liquids

169

improving the parallel scaling still somewhat over a normal calculation where this option could not be used.

5 Conclusions We have simulated a prototypical ionic liquid [C2 C1 im]AlCl4 using Car– Parrinello molecular dynamics methods. 768 atoms were included in the simulation cell. The computational eﬃciency on the NEC SX-8 at HLRS allowed us to simulate the system for about 20 ps at realistic conditions. We achieved a sustained performance of over 1 TFLOPs on 128 processor, clearly exceeding an eﬃciency of 50%. The high throughput in the NEC SX-8 allowed us to execute the simulation in a short project time. This is of great advantage, when we are not forced to wait for extended periods of time in order to execute a simulation. Our simulations indicate a distorted tetrahedral structure for the AlCl− 4 anion. The most favorable coordination site is the acidic C-H group between the two nitrogen atoms. However coordination to the other protons is also possible. Thus we are dealing most likely with an extended network. Acknowledgements We thank the HLRS for the allocation of computing time; without this our project would not have been feasible! We are grateful to Prof. J¨ urg Hutter for several discussions, and to Stefan Haberhauer (NEC) for executing the benchmarks on the NEC SX-8 and optimising CPMD on the vector machines. BK would like to thank T. Welton and J.S. Wilkes for helpful discussion. BK acknowledges the financial support of the DFG priority program SPP 1191 “Ionic Liquids”, the ERA program and the financial support from the collaborative research center SFB 624 “Templates” at the University of Bonn.

References 1. Ed, P. Wasserscheid and T. Welton. Ionic Liquids in Synthesis. VCH-Wiley, Weinheim, 2003. 2. J.H. Davis. Task-speciﬁc ionic liquids. Chem. Lett., 33:1072–1077, 2004. 3. A.E. Visser, R.P. Swatloski, W.M. Reichert, R. Mayton, S. Sheﬀ, A. Wierzbicki, J.H. Davis, and R.D. Rogers. Task-speciﬁc ionic liquids for the extraction of metal ions from aqueous solutions. Chem. Commun., 01:135–136, 2001. 4. V.A. Cocalia, K.E. Gutowski, and R. D. Rogers. The coordination chemistry of actinides in ionic liquids: A review of experiment and simulation. Coord. Chem. Rev., 150:755–764, 2006. 5. T. Welton. Room-Temperature Ionic Liquids. Solvents for Synthesis and Catalysis. Chem. Rev., 99:2071–2083, 1999.

170

B. Kirchner, A.P. Seitsonen

6. T. Welton. Ionic Liquids in catalysis. Coord. Chem. Rev., 248:2459–2477, 2004. 7. P.A. Hunt and I. Gould. J. Phys. Chem. A, 110:2269, 2006. 8. S. Kossmann, J. Thar, B. Kirchner, P.A. Hunt, and T. Welton. Cooperativity in ionic liquids. J. Chem. Phys., 124:174506, 2006. 9. Z. Liu, S. Haung, and W. Wang. A reﬁned force ﬁeld for molecular simulation of imidazolium-based ionic liquids. J. Phys. Chem. B, 108:12978, 2004. 10. J.K. Shah and E.J. Maginn. Fluid Phase Equlib, 222-223:195, 2004. 11. T.I. Morrow and E.J. Maginn. Molecular dynamics study of the ionic liquid 1-nbutyl-3-methylimidazolium hexaﬂuorophosphate. J. Phys. Chem. B, 106:12807, 2002. 12. C.J. Margulis, H.A. Stern, and B.J. Berne. Computer simulation of a “green chemistry” room-temperature ionic solvent. J. Phys. Chem. B, 106:12017, 2002. 13. J. Lopes, J. Deschamps, and A. Padua. Modeling ionic liquids using a systematic all-atom force ﬁeld. J. Chem. Phys. B, 108:2038, 2004. 14. S. Urahata and M. Ribeiro. Structure of ionic liquids of 1-alkyl-3methylimidazolium cations: A systematic computer simulation study. J. Chem. Phys., 120(4):1855, 2004. 15. T. Yan, C.J. Burnham, M.G. Del Popolo, and G.A. Voth. Molecular dynamics simulation of ionic liquids: The eﬀect of electronic polarizability. J. Phys. Chem. B, 108:11877, 2004. 16. S. Takahashi, K. Suzuya, S. Kohara, N. Koura, L.A. Curtiss, and M. Saboungi. Structure of 1-ethyl-3-methylimidazolium chloroaluminates: Neutron diﬀraction measurements and ab initio calculations. Z. fur Phys. Chem., 209:209, 1999. 17. Z. Meng, A. D¨ olle, and W.R. Carper. J. Mol. Struct., 585:119, 2002. 18. A. Chaumont and G. Wipﬀ. Solvation of uranyl(ii) and europium(iii) cations and their chloro complexes in a room-temperature ionic liquid. a theoretical study of the eﬀect of solvent “humidity”. Inorg. Chem., 43:5891, 2004. 19. F.C. Gozzo, L.S. Santos, R. Augusti, C.S. Consorti, J. Dupont, and M.N. Eberlin. Chem. Eur. J., 10:6187, 2004. 20. E.R. Talaty, S. Raja, V.J. Storhaug, A. D¨olle, and W.R. Carper. J. Phys. Chem. B, 108:13177, 2004. 21. Y.U. Paulechka, G.J. Kabo, A.V. Blokhin, A.O. Vydrov, J.W. Magee, and M. Frenkel. J. Chem. Eng. Data, 48:457, 2003. 22. J. de Andrade, E.S. B¨ oes, and H. Stassen. Computational study of room temperature molten salts composed by 1-alkyl-3-methylimidazolium cations-force-ﬁeld proposal and validation. J. Phys. Chem. B, 106:13344, 2002. 23. B. Kirchner and A.P. Seitsonen. Ionic liquids from car-parrinello simulations, part ii: Structural diﬀusion leading to large anions in chloraluminate ionic liquids. Inorg. Chem., 47:2751–2754, 2007. DOI 10.1021/ic0624874. 24. J. Hutter and D. Marx. Proceeding of the february conference in J¨ ulich. In J. Grotendorst, editor, Modern Methods and algorithms of Quantum chemistry, page 301, J¨ ulich, 2000. John von Neumann Institute for Computing. http://www.fz-juelich.de/nic-series/Volume1/. 25. J. Thar, W. Reckien, and B. Kirchner. Car–parrinello molecular dynamics simulations and biological systems. In M. Reiher, editor, Atomistic Approaches in Modern Biology, volume 268, pages 133–171, Top. Curr. Chem., 2007. Springer. 26. P. Hohenberg and W. Kohn. Inhomogeneous electron gas. Phys. Rev., 136:B864– B871, 1964. 27. W. Kohn and L.J. Sham. Self-consistent equations including exchange and correlation eﬀects. Phys. Rev., 140:A1133–A1139, 1965.

Car–Parrinello Simulations of Ionic Liquids

171

28. W.E. Pickett. Pseudo potential methods in condensed matter applications. Comput. Phys. Rep., 115, 1989. 29. J¨ urg Hutter and Alessandro Curioni. Car-parrinello molecular dynamics on massively parallel computers. ChemPhysChem, 6:1788–1793, 2005. 30. CPMD V3.8 Copyright IBM Corp 1990-2003, Copyright MPI f¨ ur Festk¨ orperforschung Stuttgart 1997-2001. see also www.cmpd.org. 31. J.P. Perdew, K. Burke, and M. Ernzerhof. Generalized gradient approximation made simple. Physical Review Letters, 77:3865–3868, 1996. original PBE. 32. N. Troullier and J.L. Martins. Eﬃcient pseudopotentials for plane-wave calculations. Physical Review B, 43:1993–2006, 1991. 33. B. Kirchner, A.P. Seitsonen, and J. Hutter. Ionic Liquids from Car–Parrinello Simulations, Part I: Liquid AlCl3 . J. Phys. Chem. B, 110:11475–11480, 2006.

DFT Modelling of Oxygen Adsorption on CoCr Surfaces Janina Zimmermann1 and Lucio Colombi Ciacchi1,2 1

2

Fraunhofer Institut f¨ ur Werkstoﬀmechanik, W¨ ohlerstraße 11, 79108 Freiburg, Germany [email protected] Institut f¨ ur Zuverl¨ assigkeit von Bauteilen und Systemen, Universit¨at Karlsruhe, 76131 Karlsruhe, Germany [email protected]

1 Introduction Oxidation phenomena on the surface of cobalt-based alloys are of large technological relevance and deep scientific importance, since the skin of “native oxide” which covers the surface governs the chemical and physical interactions of the metal with the outer environment. In particular in the case of alloys used in biomedical implantations, such as for instance CoCrMo, the structure and composition of the native oxide layers directly influence the behaviour of implants within the chemically aggressive physiological environment. Understanding and predicting this behaviour requires a precise knowledge of the structure and composition of the ultrathin oxide layer which form spontaneously when a bare alloy surface is put in contact with an oxidising environment. Traditionally, a great deal of eﬀort has been spent in the last century to develop theories able to describe quantitatively the oxidation kinetics of metals and metal alloys at high temperature (see e.g. Refs. [1,2]). The available kinetic models rely on assumptions on the nature of characteristic defects in the formed oxide, which govern the diﬀusion processes necessary to oxide growth. Compared with the amount of knowledge available in the field of high-temperature oxidation, detailed information on the oxide layers formed at room temperature or below is scarce. These layers are often glassy oxides of non-integer stoichiometry, so structural information on the oxide lattice or on the present defects cannot be inferred from the corresponding bulk oxide structures which form at more elevated temperatures or oxygen pressures [3]. Moreover, the formation of vacancies and their clustering underneath the oxide layer may have a direct influence on the stability and on the physical properties of the native oxide layers. This is an issue of particular importance when native oxides form on the surface of an alloy via the selective oxidation of

174

J. Zimmermann, L. Colombi Ciacchi

only one of its composing elements. Clustering of vacancies at the metal/oxide interface has been indeed observed experimentally upon selective Al oxidation of an intermetallic TiAl surface [4]. Our final goal is to gain a mechanistic insight into the events of initial oxide formation on CoCr surfaces, prior to the beginning of a diﬀusion-limited oxide growth process. As these events are not thermally activated and occur on a scale of picoseconds to nanoseconds, they are hardly accessible through commonly available experimental techniques. Therefore, we will address them in this work by means of “first-principles” atomistic modelling at the quantum mechanical level [5]. Examples of applications of this technique to the study of the initial oxidation reactions of metal and semiconductor surfaces can be found in [7–9] and references therein. In the present paper, we report the results of our initial investigations into the composition and structure of CoCr surfaces, bare or in the presence of an oxygen ad-layer. In particular, in this phase of the project we will focus our attention on optimising the precision and performance of the computational technique by accurate choice of the simulation parameters. This issue will be addressed here by a set of preliminary calculations of the bulk properties of pure Co and Cr metals and of their oxides. Surface properties will then be investigated initially using small surface unit cells, before constructing large models to be used in dynamical simulations of the actual oxidation reactions.

2 Computational Technique Our simulations are performed with a first-principles molecular dynamics (FPMD) approach based on the Density-Functional Theory (DFT) [5], using the PW91 exchange-correlation functional [6]. A review of this computational technique, applied in particular to the study of materials, can be found in Ref. [10]. The technique allows us to investigate the dynamical evolution of a system composed by atom nuclei and electrons at the quantum level of precision. Namely, the total energy of a given system and the forces acting on the atoms are computed from its ground-state electronic structure as obtained by the solution of the Schr¨ odinger equation within the DFT. The dynamical trajectory of the atomic system is then obtained upon numerical integration of a Newtonian equation of motion where the atomic nuclei behave as classical particles. Structural optimisations can be performed with the same formalism, applying an appropriate damping factor to the equation of motion. At the present stage, the dynamical evolution of a system composed by a few hundreds of atoms for about ten picoseconds can be routinely simulated on the parallel computers at the SSC in Karlsruhe. Although very short, this simulation time is nevertheless in many cases suﬃcient to simulate processes of formation and break of covalent bonds, as in the present study.

Oxygen Adsorption on CoCr

175

2.1 The Lautrec Code Our simulations are performed with the Lautrec code [11]. This is a highly optimised parallel code for performing molecular dynamics simulations and structure optimisations according the the Car-Parrinello scheme [12]. The code is based on the Plane-Wave Pseudopotential implementation of the Density Functional Theory, and includes generalised gradient corrections to the Local Density Approximation in the exchange-correlation potential. It has been originally developed by A. De Vita in the group of R. Car at the EPFL, Lausanne, and has been continuously updated and developed by a number of authors, in particular M. Stengel at the University of California, S. Barbara. Recent developments include the implementation of a novel algorithm to perform Car-Parrinello simulations of metallic systems [13, 14] and of Bl¨ochl’s Projector-Augmented-Wave formalism [15]. The Lautrec code has been compiled and executed on many of the commonly available supercomputers and operating systems, and in particular on the HP XC6000 cluster at the SSC Karlsruhe. Excellent parallel scaling behaviour has been obtained on this platform in typical production runs using partitions up to 32 CPUs and the MPI communication protocol. However, for larger CPU partitions, the performance becomes quickly limited by the communications between the nodes (Fig. 1). This behavior reflects the high communication demand of plane-wavebased DFT codes such as Lautrec. Parallelisation of the code is achieved at two leveles; (1) distribution of the reciprocal-space vectors, which reduces the amount of total required memory and results in a very uniform load balancing among the CPUs, (2) distribution of the set of k-points used to sample the Brillouin zone, which may be very eﬃcient for small systems, but less useful for larger systems which are sampled by only one or a few k-points. In ei-

60.0

Scaling XC1, Lautrec (DFT) code

t(4)/t

45.0 30.0 15.0 0.0

8

16

24

32

40

48

56

64

72

CPUs

Fig. 1. Inverse of the CPU time per MD iteration (normalised to 4 processors) obtained in typical production runs using the Lautrec code and diﬀerent CPU partitions of the HP XC6000 cluster at the SSC Karlsruhe. The dashed black line represents ideally linear scaling

176

J. Zimmermann, L. Colombi Ciacchi

ther cases, most of the CPU workload is due to; (i) scalar products between wavefunctions, which require fast and eﬃcient matrix-matrix multiplication routines, and (ii) Fast Fourier Transformations of the wavefunction coeﬃcients. In particular, the computational demand scales approximately with the cube of the number of atoms included in the simulations, which limits the size of the systems to a few hundred atoms in the case of transition metals. More detailed information on the performances of the Lautrec code on parallel computers may be found in Ref. [16]. As an example of the required ressources, a typical electronic structure calculation with full structural relaxation of a medium-size alloy surface model (including 40 metal atoms and 8 oxygen atoms) requires about 900 CPU hours, and a molecular dynamics simulation of the same system requires about 8000 CPU hours per simulated picosecond. Therefore, the CPU resources required for the entire present project, spanning over a total time of three years, are estimated to be of the order of 150000 to 200000 CPU hours per year.

3 Results 3.1 Preliminary Calculations Generation of PAW Datasets The simulation of transition metals within the PAW formalism requires particular care as far as the choice of the dataset for the description of the ionnucleus interactions is concerned. For Co we have generated a dataset with 9 explicit valence electrons (corresponding to the 3d and 4s atomic states), used a core radius of 1.5 au and included 3, 2 and 2 projectors for the s, p and d angular momentum channels, respectively. Through extensive tests against a more accurate dataset treating explicitly also the 3s and 3p electrons, we found that inclusion of the semicore state is not necessary in the case of Co. Instead, our Cr dataset treats explicitly 8 semi-core and 6 valence electrons, corresponding to the 3s, 3p, 3d and 4s atomic states. We have generated it using a core radius of 1.5 au and including 3, 3 and 2 projectors for the s, p and d angular momentum channels, respectively. The O dataset includes 6 valence electrons and 2 projectors in each of the s and p channels. With these datasets the bulk properties of the crystalline metals and of their oxides has been computed satisfactorily, as outlined in the following section. Calculations of Bulk Properties of Pure Metals and Oxides For each system, we have initially checked carefully the convergence of bulk properties with the kinetic-energy cut-oﬀ of the plane wave expansion of the wavefunctions. In the case of both the Co, Cr and O species we found that a value of 40 Ry produced well converged results. Namely, diﬀerences in

Oxygen Adsorption on CoCr

177

Table 1. Convergence of Co bulk properties with the plane-wave kinetic-energy cut-oﬀ, sampling the Brillouin zone with a 12 × 12 × 8 Monkhorst-Pack k-point grid

Ecut (Ry) a0 (˚ A) B0 (GPa) Ecoh (eV) 30 40 60

2.496 2.484 2.485

197 201 200

4.949 4.943 4.933

μ (μB ) 1.630 1.604 1.606

binding energies in small systems containing oxygen atoms, such as the O2 molecule and the (H2 O)2 dimer, are found to be smaller than 0.01 eV using cut-oﬀ energies of 40 or 60 Ry. Moreover, the lattice parameter and the cohesive energy of metallic Cr at 40 Ry are found to diﬀer by 0.003 ˚ A and 0.015 eV from the corresponding values at 60 Ry. In Table 1 we report the results obtained for metallic Co, using a 12 × 12 × 8 Monkhorst-Pack k-point grid to sample the Brillouin zone and a fixed ratio c/a = 1.623 between the height c and the edge a of the hexagonal unit cell. The magnetic ground state has been found to be ferromagnetic, consistently with the experimental evidence. Next, using a cut-oﬀ energy of 40 Ry we have optimised the lattice vectors a and c by performing several total energy calculations at fixed values and fitting the obtained results by the Murnaghan equation of state, obtaining a0 = 2.487 ˚ A and c/a = 1.626 (the corresponding experimental values are 2.51 and 1.622 [17]). The computed cohesive energy and magnetic moment per Co atom are 4.95 eV and 1.61 Bohr Magnetons, which are to be compared with the experimental values of 4.39 and 1.72 [17] and with the values of 4.88 and 1.66 obtained in calculations within a similar DFT formalism [18, 19]. Performing the calculations with a less dense, 6 × 6 × 4 k-point grid, we obtained the same a and c/a values, a cohesive energy of 4.98 eV and a magnetic moment of 1.65 μB . Given these very small diﬀerences, grids of density corresponding to 6 × 6 × 4 for the case of metallic Co have been used also for the calculations of the bulk properties of Cr and of the metal oxides. Figure 2 shows examples of the Murnaghan fits to the calculated total energy values at diﬀerent lattice parameters for the case of Cr and Co3 O4 . For Cr a 6 × 6 × 6 k-point grid is suﬃcient to describe the element properly. A more densely spaced 12 × 12 × 12 grid leads to variations in total energy of only 0.01 eV per atom (see Fig. 2(a)). Particular attention has been paid to consider diﬀerent magnetic solutions (see Fig. 2(b)). In all cases, the ground state solution has been found to correspond to the experimentally determined magnetic ordering at 0 K. An exception is represented by the case of bulk Cr, whose magnetic ground state is a longitudinal spin-density wave (SDW), which consists of a roughly sinusoidal modulation of the magnetic moments and an incommensurate wave vector [20]. The wavelength of the SDW is about 60 ˚ A (or 42 interlayer distances) in the low-temperature limit, increasing to about 78 ˚ A at room temperature. Us-

178

b)

-248.804

6x6x6 k-points 12x12x12 k-points

-248.808

Etot [Hartree]

Etot [Hartree]

a)

J. Zimmermann, L. Colombi Ciacchi

-248.812 -248.816 -248.82 -248.824 -248.828

-655.37

Ferromagnetic Antiferromagnetic

-655.375 -655.38 -655.385 -655.39

2.7

2.8

2.9

3

3.1

7.9

a0[Å]

8

8.1

8.2

8.3

a0 [Å]

Fig. 2. Calculated values and Murnaghan ﬁts of the total energy as a function of the lattice constant for (a) Cr and (b) Co3 O4 . In the latter, the antiferromagnetic and ferromagnetic curves are referred to the spin conﬁguration of the Co atoms on the tetrahedral sites

ing a simple BCC cubic cell we found an antiferromagnetic ground state and a magnetic moment per atom of 1.02 μB . Due to the restricted number of atoms in the primitive unit cell and the periodic boundary conditions, this state is an approximation of the incommensurate SDW. However, within this model, our magnetic solution is in agreement with a number of previous electronic structure calculations at the DFT level. All final results for the single elements and some of their oxides are summarised in Table 2. Table 2. The calculated lattice constant (a0 ), bulk modulus (B0 ), enthalpy of formation (sublimation) per metal atom (ΔHf ) and magnetic moment per atom (μ) for the elemental metals and their most important oxides. AF denotes an antiferromagnetic, and F a ferromagnetic ground state

System

Structure

a0 (˚ A) (c/a)

B0 (GPa) ΔHf (eV) μ (μB )

Co Cr Cr CoO Co3 O4 Cr2 O3

hcp bcc hcp rock salt spinel corundum

2.49 (1.626) 2.86 2.48 (1.795) 4.23 8.09 5.37

214 176 235 188 210 196

a

(−4.98) (−4.11) (−3.70) −1.29 −2.71 −5.07

1.65 1.02 0.00 2.43 2.35 2.31

(F) (AF) (AF) (AF)a (AF)

Referred to the Co atoms on the tetrahedral sites. Those on the octahedral sites are not spin

polarised.

Oxygen Adsorption on CoCr

179

3.2 Simulations of Bare Metal Surfaces All our simulations are performed under periodic boundary conditions, i.e., the simulation cell is repeated periodically in all directions of space. Within this approach, we have modelled surfaces including in the simulation cell a slab of metal atoms infinitely repeated in the xy plane and separated by a layer of vacuum in the z direction perpendicular to the surface. The thickness of the metallic slab and of the vacuum layer should be chosen so as to avoid spurious eﬀects due to the truncation of the half-crystal below the modelled surface and to interactions with the repeated images. This issue can be checked by computing the surface energy γ of a given crystal plane for various thicknesses of the metal slab. The surface energy is defined as: γ=

1 Slab Bulk (E − Nl Etot ), 2 tot

(1)

Slab is the total energy of a surface slab with Nl layers (one atom where Etot Bulk is the reference total energy per atom of the per atomic layer) and Etot bulk system. The factor 1/2 takes into account that there are two equivalent surfaces in the slab. For instance, in the case of the Co(0001) surface, using a 1 × 1 surface cell and a 6 × 6 × 1 distribution of k-points in the simulation cell, we have found that using metal slabs of thickness varying between 5 and 8 atomic layers and a vacuum layer corresponding to 6 atomic layers leads to very little variations of γ, of the order of 0.01 eV per surface atom. In Table 3 we report the computed values of the surface energy and the work function for selected Co and Cr surfaces. In our calculations, the work function φ is calculated as the diﬀerence between the Hartree potential in the middle of the vacuum regions separating the surface slab from its periodically repeated image and the Fermi energy of the system. The table reports the values computed using a 1 × 1 surface cell sampled by a 6 × 6 k-point distribution in the surface plane. After truncation of the metal bulk, the atomic layers in proximity of the created surface relax as a consequence of the undercoordination of the atoms. We have performed full structural optimisation of the surface systems, and found that for Co(0001) and Cr(110) the distance between the first and second surface layers is smaller than the bulk interlayer distance by 3.3% and 2.0%,

Table 3. Surface energy γ and work function φ for Co (hcp) and Cr (bcc and hcp) surfaces

System (1 × 1) √ Co(0001) (1 × 2) Cr(110) (1 × 1) Cr(0001)

γcalc (J/m2 ) 2.15 3.05 2.63

γexp (J/m2 ) 2.55 2.30 –

φcalc (eV) 5.06 4.89 5.30

φexp (eV) 5.0 4.5 –

180

J. Zimmermann, L. Colombi Ciacchi

respectively. Inward relaxation is due to the reinforcement of the back bonds to the subsurface layers after bond breaking at the surface and transfer of electrons to bonding d states. Moreover, we have computed the magnetic moments and charges on the atoms by integration of the electronic charge and of the spin-density within the Bader regions [22] associated with each atom. For Cr(110), the reduced coordination number at the surface induces a narrowing of the d bands, and the reduced overlap of majority and minority bands leads to a largely increased magnetic moment with respect to the bulk value (nearly 200% in our calculations) [21]. This is in contrast to Co, where the majority band is closed to being saturated, thereby reducing the moment enhancement. For Co(0001) we observed an enhancement of the magnetisation of 11.7% (the magnetic moment increases from 1.62 to 1.81 μB on the surface atoms) and a slight charge transfer from the bulk to the surface, leading to atomic charges of −0.02 e on the surface atoms. In the case of the (0001) surface of HCP Cr with the same lattice parameter a of Co, we found a nonmagnetic ground state and atomic charges of −0.03 e on the surface atoms. 3.3 Static Calculations of Oxygen Adsorption on Pure Metals To gain an insight into the oxygen aﬃnity of the Co and Cr surfaces we have calculated the heat of adsorption of O atoms on the Co(0001), Cr(110) and CrHCP (0001) surfaces with respect to the O2 molecule. The calculations have √ been performed using a 2 × 2 cell for the (0001) surfaces and a 1 × 2 cell for Cr(110) (see Figs. 3 and 4). In all cases, the surface slabs comprised 5 atomic layers and O atoms where adsorbed symmetrically on each side of the slab. On all surfaces O is found to adsorb preferentially in “hollow” sites with a threefold coordination. For instance, adsorption of O on “bridge” and “ontop” sites on Cr(110) is energetically less favourable by 0.17 and 1.44 eV, respectively. Moreover, the heat of adsorption per atom is found to decrease with increasing oxygen coverage on all surfaces (Tables in Figs. 3 and 4). This can be understood in terms of the increased Coulomb repulsion between the oxygen atoms, which accept electronic charge from the underlying surface. Consistently, the work function of the surface is found to increase with increasing oxygen coverage due to the large surface dipole which develops upon oxygen adsorption. The amount of charge transferred to the O adsorbates can be quantified by the computed Bader charges, which are found to be −0.77 e and −0.89 e for each O atom adsorbed on Cr(110) and Co(0001), respectively. Interestingly, the eﬀect of oxygen is to quench the magnetic moment of the surface atoms (Figs. 3 and 4). This is consistent with previous theoretical investigation of the adsorption of gas species on Co(0001) [23], and with the available experimental evidence [24]. We have also calculated the adsorption energy of a 1 × 1 O adlayer on hollow sites of the CrHCP (0001) surface, and obtained a value of 3.57 eV per O atom, which is roughly comparable with the value of 3.49 eV obtained for a ML of O atoms on the Cr(110) surface. In

Oxygen Adsorption on CoCr

Coverage

0.25

0.50

0.75

1.00

Work function increase (eV) 0.52 Adsorption energy (eV) 2.59

1.08 2.23

1.55 1.94

1.80 1.70

181

Fig. 3. Oxygen adsorption on the Co(0001) surface: (a) side view of the surface layers with the spin density (orange), (b) top view of the surface, to illustrate the conﬁguration with diﬀerent oxygen coverages. The Table shows the work function increase with respect to the bare Co surface, and the heat of adsorption per oxygen atom

Coverage Top 0.5 Bridge 0.5 Hollow 0.5 Hollow 1.0 Δφ (eV) Eads (eV)

1.76 2.73

0.43 4.00

0.68 4.17

2.22 3.49

Fig. 4. Oxygen adsorption on the Cr(110) surface: (a) side view of the surface layers with positive and negative spin density (blue and orange), (b) top view of the surface, to illustrate the adsorption sites. The Table shows the work function increase with respect to the bare Cr surface, and the heat of adsorption per oxygen atom

comparison, the heat of adsorption of oxygen on the Co(0001) surface is much smaller, and for a 1 × 1 O adlayer amounts to 1.70 eV per O atom. This is consistent with the high enthalpy of formation of bulk chromium oxide with respect to the cobalt oxides (see Table 2).

182

J. Zimmermann, L. Colombi Ciacchi

3.4 Surfaces of CoCr Alloys In this section we report results on the energetics and structural details of small models for surfaces of a CoCr alloy with a Co/Cr ratio equal to 2. We consider as a model an HCP crystal structure with the a lattice parameter of pure Co. The choice of this model is justified by the fact that a solid solution of Cr atoms in an hexagonal ǫ-Co matrix represents the most abundant phase in technological Co-based alloys used for biomedical implantations [25, 26]. The purpose of these initial calculations is to extract basic information on the behaviour of the system which will help the interpretation of future investigation into the dynamics of the selective alloy oxidation using larger systems. So far, calculations were performed on slab surface models containing 6 atomic layers and either a 1 × 1 or a 2 × 2 surface unit cell. The results are presented separately in the following two subsections. 1 × 1 Model The situation modelled using a 1 × 1 surface unit cell corresponds to a stacking of atomic planes each containing either pure Co or pure Cr. Considering six atomic layers, a composition of 2 Co atoms per Cr atom and a symmetric configuration of the top and bottom surfaces of the metal slab, three independent systems need to be taken into account, where the two Cr planes are in the surface (system 1), subsurface (system 2) or central positions (system 3) in the metal slab. In the case of bare surfaces, the total energy of system 2 is 0.75 eV lower than system 1 and 0.35 eV lower than system 3. However, when a 1 × 1 O adlayer is adsorbed on the surface, then the high aﬃnity of Cr for oxygen results in a stabilisation of system 1, whose energy becomes 2.79 eV lower than system 2 and 2.85 eV lower than system 3. The computed O adsorption energy on the three systems is 3.40, 1.65, 1.79 eV for system 1, 2 and 3, respectively. It is interesting to note that the adsorption energy on a pure Co surface lies between the values calculated for systems 2 and 3, where Co is at the outer surface. This can be interpreted on the basis of an analysis of the atomic charges in the outermost layers. For system 2 without oxygen there is charge flow from the Cr atom of the second layer into the Co atom of the outermost layer. In the presence of oxygen, there is a strong charge flow from the surface Co atom to the O atom, leaving positive charged Co and Cr atoms (+0.81 e and +0.26 e) in the first and second layer, respectively. The Coulomb repulsion between them induces a lowering of the adsorption energy of oxygen with respect to the pure Co surface slab, where the charges of the surface and subsurface Co atoms are +0.89 e and −0.01 e after O adsorption. On the contrary, in system 3, Co atoms are present in both the first and the second layer, which become positively and negatively charged after adsorption, respectively (+0.90 e and −0.17 e). This electrostatically favourable configuration leads to a higher adsorption energy than on the pure Co surface slab.

Oxygen Adsorption on CoCr

183

2 × 2 Model A series of five systems of equal crystal structure (HCP) and composition (Co:Cr = 2:1), but with diﬀerent arrangement of the Co and Cr atoms in the crystal positions has been constructed with 6 atomic layers and a 2 × 2 surface unit cell (Fig. 5(a)). The first system corresponds to a full segregation of Cr to the surface layer. Going from the first to the fifth system, the Cr atoms mix with Co in the inner part of the slab and Co atoms move to the surface, up to a full segregation of Co atoms to the surface layer. In the case of bare metal surfaces, the total energy of the systems decreases from system 1 to system 5 by about 3.1 eV (Fig. 5(b)). This is due to a combination of the smaller surface energy of Co(0001) with respect to CrHCP (0001) and of the favourable enthalpy of mixing of the alloy with respect to the separate components. In particular, a first estimate of the enthalpy of mixing for the 2 × 2 model can be obtained by the diﬀerence ΔEmix 2×2 = E[CoCr] −

8 16 E[Co] − E[Cr], 24 24

(2)

Fig. 5. (a) Diﬀerent models for a 2 × 2 surface unit cell of a CoCr alloy. (b) Calculated total energy for each system. (c) Calculated total energy of the same systems with 1 ML of adsorbed oxygen

184

J. Zimmermann, L. Colombi Ciacchi

where E[CoCr] is the total energy of a periodically repeated system with atomic arrangement corresponding to system 4 in Fig. 5(a) in the absence of a vacuum layer. We obtained a mixing enthalpy of −0.17 eV per atom. As in the case of the 1 × 1 model, the situation changes completely in the presence of a ML of oxygen adsorbed on hollow sites on the alloy surface. Namely, due to the higher heat of adsorption on chromium, full segregation of Cr atoms to the surface layers becomes energetically favourable with respect to the other systems (with a total energy gain of about 5 eV going from system 5 to system 1) (Fig. 5(c)). This indicates the presence of a strong thermodynamical driving force for an oxidation-driven segregation of Cr atoms towards the surface of the alloy.

4 Conclusions and Outlook The results presented in Sect. 3.1 indicate that calculations of our O/CoCr models can be safely performed using a kinetic energy cut-oﬀ for the wave function expansion of 40 Ry and k-point grids of densities correspondent to 6×6×4 points in the hexagonal unit cell of metallic Co. Surface properties and chemical reactions at surfaces can be simulated using periodically repeated surface slab models of a thickness of 5 or 6 metallic layers separated by an equally thick layer of vacuum. Calculations of surface energy reveal that for a HCP crystal structure and an a lattice parameter of 2.487 ˚ A (correspondent to the equilibrium lattice parameter of bulk Co), a Co surface is more stable than a Cr surface by about 0.16 eV per surface atom. This indicate a preference for Co segregation at the outer surface of an alloy, as we indeed found by means of total energy calculations of the energetic stability of surface models using 1 × 1 or 2 × 2 surface unit cells (Sect. 3.4). Mixing of the two elements in a pseudo-random alloy model is found to be thermodynamically favoured, and the enthalpy of mixing has been estimated to amount to about −1.7 eV using a 24-atom simulation supercell. Adsorption of oxygen atom on the pure elements has been found to have a profound influence on the magnetic state of the surface, in particular resulting in a full quenching of the magnetic moment of the surface atoms in the case of a 1 × 1 adlayer on hollow sites of the Co(0001) surface. The calculated values of heat of adsorption on Co and Cr surfaces indicate a very strong oxygen aﬃnity of Cr, consistently with the large enthalpies of formation of chromium oxide with respect to cobalt oxides. Therefore, when oxygen is adsorbed on the surface of a CoCr alloy, there exists a strong thermodynamical driving force for the surface segregation of Cr atoms. However, given that a surface segregation of Co atoms seems to be preferred in the case of bare surfaces, how the actual diﬀusion of Cr atoms will take place during the oxidation of an initially bare alloy surface remains to be investigated. This issue will be addressed in future work, where larger systems will need to be

Oxygen Adsorption on CoCr

185

taken into account and dynamical simulations of the oxidation reactions will be performed. Acknowledgements This work has been supported by the Deutsche Forschungsgemeinschaft under grants CI 144/1-1 and CI 144/2-1 (Emmy Noether Programme). L.C.C. acknowledges support from the Alexander von Humboldt foundation. The required CPU time has been allocated on the HP XC clusters at the Scientific Supercomputing Centre of the Karlsruhe university within the COCROX project.

References 1. Lawless, K. R.: The oxidation of metals. Rep. Prog. Phys., 37, 231–316 (1974) 2. Wallwork, G. R.: The oxidation of alloys. Rep. Prog. Phys., 39, 401–485 (1976) 3. Lundgren, E., Kresse, G., Klein, C., Borg, M., Andersen, J. N., De Santis, M., Gauthier, Y., Konvicka, C., Schmid, M., Varga, P.: Two-dimensional oxide on Pd(111) Phys. Rev. Lett., 88, 246103–246106 (2002) 4. Maurice, V., Despert, G., Zanna, S., Bacos, M.-P., Marcus, P.: Self-assembling of atomic vacancies at an oxide/intermetallic alloy interface. Nature Mater., 3, 687–691 (2004) 5. Payne, M. C., Teter, M. P., Allan, D. C., Arias, T. A., Joannopoulos, J. D.: Iterative minimization techniques for ab initio total-energy calculations: molecular dynamics and conjugate gradients. Rev. Mod. Phys., 64, 1045–1097 (1992) 6. Perdew, J. P., Wang, Y.: Accurate and simple analytic representation of the electron-gas correlation energy. Phys. Rev. B 45, 13244 - 13249 (1992) 7. Colombi Ciacchi, L., Payne, M. C.: Hot-atom O2 dissociation and oxide nucleation on Al(111). Phys. Rev. Lett., 92, 176104 (2004) 8. Colombi Ciacchi, L., Payne, M. C.: First-principles molecular-dynamics study of native oxide growth on Si(001). Phys. Rev. Lett., 95, 196101 (2005) 9. Piscanec, S., Colombi Ciacchi, L., Vesselli, E., Comelli, G., Sbaizero, O., Meriani, S., De Vita, A.: Bioactivity of TiN-coated titanium implants. Acta Mater., 52, 1237–1245 (2004) 10. Hafner, J.: Atomic-scale computational materials science. Acta Mater., 48, 71– 92 (2000) 11. De Vita, A. et al.: Quantum molecular dynamics on massively parallel computers. EPFL Supercomput. Rev., 6 (1994) 12. Car, R., Parrinello, M.: Uniﬁed approach for molecular dynamics and densityfunctional theory. Phys. Rev. Lett., 55, 2471–2474 (1985) 13. VandeVondele, J., De Vita, A.: First-principles molecular dynamics of metallic systems. Phys. Rev. B, 60, 13241–13244 (1999) 14. Stengel, M., De Vita, A.: First-principles molecular dynamics of metals: a Lagrangian formulation. Phys. Rev. B, 62, 15283–15286 (2000) 15. Bl¨ ochl, P. E.: Projector augmented-wave method. Phys. Rev. B, 50, 17953– 17979 (1994)

186

J. Zimmermann, L. Colombi Ciacchi

16. Gruber, R., Volgers, P., De Vita, A., Stengel, M., Tran, T.-M.: Parametrisation to tailor commodity clusters to applications. Future Gener. Comp. Sy. 19, 111120 (2003) 17. Kittel, C.: Introduction to solid state physics. Wiley, New York (1986) 18. Philipsen, P. H. T., Baerends, E. J.: Cohesive energy of 3d transition metals: density functional theory atomic and bulk calculations. Phys. Rev. B, 54, 5326– 5333 (1996) 19. Cho, J.-H., Scheﬄer, M.: Ab initio pseudopotential study of Fe, Co, and Ni employing the spin-polarized LAPW approach. Phys. Rev. B, 53, 10685–10689 (1996) 20. Corliss, L. M., Hastings, J. M., Weiss, R. J.: Antiphase antiferromagnetic structure of chromium. Phys. Rev. Lett., 3, 211–212 (1959) 21. Eichler, A., Hafner, J.: Adsorbate-induced vacancy formation and substrate relaxation on Cr(100). Phys. Rev. B, 62, 5163–5167 (2000) 22. Bader, R. F. W.: Atoms in Molecules: A Quantum Theory, Oxford University Press, Oxford (1990) 23. Pick, S., Dreysse, H.: Calculation of gas adsorption eﬀect on magnetism of Co(0001). Surf. Sci., 474, 64–70 (2001) 24. Bucher, J. P., Douglass, D. C., Bloomﬁeld, L. A.: Magnetic properties of free cobalt clusters. Phys. Rev. Lett., 66, 3052–3055 (1996) 25. Reclaru, L., L¨ uthy, H., Eschler, P.-Y., Blatter, A., Susz, C.: Corrosion behaviour of cobalt-chromium dental alloys doped with precious metals. Biomaterials, 26, 4358–4365 (2005) 26. Dearnley, P. A.: A review of metallic, ceramic and surface-treated metals used for bearing surfaces in human joint replacements. Proc. Inst. Mech. Eng., 213, 107–135 (1998)

Comparison of the Incorporation of Watson-Crick Complementary and Mismatched Nucleotides Catalyzed by DNA Polymerase I Thomas E. Exner Fachbereich Chemie, Universit¨ at Konstanz, 78457 Konstanz, Germany [email protected]

1 Introduction During the DNA replication and repair synthesis, DNA polymerases add deoxynucleotides onto the growing end of a DNA primer strand using a singlestranded DNA as a template. To maintain the genomic integrity without the expensive proofreading performed by exonucleases, these polymerases have evolved a very high fidelity with error frequencies of approximately one in 103 –106 bases synthesized. But for the demands of numerous biotechnical applications even this fidelity is not satisfactory. Especially the unnatural conditions of several techniques either restrict the use of these enzymes or demand their tedious optimization. Thus, the primary design goal for DNA polymerases with altered functions is high specificity in formation of WatsonCrick base pairs during DNA synthesis. To support this research, Summerer et al. [1] presented an eﬃcient automated high-throughput setup for the rapid parallel screening of DNA polymerase mutant libraries. With this technique, they were able to identify several active variants of the Klenow fragment of DNA polymerase I from thermus aquaticus with significant higher extension fidelity than the wild-type enzyme. These variants can now be analyzed concerning structural changes in the ternary complexes composed out of the enzyme, the DNA primer and template as well as the incoming nucleotide with the goal to rationalize the higher specificity. DNA polymerases show a highly conserved structure resembling a handlike arrangement, including a thumb, a palm, and a fingers region (see Fig. 1). The catalytic cycle for the nucleotide incorporation is composed out of several steps including a large structural rearrangement leading to a movement of the fingers towards the thumb region. The individual steps are as follows: The 2’-deoxyribonucleoside-5’-triphosphate is incorporated into the DNA polymerase forming an open substrate complex in step 1. Step 2 involves the

188

T. E. Exner Fig. 1. Hand-like structure of DNA Polymerase I. The protein and the DNA backbone are shown in a red and blue ribbon representation. The incoming nucleotide (color-coded by atom type) and the two magnesium ions (yellow ) are highlighted as CPK models

conformational changes to align catalytic groups and form a closed ternary complex. The nucleotidyl transfer reaction follows in step 3. The resulting product complex undergoes a reverse conformational change back to the open form (step 4), from which the pyrophosphate dissociates (step 5) leading to the starting point for a new DNA synthesis cycle. It was shown in theoretical studies [2, 3, 4, 5] that the chemical reaction and not the ligand binding is the rate-limiting step. But the diﬀerent activation energies for this reactive step in the Watson-Crick case compared to the mismatched base pairs are most probably related to structural changes in the educt stage. Binding of the correct nucleotide leads to the conformational changes (step 2) whereas binding of an mismatch may alter or inhibit the conformational transition [6, 7, 8, 2, 3]. Larger alterations in the mismatch complex of the mutants could in this way lead to larger reaction barriers and higher selectivity. Therefore, we decided to start our investigations regarding this fidelity increase on the closed form of the ternary complex directly before the chemical reaction. Here, we describe first results for wild-type DNA polymerase I with regard to the structural diﬀerences of the closed ternary complex when incorporating a Cytidine triphosphate (C) opposite to a Watson-Crick complementary Guanosine (G) or a mismatched Adenosine (A). For doing so, we started from the already formed, experimentally determined Watson-Crick ternary complex available from the Protein Data Bank [9] (PDB entry 3KTQ [10]). In this structure, the G opposite to the incoming C was replaced by A. This mismatched complex was chosen because for this one the largest diﬀerences in the fidelity of the wild-type and mutated polymerases were observed experimentally. Simulation were then started from the two resulting complexes to analyze the diﬀerences in the structure as well as in the stability introduced by the mismatched base pair. With the use of future simulations on the mutated polymerases, these structural changes can hopefully be correlated with the height of the reaction barrier of the nucleotide incorporation and thus the fidelity of the

Comparison of the Incorporation of Nucleotides

189

mutants. Additionally, the final structures from the simulations could be used as starting structures for simulations of the incorporation process. But for this, diﬀerent methods like mixed quantum mechanical / molecular mechanical (QM/MM) approaches have to be used to describe the chemical reaction.

2 Material and Method In the last years, a number of theoretical studies on diﬀerent DNA polymerases were published [11, 12, 13, 14, 2, 3, 15, 16, 17, 18, 6, 4, 8, 19, 20, 21, 7, 5, 22]. As already mentioned, these show that the chemical reaction and not the complex formation are rate limiting. This can be attributed to deformations of the mismatched complexes leading to unfavorable starting structures for the chemical reaction. In this sense, Flori´ an et al. [2, 3] suggested that the incorporation of mismatched nucleotides takes place in a half-open form of T7 DNA polymerase, which could even cause a diﬀerent reaction mechanism compared to the Watson-Crick incorporation. The group of Schlick [6, 7, 8] showed that DNA polymerase β also cannot fully transform into the closed structure when incorporating a mismatched nucleotide. They even demonstrated in recent publications [7, 8] that the incorporation time of the diﬀerent nucleotides is correlated with the deformation of the active site especially the coordination of the two magnesium ions. The preliminary molecular dynamic simulations performed in the work presented here are meant to show if similar eﬀects can be seen in the ternary complex of DNA polymerase I. In subsequent work, it will be tried if a correlation between incorporation time and the active-site deformation of the wild type and mutated DNA polymerases I can be obtained. 2.1 Simulation Details The simulations were performed using the AMBER 8 suite of programs [23]. The modified version of the Cornell et al. force field (parm99) [24] was used for the protein and DNA template and primer. The parameters for the polyphosphate were taken from the work of Meagher et al. [25] and the ones for magnesium from Aqvist [26]. The structure of the ternary complex in the closed form was taken from the Protein Data Bank [9] (PDB entry 3KTQ [10]). The mismatched base pair was generated by removing the base of the nucleotide of the template strand opposite to the incoming nucleotide triphosphate and renaming the residue entry of the phosphate and the sugar group. During the preparation of the input files for the molecular dynamics simulations, the missing atoms including the one of the mismatched base were added in standard position. These starting structures were placed in a periodic truncated octahedron of 19278 and 19279 TIP3P water molecules [27] for the WatsonCrick and mismatch incorporation, respectively. 171 Na+ and 143 Cl− ions

190

T. E. Exner

were added to provide a physiological ionic strength and maintain electroneutrality of the system. The borders of the truncated octahedron were chosen to be at least 12 ˚ A from every atom. In total, this resulted in system sizes of 67635 and 67637 atoms for the Watson-Crick and the mismatched simulation, respectively. The system was minimized by 10000 steps to relax unfavorable conformations generated by the standard placement of the missing atoms. The particle mesh Ewald (PME) method [28] was used to treat long-range electrostatic interactions, and the SHAKE method [29] to constrain bond lengths of bonds involving hydrogen atoms. The time step for all MD simulations was set to 2 fs with a non-bonded cutoﬀ of 9 ˚ A. For equilibration, the system was first heated from 100 K to 300 K for 100 ps and then relaxed to the density corresponding to 1 bar for 300 ps in a sequence of MD simulations using the canonical (NVT) and the isothermal isobaric (NPT) ensemble, respectively. In these simulations harmonic restraints with force constants of 5 kcal mol−1 ˚ A−2 where applied to all solute atoms. These restraints were then gradually reduced to zero during 200 ps of NVT-MD. After an additional 4 ns of simulation for further equilibration of the system, the following 3 ns were used for analysis. These simulations were again performed in the canonical (NVT) ensemble taking advantage of the fact that the pressure should not change dramatically in such a simulation of a liquid system. Therefore, it is expected that the two ensembles give approximatively the same results but that the canonical ensemble additionally minimizes the computational demand.

3 Results and Discussion The calculations were performed on the high-performance computer HP XC4000 at the Scientific Supercomputing Center in Karlsruhe using up to 16 processors. The needed resources are given in Table 1 for 100 ps of simulation time. For comparison, the corresponding numbers for two systems located at the Universit¨ at Konstanz are also shown. The scaling behavior of the AMBER package is very bad on normal computer clusters with a GBit ethernet connection (speedup of 1.7 when going from 2 to 4 processors). This is a known problem due to the large amount of data, which has to be transfered between the processes. Using shared-memory architectures strongly improves the scaling behavior (data not shown). Also the InfiniBand technology of the XC4000 system removes the communication bottleneck (speedup of 3.1 when going from 4 to 16 processors). The bad performance of the Opteron 2.0 GHz system compared to the Opteron 2.6 GHz system can be attributed to the fact that this computer is actually a 16-core server, which is not exclusively used by one program. The set up of the input file for the molecular dynamics calculations were done using the standard tools provided with the AMBER package [23]. One general problem with this set up for proteins is to decide on the protonation

Comparison of the Incorporation of Nucleotides

191

Table 1. Comparison of computer time on diﬀerent platforms for 100 ps of MD simulation of the Watson-Crick complex architecture

CPU-time by one processor

CPU-time over all processors

memory usage

a) b) c) d) e)

26:49:33 15:24:04 9:20:37 5:45:13 1:51:24

53:39:06 61:36:14 37:22:29 23:00:53 29:42:29

670 MB 800 MB 128 MB 400 MB 400 MB

a) Dual-processor Intel Xeon 2.4 GHz, 2 GB memory b) 2× Dual-processor Intel Xeon 2.4 GHz, 2 GB memory, GBit ethernet c) 4-Core Opteron 2.0 GHz, 32 GB memory d) 4-Core Opteron 2.6 GHz, 16 GB memory e) 4 × 4-Core Opteron 2.6 GHz, 16 GB memory, InfiniBand 4X DDR Interconnect

state of histidine, which cannot be obtained from the experimental structure. Its pKA is very close to the physiological pH-value and, thus, the protonation strongly depends on the surroundings. Two molecular dynamics simulations were performed using the protocol described above with all histidines in the δ-form and the fully protonated form, respectively. Both simulations resulted in stable structures, which diﬀer only slightly from each other. Thus, on this basis no decision on the protonation state can be drawn. At the end, it was decided by visual inspection of the surroundings to protonate both the δ- and ε-positions of the three histidine residues His443, His480, and His639. These are in a negative polarized environment formed by the surroundings. For all other histidines, only the δ position was protonated.

Fig. 2. Comparison of the average structures taken from the simulations of the incorporation of a Watson-Crick and a mismatched base. Yellow : backbone of the polymerase for the Watson-Crick incorporation; green: template, primer, incoming nucleotide, and magnesium ions for the Watson-Crick incorporation; blue: backbone of the polymerase for the mismatch; and red : template, primer, incoming nucleotide, and magnesium ions for the mismatch

192

T. E. Exner

Then, the simulation of the Watson-Crick and mismatched complex were performed. As described in the “Material and Method” section, the last 3 ns of the simulations were used for analysis. For doing so, snapshots of the structures were taken every 2 ps and the water molecules, Na+ and Cl− ions were removed. The structures were then aligned to the first structure of the time series and the atomic coordinates were averaged over all snapshots. The resulting average structures were minimized for 200 steps to generate valid bond lengths and angles and are shown for the Watson-Crick and the mismatched incorporation in Fig. 2. Only minor diﬀerences can be seen in the overall structures. The root mean square deviation of all atoms of the DNA polymerase I is only 1.76 ˚ A and of the Cα atoms 1.20 ˚ A. It seem that contrary to the published results [6, 7, 8, 2, 3] the mismatched structure is even more closed. But these diﬀerences should not be overstressed. To further investigate the structural changes, the active sites of the two complexes were compared in detail (see Figs. 3 and 4). The conformation of the incoming Cytidine triphosphate is almost identical in both structures. This is caused by the strong constraints imposed by the narrow active site of the closed form of the polymerase. But due to the diﬀerent hydrogen bonding networks, the orientation of the opposite Watson-Crick complementary Guanosine (G) or mismatched Adenosine (A) diverge. Additionally, the unpaired nucleotides of the templates are located at diﬀerent places in the two complexes. But this nucleotide is very flexible so that this diﬀerence could be an artefact of the simulation.

Fig. 3. Comparison of the nucleotide binding site. The average structure of the Watson-Crick and the mismatched incorporation are shown on the left and on the right, respectively. The DNA polymerase I is represented as a green ribbon and a green molecular surface, of which the front site is removed to give view into the active site. The incoming nucleotide is shown in a CPK model and the template and primer as balls-and-sticks. The magnesium ions are colored yellow

Comparison of the Incorporation of Nucleotides

193

Fig. 4. Close-up comparison between the Watson-Crick (red ) and mismatch (green) incorporation. Diﬀerences can be seen in the hydrogen-bonding network of the base pairs and in the location of the last base of the template strand, which is not paired with a base of the primer. The small changes in the coordination of the magnesium ions are hardly visible Table 2. Comparison of interatomic distances for the matched and mismatched simulation atom pair

C-G

C-A

dNTP(Pα ) - Primer(O3′ ) Mg2+ (A) - dNTP(Pα ) Mg2+ (A) - Primer(O3′ ) Mg2+ (A) - dNTP(Oα ) Mg2+ (B) - dNTP(Pα ) Mg2+ (B) - dNTP(Oα )

3.344 3.453 2.267 2.310 3.221 2.262

3.486 3.466 2.454 2.333 3.156 2.255

To describe the coordination of the two magnesium ions, some characteristic distances are given in Table 2 for the Watson-Crick and the mismatched complex. These distances are very similar in both complexes as expected from the visualization in Figs. 3 and 4. The only diﬀerences worth to mention are the slight increases in the Mg-O3′ and Pα -O3′ distances. Even if these diﬀer in the sub-˚ A range, they could be significant, because they are exactly corresponding to the reaction coordinate (Pα -O3′ distance) of the nucleophilic attack and the activation of the O3′ (Mg2+ -O3′ distance). Additionally, these distances were also used by Arora at al. [7] in the correlation with the incorporation time. But the significance has to be confirmed in additional simulations of a longer period of time. As already mentioned, the results given so far are only preliminary and should be validated by much longer simulations. This becomes even more

194

T. E. Exner

evident if one analyses the time series of the atomic fluctuations over the last 3 ns. For doing so, the root mean square deviation (rmsd) of all atoms in the complex (all atoms of the simulation except water, Na+ and Cl− ions) compared to the average structure is shown in Fig. 5. On the one hand, the rmsd for all snapshots is below 2 ˚ A indicating only small structural changes during this period of time. If one is mapping the individual fluctuations on each atom (Fig. 6) is can additionally be seen that the largest deviations occur in the DNA double helix build out of the primer and the template far

Fig. 5. Time-series of the root mean square deviation of all atoms in comparison to the average structure over the last 3 ns of the simulation

Fig. 6. Visualization of the atomic ﬂuctuations over the last 3 ns of the simulation. The size of the spheres representing each atom is proportional to the standard deviation of this atom. Additionally, color coding also representing the ﬂuctuations is used going from blue for small to red for large deviations

Comparison of the Incorporation of Nucleotides

195

Fig. 7. Time-series of the root mean square deviation of the protein (upper part) and the DNA (lower part) atoms in comparison to the average structure over the last 3 ns of the simulation

away from the incorporation site. This eﬀect is also quantified in Fig. 7. If only the rmsd of the protein is examined, it is for most snapshots even below 1.5 ˚ A. In contrast, the DNA rmsd is almost up to 5 ˚ A for some snapshots. Therefore, in future simulations some base pairs could be removed to avoid unnecessary sampling of this flexible part and to decrease the system size. On the other hand, the time series of the rmsd values of the total complex as well as of the protein show U-shaped developing. This could mean that there is a systematic drift from one structure through an intermediate (very similar to the average one) to a new one. In such a case, the simulations are not in stable states and have to be continued until these states are reached. Another possibility is that the systems perform thermal oscillations around the stable

196

T. E. Exner

states, but that the simulation is not long enough to cover all conformations of the equilibrium. In any case, because of the very slow structural changes observed in the systems, much longer simulations have to be performed to reach the equilibrium or to sample these equilibrium adequately. To summarize the results, it can be said that the two simulations of incorporation complexes of the Watson-Crick complementary and mismatched nucleotide give very similar structures with only minor diﬀerences. But the simulation time was still to short for an suﬃcient sampling of the conformational space, so that it has to be proven in longer runs if these changes are significant. Additional, future simulations must also show if the complex with the mismatched nucleotide is thermodynamical stable but cannot be access kinetically from the open form as described in the literature [6, 7, 8, 2, 3]. For doing so, simulations starting from a half-open structure as proposed by Arora et al. [6, 7, 8] will be performed. In this case, the variation in the incorporation time is caused by the diﬀerent starting structures for the chemical reactions: closed form for the Watson-Crick and half-open form for the mismatched incorporation.

4 Conclusion In this work, the first calculations on the closed ternary complex of DNA polymerase I during the incorporation of a Watson-Crick complementary and a mismatched nucleotide are described. We started from the experimentally determined Watson-Crick ternary complex available from the Protein Data Bank [9] (PDB entry 3KTQ [10]). For generating the mismatched complex, the G opposite to the incoming C was replaced by A. This mismatched complex shows in the experiment the largest diﬀerences in the fidelity between the wild-type and mutated polymerases. Simulation were then started from the two resulting complexes to analyze the diﬀerences in the structure as well as in the stability introduced by the mismatched base pair. The last 3 ns of a 7 ns long simulations were used for this analysis. Even if the overall structures are very similar, small diﬀerences in the coordination of the two magnesium ions in the active site were observed. These correspond directly with the atoms mainly involved in the chemical reaction of the incorporation. Because the catalytic activity of proteins is based on forcing the substrates into the optimal conformations for the reaction, such small changes can have a huge impact on the reaction barriers or can even change the reaction mechanism totally. But the time series of the root mean square deviation still show a drift of the conformations even after the 4 ns used for equilibration, i.e. no stable conformation has been obtained for the complexes and unforeseeable structural changes can still occur especially for the mismatched complex. A reliable concussion and future predictions can only be based on such stable conformations. Therefore, longer simulations are on their way to verify the preliminary results. With the use of future simulations on the mutated polymerases, we

Comparison of the Incorporation of Nucleotides

197

hope that then these structural changes can be correlated with the height of the reaction barrier of the nucleotide incorporation and thus the fidelity of the mutants. Additionally, the final structures from the simulations could be used as starting structures for simulations of the incorporation process. But for this, diﬀerent methods like mixed quantum mechanical / molecular mechanical (QM/MM) approaches have to be used to describe the chemical reaction. Acknowledgements We thank the Scientific Supercomputing Center in Karlsruhe for providing the CPU-time to make the simulations possible.

References [1]

Summerer, D., Rudinger N.Z., Detmer,I., Marx, A.: Enhanced Fidelity in Mismatch Extension by DNA Polymerase through Directed Combinatorial Enzyme Design. Angew. Chem. Int. Ed., 44, 4712–4715 (2005) [2] Flori´ an, J., Goodman, M.F., Warshel, A.: Computer Simulation of the Chemical Catalysis of DNA Polymerase: Discriminating between Alternative Nucleotide Insertion Mechanisms for T7 DNA Polymerase. J. Am. Chem. Soc., 125, 8163–8177 (2003) [3] Flori´ an, J., Goodman, M.F., Warshel, A.: Computer Simulations of Protein Functions: Searching for the Molecular Origin of the Replication Fidelity of DNA Polymerase. PNAS, 102, 6819–6824 (2005) [4] Radhakrishnan, R., Schlick, T.: Orchestration of Cooperative Events in DNA Synthesis and Repair Mechanism Unraveled by Transition Path Sampling of DNA Polymerase β’s Closing. PNAS, 101, 5970–5975 (2004) [5] Radhakrishnan, R.: Mechanism of Nucleotide Incorporation in DNA Polymerase β. Biochem. Biophys. Res. Commun., 347, 626–633 (2006) [6] Yang, L., Beard, W.A., Wilson, S.H., Broyde, S., Schlick,T.: Highly Organized but Pliant Active Site of DNA Polymerase β. Biophys. J., 86, 3392–3408 (2004) [7] Arora, K., Beard, W.A., Wilson, S.H., Schlick, T.: Mismatch-Induced Conformational Distortions in Polymerase β Support an Induced-Fit Mechanism for Fidelity. Biochemistry, 44, 13328–13341 (2005) [8] Arora, K., Schlick, T.: In Silico Evidence for DNA Polymerase-β’s SubstrateInduced Conformational Change. Biophys. J., 87, 3088–3099 (2004) [9] Berman, H.M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T.N., Weissig, H. et al.: The Protein Data Bank. Nuc. Acids Res., 28, 235–242 (2000) [10] Li, Y., Korolev, S., Waksman, G.: Crystal Structures of Open and Closed Forms of Binary and Ternary Complexes of the Large Fragment of Thermus Aquaticus DNA Polymerase I: Structural Basos for Nucleotide Incorporation. EMBO J. 17, 7514 (1998) [11] Flori´ an, J., Goodman, M.F., Warshel, A.: Free-Energy Perturbation Calculations of DNA Destabilization by Base Substitutions. J. Phys. Chem. B, 104, 10092–10099 (2000)

198

T. E. Exner

[12] Flori´ an, J., Goodman, M.F., Warshel, A.: Theoretical Investigation of the Binding Free Energies and Key Substrate-Recognition Components of the Replication Fidelity of Human DNA Polymerase β. J. Phys. Chem. B, 106, 5739–5753 (2002) [13] Flori´ an, M.F., Warshel, A., J., Goodman: Molecular Dynamics Free-Energy Simulations of the Binding Contribution to the Fidelity of T7 DNA Polymerase. J. Phys. Chem. B, 106, 5754–5760 (2002) [14] Flori´ an, J., Goodman, M.F., Warshel, A.: Computer Simulation Studies of the Fidelity of DNA Polymerase. Biopolymers, 68, 286–299 (2003) [15] Xiang, Y., Oelaeger, P., Flori´ an, J., Goodman, M.F., Warshel, A.: Simulating the Eﬀect of DNA Polymerase Mutations on Transition-State Energetics and Fidelity: Evaluating Amino Acid Group Contributions and Allosteric Coupling for Ionized Residues in Human Pol β. Biochemistry, 45, 7036–7048 (2006) [16] Yang, L., Beard, W.A., Wilson, S.H., Roux, B., Broyde, S., Schlick, T.: Local Deformations Revealed by Dynamics Simulations of DNA Polymerase β with DNA Mismatch at the Primer Terminus. J. Mol. Biol., 321, 459–478 (2002) [17] Yang, L., Beard, W.A., Wilson, B., Broyde, S., Schlick, T.: Polymerase β Simulations Suggest That Arg258 Rotation is a Slow Step Rather Than Large Subdomain Motions Per Se. J. Mol. Biol., 317, 651–671 (2002) [18] Yang, L., Arora, K., Beard, W.A., Wilson, S.H., Schlick,T.: Critical Role of Magnesium Ions in DNA Polymerase β’s Closing and Active Site Assembly. J. Am. Chem. Soc., 126, 8441–8453 (2004) [19] Krahn, J.M., Beard, W.A., Wilson, S.H.: Structural Insights into DNA Polymerase β Deterrents for Misincorporation Support an Induced-Fit Mechanism for Fidelity. Structure, 12, 1823–1832 (2004) [20] Radhakrishnan, R., Schlick, T.: Fidelity Discrimination in DNA Polymerase β: Diﬀering Closing Proﬁles for a Mismatched (G:A) versus a Matched (G:C) Base Pair. J. Am. Chem. Soc., 127, 13245–13252 (2005) [21] Arora, K., Schlick, T.: Conformational Transition Pathway of Polymerase β/DNA upon Binding Correct Incoming Substrate. J. Phys. Chem. B, 109, 5358–5367 (2005) [22] Rittenhouse, R.C., Apostoluk, W.K., Miller, J.H., Straatsma,T.P.: Characterization of the Active Site of DNA Polymerase β by Molecular Dynamics and Quantum Chemical Calculation. Proteins, 53, 667–682 (2003) [23] Case, D.A., Darden, T.A., Cheatham, III, T.E., Simmerling, C.L., Wang, J., Duke, R.E. et al.: AMBER8. University of California, San Francisco (2004) [24] Cornell, W.D., Cieplak, C.i., Bayly, I.R., Goul, I.R., Merz, K.M., Ferguson, D.M. et al.: A Second Generation Force Field for the Simulation of Proteins, Nucleic Acids, and Organic Molecules. J. Am. Chem. Soc, 117, 5179–5197 (1995) [25] Meagher, K.L., Redman, L.T., Carlson, H.A.: Development of Polyphosphate Parameters for Use with the AMBER Force Field. J. Comput. Chem., 24, 1016–1025 (2003) [26] Aqvist, J.: Modeling Ion Ligand Interactions in Solutions and Biomolecules. Theochem. J. Mol. Struct., 88, 135–152 (1992) [27] Jorgensen, W.L., Chandrasekhar, J., Madura, J., Klein, M.L.: Comparison of Simple Potential Functions for Simulating Liquid Water. J. Chem. Phys., 79, 926–935 (1983)

Comparison of the Incorporation of Nucleotides

199

[28] Darden, T., York, D., Pedersen,L.: Particle Mesh Ewald – A N log(N ) Method for Ewald Sums in Large Systems. J. Chem. Phys., 98, 10089–10092 (1993) [29] Ryckaert, J.P., Ciccotti, G., Berendsen, H.J.C.: Numerical-Integration of Cartesian Equations of Motions of a System with Constraints – Molecular Dynamics of n-Alkanes. J. Comput. Phys., 23, 327–341 (1977)

Assumed PDF Modeling of Turbulence Chemistry Interaction in Scramjet Combustors Markus Kindler, Peter Gerlinger, and Manfred Aigner Institut f¨ ur Verbrennungstechnik der Luft- und Raumfahrt, Universit¨ at Stuttgart Pfaﬀenwaldring 38–40, 70569 Stuttgart, Germany Summary. A multivariate assumed PDF approach together with ﬁnite-rate chemistry is used for the simulation of scramjet combustors. Because the combustor entrance conditions of scramjets at low ﬂight Mach numbers (Ma ≈ 8) are close to the ignition limit of hydrogen air mixtures, detailed kinetic schemes are required and an accurate simulation of temperature and temperature ﬂuctuations is essential. In the present paper, a lobed strut injector concept is used for hydrogen injection which enhances the mixing process by production of streamwise vorticity. The inﬂuence of the chosen combustor geometry on the mixing behaviour is investigated in this paper. Moreover a Mach 2 supersonic combustion experiment is simulated to investigate the inﬂuence of the chosen reaction mechanism and of the assumed PDF model. A detached ﬂame is obtained and the predicted ignition length is a measure for the accuracy of the used kinetic scheme and the modeling of turbulence chemistry interaction.

1 Introduction The development of scramjet (supersonic combustion ramjet) combustors strongly depends on numerical simulations because the corresponding combustor entrance conditions are hard to achieve in ground test facilities. Moreover, both tests at the ground and in flight are highly expensive. In the recent years the interest on supersonic combustion has increased. Some experimental investigations like HyShot and especially the NASA X-43A research vehicle have proven that scramjet combustors may be used in practice. But despite these achievements with model flight vehicles there are still demanding tasks. A main problem in supersonic combustions is the extremely short residence time of fuel and oxidator in the combustion chamber. Therefore diﬀerent approaches concerning mixing and combustion enhancements have to be studied. In the framework of the GRK 1095/1 at the University of Stuttgart, the IVLR is working on the optimization and on the design of scramjet combustors and hydrogen injectors. For such a task numerical simulations are an adequate

204

M. Kindler, P. Gerlinger, M. Aigner

tool. In this report numerical studies on diﬀerent geometries of the supersonic combustion chamber are presented. These calculations are performed using the scientific code TASCOM3D that is described in detail below.

2 Governing Equations and Numerical Scheme High-speed reacting flows are described by the full compressible Navier-Stokes, species and turbulence transport equations. Additionally an assumed PDF (probability density function) approach is used to take turbulence chemistry interaction into consideration. Therefore two additional equations (for the variance of the temperature and the variance of the sum of species mass fractions) have to be solved. Thus the described set of averaged equations in three-dimensional conservative form is given by ∂Q ∂(F − Fν ) ∂(G − Gν ) ∂(H − Hν ) + + + =S, ∂t ∂x ∂y ∂z

(1)

where $ %T ˜ ρ¯q, ρ¯ω, ρ¯σT , ρ¯σY , ρ¯Y˜i , Q = ρ¯, ρ¯u ˜, ρ¯v˜, ρ¯w, ˜ ρ¯E,

i = 1, 2, . . . , Nk − 1 . (2)

The variables in the conservative variable vector Q are the density ρ¯ (averaged), the velocity components (Favre averaged) u ˜, v˜ and w, ˜ the total specific √ ˜ the turbulence variables q = k and ω = ǫ/k (where k is the kinetic energy E, energy and ǫ the dissipation rate of k), the variance of the temperature σT and the variance of the sum of the species mass fractions σY and finally the species mass fractions Yi (i = 1, 2, . . . , Nk −1). Thereby Nk describes the total number of species that are used for the description of the gas composition. The vectors F, G and H specify the inviscid fluxes in x-, y- and z- direction, Fν , Gν and Hν the viscid fluxes, respectively. The source vector S in (1) results from turbulence and chemistry and is given by T S = 0, 0, 0, 0, 0, S¯q , S¯ω , S¯σT , S¯σY , S¯Yi ,

i = 1, 2, . . . , Nk − 1 ,

(3)

where Sq and Sω are the source terms of the turbulence variables, SσT and SσY the source terms of the variance variables (σT and σY ) and SYi the source terms of the species mass fractions. For turbulence closure a two-equation low-Reynolds-number q-ω turbulence-model is applied [17]. The closure of the terms concerning chemistry is described in detail below. The unsteady set of equations (1) is solved by the inhouse scientific code TASCOM3D (Turbulent All Speed Combustions Multigrid) using an implicit Lower-Upper-Symmetric Gauss-Seidel (LU-SGS) [1, 2, 7, 10] finite-volume algorithm, where the finiterate chemistry is treated fully coupled with the fluid motion. More details concerning TASCOM3D may be found in [7, 9, 10, 11, 15, 16].

Assumed PDF Modeling of Turbulence Chemistry Interaction

205

Assumed PDF Approach The momentary chemical production rate of species i in (1) is defined by

Nk Nk Nr ′ ′′ ′′ ′ νl,r νl,r , (4) νi,r − νi,r SYi = Mi kf r cl − kbr cl r=1

l=1

l=1

where kfr and kbr are the forward and backward reaction rates of reaction r that may be described by the Arrhenius function −Er (5) kr = Ar T nr exp Rm T with the constants Ar and nr (defined by the reaction-mechanism), the temperature T , the activation energy Er and the specific gas constant Rm . Furthermore (4) implies the molecular weight of a species M , the species concen′ ′′ tration ci = ρYi /Mi and the stoichiometric coeﬃcients νr and νr of reaction r. Due to the required averaging of (4) for the use in (1) higher order correlations appear that can not be modeled. An interesting approach for closure are probability density functions (PDFs). Regarding the computational cost versus accuracy the use of assumed PDFs which use a predefined shape seems to be a good compromise compared to the time comsuming calculation of the more exact shapes by solving a PDF transport equation. Via the use of an assumed PDF approach and the assumption of statistical independence of temperature, gas composition and density the averaged chemical production rate for a species i is defined by S¯Yi = Si Tˆ, cˆ1 , · · · , cˆNk P Tˆ, cˆ1 , · · · , cˆNk dTˆdˆ c1 · · · dˆ cN k , (6) where ρ − ρ¯) . P Tˆ, cˆ1 , · · · , cˆNk = PT Tˆ PY Yˆ1 , · · · , YˆNk δ (ˆ

(7)

Thereby PT defines the temperature-PDF mainly dealing with the Arrhenius function eq. (5). Usually PT is desribed by a Gaussian distribution [5] ⎡ 2 ⎤ ˆ − T) T 1 ⎢ ⎥ ′′ PT Tˆ = √ (8) exp ⎣− ⎦ , σT = T-2 2σT 2πσT

that is clipped at lower and upper temperature limits due to the limitations of eq. (5) [8]. The PDF of the gas composition PY has to be multi-dimensional and due to a feasible calculation must be resolvable analytically. The multivariate β-pdf proposed by Girimaji [6] fullfils the requirements mentioned

206

M. Kindler, P. Gerlinger, M. Aigner

above and is defined by

PY Yˆ1 , · · · , YˆNk where βm = Y˜m B,

N Nk Nk k Γ β m m=1 δ 1− = .Nk Yˆmβm −1 Yˆm Γ (β ) m m=1 m=1 m=1 )m 1 − Y)m Y m=1

⎡ N

B=⎣

k

σY

⎤

− 1⎦ ,

σY =

Nk

m=1

′′ 2 Ym .

(9)

(10)

δ in (7) and (9) simply defines the δ-function. As the PDFs are fully described by the first (T) and Y)i ) and the second (σT and σY ) order moments – that are obtained by the set of equations (1) – the averaged chemical production rates may be calculated according to (6). For further details concerning the assumed PDF approach (e.g. the complete analytically derived source Jacobian matrix) the interested reader is refered to [8].

3 Lobed Strut Injector-Mixing Enhancement In this report two diﬀerent sets of combustion chamber geometries are investigated. Figure 1 shows a sketch of the combustion chamber. Upstream of the strut injector a Mach number of 2.8 is reached after expansion in the laval nozzle. The combustor cross section indicated by “injector and mixing” in Fig. 1 (6 × 40 mm) is constant till the diverging part (expansion angle = 2.5◦ ) of the combustion chamber begins. Thereby the length of the constant part of the combustion chamber is 48 mm in case of geometry I and 100 mm in case of geometry II respectively. The constant cross section after hydrogen injection is needed in case of combustion to keep the temperature high enough to ensure auto-ignition. On the other hand a diverging channel part is needed to compensate eﬀects from heat release due to combustion and to avoid thermal choking. The eﬀect of diﬀerent lenghts of the constant channel cross section part on mixing is investigated in this paper. Geometries and inflow conditions are defined in accordance with other partners in the GRK. The injection of hydrogen in axial flow direction is performed by a lobed strut injector (see Fig. 2) which creates streamwise vortices. The enhancement of mixing in comparison to planar struts is demonstrated in [12]. The dimensions of the strut are 80 mm in length, 40 mm in width and 6 mm in height. Studies on diﬀerent strut injectors may be found in [18, 19, 20, 21]. The simulation of the combustion chamber has been realized in two steps: a two-dimensional simulation of the laval nozzle to the middle of the strut and a three-dimensional simulation from the middle of the strut to the outlet of the combustion chamber, whereas the results of the two-dimensional calculation are taken as inlet-conditions. The grid of the two-dimensional simulation has

Assumed PDF Modeling of Turbulence Chemistry Interaction

207

diverging + combustion region

injector + mixing region α=2.5° laval nozzle

H2 Tt =295 K

air Ma=1 Tt =1070 K

Fig. 1. Schematic representation of the combustion chamber

Y Z

X

Fig. 2. Geometry of the lobed strut injector

about 8000 volumes and the calculations of the three-dimensional simulation has been performed with 800000 volumes. The solution is performed in a parallized way using MPI and 8 CPUs. Because a non reactive flow is calculated the memory requirements are moderate and a small number of CPUs is used only. 3.1 Results and Discussion In this section the mixing process (without combustion) of the two diﬀerent combustion chamber geometries are investigated (set I and II) which only differ by the length of the combustor section with constant cross section. Figure 3 shows calculated distributions of hydrogen and total pressure for the combustor geometry I. The vortices created by the lobed strut are clearly visible in the hydrogen distribution. To compare the diﬀerent geometries performance

208

M. Kindler, P. Gerlinger, M. Aigner

Fig. 3. Calculated hydrogen molar fractions (upper figure) and normalized total pressure (lower figure) for combustion chamber geometry I

parameters are calculated that are averaged over the channel cross section along the channel length. The mixing eﬃciency is defined by αρuYH2 dA (11) ηmix (x) ≡ A m ˙ H2 1/Φ : Φ ≥ 1 , with α= 1 : Φ<1 where φ is the equivalence ratio, A the cross section area and m ˙ H2 the total mass flux of injected hydrogen. As the second performance parameter the total pressure loss 1 pt = pt ρu dA (12) m ˙ A for every cross section is considered, where m ˙ is the total mass flux. Both performance parameters are given in Fig. 4 for the two combustion chamber

Assumed PDF Modeling of Turbulence Chemistry Interaction

209

Fig. 4. Normalized mixing eﬃciency and total pressure loss over the channel length for combustion chamber geometry I and II

geometries. It may be observed, that the losses in total pressure are nearly identical. This is in contrast to mixing eﬃciencies where diﬀerences up to 15% occur. In case of reactive flows this may have a strong influence on ignition delay and combustion process.

4 Supersonic Combustion Experiment There are only few experiments available in the field of supersonic combustion which allow an evalution of PDF structures. To the knowledge of the authors the experiment of Cheng et al. [3] is the only one where temperature, O2 , H2 , N2 , H2 O, and OH molar fractions have been measured simultaneously. This data allows an evaluation of the quality of the assumed PDF approach used in TASCOM3D. This test case has been investigated by the authors before but now refined grids and diﬀerent kinetic schemes [13] have been tested. Moreover the influence of modeling assumptions e.g. statistical independence of temperature and species PDF is tested [14]. Detailed information may be found in the cited literature. Here only a short overview and details concerning the numerical performance of the code on the SX-8 are given. The experimental axissymmetric set up consists out of a hydrogen jet which is injected at sonic speed into a vitiated Mach 2 surrounding flow. The temperature of hydrogen and air are 545 K and 1250 K respectively. Under these conditions a lifted flame is obtained. As shown in [13] the predicted lift-oﬀ length strongly depends on the reaction mechanism used. Because the thermodynamic conditions are close to the ignition limit a detailed chemical kinetic is required. The experiment is simulated by a 2D axissymetric 5-block grid with 232 · 36, 196 · 36, 220 · 120, 196 · 16, and 232 · 66 volumes are used. Chemistry is describe by the 19-step 9-species reaction mechanism of Jachi-

210

M. Kindler, P. Gerlinger, M. Aigner

Fig. 5. Supersonic combustion experiment of Cheng et al. [3], temperature, rms (root mean square) of temperature and OH molar fraction from top to bottom

mowski [4]. Figure 5 shows calculated distributions of temperature, rms of temperature fluctuations and OH molar fractions. All dimensions are normalized using the inner nozzle diameter of the hydrogen tube D = 2.362 mm. The flow is coming from left to right, x = 0 is at the end of the tubes and y = 0 is the symmetry axis. From the temperature distribution (upper most figure) it becomes clear that ignition takes place at about x/D = 20 which is in accordance with the experiment, where the point of ignition has been localized between x/D = 20 and 25 (there was some asymmetry in the experiment). At y/D < 1 and upstream of the point of ignition the colder hydrogen jet may be seen. Spacial gradients of temperature are responsible for the production of temperature fluctuations. Thus the diﬀerences between the relatively cold hydrogen and the hot surrounding air causes an increase in temperature variance which usually decreases the ignition delay. However, the strongest production of temperature variance is caused by the gradients at the main reaction zone as may be seen from the distribution plotted in the middle of

Assumed PDF Modeling of Turbulence Chemistry Interaction

211

Fig. 5. Finally the OH distribution shows where combustion takes place. A detailed investigation of profiles as well as of PDF structures for this test case may be found in the above cited references.

5 Performance The major drawback for a good performance of the investigated testcase on the NEC SX-8 is the short size of block 4 which in one coordinate direction consists out of 16 volumes only. Because of the extreme numerical stiﬀness caused by combustion (as in the present case) implicit numerical solvers are required. Moreover highly stretched grids with cell aspect ratios of up to 10000 are very common in supersonic combustion which also require an eﬃcient numerical solver. In the present case an implicit LU-SGS algorithm is used. Due to data dependencies in the implicit part of the solver the code is vectorized along the diagonals of every block. Along these diagonals an equation system has to be inverted at every grid point. The size corresponds to the number of variables and thus strongly depends on the chosen kinetic scheme. Thus the maximum vector length in the implicit part is given by the shorter number of volumes of both coordinate directions which may be relatively short in some cases. This is the price to be paid by using an implicit solver. In the present case the situation is especially bad because due the geometric condition block 4 only has few volumes in the radial coordinate direction. For the total code 3.0 GFLOPs are reached with an Vector Operation Ratio of 97%. The average vector length is 119 for this test case. About 85% of the computational time is consumed in two subroutines only: one for the simulation of PDF averaged chemical production rates and source term Jacobians (about 45%) and the second one for the described implicit solver (also about 38%). The diﬀerences between both subroutines are dramatic. Chemistry is a local phenomena and the corresponding subroutine reaches a vector length of 253 and about 4.2 GFLOPS. This is in contrast to the implicit solver with an average vector length of 44 and about 1.3 GFLOPS. If possible the code always uses one dimensional arrays in subroutines to obtain a good vector performance. Thus the calculation of gas properties reaches an vector operation ratio of 99.8%, a vector length of 252.6 and 10 GFLOPS. As described before the greatest problem for an excellect vector performance is caused by the implicit part. We will try to perform here some optimization to achieve a further improvement. In total the performance of the code is quite good. Moreover the limitations concerning the vector performance in the implicit part are much smaller for 3D simulations due to longer vector lengths. An investigation for a 3D combustor simulation will be performed next.

212

M. Kindler, P. Gerlinger, M. Aigner

6 Conclusion In the frame of the Graduiertenkolleg 1095/1 investigations of scramjet combustors are performed using the in house code TASCOM3D. The code has been tested on the NEC SX-8 and has shown to have a good vector performance especially if it is taken into account, that a vecorization of the implicit part is possible along diagonals only. The code has been used for 2D and 3D simulations. The combustor geometry defined in the GRK has been simulated and the used strut injector is investigated for cold supersonic mixing. A detailed study concerning the employed assumed PDF approach has been performed. This method has proven to achieve quite good results and it is still eﬃcient enough to be used with finite-rate chemistry for complex 3D combustor simulations. Acknowledgements We wish to thank the Deutsche Forschungsgemeinschaft (DFG) for financial support of this work within the GRK 1095/1 at the University of Stuttgart. The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number scrcomb.

References 1. Jameson, A., Yoon, S.: Lower-Upper Implicit Scheme with Multiple Grids for the Euler Equations, AIAA Journal, 25, pp. 929–937, 1987 2. Shuen, J.S.: Upwind Diﬀerencing and LU Factorization for Chemical NonEquilibrium Navier-Stokes Equations, Journal of Computational Physics, 99, pp. 233–250, 1992 3. Cheng, T.S., Wehrmeyer, J.A., Pitz, E.W., Jarret, O., Northam, G.B.: Mixing Enhancement in Compressible Mixing Layers: An Experimental Study, Combust. Flame 99, pp. 157–173, 1994 4. Jachimowski, C.: An Analytical Study of the Hydrogen-Air Reaction Mechanism with Application to Scramjet Combustion, Nasa-tp-2791, 1988 5. Baurle, R.A., Alexopoulos, G.A., Hassan, H.A.: Modeling of Supersonic Combustion Using Assumed Probability Density Functions, J. Prop. Power 10, pp. 777–786, 1994 6. Girimaji, S.S.: A Simple Recipe for Modeling Reacting-Rates in Flows with Turbulent Combustions, AIAA paper 91–1792, 1991 7. Gerlinger, P., Br¨ uggemann, D.: An Implicit Multigrid Scheme for the Compressible Navier-Stokes Equations with Low-Reynolds-Number Turbulence Closure, Journal of Fluids Engineering, 120, pp. 257–262, 1998 8. Gerlinger, P., M¨ obus, H., Br¨ uggemann, D.: An Implicit Numerical Scheme for Turbulent Combustion Using an Assumed PDF Approach, AIAA paper 99– 3775, 1999

Assumed PDF Modeling of Turbulence Chemistry Interaction

213

9. Gerlinger, P., Br¨ uggemann, D.: Numerical Investigation of Hydrogen Strut Injections into Supersonic Air Flows, J. of Prop. and Power, 16, pp. 22–28, 2000 10. Gerlinger, P., M¨ obus, H., Br¨ uggemann, D.: An Implicit Multigrid Method for Turbulent Combustion, Journal of Computational Physics, 167, pp. 247–276, 2001 11. Gerlinger, P.: Investigations of an Assumed PDF Approach for Finite-RateChemistry, Combustion Science and Technology, 175, pp. 841–872, 2003 12. Gerlinger, P., Stoll, P., Kindler, M., Schneider, F., Aigner, M.: Numerical Investigation of Mixing and Combustion Enhancement in Supersonic Combustors by Strut Induced Steamwise Vorticity, to appear in Aerospace Science and Technology, 2007. 13. Gerlinger, P., Nold, K., Aigner, M.: Investigation of Hydrogen/Air Reaction Mechanisms for the Simulation of Supersonic Combustion, submitted for publication in Combustion and Flame. 14. Gerlinger, P, Schneider, F., Aigner, M.: Multivariate Assumed PDF Modeling of Supersonic Combustion, ISABE-2007-1315 15. Stoll, P., Gerlinger, P., Br¨ uggemann, D.: Domain Decomposition for an Impicit LU-SGS Scheme Using Overlapping Grids, AIAA-paper 97–1896, 1997 16. Stoll, P., Gerlinger, P., Br¨ uggemann, D.: Implicit Preconditioning Method for Turbulent Reacting Flows, Proceedings of the 4th ECCOMAS Conference, 1, pp. 205–212, John Wiley & Sons, 1998 17. Coakley, T. J., Huang, P. G.: Turbulence Modeling for High Speed Flows, AIAAPaper 92-0436, 1992 18. Glawe, D., Samimiy, M., Nejad, A., Cheng, T.: Eﬀects of Nozzle Geometry on Parallel Injection from Base of an Extended Strut into Supersonic Flow, AIAAPaper 95-0522, 1995 19. Strickland, J., Selerland, T., Karagozian, A. R.: Numerical Simulation of a Lobed Fuel Injector, Journal of Propulsion Power 10, pp. 2950–2964, 1998 20. Charyulu, B., Kurian, J., Venugopalan, P., Sriamulu, V.: Experimental Study on Mixing Enhancement in Two Dimensional Supersonic Flow, Experiments in Fluids 24, pp. 340–346, 1998 21. Sunami, T., Wendt, M., Nishioka, M.: Supersonic Mixing and Combustion Control Using Streamwise Vorticity, AIAA-Paper 98-3271, 1998

Simulations of Premixed Swirling Flames Using a Hybrid Finite-Volume/Transported PDF Approach Stefan Lipp1 and Ulrich Maas2 1

2

University of Karlsruhe, Institute for Technical Thermodynamics [email protected] University of Karlsruhe, Institute for Technical Thermodynamics [email protected]

Summary. The mathematical modeling of swirling ﬂames is a diﬃcult task due to the intense coupling between turbulent transport processes and chemical kinetics in particular for instationary processes like the combustion induced vortex breakdown. In this paper a mathematical model to describe the turbulence-chemistry interaction is presented. The described method consists of two parts. Chemical kinetics are taken into account with reduced chemical reaction mechanisms, which have been developed using the ILDM-Method (“Intrinsic Low-Dimensional Manifold”). The turbulencechemistry interaction is described by solving the joint probability density function (PDF) of velocity and scalars. Simulations of test cases with simple geometries verify the developed model.

1 Introduction In many industrial applications there is a high demand for reliable predictive models for turbulent swirling flows. While the calculation of non-reacting flows has become a standard task and can be handled using Reynolds averaged Navier-Stokes (RANS) or Large Eddy Simulation (LES) methods the modeling of reacting flows still is a challenging task due to the diﬃculties that arise from the strong non-linearity of the chemical source term which can not be modeled satisfactorily by using oversimplified closure methods. PDF methods (probability density function) show a high capability for modeling turbulent reactive flows, because of the advantage of treating convection and finite rate non-linear chemistry exactly [1, 2]. Only the eﬀect of molecular mixing has to be modeled [3]. In the literature diﬀerent kinds of PDF approaches can be found. Some use stand-alone PDF methods in which all flow properties are computed by a joint probability density function method [4, 5, 6, 7]. The transport equation for the joint probability density function that can be derived from the Navier-Stokes equations still contains

216

S. Lipp, U. Maas

unclosed terms that need to be modeled. These terms are the fluctuating pressure gradient and the terms describing the molecular transport. In contrast the above mentioned chemistry term, the body forces and the mean pressure gradient term already appear in closed form and need no more modeling assumtions. Compared to RANS methods the structure of the equations appearing in the PDF context is remarkably diﬀerent. The moment closure models (RANS) result in a set of partial diﬀerential equations, which can be solved numerically using finite-diﬀerence or finite-volume methods [8]. In contrast the transport equation for the PDF is a high-dimensional scalar transport equation. In general it has 7 + nS dimensions which consist of three dimensions in space, three dimensions in velocity space, the time and the number of species nS used for the description of the thermokinetic state. Due to this high dimensionality it is not feasible to solve the equation using finite-diﬀerence of finite-volume methods. For that reason Monte Carlo methods have been employed, which are widely used in computational physics to solve problems of high dimensionality, because the numerical eﬀort increases only linearly with the number of dimensions. Using the Monte Carlo method the PDF is represented by an ensemble of stochastic particles [9]. The transport equation for the PDF is transformed to a system of stochastic ordinary diﬀerential equations. This system is constructed in such a way that the particle properties, e.g. velocity, scalars, and turbulent frequency, represent the same PDF as in the turbulent flow. In order to fulfill consistency of the modeled PDF, the mean velocity field derived from an ensemble of particles needs to satisfy the mass conservation equation [1]. This requires the pressure gradient to be calculated from a Poission equation. The available Monte Carlo methods cause strong bias determining the convective and diﬀusive terms in the momentum conservation equations. This leads to stability problems calculating the pressure gradient from the Poisson equation. To avoid these instabilities diﬀerent methods to calculate the mean pressure gradient where used. One possibility is to couple the particle method with an ordinary finite-volume or finite-diﬀerence solver to optain the mean pressure field from the Navier-Stokes equations. These so called hybrid PDF/CFD methods are widely used by diﬀerent authors for many types of flames [10, 11, 12, 13, 14, 15]. In the presented paper a hybrid scheme is used. The fields for mean pressure gradient and a turbulence charactaristic, e.g. the turbulent time scale, are derived solving the Reynolds averaged conservation equations for momentum, mass and energy for the flow field using a finite-volume method. The eﬀect of turbulent fluctuations is modeled using a k-τ model [16]. Chemical kinetics are taken into account by using the ILDM method to get reduced chemical mechanisms [17, 18]. In the presented case the reduced mechanism describes the reaction with three parameters which is on the one hand few enough to limit the simulation time to an acceptable extent and on the other hand suﬃciently high to get a detailed description of the chemical reaction.

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

217

The test case for the developed model is a model combustion chamber investigated by serveral authors [19, 20, 21, 22]. With their data the results of the presented simulations are validated.

2 Numerical Model As mentioned above a hybird CFD/PDF method is used in this work. In Fig. 1 a complete sketch of the solution precedure can be found. Before explaning the details of the implemented equations and discussing consistency and numerical matters the idea of the solution procedure shall be briefly overviewed. The calulation starts with a CFD step in which the Navier-Stokes equations for the flow field are solved by a finite-volume method. The resulting mean pressure gradient together with the mean velocities and the turbulence characteristics is handed over to the PDF part. Here the joint probability density function of the scalars and the velocity is solved by a particle Monte Carlo method. The reaction progress is taken into account by reading from a lookup table based on a mechanism reduced with the ILDM method. As a result of this step the mean molar mass, the composition vector and the mean temperature field are returned to the CFD part. This internal iteration is performed until convergence is achieved. ∂p ¯ ,u ˜, v˜, w, ˜ k, τ ∂xi

?

CFD

PDF

6 R M

, ψi , T

Fig. 1. Scheme of the coupling of CFD and PDF

2.1 CFD Model The CFD code which is used to calculate the mean velocity and pressure field along with the turbulent kinetic energy and the turbulent time scale is called Sparc3 and was developed by the Department of Fluid Machinery at Karlsruhe University. It solves the Favre-averaged compressible Navier Stokes equations using a Finite-Volume method on block structured non-uniform meshes. In this work a 2D axi-symmetric solution domain is used. Turbulence closure is provided using a two equation model solving a transport equation for the turbulent kinetic energy and a turbulent time scale [16]. 3

Structured Parallel Research Code

218

S. Lipp, U. Maas

In detail the equations read ρu ˜i ) ∂ ρ¯ ∂ (¯ + =0 ∂t ∂xi ∂ ∂ (¯ ρu ˜i ) ′′ ′′ + ρ¯u ˜i u ˜j + ρui uj + ρ¯δij − τij = 0 ∂t ∂xj ∂ (∂ ρ¯e˜) ′′ ′′ + ρ¯u˜j e˜ + u˜j p¯ + uj p + ρuj e′′ + q¯j − ui τij = 0 ∂t ∂xj

(1) (2) (3)

which are the conservation equations for mass, momentum and energy in Favre average manner, respectively. Modeling of the unclosed terms in the energy equation will not be described in detail any further but can be found for example in [8]. The unclosed cross correlation term in the momentum conservation equation is modeled using the Boussinesq approximation ∂u ˜j ∂u ˜i ′′ ′′ (4) ρui uj = ρ¯μT + ∂xj ∂xi with μT = Cμ fμ kτ .

(5)

The parameter Cμ is an empirical constant with a value of Cμ = 0.09 and fμ accounts for the influence of walls. The turbulent kinetic energy k and the turbulent time scale τ are calculated from their transport equation which are [16] ∂ μT k ∂ui ∂k ∂k ∂k + ρ¯u ˜j μ+ (6) = τij − ρ¯ + ρ¯ ∂t ∂xj ∂xj τ ∂xi σk ∂xj ∂τ τ ∂τ ∂ui ρ¯ + ρ¯u ˜j + (Cǫ2 − 1) ρ¯ + = (1 − Cǫ1 ) τij ∂t ∂x j k ∂xj μT μT 2 ∂ ∂k ∂k ∂τ μ+ μ+ + − ∂xj στ 2 ∂xj k στ 1 ∂xk ∂xk μT 2 ∂τ ∂τ μ+ . (7) τ στ 2 ∂xk ∂xk Here Cǫ1 = 1.44 and στ 1 = στ 2 = 1.36 are empirical model constants. The parameter Cǫ2 is calculated from the turbulent Reynolds number Ret . Ret = Cǫ2

kτ μ

2 2 = 1.82 1 − exp (−Ret /6) 9

(8) (9)

2.2 Joint PDF Model In the literature many diﬀerent joint PDF models can be found, for example models for the joint PDF of velocity and composition [23, 24] or for the joint

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

219

PDF of velocity, composition and turbulent frequency [25]. A good overview of the diﬀerent models can be found in [12]. In most joint PDF approaches a turbulent (reactive) flow field is described by a one-time, one-point joint PDF of certain fluid properties. At this level chemical reactions are treated exactly without any modeling assumptions [1]. However, the eﬀect of molecular mixing has to be modeled. The state of the fluid at a given point in space and time can be fully T described by the velocity vector V = (V1 , V2 , V3 ) and the composition vector of nS − 1 species and the enthalpy h Ψ containing the mass fractions T Ψ = (Ψ1 , Ψ2 , . . . , Ψns −1 , h) . The probability density function is fUφ (V, Ψ; x, t) dVdΨ = Prob (V ≤ U ≤ V + dV, Ψ ≤ Φ ≤ Ψ + dΨ) (10)

and gives the probability that at one point in space and time one realization of the flow is within the interval V ≤ U ≤ V + dV

(11)

Ψ ≤ Φ ≤ Ψ + dΨ

(12)

for its velocity vector and

for its composition vector. According to [1] a transport equation for the joint PDF of velocity and composition can be derived. Under the assumption that the eﬀect of pressure fluctuations on the fluid density is negligible the transport equation writes ˜ % ∂ f˜ ∂ p ∂ $ ∂ f˜ ∂f ρ(Ψ) ρ(Ψ)Sα (Ψ)f˜ + ρ(Ψ)Uj + ρ(Ψ)gj − + ∂xj ∂xj ∂Uj ∂Ψα / 01 ∂t2 / 01 2 01 2 / 01 2 / I

II

=

∂ ∂Uj /

IV

III

3 4 ∂τij ∂Ji ∂p ∂ − − + |U, Ψ f˜ + |U, Ψ f˜ . (13) ∂xi ∂xi ∂Ψα ∂xi 01 2 01 2 / ′

V

VI

Term I describes the instationary change of the PDF, Term II its change by convection in physical space and Term III takes into account the influence of gravitiy and the mean pressure gradient on the PDF. Term IV includes the chemical source term which describes the change of the PDF in composition space due to chemical reactions. All terms on the left hand side of the equation appear in closed form, e.g. the chemical source term. In contrast the terms on the right hand side are unclosed and need further modeling. Many closing assumptions for these two terms exist. In the following only the ones that are used in the present work shall be explained further. Term V describes the influence of pressure fluctuations and viscous stresses on the PDF. Commonly a Langevin approach [26, 27] is used to close this term.

220

S. Lipp, U. Maas

In the presented case the SLM (Simplified Langevin Model) is used [1]. More sophisticated approaches that take into account the eﬀect of non-isotropic turbulence or wall eﬀects exist as well [26, 28]. But in the presented case of a swirling non-premixed free stream flame the closure of the term by the SLM is assumed to be adequate and was chosen because of its simplicity. Term VI regards the eﬀect of molecular diﬀusion within the fluid. This diﬀusion flattens the steep composition gradients which are created by the strong vortices in a turbulent flow. Several models have been proposed to close this term. The simplest model is the interaction by exchange with the mean model (IEM) [29, 30] which models the fact that fluctuations in the composition space relax to the mean. A more detailed model has been proposed by Curl [31] and modified by [32, 33] and is used in its modified form in the presented work. More recently new models based on Euclidian minimum spanning trees have been developed [34, 35] but are not yet implemented in this work. As mentioned previously it is numerically unfeasable to solve the PDF transport equation with finite-volume or finite-diﬀerence methods because of its high dimensionality. Therefore a Monte Carlo method is used to solve the transport equation making use of the fact that the PDF of a fluid flow can be represented as a sum of δ-functions. N (t) ∗ fU,φ

(U, Ψ; x, t) =

δ v − ui δ φ − Ψi δ x − xi

(14)

i=1

Appling this fact the PDF of the fluid flow can be disreticed by a set of stochastic particles. For these particles equations of motion in space, velocity space and compostion space can be derived. Now, instead of the high dimensional PDF transport equation a set of (stochastic) ordinary diﬀerential equations are solved. The evolution of the particle position X∗i reads dX∗i = U∗i (t) dt

(15)

in which U∗i is the velocity vector for each particle. The evolution of the particles in the velocity space can be calculated according to the Simplified Langevin Model [1] by 5 ∂ p¯ dt C0 k 1 3 dUi∗ ∗ =− + C0 [Ui − Ui ] + dWi . dt − (16) dt ∂xi 2 4 τ τ For simplicity the equation is here only written for the U component of the velcity vector U = (U, V, W )T belonging to the spacial coordinate x T (x = (x, y, z) ). The equations of the other components V, W look accord∂ p¯ denotes the mean pressure gradient, Ui the mean particle ingly. In (16) ∂x i velocity, t the time, dWi a diﬀerential Wiener increment, C0 a model constant, k and τ the turbulent kinetic energy and the turbulent time scale, respectively.

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

221

Finally the evolution of the composition vector can be calculated as dΨ =S+M (17) dt in which S is the chemical source term (appearing in closed form) and M denotes the eﬀect of molecular mixing. As previously mentioned this term is unclosed und needs further modeling assumptions. For this a modified Curl model is used [32]. 2.3 Chemical Kinetics The source term appearing in (17) is calculated from a lookup table which is created using automatically reduced chemical mechanisms. The deployed technique to create these tables is the ILDM method (“Intrinsic Low-Dimensional Manifold”) by Maas and Pope [17, 18]. The basic idea of this method is the identification and separation of fast and slow time scales. In typical turbulent flames the time scales governing the chemical kinetics range from 10−9 s to 102 s. This is a much larger spectrum than that of the physical processes (e.g. molecular transport) which vary only from 10−1 s to 10−5 s. Reactions that occur in the very fast chemical time scales are in partial equilibrium and the species are in steady state. These are usually responsible for equilibrium processes. Making use of this fact it is possible to decouple the fast time scales. The main advantage of decoupling the fast time scales is that the chemical system can be described with a much smaller number of variables (degrees of freedom). In our test case the chemical kinetics are described with only three parameters namely the mixure fraction, the mole fraction of CO2 and the mole fraction of H2 O instead of the 34 species (degrees of freedom) appearing in the detailed methane reaction mechanism. Further details of the method and its implementation can be found in [17, 18].

3 Results and Discussion As a test case for the presented model simulations of a premixed, swirling, confined flame are performed. A sketch of the whole test rig is shown in Fig. 2. Details of the test rig and the experimental data can be found in [20, 21, 22]. The test rig consists of a plenum containing a premixed methane-air mixture, a swirl generator, a premixing duct and the combustion chamber itself. In general three diﬀerent modes exist to stabilize flames. Flames can be stabilized by a small stable burning pilot, by bluﬀ-bodies inserted into the main flow or by aerodynamic arrangements creating a recirculation zone above the burner exit. The last possibility has been increasingly employed for flame stabilization in the gas turbine industry. The recirculation zone (often also abreviated IRZ 4 ) is a region of negative axial velocity close to the symme4

Internal Recirculation Zone

222

S. Lipp, U. Maas Combustion chamber Plenum Premix duct

Swirl generator Premixed gas Flame

Fig. 2. Sketch of the investigated combustion chamber

try line (see Fig. 2). Heat and radicals are transported upstream towards the flame tip causing a stable operation of the flame. The occurrence and stability of the IRZ depend crucially on the swirl number, the geometry, and the profiles of the axial and tangential velocity. Simulations were performed using a 2D axi-symmetric grid with approximately 15000 cells. The PDF is discretized with 50 particles per cell. The position of the simulated domain is shown in Fig. 3. Only every forth grid line is shown for clarity. In this case the mapping of the real geometry (3D) on the 2D axi-symmetric solution domain is possible since the experiments show that all essential features of the flow field exhibit the two dimensional axi-symmetric behaviour [36]. The mapping approach has shown to be valid for the modeling also in [19]. In order to consider the influence of the velocity profiles created by the swirl gernerator radial profiles of all flow quantities served as inlet boundary conditions. These profiles stem from detailed 3D simulations of the whole test rig using a Reynolds stress turbulence closure and have been taken from the literature [22]. The global operation parameters are an equivalence ratio of φ = 1, an inlet mass flow of 70 gs , a preheated temperature of 373 K and a swirl number of S = 0.5. First of all simulations of the non-reacting case were done to validate the CFD model and the used boundary conditions which are mapped from the

Fig. 3. Position of the mesh in the combustion chamber

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

223

Fig. 4. Contourplot of the axial velocity component (with steamtraces)

detailed 3D simulations. Figure 4 shows an example of the achieved results. From the steamtraces one can see two areas with negative axial velocity. One in the upper left corner of the combustion chamber is caused by the step in the geometry and one close to the symmetry line which is caused aerodynamically by the swirl. This area is the internal recirculation zone described above which is in the reactive case used to stabilize the flame. These simulations are validated with experimental results from [20, 21]. The comparison of the experimental data and the results of the simulations for one case are exemplarily shown in Fig. 5 and Fig. 6. In all figures the radial coordinate is plotted over the velocity. Both upper figures show the axial velocity, both lower show the tangential velocity. The lines denote the results of the simulations the scatters denote the results of the measurements. The two axial positions are arbitrarily chosen from the available experimental data. The (relative) x coordinates refer to the beginning of the premixing duct (Fig. 2). In both cases the profiles of the simulations seem to match reasonably well with the measured data. So the presented model gives a sound description of the flow field of the investigated test case. As an example for the reacting case the calculated temperature field of the flame is shown in Fig. 7 which can not be compared to quantitative experiments due to the lack of data. But the qualitative behaviour of the flame is predicted correctly. As one can see the tip of the flame is located at the start of the inner recirulation zone. It shows a turbulent flame brush in which the reaction occurs which can be seen in the figure by the rise of temperature. It can not be assessed whether the thickness of the reaction zone is predicted well because no measurements of the temperature field are available.

S. Lipp, U. Maas 0.04

0.04

0.03

0.03 r / (m)

r / (m)

224

0.02 0.01

0.02 0.01

−10 0 10 20 30 40 50 60 70 u / (m/s)

−10 0 10 20 30 40 50 60 70 u / (m/s)

(a) Axial velocity

0.04

0.04

0.03

0.03 r / (m)

r / (m)

(a) Axial velocity

0.02 0.01

0.02 0.01

−10 0 10 20 30 40 50 60 70 w / (m/s)

−10 0 10 20 30 40 50 60 70 w / (m/s)

(b) Tangential velocity

(b) Tangential velocity

Fig. 5. x = 29 mm

Fig. 6. x = 170 mm

Fig. 7. Temperature ﬁeld

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

225

4 Conclusion Simulations of a premixed swirling methane-air flame are presented. To account for the strong turbulence chemistry interaction occuring in these flames a hybrid finite-volume/transported PDF model is used. This model consists of two parts: a finte volume solver for the mean velocities and the mean pressure gradient and a Monte Carlo solver for the transport equation of the joint PDF of velocity and composition vector. Chemical kinetics are described by automatically reduced mechanisms created with the ILDM method. The presented results show the validity of the model. The simulated velocity profiles for the non-reacting case match well with the experimental results. The calculations of the reacting case also show a qualitatively correct behaviour of the flame. A quantitative analysis is subject of future research work. Acknowledgements The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number FLASPARC. The work of Peter Lammers (HLRS) transporting and optimizing the code on SX-8 is gratefully acknowledged.

References 1. S.B. Pope. Pdf methods for turbulent reactive ﬂows. Progress in Energy Combustion Science, 11:119–192, 1985. 2. S.B. Pope. Lagrangian pdf methods for turbulent ﬂows. Annual Review of Fluid Mechanics, 26:23–63, 1994. 3. Z. Ren and S.B. Pope. An investigation of the performence of turbulent mixing models. Combustion and Flame, 136:208–216, 2004. 4. P.R. Van Slooten and S.B. Pope. Application of pdf modeling to swirling and nonswirling turbulent jets. Flow Turbulence and Combustion, 62(4):295–334, 1999. 5. V. Saxena and S.B. Pope. Pdf simulations of turbulent combustion incorporating detailed chemistry. Combustion and Flame, 117(1-2):340–350, 1999. 6. S. Repp, A. Sadiki, C. Schneider, A. Hinz, T. Landenfeld, and J. Janicka. Prediction of swirling conﬁned diﬀusion ﬂame with a monte carlo and a presumedpdf-model. International Journal of Heat and Mass Transfer, 45:1271–1285, 2002. 7. K. Liu, S.B. Pope, and D.A. Caughey. Calculations of bluﬀ-body stabilized ﬂames using a joint probability density function model with detailed chemistry. Combustion and Flame, 141:89–117, 2005. 8. J.H. Ferziger and M. Peric. Computational Methods for Fluid Dynamics. Springer Verlag, 2 edition, 1997. 9. S.B. Pope. A monte carlo method for pdf equations of turbulent reactive ﬂow. Combustion, Science and Technology, 25:159–174, 1981.

226

S. Lipp, U. Maas

10. P. Jenny, M. Muradoglu, K. Liu, S.B. Pope, and D.A. Caughey. Pdf simulations of a bluﬀ-body stabilized ﬂow. Journal of Computational Physics, 169:1–23, 2000. 11. A.K. Tolpadi, I.Z. Hu, S.M. Correa, and D.L. Burrus. Coupled lagrangian monte carlo pdf-cfd computation of gas turbine combustor ﬂowﬁelds with ﬁnite-rate chemistry. Journal of Engineering for Gas Turbines and Power, 119:519–526, 1997. 12. M. Muradoglu, P. Jenny, S.B. Pope, and D.A. Caughey. A consistent hybrid ﬁnite-volume/particle method for the pdf equations of turbulent reactive ﬂows. Journal of Computational Physics, 154:342–370, 1999. 13. M. Muradoglu, S.B. Pope, and D.A. Caughey. The hybid method for the pdf equations of turbulent reactive ﬂows: Consistency conditions and correction algorithms. Journal of Computational Physics, 172:841–878, 2001. 14. G. Li and M.F. Modest. An eﬀective particle tracing scheme on structured/unstructured grids in hybrid ﬁnite volume/pdf monte carlo methods. Journal of Computational Physics, 173:187–207, 2001. 15. V. Raman, R.O. Fox, and A.D. Harvey. Hybrid ﬁnite-volume/transported pdf simulations of a partially premixed methane-air ﬂame. Combustion and Flame, 136:327–350, 2004. 16. H.S. Zhang, R.M.C. So, C.G. Speziale, and Y.G. Lai. A near-wall two-equation model for compressible turbulent ﬂows. In Aerospace Siences Meeting and Exhibit, 30th, Reno, NV, page 23. AIAA, 1992. 17. U. Maas and S.B. Pope. Simplifying chemical kinetics: Intrinsic low-dimensional manifolds in composition space. Combustion and Flame, 88:239–264, 1992. 18. U. Maas and S.B. Pope. Implementation of simpliﬁed chemical kinetics based on intrinsic low-dimensional manifolds. In Twenty-Fourth Symposium (International) on Combustion, pages 103–112. The Combustion Institute, 1992. 19. F. Kiesewetter, C. Hirsch, J. Fritz, M. Kr¨ oner, and T. Sattelmayer. Twodimensional ﬂashback simulation in strongly swirling ﬂows. In Proceedings of ASME Turbo Expo 2003. 20. M. Kr¨ oner. Einfluss lokaler L¨ oschvorg¨ ange auf den Flammenr¨ uckschlag durch verbrennungsinduziertes Wirbelaufplatzen. PhD thesis, Technische Universit¨ at M¨ unchen, Fakult¨ at f¨ ur Maschinenwesen, 2003. 21. J. Fritz. Flammenr¨ uckschlag durch verbrennungsinduziertes Wirbelaufplatzen. PhD thesis, Technische Universit¨ at M¨ unchen, Fakult¨ at f¨ ur Maschinenwesen, 2003. 22. F. Kiesewetter. Modellierung des verbrennungsinduzierten Wirbelaufplatzens in Vormischbrennern. PhD thesis, Technische Universit¨ at M¨ unchen, Fakult¨ at f¨ ur Maschinenwesen, 2005. 23. D.C. Haworth and S.H. El Tahry. Propbability density function approach for multidimensional turbulent ﬂow calculations with application to in-cylinder ﬂows in reciproating engines. AIAA Journal, 29:208, 1991. 24. S.M. Correa and S.B. Pope. Comparison of a monte carlo pdf ﬁnite-volume mean ﬂow model with bluﬀ-body raman data. In Twenty-Fourth Symposium (International) on Combustion, page 279. The Combustion Institute, 1992. 25. W.C. Welton and S.B. Pope. Pdf model calculations of compressible turbulent ﬂows using smoothed particle hydrodynamics. Journal of Computational Physics, 134:150, 1997. 26. D.C. Haworth and S.B. Pope. A generalized langevin model for turbulent ﬂows. Physics of Fluids, 29:387–405, 1986.

Simulations of Premixed Flames Using a Hybrid FV/PDF Approach

227

27. H.A. Wouters, T.W. Peeters, and D. Roekaerts. On the existence of a generalized langevin model representation for second-moment closures. Physics of Fluids, 8, 1996. 28. T.D. Dreeben and S.B. Pope. Pdf/monte carlo simulation of near-wall turbulent ﬂows. Journal of Fluid Mechanics, 357:141–166, 1997. 29. C. Dopazo. Relaxation of initial probability density functions in the turbulent convection of scalar ﬂieds. Physics of Fluids, 22:20–30, 1979. 30. P.A. Libby and F.A. Williams. Turbulent Reacting Flows. Academic Press, 1994. 31. R.L. Curl. Dispersed phase mixing: 1. theory and eﬀects in simple reactors. A.I.Ch.E. Journal, 9:175,181, 1963. 32. J. Janicka, W. Kolbe, and W. Kollmann. Closure of the transport equation of the probability density function of turbulent scalar ﬂieds. Journal of NonEquilibrium Thermodynamics, 4:47–66, 1979. 33. S.B. Pope. An improved turbulent mixing model. Combustion, Science and Technology, 28:131–135, 1982. 34. S. Subramaniam and S.B. Pope. A mixing model for turbulent reactive ﬂows based on euclidean minimum spanning trees. Combustion and Flame, 115(4):487–514, 1999. 35. S. Subramaniam and S.B. Pope. Comparison of mixing model performance for nonpremixed turbulent reactive ﬂow. Combustion and Flame, 117(4):732–754, 1999. 36. J. Fritz, M. Kr¨ oner, and T. Sattelmayer. Flashback in a swirl burner with cylindrical premixing zone. In Proceedings of ASME Turbo Expo 2001.

Computations of Premixed Turbulent Flames M. Lecanu1 , K. Mehravaran1, J. Fr¨ ohlich1 , H. Bockhorn1, and D. Th´evenin2 1

2

Institute for Technical Chemistry and Polymer Chemistry, University of Karlsruhe, Germany, [email protected] Laboratory of Fluid Dynamics and Technical Flows, University of Magdeburg “Otto von Guericke”, Germany

1 Motivation Until now and at least for the next few decades, combustion is and will be the major source of mechanical and electrical energy. For economic and ecological reasons, it is of primordial interest to master phenomena involved in a very important number of processes using combustion. The SFB 606, a german research initiative on unsteady combustion, established at the University of Karlsruhe aims to improve the understanding and control of such processes. The research program is divided into several parts dedicated to the study of fundamental phenomena. One numerical part of SFB 606, Project B8 is devoted to the investigation of eﬀects of diﬀerential diﬀusion on the propagation of flames in premixed gas. These characterized by the Lewis number which is the ratio between diﬀusion of heat and mass, respectively. The goal of the present reseacht is to understand these mechanisms by using Direct Numerical Simulations (DNS) in comparaison with related experiments also realized in Karlsruhe within SFB 606. This will eventually allow comparison and close cross–validation. Such studies are the basis of modeling reactions in simulations of turbulent premixed flames and could be used in less costly numerical approaches such as statistical models or Large Eddy Simulation. In DNS, no model for turbulence is used and all the diﬀerent length scales of the flow are computed, from the smallest in the flame front until the very large vortex structures. For these reasons the resolution demands are very high and hence the required CPU time. The present calculations in 2-D have been performed on the HP XC1 Cluster at the Karlsruhe Super Computing Center. 1.1 Relevance for Premixed Flames In a wide range of the physical parameters, the eﬀect of turbulence on a premixed flame is not limited to the wrinkling of the flame front. For example, when a flame front is submitted to suﬃciently strong aerodynamic stretch,

230

M. Lecanu et al.

quenching may appear locally and completely suppress the combustion process. Weaker turbulence, on the other hand, increases the propagation velocity of the flame front. From experimental data [1] or theoretical analysis, a first standard model of turbulent flame speed ST is ′ n ST u =1+α (1) SL SL where SL is the laminar flame speed, u′ is the root-mean-square (rms) of the velocity fluctuations. α and n are model constants close to unity. This is a relatively simple model, though, which neglects all other eﬀects. A more sophisticated model was proposed by Schmid [2]: ST u′ =1+ SL SL

Da2T 1 + Da2T

14

,

(2)

where DaT is the turbulent Damk¨ ohler number. Unfortunately, these models are limited and other factors, such as the flame front curvature become important when the Lewis number is diﬀerent from unity. These Lewis number eﬀects, also known as thermo-diﬀusive eﬀects, will be studied in the present project. Special attention will also be paid to eﬀects of flame curvature and stretch. Furthermore, the flame in turn aﬀects the turbulence and a universal model should include this as well. To study the eﬀects cited, it is mandatory to treat the full chemical system of reactions with all intermediate species, or at least a very good approximation of it, which then also involves many species. For each species a separate transport equation has to be solved and the chemical source terms have to be evaluated. 1.2 Experiments of Zarzalis in the Framework of SFB 606 The experimental setup of Weiss and Zarzalis, investigated in Projekt A9 of SFB 606, is considered as a basis for the present simulations. These experiments investigate isochoric premixed spherical flames, evolving in a cubic box (Fig. 1). The burning velocity and the Markstein number are determined.

Fig. 1. Experimental setup of Weiss and Zarzalis – Project A9

Computations of Premixed Turbulent Flames

231

The Markstein number M a is a suitable parameter to quantify the influence of flame strech and has to be included in the modeling of turbulent flame velocity [3, 4, 5]. 1.3 Parameter Range The volume of the vessel used in the experiment is 2.3 l and the internal pressure can be raised up to 80 or 150 bar, depending on whether the optical access is used or not. So far, H2 -Air and CH4 -Air premixed laminar and turbulent flames with diﬀerent initial compositions have been investigated with this device. The gas mixture is ignited by a spark, located in the center of the box. Turbulence is generated and maintained with the use of eight fans located at the corners of the cubic box. The range of the turbulent fluctuations intensity is 0–4 m/s. The Markstein number can be measured with a Schlieren method. 1.4 Flow Conﬁgurations The present work is concerned with 2-D configurations of premixed flames. The main parameters of the simulations are: Thermochemistry: H2 -Air mechanism, 9 species / 37 steps CH4 -Air mechanism, 50 species / 300 steps Initial conditions: Two kinds of calculations with diﬀerent intial conditions have been performed (Fig. 2). These are 2-D spherical flames and 2-D planar flames. Here, the second configuration is discussed. Its initial condition consists of a 1-D steady state solution obtained using the 1-D code Premix (employing the library Chemkin). This solution is extended to a 2-D laminar plane flame. The turbulent flow field is obtained from a 2-D turbulent kinetic energy spectrum. Velocity and concentration field are then superimposed. Domain and grid: We consider a square computational domain of 1 cm × 1 cm, discretized with an equispaced grid in both directions, typically of 501 × 501 points. Boundary conditions: Periodic boundary conditions are used along the planar flame, while nonreflecting boundary conditions are used in the direction normal to the flame on inflow and outflow boundaries. The calculations are initialized with reactants on one side of the computational domain and products on the other; they are separated by a laminar premixed flame.

232

M. Lecanu et al.

Fig. 2. Generation of initial conditions. Top row : ignition of a circular planar ﬂame in a turbulent ﬂow ﬁeld. Bottom row : superposition of a turbulent ﬂow ﬁeld and of an initially plane ﬂame front

CPU costs: For a 2-D planar flame simulation consisting of 501 × 501 grid points with H2 Air mechanism, the calculation of 1000 time steps typically takes 187 minutes for the serial computation, while an 8 processor run takes 30 minutes for the same number of iterations. The eﬃciency of the parallel simulations is discussed in the next part.

2 Numerical Method The direct numerical simulation approach consists of solving the Navier-Stokes equations in their complete form, without any averaging or filtering accounted for by a model. The code employed for this task is the code P arcomb, developed by Th´evenin and co-workers [6]. Apart from the equations for continuity, momentum and energy, Ns equations are solved for the transport of chemical species, with Ns = 9 in the present cases. High-order discretization is used in order to reduce numerical dissipation. In space, this is a sixth order central finite-diﬀerence scheme along with a third order diﬀerencing at the boundaries. In time, a fourth-order Runge-Kutta scheme is employed. The Navier-Stokes characteristic boundary conditions [7, 8] are implemented taking into account detailed chemistry and thermodynamics. Isotropic turbulence is generated in Fourier space with a von K´ arm´an energy spectrum with Pao correction [9]. In Fourier space, the definition of the kinetic energy E(k) is given by: 5 ( kke )4 u′ k 4 3 3 E(k) = A exp − α ( ) ε [1 + ( kk )2 ] 17 2 kd 6 e

(3)

Computations of Premixed Turbulent Flames

233

where k is the wave number, u′ the rms value of velocity fluctuations, ε is the dissipation, while A and α are constants of the model (A = 1.5 and α = 1.5). Furthermore, ke = 1/Le , (4) where Le is the peak energy wavelength, and kd = 1/Ld ,

(5)

where Ld is the Kolmogorov wavelength. After the initialization, no forcing of the turbulent flow field is applied and the fluctuations are allowed to decay. The propagation of the flame is relatively fast, however, so that this decay is not too strong. Examples of calculations with P arcomb and postprocessing intended towards turbulent combustion modeling are available in [10, 11, 12].

3 Issues of HPC 3.1 Parallelization: PVM versus MPI Parcomb is a fully parallel program with dynamic load balancing capabilities. The message passing library originally used in Parcomb is PVM. All the validations performed by the authors was also using PVM. Thus, it was natural to continue using the program with PVM. PVM, however, is now superseeded by MPI and no more maintained on current installations. Hence, in a first phase of the project it was converted to MPI and fully tested and validated on the XC using HP-MPI. An attempt to use MPI instead of PVM in Parcomb had been made previously during the development, using wrappers around the PVM calls. However, this revision was not validated and could not be compiled with the present installation. Most of the parallelizaiton calls were therefore rewritten. Tests with MPICH and HP-MPI served to avoid lacks of portability. The communication of the number of processors in each direction to all processes may serve as an example. The sets of original calls PVMFINITSEND PVMFPACK PVMFMCAST coupled with the receiving instructions: PVMFRECV PVMFUNPACK were replaced by a single MPI_BCAST instruction. Since these blocking instructions are in the initialization part, they do not reduce the overall performance of the code. The code was then thoroughly tested against the serial version, both in 1-D and 2-D configurations and using diﬀerent numbers of processors.

234

M. Lecanu et al.

3.2 Eﬃciency of Runs Benchmarking of the code was performed with a 2-D reacting configuration. The test case used corresponds to the 2-D ignition of a turbulent premixed hydrogen-air flame. Timings are presented for a scaled problem, which means that all nodes possess the same number of grid points (201 × 201 points and a 7 mm×7 mm domain). When increasing the number of processors, the size of the domain and the total number of grid points are increasded proportionally. The simplest definition of the parallel eﬃciency is used here, i.e. E(N ) = tCPU (1)/tCPU (N ), where N is the number of processors. Since dual nodes were employed only even numbers where chosen. Eﬃciency results are given in Fig. 3(a) and corresponding CPU times are shown in Fig. 3(b). These data show that the parallelization performs very well, with generally no decrease in the performance when using more processors. Of course, these results should not be used as a direct measurement of the performance of the machine, but they can give some insights about the practical achievement of Parcomb on the XC.

(a) Efficiency

(b) CPU times per iteration

Fig. 3. Dependence of eﬃciency (left) and CPU time per time step (right) on the number of processors

4 Sample Results Results of two calculations are presented in this section. We considered a lean premixed H2 -Air 2-D turbulent flame (λ = 2). The grid has 500 × 500 points. Two diﬀerent initial flow fields have been used: Case 1: rms-value of velocity fluctuations u′ = 4.95 m/s, turbulent integral length scale Λ = 6 10−3 m, turbulent Reynolds number Re = 1853, eddy turnover

Computations of Premixed Turbulent Flames

235

time τt = Λ/u′ = 1.22 10−3 s, laminar flame thickness δL = 3.9 10−4 m, laminar flame speed SL = 0.67 m/s, τc = δL /SL = 5.82 10−2 s. Case 2 : rms-value of velocity fluctuations u′ = 1.68 m/s, turbulent integral length scale Λ = 9.7 10−4 m, turbulent Reynolds number Re = 101, eddy turnover time τt = Λ/u′ = 0.98 10−3 s, laminar flame thickness δL = 3.9 10−4 m, laminar flame speed SL = 0.67 m/s, τc = δL /SL = 5.82 10−2 s. Figure 4 illustrates the interaction between the flame front and two diﬀerent turbulence fields. The higher turbulence level in Case 1 produces curvature, strain, stretch and even quenching on the flame front, which can be observed in Fig. 4(a). In contrast, the turbulence in Case 2 is weaker and solely wrinkling can be observed in Fig. 4(b). Theoretical studies suggest a linear relation between the displacement flame speed Sd and stretch for small amounts of stretch, i.e. Sd = 1 − M a Ka , SL

(6)

where SL is the laminar flame speed and Ka = κδF /SL is the Karlovitz number or non-dimensional flame strech by turbulence, while M a is the Markstein number and δF is the nominal flame thickness. Flame stretch κ is defined as the sum of the tangential strain rate aT and the curvature term. κ = aT + Sd ∇ · n

(7)

where ∇ · n is the flame curvature and n is the flame normal vector. The Karlovitz number Ka can be written as the sum of the strain component and

(a) time=1.72 τT

(b) time=τT

Fig. 4. Heat release contours. Case 1 on the left: u′ /SL = 7.4, Λ/δl = 15.4 – Case 2 on the right: u′ /SL = 2.5, Λ/δl = 2.5

236

M. Lecanu et al.

0.35

0.35

Ka

S

KaS

C

KaC

Ka

0.3

Ka Probability Density Function

Probabilily Density Function

0.3

0.25

0.2

0.15

0.1

0.25

0.2

0.15

0.1

0.05

0.05

0 −15

Ka

−10

−5

Ka , Ka , Ka S

0

5

C

(a) Case 1

10

0 −15

−10

−5

KaS, KaC, Ka

0

5

10

(b) Case 2

Fig. 5. Probability density functions of normalized stretch rate Ka (solid line), tangential strain rate KaS (circles) and curvature term of stretch KaC (crosses)

the curvature component: Ka =

δF (aT + Sd ∇ · n) = KaS + KaC . SL

(8)

The following DNS results in Fig. 5 and Fig. 6 are used to evaluate the flame speed in terms of the displacement speed of the flame front. The isocontour YH2 = 0.175 has been used to represent the flame front as it corresponds to the maximum heat release contour. Statistics were then accumulated along this contour line. The shape of the total stretch rate Ka changes shape substantially with Reynolds number. For low Reynolds number (Case 2) its shape is inclined towards positive values, while for higher Reynolds number (Case 1) more negative values of Ka occur. The reason is provided by the decomposition: For low Re, the PDF of the curvature term is narrow and centered around a positive value. For higher Re it broadens without substantiall shift of mean. The strain-rate contribution on the other hand narrows from low to higher Re. Figure 6 displays cross-correlations along the flame front (again identified by the YH2 = 0.175 level line). Further simulations with diﬀerent realizations of the same physical setup need to be performed in order to perform statistical averaging, but the graphs reported here allow already a first interpretation. Figures 6(a) and 6(b) show two distinct stable branches depending on the sign of the displacement speed. This is in agreement with the observations of Chen and Im [13]. By comparing correlations of displacement flame speed versus tangential strain rate (Figs. 6(c) and 6(d)) and curvature term of stretch (Figs. 6(e) and 6(f)), it can be observed that the large negative values of

Computations of Premixed Turbulent Flames 8

237

6

6

4 Displacement speed

Displacement speed

4 2 0 −2 −4

2 0 −2

−6

−4

−8 −10 −70

−60

−50

−40

−30

−20

Stretch

−10

0

10

−6 −40

20

−30

(a) Speed versus stretch

−20

−10 Stretch

0

10

20

(b) Speed versus stretch

8

6

6 4 Displacement speed

Displacement speed

4 2 0 −2 −4

2 0 −2

−6 −4 −8 −10 −4

−2

0

2 4 Tangential strain

6

−6 −4

8

(c) Speed versus tangential strain

−3

−2

−1

0 1 2 Tangential strain

3

4

5

(d) Speed versus tangential strain

8

6

6 4 Displacement speed

Displacement speed

4 2 0 −2 −4

2 0 −2

−6 −4 −8 −10 −70

−60

−50

−40 −30 −20 −10 0 Displacement speed * curvature

10

(e) Speed versus curvature term

20

−6 −40

−30

−20 −10 0 Displacement speed * curvature

10

20

(f) Speed versus curvature term

Fig. 6. Correlations of normalized displacement ﬂame speed versus stretch rate, tangential strain rate and curvature term of stretch. Results are for Case 1 on the left, u′ /SL = 7.4 and for Case 2 on the right, u′ /SL = 2.5. Note the diﬀerent scaling of the x-axis in Fig. 6(c) and 6(d)

238

M. Lecanu et al.

stretch solely result from the curvature term KaC and not from KaS . This is in line with the discussion of Fig. 5.

5 Conclusions Within SFB 606, DNS calculations for turbulent flames in premixed gas have been performed with the parallel fortran code Parcomb. The present calculations allow a better understanding of phenomena involved in turbulent premixed combustion, such as diﬀerential diﬀusion, and will be used to evaluate the turbulent flame speed taking account streching, wrinkling and quenching of the flame front. In a first phase of the project, Parcomb has been converted to MPI. It also has been validated and tested using up to 64 processors showing very good parallel eﬃciency. Further 2-D simulations are under way to accumulate statistics and to elucidate Lewis eﬀects in turbulent premixed flames. Several 3-D configurations will be studied afterwards. Acknowledgements Funding of the present research by the German Research Foundation (DFG) through SFB 606 is greatfully acknowledged. The required CPU time was kindly provided by the SSCK Karlsruhe.

References 1. R.G. Abdel-Gayed and D. Bradley. Lewis number eﬀects on turbulent burning velocity. Proceedings of the Combustion Institute, 20:505–512, 1984. 2. H.-P. Schmid, P. Habisreuther, and W. Leuckel. A model fo calculating heat release in premixed turbulent ﬂames. Combustion and Flame, 113:79–91, 1998. 3. D. Bradley, P.-H. Gaskell, and X.-J. Gu. Burning velocities, Markstein lengths and ﬂame quenching for spherical methane-air ﬂames : a computational study. Combustion and Flame, 104:176–198, 1996. 4. T. Brutscher, N. Zarzalis, and H. Bockhorn. An experimentally based approach for the space-averaged burning velocity used for modeling premixed turbulent combustion. Proceedings of the Combustion Institute, 29:1825–1832, 2002. 5. D. Bradley, P.-H. Gaskell, X.-J. Gu, and A. Sedaghat. Premixed ﬂamelet modelling : factors inﬂuencing the turbulent heat release rate source and the turbulent burning velocity. Combustion and Flame, 143:227–245, 2005. 6. D. Th´evenin, F. Behrendt, U. Maas, B. Przywara, and J. Warnatz. Development of a parallel direct simulation code to investigate reactive ﬂows. Computers and Fluids, 25:485–496, 1996. 7. T. Poinsot and S. Lele. Boundary conditions for direct simulations of compressible viscous ﬂows. Journal of Computational Physics, 101:104–129, 1992.

Computations of Premixed Turbulent Flames

239

8. M. Baum, T. Poinsot, and D. Th´evenin. Accurate boundary conditions for multicomponent reactive ﬂows. Journal of Computational Physics, 116:247– 261, 1994. 9. J. O. Hinze. Turbulence. McGraw-Hill, 2nd edition, 1975. 10. J. De Charantenay, D. Th´evenin, and B. Zamuner. Comparison of direct numerical simulations of turbulent ﬂames using compressible or low-Mach number formulations. International Journal For Numerical Methods In Fluids, 39:497– 515, 2002. 11. R. Hilbert and D. Th´evenin. Inﬂuence of diﬀerential diﬀusion on maximum ﬂame temperature in turbulent nonpremixed hydrogen/air ﬂames. Combustion and Flame, 138:175–187, 2004. 12. D. Th´evenin, P.-H. Renard, J.C. Rolon, and S. Candel. Extinction processes during a non-premixed ﬂame/vortex interaction. Proceedings of the Combustion Institute, 27:719–726, 1998. 13. J.H. Chen and H.G. Im. Correlation of ﬂame speed with stretch in turbulent premixed methane/air ﬂames. Proceedings of the Combustion Institute, 27:819– 826, 1998.

Ignition of Droplets in a Laminar Convective Environment Rainer Stauch and Ulrich Maas Institut f¨ ur Technische Thermodynamik, Universit¨ at Karlsruhe, Kaiserstraße 12, D-76128 Karlsruhe, Germany [email protected] Summary. Numerical simulations of the ignition process of methanol and nheptane droplets in a laminar convective environment are performed using detailed reaction mechanisms and detailed transport models. The ignition delay time of a single droplet is found to decrease with increasing velocity of the convective gas ﬂow. This decrease is attributed to the steepening of the spatial gradients of the proﬁles of physical variables, like species mass fractions or temperature. This steepening is originated by a stronger gas ﬂow and leads to a speed-up of the physical transport processes. A downstream movement of the local ignition point with increasing ﬂow velocity is observed. For higher ﬂow velocities an ignition in the wake of the droplet followed by an upstream ﬂame propagation is found. After ignition an envelope ﬂame is formed. The structure of this envelope ﬂame is studied.

1 Introduction For the understanding of spray combustion a detailed knowledge of the processes associated with droplet ignition and combustion is required. Especially a detailed understanding of the basic physical and chemical processes, like vaporization, transport and chemical kinetics, and their interaction is of interest. The ignition or combustion process of single droplets in a quiescent atmosphere has been investigated in detail both experimentally and numerically for diﬀerent fuels, e.g. methanol [CCD90, CYD92, Sta92] or n-heptane [HK90, TTN93, NKKN94, TKS+ 95, TBE+ 96, JA96, SDM98, CD99, MDN99, SMC+ 00, MMKE05, CMBF+ 05, SLM06]. In all technical applications fuel droplets are exposed to a convective environment and to accelerating or decelerating forces. Therefore the influence of a convective gas flow and a relative acceleration of droplet and gas phase on the ignition and combustion process of fuel droplets has to be investigated in order to achieve a more reliable modeling of spray ignition and combustion [TS90, DASN96, HC97, WYHW97, SDM98, YYWW00, AMG+ 01]. Ignition delay times have been determined experimentally by Whang et al. [WYHW97]

242

R. Stauch, U. Maas

and Yang et al. [YYWW00] for n-heptane, n-hexadecane and i-octane and numerically by Aouina et al. [AMG+ 01] for methanol and LOX (liquid oxygen) droplets. The influence of gravitational acceleration on the ignition behavior of droplets is discussed in [FO68, TKS+ 95, VCC+ 96, HC97, SMC+ 00, CGS+ 00]. In this study the ignition process of single methanol and n-heptane droplets in a convective air environment is investigated. Detailed models for the phase transition, the transport processes and the chemical kinetics are applied.

2 Numerical model To investigate the interaction of the physical and chemical processes, detailed numerical simulations are an eﬃcient tool, because they allow to evaluate the contributions of chemical reactions, flow, and molecular transport. The presented model describes a spherical fuel droplet surrounded by an ambient gas phase. The droplet is exposed to a laminar convective gas flow. This system is analogous to a moving droplet in a gas environment with a velocity which is constant in time. The studied configuration is assumed to have rotational symmetry, and the resulting computational domain is shown in Fig. 1.

inflow boundary

outflow boundary

vz,∞ droplet surface symmetry boundary Fig. 1. Computational domain and boundaries

The governing set of equations is given by the compressible Navier-Stokes equations for reacting flows [HC49, BSL60, WMD01]. Assuming a cylindrical symmetry, the results are independent of the azimuthal coordinate φ (Note, that the axisof symmetry, i. e. the z-axis, is parallel to the direction of the axisymmetric flow.). Hence, the number of independent variables is reduced to three, namely the time t, the axial coordinate z, and the radial coordinate r. f = f (x, y, z, t) → f = f (r, z, t) / 01 2 01 2 / 3D

2D

(1)

Ignition of Droplets in a Laminar Convective Environment

243

The resulting equation system reads: ∂ρ 1 ∂ ∂ + (ρrvr ) + (ρvz ) = ∂t r ∂r ∂z 1 ∂ ∂ 1 ∂ ∂ ∂ρi + (ρi rvr ) + (ρi vz ) + (r(ji )r ) + ((ji )z ) = ∂t r ∂r ∂z r ∂r ∂z ∂ ∂p ∂(ρvr ) 1 ∂ + (ρrvr vr ) + (ρvr vz ) + ∂t r ∂r ∂z ∂r ∂ Πφφ 1 ∂ (rΠrr ) − + (Πrz ) = + r ∂r r ∂z ∂ ∂ ∂p ∂(ρvz ) + (ρrvr vz ) + (ρvz vz ) + ∂t ∂r ∂z ∂z ∂ 1 ∂ (rΠrz ) + (Πzz ) = + r ∂r ∂z ∂ 1 ∂ ∂ ∂ρu 1 ∂ + (ρurvr ) + (ρuvz ) + (r(jq )r ) + ((jq )z ) ∂t r ∂r ∂z r ∂r ∂z ∂vz 1 ∂ +p (rvr ) + r ∂r ∂z ∂vz 1 vr ∂vz ∂vr ∂vr + Πφφ + Πzz + Πrz + = +Πrr ∂r r r ∂z ∂z ∂r The components of the stress tensor Π read [BSL60]: 2 1 ∂ ∂vr ∂vz − (rvr ) + Πrr = −μ 2 ∂r 3 r ∂r ∂z 2 1 ∂ vr ∂vz Πφφ = −μ 2 − (rvr ) + r 3 r ∂r ∂z 2 1 ∂ ∂vz ∂vz − (rvr ) + Πzz = −μ 2 z 3 r ∂r ∂z ∂vz ∂vr Πrz = −μ + , ∂z ∂r

0

(2)

Mi ω˙i (3)

ρgr

(4)

ρgz

(5)

0.

(6)

(7) (8) (9) (10)

where ρ denotes the density, u the specific internal energy, ρi the partial densities, p the pressure, ji the diﬀusion flux density of species i, ωi the molar rate of formation of species i, Mi the molar mass of species i, and jq the heat flux density. vr and vz are the velocity components in radial and axial direction, respectively, and gr and gz the components of the gravitational acceleration. The system of the governing equations has to be completed by an equation of state for both the gas and the liquid phase. The gas phase is assumed to be ideal. In the liquid phase the Hankinson-Brobst-Thomson technique is used to model the temperature dependence of the liquid density [HT79, RPP89]. The chemical kinetics of methanol is modeled by a detailed reaction mechanism of Chevalier and Warnatz [WMD01], consisting of 23 chemical species

244

R. Stauch, U. Maas

and 166 elementary reactions, and the chemical kinetics of n-heptane is modeled by a detailed mechanism by Golovitchev [Gol04] with 56 species and 570 reactions. The transport processes are also modeled in detail. Fourier’s law is used to determine the conductive heat fluxes. For the determination of the diﬀusion flux density the approximation of Curtiss and Hirschfelder [HCB64] is used. Details of the implementation of the transport model for the gas phase can be found in [MW88]. Convection inside the droplet is neglected. The physical properties of the liquid phase and of the phase transition are determined by the following correlations taken from Reid et al. [RPP89]: The correlation of Latini and Baroncini is used to calculate the heat conductivities [LB83], the approximation of Rowlinson and Bondi is applied to determine the specific heat capacities of the liquid phase [Row69]. The vapor pressure is calculated following the Wagner equation [WEP76], and the correlation of Riedel and Watson is used to calculate the enthalpy of vaporization [Rie54, TS66, VK67]. At the inflow boundary (cf. Fig. 1) Dirichlet conditions are used ρ = ρ∞ , vr = 0, vz = vz,∞ , wi = wi,∞ , T = T∞ ,

(11)

and at the outflow boundary the conditions read: ∂vr ∂vz ∂wi ∂T ∂p = 0, = 0, = 0, = 0, =0. ∂z ∂z ∂z ∂z ∂z

(12)

For the axis of symmetry we obtain ∂wi ∂T ∂ρ ∂vz = 0, vr = 0, = 0, = 0, =0 ∂r ∂r ∂r ∂r

r=0:

(13)

A vaporization model accounts for the coupling of the liquid phase and the gas phase. The assumption of a local phase equilibrium is modeled by interface equations. g − jj − wj Ri + Rj jvap

φvap =

i

jvap

jvap

pj Mj ¯ pM

0 = ρ · vn − φvap −

i

(14)

−1

Ri

i

0 = φvap (wi − ǫi ) + ji + wi 0=

jvap

(15) j

Rj − Ri

l g + − jq,c ǫi · φvap · Δhvap,i + jq,c

(16) i

Ri (hgi − hli )

(17)

φvap denotes the vaporization rate, wi the species mass fraction jvap the ¯ the mean index of the vaporizing species, pj the partial pressure of species j, M molar mass, vn the velocity normal to the surface, ǫi = m ˙ i /m ˙ the fraction of vaporizing mass, ji the diﬀusion flux density of species i (gas phase), Δhvap,i

Ignition of Droplets in a Laminar Convective Environment

245

the enthalpy of vaporization of species i, jq,c the conductive heat flux density and hi the enthalpy of species i. Ri is the surface reaction rate of species i, which is 0 for the calculations presented here and has only been included in the model for generality (e.g. to allow modeling of combustion of solid particles with surface oxidation). More details can be found in [SLM06]. The governing equations are solved in a fully coupled way by the method of lines using finite diﬀerence techniques. The equations are transformed into boundary-fitted curvilinear coordinates generated by Tomcat [TTM77]. The flux vector splitting upwind scheme of Steger and Warming [SW81] is applied to calculate the inviscid fluxes. The viscous fluxes are discretized using second order central diﬀerences in chain rule conservation law form [RMW93b]. The time integration of the resulting diﬀerential-algebraic equation system is realized by the linearly implicit extrapolation method Limex [DHZ87]. Details on the numerical solution can be found in [RMW93b, RMW93a, AMG+ 01].

3 Results and Discussion Based on the model presented above the auto-ignition of single methanol and n-heptane droplets in a laminar convective environment has been simulated for varying conditions of ambient gas temperature and flow velocities. The surrounding gas phase consists of air in all cases. The ambient pressure is p = 7 bar. The ambient gas temperature varies between T = 1000 K and T = 1500 K. For the auto-ignition of a fuel droplet it is important to know how the ignition process is influenced by the gas flow. In contrast to the microgravity case the flame zone does not have a spherical shape anymore [HC97, WYHW97, AMG+ 01]. Furthermore, the flow has an important impact on the ignition process. This can be seen for a methanol droplet in Fig. 2. The flame position is defined as the location of maximal OH-concentration. The gas flows from the left to the right. With increasing flow velocity, the shape of the flame deviates more and more from a spherical shape. The extension of the flame in x-direction increases with increasing gas velocity. At the same time the extension in y-direction becomes smaller. Furthermore, it is remarkable that the maximal OH-concentration increases with increasing flow velocity. This can be attributed to faster mixing of the fuel with the oxidizer. The maximum OH-concentration is located at the stagnation point upstream of the droplet. On the other hand the maximum OH-concentration downstream of the droplet decreases with increasing flow velocity. As can be seen in Fig. 2 the droplet is surrounded by a non-premixed flame. In Fig. 3 the dependence of the ignition delay time for a methanol droplet on the inflow velocity in terms of the Reynolds number is shown, where the Reynolds number is defined as: Re = ρair vair · dD /μair with dD as the droplet diameter, and ρair , vair , and μair as the density, the velocity, and the viscosity of the ambient air, respectively. A significant decrease of the ignition delay

246

R. Stauch, U. Maas vGas 0.004

0.002

0.002 y/m

v = 0.1 m/s

y/m

0.004

0

-0.002

-0.004 -0.002

0

-0.002

0 0.002 0.004 x/m

0.002

0.002

0

-0.002

-0.004 -0.002

0 0.002 0.004 x/m

0.004

y/m

v = 1 m/s

-0.004 -0.002

y/m

0.004

v = 0.5 m/s

v = 2 m/s

0

-0.002

0 0.002 0.004 x/m

-0.004 -0.002

0 0.002 0.004 x/m

v = 5 m/s 0.004

0.002

0.002 y/m

v = 3 m/s

y/m

0.004

0

-0.002

-0.004 -0.002

OH 0.006 0.005 0.004 0.003 0.002 0.001

0

-0.002

0 0.002 0.004 x/m

-0.004 -0.002

0 0.002 0.004 x/m

Fig. 2. Change of the ﬂame structure with increasing gas velocity (methanol, Tg = 1400 K, p = 7 bar, rD (0) = 200 μm, t = 9 ms, ﬂow from the left to the right)

Ignition of Droplets in a Laminar Convective Environment

247

tign / ms

10.0

T = 1300K T = 1400K T = 1500K Fit (T = 1300K) Fit (T = 1400K) Fit (T = 1500K)

1.0 0.1

1

10

100

Re

Fig. 3. Dependence of the ignition delay time for a methanol droplet on the Reynolds number (p = 7 bar, rD (0) = 200 μm) 10 T_g = 1000K, r_D(0) = 200µm

tign (0) / tign (Re)

T_g = 1200K, r_D(0) = 100µm T_g = 1400K, r_D(0) = 200µm

1 0.01

0.1

1

10

100

Re

Fig. 4. Dependence of the inverse normalized ignition delay time of a heptane droplet on the Reynolds number (p = 7 bar)

time with increasing flow velocity can be observed. As one can see a variation of the flow velocity by an order of magnitude has a comparable eﬀect on the ignition delay time like a variation of the gas temperature by ±100 K. Figure 4 shows the dependence of the normalized reciprocal ignition delay times for a heptane droplet on the flow velocity for three diﬀerent gas temperatures. Again an increase of the inverse ignition delay time with increasing Reynolds number is observed. The normalized reciprocal ignition delay times for Tg = 1200 K show a smaller slope, because a radius of rD (0) = 100 μm has been chosen and the Damk¨ohler number is smaller for smaller droplets. In general the ignition delay times of heptane droplets depend less on the gas flow, which can be attributed to a smaller Damk¨ ohler number in this case.

248

R. Stauch, U. Maas

Even if the ignition starts downstream of the droplet, the flame zone propagates upstream and forms an envelope flame. This behavior is important in technical applications, because in most sprays ignition is initiated downstream of the droplets [WLY97]. Figure 5 shows this behavior for the ignition of a methanol droplet. The left figure shows the temperature field at the end of the ignition. The arrow indicates the subsequent propagation of the flame zone, which is shown in the two other figures. The right figure shows the flame when it is located close to the droplet. After that the flame propagates further upstream, and forms an envelope flame. The ignition location is governed by the flow velocity. The local point of ignition moves gradually from the upstream region of the droplet at lower velocities to the wake of the droplet for increasing flow velocities. This is illustrated in Fig. 6, in which the diﬀerent points of ignition depending on the gas flow velocity are shown for the example of a methanol droplet. In all investigated cases up to a flow velocity of 5 m/s after ignition the formation of an envelope flame can be observed. Heptane shows a similar behavior as can be seen in Fig. 7. For low velocities the ignition occurs upstream, too, and moves downstream with increasing gas velocity. Nevertheless, the quantitative behavior is diﬀerent due to the diﬀerent chemical time scales

0.002

y/m

0.001

vGas

0

-0.001

Droplet

0

x/m

y/m

0

Droplet

0

x/m

0.002

T(K) 2400 2000 1600 1200 800 400

0.001

-0.001

0.001

-0.001

0.002

0.002

0

T(K) 2400 2000 1600 1200 800 400

0.002

y/m

T(K) 2400 2000 1600 1200 800 400

Droplet

0

x/m

0.002

Fig. 5. Upstream propagation of the reaction zone after an ignition downstream of the droplet (methanol, Tg = 1400 K, p = 7 bar, rD (0) = 200 μm, v = 3 m/s, left: t1 = 2.80 ms, right: t2 = 2.92 ms, bottom: t3 = 3.04 ms)

Ignition of Droplets in a Laminar Convective Environment

249

vGas 0.004

y/m

0.003 0.002 0.5 m/s

0.001

0.2 m/s 0.1 m/s

0 -0.004

-0.002

1 m/s 2 m/s 3 m/s Droplet 0 x/m

5 m/s 4 m/s

0.002

0.004

Fig. 6. Dependence of the location of the ignition point on convective velocity of gas ﬂow (methanol, T = 1400 K, p = 7 bar, rD (0) = 200 μm)

vGas

0.003

y/m

0.002

0.001

0

-0.002

1 m/s 2.3 m/s 1.5 m/s 0.5 m/s 2.7 m/s 0.1 m/s 0.01 m/s Droplet 0 0.002 x /m

Fig. 7. Location of the ignition for diﬀerent gas velocities (n-heptane, Tg = 1200 K, p = 7 bar, rD (0) = 100 μm)

of the two fuels. These findings show resemblance with the observations of Whang et al. [WYHW97], who observed the formation of an envelope flame in all studied cases of n-heptane and n-hexadecane droplets.

4 Conclusions The auto-ignition process of methanol and n-heptane droplets in a convective environment is investigated numerically. The simulations are performed by the developed tool Flame2D by solving the governing equations of the gas and the liquid phase in a fully coupled way including a detailed vaporization model, a detailed transport model, and a detailed reaction mechanism. The influence of a gas flow on the ignition process is investigated. The ignition delay time decreases with an increasing velocity of the gas flow velocity for diﬀerent ambient gas temperatures. The ignition process as a whole is

250

R. Stauch, U. Maas

quite remarkably aﬀected by a modified gas flow velocity. With an increasing flow velocity the location of ignition is moving from upstream the droplet, around the droplet to the wake of the droplet. To investigate the upstream flame propagation and the characteristics of the envelope flame in more detail for conditions of practical relevance, simulations have to be performed with lower gas flow velocities at lower ambient temperatures where the chemical kinetics plays a major role for the time scales of the ignition and combustion process of a droplet. This involves a considerable increase of computing time. Therefore, the further improvement of the numerical solution of the governing equations with respect to a speed-up of the solution procedure is subject of future work. Acknowledgements The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number flame2D/12792. The authors acknowledge the support of Dr. Peter Lammers from the High Performance Computing Center Stuttgart (HLRS) in optimizing the code for simulations on the NEC SX-8.

References [AMG+ 01] Y. Aouina, U. Maas, E. Gutheil, U. Riedel, and J. Warnatz. Mathematical modeling of droplet heating, vaporization and ignition including detailed chemistry. Combustion Science and Technology, 173:1–23, 2001. [BSL60] R.B. Bird, W.E. Steward, and E.N. Lightfoot. Transport Phenomena. John Wiley & Sons, New York, Chichester, Brisbane, Toronto, Singapore, 1960. [CCD90] S.Y. Cho, M.Y. Choi, and F.L. Dryer. Extinction of a free methanol droplet in microgravity. Proceedings of the Combustion Institute, 23:1611–1617, 1990. [CD99] S.Y. Cho and F.L. Dryer. A numerical study of the unsteady burning behaviour of n-heptane droplets. Combustion Theory and Modelling, 3:267–280, 1999. [CGS+ 00] C. Chauveau, I. G¨ okalp, D. Segawa, T. Kadota, and H. Enomoto. Effects of reduced gravity on methanol droplet combustion at high pressures. Proceedings of the Combustion Institute, 28:1071–1077, 2000. [CMBF+ 05] A. Cuoci, M. Mehl, G. Buzzi-Ferraris, T. Faravelli, D. Manca, and E. Ranzi. Autoignition and burning rates of fuel droplets under microgravity. Combustion and Flame, (143):211–226, 2005. [CYD92] S.Y. Cho, R.A. Yetter, and F.L. Dryer. A computer model for one dimensional mass and energy transport in and around chemically reacting particles, including complex gas-phase chemistry, multi-component molecular diﬀusion, surface evaporation, and heterogeneous reaction. Journal of Computational Physics, 102:160–179, 1992.

Ignition of Droplets in a Laminar Convective Environment [DASN96]

[DHZ87]

[FO68] [Gol04] [HC49]

[HC97] [HCB64] [HK90]

[HT79]

[JA96]

[LB83]

[MDN99]

[MMKE05]

[MW88] [NKKN94]

[Rie54] [RMW93a]

[RMW93b]

251

H.A. Dwyer, I. Aharon, B.D. Shaw, and H. Niazmand. Surface tension inﬂuences on methanol droplet vaporization in the presence of water. Proceedings of the Combustion Institute, 26:1613–1619, 1996. P. Deuﬂhard, E. Hairer, and J. Zugck. One-step and extrapolation methods for diﬀerential-algebraic systems. Numerische Mathematik, 51:501–516, 1987. G.M. Faeth and D.R. Olson. The ignition of hydrocarbon fuel droplets in air. SAE Transactions, pages 1793–1802, 1968. V. Golovitchev. http://www.tfd.chalmers.se /˜valeri/MECH.html, 2004. J.O. Hirschfelder and C.F. Curtiss. Theory of propagation of ﬂames. part i: General equations. Proceedings of the Combustion Institute, 3:121–127, 1949. L.-W. Huang and C.-H. Chen. Droplet ignition in a high-temperature convective environment. Combustion and Flame, 109:145–162, 1997. J.O. Hirschfelder, C.F. Curtiss, and R.B. Bird. Molecular Theory of Gases and Liquids. John Wiley & Sons, New York, 1964. H. Hara and S. Kumagai. Experimental investigation of free droplet combustion under microgravity. Proceedings of the Combustion Institute, 23:1605–1610, 1990. R.W. Hankinson and G.H. Thomson. A new correlation for saturated densities of liquids and their mixtures. AIChE Journal, 25:653–663, 1979. G.S. Jackson and C.T. Avedisian. Modeling of spherically symmetric droplet ﬂames including complex chemistry: Eﬀect of water addition on n-heptane droplet combustion. Combustion Science and Technology, 115:125–149, 1996. G. Latini and C. Baroncini. Eﬀect of temperature and pressure on thermal conductivity of organic liquids. High Temperatures - High Pressures, 15:407–411, 1983. A.J. Marchese, F.L. Dryer, and V. Nayagam. Numerical modeling of isolated n-alkane droplet ﬂames: Initial comparisons with ground and space-based microgravity experiments. Combustion and Flame, 116:432–459, 1999. O. Moriue, M. Mikami, K. Kojima, and C. Eigenbrod. Numerical simulations of the ignition of n-heptane droplets in the transition diameter range from heterogeneous to homogeneous ignition. Proceedings of the Combustion Institute, 30:1973–1980, 2005. U. Maas and J. Warnatz. Ignition processes in hydrogen-oxygen mixtures. Combustion and Flame, 74:53–69, 1988. R. Nakanishi, H. Kobayashi, S. Kato, and T. Niioka. Ignition experiment of a fuel droplet in high-pressure high-temperature ambient. Proceedings of the Combustion Institute, 25:447–453, 1994. L. Riedel. Kritischer koeﬃzient, dichte des ges¨ attigten dampfes und verdampfungsw¨ arme. Chemie Ingenieur Technik, 26:679–683, 1954. U. Riedel, U. Maas, and J. Warnatz. Detailed numerical modeling of chemical and thermal nonequilibrium in hypersonic ﬂows. Impact of Computing in Science and Engineering, 5:20–52, 1993. U. Riedel, U. Maas, and J. Warnatz. Simulation of nonequilibrium hypersonic ﬂows. Computers and Fluids, 22:285–294, 1993.

252

R. Stauch, U. Maas

[Row69] [RPP89] [SDM98]

[SLM06] [SMC+ 00]

[Sta92]

[SW81]

[TBE+ 96]

[TKS+ 95]

[TS66] [TS90] [TTM77]

[TTN93] [VCC+ 96]

[VK67]

[WEP76]

J.S. Rowlinson. Liquids and Liquid Mixtures. Butterworths, 2 edition, 1969. R.C. Reid, J.M. Prausnitz, and B.E. Poling. The Properties of Gases and Liquids. McGraw-Hill, 4 edition, 1989. P. Stapf, H.A. Dwyer, and R.R. Maly. A group combustion model for treating reactive sprays in i.c. engines. Proceedings of the Combustion Institute, 27:1857–1864, 1998. R. Stauch, S. Lipp, and U. Maas. Detailed numerical simulations of the auto-ignition of single n-heptane droplets in air. Combustion and Flame, 145:533–542, 2006. S. Schnaubelt, O. Moriue, T. Coordes, C. Eigenbrod, and H.J. Rath. Detailed numerical simulations of the multistage self-ignition process of n-heptane isolated droplets and their veriﬁcation by comparison with microgravity experiments. Proceedings of the Combustion Institute, 28:953–960, 2000. P. Stapf. Modellierung der Tr¨ opfchenverbrennung unter Einschluß detaillierter chemischer Reaktion. PhD thesis, Universit¨ at Stuttgart, 1992. J.L. Steger and R.F. Warming. Flux vector splitting of the inviscid gasdynamic equations with application to ﬁnite-diﬀerence methods. Journal of Computational Physics, 40:263–293, 1981. M. Tanabe, T. Bolik, C. Eigenbrod, H.J. Rath, J. Sato, and M. Kono. Spontaneous ignition of liquid droplets from a view of non-homogeneous mixture formation and transient chemical reactions. Proceedings of the Combustion Institute, 26:1637–1643, 1996. M. Tanabe, M. Kono, J. Sato, J. Koenig, C. Eigenbrod, F. Dinkelacker, and H.J. Rath. Two stage ignition of n-heptane isolated droplets. Combustion Science and Technology, 108:103–119, 1995. R.E. Thek and L.I. Stiel. A new reduced vapor pressure equation. AIChE Journal, 12:599–603, 1966. J.S. Tsai and A.M. Sterling. The combustion of linear droplet arrays. Proceedings of the Combustion Institute, 23:1405–1411, 1990. J.F. Thompson, F.C. Thames, and C.W. Mastin. Tomcat – a code for numerical generation of boundary-ﬁtted curviliniear coordinate systems on ﬁelds containing any number of arbitrary two-dimensional bodies. Journal of Computational Physics, 24:274–302, 1977. M. Takei, T. Tsukamoto, and T. Niioka. Ignition of blended-fuel droplet in high-temperature atmosphere. Combustion and Flame, 93:149–156, 1993. B. Vieille, C. Chauveau, X. Chesneau, A. Odeide, and I. G¨okalp. Highpressure droplet burning experiments in microgravity. Proceedings of the Combustion Institute, 26:1259–1265, 1996. D.S. Viswanath and N.R. Kuloor. On a generalized watson’s relation for latent heat of vaporization. The Canadian Journal of Chemical Engineering, 45:29–31, 1967. W. Wagner, J. Ewers, and W. Pentermann. New vapour-pressure measurements and a new rational vapour-pressure equation for oxygen. The Journal of Chemical Thermodynamics, 8:1049–1060, 1976.

Ignition of Droplets in a Laminar Convective Environment [WLY97]

253

S.-C. Wong, X.-X. Liao, and J.-R. Yang. A simpliﬁed theory of the ignition of single droplets under forced convection. Combustion and Flame, 110:319–334, 1997. [WMD01] J. Warnatz, U. Maas, and R.W. Dibble. Combustion. Springer, Berlin, 3 edition, 2001. [WYHW97] J.-J. Whang, C.-Y. Yukao, J.-T. Ho, and S.-C. Wong. Experimental study of the ignition of single droplets under forced convection. Combustion and Flame, 110:366–376, 1997. [YYWW00] J.-R. Yang, C.-Y. Yukao, J.-J. Whang, and S.-C. Wong. Experiments on the ignition of monocomponent and bicomponent fuel droplets in convective ﬂow. Combustion and Flame, 123:266–274, 2000.

Laminar-Turbulent Transition in a Laminar Separation Bubble: Influence of Disturbance Amplitude on Bubble Size and Bursting Olaf Marxen Center for Turbulence Research, Stanford University, Stanford, CA 94305-3035, USA [email protected] Summary. Direct numerical simulations of a short laminar separation bubble and its bursting are carried out. The bubble is developing on a ﬂat plate due to an externally imposed pressure gradient. Laminar-turbulent transition is triggered by small disturbance input with ﬁxed frequency to keep the bubble short. The short bubble reaches a statistically steady state, while switching oﬀ disturbance input yields a growing separation bubble. This phenomenon is denoted as bubble-bursting process. Disturbance input does not only prevent bursting, but can also serve to control the short bubble. Bubble size decreases with increased disturbance amplitude. Performance data on a NEC SX-8 super computer are compared for two diﬀerent resolutions.

1 Introduction A laminar separation bubble (LSB) can originate if an initially laminar boundary layer is subject to a suﬃciently strong adverse pressure gradient. Usually, laminar-turbulent transition occurs in the detached shear-layer. Further downstream, the turbulent flow reattaches in the mean, forming a closed bubble. LSBs can occur on the surface of slender bodies, e.g. wing profiles of aerial vehicles or gas-turbine blades. They may strongly aﬀect their performance, such as lift and drag, as well as their noise production. Several eﬀects related to LSBs can be observed in the absence of curved walls, that is, for pressure-induced laminar separation on a flat plate. The present investigation is restricted to this latter case. In environments with low disturbance levels, the transition process in a LSB is typically governed by a strong amplification of high-frequency, essentially 2-d fluctuating disturbances. Discussions of several aspects related to LSBs can, for instance, be found in Boiko et al. (2002); Dovgal & Kozlov (1995); Rist (2002, 2003).

262

O. Marxen

1.1 Bursting of Laminar Separation Bubbles Owen & Klanfer (1953) were the first to distinguish between short and long LSBs. Their classification is based on the bubble length in comparison to the chord of an airfoil. Tani (1964) introduced a classification of long and short LSBs depending on whether their influence on the pressure distribution is local or global. Under certain conditions, for slight changes in flow conditions, short bubbles can break-up into long ones, so-called bubble bursting. For bodies of finite dimension like airfoils, the flow might not reattach at all. On a semiinfinite flat plate, the flow will finally reattach in any case. Laminar-turbulent transition plays a key role for reattachment of a short LSB. This indicates an interconnection of laminar-turbulent transition and bubble bursting. Gaster (1966) carried out an investigation to settle characteristics of short and long bubbles. He tried to establish parameters that govern the bursting process, i.e. develop a bursting criterion based on a pressure-gradient parameter and a local Reynolds number. Pauley et al. (1990) suggested that bubble bursting coincides with the onset of vortex shedding in an otherwise fully steady separation bubble. Diwan et al. (2006) introduced the height of the bubble as an additional parameter. However, none of the criteria have found general acceptance so far (Diwan et al., 2006). For a detailed discussion of deficiencies of some bursting criteria see Marxen & Henningson (2006). Recently, Marxen & Henningson (2007c) considered bursting as a dynamical process. Moreover, they suggested to distinguish between two diﬀerent forms of bubble bursting, depending on whether the long-bubble flow at reattachment is laminar as in Pauley et al. (1990) or turbulent as in Gaster (1966). In this article, only the latter case will be considered. Even though laminar-turbulent transition in LSBs has been the subject of numerous studies in the past, both experimentally (Lang et al., 2004; Watmuﬀ, 1999) and numerically (Alam & Sandham, 2000; Spalart & Strelets, 2000; H¨aggmark et al., 2001; Marxen et al., 2003), flow dynamics of bubble bursting is not yet well understood. A commonly accepted physical explanation for bubble bursting is still lacking. 1.2 Outline This article reports results obtained within the HLRS project long lsb. Additional details can be found in Marxen & Henningson (2006, 2007a,b,c). Result reported here include an evaluation of the role of laminar-turbulent transition for bubble bursting (Sec. 3) and control of separation-bubble size (Sec. 4). Finally, some aspects of computational performance are treated (Sec. 5).

2 Physical Model and Numerical Method The basic configuration in this article is given by a flat-plate boundary layer at a suﬃciently high Reynolds number subjected to a streamwise pressure

Laminar-Turbulent Transition in a Laminar Separation Bubble

263

gradient. The set-up is similar to the one used in Gaster (1966) or Lang et al. (2004). A pressure gradient is induced via the streamwise velocity at the upper boundary. The resulting pressure distribution changes downstream from strongly favourable to strongly adverse, which is not uncommon in applications. In a laboratory experiment, such a pressure gradient could be caused by putting a displacement body into the flow at some distance from the plate as e.g. in Lang et al. (2004). 2.1 Numerical Method Direct numerical simulations (DNS) are carried out based on the threedimensional unsteady Navier-Stokes equations in vorticity-velocity formulation for an incompressible fluid. The computational code n3d serves to compute the flow field with a separation bubble. This code n3d is a solver for highly accurate computation of transitional and turbulent flows, developed at the Institut f¨ ur Aerodynamik und Gasdynamik (IAG), Universit¨ at Stuttgart. It was provided by the project partner of this HLRS project, U. Rist, Institut f¨ ur Aerodynamik und Gasdynamik (IAG), Universit¨ at Stuttgart. The method is based on a discretisation suggested and carefully analysed by Kloker (1998), and thereafter slightly refined, extended, and optimised by many others (see also Marxen & Rist, 2005). Further details of n3d, i.e. numerical method and implementation, can be found in Meyer et al. (2003). The code n3d applies a hybrid parallelisation approach employing two fundamentally diﬀerent techniques at the same time: a directive-based shared-memory intra-nodal parallelisation (similar to OpenMP) is combined with an explicitly coded distributed-memory MPI inter-nodal parallelisation. A Cartesian coordinate system with streamwise x, wall-normal y and spanwise coordinate z is located at the wall. The method uses finite diﬀerences of fourth/sixth-order accuracy for downstream (NMAX) and wall-normal (MMAX) discretisation. Grid stretching (only) in wall-normal direction allows to cluster grid points m near the wall according to the following formula (1≤m≤MMAX), with κ = 0.15:

3 m−1 m−1 ym = ymax (1 − κ) · . (1) +κ· MMAX − 1 MMAX − 1 In spanwise direction, a spectral Fourier ansatz is applied (KMAX + 1 modes) for complex modes (IMAX = 2). An explicit fourth-order Runge-Kutta scheme is used for advancing a vorticity-transport equation in time with LPER time steps per fundamental period T0 . The solution of a Poisson equation for the wall-normal velocity is obtained by a direct method based on a Fourier expansion. A grid study confirmed the resolution “lowres” given in Table 1 to be suﬃcient (Marxen & Henningson, 2007a).

264

O. Marxen Table 1. Overview on two diﬀerent resolutions used for DNS Case MMAX NMAX KMAX + 1 IMAX LPER lowres 769 3501 72 2 900 highres 1537 6001 136 2 1800

2.2 Non-Dimensionalisation and Boundary Conditions All quantities are non-dimensionalised using a reference length Lref = 1.16 · ∗ ∗ δif l , derived from the displacement thickness δif l at the inflow, and the free-stream velocity U∞ at the inflow. The global Reynolds number Re = U∞ Lref /ν amounts to 600, corresponding roughly to the local Reynolds number based on the displacement thickness at x = 0. At the inflow boundary xif l = −320 a Blasius similarity solution is prescribed with a Reynolds num∗ ber based on the displacement thickness Reδif = 515.02. l Upstream of the outflow boundary, which is located at xof l = 441.905 in the low-resolution case, a damping zone smoothly returns the flow to a steady laminar state. The useful region of the integration domain extends up to x≈ 380. At y = 0 a no-slip wall is placed. Flow through this wall does only occur within a disturbance strip (see below). Wall-normal height of the integration domain ymax is chosen to be 160. At the upper boundary, vorticity components are set to zero and the streamwise velocity u is set according to the following formula: u(ymax ) = 1 + a1 · exp(−b1 x2 ) + a2 x2 · exp(−b2 x2 ) ,

(2)

with the choice a1 = 10.0, a2 = −41.0, b1 = 200.0, and b2 = 12.0. In the homogeneous direction z the flow is assumed periodic with a spanwise extent λz = 52.360. Simulations are performed with deterministic disturbance input. A single pair of oblique time-harmonic perturbations is triggered via blowing and suction at the wall through a zero-net-mass-flux (ZNMF) disturbance strip. The streamwise location of the strip is x∈[xs = 1.5238, xe = 15.4558]. Forcing is prescribed for each wave of the pair separately within the strip, with a velocity amplitude perpendicular to the surface distributed as vˆwall = Av /U∞ (˜ x7 −3˜ x5 +3˜ x3 −˜ x)/0.238,

x ˜ = 2(x−

xe + xs )/(xe −xs ) . (3) 2

Amplitude Av will be specified below. To break spanwise symmetry, the forcing amplitude of one of the two oblique waves is increased by 10−8 . Forced fundamental circular frequency is β0 = 2π/T0 =0.3 and fundamental spanwise wavenumber is γ0 =0.12. The disturbance strip is placed downstream of the cp minimum, since the accelerated boundary layer is very stable and the forced perturbation would otherwise decay quickly. Circular frequency β0 = 0.3 is among the most amplified frequencies according to linear stability theory for the present flow (see Fig. 5 in Marxen & Henningson, 2006).

Laminar-Turbulent Transition in a Laminar Separation Bubble

265

An initial condition is composed of a blending of a slip-flow and a Blasius boundary-layer solution (Marxen & Henningson, 2007a). The computation is advanced until a statistically steady (quasi-periodic) state has been reached. Simulation results are Fourier analysed in time using available discrete time steps (equally spaced with spacing ΔL = LPER/25). Together with the spanwise Fourier transform of the numerical scheme, this double Fourier transform in time and spanwise direction yields disturbance amplitudes and phases. The notation (h, k) is used to specify modes, with h and k denoting wavenumber coeﬃcients in time and span, respectively. Data collected during four periods of the fundamental frequency were used in the analysis. Forcing frequency β0 is the respective fundamental frequency (corresponding to h = 1) for such an analysis.

3 Reference Configuration: Short Separation Bubble Certain essential features are commonly observed in short LSBs, no matter if transition has been forced (Alam & Sandham, 2000) or not (Spalart & Strelets, 2000). These features are typically: • • • •

limited/local eﬀect of the LSB on the potential flow and in particular on the pressure distribution at the wall; stagnant flow in the first, laminar part of the bubble causing a plateau in the pressure distribution at the wall; a similar size of laminar and turbulent portions of the LSB; good agreement of disturbance amplification with linear stability theory, typically with an instability of convective Kelvin-Helmholtz type.

The aim of this section is to demonstrate that for the present configuration, with disturbance input, a statistically-steady short LSB develops which exhibits these essential features. This short LSB shall serve as a reference for investigations of the role of laminar-turbulent transition. It is described in section 3.2. However, to visualise the set-up we consider a two-dimensional slip flow for the same basic configuration first (section 3.1). More details on the reference LSB can be found in Marxen & Henningson (2007a,c). 3.1 Slip Flow Replacing the lower boundary by a slip wall, the resulting slip-flow field uslip , vslip resembles a potential flow created by a cylinder (i.e. a dipole) above a wall (Marxen & Henningson, 2007a). A strong acceleration of the flow along this wall is followed by a strong deceleration. The stream function gives a good visualisation of the configuration (Fig. 1). The slip flow does not only serve to generate an initial condition for DNS or to visualise the configuration. It will also be used for comparison with DNS with respect to the corresponding wall-pressure coeﬃcient.

O. Marxen

y [-]

266

100

0 -300

-200

-100

0

x [-]100

200

300

400

500

Fig. 1. Contours of the stream function for slip ﬂow

3.2 DNS Results for Reference Separation Bubble For the reference DNS, disturbance amplitude was chosen to be Av = 0.5·10−4 for each oblique wave respectively. Mean Flow and Boundary-Layer Parameters Wall-pressure coeﬃcient cp is seen to deviate from the corresponding slip flow only in the vicinity of the LSB (Fig. 2a). This is in agreement with the expectation that a short LSB has only a local and limited eﬀect on the potential flow. Furthermore, the formation of a pressure plateau is clearly seen in the region 50 < x < 80. Skin-friction coeﬃcient cf (Fig. 2a) reveals that the separation location lies at xS ≈ 34, while mean reattachment occurs at xR ≈ 89. Resulting time-averaged streamlines are shown in Fig. 2(b). Very close to the wall, the mean dividing streamline ψ¯ = 0 visualises the short LSB. Displacement of streamlines away from the wall caused by the boundary layer can be seen when comparing DNS to the slip-flow solution. However, apart from an almost constant shift of the streamlines away from the wall due to the boundary layer, again very little impact of the separation bubble on the potential flow is seen.

1.2

cp

(a)

(b) 50

0.8

y [-]

cf -cp, cf × 10 2

0.4

25 0

-0.4 -200

Av=0.5⋅10 slip flow

-100

-4

LSB

0

100

X [-] 200

0 0

25

50 x [-]

75

100

Fig. 2. Slip-ﬂow results (blue, thin/dashed ) versus time-averaged DNS results with disturbance input 2Av = 10−4 (black ). (a) Coeﬃcients for surface pressure cp and skin-friction cf . (b) Contours of time-averaged stream function Ψ¯

Laminar-Turbulent Transition in a Laminar Separation Bubble

267

Laminar-Turbulent Transition Triggering the flow is not suﬃcient to suppress separation altogether. The transition location as judged from first appearance of small-scale structures (Fig. 3) is roughly situated at x ≈ 70. Laminar portion (34 < x ≤ 70) and turbulent portion (70 < x ≤ 89) of the separation bubble are quite similar in their streamwise extent. Strong growth of the triggered oblique waves (1, ±1) inside the LSB (Fig. 4a) is a result of a linear, essentially inviscid instability of the separated shear-layer (Kelvin-Helmholtz instability). Growth of this disturbance favourably compares with linear stability theory (LST). LST results correspond to streamwise integrated growth rates obtained from a local solution of the Orr-Sommerfeld equation (Schlichting, 1979). The frequency of the triggered perturbation can be verified a posteriori to correspond roughly to the integrally most amplified disturbance for the given spanwise wave length (Marxen & Henningson, 2006). Finally, disturbances saturate and fill up the

y [-]

10

030

40

60 x [-]

50

70

80

Fig. 3. Contours of instantaneous spanwise vorticity for spanwise harmonic k = 0 at t/T0 = 200 together with mean dividing stream line Ψ¯ = 0, DNS with 2Av = 10−4

10-1

(a)

(1,+/-1)

Au [-]

10-2

10

-3

10

-4

0

(2,0)

S

25

50

75 X [-]

100

Fig. 4. (a) Ampliﬁcation curves for the maximum (in y) streamwise velocity ﬂuctuation. Results from DNS with 2Av = 10−4 (lines) and from LST (symbols). (b) Contours of spanwise vorticity ωz for spanwise harmonic k = 0 at the wall as a function of time, DNS with 2Av = 10−4 . The black line marks the contour of vanishing vorticity (ωz = 0), while blue contours correspond to negative vorticity

268

O. Marxen

spectrum in time and span, i.e. cause transition to turbulence at x ≈ 70 as observed already from the appearance of small-scale structures. Statistically Steady State Convergence in time to a statistically steady state can be qualitatively estimated from Fig. 4(b). Soon after start-up, the separation line remains at a constant x-location. Moreover, reattachment takes place within the same streamwise region (marked as reattachment zone). This impression is confirmed in Marxen & Henningson (2007a). There it has been shown that results from two diﬀerent time intervals possess same mean quantities as well as disturbance evolution up to transition. We conclude that in case with disturbance forcing, a short LSB with a statistically-steady state has been reached. 3.3 Bursting of Reference LSB Bursting shall be considered as a dynamical process. Gradually switching oﬀ disturbance input between t = 45T0 and t = 47T0 initiates a continuously growing separation bubble (Fig. 5b). Eventually, the reattachment point has moved considerably further downstream and influence on the pressure distribution is much stronger now (Fig. 5a). This continuous deviation from the initially statistically-steady state shall be denoted as the bubble-bursting process. It has not yet stopped at the end of the bursting simulation t = 85T0 .

1.2

cp

(a) 0.8

cf -cp, cf × 10 2

0.4 0

-0.4 -200

during bursting slip flow

-100

0

100

X [-] 200

Fig. 5. (a) Coeﬃcients for surface pressure cp and skin-friction cf . Slip-ﬂow results (blue) versus time-averaged DNS results during bursting (black ) after disturbance input has been switched oﬀ around t = 46T0 , time-averaged over the interval [81T0 , 85T0 ]. (b) Contours of spanwise vorticity ωz for spanwise harmonic k = 0 at the wall as a function of time during bursting. The black line marks the contour of vanishing vorticity (ωz = 0), while blue contours correspond to negative vorticity. The white line gives results with continued disturbance input 2Av = 10−4 for reference

Laminar-Turbulent Transition in a Laminar Separation Bubble

269

y [-]

10

030

40

50

60 x [-]

70

80

Fig. 6. Contours of instantaneous spanwise vorticity for spanwise harmonic k = 0 at t/T0 = 80 together with mean dividing stream line Ψ¯ = 0 during bursting

The bubble is expected to develop towards a long-bubble state similar to those observed in Gaster (1966). Indication is that the reattachment line moves downstream much quicker than the separation line moves upstream. Thereby, transition roughly remains at the same position. This results in a shift in relation between laminar and turbulent portion. At the end of the present simulation (t/T0 = 85), the turbulent portion has already become roughly twice as long as the laminar portion of the LSB. Clearly, disturbance input is essential to maintain a short bubble in the present case. For that reason, the disturbance input can be considered a means of flow control. On the other hand, its amplitude is so small that it can as well just be regarded as a form of background noise instead. 3.4 Role of Vortical Structures for Reattachment Vortical structures shall be considered to shed some light on physical mechanisms leading to reattachment of this separated flow. Vortices will be identified using the λ2 -criterion by Jeong & Hussain (1995). Instantaneous negative λ2 isosurfaces for the short LSB and the LSB after bursting are given in Fig. 7. It can be seen that in the short bubble case (Fig. 7a), large, essentially 2-d spanwise vortices appear at transition. Since they are a result of oblique-wave disturbance forcing, they possess a checkerboard arrangement. Even though they quickly breakdown into a 3-d irregular pattern of small-scale vortices within the interval x = 70 − 80, they are at the same time quickly followed by mean reattachment at x = 89. This observation suggests their strong role for the reattachment process. In the bursting case, the vortex pattern is completely diﬀerent (Fig. 7b). The first vortices to appear are small but long streamwise vortices. Again, breakdown to 3-d small-scale irregular vortices occurs quickly. However, this time reattachment does not occur and the turbulent flow remains separated in the mean, even though occasional forward flow occurs close to the wall. These observations indicate that it is not only the increased mixing associated with turbulence that causes reattachment. Instead, occurrence of large-scale vortical structure appears essential for the reattachment process.

270

O. Marxen

Fig. 7. Isosurfaces of λ2 = −0.001 (grey) together with time-averaged dividing streamsurface (blue). (a) Instantaneous data at t/T0 = 180.0 from DNS with 2Av = 10−4 . (b) Instantaneous data at t/T0 = 80.0 from DNS during bursting after disturbance input has been switched oﬀ around t = 46T0

4 Influence of Disturbance Amplitude on Bubble Size For the purpose of flow control, Rist & Augustin (2006) applied unsteady, mono-harmonic ZNMF wall blowing/suction in a similar fashion as disturbances were triggered here. In their conclusions, they posed the question whether their flow-control “method is suﬃciently strong to prevent an “open separation””. The present separation bubble after bursting is considered such a case of “open separation”, even though due to the infinitely long flat plate the flow will eventually reattach in any case. To answer the question, Marxen & Henningson (2007b) applied the methodology of Rist & Augustin (2006) to the present set-up.

Laminar-Turbulent Transition in a Laminar Separation Bubble 1.2

271

-4

Av=0.5⋅10 -3 Av=0.5⋅10 slip flow

cp 0.8

10-1

(1,+/-1)

-cp, cf × 10 2

Au [-]

10-2

0.4

cf

(2,0)

10

-3

10

-4

S

0

-0.4 0

50

100

X [-] 150

0

25

50

75 X [-]

100

Fig. 8. (a) Coeﬃcients for surface pressure cp and skin-friction cf . Comparison between DNS results with 2Av = 10−3 (solid ) and reference DNS with 2Av = 10−4 (dashed ). (b) Ampliﬁcation curves for the maximum (in y) streamwise velocity ﬂuctuation. Results from DNS with 2Av = 10−3 (lines) and LST (symbols)

By increasing disturbance amplitude by an order of magnitude compared to the reference DNS, the size of the (short) bubble is seen to decrease (Fig. 8a). Therefore, the approach of Rist & Augustin (2006) can be applied in this case where the bubble would undergo bursting, and thus become “open”, if uncontrolled. Note that both, cp and cf , simultaneously react to flow control. Fig. 8(b) confirms that, as before, a linearly unstable mode (KelvinHelmholtz-type instability) was triggered. This was also the case in the simulations of Rist & Augustin (2006). Hence, it can be concluded that the same control mechanism was active: enhancing laminar-turbulent transition via generation of large-scale structures reduces the size of the bubble.

5 Computational Aspects Marxen & Henningson (2007a) reported that the lower of the two resolutions given in Table 1 gives suﬃciently accurate results already. But even the lower resolution with a total of 388 · 106 grid points is quite high, and a super computer is truly essential for the present study. Since a SX-specific optimised version of the code n3d was applied, all runs were done on a NEC SX-8 vector super computer at HLRS Stuttgart. For both resolutions, memory consumption and performance on this machine using combined SMP and MPI parallelisation are given in Table 2. At first glance, it does not seem surprising that with higher resolution a better performance with respect to GFLOPS rates can be achieved. In fact, they are 29% larger in the high-resolution case. A possible explanation is that the work load estimated from spatial resolution (only) is 6.47 times higher in

272

O. Marxen Table 2. Performance of n3d for a typical run on NEC SX-8 using MPI Case lowres highres

CPU×Nodes GFLOPS/CPU 8×9 8 × 17

3.4 4.4

Memory

CPU Time/(Period T0 )

105.69 GB 618.82 GB

59.11 h 712.47 h

the high resolution case, while the number of processors has only increased by 1.89. On the other hand, if we consider CPU Time/(Period T0 ), this number for the low resolution case is 59.11 h. Multiplying this by 12.94, which corresponds to the increase in resolution including the increased temporal resolution (product of all numbers in a row of Table 1 divided by each other), we obtain as an estimate for CPU Time/(Period T0 ) for the high resolution, based on lowresolution performance, a value of 765.03 h. This is opposed to the resulting 712.47 h (Table 2) actually required in the high resolution case. Comparing these two numbers suggests that the run with the higher resolution was only 1.0738, i.e. 7% more eﬃcient – despite the much higher GFLOPS rates. The reason for this discrepancy between GFLOPS rates and actual CPU Time/(Period T0 ) was found to lie in FFT routines. A significant part of the computational time is spent in transforming quantities between Fourier and physical space. Such transforms are required for the evaluation of nonlinear terms as well as for the solution of a Poisson equation for the wallnormal velocity. FFT library routines provided on the machine carry out these transformations. FFT routines are very eﬃcient in terms of GFLOPS rates. However, depending on the number of points, the number of operations within these FFTs can diﬀer considerably. The lower resolution seems better suited for the FFT: it turned out to require less operations within the respective library routines. Hence, even though a lower GFLOPS rate is achieved, the lower number of operations makes up for this in terms of total computational time. In the present case, the number of streamwise grid points must be dividable by the number of nodes. At the same time, this dimension less one was chosen such that it is factorisable with prime numbers not larger than seven. In earlier tests, this was found to result in higher eﬃciency of FFTs for the solution of the Poisson equation. Even though the higher resolved case is factorisable with more and smaller prime numbers, it turned out to be less eﬃcient.

6 Summary and Conclusions Time-dependent three-dimensional DNS of a flat-plate boundary layer subject to a strong favourable-to-adverse pressure gradient has been carried out. With small disturbance input, laminar-turbulent transition is triggered and large spanwise vortices appear at transition. The resulting LSB exhibits typical

Laminar-Turbulent Transition in a Laminar Separation Bubble

273

features of a short LSB. In particular, only a local deviation in cp from slipflow is seen as well as a roughly equal portion of laminar and turbulent flow inside the LSB. Furthermore, disturbance amplification was shown to be in favourable agreement with linear stability theory. In addition, the short LSB was converged to a periodic and thus statistically steady state. If disturbance forcing is switched oﬀ, the transition process is no longer able to immediately reattach the flow. The transition position remained roughly constant while the turbulent part of the bubble grows faster than the laminar part. This is taken as indication that the bubble develops towards a long-bubble state. Hence, disturbance input, though quite small, is essential to maintain a short bubble in the present case. To explain the diﬀerent response of the flow to laminar-turbulent transition, vortical structures were analysed around the transition location. In the forced case, these vortical structures are large spanwise vortices, while in the unforced bursting case they are small (but long) streamwise vortices. This difference suggests that not the increased turbulent mixing as such is responsible for quick reattachment, but rather the observed large-scale spanwise vortices. An approach suggested by Rist & Augustin (2006) to control separationbubble size was successfully applied in the present case. Increasing the forcing amplitude reduces the size of the short bubble. Overall performance of the simulation code on a NEC SX-8 vector super computer was found to be good for two diﬀerent resolutions. Even though a fairly large diﬀerence in GLFOPS rates between both resolutions was seen, the required computational time to advance the flow field a certain physical time was considerably less diﬀerent. The reason for this was found in a diﬀerent number of operations spent within library FFT routines. Two conclusions can be drawn from this. First, the GFLOPS rate is not necessarily a good indicator for the overall performance of a run, if library routines with a non-fixed number of operations are involved. Second, choice of the number of grid points to achieve best performance is a non-trivial task, since it can have a strong, not exactly predictable influence on performance. Acknowledgements An important part of the work presented here was carried out together with Dan Henningson, KTH Stockholm, Sweden. OM thanks Ulrich Rist and Markus Kloker, IAG, Uni Stuttgart for providing the DNS code n3d. Furthermore, OM acknowledges gratefully financial support by the Deutsche Forschungsgemeinschaft DFG under grant Ma 3916/1–1. The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number long lsb/xme12780.

274

O. Marxen

References Alam, M. & Sandham, N. D. 2000 Direct Numerical Simulation of ‘Short’ Laminar Separation Bubbles with Turbulent Reattachment. J. Fluid Mech. 410, 1–28. Boiko, A.V., Grek, G.R., Dovgal, A.V. & Kozlov, V.V. 2002 The Origin of Turbulence in Near-Wall Flows, 1st edn. Springer, Berlin, Heidelberg, New York. Diwan, S.S., Chetan, S.J. & Ramesh, O.N. 2006 On the bursting criterion for laminar separation bubbles. In Sixth IUTAM Symposium on Laminar-Turbulent Transition (ed. R. Govindarajan), Fluid Mechanics and Its Applications, vol. 78, pp. 401–407. Springer, Berlin, New York. Dovgal, A.V. & Kozlov, V.V. 1995 On nonlinearity of transitional boundary-layer ﬂows. Phil. Trans. R. Soc. Lond. A 352, 473–482. Gaster, M. 1966 The structure and behaviour of laminar separation bubbles. AGARD CP–4, pp. 813–854. H¨ aggmark, C.P., Hildings, C. & Henningson, D.S. 2001 A numerical and experimental study of a transitional separation bubble. Aerosp. Sci. Technol. 5 (5), 317–328. Jeong, J. & Hussain, F. 1995 On the identiﬁcation of a vortex. J. Fluid Mech. 285, 69–94. Kloker, M. 1998 A Robust High-Resolution Split-Type Compact FD Scheme for Spatial Direct Numerical Simulation of Boundary-Layer Transition. Appl. Sci. Res. 59, 353–377. Lang, M., Rist, U. & Wagner, S. 2004 Investigations on controlled transition development in a laminar separation bubble by means of LDA and PIV. Experiments in Fluids 36, 43–52. Marxen, O. & Henningson, D. 2006 Numerical Simulation of the Bursting of a Laminar Separation Bubble. In High Performance Computing in Science and Engineering ′ 06 (ed. W.E. Nagel, W. J¨ ager & M. Resch), pp. 253–267. Transactions of the High Performance Computing Center Stuttgart (HLRS), Springer. Marxen, O. & Henningson, D.S. 2007a Direct Numerical Simulation of a Short Laminar Separation Bubble and Early Stages of the Bursting Process. In Contributions to the 15th STAB/DGLR Symposium, Nov. 29–Dec. 1, 2006, Darmstadt, Germany. Accepted for publication. Marxen, O. & Henningson, D.S. 2007b Direct Numerical Simulation of the Bursting of a Laminar Separation Bubble and Evaluation of Flow-Control Strategies. In Proceedings of IUTAM Symposium on “Unsteady Separated Flows and Their Control”, June 18-22, 2007, Kerkyra (Corfu), Greece. Marxen, O. & Henningson, D.S. 2007c Numerical Simulation of the Bursting of a Laminar Separation Bubble. AIAA Paper 2007-0538. Marxen, O., Lang, M., Rist, U. & Wagner, S. 2003 A Combined Experimental/Numerical Study of Unsteady Phenomena in a Laminar Separation Bubble. Flow, Turbulence and Combustion 71, 133–146. Marxen, O. & Rist, U. 2005 Direct Numerical Simulation of Non-Linear Transitional Stages in an Experimentally Investigated Laminar Separation Bubble. In High Performance Computing in Science and Engineering ′ 05 (ed. W.E. Nagel, W. J¨ ager & M. Resch), pp. 103–117. Transactions of the HLRS 2005, Springer. Meyer, D., Rist, U. & Kloker, M. 2003 Investigation of the ﬂow randomization process in a transitional boundary layer. In High Performance Computing in Science

Laminar-Turbulent Transition in a Laminar Separation Bubble

275

ager), pp. 239–253. Transactions of and Engineering ′ 03 (ed. E. Krause & W. J¨ the HLRS 2003, Springer. Owen, P.R. & Klanfer, L. 1953 On the laminar boundary layer separation from the leading edge of a thin airfoil. Tech. Rep. No. Aero 2508. Royal Aircraft Establishment, UK, In: A.R.C. Technical Report C.P. No. 220, 1955. Pauley, L.L., Moin, P. & Reynolds, W.C. 1990 The structure of two-dimensional separation. J. Fluid Mech. 220, 397–411. Rist, U. 2002 On instabilities and transition in laminar separation bubbles. In Proc. CEAS Aerospace Aerodynamics Research Conference, pp. 20.1–20.14. Cambridge, UK, Jun. 10-12, 2002, Royal Aeronautical Society, London, CD-ROM 269. Rist, U. 2003 Instability and Transition Mechanisms in Laminar Separation Bubbles. In VKI/RTO-LS ”Low Reynolds Number Aerodynamics on Aircraft Including Applications in Emerging UAV Technology”, pp. 1–29. van Karman Institut, Rhode-Saint-Genese, Belgium. Rist, U. & Augustin, K. 2006 Control of Laminar Separation Bubbles Using Instability Waves. AIAA J. 44 (10), 2217–2223. Schlichting, H. 1979 Boundary-Layer Theory. McGraw-Hill, New York. Spalart, P.R. & Strelets, M.K. 2000 Mechanisms of transition and heat transfer in a separation bubble. J. Fluid Mech. 403, 329–349. Tani, I. 1964 Low-speed ﬂows involving bubble separations. Prog. Aerosp. Sci. 5, 70–103. Watmuﬀ, J.H. 1999 Evolution of a wave packet into vortex loops in a laminar separation bubble. J. Fluid Mech. 397, 119–169.

Direct Numerical Simulation on the Influence of the Nozzle Design for Water Sheets Emerged at Moderate Reynolds Numbers Wolfgang Sander and Bernhard Weigand Institut f¨ ur Thermodynamik der Luft- und Raumfahrt, Universit¨ at Stuttgart, Pfaﬀenwaldring 31, 70569 Stuttgart, Germany [email protected] Summary. The numerical simulation of liquid jets emerging from hole-type nozzles has received considerable attention since sophisticated numerical schemes and growing computational power have become available. The reason for the eﬃciency of this novel approach is associated with high numerical accuracy and the simpliﬁed numerical setup compared to experiments. Due to this fact, several numerical experiments can be conducted by varying the inﬂuencing parameters in order to identify the most inﬂuencing factors. The high numerical accuracy of a direct numerical simulation (DNS) is basically achieved by capturing all appearing lengthscales in the ﬂow which are in the order of the usually extremely ﬁne spatial discretization and therefore turbulence modeling is not required. Since preliminary results have shown that the inﬂuence of the mean velocity proﬁle can also have a strong inﬂuence on the disintegration of the jet, this paper mainly focuses on the inﬂuence of the mean axial velocity proﬁle resulting from the variation of the nozzle design. It was found that the kinetic energy per unit mass is one of the major inﬂuencing parameters. Additionally bubbly inﬂows consisting of liquid and air have been regarded in this context. With respect to the numerical accuracy of the presented results, three different grid resolutions were tested in order to identify the inﬂuence of the spatial discretization.

1 Introduction The combustion eﬃciency in engines as well as the spray formation in medical or technical devices within process engineering is of major importance. Therefore it is important to optimize the primary breakup and the disintegration of the liquid jet adjacent to the nozzle exit due to its crucial influence on spray formation further downstream. Focus has been directed on this issue by many authors in past, where experimental studies were presented by [1, 2, 3, 4] among others. Several numerical investigations were carried out using direct numerical simulation by [5] and [6], where a particular inflow generation procedure based on a digital filtering technique described in [7]

278

W. Sander, B. Weigand

was used in order to prescribe certain defined velocity conditions at the inflow related to a specific nozzle flow. Newer numerical results were presented by [8] and [9] also using the numerical procedure developed by [7], where the focus was mainly on disintegration of circular liquid jets corresponding to conditions of high pressure injection systems for combustion engines. Similar results using the in-house 3D-CFD program FS3D in combination with the numerical procedure developed by [7] were published in [10] and [11].

2 Numerical Method The in-house 3D CFD program FS3D is based on the idea of direct numerical simulation (DNS) which permits capturing turbulent fluctuations by an extremely fine spatial resolution. The program has been developed to compute the Navier-Stokes equations for incompressible flows with interfaces or free surfaces. It allows detailed simulations on high performance computers due to its capability of interprocess-communication based on the Message Passing Interface (MPI) standard and its performance shown by [12]. The flow is governed by the conservation equations for mass and momentum for two immiscible fluids in an “one-fluid formulation”: ∇·u = 0 (1) $ % ∂ (ρu) T + ∇ · (ρu) ⊗ u = −∇p + ∇ · μ ∇u + (∇u) + ρk + fγ , (2) ∂t

where u denotes the velocity vector, t the time, ρ the density, μ the dynamic viscosity and p the pressure. Furthermore, the capillary stress tensor fγ and an external body force denoted by k are added to the momentum equation. In order to describe the interaction between the two fluids it is important to consider the capillary stress tensor which is only non-zero at the interface region between the two phases. Therefore fγ is computed by the conservative continuous surface stress (CSS) model given by [13]. The presence of a liquid and gaseous phase is considered based on the Volume-of-Fluid (VOF) method [14]. Consequently an additional transport equation ∂f + ∇ · (uf ) = 0 ∂t

(3)

is defined in order to describe the temporal and spatial evolution of the two phase flow. The variable f , called the VOF-variable, represents the volume of (liquid phase) fluid fraction as follows: ⎧ 0 outside the liquid phase ⎨ 0
On the Inﬂuence of the Nozzle Design for Water Sheets

279

Since fluid properties are required every time step in order to solve the NavierStokes equations the following relations for density ρ and viscosity μ are evaluated where the subscript g indicates the gas phase and l the liquid phase. ρ(x, t) = ρg + (ρl − ρg ) f (x, t)

μ(x, t) = μg + (μl − μg ) f (x, t) .

(5) (6)

The numerical solution of all transport equations is realized by a structured finite volume scheme on a staggered grid spatial discretization of the computational domain. In each phase, the spatial discretization is of second-order accuracy due to the second order upwind approximation for the convection terms and central diﬀerences elsewhere. The formulation for the fully conservative momentum convection and volume fraction transport, the momentum diﬀusion, and the surface tension are treated explicitly. To ensure a sharp interface of all flow discontinuities and to suppress numerical dissipation of the liquid phase during each time step the interface is reconstructed by the PLIC (piecewise linear interface calculation) method proposed by [15]. The liquid phase is transported on the basis of its reconstructed distribution. In order to satisfy the incompressibility constraint (1) the pressure field is calculated implicitly during the so called “projection” step introduced by [16], by iteratively solving the Poisson equation. Due to discontinuous coeﬃcients resulting from the discontinuous density field and usually high density ratios between the two fluids a robust multigrid solver based on a Galerkin coarse grid approximation is used for solving the Poisson equation for the pressure. In order to optimize the smoothing algorithm, an eﬃcient smoothing algorithm such as Red-Black Gauss-Seidel with several pre- and post-smoothing steps in the W-cycle is used. The numerical implementation of the Red-Black scheme is realized by stride 2 loops which allow respectable vectorization ratios on high performance computers such as the NEC-SX8 platform. Furthermore the multigrid scheme is optimized by overrelaxation of the Gauss-Seidel scheme and automatically adjusting the number of pre- and post-smoothing steps during runtime, depending on the convergence rate of the scheme.

3 Numerical Setup 3.1 Discretization The typical entrainment eﬀect appearing during the spatial evolution and the spreading of the jet must be considered within the numerical setup in order to minimize numerical eﬀects caused by the numerical boundaries. Therefore Neumann boundary conditions for the velocity are applied in the vertical (y) direction and periodic boundary conditions in the homogeneous (z) direction. At the inflow boundary, the velocity data generated by the inflow generation procedure is applied via Dirichlet boundary conditions and at the outflow

280

W. Sander, B. Weigand

modified Neumann boundary conditions for the velocity were applied in order to suppress backflow eﬀects. The extent of the computational domain presented in Fig. 1 varies depending upon the typical spreading rate of the individual case. For the following presented results the extent of the domain (L × H × W) was defined to be (18D × 10D) or (10D × 10D) for 2D-simulations and (10D × 6D × 4D) for 3D-simulations. According to this, the spatial resolution for 2D-simulations is (D/64), (D/96) up to (D/128) in each direction and (D/48)×(D/64)×(D/32) in the x, y and z-directions for the presented 3D-simulations.

Fig. 1. Computational domain

3.2 Inﬂow Conditions Since preliminary results presented in [11] have shown that the influence of the mean velocity profile in combination with turbulent fluctuations (i.e. integral lengthscales and turbulence levels) can have diﬀerent eﬀects related to jet instabilities, it was concluded to focus directly on the influence of the diﬀerent mean velocity profiles in more detail. Based on the relation given by [1], the kinetic energy per unit mass is: ǫ=

1 U03 A

0

A

u3i dA

(7)

On the Inﬂuence of the Nozzle Design for Water Sheets

281

and normalized by the flat top-hat velocity profile appears to be a dominant instability enhancing parameter, with a typical value range: ⎧ plug flow ⎨ 1.0 (8) ǫ = 1.1 − 1.2 fully developed turb. profile ⎩ laminar profile 2.0

Depending on the level of kinetic energy at the inflow, ǫ, jet instabilities are expected to be enhanced starting from the lowest energy level ǫ = 1.0 up to higher values around ǫ = 2.0. Regarding this relation in more detail, it is evident that high energy levels can only be realized by introducing steep gradients in the vertical direction for the mean axial velocity profile u. This automatically implies higher vorticity levels which is in agreement with the experimental observations made by [2]. In order to meet these requirements for the numerical setup, the mean velocity profile is defined as presented in Fig. 2. The flat top-hat velocity profile resulting from a plug flow inside a short nozzle can be realized through a constant velocity level across the nozzle exit. The mean velocity profile of 0.0

0.5

1.0

1.5

D ǫ = 1.00

11111 00000 00000 11111 00000 11111 00000 11111 00000 11111 00000 11111

ǫ = 1.14

0000000000 1111111111 1111111111 0000000000 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111

D/2 1111111111 0000000000 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111

fully developed chan. profile

short nozzle with L/D < 1

top-hat profile

00000 11111 11111 00000 00000 11111 00000 11111 00000 11111 00000 11111

long nozzle with L/D > 10 3 peaks profile

ǫ = 1.65

1111111111 0000000000 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111 1111 0000 0000 1111 1111 0000 1111111111 0000000000 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111 0000000000 1111111111

long nozzle with 2 obstacles

0 u/U0

Fig. 2. Mean velocity proﬁles according to a diﬀerent nozzle design

282

W. Sander, B. Weigand

fully developed channel flows was calculated by means of a turbulent mixing length model and validated with the data given by [17]. For the mean velocity profile with three peaks it was assumed that a certain configuration of a long nozzle with two internal obstacles can issue this type of velocity profile. There was no serious eﬀort to accurately simulate the nozzle flow in order to obtain the exact mean velocity profile and the corresponding data for the integral lengthscale and the turbulence level. Instead a constant integral lengthscale Lt = D/3 and a constant value of T u = 1.5% was prescribed for the turbulence intensities. The dimensionless parameters Re = ρl U0 D/μl , W e = ρl U0 2 D/σ and t∗ = U0 t/L are all based on the film height D and the bulk inflow velocity U0 .

4 Results The following results were all obtained using the in-house CFD-program FS3D in combination with the inflow generation procedure developed by [7]. As the main focus of this work is directed to identify the inflow parameters, mostly enhancing instabilities, the mean velocity profile and its importance was studied in this context. In order to identify how far the spatial resolution influences the regarded phenomena, several attempts were made using a 2D-simulation setup due to its relatively low computational requirements compared with 3Dsimulations. Additionally, it was studied to what degree bubbly inflows can aﬀect the instability of the liquid sheet by varying the size of the gas bubbles. 4.1 Inﬂuence of the Mean Velocity Field Since numerous 2D-simulation results have indicated that the mean axial inflow velocity profile is of crucial importance for the regarded film instability the following 3D-simulation results presented in Fig. 3 clearly confirm the observations made. It can be seen that only by variation of the value for the kinetic energy ǫ at a constant mass flow rate and turbulence quantities at the inflow significant diﬀerences appear. Especially the water film emerged with the top-hat mean velocity profile and the superposed fluctuations lead to almost negligible distortions further downstream. This basically leads to the conclusion that the fluctuations at the inflow are still present even further downstream but no major deflection of the liquid film can be achieved. By increasing the kinetic energy to a level of ǫ = 1.14 using a long nozzle and accordingly the velocity profile for fully developed turbulent channel flows, the introduced disturbances are amplified further downstream and lead to stronger deflections compared to the previous configuration where ǫ = 1.0. For the case using a long nozzle with obstacles and a level of the kinetic energy at ǫ = 1.65, identical disturbances as used for the previous configurations cause much higher disturbances.

On the Inﬂuence of the Nozzle Design for Water Sheets

283

1111 0000 0000 1111 0000 1111 0000 1111 0000 1111

ǫ = 1.00

1111 0000 0000 1111 0000 1111 0000 1111

short nozzle

ǫ = 1.14

1111111 0000000 0000000 1111111 0000000 1111111 0000000 1111111

1111111 0000000 0000000 1111111 0000000 1111111 0000000 1111111

long nozzle

ǫ = 1.65

11111111 00000000 00000000 11111111 00000000 11111111 00000000 11111111 00000000 11111111 11 00 11 00 00000000 11111111 11111111 00000000 00000000 11111111 00000000 11111111 00000000 11111111

long nozzle with 2 obstacles

Fig. 3. 3D-simulation results at Re = 4000, W e = 340, t∗ = 2.5 using diﬀerent mean velocity proﬁles and a constant turbulence intensity level of T u = 1.5% with an integral lengthscale of Lt = D/3 in each direction

This leads to the conclusion that not automatically high turbulence intensities or lengthscales lead to an optimal disintegration process, in fact it seems that only the nature of the mean velocity profile influences instabilities. From a physical point of view it appears evident that the level of turbulence quantities is related to a specific nozzle design. Hence, it can be expected that these coherences may increase in reality the instabilities appearing, compared to the simulation results obtained by defining an overall constant turbulence quantity. Since the breakup process can be classified in primary and secondary atomization, according to [18], where turbulence eﬀects arising from the character of the nozzle flow cause a ruﬄing of the liquid surface and make it more

284

W. Sander, B. Weigand

susceptible to aerodynamic forces and secondary atomization eﬀects, which always involve the interaction of aerodynamic forces, it can be expected that for certain applications these instabilities can be enhanced, by simultaneously increasing the relevant dimensionless parameters (Re/W e). It is therefore reasonable to focus primarily on the very first instabilities appearing adjacent to the nozzle exit and subsequently on the aerodynamic eﬀects which represent basically the transformation of kinetic energy into surface tension energy. 4.2 Inﬂuence of Bubbly Inﬂow In the context of primary breakup it appears interesting to consider how far bubbly inflows can influence or possibly enhance the liquid film instability. For this reason, the numerical setup must be changed by prescribing also the f value for the volume concentration at the inflow. By modification of the inflow generation procedure used to prescribe the velocity inflow data also a time dependent f -value distribution can be obtained, where the approximate size of the bubbles can be controlled by defining a certain length scale. The influence of two diﬀerent bubble sizes has been considered by prescribing very small sized and medium sized bubbles at the inflow in order to compare the outcome with the results obtained without bubbly inflow. Regarding Fig. 4 it can be identified how bubbly inflows can aﬀect the time averaged spreading rate η compared to non-bubbly inflows. Apparently medium sized bubbles have only minor eﬀects on the spatial evolution of the film whereas small sized bubbles at the inflow clearly lead to a significant lower spreading rate of the liquid film. In order to identify the character of the flow for each of these configurations snapshots of the spatial evolution are shown in Fig. 5. Regarding the spatial evolution of small sized bubbles it can be recognized that 3.0

η/D

bubble size: – bubble size: medium bubble size: small

2.0

1.0

0.0

0

4

8 x/D

12

16

Fig. 4. Temporal averaged spreading rate indicating the maximum deﬂection achieved within t∗ = 16 of the liquid ﬁlm issued with Re = 5000, W e = 540 and diﬀerent bubble sizes at the inﬂow in combination with the 3 peaks mean velocity proﬁle and constant turbulence intensity of T u = 1.5% with an integral lengthscale of Lt = D/3 in each direction

On the Inﬂuence of the Nozzle Design for Water Sheets

285

Fig. 5. Instantaneous view of 2D-simulation results at Re = 5000, W e = 540, t∗ = 7 for diﬀerent bubble sizes at the inﬂow in combination with the 3 peaks mean velocity proﬁle and constant turbulence intensity of T u = 1.5% with an integral lengthscale of Lt = D/3 in each direction

most of the small sized bubbles merge further downstream to medium sized bubbles and finally also big bubbles in the order of D/2 appear. For medium sized bubbles the merging is not clearly noticeable, but deattaching of smaller droplets can appear which was not observed for the other configurations. Reviewing these results it must be stated that the liquid mass flow rate is not identical for all configurations due to the diﬀerent volume concentration at the inflow but the diﬀerences are small and can be minimized by slightly increasing the bulk inflow velocity. Therefore it can be concluded that bubbly inflows are not enhancing film instabilities, in fact the bubble size and especially small sized bubbles can reduce instabilities and the spreading rate.

286

W. Sander, B. Weigand

4.3 Inﬂuence of the Spatial Discretization The influence of the spatial discretization which is essential for the correct capturing of all scales appearing in the flow was investigated considering the typical character of the two phase flow regarded in this study. Therefore several attempts were made using a spatial resolution of D/64, D/96 and D/128. For the configuration studied here, the Reynolds- and Weber number was increased to higher values Re = 7000, W e = 1056 because it is expected to see possible diﬀerences more clearly than for the previous configurations at Re = 5000, W e = 540. The results obtained from the three diﬀerent grid resolutions have been averaged over a time period of t∗ = 12. Regarding the mean axial and vertical velocity at axial positions x = 2D and x = 6D in Fig. 6 it can be seen velocity u/U0 [−] 0.0

Resolution:

0.5

1.0

1.5

D/64

2.0

D/128

8D

0 0

2D

4D

6D

8D 0

2D

4D

6D

8D

2.0

1.0

0.0

x/D = 2

x/D = 6

0.5

D/64 D/96 D/128 D/64 D/96 D/128

v/U0 [−]

u/U0 [−]

x =0

1.5

-0.1 -0.2 -0.3

0.0 0

1

2

y/H1/2

3

0

1

2

3

y/H1/2

Fig. 6. 2D-contour plot of the mean velocity with velocity components at axial positions x/D = 2 and x/D = 6 for a liquid sheet issued at Re = 7000, W e = 1056 from a nozzle with internal obstacles (ǫ = 1.65)

On the Inﬂuence of the Nozzle Design for Water Sheets 1.00

2.0

0.50 x/D = 6

0.25

x=0 D/64 D/96 D/128 D/64 D/96 D/128

η/D [−]

x/D = 2

0.75

f [−]

287

D/64 D/96 D/128

1.5 1.0 0.5 0.0

0.00 0

1

2

y/H1/2

3

0

2

4

6

8

x/D

Fig. 7. Mean volume concentration at axial positions x/D = 2 and x/D = 6 (left) and spreading rate (right) for a liquid sheet issued at Re = 7000, W e = 1056 from a nozzle with internal obstacles (ǫ = 1.65)

that the axial velocity field further downstream is underpredicted by a lower grid resolution and especially the entrainment of the surrounding gas flow regarding the vertical mean velocity component is clearly underpredicted at an axial position of x = 6D. Regarding the mean volume concentration and the spreading rate according to Fig. 7 it can be recognized that diﬀerences between the three grid resolutions appear again further downstream at an axial position of x = 6D. These results are coherent with the results for the mean velocity field, which leads to the conclusion that underresolved numerical simulations may lead to underestimated influence of the surrounding gas flow and a lower spreading rate. Therefore it might be reasonable to use higher grid resolutions in the order of D/96 or even D/128 once the specific inflow configurations exceed the range of Re = 3000–5000. This results for the present simulations in 640 × 640 up to 1280 × 1280 grid points.

5 Computational Performance and Resources All results presented were computed on the NEC-SX8 platform at the HLRS. For 2D-simulations a serial version of FS3D was used with highest optimization level (-C hopt) possible for Fortran based programs and automatic inlining (-pi auto) was enabled. In order to demonstrate the performance and the typical requirements of the code used for this study several simulations were carried out conducting additional performance measurements by enabling the runtime flowtrace option (-ftrace) or prof profiling (-p) at link stage. The measurement results shown in Fig. 8, indicate that the CPU-load is mainly dominated by the multigrid scheme used for the final projection step in each timestep. For this type of flow, the high density ratio between the two phases and the persistent topology changes of the flow are very typical and automatically lead to higher multigrid iteration cycles until convergence. Although the number of pre- and post smoothing steps is optimized during

288

W. Sander, B. Weigand

Fig. 8. Overall CPU-time requirement for 2D-simulations on the NEC-SX8. Upper bars indicate the required CPU-time of the simulation with small sized bubbles – lower bars for the simulation with medium sized bubbles

runtime and overrelaxation of the Gauss-Seidel scheme is introduced depending on the convergence rate of the scheme, around 70% of the total CPU-time is used for the smoothing, restriction, prolongation steps and the exact solution via gauss-elimination on the coarsest grid. The significant diﬀerence between the two flows regarded here is mainly caused by the bubble size and the number of interface volumes of the numerical grid. It can be assumed that the CPU-load due to the VOF-advection and especially the surface reconstruction PLIC-algorithm is aﬀected by this relation. Surprisingly the load is higher for the case with medium sized bubbles which appears illogical at the first glance since the small sized bubbles apparently cause more interface regions. However, the higher spreading rate of the case with medium sized bubbles must be considered, which is responsible for an even higher number of interface regions. Anyhow these diﬀerences are minor compared to the load required for the multigrid scheme and even momentum diﬀusion and advection including the boundary condition setup require just around 12% of the total CPU-time. The reason for this is mainly the very high performance of these routines which was found to be in the order of 10 GFLOPS and a V. op. Ratio (%) > 99. Also the runtime control including the time marching and the timestep adjustment and especially the output of datafiles is generally low. The only diﬀerence was found to be caused by the diﬀerent performance measurement methods due to the explicit listing of system function calls using the option (-p) which were added to

On the Inﬂuence of the Nozzle Design for Water Sheets

289

category of other. The performance reported using F PROGINF=DETAIL shows the generally good performance of the program which is in the order of 4.000 MFLOPS and a V. op. Ratio (%) ≈ 97. Considering the penalty generated by the performance measurement routine for the case measured with ftrace, a diﬀerence of 600 MFLOPS was identified. Several other cases reach values up to 5.000 MFLOPS depending on the character of the flow and the numerical grid used. For 3D-simulations a parallel version of the program using the (-P auto) option was used on one node. The performance measurement given in Fig. 9 shows a CPU-load similar to the 2D-simulation cases if the inflow-generation is not considered. As the program allows to read already generated velocity data it can be decided to generate this separately and reuse previously generated data or generate the inflow velocity data prior to the simulation. The CPU-load for this case is significantly dominated by the inflow-generation procedure which is mainly caused by the two-dimensional inflow side and the high number of filtered datasets (≈ 5000 for t∗ = 1) necessary, for each timestep of the simulation. Considering the performance of 3D-simulations on 8 CPUs also values around 4.000 MFLOPS and a V. op. Ratio (%) ≈ 98 were reached. The CPU-time required for the cases presented here of course varies depending on the specific numerical setup as it was demonstrated previously. The CPU-hours used for each single flow through (t∗ = 1) of the liquid film based

Fig. 9. Overall CPU-time requirement for 3D-simulations on the NEC-SX8. Upper bars indicate the required CPU-time for simulations without using the inﬂowgeneration procedure – lower bars include the CPU-time required for the generation of the inﬂow velocity data

290

W. Sander, B. Weigand

on the axial extent of the domain requires approximately 10 CPUh and typically 100–150 CPUh for t∗ = 10–15 in each single 2D-simulation with a grid spacing of D/64. The requirement for the 3D-simulations and the grid spacing presented was in the order of 64 CPUh for t∗ = 1 including the generation of inflow velocity data.

6 Concluding Remarks Since the parameters considered within this study have partially shown drastic influences on the instability of liquid sheets, it can be concluded that the optimal nozzle geometry for injection systems must be designed with respect to the kinetic energy per unit mass of the flow and the coherent turbulence quantities. As it is demonstrated, not necessarily high velocities and Reynolds or Weber number can enhance instabilities, in fact the character of the nozzle flow leads to first disturbances of the liquid surface which are amplified further downstream depending on the mean velocity profile. Since 2D- and 3D-simulations have qualitatively similar results it is justified to perform 2Dsimulations as numerical experiments and 3D-simulations to exactly determine the influence of the 2D-simplification for certain configurations. Additionally it was shown how far bubbly inflows may aﬀect instabilities and which eﬀect the spacing of the numerical grid can have especially at higher Reynolds-numbers. The range of higher Reynolds numbers, interesting for technical applications is therefore related to these first results as the performance of the program and the results for the moderate Reynolds number range may be relevant for further studies. Future studies will also focus on this field and the comparison between experimental and numerical data. For future simulations it is also planned to modify the inflow generation procedure and the features in FS3D in order to study circular liquid jets which are widely used in technical applications. Acknowledgements The authors greatly appreciate the High Performance Computing Center Stuttgart (HLRS) for support and supply of computational time on the NEC-SX8 platform.

References 1. M.J. McCarthy and N.A. Molloy. Review of Stability of Liquid Jets and the inﬂuence of Nozzle Design. Chemical Engineering Journal, Vol. 7, pp. 1–20, 1974. 2. P. Wu, R. Miranda, and G. Faeth. Eﬀect of initial inﬂow conditions on primary break-up of nonturbulent and turbulent jets. Atomization and Sprays, Vol. 5, pp. 175–196, 1995.

On the Inﬂuence of the Nozzle Design for Water Sheets

291

3. H. Hiroyasu. Spray breakup mechanism from the hole-type nozzle and its applications. Atomization and Sprays, Vol. 10, pp. 511–527, 2000. 4. K. Heukelbach. Untersuchung zum Einfluss der D¨ useninnenstr¨ omung auf die Stabilit¨ at von fl¨ achigen Fl¨ ussigkeitsstrahlen. PhD thesis, Technische Universit¨ at Darmstadt, 2003. 5. M. Klein. Direkte numerische Simulation des prim¨ aren Strahlzerfalls in Einstoffzerst¨ auberd¨ usen. PhD thesis, Technische Universit¨ at Darmstadt, 2002. 6. M. Klein. Direct numerical simulation of a spatially developing water sheet at moderate Reynolds number. Journal of Heat and Fluid Flow, Vol. 26, pp. 722–731, 2005. 7. M. Klein, A. Sadiki, and J Janicka. A digital ﬁlter based generation of inﬂow data for spatially developing direct numerical or large eddy simulations. Journal of Computational Physics, Vol. 186, pp. 652–665, 2003. 8. T. M´enard, P.A. Beau, S. Tanguy, F.X. Demoulin, and A. Berlemont. Primary Break up modeling Part A: DNS, a tool to explore primary break up. In Proceedings 10th ICLASS, Kyoto, Japan, 2006. 9. Y. Pan and K. Suga. LARGE EDDY SIMULATION OF TURBULENT LIQUID JETS INTO AIR. In Proceedings 10th ICLASS, Kyoto, Japan, 2006. 10. W. Sander, B. Weigand, K. Jellinghaus, and K.D. Beheng. Direct Numerical Simulation of Breakup Phenomena in Liquid Jets and of Colliding Raindrops. In High-Performance Computing in Science and Engineering 2005: Transactions of the High Performance Computing Center Stuttgart (HLRS), pp. 129–142. Springer-Verlag, 2005. 11. W. Sander and B. Weigand. Direct Numerical Simulation of Breakup Phenomena in Liquid Sheets. In High-Performance Computing in Science and Engineering 2006: Transactions of the High Performance Computing Center Stuttgart (HLRS), pp. 223–236. Springer-Verlag, 2006. 12. M. Rieber and A. Frohn. Parallel computation of interface dynamics in incompressible two-phase ﬂows. In High-Performance Computing in Science and Engineering 99: Transactions of the High Performance Computing Center Stuttgart (HLRS), pp. 241–252. Springer Verlag, 2000. 13. B. Lafaurie, C. Nardone, R. Scardovelli, S. Zaleski, and G. Zanetti. Modelling merge and fragmentation in multiphase ﬂows with SURFER. Journal of Computational Physics, Vol. 113, pp. 134–147, 1994. 14. C.W. Hirt and B.D. Nichols. Volume of ﬂuid (VOF) method for the dynamics of free boundaries. Journal of Computational Physics, Vol. 39, pp. 201–225, 1981. 15. W.J. Rider and D.B. Kothe. Reconstructing volume tracking. Journal of Computational Physics, Vol. 141, pp. 112–152, 1998. 16. J.B. Bell, P. Colella, and H.M. Glaz. A second-order projection method for the incompressible navier-stokes equations. Journal of Computational Physics, Vol. 85, pp. 257–283, 1989. 17. R.D. Moser, J. Kim, and N. N. Mansour. Direct numerical simulation of turbulent channel ﬂow up to Reτ = 590. Physics of Fluids, Vol. 11, pp. 943–945, 1999. 18. A. Lefebvre. Atomization and Sprays. Taylor & Francis, 1989.

DNS of Heat Transfer from a Flat Plate affected by Free-Stream Fluctuations Jan G. Wissink and Wolfgang Rodi Institute for Hydromechanics, University of Karlsruhe Kaiserstraße 12, D-76128 Karlsruhe, Germany [email protected], [email protected] Summary. Direct Numerical Simulations (DNS) of heat transfer from a ﬂat plate aﬀected by free-stream ﬂuctuations are performed at three diﬀerent Reynolds numbers. A contoured upper wall is employed to generate a favourable pressure gradient along a large portion of the ﬂat plate. The free-stream ﬂuctuations stem from a separate LES of isotropic turbulence in a box. In the laminar portions of the accelerating boundary layer ﬂow, the formation of streaks was observed to induce an increase in heat transfer by exchange of hot ﬂuid near the surface of the plate and cold ﬂuid from the free-stream. Compared to the accompanying fully laminar simulations (without free-stream ﬂuctuations), in the simulations with free-stream ﬂuctuations the increase in heat transfer in the fully turbulent part of the boundary layer was found to be signiﬁcantly larger than the increase in heat transfer induced by the formation of streaks in the laminar part. The computations were carried out on the HP-XC1 cluster in Karlsruhe using up to 64 processors and 137 million grid points.

1 Introduction Free-stream turbulence in incoming flow can influence strongly heat transfer to turbine blades. The physical mechanisms involved are only just beginning to be understood (see Mayle et al. [M98]). Mainly two eﬀects of free-stream turbulence impinging on a blade boundary-layer can be distinguished: 1. Earlier transition on the suction side which leads to an increase in heat transfer due to turbulent motion of the boundary layer flow 2. Increase in heat transfer in regions where the boundary layer remains laminar, where one can diﬀerentiate between: • Pre-transitional/inactive regions of the suction side boundary layer • The entire pressure side boundary layer Free-stream turbulence can be either uniformly distributed or concentrated in wakes. To be able to predict the eﬀects of free-stream turbulence impinging on

294

J.G. Wissink, W. Rodi

a boundary layer one needs to know both the integral length-scale of this turbulence and its turbulence-level. When free-stream turbulence is concentrated in wakes, also the frequency of these wakes should be taken into account. Several experiments have been performed to study the influence of freestream turbulence on “laminar” heat transfer – that is: the heat transfer in regions where the boundary layer is laminar – and on boundary layer transition. In their experimental studies of laminar heat transfer to a flat plate aﬀected by free-stream fluctuations, Kestin et al. [K61] and Junkhan and Serovy [J67] discovered that in order for these fluctuations to be able to increase heat transfer, the aﬀected laminar boundary layer flow needs to be accelerating. Liu and Rodi [L94a, L94b] performed heat transfer measurements of flow in a turbine cascade with incoming wakes. With increasing wake-frequency, both the heat transfer along the pressure side, with its strong acceleration and the heat transfer along the suction side, which exhibited flow-acceleration upstream of s = 0.65 – where s is the normalised surface coordinate - and transition downstream of s = 0.65, was found to increase. In the recent experiments of Choi et al. [C04], studying the eﬀect of free-stream turbulence on turbine blade heat transfer, the grid-generated turbulence was shown to aﬀect the heat transfer along both areas of the turbine blade with flow acceleration (i.e. the entire pressure side and approximately the upstream half of the suction side) and also along those areas which exhibit transition to turbulence. Increasing the free-stream turbulence level – while keeping the Reynolds number fixed – was shown to lead to an increase in heat transfer. As already noted above, the physical mechanisms that play a role in the interaction of free-stream fluctuations and an accelerated laminar boundary layer flow – causing a significant increase in laminar heat transfer – are not well understood. To clarify the physical mechanisms, Direct Numerical Simulations (DNS) of laminar heat transfer are currently performed on the HP-XC1 cluster at the computing centre in Karlsruhe. Some of the results are reported in this paper.

2 Computational Details The three-dimensional, incompressible Navier-Stokes equations were discretised using a finite-volume method with collocated variables. In space a secondorder central discretisation was used, combined with a three-stage RungeKutta method for the time-integration. To avoid the decoupling of the velocity field and the pressure field, the momentum interpolation procedure of Rhie and Chow [R83] was employed. The Poisson equation for the pressure was solved using the SIP solver of Stone [S68]. A more complete description of the computational code can be found in Breuer and Rodi [B96]. For the heat transfer, a convection-diﬀusion equation for the temperature, T , was solved using a second-order accurate central finite-volume method which was again

DNS of Heat Transfer from a Flat Plate

295

combined with a three-stage Runge-Kutta method for the time-integration. The Prandtl-number was chosen to be P r = 0.71. The code was parallelised using the standard Message Passing Interface (MPI) protocol. The larger simulations, performed on the HP-XC1, used 64 processors. Compared to the IBM SP SMP – also at the computing centre in Karlsruhe – a speedup of more than a factor of 3 was obtained. ue u=ublas(1+u’/ue) v=ublasv’/ue w=ublasw’/ue

f re es

l ip

T=0.7T0 convective outflow

no-slip, T=T0 0

1

2

3

4

5

x/L Fig. 1. Spanwise cross-section through the computational domain

Figure 1 shows a spanwise cross-section through the computational domain. The flat plate was heated to a fixed temperature of T = T0 , while the cool outer flow had a temperature of T = 0.7T0 At the surface of the flat plate (i.e. the lower wall) a no-slip boundary condition was prescribed, at the outlet a convective outflow boundary condition was used and in the spanwise direction periodic boundary conditions were used. Along the contoured upper wall a free-slip boundary condition was employed. The shape of the upper wall is such that a strongly accelerating boundary layer flow is obtained along a significant portion of the flat plate. The Reynolds number employed in the simulations was Re = 80 000, based on the free-stream velocity at the inlet, ue , and the length scale L, which is implicitly defined in Fig. 1. The time-step smployed in the simulations was δt = 0.0001L/ue. The spanwise size of the computational domain was varied to accommodate free-stream fluctuations with various integral length-scales Λ. At the inflow plane, located at x/L = 0.0, free-stream fluctuations (u′ , v ′ , w′ ) which were generated in a separate LES of isotropic turbulence in a box - were superposed on a Blasius profile in which the v-velocity component was set to zero, i.e. (uin , vin , win ) = ub [(1, 0, 0) + 1/ue (u′ , v ′ , w′ )], where (uin , vin , win ) is the velocity at the inlet and ub is the u-component of the Blasius profile. The multiplication by ub guarantees that the no-slip boundary condition was fulfilled at y/L = 0. As shown in Table 1, three simulations with varying inflow conditions were performed. Simulation A – without inflow fluctuations – was eﬀectively two-dimensional and its purpose was to obtain laminar reference data. Simulations B and C were performed using free-stream fluctuations with a level of T u = 5%. In Simulation B the integral length-scale of the fluctuations was

296

J.G. Wissink, W. Rodi

Table 1. Overview of the simulations performed. Λ is the integral length-scale of the free-stream ﬂuctuations. The level of the free-stream ﬂuctuations added at the inﬂow plane was T u = 5.0%, and the Reynolds number was Re = 80 000 Simulation grid A 902 × 294 × 4 902 × 294 × 512 B 902 × 294 × 256 C

spanwise size 0.0003L 1.4L 0.7L

Λ – 0.0830L 0.0415L

Λ = 0.0830L, while in Simulation C an integral length-scale of Λ = 0.0415L was prescribed. Figure 2 shows every eighth grid-line in a cross section through the computational domain at midspan. The distribution of grid points in the mesh was based on experience gained earlier in performing various DNS of flow in low-pressure turbine cascades (see Michelassi et al. [M02] and Wissink [W03]) and produced a satisfactory resolution of the flat plate boundary layer. A gridrefinement study will be performed at a later stage.

Fig. 2. Mesh at midspan showing every eighth grid-line

3 Performance of the Code on the XC1 In this section, the performance of the code on the XC1 will be analyzed for both Simulations B and C (see also Table 1). Both simulations took 100 000 time steps to finish, which corresponds to 673 clock hours of continuous running for Simulation B and 433 clock hours for Simulation C. Some of the statistics of the runs are listed in Table 2. As already mentioned above, in both simulations 64 processors and a time step of δt = 0.0001 were employed, which simplifies the comparison of the performance of the numerical simulations. The most significant quantitity for the assessment of the performance is the computing time per grid point

DNS of Heat Transfer from a Flat Plate

297

Table 2. Some characteristics related to the performance of the Simulations B and C on the HP-XC1 cluster in Karlsruhe. t1 is the computing time needed to perform one time-step, t1000 is the computing time needed to perform one time-unit and tpoint is the computing time needed per grid point to perform one time-unit Simulation points per block B 2121504 C 1060752

t1 24.228 s. 15.578 s.

t1000 242280 s. 155780 s.

tpoint 0.0017855 s. 0.0022946 s.

needed to perform 1 time-unit (which corresponds to 10000 time steps in each simulation). In Simulation B, this time was tpoint = 0.0017855 s., while in Simulation C the time was tpoint = 0.0022946 s, which was slightly larger. Hence, compared to Simulation C, the increase by 55.5% of the number of grid points per block in Simulation B resulted in an increase in performance of the code on the XC1 of 28.5%. The main reason of this increase is the reduction in communication time per grid point associated with the usage of larger blocks in Simulation B, since the percentage of the data per block that is exchanged each time-step gets smaller as the block size increases. From this we may conclude that it is computationally more eﬃcient to use as few processors as possible when performing simulations.

4 Results Figure 3 shows the wall static-pressure distribution Cp =

p− ¯ p¯ref 1 2 2 Uref

along the

flat plate from Simulation A. The Cp -distribution shown is characteristic for all simulations. The reference values for the mean pressure p¯ref and the mean velocity Uref were extracted at x/L = 0.2, y/L = 1.4. Around x/L = 0.5 the pressure gradient is virtually neutral. In between x/L ≈ 1 and x/L ≈ 4 the pressure gradient is strongly favourable and downstream of x/L = 4 it again becomes neutral. In the strongly Favourable Pressure Gradient (FPG) region the boundary layer flow is stabilised and transition is suppressed. 0

Cp

-6

-12

-18

0

1

2

3

4

5

x/L

Fig. 3. Wall static-pressure coeﬃcient Cp along the ﬂat plate from Simulation A

298

J.G. Wissink, W. Rodi 0.02

δ99/L

0.015 maxx/L(δ99) = 0.01583L

minx/L(δ99) = 0.00723L

0.01 0.005 0

0

1

2

3

4

5

x/L

Fig. 4. Boundary layer thickness δ99 in the laminar simulation

Figure 4 shows the boundary layer thickness δ99 along the flat plate obtained in the Simulation A. In the FPG region, at x/L ≈ 3.3, δ99 reaches a minimum value of 0.00723L – a factor 11.5 times smaller than the integral length-scale, Λ, of the free-stream fluctuations in Simulation B and a factor 5.74 times smaller than Λ in Simulation C. Figure 5 shows snapshots with contours of the streamwise velocity fluctuations u′ and the temperature T /T0 in a grid-plane close to the surface of the flat plate from Simulations B. The snapshot of the streamwise velocity fluctu-

Fig. 5. Simulation B: Contours of the streamwise velocity ﬂuctuations (upper ) and the instantaneous temperature (lower ) – at the same instance in time – in a gridplane close to the ﬂat plate

DNS of Heat Transfer from a Flat Plate

299

ations clearly shows the presence of low-speed streaks (blue) and high-speed streaks (red) in the FPG region of the flat plate. Upstream of x/L = 2 some of the streaks are unstable and some small turbulent spots can be observed. Because of the strong streamwise FPG downstream of this location, the flow re-laminarizes and the turbulent spots disappear. Between of x/L = 2 and x/L = 4, there is no sign of further transition to turbulence. Low-speed streaks form as low-speed (hot) flow from the bottom of the boundary layer moves upwards, while the high-speed streaks form as highspeed (cold) flow from the upper part of the boundary layer moves downwards. This is confirmed by comparing the u′ -snapshot to the snapshot with contours of the instantaneous temperature T /T0 , showing a correlation between the presence of low speed streaks and regions of increased temperature on the one hand and high speed streaks and regions of decreased temperature on the other hand. The flat plate boundary-layer from Simulation B undergoes further transition to turbulence immediately upstream of x/L = 5 (not shown in the figure). Figure 6 shows snapshots with contours of the streamwise velocity fluctuations and the instantaneous temperature from Simulation C. As in Fig. 5, some turbulent spots can be observed around and upstream of x/L = 2. Farther downstream, around x/L = 3 (where the FPG is strongest), all traces of turbulence have disappeared. Again a correlation can be seen between low speed streaks and regions of high temperature and high-speed streaks and regions of low temperature. Opposed to the situation in Simulation B, the boundary layer flow from Simulation C undergoes transition to turbulence much earlier. Already at x/L ≈ 3.5 transition can be seen to begin and at x/L ≈ 4 the boundary layer flow has turned completely turbulent. The reason for the early transition observed in Simulation C is the diﬀerent spectral contents of the free-stream fluctuations. Compared to Simulation B – with Λ = 0.0830L –, in Simulation C – with Λ = 0.0415L – the smaller length-scales contain more

Fig. 6. Simulation C: Contours of the streamwise velocity ﬂuctuations (upper ) and the instantaneous temperature (lower ) – at the same instance in time – in a gridplane close to the ﬂat plate. The legends are identical to the ones displayed in Fig. 5

300

J.G. Wissink, W. Rodi

energy. While the larger length scales are responsible for the triggering of streaks, the smaller length scales are responsible for triggering instabilities in the low speed streaks, thereby initiating further transition to turbulence. Hence, in Simulation C a stronger triggering of low-speed streaks instabilities takes place resulting in an earlier transition to turbulence. The instantaneous Nusselt number N u = hL k , where h is the heat transfer coeﬃcient and k is the thermal diﬀusivity, is used as a measure for the heat transfer from the flat plate to the outer flow. In Figure 7, the Nusselt numbers obtained in the Simulations A, B, C with and without free-stream fluctuations are compared to one another. Compared to the laminar Simulation A, a significant increase in N u is only observed in the region where the boundary layer flow from Simulation C undergoes transition to turbulence (between x/L = 3.5 and x/L = 4.0. As already noted above, in Simulation B the onset of transition is relatively far downstream, resulting in a slight increase in N u immediately upstream of x/L = 5. Along the predominanly laminar region of the flat plate, the Nusselt number is also found to increase. In the FPG region around x/L = 3, in Simulation B N u is found to increase by 17.8%, while in Simulation C an increase of 22.5% is found. 800 Simulation A Simulation B Simulation C

Nu

600

400

200

0

0

1

2

x/L

3

4

5

Fig. 7. Instantaneous Nusselt number along the ﬂat plate 1500 Simulation A Simulation B Simulation C

Cf

1000

500

0

1

2

3

4

5

x/L

Fig. 8. Mean friction coeﬃcient along the ﬂat plate for the simulations listed in Table 1

DNS of Heat Transfer from a Flat Plate

301

Figure 8 shows the mean friction-coeﬃcient, Cf , along the flat plate for all simulations listed in Table 1. Compared to the Simulation A – which shows the laminar value of Cf – the location of transition in Simulation C is easily identified by a sudden growth in the Cf -level starting at x/L ≈ 3.6. Concerning the transition to turbulence, the Cf -distribution is correlated with the Nusselt number shown in Fig. 7, confirming that the observed significant increase in N u in Simulation C can be attributed almost entirely to the boundary layer flow undergoing transition to turbulence. Upstream of x/L = 2.5, the Cf -level in Simulations B and C is slightly higher than the level in Simulation A, which indicates that the boundary layer flow is not completely laminar. Around x/L = 3, where the FPG is strongest, the friction coeﬃcient in both Simulations B and C assumes the laminar value of Simulation A. Hence, at this location the boundary layer flow is completely laminar in all simulations, such that the increase in heat transfer is completely due to laminar eﬀects.

5 Conclusions Three-dimensional Direct Numerical Simulations of flow over and heat transfer from a heated flat plate with and without free-stream fluctuations at a Reynolds number of Re = 80 000 have been performed on the HP-XC1 cluster in Karlsruhe. The special shape of the upper wall of the computational domain induced a strong, favourable pressure gradient along a significant portion of the flat plate. The free-stream fluctuations (T u = 5% at the inflow plane) were generated in a separate large-eddy simulation of isotropic turbulence in a box. By rescaling the size of the box, the integral length-scale of the free-stream fluctuations was varied. In all cases, the presence of free-stream fluctuations caused the boundary layer flow to undergo by-pass transition. Initially, in both Simulations B and C, streaks were triggered in the region around x/L = 0.5 – immediately upstream of the region with the strong Favourable Pressure Gradient (FPG) – by free-stream fluctuations with a relatively large length-scale. In the FPG region itself, further transition to turbulence was delayed by the action of the accelerating boundary-layer flow which tends to damp disturbances. In the region around x/L = 3, the presence of streaks was found to promote heat transfer from the heated flat plate to the cold outer flow by 17.8–22.5%. In Simulations B and C, initially traces of turbulent spots were found in the region upstream of x/L ≈ 2.5, leading to a slight increase in the friction coeﬃcient as compared to its laminar value. Around x/L = 3, where the favourable pressure gradient is strongest, any traces of turbulent spots were found to have completely disappeared. Farther downstream, the exact location of transition was found to depend on the integral length-scale Λ of the free-stream fluctuations. A decrease in Λ resulted in a stronger triggering of the instability of the low-speed streaks by free-stream fluctuations with

302

J.G. Wissink, W. Rodi

a small length-scale. As a result the location of transition was found to move upstream. The most significant increase in heat transfer was observed in Simulation C in the region where the boundary layer flow undergoes transition to turbulence.

References [B96]

[C04]

[J67]

[K61]

[L94a] [L94b]

[M98] [M02]

[R83] [S68]

[S86]

[W03]

Breuer, M. and Rodi, W.: Large eddy simulation for complex ﬂow of practical interest. In: Flow Simulation with High-Performance Computers II, Notes on Num. Fluid Mech., 52, Vieweg Verlag, Braunschweig (1996) Choi, J., Teng, S., Ladeinde, F.: Eﬀect of free-stream turbulence on turbine blade heat transfer and pressure coeﬃcients in low Reynolds number ﬂows. Int. J. Heat and Mass Transfer, 47, 3441–3452 (2004) Junkhan, G.H. and Serovy, G.K.: Eﬀects of free-stream turbulence and pressure-gradient on ﬂat plate boundary layer velocity proﬁles and on heat transfer. ASME J. Heat Transfer, 89, 169–176 (1967) Kestin, J., Maeder, P.F., Wang, H.E.: Inﬂuence of turbulence on heat transfer from plates with and without a pressure gardient. Int. J. Heat and Mass Transfer, 3, 133–154 (1961) Liu, X. and Rodi, W.: Velocity measurements in wake-induced unsteady ﬂow in a linear turbine cascade. Exp. in Fluids 17, 45–58 (1994) Liu, X. and Rodi, W.: Surface pressure and heat transfer measurements in a turbine cascade with unsteady oncoming wakes. Exp. in Fluids 17, 171–178 (1994) Mayle, R.E., Dullenkopf, K., Schulz, A.: The turbulence that matters. ASME J. of Turbomachinery, 120, 402–409 (1998) Michelassi, V., Wissink, J.G., Rodi, W.: Analysis of DNS and LES of ﬂow in a low-pressure turbine cascade with incoming wakes and comparison with experiments. Flow, Turbulence and Combustion, 69, 295–329 (2002) Rhie C.M. and Chow, W.L.: Numerical study of the turbulent ﬂow past an airfoil with trailing edge separation. AIAA J., 21, 1525–1532 (1983) Stone, H.L.: Iterative Solution of Implicit Approximations of Multidimensional Partial Diﬀerential Equations. SIAM J. Numerical Analysis, 5, 530–558 (1968) Schulz, A.: “Zum Einﬂuß h¨ oher Einstromturbulenz, intensiver K¨ uhlung und einer Nachlaufstr¨ omung auf den außeren W¨ arme¨ ubergang einer konvektiv gek¨ uhlten Gasturbinenschaufel”, PhD. Thesis, U. Karlsruhe, Karlsruhe, Germany (1986) Wissink, J.G.: DNS of separating, low-Reynolds number ﬂow in a turbine cascade with incoming wakes. Int. J. Heat and Fluid Flow, 24, 626–635 (2003)

Direct Numerical Simulation of Turbulent Flow Over Dimples – Code Optimization for NEC SX-8 plus Flow Results M. Breuer1 , P. Lammers2 , T. Zeiser3 , G. Hager3 , and G. Wellein3 1

2

3

Institute of Fluid Mechanics, University of Erlangen-N¨ urnberg, Cauerstraße 4, D-91058 Erlangen, Germany [email protected] H¨ ochstleistungsrechenzentrum Stuttgart (HLRS), Nobelstraße 19, D-70569 Stuttgart, Germany Regionales Rechenzentrum Erlangen (RRZE), Martensstraße 1, D-91058 Erlangen, Germany

Summary. The objective of the project DIMPLE is to study complex turbulent ﬂows over a dimpled surface which is expected to lead to drag reduction compared to a smooth surface. In order to understand the physical mechanism of the ﬂow over an arrangement of shallow dimples, CPU time-consuming direct numerical simulations were carried out using two diﬀerent grid resolutions and three diﬀerent conﬁgurations. As a ﬁrst step within this project, the code LESOCC was evaluated and optimized for the new vector machine NEC SX-8. Classical vector systems still combine excellent performance with a well-established optimization approach. On the other hand, clusters based on commodity microprocessors oﬀer comparable peak performance at much lower costs. In the context of the introduction of the NEC SX-8 vector computer series we compare single and parallel performance of a computational ﬂuid dynamics application on the SX-8 and on the SGI Altix architecture demonstrating the potential of the SX-8 for teraﬂop computing in the area of turbulence research and ﬂow control. Finally, results of the ﬂow predictions over a dimpled surface are presented and compared with the reference case of plane channel ﬂow with smooth walls. Key words: Dimples, Drag Reduction, Flow Control, Turbulence, Direct Numerical Simulation, High-Performance Computing, Sustained Performance, NEC SX-8

1 Introduction The objective of the present project DIMPLE is to investigate and analyze the so-called ‘dimple’ eﬀect based on direct numerical simulations (DNS) of turbulent flows. Some years back Russian scientists (see, e.g., [1]) found out that

304

M. Breuer et al.

by a special structure of the surface the skin friction drag of a turbulent flow can be decreased by up to 20% (no golf ball eﬀect). Their findings are based on experimental measurements of turbulent flows over surfaces with a regular arrangement of troughs called dimples. However, up to now no clear explanation was provided which can illuminate this eﬀect. Hence some doubts remain regarding the experimental investigations and their outcome. Compared to classical devices for drag reduction such as riblets, shallow dimples were advantageous since they are composed of macroscopic structures which are less sensitive to dirt, mechanical degradation and they do not depend on the flow direction. Since the drag reduction promised by dimples would have a tremendous economical influence, it is worth to study it in more detail. For that purpose, direct numerical simulations of the turbulent flow inside a channel with dimpled walls were carried out to investigate the physical mechanism in-depth. As the reference test case the fully developed turbulent channel flow was chosen because it oﬀers several significant advantages: there are no other drag components involved but skin friction, the boundary conditions are very well defined and the wall shear stress can be deduced from relatively simple and reliable pressure drop measurements in a complementary experimental set-up [8]. However, these kinds of simulations require several millions of grid points and hundred thousands of time steps to achieve reliable statistics of turbulent quantities. Thus they demand for high-performance computers (HPC) and huge amounts of CPU resources. Consequently, it is useful to spend some eﬀort towards the optimization of the code for the special architecture of HPC machines used. Even though massively parallel systems or clusters based on microprocessors deliver high peak performance and large amounts of compute cycles at a very low price tag, it has been emphasized recently that vector technology is still extremely competitive or even superior to massively parallel systems if application performance for memory intensive codes is the yardstic [9–11]. Introducing the new NEC SX-8 series in 2005, the powerful technology used in the NEC Earth Simulator (ES) has been pushed to new performance levels by doubling all important performance metrics like peak performance, memory bandwidth and interconnect bandwidth. Since the basic architecture of the system itself did not change at all from a programmer’s point of view, the new system is expected to run most applications roughly twice as fast as its predecessor, even using the same binary. In this report we test the potentials of the new NEC SX-8 architecture using the real world application described above and compare the results with the predecessor system (NEC SX-6+) as well as a microprocessor based system. For the latter we have chosen the SGI Altix, which uses Intel Itanium 2 processors and usually provides high eﬃciencies for the applications under consideration in this report. We focus on a CFD code from turbulence research which is a classical finite-volume code called LESOCC (Large-Eddy Simulation On Curvi-

Project DIMPLE: DNS of Turbulent Flow Over Dimples

305

linear Co-ordinates [2–5]), mainly written in FORTRAN. The code is MPIparallelized using domain decomposition and has been optimized for a wide range of computer architectures (see e.g. [4, 7]). As test cases we run simulations of flow in a plane channel and over a single flat plate. These flow problems are intensively studied in the context of wall-bounded turbulent flows, flow control, and drag reduction.

2 Architecture of the NEC SX-8 From a programmer’s view, the NEC SX-8 CPU is a traditional vector processor with 4-track vector pipes running at 2 GHz. One multiply and one add instruction per cycle can be sustained by the arithmetic pipes, delivering a theoretical peak performance of 16 GFlop/s. The memory bandwidth of 64 GByte/s allows for one load or store per multiply-add instruction, providing a balance of 0.5 Word/Flop. The processor has 64 vector registers, each holding 256 64-bit words. Basic changes compared to its predecessor systems are a separate hardware square root/divide unit and a “memory cache” which lifts stride-2 memory access patterns to the same performance as contiguous memory access. An SMP node comprises eight processors and provides a total memory bandwidth of 512 GByte/s, i.e., the aggregated single processor bandwidths can be saturated. The SX-8 nodes are networked by an interconnect called IXS, providing a bidirectional bandwidth of 16 GByte/s and a latency of about 5 microseconds. For a comparison with the technology used in the ES, we have chosen a NEC SX-6+ system which implements the same processor technology as used in the ES but runs at a clock speed of 565 MHz instead of 500 MHz. In contrast to the NEC SX-8, this vector processor generation is still equipped with two 8-track vector pipelines allowing for a peak performance of 9.04 GFlop/s per CPU for the NEC SX-6+ system. Note that the balance between main memory bandwidth and peak performance is the same as for the SX-8 (0.5 Word/Flop) both for the single processor and the 8-way SMP node. Thus, we expect most application codes to achieve a speed-up of around 1.77 when going from SX-6+ to SX-8. Due to the architectural changes described above, the SX-8 should be able to show even a better speed-up on some selected codes. As a competitor we have chosen the SGI Altix architecture which is based on the Intel Itanium 2 processor. This CPU has a superscalar 64-bit architecture providing two multiply-add units and uses the Explicitly Parallel Instruction Computing (EPIC) paradigm. Contrary to traditional scalar processors, there is no out-of-order execution. Instead, compilers are required to identify and exploit instruction level parallelism. Today clock frequencies of up to 1.6 GHz and on-chip caches with up to 9 MBytes are available. The basic building block of the Altix is a 2-way SMP node oﬀering 6.4 GByte/s memory

306

M. Breuer et al.

bandwidth to both (single-core) CPUs, i.e., a balance of 0.06 Word/Flop per CPU. The benchmark results presented in this paper were measured on the NEC SX-8 system (576 CPUs) at High Performance Computing Center Stuttgart (HLRS), the SGI Altix3700Bx2 (128 single-core CPUs, 1.6 GHz/6 MB L3) at Leibniz Rechenzentrum M¨ unchen (LRZ) and the SGI Altix3700 (128 singlecore CPUs, 1.5 GHz/6 MB L3) at CSAR Manchester. All performance numbers are given in GFlop/s.

3 Finite–Volume Code LESOCC The CFD code LESOCC was developed for the simulation of complex turbulent flows using either the methodology of direct numerical simulation (DNS), large-eddy simulation (LES), or hybrid LES-RANS coupling such as the detached-eddy simulation (DES). LESOCC is based on a 3-D finite-volume method for arbitrary nonorthogonal and non-staggered, block-structured grids [2–5]. The spatial discretization of all fluxes is based on central diﬀerences of second-order accuracy. A low-storage multi-stage Runge–Kutta method (second-order accurate) is applied for time-marching. In order to ensure the coupling of pressure and velocity fields on non-staggered grids, the momentum interpolation technique is used. For modeling the non-resolvable subgrid scales, a variety of diﬀerent models is implemented, cf. the well-known Smagorinsky model with Van Driest damping near solid walls, the dynamic approach with a Smagorinsky base model, and others. LESOCC is highly vectorized and additionally parallelized by domain decomposition using MPI. The block structure builds the natural basis for grid partitioning. If required, the geometric block structure can be further subdivided into a parallel block structure in order to distribute the computational load to a number of processors (or nodes). Because the code was originally developed for high-performance vector computers such as CRAY, NEC or Fujitsu, it achieves high vectorization ratios (> 99.8%). In the context of vectorization, three diﬀerent types of loop structures have to be distinguished: • Loops running linearly over all internal control volumes in a grid block (3-D volume data) and exhibit no data dependencies. These loops are easy to vectorize, their loop length is much larger than the length of the vector registers and they run at high performance on all vector architectures. They show up in large parts of the code, e.g. in the calculation of the coeﬃcients and source terms of the linearized conservation equations. • The second class of loops occurs in the calculation of boundary conditions. Owing to the restriction to 2-D surface data, the vector length is shorter than for the first type of loops. However, no data dependence prevents the vectorization of this part of the code.

Project DIMPLE: DNS of Turbulent Flow Over Dimples

•

307

The most complicated loop structure occurs in the solver for the linear systems of equations in the implicit part of the code. Presently, we use the strongly implicit procedure (SIP) of Stone [12], a variant of the incomplete LU (ILU) factorization. All ILU type solvers of standard form are aﬀected by recursive references to matrix elements which would in general prevent vectorization. However, a well-known remedy for this problem exists. First, we have to introduce diagonal planes (hyper-planes, see Fig. 1) defined by i + j + k = constant, where i, j, and k are the grid indices. Based on these hyper-planes we can decompose the solution procedure for the whole domain into one loop over all control volumes in a hyper-plane, where the solution is dependent only on the values computed in the previous hyperplane and an outer do-loop over the imax + jmax + kmax − 8 hyper-planes.

Hyper-plane: i + j + k = const. Start of backward substitution

k

Start of Forward substitution i

j

Fig. 1. Hyper-plane concept for the vectorization of the SIP solver

4 Performance of LESOCC The most time-consuming part of the solution procedure is usually the implementation of the incompressibility constraint. As mentioned, in LESOCC the SIP-solver is used to solve the linear system of equations which appears in this part of the solution procedure. Profiling reveals that LESOCC spends typically 20–60% of the total runtime in the SIP-solver, depending on the actual flow problem and computer architecture. For that reason, we have established a benchmark kernel for the SIP-solver called SipBench [6], which contains the performance characteristics of the solver routine and is easy to analyze and modify. In order to test for memory bandwidth restrictions, we have also added an OpenMP parallelization to the diﬀerent architecture-specific implementations.

308

M. Breuer et al. 6000

NEC SX-8 (1 CPU) SGI Altix (1 Thread) SGI Altix (2 Threads) SGI Altix (4 Threads) SGI Altix (8 Threads) SGI Altix (16 Threads)

5000

MFlop/s

4000

3000 2000 1000 0

127

128

129

130

131

3000

2000

1000

0

0

100

200

300

domain size

Fig. 2. Performance of SipBench for diﬀerent (cubic) domains on SGI Altix using up to 16 threads and on NEC SX-8 (single CPU performance only)

In Fig. 2 performance numbers for the NEC SX-8 using a hyper-plane implementation are shown together with the performance of the SGI Altix which uses a pipeline-parallel implementation (cf. [7]) on up to 16 threads. On both machines we observe start-up eﬀects (vector pipeline or thread synchronization), yielding low performance on small domains and saturation at high performance on large domains. For the pipeline-parallel (SGI Altix) 3-D implementation a maximum performance of 1 GFlop/s can be estimated theoretically, if we assume that the available memory bandwidth of 6.4 GByte/s is the limiting factor and caches can hold at least two planes of the 3-D domain for the residual vector. Since two threads (sharing a single bus with 6.4 GByte/s bandwidth) come very close (800 MFlop/s) to this limit, we assume that our implementation is reasonably optimized and pipelining as well as latency eﬀects need not be further investigated for this report. For the NEC SX-8 we use a hyper-plane implementation of the SIP-solver. Compared to the 3-D implementation additional data transfer from main memory and indirect addressing is required. Ignoring the latter, a maximum performance of 6–7 GFlop/s can be expected on the NEC SX-8. As can be seen in Fig. 2, with a performance of roughly 3.5 GFlop/s the NEC system falls short of this expectation. Removing the indirect addressing, one can achieve up to 5 GFlop/s, however, at the cost of substantially lower performance for small/intermediate domain sizes or non-cubic domains. Since this is the application regime for our LESOCC benchmark scenario, we do not discuss the latter version in this report. The inset of Fig. 2 shows the performance impact of slight changes in the domain size. It reveals that solver performance can drop by a factor of 10 for specific memory access patterns, indicating severe memory bank conflicts.

Project DIMPLE: DNS of Turbulent Flow Over Dimples

309

The other parts of LESOCC perform significantly better. However, although the code was already vectorized before, two minor routines were found to decrease the overall performance dramatically (to about 50% of the final value). After optimizing these critical parts, the total single processor performance for a cubic plane channel flow scenario with 1303 grid points was lifted to 8.2 GFlop/s on the SX-8, i.e., more than half of the peak performance of the NEC SX-8. That is a typical value also found on percursor NEC machines or other vector architectures. Using the same executable we measured a performance of 4.8 GFlop/s on a single NEC SX-6+ processor, i.e., the SX-8 provides a speed-up of 1.71 which is in line with our expectations based on the pure hardware numbers (see Sect. 2). Applying the code in parallel (MPI) on 8 processors of one node for the DNS predictions described below, a total performance of about 52.7 GFlop/s was found which is equivalent to an average sustained performance per processor of about 6.6 GFlop/s. For our strong scaling parallel benchmark measurements, we have chosen a boundary layer flow over a flat plate with 11 × 106 grid points and focus on moderate CPU counts (6, 12 and 24 CPUs), where the domain decomposition for LESOCC can be reasonably done. For the run with 6 CPUs the domain was cut in wall-normal direction only; at 12 and 24 CPUs streamwise cuts have been introduced, lowering the communication-to-computation ratio. 150

NEC SX-8 SGI Altix

GFlop/s

100

50

0

0

5

10

15

20

25

30

Numbers of CPUs

Fig. 3. Speed-up (strong scaling) for a boundary layer ﬂow with 11×106 grid points up to 24 CPUs

The absolute parallel performance for the NEC SX-8 and the SGI Altix systems is depicted in Fig. 3. The parallel speed-up on the NEC machine is obviously not as perfect as on the Altix system. Mainly two eﬀects are responsible for this behavior. First, the baseline measurements with 6 CPUs were done in a single node on the NEC machine ignoring the eﬀect of com-

310

M. Breuer et al.

Table 1. Fraction of the SIP-solver and its performance in comparison with the overall performance. Data from the boundary layer set-up with 24 CPUs

Platform Intel Itanium 2 (1.6 GHz) NEC SX-8 (2 GHz) NEC SX-8 (2 GHz) NEC SX-8 (2 GHz)

Time SIP-solver L3 cache-hit LESOCC CPUs SIP-solver GFlop/s/ rate GFlop/s/ (%) CPU (%) CPU 24 25 0.39 97 0.73 6 12 24

31.5 32.9 33.6

3.25 2.83 2.5

— — —

7.02 6.6 5.2

munication over the IXS. Second, but probably more important, the single CPU performance (cf. Table 1) of the vector machine is almost an order of magnitude higher than on the Itanium 2 based system, which substantially increases the impact of communication on total performance due to strong scaling. A more detailed profiling of the code further reveals that also the performance of the SIP-solver is reduced with increasing CPU counts on the NEC machine due to reduced vector length (i.e. smaller domain size per CPU). The single CPU performance ratio between vector machine and cache based architecture is between 7 and 9.6. Note that we achieve a L3 cache hit ratio of roughly 97% (i.e. each data element loaded from main memory to cache can be reused at least once from cache), which is substantially higher than for purely memory-bounded applications.

5 Flow Predictions 5.1 Description of the Conﬁgurations In order to investigate the eﬀect of dimples, a classical wall-bounded flow often studied in literature is considered, i.e., a turbulent plane channel flow at Re = Ub δ/ν = 10, 935 or Reτ = 590. This test case has several advantages. With smooth non-dimpled walls the flow is homogeneous in streamwise and spanwise direction. That issue allows to apply periodic boundary conditions in both directions and thus avoids the definition of appropriate inflow and outflow boundary conditions. The pressure gradient in streamwise direction is adjusted such that a fixed mass flow rate (Ub = const.) is assured. Furthermore, no-slip boundary conditions are used at both walls. Since the simulation cannot cover the entire experimental set-up [8], the computational domain consists of a cutout of the channel as sketched in Fig. 4 allowing to apply periodic boundary conditions in streamwise and spanwise direction. The extensions of the domain are 4.1744 δ × 2 δ × 2.4096 δ, while all

Project DIMPLE: DNS of Turbulent Flow Over Dimples

311

Fig. 4. Geometric conﬁguration of the channel ﬂow with multiple shallow dimples at the lower wall

geometrical quantities are scaled by the channel half-width δ. Three cases are considered: L: Plane channel with multiple dimples at the Lower wall B: Plane channel with multiple dimples at Both walls R: Plane channel without dimples, i.e., smooth walls as Reference case As shown in Fig. 4(a) multiple shallow dimples (depth to print diameter of h/D = 0.05; D/δ = 0.6) are arranged regularly on the surface of the lower and/or upper channel walls. Using block-structured curvilinear grids, two diﬀerent grid resolutions are taken into account for the DNS predictions, see Table 2. The first denoted coarse grid consists of about 5.5 million control volumes (CVs) and thus is not really coarse. For the plane channel case the grid is Cartesian with equidistant grid spacing in streamwise and spanwise directions and a stretched distribution towards both walls according to a geometric series. However, the expansion factor of the geometric series is mild (r = 1.04) and the first node is located at a dimensionless distance of + Δymin = 0.587. Hence several grid points are found in the viscous sublayer Table 2. Overview on simulation parameters for the ﬁve simulations carried out Parameter Resolution (# CVs) Δymin /δ expansion ratio r in y-direct. Δx+ + Δymin + Δz Δt Cases considered

Coarse Grid 260 × 160 × 132 10−3 1.040 9.58 0.587 11.05 4.2 × 10−3 L, B, R

Fine Grid 516 × 320 × 260 5 × 10−4 1.023 4.79 0.293 5.52 1.6 × 10−3 B, R

312

M. Breuer et al.

and the resolution in the homogeneous direction expressed in wall coordinates is of the order O(10). For the cases with dimples the grid is clinging to the curvilinear geometry of the walls and thus locally no longer Cartesian. For the second grid denoted fine grid, the number of control volumes is exactly doubled in each direction for the computational domain sketched in Fig. 4, but due to some extra cells required for the periodic boundary conditions, the numbers given in Table 2 are not exactly doubled. In total about 43 million CVs are used and the resolution in each direction is halved as vis+ ible from the values for Δx+ , Δymin , and Δz + summarized in Table 2. The dimensionless time step sizes are Δt = 4.2 × 10−3 and Δt = 1.6 × 10−3 for the coarse and fine grid, respectively. To achieve reliable statistical data, the flow is averaged over dimensionless time intervals of up to Tavg = 17, 000 which is equivalent to about 4000 flow-through times. 5.2 Results and Discussion Figure 5 depicts results obtained for the case L using a regular arrangement of shallow dimples as shown in Fig. 4 at the lower wall only. The time-averaged pressure distribution is shown in Fig. 5(a). The influence of the dimples on the pressure distribution at the lower wall can clearly be seen. It has to be mentioned here that the linear pressure gradient in the streamwise direction of the channel is not included in the figure. In Fig. 5(b) the time-averaged wall shear stress distribution is displayed. Obviously the wall shear stresses are decreasing within the dimples, leading to a small recirculation region at the falling edge (see Fig. 6). However, at the border of the dimples where the fluid flow is leaving the troughs again, large values of the wall shear stress are found. In Fig. 6 the wall streamlines at the lower walls are displayed together with the distribution of the time-averaged spanwise (w) and streamwise (u) velocities close to the wall. In front of the dimple and partially also in the dimple the streamlines are converging. A tiny recirculation region with negative values of u visible in Fig. 6(b) is found inside the dimple (coarse grid result, but confirmed by fine grid results). At the side and past the recirculation region the streamlines are diverging again. Figures 7(a)–(b) display the time-averaged flow in a x-y midplane of one dimple. Based on the velocity vectors the reduction of the velocity gradient near the wall and thus of the wall shear stress is visible. On the other hand it is obvious that the structure of the wall leads to a modified pressure distribution compared with the case of a smooth wall. At both borders of the dimple when the flow is either entering or leaving the trough, the pressure is slightly decreasing. However, more important is the observation that the pressure is increasing on the rising edge of the dimple yielding a contribution to the overall drag resistance as will be shown below. The strongest flow structures

Project DIMPLE: DNS of Turbulent Flow Over Dimples

313

Fig. 5. Channel ﬂow with multiple dimples at the lower wall, case L, Re = 10, 935

in the time-averaged flow visualized by iso-surfaces of λ2 = −0.05 are located at the borders of the dimple where the flow is entering or leaving the dimple. Finally, Fig. 8 depicts time histories of the forces acting on the lower dimpled wall (south) and on the non-dimpled smooth wall (north). For the upper wall the situation is simple since on this surface only the wall shear stress leads to a skin friction drag. This force also acts on the lower dimpled wall where the time-average is marginally smaller on the dimpled wall than on the smooth wall. However, at the lower wall the pressure distribution on the wavy surface (Fig. 7(b)) yields a pressure force in the main flow direction (about 5% of total force) which contributes to the overall drag force. Both eﬀects, the slight decrease of the shear force in average and the additional

314

M. Breuer et al.

Fig. 6. Channel ﬂow with multiple dimples at the lower wall, case L, Re = 10, 935

contribution of the pressure force approximately compensate each other so that no net gain for drag reduction due to dimples remains. Presently, the total drag resistance even increases about 1.9% or 3.8% compared to the reference case R for the cases L and B (coarse grid results), respectively. Hence based on these simulations (and corresponding measurements [8]), the question whether dimples lead to drag reduction or not, cannot be finally answered. For that purpose, the results using the fine grid presently running on NEC SX-8 and additional boundary layer measurements integrated for momentum loss [8] remain to be seen. However, although statistically not fully converged, the fine grid simulations seem to confirm the trend observed on the coarse grid.

Project DIMPLE: DNS of Turbulent Flow Over Dimples

315

0.2 0.15

y

0.1 0.05 0 2.3

2.4

x

2.5

2.6

(a) Zoom of the dimple region in a x–y midplane 0.6 0.034 0.026 0.018 0.01 0.002 -0.006 -0.014

y

0.4

0.2

0 2

2.2

2.4

2.6

x

2.8

3

3.2

(b) Contours of p in the dimple in a x–y midplane

(c) Iso-surfaces of λ2 = −0.05 Fig. 7. Channel ﬂow with multiple dimples at the lower wall, case L, Re = 10, 935

316

M. Breuer et al.

Fig. 8. Channel ﬂow with multiple dimples at the lower wall, case L, Re = 10.935

6 Conclusions Using the finite-volume CFD code LESOCC we have demonstrated that the latest NEC SX-8 vector computer generation provides unmatched performance levels for applications which are data and computationally intensive. Another striking feature of the NEC vector series has also been clearly demonstrated: Going from the predecessor vector technology (SX-6+) to the SX-8, we found a performance improvement of about 1.71 which is roughly the same as the ratio of the peak performance numbers (1.77). To comment on the long standing discussion about the success of cache based microprocessors, we have compared the NEC results with the SGI Altix system, being one of the best performing microprocessor systems for the applications under review here (i.e., peak performance ratio of NEC SX-8 vs. SGI Altix [1.6 GHz]= 2.5). We found that the sustained per-processorperformance is in average almost one order of magnitude higher for the vector

Project DIMPLE: DNS of Turbulent Flow Over Dimples

317

machine (i.e., sustained LESOCC performance ratio of NEC SX-8 vs. SGI Altix [1.6 GHz]= 7.5–9.0), clearly demonstrating that vector architectures still provide a class of their own if application performance for vectorizable problems is the measure. The extremely good single processor performance does not force the scientist to scale their codes and problems to thousands of processors in order to reach the teraflop regime. Furthermore, it should be emphasized that there has been a continuity of the basic principles of vector processor architectures for more than 20 years. This has provided highly optimized applications and solid experience in vector processor code tuning. Thus, the eﬀort to benefit from technology advancements is minimal from a user’s perspective. For the microprocessors, on the other hand, we suﬀer from a lack of continuity even on much smaller time scales. In the past years we have seen the rise of a completely new architecture (Intel Itanium). With the introduction of dual-/multi-core processors a new substantial change is just ahead, raising the question whether existing applications and conventional programming approaches are able to transfer the technological advancements of these machines to application performance. The application studied within this project is the potential drag reduction by a regular arrangement of shallow dimples. However, up to now only a minor reduction of the skin friction drag was found by DNS predictions which is compensated by an additional pressure drag contribution on the wavy wall so that no net gain for drag reduction due to dimples remains. Further investigations including measurements are required to figure out the reasons. Acknowledgements This work is financially supported by the Deutsche Forschungsgemeinschaft under contract number BR 1847/9. The computations were carried out on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (grant no.: DIMPLE/12760) and on the SGI Altix machines at RRZE Erlangen, LRZ M¨ unchen, and CSAR Manchester, which are gratefully acknowledged.

References 1. V.V. Alekseev, I.A. Gachechiladze, G.I. Kiknadze, and V.G. Oleinikov. Tornadolike energy transfer on three-dimensional concavities of reliefs-structure of self-organizing ﬂow, their visualisation, and surface streamlining mechanisms. In Transactions of the Second Russian Nat. Conf. of Heat Transfer, vol. 6, Heat Transfer Intensification Radiation and Complex Heat Transfer, Publishing House of Moscow Energy Institute (MEI), Moscow, pages 33–42, 1998. 2. M. Breuer. Large-eddy simulation of the sub-critical ﬂow past a circular cylinder: Numerical and modeling aspects. Int. J. for Numer. Methods in Fluids, 28:1281– 1302, 1998.

318

M. Breuer et al.

3. M. Breuer. A challenging test case for large-eddy simulation: High Reynolds number circular cylinder ﬂow. Int. J. of Heat and Fluid Flow, 21(5):648–654, 2000. 4. M. Breuer. Direkte Numerische Simulation und Large-Eddy Simulation turbulenter Str¨ omungen auf Hochleistungsrechnern. Berichte aus der Str¨ omungstechnik, Habilitationsschrift, Universit¨ at Erlangen–N¨ urnberg, Shaker Verlag, Aachen, 2002. ISBN: 3-8265-9958-6. 5. M. Breuer and W. Rodi. Large-eddy simulation of complex turbulent ﬂows of practical interest. In E.H. Hirschel, editor, Flow Simulation with HighPerformance Computers II, volume 52, pages 258–274. Vieweg Verlag, Braunschweig, 1996. 6. F. Deserno, G. Hager, F. Brechtefeld, and G. Wellein. Basic Optimization Strategies for CFD-Codes. Technical report, Regionales Rechenzentrum Erlangen, 2002. 7. F. Deserno, G. Hager, F. Brechtefeld, and G. Wellein. Performance of scientiﬁc applications on modern supercomputers. In S. Wagner, W. Hanke, A. Bode, and F. Durst, editors, High Performance Computing in Science and Engineering, Munich 2004. Transactions of the Second Joint HLRB and KONWIHR Result and Reviewing Workshop, March 2nd and 3rd, 2004, Technical University of Munich, pages 3–25. Springer Verlag, 2004. 8. H. Lienhart, M. Breuer, and C. K¨ oksoy. Drag reduction by dimples? – a complementary experimental/numerical investigation –. In R. Friedrich et al., editor, Fifth Int. Symposium on Turbulence and Shear Flow Phenomena, August 27-29, 2007, Garching, Germany, 2007. 9. L. Oliker, J. Carter A. Canning, J. Shalf, and S. Ethier. Scientiﬁc computations on modern parallel vector systems. In Proceedings of SC2004, CD-ROM, 2004. 10. L. Oliker, J. Carter A. Canning, J. Shalf, D. Skinner, S. Ethier, R. Biswas, J. Djomehri, and R. V. d. Wijngaart. Evaluation of cache-based superscalar and cacheless vector architectures for scientiﬁc computations. In Proceedings of SC2003, CD-ROM, 2003. 11. T. Pohl, F. Deserno, N. Th¨ urey, U. R¨ ude, P. Lammers, G. Wellein, and T. Zeiser. Performance evaluation of parallel large-scale lattice Boltzmann applications on three supercomputing architectures. In Proceedings of SC2004, CD-ROM, 2004. 12. H.L. Stone. Iterative solution of implicit approximations of multidimensional partial diﬀerential equations. SIAM J. Num. Anal., 91:530–558, 1968.

Direct Numerical Simulation of a Serrated Nozzle End for Jet-Noise Reduction A. Babucke, M. Kloker, and U. Rist Institut f¨ ur Aerodynamik und Gasdynamik, Universit¨at Stuttgart, Pfaﬀenwaldring 21, D-70550 Stuttgart, Germany [email protected] Summary. Sound generation of a subsonic laminar jet has been investigated using direct numerical simulation (DNS). The simulation includes the nozzle end, modelled by a ﬁnite ﬂat splitter plate with Mach numbers of MaI = 0.8 above and MaII = 0.2 below the plate. Behind the nozzle end, a combination of wake and mixing layer develops. Due to its instability, roll up and pairing of spanwise vortices occur, with the vortex pairing being the major acoustic source. As a ﬁrst approach for noise reduction, a rectangular notch at the trailing edge is investigated. It generates longitudinal vortices and a spanwise deformation of the ﬂow downstream of the nozzle end. This leads to a an early breakdown of the large spanwise vortices and accumulations of small-scale structures. Compared to a two-dimensional simulation performed earlier [3], the emitted sound is reduced by 6 dB.

1 Introduction Noise reduction is of special interest for many technical problems, as high acoustic loads lead to a reduced quality of life and may cause stress for persons concerned permanently. The current investigation focuses on jet noise as it is a major noise source of aircrafts. As the major airports are typically located in highly populated areas, noise reduction would improve the situation of many people. Direct aeroacoustic simulations are a relatively new field in computational fluid dynamics, facing several diﬃculties due to largely diﬀerent scales. The hydrodynamic fluctuations are small-scale structures containing high energy compared to the acoustics with relatively long wavelengths and small amplitudes. Therefore, high resolution is required to compute the noise sources accurately. On the other hand a large computational domain is necessary to obtain the relevant portions of the acoustic far-field. Due to the small amplitudes of the emitted noise, boundary conditions have to be chosen carefully, in order not to spoil the acoustic field with reflections. Up to now, large-eddy or direct numerical simulations of jet noise have been focusing on either pure mixing layers [2, 5, 7] or low Reynolds number jets [9], where an S-shaped velocity profile is prescribed at the inflow. Our

320

A. Babucke, M. Kloker, U. Rist

approach is to include the nozzle end, modelled by a thin finite flat plate with two diﬀerent free-stream velocities above and below. Including the nozzle end shifts the problem to a more realistic configuration, leading to a combination of wake and mixing layer behind the splitter plate. Additionally, wall-bounded actuators for noise reduction can be tested without the constraint to model them by artifical volume forces. In the current investigation, a passive ’actuator’ is considered as a first realistic approach for noise reduction.

2 Numerical Method 2.1 Simulation Framework Obtaining DNS results does not only require the simulation itself, it also includes pre- and postprocessing as well as stability analysis of the considered baseflow. By far, most computer resources are acquired by the DNS code. Nevertheless the user spends a lot of time in setup of the problem and analysis of the computed data. Therefore a complete framework of tools with a common structure of in- and outputfiles, based on the EAS3 framework [8], has been developed. This does not only lead to a more eﬃcient work for the scientist himself, it also provides reproducibility of the complete set of results. The typical workflow for a DNS is illustrated in Fig. 1. The first step is the definition of the problem itself. Here, the flow conditions, the grid and the domain decomposition are specified, providing initial data for the DNS. This process strongly depends on the problem to be investigated, resulting in specific tools. A typical approach is to use the solution of the boundary-layer equations and interpolate them on the grid. If the problem allows to obtain Initial Condition

DNS

Linear Stability Theory

DNS

Postprocessing

grid, flow conditions (e.g. boundary layer equations)

converged baseflow

eigenfunctions, amplification rates

unsteady flow−field

amplitudes, vortices, acoustics, ...

Fig. 1. Workﬂow for DNS including pre- and postprocessing as well as stability analysis

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

321

a steady-state solution, the DNS code may be used here to obtain a converged solution of the Navier-Stokes equations. The initial condition or a converged solution obtained from the previous step can be used in linear stability theory providing amplification rates and eigenfunctions. This information is used to define disturbance generation for the actual simulation performed by the DNS code. The resulting output is raw binary data with the flow variables given on the computational grid for several time steps. To get a better understanding of the flow physics, postprocessing is used to compute e.g. spectra and amplitudes or vortex criteria. As the initial condition requires relatively few computations, it is run interactively on the Itanium frontend machine of the NEC-SX8 vector computer. Linear stability theory and the DNS code are executed on the SX8 vector computer. For both codes, the usability has been improved by startscripts running on the frontend machine. These scripts gather all input files, compile the code with the required array sizes, create the jobfile and submit the actual job to the queueing system of the vector computer. Additionally the input files and the source code are archived in the output directory. Thus, the whole computation can be reproduced easily. Postprocessing is done on the frontend machine and consists mainly of a collection of shell-scripts which can be selected via a common input file. These scripts basically trigger the EAS3 command line interface [8]. If an operation needs to be done for multiple files, multiple entities of EAS3 are started, simply by adding an ‘&’ to its call in the script. Due to the common file system of the frontend and the vector machine, no unnecessary copying of data is required. 2.2 DNS Code The main part of this framework is the DNS code NS3D, solving the full three-dimensional unsteady compressible Navier-Stokes equations on multiple domains. The purpose of domain decomposition is not only to increase computational performance. The combination with grid transformation and the concept of modular boundary conditions allows to compute a wide range of problems. Computation is done in non-dimensional quantities: velocities are normalised by the reference velocity U ∞ and all other quantities by their inflow values, marked with the subscript ∞ . Length scales are made dimensionless with a reference length L and the time t with L/U ∞ , where the overbar denotes dimensional values. Temperature dependence of viscosity μ is modelled using the Sutherland law: μ(T ) = μ(T ∞ ) · T 3/2 ·

1 + Ts , T + Ts

(1)

where Ts = 110.4 K/T ∞ and μ(T ∞ = 280 K) = 1.735 · 10−5 kg/(ms). Thermal conductivity ϑ is obtained by assuming a constant Prandtl number P r = cp μ/ϑ. The most characteristic parameters describing a compressible viscous

322

A. Babucke, M. Kloker, U. Rist

1111111111111111 0000000000000000 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111 0000000000000000 1111111111111111

y x z

λ z,0 Fig. 2. Integration domain for jet noise computation with splitter plate and sponge zone

flow-field are the Mach number Ma = U∞ /c∞ and the Reynolds number Re = ρ∞ U∞ L/μ∞ . We use the conservative formulation of the Navier-Stokes equations which results in the solution vector Q = [ρ, ρu, ρv, ρw, E] containing the density, the three momentum densities and the total energy per volume E = ρ · cv · T +

ρ 2 · u + v 2 + w2 . 2

(2)

The complete set of equations is given in [4]. The simulation is carried out in a rectangular domain with x, y, z being the coordinates in streamwise, normal and spanwise direction, respectively. A typical setup for jet noise computation is shown in Fig. 2. Since the flow is assumed to be periodic in spanwise direction a spectral discretization in z-direction is used: f (x, y, z, t) =

K

k=−K

Fˆk (x, y, t) · ei(kγ0 z) .

(3)

f denotes any flow variable, Fˆ√ k its complex Fourier coeﬃcient, K the number of spanwise modes and i = −1. The fundamental spanwise wavenumber γ0 is given by the fundamental wavelength λz,0 representing the width of the integration domain by γ0 = 2π/λz,0 . Spanwise derivatives are computed by transforming the respective variable into Fourier space, multiplying its spectral components with their wavenumbers (i · k · γ0 ) for the first derivatives or square of their wavenumbers for the second derivatives and transforming them back into physical space. Due to

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

323

the non-linear terms in the Navier-Stokes equations, higher harmonic spectral modes are generated at each time step. To suppress aliasing, only 2/3 of the maximum number of modes for a specific z-resolution are used [6]. If a two-dimensional baseflow is used and disturbances of u, v, ρ, T , p are symmetric and disturbances of w are antisymmetric, flow variables are symmetric/antisymmetric with respect to z = 0. Therefore only half the number of points in spanwise direction are needed (0 ≤ z ≤ λz /2) and (3) is transferred to f (x, y, z, t) = F0r (x, y, z, t) +2 ·

K

Fkr (x, y, t) · cos (kγ0 z)

K

Fki (x, y, t) · sin (kγ0 z)

k=1

(4)

for f ∈ [u, v, ρ, T, p] f (x, y, z, t)

−2 ·

=

k=1

(5)

for f ∈ [w] . The spatial discretization in streamwise (x) and normal (y) direction is done by 6th -order compact finite diﬀerences. The tridiagonal equation systems of the compact finite diﬀerences are solved using the Thomas algorithm. To reduce the aliasing error, alternating up- and downwind-biased finite differences are used for convective terms as proposed by Kloker [11]. The second derivatives are evaluated directly which distinctly better resolves the second derivatives compared to applying the first derivative twice. The square of the ∗ over k ∗ is shown for diﬀerent schemes in Fig. 3. modified wavenumber kmod The exact solution for the second derivative is k ∗2 . While the relative error of the second derivatives is less than 1% up to a modified wavenumber of k ∗ = 1.00 or k ∗ = 1.74 for 4th and 6th order, respectively, the result of computing the first derivative twice is only good up to k ∗ = 0.62 and k ∗ = 1.42 accordingly. Due to that, our scheme provides a resolution advantage of factor 3 for the viscous terms compared to a standard scheme of 4th order, often

exact 2nd derivative CFD O6 2nd derivative FD O4 2x 1st derivative CFD O6 2x 1st derivative FD O4

k*2mod

8 6 4 2 0

0

0.5

1

1.5

k*

2

2.5

3

Fig. 3. Comparison of second derivative versus twice the ﬁrst derivative for a wave with wave number k∗ = k · Δx

324

A. Babucke, M. Kloker, U. Rist

used for compressible flows. Moreover, another advantage of direct second∗2 derivatives computation is the fact, that kmod does not vanish for the least ∗ resolved waves with k = π providing better accuracy and stability of the code. Arbitrary grid transformation in the x-y plane is provided by mapping the physical grid on an equidistant computational ξ-η grid: x = x(ξ, η) , y = y(ξ, η) . According to [1], the first derivatives can be computed as: 1 ∂ ∂y ∂ ∂y ∂ − = ∂x J ∂ξ ∂η ∂η ∂ξ 1 ∂ ∂ ∂x ∂ ∂x = − ∂y J ∂η ∂ξ ∂ξ ∂η ∂x ∂y ∂x ∂y ∂y ∂x ∂ξ ∂ξ · − · J = ∂x = ∂η ∂y ∂ξ ∂η ∂ξ ∂η ∂η

(6)

(7) (8) (9)

with the metric coeﬃcients (∂x/∂ξ), (∂y/∂ξ), (∂x/∂η), (∂y/∂η) and J being the determinant of the Jacobi matrix. To compute second spatial derivatives, equations (7) and (8) are applied twice. Here one has to take into account that the metric coeﬃcients and by that also the Jacobi determinant are a function of ξ and η as well. Time integration of the Navier-Stokes equations is done using the classical 4th-order Runge–Kutta scheme as described in [11]. At each time step and each intermediate level the biasing of the finite diﬀerences for the convective terms is changed. The ability to perform computations not only in total value but also in disturbance formulation is provided by subtracting the spatial operator of the baseflow from the time derivatives of the conservative variables Q. 2.3 Parallelization To use the full potential of the NEC-SX8 vector computer at HLRS, we have chosen a hybrid parallelization of both MPI and Microtasking. As shared memory parallelization, Microtasking is used along the spanwise direction. The second branch of the parallelization is domain decomposition using MPI. Due to the fact that the Fourier transformation requires data over the whole spanwise direction, a domain decomposition in z would have caused high communication overhead. Therefore domain decomposition is applied only in the ξ-η plane. At each boundary (left, right, top, bottom) of a domain, one can specify the neighbour or one of the implemented boundary conditions. The explicit formulation of the boundary conditions allows to easily implement new boundary conditions or modifications of them. As the domain decomposition must not influence the result, the compact finite diﬀerences are used

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

325

in the whole computational domain. Solving the resulting tridiagonal equation system is the crucial part of the parallelization concept. The Thomas algorithm requires two recursive loops (forward and backward) [4]. Therefore each MPI process has to wait until the previous neighbour has finished its part of the recursive loop. An ad-hoc implementation would result in a serialisation of the Thomas algorithm. To avoid that, we make use of the fact that we have to compute not only one but up to 25 spatial derivatives depending on the spatial direction. The procedure is implemented as follows: the first domain starts with the forward-loop of derivative one. After its completion, the second domain continues the computation of derivate one while the first domain starts to evaluate derivative number two simultaneously. For the following steps, the algorithm continues accordingly. The resulting pipelining is shown exemplary for the forward-loop in Fig. 4, the backward loop works in the opposite direction, accordingly. If communication time is neglected, the theoretical speedup for forwardand backward-loop is expressed by: speedup =

m·n m+n−1

(10)

with n being the number of domains in a row or column, respectively, and m the number of equations to be solved. Theoretical speedup and eﬃciency of the pipelined Thomas algorithm are shown in Fig. 5 for 25 equations as a function of the number of domains. For 30 domains, eﬃciency of the algorithm decreases to less than 50 percent. Note that all other computations, e.g. Fourier transformation, Navier-Stokes equations and time integration, are local for each MPI process. Therefore the eﬃciency of the pipelined Thomas algorithm does not aﬀect the speedup of the entire code that severely. The alternative to the current scheme would be an iterative solution of the equation system. The advantage would be to have no dead times, but quite a number of iterations would be necessary for a converged solution. This would result in higher CPU time up to a moderate number of domains. As shared memory parallelization is implemented additionally, the number of domains corresponds to the number of nodes and therefore only a moderate number of domains will be used.

Fig. 4. Illustration of pipelining showing the forward-loop for three spatial derivatives on three domains. Green colour is denoted to computation, red to communication and grey colour shows dead time

A. Babucke, M. Kloker, U. Rist

13 12 11 10 9 8 7 6 5 4 3 2 1

Fig. 5. Theoretical speedup and eﬃciency of the pipelined Thomas algorithm versus number of domains n for 25 equations

1 0.8 0.6 0.4 speedup thomas alg. ideal speedup efficiency

1

5

10

15

n

20

25

efficiency

speedup

326

0.2 0 30

2.4 Boundary Conditions At the borders of each domain where no neighbour exists, one can select a specific boundary condition. Up to now a variety of boundary conditions have been implemented. According to their properties, the code knows where time integration has to be done and where values are prescribed. This allows to easily implement new boundary conditions or modifications of existing ones, e.g. for disturbance generation. For the jet-noise investigation, we use a one-dimensional characteristic boundary condition [10] at the freestream. This allows outward-propagating acoustic waves to leave the domain. An additional damping zone forces the flow variables smoothly to a steady state solution, avoiding reflections due to oblique waves. Having a subsonic flow, we also use a characteristic boundary condition at the inflow, allowing upstream propagating acoustic waves to leave the domain. Additionally amplitude and phase distributions from linear stability theory (see 2.6) can be prescribed to introduce defined disturbances. The outflow is the most crucial part as one has to avoid large structures passing the boundary and contaminating the acoustic field. Therefore, a combination of grid stretching and spatial low-pass filtering is applied in the sponge region. Disturbances become increasingly badly resolved as they propagate through the sponge region. As the spatial filter depends on the step size in x-direction, perturbations are smoothly dissipated before they reach the outflow boundary. This procedure shows very low reflections and has been already applied by Colonius et al. [7]. For the splitter plate representing the nozzle end, an isothermal boundary condition is used with the wall temperature being fixed to its value from the initial condition. The pressure is obtained by extrapolation from the interior gridpoints. An extension of the wall boundary condition is the modified trailing edge, where the end of the splitter plate is no more constant along the spanwise direction. As we have grid transformation only in the x-y plane and not in z-direction, the spanwise dependency of the trailing edge is achieved by modifying the connectivity of the aﬀected domains. Instead of regularly prescribing the wall boundary condition along the whole border of the re-

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

327

spective subdomain, we can also define a region without wall, now. At these gridpoints, the spatial derivatives in normal direction are recomputed, now using also values from the domain on the other side of the splitter plate. The spanwise derivatives are computed in the same manner as inside the flowfield with the Fourier-transformation being applied along the whole spanwise extent of the domain. The concept of modular boundary conditions, chosen because of flexibility and maintainability, requires explicit boundary conditions and by that a non-compact finite-diﬀerence scheme, here. Therefore explicit finite diﬀerences have been developed with properties quite similar to the compact scheme used in the rest of the domain. The numerical properties of the chosen 8th-order scheme are compared with standard explicit 6th-order finite diﬀerences and the compact scheme of 6th order, regularly used in the flowfield. For the first derivative, the real and imaginary parts of the modified ∗ wavenumber kmod are shown in Fig. 6: the increase from order six to eight does not fully reach the good dispersion relation of the 6th-order compact scheme ∗ by 10% compared with an ad but at least increases the maximum of kmod hoc explicit 6th-order implementation. The imaginary part of the modified wavenumber, responsible for dissipation, shows similar characteristics as the compact scheme with the same maximum as for the rest of the domain. Also for the second derivative, shown by the square of the modified wavenumber ∗2 kmod in Fig. 7, the increase of its order improves the properties of the explicit finite diﬀerence towards the compact scheme.

exact Re(k*mod) FD-O8 Re(k*mod) CFD-O6 Re(k*mod) FD-O6 Im(k* mod) FD-O8 Im(k* mod) CFD-O6

exact 2 Re(k* mod) FD O8 2 Re(k* mod) CFD O6 Re(k* 2mod) FD O6

8

2

k* 2mod

k*mod

3

6 4

1

2 0

0

0.5

1

1.5

k*

2

2.5

3

Fig. 6. Real and imaginary part of ∗ the modiﬁed wavenumber kmod for the ﬁrst derivative based on a wave with wave number k∗ = k · Δx. Comparison of 8th-order explicit ﬁnite diﬀerence with 6th-order explicit and compact scheme

0

0

0.5

1

1.5

k*

2

2.5

3

Fig. 7. Square of the modiﬁed wavenumber of the second derivative for a wave with wave number k∗ = k · Δx. Comparison of 8th-order explicit ﬁnite diﬀerence with 6th-order explicit and compact scheme

328

A. Babucke, M. Kloker, U. Rist

2.5 Initial Condition For the current investigation, an isothermal laminar subsonic jet with the Mach numbers MaI = 0.8 for the upper and MaII = 0.2 for the lower stream has been selected. As both temperatures are equal (TI = TII = 280K), the ratio of the streamwise velocities is UI /UII = 4. This large factor leads to strong instabilities behind the nozzle end, so a moderate number of grid points in x-direction is suﬃcient to simulate the aeroacoustic source. The Reynolds number Re = ρ∞ UI δ1,I /μ∞ = 1000 is based on the displacement thickness δ1,I of the upper stream at the inflow. With δ1,I (x0 ) = 1, length scales are normalized with the displacement thickness of the fast stream at the inflow. The boundary layer of the lower stream corresponds to the same origin of the flat plate. The cartesian grid is decomposed into sixteen subdomains as illustrated in Fig. 8: eight in streamwise and two in normal direction. Each subdomain contains 325×425×65 points in x-, y- and z-direction, resulting in 42 spanwise modes (dealiased) and a total number of 143.6 million gridpoints. The mesh is uniform in streamwise direction with a step size of Δx = 0.15 up to the sponge region, where the grid is highly stretched. In normal direction, the finest step size is Δy = 0.15 in the middle of the domain with a continuous stretching up to a spacing of Δy = 1.06 at the upper and lower boundaries. In spanwise direction, the grid is uniform with a spacing of Δz = 0.2454 which is

0.4

y

0.2 0

-0.2 -0.4

100

-0.6

y

-0.5

x0

0.5

0

-100

0

100

x

200

300

Fig. 8. Grid in x-y plane showing every 25th gridline and illustrating the domain decomposition. Insert: implementation of the splitter plate at the borders of the corresponding domain

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

329

equivalent to a spanwise wavenumber γ0 = 0.2, where λz /2 = π/γ0 = 15.708 is the spanwise extent of the domain. The origin of the coordinate system (x = 0, y = 0) is located at the end of the nozzle. The nozzle end itself is modeled by a finite thin flat plate with a thickness of Δy. Due to the vanishing thickness of the nozzle end, an isothermal boundary condition at the wall has been chosen. The temperature of the plate is T wall = 296 K, being the mean value of the adiabatic wall temperatures of the two streams. The initial condition along the flat plate is obtained from similarity solutions of the boundary-layer equations. Further downstream, the full boundarylayer equations are integrated downstream, providing a flow-field suﬃcient to serve as an initial condition and for linear stability theory. The resulting streamwise velocity profiles of the initial condition are shown in Figs. 9 and 10. Behind the nozzle end, the flow field keeps its wake-like shape for a long range. As high amplification rates occur here, the flow is already unsteady before a pure mixing layer has developed. This means that the pure mixing layer investigated earlier [2, 5, 7] has to be considered as a rather theoretical approach.

15

15

10

10 x = -97.5 Reδ1 = 1000

5

0

y

y

5

-5

-5

-10

-10

-15

0

0.2

0.4

u

0.6

0.8

1

Fig. 9. Proﬁle of the streamwise velocity u for the upper and lower boundary layer at the inﬂow

x = 0 ... 96.75

0

-15

0

0.2

0.4

u

0.6

0.8

1

Fig. 10. Downstream evolution of the streamwise velocity proﬁle behind the nozzle end

2.6 Linear Stability Theory Spatial linear stability theory (LST) [12] is based on the linearisation of the Navier-Stokes equations, split into a steady two-dimensional baseflow and wavelike disturbances Φ = Φˆ(y) · ei(αx+γz−ωt) + c.c.

(11)

with Φ = (u′ , v ′ , w′ , ρ′ , T ′ , p′ ) representing the set of fluctuations of the primitive variables. As only first derivatives in time occur, the temporal problem, where the streamwise wavenumber α = αr is prescribed, is solved first by a 4th-order matrix solver providing the complex eigenvalues (ωr , ωi ), with ωr

330

A. Babucke, M. Kloker, U. Rist

-0.01 -0.2 -0.1

αi .5

ω0 = 0.0688

x = 13.35

0.1

5

ωr

x = 1.35

0

-9 7

-0.1

0.05

x=

0.01

x=

αi

0

ω0 = 0.0688

being the frequency and ωi the temporal amplification. Once an amplified eigenvalue is found, the Wielandt iteration iterates the temporal to the spatial problem by varying the spatial amplification −(αi ) such that ωi = 0. This can also be done for a range of streamwise wavenumbers αr and x positions to obtain a stability diagram. A selected spatial eigenvalue (αr , αi ) can be fed into the matrix solver to obatin the eigenfunction, being the amplitude and phase distribution of the primitive variables along y. The eigenfunctions can be used directly in the DNS-code for disturbance generation at the inflow. As the flow is highly unsteady behind the nozzle end and enforcing an artificial steady state does not work properly, we use the initial condition derived from the boundary-layer equations to compute eigenvalues and eigenfunctions. According to Fig. 11, a fundamental angular frequency of ω = 0.0688 was chosen for the upper boundary layer. The amplification keeps almost constant in downstream direction. As the two boundary layers emerge from the same position, the lower boundary layer is stable up to the nozzle end. Behind the edge of the splitter plate, amplification rates 50 times higher than in the upper boundary layer occur due to the inflection points of the streamwise-velocity profile. Maximum amplification in the mixing layer takes place for a frequency of roughly three to four times of the fundamental frequency of the boundary layer as illustrated in Fig. 12.

0.1

0.15

Fig. 11. Ampliﬁcation rates of the upper (fast) boundary layer given by linear stability theory for various xpositions

0.2

0.1

ωr0.2

0.3

Fig. 12. Ampliﬁcation rates for various x-positions behind the splitter plate predicted by linear stability theory.

3 Numerical Results For the pure mixing layer without splitter plate [2], we already found that introducing a steady longitudinal vortex leads to a break-up of the big spanwise vortices and may reduce the emitted sound originating from vortex pairing. A variety of wall-mounted actuators are cogitable for the generation of streamwise vortices, our approach is to engrail the trailing edge of the splitter plate. Here, a rectangular spanwise profile of one notch per spanwise wavelength

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

331

Fig. 13. Perspective view of the engrailed trailing edge and the vortical structures in the instantaneous ﬂow ﬁeld, visualised by the isosurface λ2 = −0.005. The distance from the plane of the splitter plate (y = 0) is coloured from blue to red

with a depth of 10 in x-direction has been chosen as a first choice. At the inflow of the upper boundary layer, the flow is disturbed with the TollmienSchlichting (TS) wave (1, 0) with the fundamental frequency and an amplitude of u ˆmax = 0.005, being the same as for the two-dimensional simulation, performed earlier [3]. The TS wave generates higher harmonics in the upper boundary layer, driving the roll-up of spanwise vortices (Kelvin-Helmholtz instability) and the subsequent vortex pairing behind the splitter plate. An additional oblique wave (1, 1) with a small amplitude of u ˆmax = 0.0005 is intended to provide a more realistic inflow disturbance than a purely twodimensional forcing. A total number of 80000 time steps with Δt = 0.018265 has been computed, corresponding to an non-dimensional elapsed time of t = 1461, with the last four periods of the fundamental frequency used for analysis. The instantaneous flowfield is illustrated in Fig. 13, showing the λ2 vortex criterion. Small vortices emerge from the longitudinal edges, slightly deforming the first spanwise vortex of the Kelvin-Helmholtz instability. Further downstream, multiple streamwise vortices exist per λz,0 , being twisted around the spanwise vortices. This vortex interaction leads to a breakdown of the big spanwise vortices. From x ≈ 120 onwards, the Kelvin-Helmholtz vortices known from the two-dimensional investigations are now an accumulation of small-scale structures. A spectral decomposition is shown in Figs. 14 and 15, based on the maximum of v along y. The normal velocity has been chosen as it is less associated with upstream propagating sound. The modes are denoted as (h, k) with h and k being the multiple of the fundamental frequency ω0 and the spanwise wavenumber γ0 , respectively. As Fig. 14 shows, the nonlinear interaction of the introduced disturbances (1, 0) and (1, 1) in the upper boundary layer generates nonlinearly the mode (0, 1) up to an amplitude of vˆ = 2 · 10−5 . From

332

A. Babucke, M. Kloker, U. Rist

10-1 10-1

10

-2

10-2

10-3

v’

v’

10-3

10

-4

(0,1) (0,2) (0,4) (0,8)

10-5 10-6

-50

0

x

50

100

Fig. 14. Generation of the steady modes (0, k) at the trailing edge, based on the maximum of v over y

(1,0) (1,1) (2,0) (2,1) (3,0) (3,1)

10-4 10-5 10-6

0

x 100

200

Fig. 15. Maximum amplitude of normal velocity v along y for unsteady modes (h, k)

x = −25 onwards, the upstream eﬀect of the notch at the end of the splitter plate prevails. The engrailment at the end of the splitter plate (−10 ≤ x ≤ 0) generates steady spanwise disturbances (0, k) with peaks up to vˆ = 8 · 10−3 at the corners. In the notch (7.8 ≤ z ≤ 23.6), the combination of wake and mixing layer originates further upstream at x = −10 instead of x = 0. This results in a spanwise deformation, corresponding to the disturbance (0, 1). Its amplitude decreases behind the splitter plate up to x = 15. Higher harmonics in spanwise direction (0, 2) and (0, 4) are generated at the notch as well, but only mode (0, 2) shows a similar upstream eﬀect as mode (0, 1). Behind the splitter plate, the amplitudes of the first two higher harmonics in spanwise direction stay almost constant at an amplitude of vˆ ≈ 6·10−4 and vˆ ≈ 4·10−4 , respectively. As two streamwise vortices per λz,0 emerge from the longitudinal edges, the steady, spanwise higher harmonics mainly correspond to these streamwise vortices. The similar amplitudes behind the splitter plate indicate that the engrailed trailing edge introduces a spanwise deformation due to the diﬀerent origin of the mixing layer as well as longitudinal vortices. For x > 40, all steady modes grow due to non-linear interaction with the travelling waves, resulting in a spanwise deformation of the mixing layer. The introduced two-dimensional TS wave grows slowly in the upper boundary layer. Figure 16 reveals the good agreement of its amplification rate with linear stability theory. Near the end of the splitter plate, the amplification rate diﬀers from LST due to the discontinuity in geometry. With an amplitude of the driving TS wave of vˆ ≈ 2 · 10−3 , shown in Fig. 15, the generated higher harmonic modes (2, 0), (3, 0) reach an amplitude of vˆ ≈ 3 · 10−4 and vˆ ≈ 2 · 10−5 , respectively. According to the forcing at the inflow, only lowamplitude oblique disturbances (2, 1) and (3, 1) are generated in the upper boundary layer. Behind the splitter plate, the growth of two-dimensional disturbances (h, 0) is only weakly aﬀected by the engrailed trailing edge. The growth rate of the fundamental frequency shows excellent agreement with lin-

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

Fig. 16. Ampliﬁcation rate of the Tollmien-Schlichting wave, compared with linear stability theory (marked with symbols)

333

Fig. 17. Ampliﬁcation rates of twodimensional disturbances behind the splitter plate, compared with linear stability theory (marked with symbols)

ear stability theory. The higher the frequency of the disturbances, the more diﬀers their amplification rate with a slightly lower mean amplification value compared to LST. The initially small three-dimensional disturbances (h, 1) grow instantaneously at the beginning of the notch (x = −10) by approximately one order of magnitude. Further downstream, they are driven by their two-dimensional counterparts (h, 0). Saturation of the first two higher harmonics (2, 0) and (3, 0) occurs at x ≈ 70, the position of the first vortex roll up. The two-dimensional fundamental disturbance (1, 0) saturates at x ≈ 140. This corresponds to the pairing of the accumulated small-scale structures. In order to evaluate the eﬀect of the modified trailing edge, the emitted sound is compared with a two-dimensional simulation with the same flow parameters, performed earlier [3]. The acoustic field, visualised by the dilatation ∇u, is given for the two cases in Figs. 18 and 19 for the two-dimensional simulation and the engrailed trailing edge, respectively. In both cases, no reflections from the boundaries are visible. For the two-dimensional simulation, the acoustic field is determined by long-wave sound, originating mainly from x ≈ 150 and x ≈ 220. This corresponds to the positions of vortex pairing [3]. The emitted sound for the engrailed trailing edge is mainly high-frequency noise with short wavelengths. Despite being a two-dimensional simulation, the acoustic field in Fig. 18 is less clearer than for the pure mixing layer [7]. Nevertheless two main sources can be determined at x ≈ 150 and x ≈ 220, corresponding to the positions of vortex pairing [3]. The emitted sound for the engrailed trailing edge is mainly high-frequency noise with short wavelengths as shown in Fig. 19. The main sources are located at x ≈ 140 and x ≈ 200 which is equivalent to the pairing of the allocations of the small-scale structures. For both, the two-dimensional case and the modified trailing edge, sound generation takes place not directly at the edge of the splitter plate, but further downstream in the mixing layer.

334

A. Babucke, M. Kloker, U. Rist

Fig. 18. Snapshot of the far-ﬁeld sound for the two-dimensional simulation showing the dilatation ∇u in a range of ±3 · 10−4

Fig. 19. Snapshot of the dilatation ﬁeld ∇u for the engrailed trailing edge at spanwise position z = 0. Contour levels are the same as in Fig. 18. The position of the acoustic observer is marked by a cross

The dilatation plots themselves do not show clearly whether the emitted sound is reduced. By placing an observer in the acoustic far-field (x = 195, y = −121.8, z = 0), marked by a cross in Fig. 19, the sound pressure level can be evaluated more precisely. The time-dependent pressure fluctuations are shown in Fig. 20 over four periods of the fundamental frequency. For both cases, the pressure fluctuations are almost random. The two-dimensional sample is dominated by low-frequency fluctuations compared to the engrailed-trailingedge case. The pressure fluctuations of the two- and three-dimensional case are p′2D = 0.0139 and p′3D = 0.00693, respectively. This means that the engrailed nozzle end leads to a reduction by a factor two, corresponding to a decrease of the noise by −6 dB.

1.115

p

1.110 1.105 1.100 1.095

mod. trailing edge 2-d simulation

0

100

t-t0

200

300

Fig. 20. Acoustic pressure ﬂuctuations in the far-ﬁeld at the observer’s position (x = 195, y = −121.8) for 2-d and 3-d trailing-edge simulation. The plotted time interval corresponds to four periods of the fundamental frequency

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

335

4 Computational Aspects The simulation was run on the NEC-SX8 Supercomputer of the hww GmbH, Stuttgart, using 16 nodes which corresponds to a total number of 128 processors. On each node, one MPI process was executed, each with shared-memory parallelization having eight tasks. The computation of 80000 time steps required 46 hours wall-clock time. This leads to a total CPU-time of nearly 6000 hours and a specific computational time of 1.8 μs per gridpoint and time step (including four Runge-Kutta subcycles), being only the double value of the incompressible N3D code of the IAG [13]. With a sustained performance of 694.7 GFLOP/s, 34% of the theoretical peak performance of the computer are reached. The code shows a vector operation ratio of 99.47% with an average vector length of 206 and a total memory size of 162 GB. As the array sizes of each domain are equal, only slight performance diﬀerences between the MPI processes exist. Profiling shows that the main computational time (40%) is spent in the Fourier-transformation, followed by the evaluation of the Navier-Stokes equations with 17.4% CPU-time. The computation of the spatial derivatives in streamwise and normal direction requires 10% and 16%, respectively. Time-integration is only a minor part with 1.7% CPU-time. Having 21 derivatives in x- and 25 derivatives in y-direction, the theoretical speedup of the pipelined Thomas algorithm is, according to (10), 6 for the streamwise and 1.92 for the normal direction. This means that the eﬃciency of the corresponding parts of the code is 75% and 96%, respectively. Taking into account that the major parts of the code (FFT, Navier-Stokes equations) are local for each MPI process, the overall eﬃciency regarding MPI parallelization is around 96%. As mentioned in Sect. 2.3, this does not include communication times. Nevertheless, profiling shows that the time spent for data transfer is negligible. A limiting factor for the shared-memory parallelization is the spanwise resolution. For a symmetric computation, the number of gridpoints in z-direction is (2n + 1), with the integer n depending on the number of spanwise modes (here n = 6). By that, the spanwise resolution is not a multiple of 8 (the number of shared-memory tasks). The spanwise resolution of 65 points used here means that seven processors pass a Microtasking-loop eight times and 1 processor nine times. By that the eﬃciency of a shared-memory parallelized loop decreases to 90%. This value corresponds well to the profiling results, showing that 10.5% of the CPU-time is spent in the barriers, framing a Microtasking-loop (e.g. subroutines ex lpminit, ex lpmterm). This problem does not exist for non-symmetric simulations, the spanwise resolution is then an exact multiple of eight. Including the losses of both types of parallelization, an overall eﬃciency of 86% on 128 processors was reached. To achieve further gains in performance, we intend to optimise the code further, with the main focus on the Fourier-transformation as it requires the most of the computational time. By using the FFT from the SX8 internal library instead of our own subroutines [8], we hope to achieve improvements, since this increased the speed of the incompressible N3D code by 20–30% [13].

336

A. Babucke, M. Kloker, U. Rist

5 Conclusion The sound generation of an isothermal subsonic jet with Mach numbers MaI = 0.8 and MaII = 0.2 has been simulated using spatial DNS. The nozzle end is modelled by a thin finite flat plate with spanwise engrailment at its trailing edge. This modification of the nozzle end serves as a first example of an actuator for noise reduction, generating streamwise vortices and a spanwise deformation of the flow. Further downstream, the induced longitudinal vortices are bended around the spanwise vortices of the Kelvin-Helmholtz instability, leading to a breakdown of the large coherent structures. By that, the spanwise vortices, known from two-dimensional simulations are now an accumulation of small scale structures. The emitted sound is compared to a two-dimensional simulation with the same flow parameters. The engrailed trailing edge leads to higher-frequency noise, while the generated sound of the two-dimensional simulation is dominated by low-frequency noise. Despite the parameters of the notch were chosen arbitrarily, a noise reduction of 6dB could be achieved. Therefore, we are confident that further improvements in jet-noise reduction are possible. Besides finding the optimal parameters for the engrailment (shape and dimensions), we also intend to test diﬀerent types of active and passive actuators. The investigation was performed using our complete simulation framework of preprocessing, stability analysis, DNS and postprocessing. Due to the combination of frontend machine and supercomputer, a comfortable combination of our tools is possible. This does not only increase the usability, it also provides reproducibility of the complete set of results. The DNS code NS3D uses a hybrid parallelization of MPI and Microtasking. The achieved computational performance is 694.7 GFLOP/s on 16 nodes of the NEC-SX8 vector computer, corresponding to 34% of its theoretical peak performance. The parallel eﬃciency of the code was investigated by a combination of theoretical analysis and profiling. For this simulation, a scaling of 86% on the 128 processors was reached. Acknowledgements The authors would like to thank the Deutsche Forschungsgemeinschaft (DFG) for its financial support within the subproject SP5 in the DFG/CNRS research group FOR-508 “Noise Generation in Turbulent Flows”. The provision of supercomputing time and technical support by the H¨ ochstleistungsrechenzentrum Stuttgart (HLRS) within the projects “Lamtur” and “Teraflop Workbench” is gratefully acknowledged.

DNS of an Engrailed Nozzle End for Jet-Noise Reduction

337

References 1. J. D. Anderson. Computational Fluid Dynamics. McGraw-Hill, 1995. 2. A. Babucke, M.J. Kloker, and U. Rist. DNS of a plane mixing layer for the investigation of sound generation mechanisms. to appear in Computers and Fluids, 2007. 3. A. Babucke, M.J. Kloker, and U. Rist. Numerical investigation of ﬂow-induced noise generation at the nozzle end of jet engines. In to appear in: New Results in Numerical and Experimental Fluid Mechanics VI, Contributions to the 15. STAB/DGLR Symposium Darmstadt, 2007. 4. A. Babucke, J. Linn, M. Kloker, and U. Rist. Direct numerical simulation of shear ﬂow phenomena on parallel vector computers. In High performance computing on vector systems: Proceedings of the High Performance Computing Center Stuttgart 2005, pages 229–247. Springer Verlag Berlin, 2006. 5. C. Bogey, C. Bailly, and D. Juve. Numerical simulation of sound generated by vortex pairing in a mixing layer. AIAA J., 38(12):2210–2218, 2000. 6. C. Canuto, M.Y. Hussaini, and A. Quarteroni. Spectral methods in fluid dynamics. Springer Series of Computational Physics. SpringerVerlag Berlin, 1988. 7. T. Colonius, S.K. Lele, and P. Moin. Sound generation in a mixing layer. J. Fluid Mech., 330:375–409, 1997. 8. EAS3 project. http://sourceforge.net/projects/eas3. 9. J.B. Freund. Noise sources in a low-Reynolds-number turbulent jet at Mach 0.9. J. Fluid Mech., 438:277–305, 2001. 10. M.B. Giles. Nonreﬂecting boundary conditions for Euler equation calculations. AIAA J., 28(12):2050–2058, 1990. 11. M.J. Kloker. A robust high-resolution split-type compact FD scheme for spatial DNS of boundary-layer transition. Appl. Sci. Res., 59:353–377, 1998. 12. L. Mack. Boundary-layer linear stability theory. In AGARD Spec. Course on Stability and Transition of Laminar Flow, volume R-709, 1984. 13. R. Messing, U. Rist, and F. Svenson. Control of turbulent boundary-layer ﬂow using slot actuators. In High performance computing on vector systems: Proceedings of the High Performance Computing Center Stuttgart 2006. Springer Verlag Berlin, 2007.

Direct Numerical Simulation of a Round Jet into a Crossflow – Analysis and Required Resources J.A. Denev1 , J. Fr¨ ohlich2 , and H. Bockhorn1 1

2

Institute for Technical Chemistry and Polymer Chemistry, University of Karlsruhe (TH), Kaiserstraße 12, D-76128 Karlsruhe, Germany denev,[email protected] Institute for Fluid Mechanics, Technical University of Dresden, George-B¨ ahr Straße 3c, D-01062 Dresden, Germany [email protected]

Summary. Results from two Direct Numerical Simulations of a round jet in crossﬂow with velocity ratio of 3.3 are presented. The Reynolds number was 650 and 325. A passive scalar with Schmidt number of unity is introduced with the jet. The boundary conditions for both, jet and crossﬂow are laminar. This provides an unambiguous deﬁnition of the setup and favours its use as a test case. Transition of the jet was identiﬁed by an abrupt expansion of the average scalar ﬁeld. The higher Reynolds number leads to a transition at 3.49 diameters downstream of the jet exit, the lower one – at 4.41 diameters. The higher Reynolds number ﬂow exhibits smaller turbulent structures, but despite this and the diﬀerent location of the transition, the trajectories of the two ﬂows are close to each other. The computational technique employed is a block-structured Finite-Volume method with local grid reﬁnement at block boundaries implemented in the code LESOCC2. This allowed eﬃcient distribution of cells so that 89% of them could be clustered in the vicinity of the jet exit and in the transition region. Issues of parallelization and eﬃciency are addressed in the text.

1 Introduction The configuration of a jet issuing from a pipe into a crossflow (JICF) appears frequently in chemical, pharmaceutical, environmental and combustion engineering, to name but a few application areas. The complex vortical structures of this flow and its good mixing capabilities make it a target of intense investigation for both experimental and numerical groups [7]. Previous work of the authors showed results from a Direct Numerical Simulations (DNS) of the jet in crossflow at Reynolds number Re = 275 (defined with the jet-

340

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

diameter D and the crossflow velocity u∞ ) and jet-to-crossflow velocity ratio R = wjet /u∞ = 2.4. In the present study the results from two recent Direct Numerical Simulations (DNS) with velocity ratio R = 3.3 and Reynolds numbers Re = 650 and Re = 325 are compared. The influence of the Reynolds number on the vortex structures of the flow and its jet trajectory is presented in detail. As the jet in crossflow has a relatively complex structure, it is not trivial to generate a block-structured grid which is well-suited for this flow. As the eﬃcient use of computer resources is specially important for studies with DNS, the present work focuses also on this issue.

2 Flow Configuration The flow configuration of a round transverse jet into a crossflow has been presented in detail in a previous work of the authors [1]. For brevity only the main parameters will be discussed here. The pipe flow, from which the jet issues is laminar, as well as the crossflow. The laminar flow conditions allow to keep the boundary conditions of the investigation simple and unique (in contrast to, e.g. turbulence flow generators). However, during the interaction of the two streams transition to turbulence occurs and one of the targets of the present work is to investigate the characteristics of this transition and to establish its dependence on the Reynolds number in the range being studied. As the equations solved are in a non-dimensional form (with the jet diameter D and the crossflow velocity u∞ set to unity in the computations), the Reynolds number is set directly as an input parameter for the computations. In order to allow the simultaneous study of the mixing process and of chemical reactions in the configuration of the jet in crossflow, transport equations for passive and reactive scalars have been solved additionally to the incompressible Navier-Stokes equations. These equations govern the transport of non-reactive scalars and for them the Schmidt number has been varied and also the transport of reactive scalars in three simple chemical reactions and for them the influence of the Damkoehler number has been studied. Although according to the targets of the present investigation results only from one scalar transport equation are shown, the presence of a large set of partial differential equations for the remaining scalars should be beard in mind because it aﬀects seriously the required computer resources for the investigation.

3 Numerical Method The simulation has been carried out with the collocated block-structured Finite Volume Code LESOCC2, developed at the Institute for Hydromechanics (IfH) of the University of Karlsruhe [5]. Second-order accurate schemes have

DNS of a JICF – Analysis and Required Resources

341

been used for spatial discretization. The flow is treated as incompressible and a Poisson equation is solved for the pressure-correction equation. No-slip boundary conditions have been applied at the walls of the channel crossflow with a boundary layer with a size larger than one pipe diameter. The simulated length of the pipe generating the jet is two diameters which has been found to be suﬃcient according to literature data and own preliminary simulations.

4 Local Grid Refinement for Optimum Distribution of Grid Nodes The present investigation uses a block-structured curvilinear grid. Structured numerical grids have the drawback that finely resolved regions inevitably require fine mesh resolution throughout the whole computational domain. This can lead to an unnecessary increase of the total number of control volumes. To overcome this drawback, the present study uses the local grid refinement technique, recently implemented in the code LESOCC2 [5]. The implementation allows two neighbour blocks to have a diﬀerent number of control volumes at the two sides of their common boundary, see Fig. 1. Using this feature, the region near the jet exit is covered by small control volumes – in the present study 89% of the total number of control volumes are located there. The factor of refinement used is 3:1 for in all spatial directions.

Fig. 1. The reﬁned region of the grid in the midplane (y = 0), shown within the white boundaries. The local grid reﬁnement factor is 3:1 for all spatial directions of the numerical blocks near the jet-exit. The direction of the jet is upwards along the z-axes, the crossﬂow is along the x-axes (from left to right)

342

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

5 Parallel Efficiency of the Computations The computations use a grid with 219 numerical blocks. At each Runge-Kutta step the computed variables are exchanged over the block boundaries. The algorithm requires that at each side of the block boundary two rows of additional halo cells are positioned. Those cells store the information for exchange by MPI (Message Passing Interface). Consequently, in a three-dimensional computation the percentage of halo cells can become large. For the present investigation the amount of the halo cells is 4 342 976, or 24% of the control volumes: this is the diﬀerence between the total number of computational nodes (22 320 832) and the number of control volumes (17 977 856). The gridding process and the local grid refinement of the blocks near the jet outlet (Fig. 1) lead to a disparity in the amount of control volumes per block. The histogram in Fig. 2 shows the number of blocks for a certain range of control volumes. A large number of small blocks exist, together with an increased number of large blocks. The maximum number of processors to be used is generally limited by the size of the largest numerical block. For the present investigation the largest block (which has to be computed on a single processor) has 606 528 control volumes and there are 8 blocks of that size (cf. Fig. 2). Figure 3 shows the load balancing eﬃciency of the computations as a function of the number of processors for the present investigation. The graph is based on the statistics reported by the LESOCC2 code – after an optimization tool has been used for the distribution of the blocks over the processors. The load balancing eﬃciency does not take into account the computer hardware, but rather is a measure for the best achievable parallel eﬃciency. However, for the use of 31 processors, e.g., this information fully complies with the perfor-

Fig. 2. Histogram: distribution of the number of control volumes for the 219 blocks

DNS of a JICF – Analysis and Required Resources Load balance efficiency

100

343

Fig. 3. Load balance eﬃciency for the present problem as a function of the number of processors

80 60 40 20 0 0

4

8 12 16 20 24 28 32 36 40 Number of processors

mance statistics of the HP XC4000 – both give 91% eﬃciency (i.e. 91% user time and 9% communication time). From the graph it is seen that the use of 31–32 processors presents an optimum for the present problem grid. Using more than 36 processors would decrease the parallel eﬃciency considerably, because of the restriction connected with the largest block size as mentioned above. In order to increase the number of processors, a new blocking structure with more blocks and less control volumes per block would be required. This in turn would increase the number of halo cells and the communication time for MPI, thus decreasing the parallel eﬃciency of the computations. Some restrictions on the increase of processor number and control volume number currently apply due to the use of a commercial grid generator – ANSYS ICEM CFD which was employed in this project.

6 Other Numerical Statistics from the Computations The computations have been carried out on the new HP XC4000 high performance computer at the Computer Centre of the University of Karlsruhe (TH). Thirty one processors (AMD Opteron, 2.6 GHz) have been used for the computations. In order to accumulate the required statistical data for the averaged variables, 105.7 dimensionless time units have been calculated (time units are based on the diameter of the jet-pipe and the velocity of the crossflow). For the computations of these statistics the CPU-time (sum over all processors) was 35 708 hours for the run with Re = 650 (less than 2 months clock-time and practically no queuing time!). The lower Reynolds number case (Re = 325) required about 13.5% more CPU-time. This is due to the smaller time-step which is computed by the code based on stability criteria. The CPU-time required per grid node and per time-step is 5.3E–05. This is based on the computation of the incompressible Navier-Stokes equations together with 3 equations for non-reacting passive scalars and 6 equations

344

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

for reacting (but also passive in relation to the velocity field) scalars. The computation of all 9 scalar equations takes more than 2/3 of the CPU-time of the entire computation. No restrictions connected with the RAM of the HP XC4000 have been noticed during the computations (required RAM per processor was 4 Gb, the computations use practically below 25% of this). Finally, during the test stage of the HP XC4000 its performance (still with the single rate (SDR) of the Infiniband 4X DDR interconnect) have been compared to the performance of HP XC6000. For the present computations the performance of the two machines has been practically identical. Also the comparison between the three FORTRAN 90 compilers available on the HP XC4000 (Intel, PGI and PathScale) showed the same performance (diﬀerences found were less than 1%).

7 Flow Structures at Reynolds Number 650 The jet in crossflow exhibits complex vortex structures, which have been the subject of research by many authors, e.g. ([3, 4, 6, 8]). Diﬀerent variables are used here to allow a clear presentation of the corresponding vortices for the case with Re = 650 in Fig. 4. Figure 4a shows the instantaneous concentration field of scalar c1 issuing from the pipe. The position of the transition as well as the formation of the first two ring-like vortices (the first one being in the zone prior to transition) is well identified on the figure. Fig. 4b shows an isosurface of the Q-criterion [2]. Additional to the transition region, a clear picture of the upright vortices formed in the wake of the jet is obtained. In the boundary layer near the bottom wall two vortices directed downstream the crossflow are also observed. Large vortices are characterized by regions of pressure minima. Fig. 4c shows the isosurface of the pressure fluctuation. The pressure fluctuation is defined as the averaged pressure, p, subtracted from the instantaneous pressure, p, thus showing the locations of pressure minima. Three ring-like vortices are seen in the figure, with the third one being downstream of the transition region. Figure 4d shows the same variable (p − p) in the plane of symmetry y = 0 (the symmetry applies only to the averaged flow field!). Beside the two pressure minima, which coincide with the ring-like vortices in Fig. 4c, the locations of the upright vortices behind the jet are clearly seen (cf. Fig. 4b). All variables shown in Figs. 4a–d were extracted for the same instant in time. They all show the clear presence of small-scale turbulent structures behind the region of transition together with the disorder and irregularity of the vortex structures typical for turbulent flows. The fifth figure (Fig. 4e), unlike the previous four, shows an averaged isosurface of the scalar concentration together with some three-dimensional streamlines of the averaged velocity field. It allows the identification of the counter-rotating vortex pair (CVP) which is

DNS of a JICF – Analysis and Required Resources

345

Fig. 4. Vortex structures of the jet in crossﬂow, simulation with Re = 650. a scalar concentration, c1 = 0.18; b the Q-criterion, Q = 1.0; c pressure ﬂuctuation, p−p = −0.1; d pressure ﬂuctuation in the plane, y = 0; e average scalar concentration, c1 = 0.24; f 2D (white lines) and 3D, (ribbons coloured by the concentration of the scalar) streamlines. The plane z = 0.1 is coloured by the z-vorticity

346

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

the most typical vortex structure of the jet in crossflow. In this figure the jet is oriented towards the observer and the streamlines show the strong upward flow between the two vortices of the CVP. This upward flow contributes to the increased mixing capabilities of the transverse jet compared to a straight jet without crossflow. In the figure, the two downstream oriented vortices from Fig. 4b (close to the bottom wall) are visualized by two streamlines. Finally, Fig. 4f shows some two- and three-dimensional streamlines of the averaged flow field from which again the complex vortex structure of the jet in crossflow can be deduced. The four “ribbons” originate from the pipe (positioned symmetrically close to the pipe walls at z = 0). Two of them, starting from the “side” of the jet (coordinates of their origins being (0.;0.45;0.) and (0.;–0.45;0.)) are twisted by the vortices at the rear side of the jet and then taken upwards from the same upward flow shown in Fig. 4e. The vortices at the rear side of the jet are also seen on the 2d streamlines drawn in a horizontal plane at z = 0.1. The same plane is coloured by the vorticity component in direction of the z-axes, showing again the upright vortices from Figs. 4b and d.

8 The Influence of the Reynolds Number In Fig. 5 the corresponding vortex structures for the case with two times lower Reynolds number (Re = 325) are presented and compared to the flow with Re = 650 (the dimensionless time of the accumulated statistics for this computation is 50). On this figure the same levels of the isosurfaces as in Fig. 4 are presented. Basically two important features can be seen from the comparison of the two figures. First, as it should be expected, the transition region for the jet at Re = 325 is further downstream than that for Re = 650. Second, all the turbulent structures are clearly larger than the corresponding structures for Re = 650. Despite this, Fig. 5a,b,c shows that even in the case with Re = 325 the flow downstream of the transition region is irregular and turbulent. Figure 5d shows the influence of the Reynolds number on the twodimensional streamline originating from the origin of the jet (coordinates (0.;0.;0.)). This definition of the jet streamline which sometimes also is called “streamline trajectory” ([9]) is computed from the averaged flowfield. The figure shows that the trajectory for the case Re = 325 is a bit lower than at Re = 650 and also that there is only little impact of the Reynolds number on the shape of the streamline trajectory. Here it should be noticed that the shape of the boundary layer at the channel walls has been exactly the same for the two flows studied in order to exclude the influence of all other parameters except the Reynolds number.

DNS of a JICF – Analysis and Required Resources a)

b)

c)

d)

347

12 10

z

8 6 4

Re = 650 Re = 325

2 0

0

5

x

10

15

Fig. 5. Vortex structures for the case with Re = 325 and the central streamline (averaged velocity ﬁeld) for the two DNS. a scalar concentration, c1 = 0.18; b the Q-criterion, Q = 1.0; c pressure ﬂuctuation, p−p = −0.1; d the central streamlines (0.;0.;0.) at y = 0

9 Location of the Transition It can be deduced from the spatial distribution of the plotted variables in Figs. 4 and 5, that the transition occurs at diﬀerent heights (i.e. diﬀerent z-values) for the upstream and downstream parts of the jet. In order to seek for a universal measure to define the location of transition, the concentration distribution in the symmetry plane y = 0 together with the central streamline (0.;0.;0.) of the averaged flowfield are plotted in Fig. 6. Now, from Figs. 6a and b the location of transition in the symmetry plane (y = 0) is easy to deduce. However, although not shown here explicitly, the transition location changes with time. Therefore, it was decided that the averaged scalar field presents a better measure for the transition point. For this reason the averaged scalar distribution along the streamline trajectory is considered. As it can be seen from Figs. 5c and d, the transition is characterised by a sudden enlargement of the scalar area along the streamline. This area is marked by the white “terminating” line for the streamline in Figs. 6c and d.

348

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

Fig. 6. Location of the transition point – visualized by the scalar concentration. a Instantaneous scalar concentration at plane y = 0 (Re = 650), b instantaneous scalar concentration at plane y = 0 (Re = 325), c averaged scalar concentration at plane y = 0 (Re = 650) and the location of transition, d averaged scalar concentration at plane y = 0 (Re = 325) and the location of transition

From Fig. 6 it becomes evident that the Reynolds number has a clear influence on the location of transition: transition occurs earlier in the case of higher Reynolds number. For Re = 325 the transition at the streamline trajectory occurs at s/D = 4.41 and for Re = 650 – at s/D = 3.49, or in other words, the transition occurs approximately one diameter earlier for the higher Reynolds number. Here, “s” is the path along the streamline starting from the point (0.;0.;0.).

10 Conclusions Direct Numerical Simulation has been used to study the vortex structures of the jet in crossflow. The simultaneous computation and averaging of all important flow variables and scalar concentrations allows a detailed and concurrent analysis of the turbulent flow structures and the occurrence of transition.

DNS of a JICF – Analysis and Required Resources

349

A local grid refinement technique with a factor 3:1 allows an optimal distribution of the grid nodes of the block-structured grid within the computational domain. This contributes considerably to the eﬃciency of the numerical investigation and to the overall reduction of computational costs. The use of a complex curvilinear grid and complex domain shape for the jet in crossflow requires a large number of processors optimising at the same time the distribution of the control volumes over the processors. The optimisation process applied here in a separate preprocessing step takes into account the histogram of the distribution of the number of control volumes per block, the number of control volumes in the largest blocks and the parallel eﬃciency, which in turn depends on the number of processors. For the present DNS the optimum distribution of control volumes per processor was found when using 31 or 32 processors, allowing parallel eﬃciency of 91% to be achieved on the XC HP 4000 high performance computer. The Reynolds number has been found to be an important factor for the location of the transition region. At the same time its influence on the trajectory streamline of the flow is small. Acknowledgements The simulations were performed on the national super computer HP XC4000 at the High Performance Computing Center Stuttgart (HLRS) under the grant with acronym “DN S − jet”.

References 1. J.A. Denev, J. Fr¨ ohlich, and H. Bockhorn. Direct numerical simulation of mixing and chemical reactions in a round jet into a crossﬂow. In W.E. Nagel, W. Jaeger, and M. Resch, editors, High Performance Computing in Science and Engineering 06, Transactions of the High Performance Computing Center Stuttgart, pp. 237– 251. Springer, Heidelberg New York, 2006. 2. Y. Dubief and F. Delcayre. On coherent-vortex identiﬁcation in turbulence. J. Turbulence, 1(11), 2000. 3. T.F. Fric and A. Roshko. Vortical structure in the wake of a transverse jet. J. Fluid Mech., 279:1–47, 1994. 4. B.A. Haven and M. Kurosaka. Kidney and anti-kidney vortices in crossﬂow jets. J. Fluid Mech, 352:27–64, 1997. 5. C. Hinterberger. Dreidimensionale und tiefengemittelte Large–Eddy–Simulation von Flachwasserstr¨ omungen. PhD thesis, Institute for Hydromechanics, University of Karlsruhe, http://www.uvka.de/univerlag/volltexte/2004/25/, 2004. 6. T.T. Lim, T.H. New, and S.C. Luo. On the development of large-scale structure of a jet normal to a cross ﬂow. Phys. Fluids, 13(3):770–775, March 2001. 7. R.J. Margason. 50 years of jet in cross ﬂow research. In Computational and experimental assessment of jets in crossflow, pages 1.1–1.41, AGARD-CP-534, 1993.

350

J.A. Denev, J. Fr¨ ohlich, H. Bockhorn

8. S. Narayanan, P. Barooah, and J.M. Cohen. Experimental study of the coherent structure dynamics and control of an isolated jet in crossﬂow. AIAA Paper 20020272, 2002. 9. L.L. Yuan and R.L. Street. Trajectory and entrainment of a round jet in crossﬂow. Phys. Fluids, 10(9):2323–2335, 1998.

Transport of Heavy Spherical Particles in Horizontal Channel Flow Markus Uhlmann1 and Jochen Fr¨ ohlich2 1

2

Modeling and Numerical Simulation Unit, CIEMAT, Av. Complutense 22, 28040 Madrid, Spain [email protected] Institut f¨ ur Technische Chemie und Polymerchemie, Universit¨at Karlsruhe, Kaiserstraße 12, 76128 Karlsruhe, Germany [email protected]

Summary. We numerically simulate turbulent ﬂow in a horizontal plane channel over a bed of mobile spherical particles. All scales of ﬂuid motion are resolved without modeling and the phase interface is accurately represented. Direct contact between particles is taken into account by an artiﬁcial short-range repulsion force. Our results indicate possible scenarios for the onset of erosion through collective motion induced by buﬀer-layer streaks and subsequent saltation of individual particles.

1 Introduction The motion of rigid particles suspended in horizontal boundary-layer-type flow is a feature which is encountered in a wide range of applications of technological and environmental interest. Examples include pneumatic conveying and bedload transport in rivers. The understanding and modeling of the latter case is one of the mayor topics in civil and environmental engineering since inhomogeneous erosion and particle transport lead to a modification of the shape of the bed. The basic question in this context concerns the critical conditions for the onset of erosion, most commonly described in terms of the critical non-dimensional shear stress as a function of the normalized particle diameter. Shields [1] was the first to provide an empirically determined curve for the critical shear stress which is still widely used by engineers. Other authors have developed diﬀerent correlations some of which disagree considerably [2]. Up to this date, however, the detailed mechanisms underlying the resuspension phenomenon have not been fully clarified. In the present study we approach this problem by generating flow data with the aid of direct numerical simulation (DNS). This technique enables us to analyze the quantities of interest with great spatial and temporal precision, albeit at a large computational cost and for somehow idealized situations.

352

M. Uhlmann, J. Fr¨ ohlich

A considerable number of experimental studies focused upon the hydrodynamic erosion process can be found in the literature. The following is a small selection of the most relevant references for the present investigation. Kaftori, Hetsroni and Banerjee have studied the motion of particles in a horizontal turbulent boundary layer [3, 4]. In these experiments, particles were nearly buoyant, and their interaction with the coherent structures of the near-wall region was documented in detail. By analysis of high-speed video images, Ni˜ no and Garcia [5] have identified the upward motion downstream of strong inclined shear layers as the dominant mechanism which lifts heavy particles away from the wall in turbulent open channel flow. Ni˜ no, Lopez and Garcia [2] present parametrical data for the onset of erosion. Finally, let us mention that Kiger and Pan [6] found evidence for a correlation between the existence of hairpin vortex packets and vertical motion of heavy particles. Reliable numerical studies of particulate flow have only recently come into reach of available supercomputing capabilities. Zeng, Balachandar and Fischer have studied the forces acting upon a single sphere which translates parallel to a wall in a quiescent fluid [7]. Patankar et al. have studied the lift-oﬀ of a single circular disc in horizontal (laminar) Poiseuille flow [8]. Also in two dimensions, Choi and Joseph simulated the fludization of 300 closely packed circular discs in the same configuration [9]. They found that – for suﬃciently high values of the driving pressure gradient – the bed of particles is “inflated” in an early stage (meaning that fluid enters the bed) which in turn causes dislocation of individual particles in the top row, finally leading to full resuspension of particles. It should be noted that the particles were located on a smooth wall. Furthermore, it is not clear whether their conclusions extend to the three-dimensional case. Finally, it should be kept in mind that the flow was laminar, which means that there are potentially less mechanisms available for the lift-oﬀ of particles. Pan and Banerjee [10] were probably the first to conduct DNS of turbulent plane channel flow with suspended spherical particles of finite size. However, they mainly considered fixed particles and the spatial resolution of each sphere was rather coarse. The particles in their study were nearly neutrally buoyant, and, therefore, the mechanism for re-suspension was not considered. To our knowledge, the re-suspension of heavy spherical particles in horizontal turbulent channel flow was for the first time simulated via the DNS approach in a preliminary study documented in [11]. Therein it was observed that particles are set into motion collectively, i.e. in streamwise organized trains, by strong shear events (high-speed streaks), eventually leading to saltation of individual particles. In the present contribution, we expand upon those results in two respects. Firstly, a substantially larger computational domain is chosen in order to provide a more realistic representation of the largest scales of the turbulent motion; secondly, an additional configuration is simulated where some of the particles in the top layer of the bed are kept in a fixed position in order to inhibit the collective sliding motion found in the previous study.

Transport of Particles in Horizontal Channel Flow

353

In the following we will first outline our numerical method. In § 3.2 we consider the problem of turbulent flow over spherical particles which are fixed to the wall. In § 3.3 we present the numerical experiments involving the erosion of mobile particles. A short discussion of our results can be found in § 5.

2 Numerical Method The numerical method used for the direct simulation of sediment transport has been proposed in [12]. For clarity we will first present the general idea in the framework of a single-step time discretization. The final algorithm as implemented in the code SUSPENSE (Suspended Particle Evolution by Navier-Stokes Equations) will be given in (6) below. For this purpose, let us write the momentum equation in the following form un+1 − un = rhsn+1/2 + f n+1/2 , Δt

(1)

where rhsn+1/2 regroups all usual forces (convective, pressure-related, viscous) and f n+1/2 is the fluid-solid coupling term, both evaluated at some intermediate time level. Since the work of [13] it is common to express the additional force by simply rewriting the above equation as f n+1/2 =

u(d) − un − rhsn+1/2 Δt

(2)

where u(d) is the desired velocity at any point where forcing is to be applied (i.e. at a point inside a solid body). Formula (2) is characteristic for direct forcing methods. Problems arise from the fact that in general the solid-fluid interface does not coincide with the Eulerian grid lines, meaning that interpolation needs to be performed in order to obtain an adequate representation of the interface. In [12] the definition of the force term was instead formulated at Lagrangian positions attached to the surface of the particles, viz. Fn+1/2 =

U(d) − Un − RHSn+1/2 , Δt

(3)

where upper-case letters indicate quantities evaluated at Lagrangian coordinates. Obviously, the velocity in the particle domain S is simply given by the solid-body motion, U(d) (X) = uc + ω c × (X − xc )

X∈S,

(4)

as a function of the translational and rotational velocities of the particle, uc , ω c , and its center coordinates, xc . The final element of the method of [12] is the transfer of the velocity (and r.h.s. forces) from Eulerian to Lagrangian positions as well as the inverse

354

M. Uhlmann, J. Fr¨ ohlich

transfer of the forcing term to the Eulerian grid positions. For this purpose we define a Cartesian grid xijk with uniform mesh width h in all three directions. Furthermore, we distribute so-called discrete Lagrangian force points Xl (with 1 ≤ l ≤ NL ) evenly on the particle surface. An ‘even’ distribution of points on the surface of a sphere can be obtained in a pre-processing step by an iterative procedure [App. A, 12]. Using the regularized delta function formalism of [14], the transfer can be written as: U(Xl ) = u(xijk ) δh (xijk − Xl ) h3 , (5a) ijk

f (xijk ) =

l

F(Xl ) δh (xijk − Xl ) ΔVl ,

(5b)

where ΔVl designates the forcing volume assigned to the lth force point. We use the particular function δh given in [15] which has the properties of continuous diﬀerentiability, second order accuracy, support of three grid nodes in each direction and consistency with basic properties of the continuous delta function. It should be underlined that the force points are distributed on the interface between fluid and solid, Xl ∈ ∂S, and not throughout the whole solid domain S. The reason for this is eﬃciency: the particle-related work scales as (D/h)2 instead of (D/h)3 , where D is the particle diameter. The consequences for the eﬃciency of the forcing due to these two alternative placements of the forcing points have been discussed in a previous study [16]. The above method has been implemented in a staggered finite-diﬀerence context, involving central, second-order accurate spatial operators, an implicit treatment of the viscous terms and a three-step Runge-Kutta procedure for the non-linear part. Continuity in the entire domain is enforced by means of a projection method. For completeness, the full semi-discrete equations for each Runge-Kutta sub-step (indicated by superscript k ) are given in the following (superscripts (m) refer to a particle in the range 1 ≤ m ≤ Np ): ˜ = uk−1 + Δt 2αk ν∇2 uk−1 − 2αk ∇pk−1 u k−1 k−2 (6a) −γk [(u · ∇)u] − ζk [(u · ∇)u] (m) ˜β (X(m) ) = U u˜β (xijk ) δh (xijk − Xl ) h3 ∀ l; m; 1 ≤ β ≤ 3 (6b) l ijk

(m)

F(Xl

(m)

)=

fβ (xijk ) =

U(d) (Xl Np NL

m=1 l=1

˜ (m) ) ) − U(X l Δt (m)

Fβ (Xl

∀ l; m (m)

) δh (xijk − Xl

(m)

) ΔVl

(6c) ∀ i, j, k

1≤β≤3

(6d)

Transport of Particles in Horizontal Channel Flow

∇2 u∗ −

˜ 1 u∗ u =− + f k + ∇2 uk−1 αk νΔt ναk Δt ∇ · u∗ , ∇2 φk = 2αk Δt uk = u∗ − 2αk Δt∇φk , pk = pk−1 + φk − αk Δt ν∇2 φk

355

(6e) (6f) (6g) (6h)

where the set of coeﬃcients αk , γk , ζk (1 ≤ k ≤ 3) is given in [17]. The intermediate variable φ is the so-called “pseudo-pressure”, u∗ the predicted velocity field; both are discarded after each step. The particle motion is determined by the Runge-Kutta-discretized Newton equations for rigid-body motion, which are weakly coupled to the fluid equations. During the course of a simulation, particles can approach each other closely. However, very thin liquid inter-particle films cannot be resolved by a typical grid and therefore the correct build-up of repulsive pressure is not captured which in turn can lead to possible partial “overlap” of the particle positions in the numerical computation. In practice, various authors use artificial repulsion potentials which prevent such non-physical situations [18–20]. Here we apply the collision strategy of Glowinski et al. [21], relying upon a short-range repulsion force (with a range of 2Δx). The stiﬀness parameter appearing in the definition of the repulsion force has been calibrated in simulations of two sedimenting particles and particle-wall interactions. The current algorithm has been implemented for parallel machines with distributed memory, using the MPI library. For reasons of eﬃciency, the Helmholtz problems in (6e) are simplified by second-order-accurate approximate factorization and the Poisson problem in (6f) is solved by a multi-grid technique. We use a domain decomposition technique for distributing the Eulerian nodes over a three-dimensional processor grid. Each processor treats the particles currently located in its sub-domain. Additionally, the neighbor processors need to contribute to the transfer operations (6b, 6d) whenever particle domains happen to overlap sub-domains of the distributed grid. The particle treatment can therefore be described as a “master-slave” technique. Our method has been subjected to a wide array of validation cases: (i) Taylor-Green flow in an immersed region; (ii) uniform flow around a stationary cylinder; (iii) uniform flow around an oscillating cylinder; (iv) a freely rotating circular disc in Couette flow; (v) a single sedimenting circular disc; (vi) a freely-rotating cylinder in Couette flow; (vii) drafting-kissing-tumbling of two circular discs; (viii) pure wake interaction of two circular discs; (ix) sedimentation of a single spherical particle; (x) pure wake-interaction of two spherical particles; (xi) drafting-kissing-tumbling of two spherical particles. (xii) grid convergence of vertical particulate channel flow. Most of these simulations have been documented in [12, 16, 22–26].

356

M. Uhlmann, J. Fr¨ ohlich

3 Results 3.1 Flow Conﬁguration We consider the flow in a plane channel bounded by one solid wall and one slip wall (cf. Fig. 1). The following boundary conditions are used for velocity: y=0:

u = 0,

y=h:

u ·n = 0,

∂(u − u · n) =0 ∂n

(7)

along with ∂p/∂n = 0 on both walls. The flow is periodic in the x- and zdirections and driven by a negative streamwise pressure gradient. We impose a constant volume flow rate, i.e. the bulk velocity Ub is maintained constant. The Reynolds number corresponding to the flow around an individual sphere is defined as ReD = UbνD . In the following · denotes an average over time and planes parallel to the wall, and a normalization in wall units is indicated by the familiar superscript + .

slip wall ..... ..... ..... ..... ..... ..... ..... ..... ..... ....

... .............. ..... ..... ..... ..... ..... ..... ..... .

h . .....

no slip wall

y

. . . . . . . . . . . . . .... . ........ ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... .......... ...... . . . . . ..... . . ..... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... .......... .......... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ..... ...... ...... ..... . . . ..... . . . . . . . . . . . . ..... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... ......... .......... ......... . ..... . . . . . . . . . . . . . . ..... ........ ......... ........ ......... ........ ......... ........ ......... ........ ......... ........ ......... ........ .......... ............ ........ ... ....... .......

Lz

x

...... ..... ..... ...... ....... ...

z

Lx

Fig. 1. Schematic of the geometry for the case of turbulent ﬂow in a plane halfchannel

3.2 Flow Over an Array of Fixed Wall-Mounted Spheres A first set of simulations was run with immobile particles, hence a situation of a rough wall with a particular roughness pattern. This pattern is the same as used below, where some of the particles are allowed to move. Spherical particles were located near the solid wall of the plane halfchannel configuration of Fig. 1. Figure 2 shows the arrangement of the spheres in two layers: the first one located just above the solid wall and the other one underneath it such that only a small fraction protrudes into the fluid domain. This second layer is arranged in a staggered manner with respect to the top layer, and, therefore, it will prevent that the top layer will simply “slide away”

Transport of Particles in Horizontal Channel Flow

↑ y

357

↑ z x −→ x −→ Fig. 2. Staggered two-layer particle arrangement used in the present study

in the subsequent cases with mobile particles. It should be noted that the distance between particle centers in each layer amounts to D + 2Δx in both the x- and z-direction with Δx being the step size of the Eulerian grid. Therefore, the particles in each layer do not touch other particles in the same layer. Also note that the bulk Reynolds number of the channel flow is defined with the fluid height above the top layer of the spheres, i.e. Reb = Ub (h−D) . Finally, ν the drag coeﬃcient is obtained from the following definition of the streamwise force: 1 (8) Fx = cD ρf Ub2 A , 2 with A = π(D/2)2 , and similarly for the remaining components. Fx is obtained from the immersed boundary method as the sum of all fluid-solid coupling contributions (i.e. a sum over all Lagrangian points in relation 6c above). Table 1 shows all the relevant parameters of our runs. These simulations were run at a bulk Reynolds number Reb = 2700 (the value of reference [27]). The time step was adjusted such that CF L ≈ 0.75. Table 1 also includes the two cases (number 13 and 16) of reference [25] for convenience, since they will be discussed along the present case (number 20) in the following. Note that the domain size in case 20 is substantially longer and wider than previously investigated, measuring 2424 × 202 × 303 wall units. For the present configuration, the streamwise momentum, averaged in time and over wall-parallel planes and integrated over the channel-height reads: Table 1. Parameters of the simulation of turbulent ﬂow over a rough wall. Note that the bulk Reynolds number is deﬁned as Reb = Ub (h − D)/ν case Lx /h × Ly /h × Lz /h Nx × Ny × Nz 13 16 20

1.5 × 1 × 0.75 1.5 × 1 × 0.75 12 × 1 × 1.5

Np

Reτ

D/h D/∆x NL

384 × 257 × 192 36 445.8 0.2422 384 × 257 × 192 576 224.5 0.0547 3072 × 257 × 384 9216 202.3 0.0547

62 14 14

D + ∆x+

12076 108.0 1.74 616 12.3 0.88 616 11.1 0.8

358

M. Uhlmann, J. Fr¨ ohlich Npf ixed

∂x p h = −ν(∂y u)y=0 + /

i=1

0

01 = −τw = −u2τ

h

Fx(i) dy ,

(i)

(9)

2

where the sum runs over all fixed particles and Fx is the sum of all solidfluid coupling force contributions of the ith particle in the x-direction (cf. 6c). In (9) we associate the right-hand-side with the wall shear τw which is equal to the sum of the viscous drag on the wall and the total drag (i.e. viscous and pressure-induced) on the spheres (cf. [28]). The time-average of the wall-shear lets us compute a value for the friction velocity and Reτ (cf. Table 2). Figure 3(a) depicts the mean profile of the total shear stress τtot = u′ v ′ − ν∂y u after accumulation of tstat ub /h ≈ 71 in case 20. The variation approximately follows a straight line, appart from a small undershoot around y ≈ 0.1. For wall distances above the top layer of roughness elements, y > D a straight line is expected for theoretical reasons, so that the present statistics are reasonably converged. A comparable convergence has not been not achieved in cases 13 and 16, simulated in reference [25]. We will therefore not further discuss the Eulerian statistics obtained in that study. The remaining plots in Fig. 3 show the first and second moments of fluid velocity for case 20. Note that we should obtain a mean flow with a logarithmic region of the form [29]: u 1 (y − y0 )uτ +A (10) = log uτ κ ν

which means that, given a data-set for u = f (y), we have in principle 4 unknowns: uτ , κ, y0 and A. Here we will assume the standard value for the slope (κ = 0.41) and use uτ as defined in (9). Concerning the oﬀset of the origin, y0 , it will lie in the range 0 ≤ y0 ≤ D [28]. Using a visual fit of the data, we obtain the values for y0 and A shown in Table 2; the graph of the logarithmic law is given in Fig. 3(c). The mean velocity profile (cf. Fig. 3b) as well as the spanwise velocity fluctuation intensity (Fig. 3d) exhibit a slight ‘dip’ around y/h = 0.2, which will require further investigation in the future. Table 2. Results of the simulation of turbulent ﬂow over a rough wall. Please note that the results for cases 13 and 16 cannot be considered as converged in a statistical sense, due to insuﬃcient sample case

y0

A

13 0.210 −2.5 16 0.035 +2 20 0.035 +2.5

tstat ub /h 59.2 56.9 71.0

fully rough transitionally rough transitionally rough

Transport of Particles in Horizontal Channel Flow (b)

(a) 0

1

−0.2

0.8

−0.4

+ τtot

359

0.6

u

−0.6

0.4

−0.8

0.2

−1 0

0.2

0.4

0.6

0.8

0

1

0

0.2

0.4

y/h (c)

0.6

0.8

1

0.6

0.8

1

y/h (d)

18

3

16 2.5

14 12

2

10

u+

u′α u′α +

8 6

1.5 1

4 0.5

2 0 0 10

1

(y − y0 )

0

2

10

0

10

0.2

0.4

+

y/h (e) 0.5

0

u′ v ′ + −0.5

−1 0

0.2

0.4

0.6

0.8

1

y/h Fig. 3. Statistics from the simulation of wall-mounted spheres in a half-plane channel, case 20. The log-law u+ = 2.5 log((y − y0 )+ ) + A in (c) is ﬁtted as indicated in Table 2. The components of the velocity ﬂuctuations in (d) are: , α = 1; , α = 2; ,α=3

Histograms for the three force components acting upon the top layer of spheres in the two simulations are shown in Fig. 4 in terms of the drag coeﬃcient cD defined in (8) and lift coeﬃcients cLy , cLz defined analogously. The curve for the spanwise component is symmetric with respect to the origin, as expected. Both lift and drag show a bias towards positive values, i.e. forces which would tend to induce forward and upward motion in free particles.

360

M. Uhlmann, J. Fr¨ ohlich

probability

0.2

0.1

0 −0.05

0

0.05

c∗ Fig. 4. Normalized histograms of the three force components acting upon the top layer of spheres in the simulation of turbulent ﬂow over spherical roughness elements in case 20. The lines correspond to: , cD ; , cLy ; , cLz . Note that the coeﬃcients are deﬁned with the bulk velocity, e.g. Fx = 1/2πD2 /4u2b cD

It is interesting to compare these forces with those necessary to induce actual particle motion would the particles be free. As will be seen below in the simulations with free particles, erosion is initiated not by vertical motion but by particle motion following the surface of the fixed bed [30]. This means that we need to consider the forces acting in the tangential direction at the contact point between the two layers of spheres. The projection of this direction into the (x,y)-plane forms an angle α = 35.3◦ with the horizontal. The resulting force in this direction then reads (cf. schematic in Fig. 5): Ft = Fx cos(α) + Fy sin(α) .

(11)

The histogram in Fig. 6(a) shows the sum of the drag and lift contributions to (11) and compares it to −G sin(α), the gravitational contribution in tangential direction for the particles used in § 3.3. It can be seen that suﬃciently strong hydrodynamical forces occur in order to overcome the gravitational F. y

Fx

. ...... t ..... ..... ......... . . . . . . . . . . . . . . . . . . . . . . . . . . . ......... ... ..... ...... ........ ..... ..... .......... ...... ..... .......... .... ... ..... .......... . ... . . . . . . . .. ... ..... ... .. ..... ...... .. ... ...... ..... .. .... ...... .... ................... . ... . .... .......... .. ... .. ... .. ... . ... ... ... ... .. .. ..... . .. ... ... .......... ... ... . .... .... . . . ...... ... . .. . . ........ . .. . . . . . ............................... .. . ..... ..... ..... . ..... ..... . . . .. .....

F

α

G

Fig. 5. Schematic of the forces acting upon a particle in the (x,y)plane and the resulting force Ft in the direction tangential to the surface of the supporting particle at the point of contact forming an angle α with the horizontal plane

Transport of Particles in Horizontal Channel Flow (a)

(b) 0.2

probability

0.2

probability

361

0.1

0

0.1

0 −0.05

0

ctan

0.05

−0.05

0

0.05

ctan

Fig. 6. (a) Normalized histograms of the hydrodynamical force component in the tangential direction at the contact points between the two layers, Ft = Fx cos(α) + Fy sin(α), acting upon the top layer of spheres in case 20. (b) The same as (a), but computed for the forces averaged over streamwise rows of particles. The dashed line corresponds to the gravitational force (projected onto the tangential direction) acting upon the particles simulated in the erosion experiment of § 3.3 below

force projected upon the sloped line. This would suggest that particle motion can be initiated in this case. However, further points might also play a–presumably minor–role. First, the force balance in (11) does not take into account frictional forces during particle contact. This would tend to delay the onset of particle motion. However, the simulations below do not consider this phenomenon either, as it is currently outside the scope of our study. Second, the historgram does not provide information about the duration of the force applied to the particle. Third, the histogram for the force balance is evaluated for individual spheres in the top layer as if they were isolated from their neighbors in that layer. In reality, particle motion over longer distances than the small gap between direct neighbors can only take place when a group of particles moves at the same time, since particles aligned in the streamwise direction are blocking each other. This basically means that the force balance needs to be evaluated collectively for a streamwise “row” of particles in order to obtain a histogram with relevance to such group motion, which brings into play its spatial correlation.3 Figure 6(b) shows the result when averaging the data over such rows. It can be seen that the number of events which have the potential to overcome the gravitational force in the tangential direction collectively is substantially smaller than indicated by the individual evaluation. The plot indicates that the spheres considered in § 3.3 would not be set into collective motion over distances larger than the inter-particle gap.

3

For inﬁntely long domains the force ﬂuctuations acting upon individual particles would obviously cancel out and only a suﬃciently strong mean ﬂow would be able to initiate the collective motion.

362

M. Uhlmann, J. Fr¨ ohlich

3.3 The Onset of Erosion In a second set of computations, the initial particle arrangement was the same as in § 3.2. In contrast, the positions of the top layer of spheres (those which have their center located at y = D/2) were not fixed, i.e. these particles can move freely under the influence of hydrodynamical and gravitational forces. The initial field for this series was taken from the simulation with fixed spheres at a time when the flow is considered fully developed. The simulations were then run for a certain time interval, say tobs . In the case of an observed particle motion, the simulation can be terminated; conversely, some upper bound for tobs needs to be set beyond which erosion is not expected to take place any more (or if it would do so, would be considered marginal anyway). Note that this time interval depends on the size of the box which is used, since the latter determines the number of potential turbulent “events” which take place in the numerical experiment in a given interval. The observation interval is given in Table 3. In reference [25] the value of the gravity was varied for a fixed density ratio ρp /ρf = 1.25, thereby allowing for an exploration of the bounds for the onset of erosion. The set of values for |g| is given in Table 3, where the two series of simulations of [25] (cases 13c–13e and 16c–16d) are included for comparison. In the present contribution we have performed two additional simulations, cases 20b and 20c, with parameters corresponding to the fixed-sphere case 20. The former case (20b) is equivalent to case 16e of [25], except for the enlarged domain size. The latter (case 20c) is equivalent to case 20b, except that in addition to the lower layer, the positions of one spanwise row of particles in Table 3. Parameters for the simulation of sediment erosion. The terminal velocity was evaluated according to the procedure given in [25] case

D/h ρp /ρf |g|

u∞ tobs ub /h row-motion? saltation?

13c of [25] 13d of [25] 13f of [25] 13e of [25]

0.2422 0.2422 0.2422 0.2422

1.25 1.25 1.25 1.25

0.15 0.20 0.30 0.40

0.130 0.160 0.213 0.262

16c of [25] 16g of [25] 16f of [25] 16e of [25] 16d of [25]

0.0547 0.0547 0.0547 0.0547 0.0547

1.25 1.25 1.25 1.25 1.25

0.20 0.40 0.60 0.80 1.00

0.026 0.043 0.058 0.071 0.083

20b 20c

0.0547 1.25 0.80 0.071 0.0547 1.25 0.80 0.071

16.9

y y y n

y y y n

22.2

y y y y n

y y y y n

25 20

y n

n y

spanwise row ﬁxed

Transport of Particles in Horizontal Channel Flow

363

the top layer was fixed in order to prevent the motion of entire streamwise rows of particles. The motion of the “most mobile” particles, i.e. those which reach the largest wall-normal distance in each case, is visualized in Fig. 7; the positions of all particles at a few selected instants in time are shown in Figs. 8 and 9. We observe non-negligible motion in both cases 20b and 20c, in correspondence with our earlier results in a smaller computational domain [25]. However, the nature of the particle motion diﬀers between the present two cases. In case 20b, the most mobile particle travels relatively small distances during a period of approximately 5 bulk time units, corresponding to a collective sliding motion over the protrusions formed by the lower particle layer, before again coming to rest. No saltation of individual particles is observed in this case during the current observation interval. We recall that in a smaller computational domain, the corresponding case (16e) did show saltation after an initial phase of collective sliding. A priori it was expected that collective sliding would not take place in the current much longer box, due to decorrelation of coherent structures over the streamwise period. Also, the evaluation of the force histogram for fixed spheres, averaged over streamwise rows of particles (cf. Fig. 6b) indicated that the probability for the onset of sliding motion (b)

(a)

yc D

3.5

3.5

3

3

2.5

2.5

2

2

1.5

1.5

1

1

0.5

0.5

0 75

80

85

90

95

100

0 75

80

tub /h

85

95

(d)

(c) yc D

90

tub /h 20

1 0.5

10

0 1

2

3

4

xc /D

5

6

7

0 120

140

160

180

200

220

xc /D

Fig. 7. Motion of the “most mobile” particle (i.e. the one which reaches the highest value of yc during the observation time). a) and c) case 20b exhibiting row-motion b) and d) case 20c with saltation. The plots in (a) and (b) show the temporal evolution of the y-coordinate of the particle’s center (solid line) and the height of the “obstacles” (dashed line); (c) and (d) show the particle trajectories in the (x, y)-plane. Note that the scale diﬀers between those two latter plots since the vertical range was increased proportional to the horizontal scale in order to convey an undistorted image of the trajectory

364

M. Uhlmann, J. Fr¨ ohlich (a)

(b)

Fig. 8. Particle positions during the initial stages of the simulation of case 20b. In (a), the ﬂow is from left to right; in (b) the mean ﬂow direction is away from the observer. The arrows in (b) point to the row of particles which is set into motion

of the current particles is negligible. However, as it turns out, this motion is observed, but only over a short interval, during which no saltation of individual particles takes place, probably due to the absence of local boundary layer events with a suﬃciently strong lift component. In case 20c, the onset of the particle motion is again in form of collective sliding (up to tub /h ≈ 80) whence the streamwise motion of the two streamwise-organized trains is blocked by the fixed spanwise row of particles (i.e. particles lined up in each ’train’ are touching each other). At this point, saltation of two pairs of particles first sets in, followed by additional particles which are entrained by the flow and rapidly carried downstream. While it has already been established in [25] that the initial sliding motion is initiated by high-shear regions located underneath of high-speed streaks, the type of coherent flow structures responsible for the saltation has not been determined. This question will be addressed in a future investigation.

Transport of Particles in Horizontal Channel Flow (a)

365

(b)

Fig. 9. As ﬁgure 8, but for case 20c

4 Performance of the Code During the initial stage of the present project, the performance of the code SUSPENSE on the NEC-SX8 platform at HLRS was evaluated in detail. Two bottlenecks were encountered. First, it was found that on this specific platform the solution of the implicit predictor step (6e) posed a severe restriction on the performance of the code. In the corresponding part of the code, approximate factorization is used to reduce the 3D matrix inversion to a sequence of 1D tridiagonal problems, which are solved by the parallel method of [31]. Although all significant loops were vectorized and function calls were inlined, no more than 800MFlops per processor could be obtained for representative configurations. As a comparison, switching to an explicit solution of the predictor step increased the performance to approximately 5.6GFlops. We recall that this latter case still involves the iterative solution of the Poisson problem for pressure by means of a multi-grid procedure on a 3D Cartesian processor grid. An explicit solution is, however, not feasible in production runs since the maximum time step for stable integration shrinks considerably. The reason for the observed bottleneck was strongly related to the particular method of parallel solution of the tridiagonal problems, where successive sweeps in one coordinate direction

366

M. Uhlmann, J. Fr¨ ohlich Table 4. Execution time per full time step on the NEC SX-8 at HLRS

test case nproc Nx × Ny × Nz A B

8 8

Np

texec MFlops Vector op. % Vector length

3072 × 257 × 384 0 28sec 3072 × 257 × 384 9216 114sec

2975 882

96.6 92.6

166 40

where performed (short data-length, large number of short messages). The problem has since been solved by a complete reimplementation of the algorithm, now carrying out loops over all three coordinate directions at the lowest functional level while minimizing the number of messages and increasing the vector length. This modified approach yields a performance of 3GFlops in the single-phase example A shown in Table 4. A second restriction on the present platform was detected when adding a substantial amount of particles to the flow, as in the test case B (see Table 4). It can be seen that the average vector length drops significantly, leading to a reduced performance of 880MFlops when considering 9216 particles. The reason for this loss of performance lies in the evaluation of the convolution sums in (5), which are in practice not performed over the full range of indices of the Eulerian grid, but only over the 3-node support of each discrete delta function. This procedure leads to overall savings in execution time (with respect to an evaluation of the loops over the entire Eulerian grid), but implies shorter vector lengths. No solution for overcoming the bottleneck due to this intrinsic feature of the present algorithm could be found so far.

5 Conclusion We have simulated turbulent flow over a rough wall formed by a closely packed two-layer bed of spheres (with diameters equal to approximately 11 wall units) at a bulk Reynolds number of 2700. Flow statistics were accumulated over an interval of 70 bulk time units in a computational box of considerable size, measuring approximately 2400 × 200 × 300 wall units. The simulations with fixed spheres have provided important information about the forces acting on the roughness elements. The projection of these forces upon the tangential direction at the contact point between the two layers of spheres needs to be larger than the corresponding gravitational component, in order to be able to initiate particle motion along the surface. We have used histograms of these forces accumulated in our simulations for evaluating the probability of such motion: a lower bound for the onset of erosion is given by the probability of the motion induced by the forces acting upon an individual particle; an upper bound can be determined from the evaluation of the average forces acting upon streamwise trains of particles. The behavior which was observed

Transport of Particles in Horizontal Channel Flow

367

in our subsequent simulations with mobile particles was situated somewhere in between those two limits. We have performed direct numerical simulations where the top layer of spheres of the roughness cases can translate and rotate freely (according to the balance of hydrodynamic and gravitational forces). For particles with critical buoyancy (as determined in [25]) it has been observed that the collective motion in form of streamwise trains of particles only takes place rather seldomly in the present large box, probably due to the streamwise decorrelation of coherent structures. A full entrainment of individual spheres (saltation) was only observed in a simulation where the position of some of the spheres in the top layer of the bed was fixed, thereby blocking the otherwise occurring row-wise motion. In the future, the data from the fixed-sphere simulations will be analyzed with respect to their implications for the dynamics of turbulent flow over rough surfaces. In particular, there remain some open questions pertaining to the regime of transitionally rough surfaces [32]. Further analysis of the data generated up to this point will be particularly targeted at the mechanisms responsible for the entrainment of individual particles into the bulk flow. For this purpose we will analyze in detail the local coherent structures inducing the saltation process. It will also be checked whether the observed mechanism of erosion is robust with respect to perturbations of the initial bed configuration. In a next step we plan to study possible implementations for tangential contact forces in our formulation. This additional feature will help to assess the importance of friction in the process of erosion. Acknowledgements The attribution of CPU time on the SX-8 of HLRS and the support by its staﬀ is greatfully acknowledged. M.U. was supported by the Spanish Ministry of Education and Science under contract DPI-2002-040550-C07-04.

References ¨ [1] A. Shields. Anwendungen der Ahnlichkeitsmechanik und der Turbulenzforschung auf die Geschiebebewegung. Technical Report 26, Mitteilungen der Preußischen Versuchsanstalt f¨ ur Wasser- Erd- und Schiﬀbau, Berlin, 1936. [2] Y. Ni˜ no, F. Lopez, and M. Garcia. Threshold for particle entrainment into suspension. Sedimentology, 50:247–263, 2003. [3] D. Kaftori, G. Hetsroni, and S. Banerjee. Particle behaviour in the turbulent boundary layer. I. Motion, deposition and entrainment. Phys. Fluids, 7(5):1095–1106, 1995. [4] D. Kaftori, G. Hetsroni, and S. Banerjee. Particle behaviour in the turbulent boundary layer. II. Velocity and distribution proﬁles. Phys. Fluids, 7(5):1107– 1121, 1995.

368

M. Uhlmann, J. Fr¨ ohlich

[5] Y. Ni˜ no and M.H. Garcia. Experiments on particle-turbulence interactions in the near-wall region of an open channel ﬂow: implications for sediment transport. J. Fluid Mech., 326:285–319, 1996. [6] K.T. Kiger and C. Pan. Suspension and turbulence modiﬁcation eﬀects of solid particulates on a horizontal turbulent channel ﬂow. J. Turbulence, 3(19):1–17, 2002. [7] L. Zeng, S. Balachandar, and P. Fischer. Wall-induced forces on a rigid sphere at ﬁnite Reynolds numbers. J. Fluid Mech., 536:1–25, 2005. [8] N.A. Patankar, P.Y. Huang, T. Ko, and D.D. Joseph. Lift-oﬀ of a single particle in Newtonian and viscoelastic ﬂuids by direct numerical simulation. J. Fluid Mech., 438:67–100, 2001. [9] H.G. Choi and D.D. Joseph. Fluidization by lift of 300 circular particles in plane Poiseuille ﬂow by direct numerical simulation. J. Fluid Mech., 438:101– 128, 2001. [10] Y. Pan and S. Banerjee. Numerical investigation of the eﬀects of large particles on wall-turbulence. Phys. Fluids, 9(12):3786–3807, 1997. ´ ´ [11] M. Uhlmann. Etude des Ecoulements en Conduite. M´ethodes Num´ eriques et ´ Analyse. Habilitation ` a diriger des recherches, Ecole Centrale de Lyon – Universit´e Claude Bernard Lyon 1, 2006. [12] M. Uhlmann. An immersed boundary method with direct forcing for the simulation of particulate ﬂows. J. Comput. Phys., 209(2):448–476, 2005. [13] E.A. Fadlun, R. Verzicco, P. Orlandi, and J. Mohd-Yusof. Combined immersedboundary ﬁnite-diﬀerence methods for three-dimensional complex ﬂow simulations. J. Comput. Phys., 161:35–60, 2000. [14] C.S. Peskin. The immersed boundary method. Acta Numerica, 11:479–517, 2002. [15] A.M. Roma, C.S. Peskin, and M.J. Berger. An adaptive version of the immersed boundary method. J. Comput. Phys., 153:509–534, 1999. [16] M. Uhlmann. An improved ﬂuid-solid coupling method for DNS of particulate ﬂow on a ﬁxed mesh. In M. Sommerfeld, editor, Proc. 11th Workshop TwoPhase Flow Predictions, Merseburg, Germany, 2005. Universit¨ at Halle. ISBN 3-86010-767-4. [17] M.M. Rai and P. Moin. Direct simulation of turbulent ﬂow using ﬁnitediﬀerence schemes. J. Comput. Phys., 96:15–53, 1991. [18] K. H¨ oﬂer and S. Schwarzer. Navier-Stokes simulation with constraint forces: Finite-diﬀerence method for particle-laden ﬂows and complex geometries. Phys. Rev. E, 61(6):7146–7160, 2000. [19] H.H. Hu, N.A. Patankar, and N.Y. Zhu. Direct numerical simulation of ﬂuidsolid systems using the arbitrary Lagrangian Eulerian technique. J. Comput. Phys., 169:427–462, 2001. [20] R. Glowinski, T.W. Pan, T.I. Hesla, D.D. Joseph, and J. P´eriaux. A ﬁcticious domain approach to the direct numerical simulation of incompressible viscous ﬂow past moving rigid bodies: Application to particulate ﬂow. J. Comput. Phys., 169:363–426, 2001. [21] R. Glowinski, T.-W. Pan, T.I. Hesla, and D.D. Joseph. A distributed Lagrange multiplier/ﬁctitious domain method for particulate ﬂows. Int. J. Multiphase Flow, 25:755–794, 1999. [22] M. Uhlmann. New results on the simulation of particulate ﬂows. Technical Report No. 1038, CIEMAT, Madrid, Spain, 2004. ISSN 1135-9420.

Transport of Particles in Horizontal Channel Flow

369

[23] M. Uhlmann. Numerical simulation of particulate ﬂows: comparison of ﬁctitious domain methods with direct and indirect forcing. In H.I. Andersson and P.A. Krogstad, editors, Advances in Turbulence X, Proc. 10th Eur. Turb. Conf., pages 415–418, Trondheim, Norway, 2004. [24] M. Uhlmann and A. Pinelli. Performance of various ﬂuid-solid coupling methods for DNS of particulate ﬂow. In S. Balachandar and A. Prosperetti, editors, IUTAM Symposium on Computational Approaches to Multiphase Flows, volume 81 of Fluid Mechanics and its Applications, pages 215–223. Springer, 2006. [25] M. Uhlmann. Direct numerical simulation of sediment transport in a horizontal channel. Technical Report No. 1088, CIEMAT, Madrid, Spain, 2006. ISSN 1135-9420. [26] M. Uhlmann. Experience with DNS of particulate ﬂow using a variant of the immersed boundary method. In P. Wesseling, E. O˜ nate, and J. P´eriaux, editors, Proc. ECCOMAS CFD 2006, Egmond aan Zee, The Netherlands, 2006. TU Delft. ISBN 90-9020970-0. [27] J. Kim, P. Moin, and R. Moser. Turbulence statistics in a fully developed channel ﬂow at low Reynolds number. J. Fluid Mech., 177:133–166, 1987. [28] S. Leonardi, P. Orlandi, R.J. Smalley, L. Djenidi, and R.A. Antonia. Direct numerical simulations of turbulent channel ﬂow with transverse square bars on one wall. J. Fluid Mech., 491:229–238, 2003. [29] H. Schlichting and K. Gersten. Grenzschicht–Theorie. Springer, 9 edition, 1996. [30] J.R.L. Allen. Principles of Physical Sedimentology. George Allen & Unwin, London, 1985. [31] N. Mattor, T.J. Williams, and D.W. Hewett. Algorithm for solving tridiagonal matrix problems in parallel. Parallel Computing, 21:1769–1782, 1995. [32] J. Jim´enez. Turbulent ﬂows over rough walls. Ann. Rev. Fluid Mech., 36:173– 96, 2004.

Analysis of Turbulent Structures in a Czochralski System Using DNS and LES Predictions A. Raufeisen1 , M. Breuer2 , T. Botsch1 , and A. Delgado2 1

2

Process Engineering Department (VT), University of Applied Sciences Nuremberg, Wassertorstraße 10, 90489 Nuremberg, Germany [email protected] Institute of Fluid Mechanics (LSTM), University of Erlangen-Nuremberg, Cauerstraße 4, 91058 Erlangen, Germany [email protected]

Summary. A DNS of the turbulent ﬂow and heat transfer in an idealized Czochralski conﬁguration was conducted using a very ﬁne grid. The results show the irregular forming of buoyant thermal plumes as well as the occurrence of a large, stable vortical structure in the bulk of the melt. In the averaged ﬂow ﬁeld a B´enard-cell-like pattern can be recognized. The DNS data served also as a reference for the validation of the LES method. The LES computations were conducted using diﬀerent grid sizes, SGS models, and discretization methods. For relatively ﬁne grids and central diﬀerences, the results agree very well with the DNS. Using an upwind discretization introduces numerical errors. In combination with coarser grids, this leads to large deviations and even qualitative diﬀerences. However, overall the computational effort could be reduced signiﬁcantly by LES. Thus a compromise between accuracy achieved and eﬀort required has to be found.

1 Introduction The numerical simulation of complex flows including turbulence is a diﬃcult task. Highly accurate Direct Numerical Simulations (DNS) require a high spatial and temporal resolution and therefore use massive computational resources. Thus, parametric studies for optimization procedures are infeasible. To reduce the computational eﬀort for that purpose, turbulence modeling is necessary. However, it was shown that for anisotropic turbulence, which occurs in most complex flow systems, classical statistical models based on the Reynolds-Averaged Navier-Stokes (RANS) equations are not applicable. On the contrary, the Large-Eddy Simulation method (LES) has proven in many cases that it is capable of predicting complex flows in a very eﬃcient way. It was the aim of the present project to assess the accuracy of LES computations in relation to the computational eﬀort necessary.

372

A. Raufeisen et al.

As the validation case the flow and heat transfer in the melt during the Czochralski (Cz) process for the growth of silicon single crystals was chosen. In this process, the liquid silicon is contained in an open crucible heated from the side. The crucible is rotating, while the counterrotating crystal is slowly pulled from the melt. Due to this setup, centrifugal and Coriolis forces, buoyancy, and Marangoni convection [8] occur in the fluid as well as thermal radiation from the surfaces and the phase change due to crystallization. Furthermore, oxygen can be dissolved and transported in the melt [13]. This complex configuration generates a turbulent, three-dimensional and time-dependent flow [1]. To obtain reference data, a large DNS was computed first [11] due to the lack of suitable results from the literature. Then a variety of diﬀerent LES computations was conducted and compared with the reference [12]. In the following sections, the cases are described and the results are presented.

2 Problem Details Due to the limited space, only a basic description of the case is presented here. Details and the full set of equations can be found in [11]. The simulation model consists of an idealized Cz-crucible with a cylindrical geometry of 170 mm radius with a crystal of 50 mm radius on top. The flow and heat transfer in the melt are governed by the three-dimensional Navier-Stokes equations for an incompressible fluid expressing the conservation of mass, momentum and energy. Buoyancy is taken into account by the Boussinesq approximation. The simulations are conducted in a rotating frame of reference with the crucible as the reference system, thus additional source terms arise in the momentum equation due to Coriolis and centrifugal forces. The equations are non-dimensionalized with appropriate normalizations and read ∂ui = 0, ∂xi

(1)

∂P 1 ∂τijmol Gr ∂ui ∂(ui uj ) + =− − − T ∂t ∂xj ∂xi Re ∂xj Re2 Re2ω Reω ǫijk ωj uk − ǫijk ωj (ǫklm ωl xm ), Re Re2 ∂(ui T ) 1 ∂T ∂2T + = , ∂t ∂xi Re P r ∂xi ∂xi +

(2) (3)

Thermal radiation from the free surface of the melt is considered by the Stefan-Boltzmann equation. Marangoni convection is accounted for by stating a force balance across the free surface including temperature gradients. At the sidewalls of the crucible, a temperature profile is applied, which is interpolated from experimental data [7]. The crystal is fixed to the melting temperature of Si and is rotating in the opposite direction of the crucible. The

Analysis of Turbulence in Cz Using DNS and LES

373

Table 1. Dimensionless numbers Number Prandtl

Symbol Formula ν Pr α

Reynolds

Re

Grashof

Gr

Rc u b ν

Marangoni M a

κgR3 c ∆T ν2 dσ Rc ∆T dT µα

Rayleigh

GrP r

Ra

Value 0.0128 4.7 × 104 2.21 × 109 −2.82 × 104 2.83 × 107

relevant material properties of Si can be found in [11]. From these values and the boundary conditions, the dimensionless numbers compiled in Table 1 are derived. The cylindrical geometry of the crucible was spatially discretized by a block-structured mesh of o-grid type to avoid the singularities of a fully radial or Cartesian grid. The mesh is divided into 8 blocks for parallelization and to fit the system architecture of the NEC SX-8, whose computational nodes contain 8 processors each. To resolve all turbulent scales in the DNS, a very fine grid was constructed containing almost 8.4 × 106 control volumes (CVs). The grid was refined towards the walls to be able to resolve the thin boundary layers. Furthermore, smoothing algorithms were employed to rule out sharp transitions, especially between the o-grid blocks, to ensure a fast convergence and avoid numerical errors. To verify the suﬃciency of the resolution, the Kolmogorov criterion for the characteristic length of the smallest turbulent scales hmax was computed using the dissipation rate and applying the additional estimations by Pope [10], and then compared with the actual cell width of the grid h. The result can be seen in Fig. 1. This graph depicts a vertical cut through the half crucible, with the data averaged in circumferential direction. The crucible axis is located on the left, the sidewall on the right side. The crystal resides on the top, extending from the axis to r/Rc = 0.294.

Fig. 1. Quotient of the actual and maximum mean cell edge length h/hmax . Calculated from circumferentially and time-averaged quantities

374

A. Raufeisen et al.

Values of h/hmax ≤ 1 mean that according to the estimates of Kolmogorov and Pope, respectively, all relevant turbulent scales of the flow are resolved. This is the case in the whole domain. Especially in the center of the crucible, the grid shows a very fine resolution with h/hmax ≤ 0.2. Near the walls and the free surface as well as below the crystal, the dissipation is very high, so that the values of h/hmax are bigger. Nevertheless, the computational cells are suﬃciently small there. The time step was chosen according to the spatial resolution as Δt = 2.5 × 10−4 to ensure numerical stability. This is quite small due to the explicit time marching scheme used here and thus easily fulfills the Kolmogorov criterion for the temporal resolution. For the LES computations, diﬀerent grids were constructed. For the first validation, a relatively fine grid containing ca. 106 CVs was chosen and different SGS models and discretization methods were applied (for details see next section). Also a case with a refined grid was computed (LES-BIG). To assess the achievable accuracy with LES on coarser grids, additional cases were simulated, taking approximately half the amount of CVs of the previous case. An overview of the conducted simulations is presented in Table 2. For more details refer to [12]. Table 2. Main parameters of DNS and LES predictions. (Smag. = Smagorinsky model with van Driest damping) Case Name Grid Size Nr × Nφ × Nz × Nblk DNS 128 × 128 × 64 × 8 LES-SMAG 64 × 64 × 32 × 8 LES-HLPA 64 × 64 × 32 × 8 LES-DYN 64 × 64 × 32 × 8 LES-BIG 64 × 64 × 40 × 8 LES-S 48 × 48 × 30 × 8 LES-XS 36 × 36 × 22 × 8 LES-XXS 30 × 30 × 18 × 8

Discretization CDS-2 CDS-2 HLPA CDS-2 CDS-2 HLPA HLPA HLPA

SGS model Time Step Size [–] – 2.5 × 10−4 Smag. 10−3 Smag. 10−3 Dynamic 10−3 Smag. 10−3 Smag. 1.5 × 10−3 Smag. 2 × 10−3 Smag. 3 × 10−3

3 Numerical Method In the LES, the Navier-Stokes equations are filtered in space, i.e. the flow quantities are divided into a “grid-scale” and a “subgrid-scale” (SGS) part. Here this is done implicitly, i.e. the filter width is the cell width of the computational grid. After filtering and reformulation one obtains the Navier-Stokes equations for the large scales and additionally a subgrid-scale stress tensor τijSGS and heat flux qiSGS , which have to be approximated. Using Smagorinsky’s model [15], this is done by determining a turbulent eddy viscosity μT

Analysis of Turbulence in Cz Using DNS and LES

375

using the so-called Smagorinsky constant Cs and calculating a turbulent eddy diﬀusivity kt using a turbulent Prandtl number P rt . Here, these values are set to Cs = 0.1 and P rt = 0.9, which has proven to deliver good results in practical applications. In the near-wall region the SGS stress tensor must tend to zero. To achieve this, the van Driest damping function is used. The idea of the dynamic procedure invented by Germano et al. [6] is to dynamically determine the SGS model parameters (e.g. Cs if the Smagorinsky model is chosen as the base model) from the smallest resolved scales of the filtered domain. Therefore, another (coarser) filter function, a so-called testfilter (TF), is applied to the basic equations. Thus, another subgrid-scale stress tensor arises, which is used to determine Cs dynamically during the simulation. For the numerical simulations the general-purpose CFD package LESOCC (Large–Eddy Simulation On Curvilinear Coordinates) [2, 3, 4, 5] is used. LESOCC is based on a 3–D finite–volume method for arbitrary non–orthogonal and block–structured grids. All viscous fluxes are approximated by central diﬀerences of second–order accuracy, which fits the elliptic nature of the viscous eﬀects. As shown in [3, 5] the quality of LES predictions is strongly dependent on low–diﬀusive discretization schemes for the non–linear convective fluxes in the momentum equation. Although several schemes are implemented in the code, the central scheme of second–order accuracy (CDS–2) is preferred for the LES predictions in the present work. For testing purposes, the HLPA scheme (Hybrid Linear/Parabolic Approximation) by Zhu [17] was also used. It combines a second-order upstream-weighted approximation with first-order upwind diﬀerencing under the control of a convective boundedness criterion. Time advancement is performed by a predictor–corrector scheme. A low– storage multi–stage Runge–Kutta method (three sub–steps, second–order accuracy) is applied for integrating the momentum equations in the predictor step. Within the corrector step the Poisson equation for the pressure correction is solved implicitly by the incomplete LU decomposition method of Stone [16]. Explicit time marching works well for LES and DNS with small time steps which are necessary to resolve turbulence motion in time. The pressure and velocity fields on a non–staggered grid are coupled by the momentum interpolation technique of Rhie and Chow [14]. A variety of diﬀerent test cases (see, e.g., [2, 3, 4, 5]) served for the purpose of the code validation. The algorithm is highly vectorized and additionally parallelized by domain decomposition with explicit message–passing based on MPI (Message Passing Interface) allowing eﬃcient computations especially on vector–parallel machines and SMP (Symmetric Multi–Processing) clusters. Due to its recursive data structure, the SIP solver for the algebraic system of equations in LESOCC is not vectorizable in a straightforward manner. However, Leister and Peri´c [9] showed that vectorization of the SIP solver can be achieved by avoiding data dependencies through indirect addressing and sweeping through the computational domain along diagonal planes, so called hyper-planes. Thus one sweep through the entire domain consists of hyper-planes having diﬀerent vector

376

A. Raufeisen et al.

length. Due to this variable vector length and the indirect addressing used, the performance of the vectorized SIP solver is slightly lower than the other parts of the code. The performance data of the diﬀerent cases are listed in Table 3. The DNS achieved a very high average computing eﬃciency of 8.2 GFlop/s per processor (more than 50% of the peak performance) on 8 processors (1 node) of the NEC SX-8. The same case distributed on 2 nodes / 16 processors delivered a speedup factor of ca. 1.7 (theoretically 2). This decreased performance is due to the slower interconnect between the nodes and the smaller number of CVs per processor. The whole simulation with more than 4 million time steps computed took more than 1000 hours of real time. The LES cases featuring 1 million CVs achieved a speedup of approximately 20 compared to the DNS, whereas theoretically, due to the grid being 8 times coarser (twice in each spatial direction) and the time step being 4 times larger, an increase in performance of a factor of 32 was calculated. This is mostly caused by a less eﬃcient calculation owing to the reduced number of cells per processor (only about 6.2 GFlop/s). Also the solution of additional equations from the LES procedure, e.g. to compute the eddy viscosity, has an eﬀect, especially in case of the dynamic model. The other LES cases were computed on the Linux Cluster at the RRZE in Erlangen, using diﬀerent systems (Xeon or Opteron) with diﬀerent interconnects dependent on availability. Due to this variety, the performance data of the computations are widespread, but on the average, the LES computations on the same grid using the same SGS model took approximately 20–25 times longer on the Linux Cluster than on the NEC SX-8 owing to the slower processors and smaller bandwidth of the systems. For the LES computations correction factors were calculated accounting for the diﬀerent system architectures. As a result, the diﬀerent LES cases showed a speedup of around three after coarsening. The real factor does not deviate much from the theoretical value due to the fact that for the LES predictions the same model and discretization was chosen. For LES-XXS, the real speedup factor is even higher than the calculated one, which can be attributed to an increase in performance for certain optimum amounts of CVs per processor. If the speedup factors are accumulated, it can be seen that the smallest LES case is theoretically almost 780 times (more than 460 times real) faster than the Table 3. Computing times of DNS and LES predictions Case Name Grid Size Time Step Speedup [-] theor. accum. real accum. DNS 8,388,608 2.5 × 10−4 1 1 1 1 LES-HLPA 1,048,576 10−3 32 32 20.5 20.5 2.8 91.0 2.6 53.6 LES-S 552,960 1.5 × 10−3 3.2 294.2 3.5 188.3 LES-XS 228,096 2 × 10−3 LES-XXS 129,600 3 × 10−3 2.6 776.7 2.5 461.5

Analysis of Turbulence in Cz Using DNS and LES

377

DNS. Thus, it is possible to save huge amounts of computational time using LES, however, only in connection with a loss in accuracy, as will be discussed in the next section.

4 Results and Discussion 4.1 DNS The instantaneous quantities of the DNS demonstrate the turbulent nature of the flow and heat transfer in the Si melt. The typical irregular forming of buoyant plumes rising from the bottom of the crucible to the free surface can be observed in Fig. 2. Around the center of the crucible, a large vortical structure forms, which can be identified in the picture series of pressure isosurfaces and streamlines in Fig. 4. This is caused by the interaction of the crucible rotation inducing centrifugal and Coriolis forces and the buoyancy, leading to a spiral or swirling flow directed towards the center and the bottom. This structure remains stable throughout the whole process, moving very slowly with respect to the crucible in the same direction of rotation (ca. 1 revolution in 15 crucible revolutions). Furthermore, in Fig. 3 the very small turbulent scales of the velocity field can be observed in contrast to the larger thermal scales. Due to the turbulence, the flow field breaks up into very small vortices where the kinetic energy is finally dissipated into heat. To obtain statistical data, the flow quantities were averaged in time and additionally in circumferential direction to establish axisymmetry. In the averaged flow field, a cellular structure as in Rayleigh-B´enard convection can be identified (see Fig. 5(c)). Three large convection rolls are formed in the bulk of the melt and a secondary vortex at the top of the sidewall. Furthermore, by the rotation thin Ekman layers at the crucible bottom and the crystal are built.

Fig. 2. Instantaneous temperature distribution at the free surface, showing the appearance of thermal plumes. In the center, the cold crystal is located

378

A. Raufeisen et al.

Fig. 3. Instantaneous temperature distribution and velocity ﬁeld (using the Line Integral Convolution method LIC) in a vertical cut through the crucible (y = 0) depicting the small velocity structures in contrast to the larger temperature scales

The mean temperature field exhibits a very smooth transition from the hot crucible sidewall to the cold crystal with almost diﬀusive character. This emphasizes the fact that the turbulent thermal scales are quite large. The RMS temperature fluctuations show their maximum at the crystal corner. This is caused by the large temperature diﬀerence between the free surface and the crystal, which induces a strong Marangoni flow at the surface, counteracting the centrifugal forces due to the rotation of the fluid. At the walls, the fluctuations tend to zero owing to the fixed temperature profile. The turbulent kinetic energy, depicted in Fig. 5(d), is defined as the trace of the Reynolds stress tensor: k = 1/2u′i u′i and is a good measure of the overall contents of turbulence in the melt due to flow. As expected from the plots of the single fluctuations, the maximum turbulent kinetic energy is located at the free surface and below the crystal, where the highest shear rates occur and the flow is highly instationary. At the walls, the values of k are very low due to the boundary layers. All details of the DNS can be found in [11]. 4.2 LES Firstly, the LES predictions on the fine grid (106 CVs) are analyzed. Three variants were computed: One using the Smagorinsky model with van Driest damping (LES-SMAG), one using the dynamic model (LES-DYN), and one also using Smagorinsky/van Driest, but with the HLPA discretization instead of CDS-2 (LES-HLPA). The latter was computed to smooth minor numerical oscillations, which had occurred in the vertical velocity component of LES-

Analysis of Turbulence in Cz Using DNS and LES

379

Fig. 4. Instantaneous pressure contours and streamlines in the melt in a horizontal cut through the crucible (z/Rc = 0.25). The crucible is rotating in counter-clockwise direction, the pictures show the movement relative to it. Intervals of 10 dimensionless time units (≈ 18 s or 1.5 crucible revolutions) are between consecutive pictures (from top left to bottom right line by line), giving a total time of 50 dimensionless units or ≈ 90 s or 7.5 rev., while the large vortical structure moves less than one half revolution relative to the crucible

380

A. Raufeisen et al.

Fig. 5. DNS computation: Circumferentially and time-averaged temperature ﬁeld p T ′ T ′ , streamlines of the mean ﬂow and T , RMS temperature ﬂuctuations turbulent kinetic energy k in the melt in a vertical cut through the center of the crucible (axis of symmetry on the left border, crucible wall on the right border)

Analysis of Turbulence in Cz Using DNS and LES

381

SMAG. Additionally, for the same reason a case with a grid refined in zdirection was run using CDS-2 and Smagorinsky/van Driest. The instantaneous data agree well with the DNS results regarding the main flow phenomena: Buoyant plumes dominate the flow and heat transfer in the melt, and also the occurrence of a large, slowly moving structure could be reproduced, even with the same angular velocity. Furthermore, the 2D contour plots of the averaged velocities, temperature, and fluctuations show a good overall agreement. However, this analysis can only lead to qualitative statements. For a detailed quantitative assessment, data were extracted at selected horizontal and vertical lines, e.g. at the critical position r/Rc = 0.26 close to the crystal edge, see Fig. 6, in a horizontal cut close to the surface at z/Rc = 0.46 and almost in mid-height at z/Rc = 0.32, see Fig. 7. The graphs show a good accordance between the LES and DNS results. LES-SMAG and LES-DYN exhibit almost the same values, which proves that the resolution of the grid is very fine for LES, such that the SGS model does not have much influence on the results. LES-HLPA shows some larger deviations, which is due to the diﬀusive character of the upwind discretization scheme used. Contrarily, the results of LES-BIG (not shown here) are very accurate and also smooth, however, additional computational eﬀort is necessary due to the finer grid. For the LES predictions on coarser grids, always the HLPA discretization scheme was used due to the fact that with CDS-2, strong oscillations would occur and thus make the results unusable. However, as seen before, HLPA adds numerical errors due to numerical diﬀusion. In combination with the coarse grids, these become significant, as demonstrated in the line graphs in Fig. 8. Especially the turbulent quantities show large deviations and even qualitative diﬀerences, see the plot of k. Thus, some important flow features are not captured. For an extensive analysis refer to [12].

5 Summary and Conclusions The results of a DNS of an idealized Cz-system were presented, which is a good example for complex turbulent flow including rotation, buoyancy, and surface tension. This simulation was carried out in order to obtain reference data for the validation of the LES method, which is used for turbulence modeling in order to reduce the computational eﬀort of the predictions. The DNS was computed on a very fine grid containing almost 8.4 × 106 CVs using 1 node (8 processors) of the NEC SX-8 at the HLRS. Due to the highly vectorized code LESOCC, an outstanding performance of 8.2 GFlop/s per processor could be achieved. However, the simulation had to be run for more than 4.5 × 106 time steps in order to obtain well-averaged data. The results show the characteristic turbulent flow structure featuring buoyant thermal plumes. Furthermore, a large vortical structure rotating faster than the crucible could be recognized. The averaged data exhibit a B´enard cell-like convection pattern consisting of four vortices.

382

A. Raufeisen et al.

0.4

0.4

0.3

0.3

z/R c

0.5

z/R c

0.5

0.2

0.2

DNS LES-SMAG LES-DYN LES-HLPA

0.1

0.1

0

0 -0.5

-0.4

-0.3

-0.2

T

-0.1

0

0

0.02

(a) T

0.4

0.4

0.3

0.3

0.06

0.08

0.1

pTTRMS (b) T ′ T ′

0.12

z/R c

0.5

z/R c

0.5

0.04

0.2

0.2

0.1

0.1

0

0

-0.14

-0.12

-0.1

-0.08

-0.06

ur

-0.04

-0.02

0

-1

-0.8

-0.6

(c) ur

-0.4

-0.2

ut

0

0.2

0.4

(d) ut

0.4

0.4

0.3

0.3

z/R c

0.5

z/R c

0.5

0.2

0.2

0.1

0.1

0

0 0

0.01

0.02

k

(e) k

0.03

0.04

0.05

-0.02

-0.015

(f)

-0.01

urut u′r u′t

-0.005

0

Fig. 6. Comparison of DNS and three LES variants (see Tab.p2): Circumferentially and time-averaged temperature and RMS ﬂuctuations T , T ′ T ′ , mean radial and tangential velocity ur , ut , turbulent kinetic energy k and Reynolds shear stress u′r u′t in the melt in a vertical intersection line at r/Rc = 0.26

Analysis of Turbulence in Cz Using DNS and LES 0.3

383

0.1

DNS LES-SMAG LES-DYN LES-HLPA

0.2

0

-0.1

T

T

0.1

-0.2 0

DNS LES-SMAG LES-DYN LES-HLPA

-0.3 -0.1

0

0.2

0.4

r/R c

0.6

0.8

-0.4

1

0

0.2

(a) T , z/Rc = 0.32

0.4

r/R c

0.6

0.8

1

(b) T , z/Rc = 0.46

0.12

0.1

0.1

TTRMS

TTRMS

0.08

0.06

0.08

0.06

0.04 0.04 0.02

0.02

0

0 0

0.2

(c) 0.014

p

0.4

r/R c

0.6

0.8

1

0

T ′ T ′ , z/Rc = 0.32

0.2

(d)

p

0.4

r/R c

0.6

0.8

1

T ′ T ′ , z/Rc = 0.46

0.025

0.012

0.01

k

0.02

0.008

k

0.015

0.006 0.01

0.004 0.005

0.002

0

0

0

0.2

0.4

r/R c

0.6

(e) k, z/Rc = 0.32

0.8

1

0

0.2

0.4

r/R c

0.6

0.8

1

(f) k, z/Rc = 0.46

Fig. 7. Comparison of DNS and three LES variants (see Tab. p 2): Circumferentially and time-averaged temperature and RMS ﬂuctuations T , T ′ T ′ , and turbulent kinetic energy k in the melt in the horizontal intersection lines at z/Rc = 0.32 and z/Rc = 0.46.

384

A. Raufeisen et al. 0.1

0.3

DNS LES-HLPA LES-S LES-XS LES-XXS

0.2

-0.1

0

T

T

0.1

0

-0.2 -0.1 -0.3 -0.2 -0.4 -0.3 0

0.2

0.4

r/R c

0.6

0.8

1

0

(a) T , z/Rc = 0.32

0.2

0.4

r/R c

0.6

0.8

1

0.8

1

(b) T , z/Rc = 0.46 0.12

0.1

0.1

TTRMS

TTRMS

0.08

0.06

0.08

0.06

0.04 0.04 0.02

0.02

0

0 0

(c)

0.2

p

0.4

r/R c

0.6

0.8

1

0

T ′ T ′ , z/Rc = 0.32

(d)

0.02

0.02

0.015

0.015

0.4

r/R c

0.6

T ′ T ′ , z/Rc = 0.46

k

0.025

k

0.025

0.2

p

0.01

0.01

0.005

0.005

0

0 0

0.2

0.4

r/R c

0.6

(e) k, z/Rc = 0.32

0.8

1

0

0.2

0.4

r/R c

0.6

0.8

1

(f) k, z/Rc = 0.46

Fig. 8. Comparison of coarse grid LES (LES-HLPA, LES-S, LES-XS, LES-XXS): Circumferentially and time-averaged temperature and RMS ﬂuctuations T and p T ′ T ′ and turbulent kinetic energy k in the melt in horizontal intersection lines

Analysis of Turbulence in Cz Using DNS and LES

385

The LES predictions were carried out on diﬀerent grids using diﬀerent SGS models to assess the achievable accuracy in relation to the computational effort needed. The fine grid computations (ca. 106 CVs) agree very well with the reference DNS data. Only the usage of an upwind discretization scheme (HLPA) shows some deviations. However, in the simulations on coarser grids (down to less than 130,000 CVs) large aberrations and even qualitative differences appear, especially in the turbulent quantities. Thus, it can be concluded that LES is an eﬃcient method to compute complex turbulent flow like in Cz-melts, however, the grid resolution and the discretization scheme have to be chosen appropriately to achieve a reasonable accuracy. Acknowledgements The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number CZ/12761.

References 1. Basu, B., Enger, S., Breuer, M., and Durst, F. (2000) Three–Dimensional Simulation of Flow and Thermal Field in a Czochralski Melt Using a Block– Structured Finite–Volume Method, Journal of Crystal Growth, vol. 219, pp. 123–143 2. Breuer, M., Rodi, W. (1996) Large–Eddy Simulation of Complex Turbulent Flows of Practical Interest, In: Flow Simulation with High–Performance Computers II, ed. E.H. Hirschel, Notes on Numer. Fluid Mech., vol. 52, pp. 258–274, Vieweg Verlag, Braunschweig 3. Breuer, M. (1998) Large–Eddy Simulation of the Sub–Critical Flow Past a Circular Cylinder: Numerical and Modeling Aspects, Int. J. for Numer. Methods in Fluids, vol. 28, pp. 1281–1302 4. Breuer, M. (2000) A Challenging Test Case for Large–Eddy Simulation: High Reynolds Number Circular Cylinder Flow, Int. J. of Heat and Fluid Flow, vol. 21, no. 5, pp. 648–654 5. Breuer, M. (2002) Direkte Numerische Simulation und Large–Eddy Simulation turbulenter Str¨ omungen auf Hochleistungsrechnern, Habilitationsschrift, Universit¨ at Erlangen–N¨ urnberg, Berichte aus der Str¨ omungstechnik, ISBN: 3– 8265–9958–6, Shaker Verlag, Aachen 6. Germano, M., Piomelli, U., Moin, P., and Cabot, W.H. (1991) A Dynamic Subgrid-Scale Eddy Viscosity Muodel, Phys. Fluids A, vol. 3, pp. 1760–1765 7. Gr¨ abner, O., M¨ uller, G., Virbulis, J., and Tomzig, E. (2001) Eﬀects of Various Magnetic Field Conﬁgurations on Temperature Distributions in Czochralski Silicon Melts, Microelectronic Engineering, vol. 56, pp. 83–88 8. Kumar, V., Basu, B., Enger, S., Brenner, G., and Durst, F. (2003) Role of Marangoni Convection in Si-Czochralski Melts, Part II: 3D Predictions with Crystal Rotation, Journal of Crystal Growth, vol. 255, pp. 27–39

386

A. Raufeisen et al.

9. Leister, H.J. and Peri´c, M. (1993) Vectorized Strongly Implicit Solving Procedure for a Seven-Diagonal Coeﬃcient Matrix, Int. Journal for Heat and Fluid Flow, vol. 4, pp. 159–172 10. Pope, S.B. (2000) Turbulent flows, Cambridge University Press 11. Raufeisen, A., Breuer, M., Botsch, T., and Delgado, A. (2007) DNS of Rotating Buoyancy- and Surface Tension-Driven Flow, International Journal of Heat and Mass Transfer, submitted 12. Raufeisen, A., Breuer, M., Botsch, T., and Delgado, A. (2007) LES Validation of Turbulent Rotating Buoyancy– and Surface Tension–Driven Flow Against DNS, International Journal of Heat and Fluid Flow, submitted 13. Raufeisen, A., Jana, S., Breuer, M., Botsch, T., and Durst, F. (2007) 3D Computation of Oxygen Transport in Czochralski Crystal Growth of Silicon Considering Evaporation, Journal of Crystal Growth, vol. 303, pp. 146–149 14. Rhie, C.M., Chow, W.L. (1983) A Numerical Study of the Turbulent Flow Past an Isolated Airfoil with Trailing Edge Separation, AIAA Journal, vol. 21, pp. 1525–1532 15. Smagorinsky, J. (1963) General Circulation Experiments with the Primitive Equations, I, The Basic Experiment, Mon. Weather Rev., vol. 91, pp. 99–165 16. Stone, H.L. (1968) Iterative Solution of Implicit Approximations of Multidimensional Partial Diﬀerential Equations, SIAM Journal of Numerical Analyses, vol. 5, pp. 530–558 17. Zhu, J. (1991) A Low–Diﬀusive and Oscillation–Free Convection Scheme, Commun. Appl. Numerical Methods, vol. 7, pp. 225–232

Aeroacoustic Prediction of Jet and Slat Noise E. Gr¨ oschel, D. K¨onig, S. Koh, W. Schr¨ oder, and M. Meinke Institute of Aerodynamics, RWTH Aachen University, W¨ ullnerstraße zw. 5 u. 7, 52062 Aachen, Germany {e.groeschel,d.koenig}@aia.rwth-aachen.de

Summary. The ﬂow ﬁeld and the acoustic ﬁeld of various jet ﬂows and a high-lift conﬁguration consisting of a deployed slat and a main wing are numerically analyzed. The ﬂow data, which are computed via large-eddy simulations (LES), provide the distributions being plugged in the source terms of the acoustic perturbation equations (APE) to compute the acoustic near ﬁeld. The investigation emphasizes the core ﬂow to have a major impact on the radiated jet noise. In particular the eﬀect of heating the inner stream generates substantial noise to the sideline of the jet, whereas the Lamb vector is the dominant noise source for the downstream noise. Furthermore, the analysis of the airframe noise shows the interaction of the shear layer of the slat trailing edge and the slat gap ﬂow to generate higher vorticity than the main airfoil trailing edge shear layer. Thus, the slat gap is the more dominant noise region for an airport approaching aircraft.

1 Introduction In the recent years the emitted sound by aircraft has become a very contributing factor during the development process. This is due to the predicted growth of air-traﬃc as well as the stricter statutory provisions. The generated sound can be assigned to engine and airframe noise, respectively. The present paper deals with two specific noise sources, the jet noise and the slat noise. Jet noise constitutes the major noise source for aircraft during take-oﬀ. In the last decade various studies [5, 6, 12, 25] focused on the computation of unheated and heated jets with emphasis on single jet configurations. Although extremely useful theories, experiments, and numerical solutions exist in the literature, the understanding of subsonic jet noise mechanisms is far from perfect. It is widely accepted that there exist two distinct mechanisms, one is associated with coherent structures radiating in the downstream direction and the other one is related to small scale turbulence structures contributing to the high frequency noise normal to the jet axis. Compared with single jets, coaxial jets with round nozzles can develop flow structures of very diﬀerent topology, depending on environmental and initial conditions and, of course,

388

E. Gr¨ oschel et al.

on the temperature gradient between the inner or core stream and the bypass stream. Not much work has been done on such jet configurations and as such there are still many open questions [3]. For instance, how is the mixing process influenced by the development of the inner and outer shear layers What is the impact of the temperature distribution on the mixing and on the noise generation mechanisms The current investigation contrasts the flow field and acoustic results of a high Reynolds number cold single jet to a more realistic coaxial jet configuration including the nozzle geometry and a heated inner stream. During the landing approach, when the engines are near idle condition, the airframe noise becomes important. The main contributor to airframe noise are high-lift devices, like slats and flaps, and the landing gear. The paper focuses here on the noise generated by a deployed slat. The present study applies a hybrid method to predict the noise from turbulent jets and a deployed slat. It is based on a two-step approach using a large-eddy simulation (LES) for the flow field and approximate solutions of the acoustic perturbation equations (APE) [10] for the acoustic field. The LES comprises the neighborhood of the dominant noise sources such as the potential cores and the spreading shear layers for the jet noise and the slat cove region for the airframe noise. In a subsequent step, the sound field is calculated for the near field, which covers a much larger area than the LES source domain. Compared to direct methods the hybrid approach possess the potential to be more eﬃcient in many aeroacoustical problems since it exploits the diﬀerent length scales of the flow field and the acoustic field. To be more precise, in subsonic flows the characteristic acoustic length scale is definitely larger than that of the flow field. Furthermore, the discretization scheme of the acoustic solver is designed to mimic the physics of the wave operator. The paper is organized as follows. The governing equations and the numerical procedure of the LES/APE method are described in Sect. 2. The simulation parameters of the cold single jet and the heated coaxial jet are given in the first part of Sect. 3 followed by the description of the high-lift configuration. The results for the flow field and the acoustical field are discussed in detail in Sect. 4. In each section, the jet noise and the slat noise problem are discussed subsequently. Finally, in Sect. 5, the findings of the present study are summarized.

2 Numerical Methods 2.1 Large-Eddy Simulations The computations of the flow fields are carried out by solving the unsteady compressible three-dimensional Navier-Stokes equations with a monotoneintegrated large-eddy simulation (MILES) [7]. The block-structured solver

Aeroacoustic Prediction of Jet and Slat Noise

389

is optimized for vector computers and parallelized by using the Message Passing Interface (MPI). The numerical solution of the Navier-Stokes equations is based on an vertex centered finite-volume scheme, in which the convective fluxes are computed by a modified AUSM method with an accuracy is 2nd order. For the viscous terms a central discretization is applied also of 2nd order accuracy. Meinke et al. showed in [21] that the obtained spatial precision is suﬃcient compared to a sixth-order method. The temporal integration from time level n to n + 1 is done by an explicit 5-stage Runge-Kutta technique, whereas the coeﬃcients are optimized for maximum stability and lead to a 2nd order accurate time approximation. At low Mach number flows a preconditioning method in conjunction with a dual-time stepping scheme can be used [2]. Furthermore, a multi-grid technique is implemented to accelerate the convergence of the dual-time stepping procedure. 2.2 Acoustic Simulations The set of acoustic perturbation equations (APE) used in the present simulations corresponds to the APE-4 formulation proposed in [10]. It is derived by rewriting the complete Navier-Stokes equations as ∂p′ p′ 2 ′ + c¯ ∇ · ρ¯u + u = c¯2 qc ¯ 2 (1) ∂t c¯ ∂u′ p′ + ∇ (¯ u · u′ ) + ∇ = qm . (2) ∂t ρ¯ The right-hand side terms constitute the acoustic sources ′

qc = −∇ · (ρ′ u′ ) + qm

¯ ′ ρ¯ Ds cp Dt

(3)

′ ′ (u′ )2 ∇·τ ¯ = − (ω × u) + T ∇¯ s − s ∇T − ∇ . + 2 ρ ′

′

′

(4)

To obtain the APE system with the perturbation pressure as independent variable the second law of thermodynamics in the first-order formulation is used. The left-hand side constitutes a linear system describing linear wave propagation in mean flows with convection and refraction eﬀects. The viscous eﬀects are neglected in the acoustic simulations. That is, the last source term in the momentum equation is dropped. The numerical algorithm to solve the APE-4 system is based on a 7-point finite-diﬀerence scheme using the well-known dispersion-relation preserving scheme (DRP) [24] for the spatial discretization including the metric terms on curvilinear grids. This scheme accurately resolves waves longer than 5.4 points per wave length (PPW). For the time integration an alternating 5-6 stage low-dispersion low-dissipation Runge-Kutta scheme [15] is implemented. To eliminate spurious oscillations the solution is filtered using a 6th-order explicit commutative filter [23, 26] at every tenth iteration step. As the APE

390

E. Gr¨ oschel et al.

system does not describe convection of entropy and vorticity perturbations [10] the asymptotic radiation boundary condition by Tam and Webb [24] is suﬃcient to minimize reflections on the outer boundaries. On the inner boundaries between the diﬀerent matching blocks covering the LES and the acoustic domain, where the transition of the inhomogeneous to the homogeneous acoustic equations takes place, a damping zone is formulated to suppress artificial noise generated by a discontinuity in the vorticity distribution [22].

3 Computational Setup 3.1 Jet The quantities uj and cj are the jet nozzle exit velocity and sound speed, respectively, and Tj and T∞ the temperature at the nozzle exit and in the ambient fluid. Unlike the single jet, the simulation parameters of the coaxial jet have additional indices “p” and “s” indicating the primary and secondary stream. An isothermal turbulent single jet at Mj = uj /c∞ = 0.9 and Re = 400, 000 is simulated. These parameters match with previous investigations performed by a direct noise computation via an acoustic LES by Bogey and Bailly [6] and a hybrid LES/Kirchhoﬀ method by Uzun et al. [25]. The chosen Reynolds number can be regarded as a first step towards the simulation of real jet configurations. Since the flow parameters match those of various studies, a good database exists to validate our hybrid method for such high Reynolds number flows. The inflow condition at the virtual nozzle exit is given by a hyperbolic-tangent profile for the mean flow, which is seeded by random velocity fluctuations into the shear layers in form of a vortex ring [6] to provide turbulent fluctuations. Instantaneous LES data are sampled over a period of T¯ = 3000 · Δt · uj /R = 300.0 corresponding to approximately 6 times the time interval an acoustic wave needs to travel through the computational domain. Since the source data is cyclically fed into the acoustic simulation a modified Hanning windowing [20] has been performed to avoid spurious noise generated by discontinuities in the source term distribution. More details on the computational set up can be found in Koh et al. [17] The flow parameters of the coaxial jet comprises a velocity ratio of the secondary and primary jet exit velocity of λ = ujs /ujp = 0.9, a Mach number 0.9 for the secondary and 0.877 for the primary stream, and a temperature ratio of Tjs /Tjp = 0.37. An overview of the main parameter specifications is given in Table 1. To reduce the computational costs the inner part of the nozzle was not included in the simulation, but a precursor RANS simulation was set up to generate the inflow profiles for the LES. For the coaxial jet instantaneous data are sampled over a period of T¯s = 2000 · Δt · c∞ /rs = 83. This period corresponds to roughly three times the time interval an acoustic wave needs to propagate through the computational domain. As in the single jet computation, the source terms are cyclically inserted into the acoustic simulation.

Aeroacoustic Prediction of Jet and Slat Noise

391

Table 1. Flow properties coaxial jet Jet ﬂow conditions of the (p)rimary and (s)econdary stream notation dimension SJ CJ parameter M ap 0.9 Mach number primary jet M as 0.9 0.9 Mach number secondary jet U M aac 0.9 1.4 Acoustic Mach number ( c∞p ) Tp K 775 Static temperature primary jet Ttp K 879.9 Total temperature primary jet Ts K 288. 288. Static temperature secondary jet Tts K 335 335. Total temperature secondary jet T∞ K 288. 288. Ambient temperature Re 4 · 105 2 · 106 Reynolds number Fig. 1. The grid topology close to the nozzle tip is “bowl” shaped, i.e., grid lines from the primary nozzle exit end on the opposite side of the primary nozzle. Every second grid point is shown

The grid topology and in particular the shape of the short cowl nozzle are shown in Fig. 1. The computational grid has about 22 · 106 grid points. 3.2 High-Lift Conﬁguration Large-Eddy Simulation The computational mesh consists of 32 blocks with a total amount of 55 million grid points. The extent in the spanwise direction is 2.1% of the clean chord length and is resolved with 65 points. Figure 2 depict the mesh in the slat cove area. To assure a suﬃcient resolution in the near surface region of Δx+ ≈ 100, Δy + ≈ 1, and Δy + ≈ 22 [1] the analytical solution of a flate plate was used during the grid generation process to approximate the needed step sizes. On the far-field boundaries of the computational domain boundary conditions based on the theory of characteristics are applied. A sponge layer following Israeli et al. [16] is imposed on these boundaries to avoid spurious reflections, which would extremely influence the acoustic analysis. On the

392

E. Gr¨ oschel et al. Fig. 2. LES grid in the slat cove area of the high-lift conﬁguration. Every 2nd grid point is depicted

walls an adiabatic no-slip boundary condition is applied and in the spanwise direction periodic boundary conditions are used. The computation is performed for a Mach number of M a = 0.16 at an angle of attack of α = 13◦ . The Reynolds number is set to Re = 1.4 · 106 . The inital conditions were obtained from a two-dimensional RANS simulation. Acoustic Simulation The acoustic analysis is done by a two-dimensional approach. That is, the spanwise extent of the computational domain of the LES can be limited since especially at low Mach number flows the turbulent length scales are significantly smaller then the acoustic length scales and as such the noise sources can be considered compact. This treatment tends to result in somewhat overpredicted sound pressure levels which are corrected following the method described by Ewert et al. in [11]. The acoustic mesh for the APE solution has a total number of 1.8 million points, which are distributed over 24 blocks. Figure 3 shows a section of the

Fig. 3. APE grid of the highlift conﬁguration. Every 2nd grid point is depicted

Aeroacoustic Prediction of Jet and Slat Noise

393

used grid. The maximum grid spacing in the whole domain is chose to resolve 8 kHz as the highest frequency. The acoustic solver uses the mean flow field obtained by averaging the unsteady LES data and the time dependent perturbed Lamb vector (ω × u)′ , which is also computed from the LES results, as input data. A total amount of 2750 samples are used which describe a non-dimensional time period of T ≈ 25, non-dimensionalized with the clean chord length and the speed of sound c∞ . To be in agreement with the large-eddy simulation the Mach number, the angle of attack and the Reynolds number are set to M a = 0.16, α = 13◦ and Re = 1.4 · 106 , respectively.

4 Results and Discussion The results of the present study are divided into two parts. First, the flow field and the acoustic field of the cold single jet and the heated coaxial jet will be discussed concerning the mean flow properties, turbulent statistics and acoustic signature in the near field. To relate the findings of the coaxial jet to the single jet, the flow field and acoustic field of which has been validated in current studies [17] against the experimental results by [27] and numerical results by Bogey and Bailly [6], comparisons to the flow field and acoustic near field properties of the single jet computation are drawn. This part also comprises a discussion on the results of the acoustic near field concerning the impact by the additional source terms of the APE system, which are related to heating eﬀects. The second part describes in detail the airframe noise generated by the deployed slat and the main wing element. Acoustic near field solutions are discussed on the basis of the LES solution alone and the hybrid LES/APE results. 4.1 Jet Large-Eddy Simulation In the following the flow field of the single jet is briefly discussed to show that the relevant properties of the high Reynolds number jet are well computed when compared with jets at the same flow condition taken from the literature. In Fig. 4 the half-width radius shows an excellent agreement with the LES by Bogey and Bailly [6] and the experiments by Zaman [27] indicating a potential core length of approximately 10.2 radii. The jet evolves downstream of the potential core according to experimental findings showing the quality of the lateral boundary conditions to allow a correct jet spreading. Furthermore, 9 9 in Figs. 5 and 6 the turbulent intensities u′ u′ /u2j and v ′ v ′ /u2j along the center line rise rapidly after an initially laminar region to a maximum peak

394

E. Gr¨ oschel et al.

near the end of the potential core and decrease further downstream. The obtained values are in good agreement with those computed by Bogey and Bailly [6] and the experimental results by Arakeri et al. [4] and Lau et al. [19]. The self-similarity of the jet in Fig. 7 is well preserved. From these findings it seems appropriate to use the present LES results for jet noise analyses, which are performed in the next subsection. The flow field analysis of the coaxial jet starts with Fig. 8 showing instantaneous density contours with mapped on mean velocity field. Small vortical and slender ring-like structures are generated directly at the nozzle lip. Further downstream, these structures start to stretch and become unstable, eventually breaking into smaller structures. The degree of mixing in the shear layers between the inner and outer stream, the so-called primary mixing region, is generally very high. This is especially noticeable in Fig. 9 with the growing shear layer instability separating the two streams. Spatially growing vortical structures generated in the outer shear layer seem to aﬀect the inner shear layer instabilities further downstream. This finally leads to the collapse and break-up near the end of the inner core region. Figure 10 shows mean flow velocity profiles based on the secondary jet exit velocity of the coaxial jet at diﬀerent axial cross sections ranging from x/RS = 0.0596 to x/Rs = 14.5335 and comparisons to experimental results. A good agreement, in particular in the near nozzle region, is obtained, however, the numerical jet breaks-up earlier than in the experiments resulting in a faster mean velocity decay on the center line downstream of the potential core. The following three Figs. 11 to 13 compare mean velocity, mean density, and Reynolds stress profiles of the coaxial jet to the single jet in planes normal to the jet axis and equally distributed in the streamwise direction from x/Rs =

Fig. 4. Jet half-width radius in comparison with numerical [6] and experimental results [27]

q Fig. 5. Reynolds stresses u′ u′ /u2j normalized by the nozzle exit velocity in comparison with numerical [6] and experimental [4, 19] results

Aeroacoustic Prediction of Jet and Slat Noise

395

q Fig. 6. Reynolds stresses v ′ v ′ /u2j normalized by the nozzle exit velocity in comparison with numerical [6] results

q Fig. 7. Reynolds stresses u′ u′ /u2j normalized by the nozzle exit velocity over jet half-width radius at x/R = 22 in comparison with numerical [6] results

Fig. 8. Instantaneous density contours with mapped on velocity ﬁeld

Fig. 9. Instantaneous temperature contours (z/Rs = 0 plane)

1 to x/Rs = 21. In the initial coaxial jet exit region the mixing of the primary shear layer takes place. During the mixing process, the edges of the initially sharp density profile are smoothed. Further downstream the secondary jet shear layers start to break up causing a rapid exchange and mixing of the fluid in the inner core. This can be seen by the fast decay of the mean density profile in Fig. 12. During this process, the two initially separated streams merge and show at x/Rs = 5 a velocity profile with only one inflection point roughly at r/Rs = 0.5. Unlike the density profile, the mean axial velocity profile decreases only slowly downstream of the primary potential core. In the self-similar region the velocity decay and the spreading of the single and the coaxial jet is similar. The break-up process enhances the mixing process yielding higher levels of turbulent kinetic energy on the center line. The axial velocity fluctuations of the coaxial jet starts to increase at x/Rs = 1 in the outer shear layer and reach at x/Rs = 9 high levels on the center line, while the single jet axial

396

E. Gr¨ oschel et al.

Fig. 10. Mean ﬂow development of coaxial jet in parallel planes perpendicular to the jet axis in comparison with experimental results

Fig. 11. Axial velocity proﬁles for cold single jet and heated coaxial jet

Fig. 12. Density proﬁles for cold single jet and heated coaxial jet

Fig. 13. Reynolds stresses proﬁles for cold single jet and heated coaxial jet

fluctuations start to develop not before x/rs = 5 and primarily in the shear layer but not on the center line. This diﬀerence is caused by the density and entropy gradient, which is the driving force of this process. This is confirmed by the mean density profiles. These profiles are redistributed beginning at x/rs = 1 until they take on a uniform shape at approx. x/rs = 9. When this process is almost finished the decay of the mean axial velocity profile sets in. This redistribution evolves much slower over several radii in the downstream direction.

Aeroacoustic Prediction of Jet and Slat Noise

397

Acoustic Simulation The presentation of the jet noise results is organized as follows. First, the main characteristics of the acoustic field of the single jet from previous noise [13], [17] computations are summarized, by which the present hybrid method has been successfully validated against. Then, the acoustic fields for the single and coaxial jet are discussed. Finally, the impact of diﬀerent source terms on the acoustic near field is presented. Unlike the direct acoustic approach by an LES or a DNS, the hybrid methods based on an acoustic analogy allows to separate diﬀerent contributions to the noise field. These noise mechanisms are encoded in the source terms of the acoustic analogy and can be simulated separately exploiting the linearity of the wave operator. Previous investigations of the single jet noise demonstrated the fluctuating Lamb vector to be the main source term for cold jet noise problems. An acoustic simulation with the Lamb vector only was performed and the sound field at the same points was computed and compared with the solution containing the complete source term. The overall acoustic field of the single and coaxial jet is shown in Figs. 14 and 15 by instantaneous pressure contours in the near field, i.e., outside the source region, and contours of the Lamb vector in the acoustic source region. The acoustic field is dominated by long pressure waves of low frequency radiating in the downstream direction. The dashed line in Fig. 14 indicates the measurement points at a distance of 15 radii from the jet axis based on the outer jet radius at which the acoustic data have been sampled. Fig. 16 shows the acoustic near field signature generated by the Lamb vector only in comparison with an, in terms of number of grid points, highly resolved LES and the direct noise computation by Bogey and Bailly. The downstream noise is well captured by the LES/APE method and is consistent with the highly resolved LES results. The increasing deviation of the overall sound pressure level at obtuse angles with respect to the jet axis is due to missing contributions from nonlinear and entropy source terms. A detailed investigation can be found in Koh et al. [17]. Note that the results by Bogey and Bailly are 2 to 3 dB too high compared to the present LES and LES/APE distributions. Since diﬀerent grids (Cartesian grids by Bogey and Bailly and boundary fitted grids in the present simulation) and diﬀerent numerical methods for the compressible flow field have been used resulting resulting in varying boundary conditions, e.g.,the resolution of the initial momentum thickness, diﬀerences in the sensitive acoustic field are to be expected. The findings of the hybrid LES/Kirchhoﬀ approach by Uzun et al. [25] do also compare favorably with the present solutions. The comparison between the near field noise signature generated by the Lamb vector only of the single and the coaxial jet at the same measurement line shows almost the same characteristic slope and a similar peak value lo-

398

E. Gr¨ oschel et al.

Fig. 14. Pressure contours of the single jet by LES/APE generated by the Lamb vector only. Dashed line indicates location of observer points to compute the acoustic near ﬁeld

Fig. 15. Pressure contours outside the source domain and the y-component of the Lamb vector inside the source domain of the coaxial jet

Fig. 16. Overall sound pressure level (OASPL) in dB for r/R = 15. Comparison with data from Bogey and Bailly [6]

Fig. 17. Comparison of the acoustic ﬁeld between the single jet and the coaxial jet generated by the Lamb vector only. Comparison with data from Bogey and Bailly [6]

cation along the jet axis. This is surprising, since the flow field development of both jets including mean flow and turbulent intensities diﬀered strongly. Finally, Figs. 18 and 19 show the predicted far field directivity at 60 radii from the jet axis by the Lamb vector only and by the Lamb vector and the entropy source terms, respectively, in comparison with numerical and exper-

Aeroacoustic Prediction of Jet and Slat Noise

Fig. 18. Directivity at r/Rs = 60 generated by the Lamb vector only. Comparison with experimental and numerical results

399

Fig. 19. Directivity at r/Rs = 60 generated by the Lamb vector and entropy sources. Comparison with experimental and numerical results

imental results at the same flow condition. To obtain the far field noise signature, the near field results have been scaled to the far field by the 1/r-law assuming the center of directivity at x/Rs = 4. The acoustic results generated by the Lamb vector only match very well the experimental results at angles lower than 40 degree. At larger angles from the jet axis the OASPL falls oﬀ more rapidly. This deviation is due to the missing contributions from the entropy source terms. When including those source terms in the computation, the LES/APE are in good agreement with the experimental results up to angles of 70 degree. That observation confirms previous studies [14] on the influence of diﬀerent source terms. To be more precise, the Lamb vector radiates dominantly in the downstream direction, whereas the entropy sources radiate to obtuse angles from the jet axis. Computational Resources The determination of the acoustic field of the jet requires both an LES of the flow field and the solution of the acoustic perturbation equations. Both simulations were run on the high-performance computer NEC-SX8 installed at the HLRS of the university of Stuttgart. The overall computing time for the jet is determined by the requirement to resolve acoustic frequencies with a minimum Strouhal number of 0.05 and a maximum of 1.5, based on the jet diameter and the centerline velocity. To obtain a fully developed acoustical field we further required the lowest frequencies to pass approx. 7 times through

400

E. Gr¨ oschel et al.

a given point in the domain of integration. With a time step of about Δt=0.002 based on the speed of sound and the radius of the jet about 150.000 time steps for a full acoustic simulation are required. The total CPU time for one acoustic simulation is then about 100.000 single processor CPU hours. Additionally, the time for the LES of a jet amounts to about 50.000 CPU hours, so that the determination of the acoustic field needs 150.000 CPU hours. About 8 diﬀerent jet cases were investigated so that, including some trial solutions, in total approx. 1.500.000 hours were used within this research project. 4.2 High-Lift Conﬁguration Large-Eddy Simulation The large-eddy simulation has been run for about 5 non-dimensional time units based on the freestream velocity and the clean chord length. During this time a fully developed turbulent flow field was obtained. Subsequently, samples for the statistical analysis and also to compute the aeroacoustic source terms were recorded. The sampling time interval was chosen to be approximately 0.0015 time units. First of all, the quality of the results should be assessed on the basis of the proper mesh resolution near the walls. Figures 20 to 23 depict the determined values of the grid resolution and shows that the flat plate approximation yields satisfactory results. However, due to the accelerated and decelerated flow on the suction and pressure side, respectively, the grid resolution departs somewhat from the approximated values. In the slat cove region the resolution reaches everywhere the required values for large-eddy simulations of wall bounded flows (Δx+ ≈ 100, Δy + ≈ 2, and Δy + ≈ 20 [1]). The Mach number distribution and some selected streamlines of the time and spanwise averaged flow field is presented in Fig. 24. Apart form the two stagnation points one can see the area with the highest velocity on the suction side short downstream of the slat gap. Also recognizable is a large recirculation domain which fills the whole slat cove area. It is bounded by a shear layer which develops form the slat cusp and reattaches close to the end of the slat trailing edge. The pressure coeﬃcient cp computed by the time averaged LES solution is compared in Fig. 25 with RANS results [9] and experimental data. The measurements were carried out at DLR Braunschweig in an anechoic wind tunnel with an open test section within the national project FREQUENZ. These experiments are compared to numerical solutions which mimic uniform freestream conditions. Therefore, even with the correction of the geometric angle of attack of 23◦ in the measurements to about 13◦ in the numerical solution, no perfect match between the experimental and numerical data can be expected.

Aeroacoustic Prediction of Jet and Slat Noise

401

Fig. 20. Grid resolution near the wall: Suction side of the main wing

Fig. 21. Grid resolution near the wall: Pressure side of the main wing

Fig. 22. Grid resolution near the wall: Suction side of the slat

Fig. 23. Grid resolution near the wall: Slat cove

402

E. Gr¨ oschel et al.

Fig. 24. Time and spanwise averaged Mach number distribution and some selected streamlines

Fig. 25. Comparison of the cp coeﬃcient between LES, RANS [9] and experimental data [18]

Figures 26 to 28 show the turbulent vortex structures by means of λ2 contours. The color mapped onto these contours represents the Mach number. The shear layer between the recirculation area and the flow passing through the slat gap develops large vortical structures near the reattachment point. Most of these structures are convected through the slat gap while some vortices are trapped in the recirculation area and are moved upstream to the cusp. This behavior is in agreement with the findings of Choudhari et al. [8]. Furthermore, like the investigations in [8] the analysis of the unsteady data indicates a fluctuation of the reattachment point. On the suction side of the slat, shortly downstream of the leading edge, the generation of the vortical structures in Fig. 26 visualizes the transition of the boundary layer. This turbulent boundary layer passes over the slat trailing edge and interacts with the vortical structures convected through the slat gap. Figure 28 illustrates some more pronounced vortices being generated in the reattachment region and whose axes are aligned with the streamwise direction. The distribution of the time and spanwise averaged turbulent kinetic en ergy k = 12 u′2 + v ′2 + w′2 is depicted in Fig. 29. One can clearly identify the shear layer and the slat trailing edge wake. The peak values occur, in agreement with [8], in the reattachment area. This corresponds to the strong vortical structures in this area evidenced in Fig. 27. Computational Resources The simulations were carried out on the high-performance computer NECSX8 installed at the HLRS of the university of Stuttgart. A single computation was run for about 12 non-dimensional time units, where the simulation of one

Aeroacoustic Prediction of Jet and Slat Noise

403

Fig. 26. λ2 contours in the slat region

Fig. 27. λ2 contours in the slat region

Fig. 28. λ2 contours in the slat gap area

Fig. 29. Time and spanwise averaged turbulent kinetic energy in the slat cove region

time unit required approx. 180 CPU hours on 32 processors, i.e. 5760 single processor CPU hours. The resulting total computing time is therefore in the order of 70.000 single processor CPU hours for a single simulation. Due to grid refinement studies and the adaption of diﬀerent numerical parameters, several additional runs were carried out, which required additional CPU time in the order of 300.000 CPU hours. A total of 4000 data sets using 7 Terabyte of disk space have been collected which cover an overall time of approximately 6 non-dimensional time units. The maximum obtained floating point operations per second (FLOPS) amounts 6.7 GFLOPS, the average value was 5.9 GFLOPS. An average vectorization ratio of 99.6% was achieved with a mean vector length of 247.4.

404

E. Gr¨ oschel et al.

Fig. 30. Snapshot of the x-component of the Lamb Vector

Fig. 31. Snapshot of the y-component of the Lamb Vector

Acoustic Simulation A snapshot of the distribution of the acoustic sources by means of the perturbed Lamb vector (ω × u)′ is shown in Figs. 30 and 31. The strongest acoustic sources are caused by the normal component of the Lamb vector. The peak value occurs on the suction side downstream of the slat trailing edge, whereas somewhat smaller values are determined near the main wing trailing edge. Figures 32 and 33 illustrate a snapshot of the pressure fluctuations based on the APE and the LES solution. Especially in the APE solution the interaction between the noise of the main wing and that of the slat is obvious. A closer look reveals that the slat sources are dominant compared to the main airfoil trailing edge sources. It is clear that the LES mesh is not able to resolve the high frequency waves in some distance from the airfoil. The power spectral density (PSD) for an observer point at x = −1.02 and y = 1.76 compared to experimental results are shown in Fig. 34 [18]. The

Aeroacoustic Prediction of Jet and Slat Noise

405

Fig. 32. Pressure contours based on the LES/APE solution

Fig. 33. Pressure contours based on the LES solution

magnitude and the decay of the PSD at increasing Strouhal number (Sr) is in good agreement with the experimental findings. A clear correlation of the tonal components is not possible due to the limited period of time available for the Fast Fourier Transformation which in turn comes from the small number of input data. The directivities of the slat gap noise source and the main airfoil trailing edge source are shown in Fig. 35 on a circle at radius R = 1.5 centered near the trailing edge of the slat. The following geometric source definitions were used. The slat source covers the part from the leading edge of the slat through 40% chord of the main wing. The remaining part belongs to the main wing trailing edge source. An embedded boundary formulation is used to ensure that no artificial noise is generated [22]. It is evident that the sources located near the slat cause a stronger contribution to the total sound field than the main wing trailing edge sources. This behavior corresponds to the distribution of the Lamb vector.

406

E. Gr¨ oschel et al. Fig. 34. Power spectral density for a point at x = −1.02 and y = 1.76

Fig. 35. Directivities for a circle with R = 1.5 based on the APE solution

5 Conclusion In the present paper we successfully computed the dominant aeroacoustic noise sources of aircraft during take-oﬀ and landing, that is, the jet noise and the slat noise by means of a hybrid LES/APE method. The flow parameters were chosen to match current industrial requirements such as nozzle geometry, high Reynolds numbers, heating eﬀects etc. The flow field and acoustic field were computed in good agreement with experimental results showing the correct noise generation mechanisms to be determined. The dominant source term in the APE formulation for the cold single jet has been shown to be the Lamb vector, while for the coaxial jets additional source terms of the APE-4 system due to heating eﬀects must be taken into account. These source terms are generated by temperature and entropy fluctuations and by heat release eﬀects and radiate at obtuse angles to the far field. The comparison between the single and coaxial jets revealed diﬀerences in the

Aeroacoustic Prediction of Jet and Slat Noise

407

flow field development, however, the characteristics of the acoustic near field signature was hardly changed. The present investigation shows that the noise levels in the near field of the jet are not directly connected to the statistics of the Reynolds stresses. The analysis of the slat noise study shows the interaction of the shear layer of the slat trailing edge and slat gap flow to generate higher vorticity than the main airfoil trailing edge shear layer. Thus, the slat gap is the dominant noise source region. The results of the large-eddy simulation are in good agreement with data from the literature. The acoustic analysis shows the correlation between the areas of high vorticity, especially somewhat downstream of the slat trailing edge and the main wing trailing edge, and the emitted sound. Acknowledgements The jet noise investigation, was funded by the Deutsche Forschungsgemeinschaft and the Centre National de la Recherche Scientifique (DFG-CNRS) in the framework of the subproject “Noise Prediction for a Turbulent Jet” of the research group 508 “Noise Generation in Turbulent flows”. The slat noise study was funded by the national project FREQUENZ. The APE solutions were computed with the DLR PIANO code the development of which is part of the cooperation between DLR Braunschweig and the Institute of Aerodynamics of RWTH Aachen University.

References 1. LESFOIL: Large Eddy Simulation of Flow Around a High Lift Airfoil, chapter Contribution by ONERA. Springer, 2003. 2. N. Alkishriwi, W. Schr¨ oder, and M. Meinke. A large-eddy simulation method for low mach number ﬂows using preconditioning and multigrid. Computers and Fluids, 35(10):1126–1136, 2006. 3. N. Andersson, L.-E. Eriksson, and L. Davidson. Les prediction of ﬂow and acoustcial ﬁeld of a coaxial jet. Paper 2005-2884, AIAA, 2005. 4. V. Arakeri, A. Krothapalli, V. Siddavaram, M. Alkislar, and L. Lourenco. On the use of microjets to suppress turbulence in a mach 0.9 axissymmetric jet. J. Fluid Mech., 490:75–98, 2003. 5. D. J. Bodony and S. K. Lele. Jet noise predicition of cold and hot subsonic jets using large-eddy simulation. CP 2004-3022, AIAA, 2004. 6. C. Bogey, C.and Bailly. Computation of a high reynolds number jet and its radiated noise using large eddy simulation based on explicit ﬁltering. Computers and Fluids, 35:1344–1358, 2006. 7. J. P. Boris, F. F. Grinstein, E. S. Oran, and R. L. Kolbe. New insights into large eddy simulation. Fluid Dynamics Research, 10:199–228, 1992. 8. M. M. Choudhari and M. R. Khorrami. Slat cove unsteadiness: Eﬀect of 3d ﬂow structures. In 44st AIAA Aerospace Sciences Meeting and Exhibit. AIAA Paper 2006-0211, 2006.

408

E. Gr¨ oschel et al.

9. M. Elmneﬁ. Private communication. Institute of Aerodynamics, RWTH Aachen University, 2006. 10. R. Ewert and W. Schr¨ oder. Acoustic pertubation equations based on ﬂow decomposition via source ﬁltering. J. Comput. Phys., 188:365–398, 2003. 11. R. Ewert, Q. Zhang, W. Schr¨ oder, and J. Delfs. Computation of trailing edge noise of a 3d lifting airfoil in turbulent subsonic ﬂow. AIAA Paper 2003-3114, 2003. 12. J. B. Freund. Noise sources in a low-reynolds-number turbulent jet at mach 0.9. J. Fluid Mech., 438:277 – 305, 2001. 13. E. Gr¨ oschel, M. Meinke, and W. Schr¨ oder. Noise prediction for a turbulent jet using an les/caa method. Paper 2005-3039, AIAA, 2005. 14. E. Gr¨ oschel, M. Meinke, and W. Schr¨ oder. Noise generation mechanisms in single and coaxial jets. Paper 2006-2592, AIAA, 2006. 15. F. Q. Hu, M. Y. Hussaini, and J. L. Manthey. Low-dissipation and low-dispersion runge-kutta schemes for computational acoustics. J. Comput. Phys., 124(1):177– 191, 1996. 16. M. Israeli and S. A. Orszag. Approximation of radiation boundary conditions. Journal of Computational Physics, 41:115–135, 1981. 17. S. Koh, E. Gr¨ oschel, M. Meinke, and W. Schr¨ oder. Numerical analysis of sound sources in high reynolds number single jets. Paper 2007-3591, AIAA, 2007. 18. A. Kolb. Private communication. FREQUENZ, 2006. 19. J. Lau, P. Morris, and M. Fisher. Measurements in subsonic and supersonic free jets using a laser velocimeter. J. Fluid Mech., 193(1):1–27, 1979. 20. D. Lockard. An eﬃcient, two-dimensional implementation of the ﬀowcs williams and hawkings equation. J. Sound Vibr., 229(4):897–911, 2000. 21. M. Meinke, W. Schr¨ oder, E. Krause, and T. Rister. A comparison of secondand sixth-order methods for large-eddy simulations. Computers and Fluids, 31:695–718, 2002. 22. W. Schr¨ oder and R. Ewert. LES-CAA Coupling. In LES for Acoustics. Cambridge University Press, 2005. 23. J. S. Shang. High-order compact-diﬀerence schemes for time dependent maxwell equations. J. Comput. Phys., 153:312–333, 1999. 24. C. K. W. Tam and J. C. Webb. Dispersion-relation-preserving ﬁnite diﬀerence schemes for computational acoustics. J. Comput. Phys., 107(2):262–281, 1993. 25. A. Uzun, A. S. Lyrintzis, and G. A. Blaisdell. Coupling of integral acoustics methods with les for jet noise prediction. Pap. 2004-0517, AIAA, 2004. 26. O. V. Vasilyev, T. S. Lund, and P. Moin. A general class of commutative ﬁlters for les in complex geometries. J. Comput. Phys., 146:82–104, 1998. 27. K. B. M. Q. Zaman. Flow ﬁeld and near and far sound ﬁeld of a subsonic jet. Journal of Sound and Vibration, 106(1):1–16, 1986.

Investigation of the Turbulent Flow Separation from an Axisymmetric Hill Manuel Garc´ıa-Villalba and Wolfgang Rodi Institut f¨ ur Hydromechanik, Universit¨ at Karlsruhe, Kaiserstraße 12, 76128 Karlsruhe, Germany [email protected], [email protected] Summary. A highly-resolved Large Eddy Simulation of high Reynolds number ﬂow over and around a three-dimensional hill is currently being performed on the NEC SX-8. The principal aim of the study is to generate target results against which Hybrid LES-RANS methods can be validated. The complex ﬂow separation in the lee of the hill is illustrated by displaying streamlines of the mean ﬂow at diﬀerent heights. The instantaneous structures which appear in the wake of the hill are also discussed.

1 Introduction The turbulent flow separation from a three-dimensional curved body is a complex problem which plays an important role in practical applications. In recent laboratory studies Simpson and co-workers [1, 2, 3] investigated extensively the separated flow over and around a 3D hill at high Reynolds number using LDV measurement techniques, oil flow visualisation and hot-wire anemometry. The examinations reveal the complex flow physics associated with the geometry of the hill. Complex separation occurs on the leeside and the evolving vortical structures merge into two large counter-rotating streamwise vortices downstream. There is also some evidence of low frequency spanwise meandering of the vortices in the wake. This type of flow is extremely challenging for any numerical model and has served as a test or validation case for statistical Reynolds Averaged Navier Stokes (RANS) models as well as for time-dependent Large Eddy Simulations (LES). It was shown e.g. in the framework of the European project FLOMANIA (Flow Physics Modelling – An Integrated Approach) and in the 11th ERCOFTAC/IAHR Workshop in Gothemburg 2005, where this flow was chosen as a test case, that RANS type models provide a qualitatively fairly good approximation of the flow, however fail to quantitatively calculate separation and reattachment points as well as the distribution of mean velocities, turbulence intensities and Reynolds stresses in the recovery region behind the hill.

410

M. Garc´ıa-Villalba, W. Rodi

This was also shown by Wang et al. [4] who used diﬀerent non-linear eddy viscosity and second moment closure models to simulate this flow and reported unsatisfactory predictions. The main reason for this failure of RANS schemes is the fact that in this type of flow unsteady separation along a smooth surface occurs which means that the point of separation varies in time and the detachment and attachment process is an unsteady feature of the fully turbulent flow. Moreover, in the lee of the hill large scale organised structures occur which RANS methods cannot predict. However, this suggests that the method to be used in such cases is LES, where these large scale structures are simulated directly and only small scales are modelled via a subgrid scale model. This method avoids the high cost of fully resolving all turbulent scales as in a Direct Numerical Simulation (DNS) and makes accurate predictions of unsteady flow processes aﬀordable. This has been shown in many LES predictions of the flow around bluﬀ bodies with surface curvature, such as long and short cylinders [5, 6] or wavy walls [7] and also for the flow over the hill to be investigated herein. However, the latter case is associated with some diﬃculties regarding the high Reynolds number, which is, similar to a DNS, still a limiting factor in the use of Large Eddy Simulation methods. For high values such as the Reynolds number in this case (130 000) LES requires very fine resolution, especially near the walls so that several attempts to simulate this test case did not improve the predictions to a satisfying degree [8, 9, 10, 11, 12, 13]. Recently, hybrid RANS-LES and zonal LES/RANS methods have been developed to overcome the immense resolution requirements, especially near solid walls, of Large Eddy Simulations of high Reynolds number flows. In the case of the hybrid RANS-LES scheme a RANS eddy viscosity turbulence model for the computation of the flow in the near wall region is used. The zonal LES/RANS method uses a simplified turbulent boundary layer equation that is applied to regions up to y+ = 50. For the distribution of mean velocities in regions behind the hill, Davidson and Dahlstr¨ om [14] produced promising results with a hybrid LES-RANS method and Tessicini et al. [15] with a zonal LES/RANS method. However, the treatment of the interface between the LES and the RANS regions is very important for the method to be successful and is still the subject of ongoing research. In the experiments of Simpson and co-workers some discrepancies have been observed between measurements obtained with diﬀerent techniques (LDA [3] and hot-wire anemometry [2]). For the validation of hybrid RANSLES approaches it is very important that the benchmark data are of good quality, especially since there is some doubt about the experiments. Therefore, with the aim of generating target results against wich hybrid methods can be validated, a highly-resolved Large Eddy Simulation of the experimental configuration is currently being carried out on the NEC SX-8. A second aim of the study is to improve the understanding of the turbulent flow separation from a three-dimensional curved body. The configuration mentioned above consists of flow over and around an axisymmetric hill of height H = 78 mm and base-to-height ratio of 4; the

Turbulent Flow Separation from an Axisymmetric Hill

411

approach-flow turbulent boundary-layer has a thickness of δ = 0.5H. The Reynolds number of the flow based on the free-stream velocity Uref = 27.5 m/s and the hill height H is Re = 1.3 · 105 .

2 Numerical Model The LES is performed with the in-house code LESOCC2 (Large Eddy Simulation On Curvilinear Coordinates). The code has been developed at the Institute for Hydromechanics. It is the successor of the code LESOCC developed by Breuer and Rodi [5] and is described in its most recent status in [16]. The code solves the Navier-Stokes equations on body-fitted, curvilinear grids using a cell-centered Finite Volume method with collocated storage for the cartesian velocity components. Second order central diﬀerences are employed for the convection as well as for the diﬀusive terms. The time integration is performed with a predictor-corrector scheme, where the explicit predictor step for the momentum equations is a low-storage 3-step Runge-Kutta method. The corrector step covers the implicit solution of the Poisson equation for the pressure correction (SIMPLE). The scheme is of second order accuracy in time because the Poisson equation for the pressure correction is not solved during the sub-steps of the Runge-Kutta algorithm in order to save CPU-time. The Rhie and Chow momentum interpolation [17] is applied to avoid pressurevelocity decoupling. The Poisson equation for the pressure increment is solved iteratively by means of the ‘strongly implicit procedure’ [18]. Parallelization is implemented via domain decomposition, and explicit message passing is used with two halo cells along the inter-domain boundaries for intermediate storage. The geometry of the computational domain is shown in Fig. 1. The size of the domain is 20H × 3.2H × 11.7H in streamwise, wall-normal and spanwise directions, respectively. The grid consists of 770 × 240 × 728 cells in these directions. Three views of the grid in the symmetry plane are shown in Fig. 2. Note that to minimize numerical errors, the grid is quasi-orthogonal close to the hill wall. The inflow conditions are obtained by performing simultaneously a separate periodic LES of channel flow in which the mean velocity is forced to assume the experimental vertical distribution using a body-force technique [19]. The length of the channel is 1.8H = 3.6δ and the number of

Fig. 1. Sketch of the computational domain and inﬂow generator

412

M. Garc´ıa-Villalba, W. Rodi 3

y

2 1 0

0

5

10

x

15

1

2

y

y

3

1 0-4

0.5 -2

0

x

2

4

0.5

1

x

1.5

Fig. 2. Grid in the symmetry plane. Three diﬀerent views. Every 4th grid line is shown

cells in streamwise direction is 110. The cost of the precursor simulation is, therefore, 1/8 of the total cost. A no-slip condition is employed at the bottom wall while the Werner-Wengle wall function is used at the top wall, so that the boundary layer there is not well resolved. Free-slip conditions are used at the lateral boundaries and convective conditions at the exit boundary. The quality of the grid resolution is judged by determining the cell size in wall units. The centre of the wall-adjacent cell is located at y1+ ∼ 2. The streamwise and spanwise cell sizes in wall units are roughly 70 and 30, respectively. These values are just within the limits of the recommendations given by Piomelli & Chasnov [20] for wall resolving LES. As for the quality of the inflow conditions, the mean streamwise velocity, turbulent kinetic energy and shear-stress profiles (Fig. 3) are in good agreement with experimental data.

Fig. 3. Proﬁles at the inlet obtained from the precursor simulation. Left, mean streamwise velocity u. Centre, turbulent kinetic energy k. Right, shear stress −u′ v ′ . Solid line, LES. Symbols, experimental data

Turbulent Flow Separation from an Axisymmetric Hill

413

3 Computational Details Several tests runs were necessary in order to determine the grid resolution and inflow boundary conditions described above. The following data excludes the test runs and concentrates on the final run, which is still underway at the time of writing. To obtain a near-optimal load-balancing, the computational grid is subdivided into sub-grids of equal size. Each sub-grid is assigned to a single-processor. For the present configuration a total of 64 blocks are used so that 8 nodes of the NEC SX-8 are needed, using 190 Gb of memory in total. The average vector operation ratio achieved with LESOCC2 is between 98% and 99.3%. The average speed per processor is between 2900 and 3100 Mflops. At the time of writing 700 000 time steps of the simulation have been computed. This corresponds to a clock time of approximately 3 months and a total number of CPU hours of about 125 000.

4 Results After discarding initial transients, statistics have been collected so far for a time span of roughly 160 H/Uref . This corresponds approximately to 8flow-trough times of the computational domain and 700 000 time steps of the simulation. An illustration of the mean flow obtained in the simulation is displayed in Fig. 4 by showing streamlines of the flow in the symmetry plane (left) and in a plane close to the hill wall at y + ∼ 40(right). With respect to the experiments, the thickness of the recirculation zone is well predicted although the separation occurs somewhat earlier in the simulation. The flow topology is also rather well predicted with the two counter-rotating vortices appearing roughly at the same location as in the experiment. Figure 5 displays streamlines of the mean flow in four surfaces very close to the bottom wall, at y + = 10, 40, 200 and 400, respectively. Contours of

Fig. 4. Left, streamlines in midplane. Right, streamlines at y + = 40

414

M. Garc´ıa-Villalba, W. Rodi

Fig. 5. Streamlines at y + = 10,40, 200 and 400. Color represents turbulent kinetic energy

turbulent kinetic energy are also shown. This figure reveals the complex flow topology. Very close to the wall, at y + = 10 and 40 (top of the figure), one pair of counter-rotating vortices detaching from the leeward side of the hill is present. With increasing height y + ∼ 200 (bottom left), the flow is even more complex, where an extra pair of counter-rotating vortices is formed closer to the symmetry plane. Unfortunately, the mean flow is still not fully converged, as can be observed from the asymmetries in the vortex patterns with respect to the midplane at y + = 400 (bottom right). The convergence of the second order moments is less good than the convergence of the mean flow, as can be observed from the contours of turbulent kinetic energy, which are not very smooth yet. However some clear trends can already be detected from these contours. First, it is clearly visible in all 4 plots in Fig. 5 that the turbulent kinetic energy is concentrated in the separated shear layer formed in the boundary between the incoming flow and the recirculation region. Second, very close to the wall at y + = 10 and 40, there is also a region of high turbulent kinetic energy where the re-attachment of the flow

Turbulent Flow Separation from an Axisymmetric Hill

415

occurs. In this zone impinging fluid is reoriented in all directions tangential to the wall. A visualization of the instantaneous coherent structures of the flow is displayed in Fig. 6. It shows an iso-surface of pressure fluctuations at three diﬀerent instants; the color represents the y-coordinate. Coherent structures are observed to form in the lee of the hill and are convected downstream. Many of them have the shape of a hairpin vortex although due to the high level

Fig. 6. Iso-surface of pressure ﬂuctuations at three instants

416

M. Garc´ıa-Villalba, W. Rodi

of turbulence they are usually rapidly deformed. It happens frequently that structures appear only on one side of the wake. In the first instant observed in Fig. 6 top all the structures are in the left part of the wake, while no activity is observed in the right part. The opposite happens in the instant displayed in Fig. 6 bottom. There are also instants in which structures appear everywhere in the wake, Fig. 6 middle. Time signals of velocity recorded in the near wake show pronounced peaks when one of these big structures crosses the recording point.

5 Conclusions In this paper, preliminary results of a highly-resolved LES of flow over and around a three-dimensional hill have been presented. This is a very important generic configuration in the context of external aerodynamics. The complex mean flow on the leeward side of the hill and the near wake has been visualized. Illustrations of the instantaneous flow have been also provided. It is expected that once the flow is fully converged, a very complete database can be generated for the validation of hybrid LES-RANS methods. Acknowledgements The authors are grateful to Dr. J.G. Wissink for his help with the grid generation, to Prof. J. Fr¨ ohlich, Prof. T. Stoesser and Dr. D. von Terzi for fruitful discussions, and to the steering committee of the supercomputing facilities in Stuttgart for granting computing time on the NEC SX-8.

References 1. R.L. Simpson, C.H. Long, and G. Byun. Study of vortical separation from an axisymmetric hill. Int. J. Heat Fluid Flow, 23:582–591, 2002. 2. R. Ma and R.L. Simpson. Characterization of turbulent ﬂow downstream of a three-dimensional axisymmetric bump. In Proc. 4th Int. Symposium on Turbulence and Shear Flow Phenomena. Williamsburg. USA, 2005. 3. G. Byun and R. L. Simpson. Structure of three-dimensional separated ﬂow on an axisymmetric bump. AIAA J., 44(5):999–1008, 2006. 4. C. Wang, Y. J. Jang, and M. A. Leschziner. Modelling two and threedimensional separation from curved surfaces with anisotropy-resolving turbulence closures. Int. J. Heat Fluid Flow, 25:499–512, 2004. 5. M. Breuer and W. Rodi. Large eddy simulation of complex turbulent ﬂows of practical interest. In E.H. Hirschel, editor, Flow simulation with high performance computers II, volume 52 of Notes on Numerical Fluid Mechanics, pages 258–274. Vieweg, Braunschweig, 1996. 6. J. Fr¨ ohlich and W. Rodi. LES of the ﬂow around a cylinder of ﬁnite height. Int. J. Heat Fluid Flow, 25:537–548, 2004.

Turbulent Flow Separation from an Axisymmetric Hill

417

7. J. Fr¨ ohlich, C. P. Mellen, W. Rodi, L. Temmerman, and M. A. Leschziner. Highly resolved large-eddy simulation of separated ﬂow in a channel with streamwise periodic constrictions. J. Fluid Mech., 526:19–66, 2005. 8. N. Patel, C. Stone, and S. Menon. Large-Eddy Simulation of turbulent ﬂow over an axisymmetric hill. AIAA paper no. 2003-0967, 2003. 9. C. Fureby, N. Alin, N. Wikstr¨om, S. Menon, N. Svanstedt, and L. Persson. Large-Eddy Simulation of high-Reynolds number wall-bounded ﬂows. AIAA J., 42(3):457–468, 2004. 10. N. Li, C. Wang, M.A. Leschziner, and L. Temmerman. Large eddy simulation of separation from a three-dimensional hill and comparison with second-moment closure RANS modelling. In Proc. 4th Int. Symposium on Turbulence and Shear Flow Phenomena. Williamsburg. USA, 2005. 11. S. Benhamadouche, J. Uribe, N. Jarrin, and D. Laurence. Large eddy simulation of a symmetric bump on structured and unstructured grids, comparison with RANS and T-RANS models. In Proc. 4th Int. Symposium on Turbulence and Shear Flow Phenomena. Williamsburg. USA, 2005. 12. T. Persson, M. Liefvendahl, R. E. Benson, and C. Fureby. Numerical investigation of the ﬂow over an axisymmetric hill using LES, DES and RANS. J. Turbulence, 7(4):1–17, 2006. 13. S. Krajnovi´c. Large eddy simulation of the ﬂow around a three-dimensional axisymmetric hill. In K. Hanjali´c, Y. Nagano, and S. Jakirli´c, editors, Proc. Turbulence, Heat and Mass Transfer 5, 2006. 14. L. Davidson and S. Dahlstr¨ ohm. Hybrid LES-RANS: computation of the ﬂow around a three-dimensional hill. In W. Rodi and M. Mulas, editors, Engineering Turbulence Modelling and Experiments 6. Elsevier, 2005. 15. F. Tessicini, N. Li, and M. A. Leschziner. Simulation of three-dimensional separation with a zonal near-wall approximation. In P. Wesseling, E. Onate, and J. P´eriaux, editors, Proc. ECCOMAS CFD 2006, 2006. 16. C. Hinterberger. Dreidimensionale und tiefengemittelte Large-Eddy-Simulation von Flachwasserstr¨ omungen. PhD thesis, University of Karlsruhe, 2004. 17. C.M. Rhie and W.L. Chow. Numerical study of the turbulent ﬂow past an airfoil with trailing edge separation. AIAA J., 21(11):1061–1068, 1983. 18. H.L. Stone. Iterative solution of implicit approximations of multidimensional partial diﬀerential equations for ﬁnite diﬀerence methods. SIAM J. Numer. Anal., 5:530–558, 1968. 19. C.D. Pierce. Progress-variable approach for large-eddy simulation of turbulent combustion. PhD thesis, Stanford University, 2001. 20. U. Piomelli and J. R. Chasnov. Large eddy simulation: theory and applications. In Turbulence and transition modelling, pages 269–331. Kluwer, 1996.

Large Eddy Simulation (LES) with Moving Meshes on a Rapid Compression Machine: Part 2: Numerical Investigations Using Euler-Lagrange-Technique Franco Magagnato1, Martin Gabi2 , Thomas Heidenreich3 , Amin Velji4 , and Ulrich Spicher5 1

2

3

4

5

Institute of Fluid Machinery, University of Karlsruhe, 76128 Karlsruhe, Germany [email protected] Institute of Fluid Machinery, University of Karlsruhe, 76128 Karlsruhe, Germany [email protected] Institute of Reciprocating Engines, University of Karlsruhe, 76128 Karlsruhe, Germany [email protected] Institute of Reciprocating Engines, University of Karlsruhe, 76128 Karlsruhe, Germany [email protected] Institute of Reciprocating Engines, University of Karlsruhe, 76128 Karlsruhe, Germany [email protected]

Summary. The ﬂow inside a simpliﬁed one-stroke engine with squared cross section has been calculated with compressible Large Eddy Simulation (LES) using our code SPARC and compared with the measurements on the same geometry. The one-stroke engine has a turbulence generator, which can ether generate a tumble or homogenous turbulence depending on the conﬁguration. By waiting diﬀerent amount of time after the turbulence generation process a variable turbulence level can be achieved. During the up going motion of the piston the turbulent fuel mixture is compressed and ignited by a row of spark plugs. The simulation has been using more then 8 million points for the space discretization. A space conservation law was used to calculate the grid motion with Euler-Lagrange technique. The mesh was reﬁned in the shear layers and close to the wall so that y + < 1 results almost everywhere. A comparison between Miles (monotonically integrated large eddy simulation) approach and conventional subgrid scale modelling (dynamic Smagorinsky) showed very similar solutions. Mean and ﬂuctuating velocities at TDC are compared with available experimental ﬁndings.

420

F. Magagnato et al.

1 Introduction The development and optimization of DI engines is nowadays mainly based on experiments and URANS (Unsteady Reynolds Averaged Navier-Stokes) computations. The impressive results of LES in other branches of fluid mechanics has encouraged the application of LES also in the field of internal combustion engine in the last few years [1, 2, 3]. The severe restrictions of the underlying turbulence model for URANS simulation especially in the context of the complicated situation in In-cylinder flows with two phase flows, mixing processes, moving grids and combustion processes makes the use of URANS at least questionable. LES has already demonstrated that the combustion simulation in statistical steady state flows is far more reliable then URANS [4] since the modelling of the turbulence eﬀects in combination with reactive flows depends mainly on the correct description of the large coherent structure of the flow. The Large Eddy Simulation resolves these large structures accurately and models the small structures by a subgrid scale model. The superiority of LES compared to URANS lies in the fact that the small scale structures are more universal then the large ones and can be easily modelled with an explicit or an implicit subgrid scale model. In the case of an explicit subgrid scale model one uses usually the eddy viscosity concept and adds a turbulent viscosity to the laminar viscosity in order to model the turbulent kinetic energy dissipation from the large scales to the smaller (unresolved) scales. The majority of these models use the grid resolution of the numerical grid and the deformation tensor to compute the eddy viscosity. It has been found that the unavoidable constants in these subgrid scale models depend somewhat on the flow field (for example the Smagorinsky model). A better strategy has been found by Germano et al. [5]. He proposed to extract the value of the constants locally from the resolved flow field. This is called the dynamic approach. A totally diﬀerent strategy (Miles) is to use the numerical dissipation of the numerical scheme to transfer the energy from the large scales to the smaller scales in contrast to the explicit modelling where extremely low numerical dissipation and dispersion errors of the numerical scheme are required. It has been found by Boris et al. [6] that the numerical inaccuracies of the scheme behaves in a manner that is qualitatively similar to an explicit SGS model. Both strategies have been recently used and compared by Magagnato et al. [7] for the challenging case of predicting the transitional flow on a highly loaded turbine blade. It has been shown that the explicit modelling can predict the transition region and the fully turbulent region quite accurately but the Miles approach performed somewhat better when compared to the experimental findings. Beyond that the author believes that in the case of InCylinder flows the advantage of the Miles approach will be even more significant because of the possible appearance of transonic regions, flame front of the combustion process and mixing eﬀects in case of DI engines. The reason is that in regions of very high velocity gradients (shock waves, flame fronts

LES on a Rapid Compression Machine: Part 2

421

etc.) any compressible numerical scheme needs a certain amount of numerical dissipation to stabilize the computation but this would also falsify the energy transport process when modelled with an explicit SGS model. Another very important aspect of an accurate Large Eddy Simulation is the use of nonreflecting or absorbing boundary conditions. Since in the simulation of internal flow fields the computational grid has always to be truncated for numerical eﬃciency the inlet and outlet BC are always very close to the domain of interest of the flow field. A reflecting boundary condition will contaminate the flow field with nonphysical pressure reflections. In the case of the turbine blade calculation mentioned above the pressure reflections at the outlet caused resonant behavior at the suction side of the blade and falsified the turbulent velocity profiles on the blade. It is sure that in the case of the computation of reciprocating engines the highly unsteady acoustic and convective waves generated by the opening of the valves these is even more important then in other applications. A possible solution of this problem can be found in Magagnato et al. [8]. For the computation of In-cylinder flows the solution of the Navier-Stokes equations requires the description and solution of the deforming domain. It is clear that the improper formulation of the grid velocities due to the deforming grid is of great importance. A detailed explanation of the diﬀerent approaches can be found in Moreau et al. [2]. In our code SPARC [9] an alternative approach is implemented. The method proposed by Lai and Przekwas [10] can be categorized as an arbitrary Lagragian-Eulerian (ALE) method which solves the Space Conservation Law in an accurate and conservative way. The Space conservation law reads: ∂ dV − ρug · ndS = 0 (1) ∂t V

S

In order to evaluate the volume, cell face area and cell face normal accurately they propose to define a fifth point on a cell face according to Fig. 1. The formulas are reads: 1 1 ni Ai r5 = Ai (2) , V ol = ri , n = (r5 · n) · 4 3 Ai

The error of a non-conservative formulation can be significant and is demonstrated below. Here we have computed the deformation of a grid inside a 3D square box in stationary flow. Obviously the fluid should remain stationary when we solve the Navier-Stokes equation in a moving control volume.

422

F. Magagnato et al. Fig. 1. Illustration of a cell face construction

∂ ρdV + ρ(u − ug ) · ndS = 0 (3) ∂t V S ∂ σ · ndS (4) ρudV + ρu(u − ug ) · ndS = ∂t V S S ∂ q · ndS (5) ρEdV + ρE(u − ug ) · ndS = uσ · ndS − ∂t V

S

S

S

The initial grid and the final grid are shown in Fig. 2a and 2b. The non-conservative formulation by using the cell face area computed in the usual way produce artificial mass fluxes and the stationary flow becomes unsteady. In contrast to that the flow remains steady in case one uses the formulation of Lai and Przekwas. In the experiments two main configurations have been investigated. One is equipped with a four-hole turbulence generator (Fig. 3), the other has a twohole turbulence generator. Both configurations use a pancake-chamber. It exists also configurations with diﬀerent shapes of the combustion chamber which will be subject of further investigations in the near future. While the former produces relatively homogeneous turbulence for the compression phase the later produces a tumble which is more appropriate for SI-engines. We have generated the four-hole configuration and obtained the two-hole configuration by eliminating two nozzles from the solution process.

LES on a Rapid Compression Machine: Part 2

423

Fig. 2. Initial grid (a) and ﬁnal grid (b)

Fig. 3. Geometry of the simpliﬁed one-stroke engine with turbulence generator in an intermediate position

2 Results of the Four-Hole Configuration The down going motion of the turbulence generator has been measured in the experiments and provided for the calculation in a digital format. The motion is not uniform since the piston is driven by a pneumatic device. At the end of the down going motion some disagreement between the experiments and the computations arises since the mechanical device reacts in an elastic way due to the extremely high pressure which evolves when the turbulence generator

424

F. Magagnato et al.

reaches the base plate. This is due to the fact that the compressed air in the gap can not be evacuated fast enough throw the holes because the flow becomes choked in the smallest gap (Ma = 1) while the turbulence generator still wants to move down. As result in the experiment the material reacts in an elastic way because the pressure inside the gap becomes several hundreds bar. In the experiment this phenomenon has been observed by visual inspection. That means that the turbulence generator bounced back once it reaches the bottom. Since no detailed measurements could be made we had to neglect this detail. We believe that it has only a small impact on the compression phase since the turbulence field is homogenized during the holding time before compression. At the beginning of the motion of the turbulence generator pressure waves moves up and down through the chamber reflecting the disturbances of the stagnant air-methane mixture. At about 5% of the displacement the maximum velocity at the nozzle exit is reached. In this configuration we could see speeds in the order of 240 m/s (Ma ≈ 0.7). It is clear that this causes large density and temperature diﬀerences. On Fig. 5 we show the density distribution in the symmetry plane of the nozzles at about 20% of the motion. The flow in the jets is already turbulent and influenced by the neighboring jets and by the proximity of the walls of course. A typical iso-surface distribution of the so called Q-criterion is shown in Fig. 6. The Q-criterion (positive values of Q) is a necessary condition for the existence of thin convex low-pressure tubes and therefore allows visualization of coherent vortices. It is defined as (Ωij = vorticity tensor, Sij = strain rate tensor): Q=

1 (Ωij Ωij − Sij Sij ) 2

Fig. 4. Grid in the symmetry plane of a nozzle

(6)

LES on a Rapid Compression Machine: Part 2

425

Fig. 5. Density distribution in the symmetry plane of the nozzles at 20% motion

Fig. 6. Iso-surfaces of the Q-criterion in the combustion chamber at 70% motion

The velocity vectors in the symmetry plane of the nozzles are shown in Fig. 7. The large variation of the velocity fluctuations in the jet and especially close to the wall is impressive. It appears that the low resolutions used by many

426

F. Magagnato et al.

Fig. 7. Velocity vectors in the symmetry plane of the nozzles

researchers in the past seem to be inappropriate to account for the wall influence on the turbulent field for in-cylinder flows. It is clear that an even higher influence due to the flow field resolution in the combustion phase and more importantly on the heat transfer rate to the wall could be expected. When the turbulence generator has reached the bottom a certain amount of time will elapse in order to allow the turbulent degree to decrease. This allows investigating the combustion process with a variable turbulence level. In the experiment this waiting time has been varied between 15 ms and 50 ms. In our first computation we have chosen 15 ms for convenience. It is clear that in the near future we will also vary the waiting time of our simulation. The compression phase is rather unspectacular compared with the first phase but nevertheless more important for the understanding of the pre-combustion phase. An impression of the turbulence structures during this phase can be seen on Fig. 8. During the compression the large turbulent structures are transformed into smaller and smaller structures due to the wall proximity eﬀect. At the end of the compression phase (the compression ratio in this case is 11.5) the temperature has reached about 600 K and the pressure reached 40 MPa. This is considerable more than in the experiment (30 MPa) because the simulation is performed on a leakages free compression process while in the experiments due to blow by on the piston sealing the maximum pressure can not be expected. Unfortunately, no detailed measurements of the pressure

LES on a Rapid Compression Machine: Part 2

427

Fig. 8. Iso-surfaces of Q-criterion during the compression phase

Fig. 9. Mach-number distribution in a z-const. plane at TDC

losses are available but some former coarse estimation of Hunzinger [14] gives an error of about 10%. Figure 9 shows the extremely low Mach-number distribution in the combustion chamber at TDC. The maximum Mach-Number is below Ma = 0.002 which is known to create big numerical convergence problems due to math-

428

F. Magagnato et al.

ematical stiﬀness associated with the large disparity of the convective and acoustic speeds for full compressible formulations. We have overcome this problem by using a preconditioning scheme according to Weiss et al. [15]. This technique allows us to compute extremely low Mach-numbers down to Ma = 0.001 without degradation of the accuracy and convergence behavior.

3 Two-Hole Configuration In the case of the two-hole configuration the initial phase of the down going motion of the turbulence generator is more critical. The reason is that the speed of the turbulence generator and hitherto the mass displacement is the same as in the four-hole configuration but the evacuation area is reduced by a factor of two. As a result the maximum speed is increased until app. 340 m/s that means the flow becomes choked during the initial phase. After a few moments the situations relax and the max flow velocity becomes subsonic again. In Fig. 10 the generated tumble can be observed in the symmetry plane of the nozzle. Due to the asymmetry a strong tumble is generated. The strength of the tumble is reduced during the waiting time of the turbulence generator at BTC controlled by the amount of waiting time in this phase. During the compression stroke the tumble is further reduced due to the influence of the wall and the reduced space available inside the chamber.

Fig. 10. Velocity component in vertical direction at 50% motion

LES on a Rapid Compression Machine: Part 2

429

4 Computational Efficiency The computations were done on a parallel cluster of Itanium processors on our supercomputing center at the University of Karlsruhe. By using 34 processors the whole computation consisting of the turbulence generator motion (Δt = 20 ms), the waiting time (Δt = 15 ms) and the compression stroke (Δt = 16 ms) took app. 150 wallclock hours. The computational eﬃciency for this application was app. 96% while the load balancing was 97% by distributing 604 on 34 processors. The mayor part was spent on the up going motion of the turbulence generator (about 50%) while the waiting time (20%) and the compression phase were done quicker due to the reduced number of mesh points.

5 Conclusion The flow inside a simplified one-stroke engine with squared cross section was calculated with a compressible Large Eddy Simulation. The grid resolution was chosen very high (about 8 million points) in order to resolve accurately the flow structures in the jet shear layers and the wall layer (Fig. 4). The simulations were done using the dynamical Smagorinsky subgrid scale model and the Miles approach according to Boris et al. No significant diﬀerences in the quality of the results between them could be observed. Nevertheless the Miles approach stabilized the computation especially when the turbulence generator was very close to the ground plate. The Mach number range in the calculation was found to vary very much. During the initial phase of the turbulence generator motion we could observe Ma = 1 while at TDC of the compression stroke the Mach number felt below Ma = 0.002. This required improving the conventional compressible scheme with the preconditioning technique. The measurements of the one-stroke engine have mainly been done but the evaluation and comparison with the simulation is still not available. Several combustion models (reaction progress models, flame surface density models) are being implemented in the numerical scheme and are at the moment in the validation phase. The ignition and combustion of the methaneair mixture will then be calculated and compared with the measurements. Acknowledgements The present work is a part of the subproject B2 of the Collaborative Research Centre (CRC) 606 – “Unsteady Combustion: Transport phenomena, Chemical Reactions, Technical Systems” at the University of Karlsruhe. The project is supported by the German Research Foundation (DFG).

430

F. Magagnato et al.

References 1. Haworth, D.C.: Large-Eddy-Simulation of In-Cylinder Flows. Oil & Gas Science and Technology-Rev. IFP, 54 (2), pp. 175–185 (1999) 2. Moreau, V., Lartique, G., Sommerer, Y., Angelberger, C., Colin, O., Poinsot, T.: Numerical methods for unsteady compressible multi-component reacting ﬂows on ﬁxed and moving grids. Journal of Computational Physics , 202, pp. 710–736 (2005) 3. Jhavar, R., Rutland, C.: Using Large Eddy Simulation to Study Mixing Eﬀects in Early Injection Diesel Engine Combustion. SAE Paper, 2006-01-0871 4. Poinsot, T., Veynante, D.: Theoretical and Numerical Combustion. 2nd ed., Edwards, ISBN: 1-930217-10-2 (2005) 5. Germano, M.: The Filtering Approach. Journal of Fluid Mechanics, 238, pp. 325–336 (1992) 6. Boris, J.P., Grinstein, F.E., Oran, E.S., Kolbe, R.L.: New insights into largeeddy simulation. Fluid Dyn. Res. 10, pp. 199–228 (1992) 7. Magagnato, F., Pritz, B., Gabi, M.: Calculation of a Turbine Blade at high Reynolds number by LES. Proceedings of the 11ˆth International Symposium on Transport Phenomena and Dynamics of Rotating Machinery, Honolulu, Hawaii (2006) 8. Magagnato, F., Pritz, B., Bchner, H., Gabi, M.: Prediction of the Resonance Characteristic of Combustion Chambers on the Basis of Large-Eddy-Simulation. Journal of Thermal Sciences, 14 (2), pp. 156–161 (2005) 9. Magagnato, F.: KAPPA - Karlsruhe Parallel Program for Aerodynamics. TASK Quarterly 2 (2), pp. 215–270 (1998) 10. Lai, Y.G., Przekwas, A.J.: A Finite-Volume Method for ﬂuid ﬂow Simulations with Moving Boundaries. Comp. Fluid Dynamics, 2, pp. 19–40 (1994) 11. Franke, J.: Untersuchung zur Grobstruktur-simulation kompressibler Strmungen mit der Volumenﬁltermethode auf bewegten Gittern. Dissertation, University of Karlsruhe (1998) 12. Merkel, S., Hunzinger, M., Nauwerck, A., Hensel, S., Velji, A., Spicher, U.: Einﬂuss der Turbulenz, des Luftverhltnisses und der Wandtemperatur auf die Flammenausbreitung unter ottomotorischen Bedingungen. VII. Tagung ”Motorische Verbrennung” Mnchen, (2005) 13. Swanson, R.C., Turkel, E.: Computational ﬂuid dynamics: Multistage central diﬀerence schemes for the Euler and Navies-Stokes equations. Lecture Notes of von Karman Institute for Fluid Dynamics (1996) 14. Hunzinger, M.: Private Communication. (2006) 15. Weiss, J.M., Smith, W.A.: Preconditioning applied to variable and constant density ﬂows. AIAA Journal, 33 (2), pp. 2050–2057 (1995)

Numerical Characterization of the Non-Reacting Flow in a Swirled Gasturbine Model Combustor A. Widenhorn, B. Noll, M. St¨ ohr, and M. Aigner Institut f¨ ur Verbrennungstechnik der Luft- und Raumfahrt, Universit¨ at Stuttgart, Pfaﬀenwaldring 38–40, 70569 Stuttgart, Germany [email protected] Summary. In this paper the three-dimensional non-reacting turbulent ﬂow ﬁeld of a swirl-stabilized gas turbine model combustor is analysed with compressible CFD. For the ﬂow analysis URANS and Hybrid RANS/LES (DES, SAS) turbulence models were applied. The governing equations and the numerical method are described. The simulations were performed using the commercial CFD software package ANSYS CFX-10.0. The numerically achieved velocity components show a good agreement with the experimental values obtained by Particle Image Velocimetry (PIV). Furthermore, a precessing vortex core (PVC) could be found in the combustion chamber. The simulations were performed on the HP XC4000 system of the High Performance Computing Centre Karlsruhe.

1 Introduction In order to achieve low levels of pollutants modern gas turbine swirl stabilized combustion systems are operating in lean and premixed mode. The combustor design is commonly based on the common injection of air and gaseous fuel in the form of a swirling jet. The central recirculation zone, which arises due to the swirl of the incoming flow, serves to anchor the flame within the combustion zone. However, especially under lean premixed conditions often severe self-excited combustion oscillations arise. These unwanted oscillations are in conjunction with high amplitude pressure oscillations which can decrease the lifetime and availability of the gas turbine. Depending on the swirl number swirling flows can exhibit diﬀerent topologies [1, 2, 3, 4]. A typical flow instability at a high swirl number is the precessing vortex core (PVC). This phenomenon can be found at the outlet of the injector system and rotates around the swirl flow axis at a given frequency. A further typical instability of swirling flows that can have an impact on combustor dynamics is vortex shedding which can be caused by the swirler vanes.

432

A. Widenhorn et al.

The design process of modern gas turbine combustion systems relies more and more on numerical simulation. In order to allow a reliable design the CFD methods have to predict the aerodynamics and the combustion driven dynamics accurately. However, there is still a lack in the field of turbulence and combustion modelling as well as in the definition of appropriate boundary conditions [5, 6, 7]. Nowadays diﬀerent approaches are available to capture unsteady flow fields. These are the Unsteady Reynolds Averaged Simulation (URANS), Large Eddy Simulation (LES) and hybrid RANS/LES methods. The URANS approach, which is commonly used in practical applications, uses complete statistical averaging [8, 9]. This allows the prediction of the time-mean or ensemble-averaged quantities for velocity, temperature and species distribution of non-reacting and reacting flow fields. However, experience shows that the URANS approach can lead to an excess of turbulent dissipation. For this reason there is the risk that important flow structures are dissipated. In order to avoid such errors in the present work turbulence models were applied which resolve parts of the large turbulent structures. Thus only the small structures have to be modelled. LES seems to be more appropriate to simulate gas turbine combustion systems and the potential of LES in this field was demonstrated by several authors in representative academic and semi industrial combustion chambers [10, 11]. However, due to the immense computational eﬀort of LES methods it is unclear if LES will get widely accepted for gas turbine research and development in the medium term. Beside LES promising approaches for the prediction of unsteady combustor phenomena are hybrid turbulence models which combine the strength of LES and RANS. Such hybrid approaches are the Detached Eddy Simulation (DES) and the Scale Adaptive Simulation (SAS). In the present work the non-reacting dynamic turbulent flow field of a model gas turbine combustor has been investigated using diﬀerent turbulence models. The first goal was to analyse the capability and the limits of the diﬀerent turbulence models by comparing the simulation data against the experimental data set. Furthermore, the turnaround times were assessed. For the numerical simulation the commercial CFD package ANSYS CFX 10.0 was used.

2 Physical Model 2.1 Conservation Equations The initial set for the numerical simulation of non-reacting flows includes the continuity, momentum, energy and turbulence equations. In this paper the compressible formulation is used. The equations are given by: ∂Q ∂(F − Fv ) ∂(G − Gv ) ∂(H − Hv ) + + + =S ∂t ∂x ∂y ∂z

(1)

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

433

The conservative variable vector Q consists of the density, the velocity components, the total specific energy, the turbulent kinetic energy and the specific dissipation rate and is defined as: ˜ ρ¯k, ρ¯ω]T Q = [¯ ρ, ρ¯u˜, ρ¯v˜, ρ¯w, ˜ ρ¯E,

(2)

Here, Favre-averaged quantities are used. F , G and H are the inviscid and Fv , Gv and Hv are the viscous fluxes in x-, y- and z-direction, correspondingly. The vector S in (1) contains the source terms and is defined as: S = [0, Su , Sv , Sw , SE , Sk , Sω ]T

(3)

2.2 Turbulence Modelling For the closure of the above system of partial diﬀerential equations for turbulent flows the Boussinesq hypothesis is used. The required values for the eddy viscosity can be obtained by diﬀerent turbulence models. Shear Stress Transport Model (SST) The Shear Stress Transport (SST) model of Menter [12] belongs to the eddyviscosity models. It combines the advantages of the k-ǫ and k-ω turbulence models. The k-ω formulation is solved at the wall and k-ǫ in the bulk flow. The wall treatment is taken into account by the near wall formulation of the low Reynolds number k-ω model and switches automatically to a wall function treatment based on the grid refinement. The transport equation for the turbulent kinetic energy k and the specific dissipation rate ω are given by (4) and (5). ¯j k) ∂(ρk) ∂(ρU ∂ μt ∂k + = Pk − β ∗ ρkω + [(μ + ) ] ∂t ∂xj ∂xj σ ˜k ∂xj

(4)

¯j ω) ∂(ρω) ∂(ρU ˜ 2 + ∂ [(μ + μt ) ∂ω ] + = αρS ˜ 2 − βρω ∂t ∂xj ∂xj σ ˜ω ∂xj 2 ∂k ∂ω + (1 − F1 )ρ σω ∂xj ∂xj

(5)

In (5) F1 corresponds to a blending function which ensures a smooth transition between both models. This blending function equals to zero in regions outside the boundary layers and switches to one inside the boundary layer. Pk is a production limiter which prevents the build up of turbulence in stagnation regions. The turbulent eddy viscosity is defined by (6). F2 corresponds to a second blending function and S considers the invariant measure of the strain rate.

434

A. Widenhorn et al.

μt =

a1 k max(a1 ω, SF2 )

(6)

The definition of both blending functions, production limiter and the constants of the SST turbulence model can be found in [12, 13]. The hybrid RANS/LES turbulence models are based on the SST model. Detached Eddy Simulation Model (DES) Generally DES is not bounded on a certain underlying statistical turbulence model. Spalart introduced a DES approach which is based on a one equation RANS model [14]. The model of Strelets [15], which is used in this work, combines the two equation SST-RANS model with elements of LES methods. The SST-RANS model is used to cover the boundary layer and switches to a LES mode in regions where the turbulent length Lt predicted by the RANS model is larger than the local grid spacing. In this case the length scale used to compute the dissipation rate in the equation for the turbulent kinetic energy is replaced by the local grid spacing Δ. In contradiction to the SST model the destruction term in the k-equation depends on the turbulent length scale. ¯j k) ∂ μt ∂k ∂(ρk) ∂(ρU + = Pk − β ∗ ρkωFDES + [(μ + ) ] ∂t ∂xj ∂xj σ ˜k ∂xj

(7)

The factor FDES which is the multiplier to the destruction term in (7) is defined as: √ Lt,SST k FDES = max( ; Lt,DES = CDES Δ , 1); Lt,SST = (8) Lt,DES βω An issue of this approach for some applications is the grid induced separation of the flow in the boundary layer [16]. This phenomenon, which can occur when the LES mode in the boundary layer becomes active, is known for models which rely on the modelling of the turbulent length scale as a function of the local grid size. To reduce this issue a zonal DES limiter, which includes the blending function of the SST turbulence model, is introduced by (9). FDES−CF X = max(

Lt,SST (1 − FSST ), 1) Lt,DES

(9)

The model constants are given in [13, 15, 16] Scale Adaptive Simulation Model (SAS) In principle the SAS is similar to the DES model and allows resolving partially the turbulent spectrum. In contradiction to the standard turbulence models which provide a length scale proportional to the thickness of the shear layer,

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

435

SAS adjusts dynamically to the length scale of the resolved structures. The length scale of the resolved eddies is taken into account by the introduction of the von Karmann length scale into the turbulence scale equation. This information allows the SAS model to operate in LES like mode. Thus, usually in attached boundary layers the RANS model is solved. The model is based on the k-kL formulation given in Menter and Egorov [17, 18]. Menter [19] transformed the term containing the von Karmann length scale to the SST model. This transformation results in a modified transport equation for the specific dissipation rate ω of the SST model. The new source term contains two independent scales. In addition to the standard velocity gradient tensor the von Karmann length scale, which is computed from the second derivative of the velocity field is introduced. ¯j ω) ∂(ρω) ∂(ρU ˜ 2 + ∂ [ μt ∂ω ] + = αρS ˜ 2 − βρω ∂t ∂xj ∂xj σ ˜ω ∂xj 2ρ 1 ∂k ∂ω + + FSAS−SST σΦ ω ∂xj ∂xj

(10)

The additional term is given by (11): FSST −SAS = −

2ρ k ∂ω ∂ω L + ζ¯2 κρS 2 2 σΦ ω ∂xj ∂xj Lνk

(11)

In order to preserve the SST model in the RANS regions a modified formulation of (11) is used. L FSST −SAS = ρFSAS max[ζ¯2 κS 2 Lνk 2 1 ∂ω ∂ω 1 ∂k ∂k − kmax( 2 , ; 0] σΦ ω ∂xj ∂xj k 2 ∂xj ∂xj

(12)

Since the grid spacing is not an explicit term in (12) the SAS model can operate in scale resolving mode without explicit grid information. The issue of grid induced separation of the flow in the boundary layer as it can appear in the DES model is eliminated. The model constants are given in [19].

3 Numerical Method The simulations were performed appling the commercial software package ANSYS CFX 10.0. The fully implicit solver is based on a finite volume formulation for structured and unstructured grids. Mulitgrid strategy is used to solve the linear set of coupled equations. For the spatial discretization a high order resolution scheme which is essentially second order accurate and bounded is used. For DES and SAS simulations a non-dissipative second order central scheme is applied in the detached regions. This is necessary to avoid excessive numerical

436

A. Widenhorn et al.

diﬀusion which would interfere the resolution of the turbulent structures. In RANS regions the hybrid turbulence model switches back to the second order accurate upwind based scheme. For the time discretization an implicit second order time diﬀerencing scheme is used. The parallelisation in CFX is based on the Single Program Multiple Data (SPMD) concept. The numerical domain is decomposed into tasks which can be executed separately. The communication between the processes is realized using the Message Passing Interface (MPI) utility. The partitioning process is fully automated and the memory usage is equally distributed among all processors.

4 Results and Discussion 4.1 Test Case The simulated gas turbine model combustor is schematically illustrated in Fig. 1. The combustor is a modified version of an aero engine combustor [20]. Air at room temperature and atmospheric pressure is supplied from a common plenum and admitted through an approximately annular nozzle to the flame. Non-swirling gaseous fuel is injected between the two co-rotating air flows. The annular fuel injection slot is divided in 72 sections with an area of each segment 0.5 × 0.5 mm2 . The exit plane of the outer air nozzle is taken as the reference height x = 0 mm. The combustion chamber, which permits a good optical access, consists of 4 quartz plates held by four posts in the corners. The square cross section of the chamber is 85 × 85 mm2 and the height is 114 mm. The combustion chamber is connected via a conical top plate to a central exhaust pipe (diameter 40 mm, length 50 mm). Microphones are installed in

Fig. 1. Schematic of the gas turbine model combustor

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

437

the plenum and the combustion chamber to detect the pressure fluctuations. Depending on the load conditions the combustion within this combustor is stable or strongly oscillating. In this work a non-reacting case is investigated. 4.2 Numerical Setup The computational grid consists of 1.6 million grid points. For the nozzle and the combustion chamber an unstructured hexaeder grid with 1.4 million grid points was created. In regions of potential turbulence generation and large velocity gradients a fine mesh was used in order to fulfil the LES requirements. Furthermore, the growth of the adjacent cells was limited to 10% in these zones. For the plenum an unstructured tetrahedral mesh was used. It consists of 1.2 million tetrahedral elements and 0.2 million grid points. At the air inflow boundary condition a total pressure boundary of 256 Pa is set. This corresponds to an air mass flow of 0.00468 kg/s. The air temperature is set to

Fig. 2. Computational grid of the gas turbine model combustor on a cutting plane

Fig. 3. Computational domain of the gas turbine model combustor with numerical boundary conditions

438

A. Widenhorn et al.

293 K. At the fuel inlet an equivalent air mass flow was used instead of CH4. This was also done in the experimental work. The numerical boundary condition at the fuel inlet specifies a total pressure of 66 Pa which corresponds to a mass flow of 0.000362 kg/s at 293 K. The wall of the plenum, the nozzle and the combustion chamber is assumed to be adiabatic. The reference pressure is set to 100000 Pa. 4.3 Averaged Proﬁles Figure 4 shows the calculated flow pattern of the time-averaged axial velocity. The black lines represent the locations of zero velocity. The negative values in the centre indicate the inner recirculation zone (IRZ) which occurs due to vortex break down. The outer recirculation zone (ORZ) is developing in the corners of the combustion chamber. The IRZ and ORZ zones are unsteady and their positions oscillate with time. The jet between the inner and outer recirculation zone represents the region of the inflow of the fresh gas. Note that in the complex flow inside the plenum further recirculation zones are formed. Figure 5 shows the location of positions where a comparison of measured and calculated averaged velocity profiles will be presented. Furthermore, a representative blending function of the DES model is visualized. The blending function controls the local proportion of the RANS and DES mode. In regions where the blending function is zero, the LES mode of the model is switched on. If the blending function is one the underlying RANS model is used. Since the flow is unsteady the blending function changes with time. It is clearly visible that the model changes near the wall and at the inlet and outlet boundaries to the RANS mode. In the region where strong turbulence generation is expected and high velocity gradients as well as large turbulent length scales appear the model switches into the LES mode. This allows resolving the turbulent structures. The transient simulation was started from a steady state solution. After a start-up phase of 2 combustor residence times the averaging of the velocities was started. The averaging was performed over two residence times. Figure 6

Fig. 4. Averaged axial velocity on a cutting plane (DES)

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

439

Fig. 5. Position of measurements and DES blending function

(a) x = 5 mm

(b) x = 10 mm

(c) x = 20 mm

(d) x = 60 mm

Fig. 6. Averaged axial velocity proﬁles at x = 5 mm, x = 10 mm, x = 20 mm, x = 60 mm

440

A. Widenhorn et al.

(a) x = 5 mm

(b) x = 10 mm

(c) x = 20 mm

(d) x = 60 mm

Fig. 7. Averaged tangential velocity proﬁles at x = 5 mm, x = 10 mm, x = 20 mm, x = 60 mm

show the comparison of the numerically obtained axial averaged velocity profiles with appropriate PIV measurements. Numerical simulations were carried out applying the SST, DES and SAS turbulence model. At the position x = 5 mm which is close to the burner mouth the simulated averaged axial velocity agree well with the experiment. Whereas the DES and SAS models can reproduce the velocity profiles very accurately the SST model slightly under predicts the strength of the recirculation zone. The same behaviour can be observed at the position x = 10 mm and x = 20 mm. At x = 60 mm all models predicts the axial velocity again very well for the regions near the burner centreline but deviates towards the combustion chamber wall. The comparison of the simulated averaged tangential velocity profiles obtained by using the DES and SAS turbulence model shows an excellent agreement with the experiment at x = 5 mm, x = 10 mm, x = 20 mm and x = 60 mm. For the position at x = 60 mm no experimental data for the tangential velocity component are available. The SST turbulence model predicts always the peeks very well but slightly under predicts the tangential component toward the wall. This region altogether is better covered by the hybrid RANS/LES approaches.

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

441

4.4 Unsteady Analysis Time averaged data are informative to assess the general feature of the flow, but do not describe the full flow characteristics which is highly unsteady. Exemplarily Fig. 8 shows a planar cut coloured with the instantaneous axial velocity contours and the zero axial velocity isolines. A typical instability in non-reacting swirling flows at high swirl numbers is the precessing vortex core (PVC). This vortex system can be extracted from the results of the DES and SAS calculations. Figure 9 shows the instantaneous views of the DES and SAS results using a low pressure isosurface. The precessing vortex core (PVC) is rotating around the burner axis in the same direction as the imposed swirl. Figure 10 shows a further unsteady behaviour of the flow. For the visualisation of the flow an instantaneous velocity vector plot is used. In the inner shear layers vortex structures, which propagate in time are clearly visible. The vortices are formed inside the fuel/air nozzle in the inner air flow section. Due to these vortices the turbulent mixing inside the combustor is

Fig. 8. Instantaneous axial velocity

Fig. 9. Precessing vortex core visualised by a low pressure isosurface (left DES, right SAS)

442

A. Widenhorn et al.

Fig. 10. Instantaneous vortex structures

enhanced significantly. Thus the resolution of such flow phenomena might be decisive for the quality of a simulation of the turbulent combustion processes. This eﬀect will be elaborated further during the next simulation runs for the reacting flow case.

5 Computational Resources The simulations have been performed on the HP XC4000 system. According to our experience to obtain a statistically converged solution a total integration time of at least 4 combustor residence times is required. This corresponds for the applied test case to a simulated real time of 0.36s. Furthermore, since the SAS and DES turbulence models are applied a relatively fine mesh have to be used to achieve low CFL numbers and hence the numerical eﬀort is very large. A typical grid which is applied for the simulation has 1.6 million grid points. In order to perform the calculations within adequate turnaround times the CPU numbers diﬀers between 20 and 24. Table 1 contains the turnaround times which were needed to perform the numerical runs. In all simulations one time step consists of three inner iterations loops. The time step is set to 1e-5s. This leads to 36000 time steps and 108000 inner iteration loops per run. Figure 11 illustrates the speed up of parallelization for the applied test case. The chart has been generated using the DES turbulence model. Table 1. Comparision of the required turnaround times Turbulence Model SST Turbulence Model DES Turbulence Model SAS Turbulence Model

Elapsed Time 28.12 days 34.86 days 35.12 days

Non-Reacting Flow in a Swirled Gasturbine Model Combustor

443

Fig. 11. Speed up of parallelization

6 Conclusions The potential of the hybrid RANS/LES turbulence models for the simulation of the non-reacting flow in gas turbine combustion chambers has been worked out. In general the results show a remarkable predictive capacity of these methods and also indicate the gain of such turbulence models which might be decisive in reacting flow calculations. Compared to the classical LES approach the DES and SAS models saves order of magnitudes of computing power, since the boundary layers are RANS regions. Another advantage is the stationary RANS boundary conditions formulation which can be applied instead of the unsteady LES boundary conditions. Nevertheless, high performance computing is necessary to perform the calculations within adequate turnaround times. The numerically obtained averaged velocity components match very well for all applied turbulence models appropriate PIV measurements. Especially, the hybrid RANS/LES models are able to capture the inner recirculation zone very well. The dynamic behaviour of the non-reacting flow is visualised. A precessing vortex core (PVC) could be found in the combustion chamber. Acknowledgements The authors would like thanks the High-Performance Computing Centre Karlsruhe for the always helpful support and the computation time on the high performance computers.

444

A. Widenhorn et al.

References 1. Liang, H., Maxworthy, T.: An Experimental Investigation of Swirling Jets. J. Fluid Mech., 525, pp. 115–159 (2005) 2. Cala, C.E., Fernandez, E.C., Heitor, M.V., Shtork, S.I.: Coherent Structures in Unsteady Swirling Jet Flows. Exp. Fluids, 40, pp. 267–276 (2006) 3. Fernandez, E.C., Heitor, M.V., Shtork, S.I.: An Analysis of Unsteady Highly Turbulent Swirling Flow in a Model Vortex Combustor. Exp. Fluids, 40, pp. 177–187 (2006) 4. Midgley, K., Spencer, A., McGuirk, J.J.: Unsteady Flow Structures in Radial Swirler Fed Fuel Injectors. ASME Turbo Expo 2004, GT2004-53608 5. Thompson, K.W.: Time Dependent Boundary Conditions for Hyperbolic Systems. J. of Comput. Phys., 68, pp. 1–24 (1987) 6. Poinsot, T., Lele, S.: Boundary Conditions for Direct Simulations of Compressible Viscous Flows. J. of Comput. Phys., 101, pp. 104–129 (1992) 7. Widenhorn, A., Noll, B., Aigner, M.: Accurate Boundary Conditions for the Numerical Simulation of Thermoacoustic Phenomena in Gas-Turbine Combustion Chambers. ASME Turbo Expo 2006, GT2006-90441 8. Noll, B.: Numerische Str¨ omungsmechanik. Springer Verlag (1993) 9. Noll, B., Sch¨ utz, H., Aigner, M.: Numerical Simulation of High Frequency Flow Instabilities Near an Airblast Atomizer. ASME Turbo Expo 2001, GT2001-0041 10. Schl¨ uter, J., Sch¨ onfeld, T., Poinsot, T., Krebs, W., Hoﬀmann, S.: Characterization of Conﬁned Swirl Flows Using Large Eddy Simulation. ASME Turbo Expo 2001, GT2001-0060 11. Selle, L., Lartique, G., Poinsot, T., Koch, R., Schildmacher, K.U., Krebs, W., Kaufmann, P., Veynante, D.: Compressible Large Eddy Simulation of Turbulent Combustion in Complex Geometry on Unstructered Mesh. Comb. and Flame, 137, pp. 489–505 (2004) 12. Menter, F.R.: Two Equation Eddy Viscosity Turbulence Models for Engineering Applications. AIAA Journal 32(8), pp. 269–289 (1995) 13. CFX 10.0 User Guide 14. Spalart, P.R.: Strategies for Turbulence Modelling and Simulations. Int. J. Heat Fluid Flows 21, pp. 252–263(2000) 15. Strelets, M.: Detached Eddy Simulation of Massively Separated Flows, AIAA Paper, 2001-0879 16. CFX Validation Report: Development and application of a zonal DES turbulence model for CFX5. 17. Menter, F.R., Egorov, Y.: Re-visting the Turbulent Scale Equation. IUTAM Symposium (2004) 18. Menter, F.R., Egorov, Y.: A Scale-Adaptive Simulation Model Using TwoEquation Models. AIAA-Paper, 2005-1095 19. Menter, F.R., Kuntz, M., Bender, R.: A Scale Adaptive Simulation Model for Turbulent Flow Prediction. AIAA Paper, 2003-0767 20. Giezendanner, R., Keck, O., Weigand, P., Meier, W., Meier, U., Stricker, W., Aigner, M.: Periodic Combustion Instabilities in a Swirl Burner Studied by Phase-Locked Planar Laser-Induced Fluorescence. Combust. Sci. Technol., 175, pp. 721–741 (2003)

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers for Two-Phase Flow Simulations D. Gaudlitz and N.A. Adams Institute of Aerodynamics, Technische Universit¨ at M¨ unchen, D-85748 Garching, Germany [email protected] Summary. The hybrid particle-level-set method (HPLS) is an extension of the established level-set technique and allows for an eﬃcient description of moving interfaces. With level-set methods phase interfaces are treated implicitly and hence complex shape changes as well as merging and breaking up of geometries can be handled. The HPLS-method additionally employs marker particles to improve massconservation properties of the classical level-set scheme. Subject of the present paper is the eﬃcient implementation and the application to large-scale computations of this method. In simulations of two-phase ﬂows the major part of computational operations for the multi-phase model occur in the vicinity of the interface. The implementation of these operations on parallel vector systems requires special attention. Computational results of gas bubbles rising in liquids show good agreement with the experimental data and conﬁrm the eﬃciency and accuracy of the HPLS-scheme.

1 Introduction For the computation of multi-phase flows diﬀerent numerical methods have been developed. An overview of the most widely used schemes can be found e.g. in Kothe [11], Unverdi and Tryggvason [23], Sethian and Smereka [18], and Scardovelli and Zaleski [17]. Since their initiation by Osher and Sethian [14] level-set schemes have been subject to further development and improvement [13, 18] and are nowadays an attractive alternative to volume-of-fluid or particle-tracking approaches. Level-set methods represent phase boundaries implicitly as the zero-level-set of a scalar grid function φ(x, t). Away from the interface the level-set function φ gives the signed distance to the interface, being negative in the gas phase and positive in the liquid phase. Using the flow velocities the levelset distribution is evolved in time by an advection equation and after each time step a reinitialization-procedure [13, 18] is carried out to maintain the signed-distance property. In contrast to volume-of-fluid methods and particle-

446

D. Gaudlitz, N.A. Adams

tracking techniques formally no reconstruction process of the interface is necessary with level-set methods. In addition, the computation of quantities at the interface, e.g., the local curvature and the unit normal vector is straightforward due to the continuous definition of the level-set function. However, in regions of high curvature the interface might become marginally resolved by the underlying computational mesh. In these areas level-set methods are known to suﬀer from mass losses. Among the proposed remedies for this numerical error are the use of higher-order discretization schemes [13, 18], the introduction of additional constraints in the reinitialization process of the level-set function [19], as well as employing modified sign-functions during reinitialization [15, 20]. The hybrid particle-level-set method (HPLS) [6] uses marker particles to correct mass errors in the level-set representation of the interface. In general, Lagrangian marker-particle schemes [16] allow for a highly accurate interface description but the reconstruction of complex interfaces in 3D and the merging or breaking up of geometries involves rather large computational eﬀorts. The HPLS-method takes advantage of the grid-independent accuracy given by marker particles. Therefore, two sets of massless particles are placed next to the phase boundary: Positive particles are seeded in the liquid phase (φ > 0) and negative ones in the gas phase (φ < 0). The particles are advected with the flow field. If after computing a time step there are particles appearing on the wrong side of the interface, e.g. negative particles are found in the liquid phase, this indicates that the interface prediction of the level-set method has to be corrected to maintain mass conservation. In Ref. [6, 8] details on the correction procedure using these so-called ’escaped’ particles can be found. The superiority of the HPLS-scheme regarding eﬃciency and accuracy compared to pure level-set approaches has been shown in [7] using a generic numerical test case. In summary, with the HPLS-scheme the simple and smooth interface description by level-set methods is maintained but the accuracy of the method has been increased by means of particle tracking. In Sect. 2 of the present paper we review the basic equations of the HPLSmethod. We focus on eﬃcient algorithms for the implementation of level-set methods in Sect. 2.1. In Sect. 2.2 optimized and fully parallel algorithms for the correction step employing the marker particles as well as for particle reseeding are described. Results of 3D numerical simulations of gas bubbles rising in quiescent liquids are compared with experimental data in Sect. 3.

2 An Optimized HPLS-Method Before going into details of the HPLS-scheme we recall the basic set of equations for computing two-phase flows. The Navier-Stokes equations for incompressible, viscous, non-reacting, immiscible two-phase flows read

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers

1 ∇p + ∇ · (2μ (φ) D) + ρ (φ) Re · ρ (φ) g 1 κ (φ) δ (φ) N (φ) , + u2 − Fr W e · ρ (φ)

447

ut + u · ∇u = −

∇·u = 0 .

(1) (2)

where u is the velocity vector, ρ and μ are the density and viscosity, respectively. D is the rate-of-deformation tensor, whose components are Dij = 1 2 (ui,j + uj,i ). g u is a unit gravity vector, κ denotes the curvature, δ is a smoothed delta function being zero everywhere except near the interface, N is the unit normal vector at the interface, and φ is the level-set function. The above equations are non-dimensionalized leading to a Reynolds number Re = √ ρc LU/μc , a Weber number W e = ρc LU 2 /σ, and a Froude number F r = U/ gL. Herein ρc and μc are the density and dynamic viscosity of the continuous phase, respectively. L is a reference length, U a reference velocity and σ is the surface-tension coeﬃcient. For solving the Navier-Stokes equations a projection method [4] is used. The equations are discretized using finite diﬀerences on an equidistant staggered grid. For the results presented in this paper periodic boundary conditions are applied in all three coordinate directions of the Cartesian domain. Explicit time advancement is done using a third-order Runge-Kutta scheme. For the simulation of two-phase flows we employ the HPLS-method which combines a level-set scheme with a particle algorithm. In the following both parts are discussed individually with respect to an eﬃcient implementation on high-performance computing systems. 2.1 Eﬃcient Implementation of a Level-Set Scheme The level-set function ist evolved in time by φt + u · ∇φ = 0.

(3)

We employ a high-order WENO scheme [9, 10] for spatial derivatives and third-order interpolation to obtain the velocities in the cell center. The vectorization and parallelization by domain decomposition of the computational algorithms for solving (3) is straightforward. The same holds for the reinitialization equation φ φτ + sign (φ) · (|∇φ| − 1) = 0 , where sign(φ) = : 2 φ + △x2

(4)

involving similar discretization schemes as in (3) for its numerical solution. When solving the Navier-Stokes equations the spatially varying density and viscosity distribution are given by ρ(φ) = ρd + (ρc − ρd ) · H(φ) ,

(5)

448

D. Gaudlitz, N.A. Adams

μ(φ) = μd + (μc − μd ) · H(φ) .

(6)

To avoid numerical instabilities the jump of these variables at the interface is smoothed within a distance ǫ using a smoothed Heaviside function [13, 18] ⎧ , if φ < −ǫ ⎨0 φ πφ 1 1 (7) H(φ) = 2 (1 + ǫ + π sin( ǫ )) , if −ǫ ≤ φ ≤ ǫ . ⎩ 1 , if φ > ǫ

A typical smoothing length is two grid spacings, ǫ = 2 · h with h being the mesh spacing. The original implementation of (7) involved an evaluation of conditional if -statements at every grid node and therefore reduced computational performance significantly. To increase eﬃciency of the method a band at the interface is defined with a bandwidth based on the chosen value for ǫ, see Fig. 1. All grid points located within this band form a proximity-list which is updated after every time step. Now a simple flag field, specifying wether a point is located in the gas or the liquid phase, is used for generating the density and viscosity distributions. The resulting discontinuity of these variables at the interface is smoothed by evaluating (5), (6), (7) only for the grid points contained in the proximity-list. In (1) the surface-tension force Fsurf =

1 κ (φ) δ (φ) N (φ) W e · ρ (φ)

(8)

is included as an additional source term. Again, the proximity-list is used to compute the unit normal vector N (φ) = and the curvature κ(φ) = ∇ ·

∇φ |∇φ|

∇φ |∇φ|

(9)

(10)

Fig. 1. Deﬁnition of a band at the interface. Thick black line describes the interface, green region contains all points of the proximity-list

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers

449

only at points near the interface, thus reducing the computational work. With the continuum-surface-force approach [2] a mollified delta-Dirac function πφ 1 1 + cos , if |φ| < ǫ ǫ δ(φ) = 2ǫ (11) 0 , otherwise, is employed to smooth the singular surface-tension force at the phase boundary. Computational costs can be reduced further by initializing the delta function with zero and evaluating (11) with if -statements and trigonometric operations only within the band at the interface. After implementing the mentioned modifications and parallelizing the code based on the OpenMP standard a numerical simulation of an air bubble rising in water has been conducted. This computation was performed with a pure level-set scheme on a grid of 6.75 Mio. points and using 4 CPUs on one node of the NEC SX-8 supercomputer at HLRS1 . This simulation revealed a performance of 6.2 GFLOPs per CPU. We note, that a parallel version of the code following a hybrid approach consisting of a domain decomposition between nodes with the use of MPI libraries for communication and a shared-memory parallelization using OpenMP within each node is available by now. 2.2 Particle Algorithm After initialization of the level-set function negative and positive particles are seeded within the gas and liquid phase respectively. The particles are associated with a radius rP which is adjusted after each time step according to ⎧ , if signP · φ(xP ) > rmax ⎨ rmax rP = signP · φ(xP ) , if rmin ≤ signP · φ(xP ) ≤ rmax , (12) ⎩ , if signP · φ(xP ) < rmin rmin

where signP denotes the sign of the particle and rmin = 0.1 · h is the lower and rmax = 0.5 · h is the upper limit of the particle radii. The particle positions xP are evolved in time by dxP = u(xP ) , (13) dt where particle velocities are trilinearly interpolated from the underlying grid. After each time step the particles are used to correct possible errors in the level-set function. This process consists of two steps. First, errors of the interface representation given by the zero-level-set have to be identified. Therefore the level-set values at all particle positions φP are interpolated which is equivalent to computing the distance of the particles to the interface. In case a particle has crossed the interface by a distance of more than its radius this particle is considered to be ’escaped’. Second, to correct these errors a new 1

H¨ ochstleistungsrechenzentrum Stuttgart

450

D. Gaudlitz, N.A. Adams

prediction of the level-set value φP C (x) at a grid point with the coordinates x using all escaped particles is computed by φP C (x) = signP (rP − |x − xP |) .

(14)

The correction algorithm has the following rules: The set of escaped positive particles E + is used to reconstruct the φ > 0 region, while the φ < 0 region is reconstructed by the escaped negative particles E − , and for each grid point the closest escaped particle is used to correct the level-set function. Hence for each point we compute φ+ = max+ (φP C , φ) , and

(15)

φ− = min− (φP C , φ) .

(16)

p∈E

p∈E

Finally, the corrected local level-set value is obtained by + φ , if |φ+ | ≤ |φ− | φ= . φ− , if |φ+ | > |φ− |

(17)

As only the grid points included in the proximity-list are considered for particle correction computational work is decreased. All points outside the band will be updated by the next reinitialization step. Furthermore, for finding the maximum of (φP C , φ) in (15) the list of escaped positive particles E + can be split into sublists for parallelization. A reduction operation at the end of the search will then give the global maximum φ+ . The same holds for (16) and the set of escaped negative particles E − . Another challenging task is the eﬃcient implementation of reseeding algorithms for particles. When progressing in time shear flow and bubble deformations will lead to an inhomogeneous particle-density distribution along the interface. Therefore, we periodically perform an adaptive reseeding procedure featuring three major tasks: First, particles which have moved too far away from the interface to eﬀect a correction of the latter are deleted. Second, in cells where there are less particles than a prescribed particle-number per cell new particles are seeded randomly. And last, in overcrowded cells the particles with the largest distance to the interface are deleted until the chosen particle-number per cell is reached. A very first implementation of the reseeding procedure reduced the overall performance of the code to some hundred MFLOPs per CPU, although reseeding has been carried out every 40 time steps only. In order to achieve a vectorizable algorithm we introduced a list holding the global indices iGC = i + j · N 1 + k · N 1 · N 2 of all cells contained in the proximity-list. Here i, j and k are the indices in the three coordinate directions having a total number of cells of N 1, N 2, and N 3, respectively. Based on their coordinates xP the particles are associated with computational cells and hence a list containing the global cell index iGP of each particle can be

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers

451

created. Now the total number of particles contained in cell iGC can be determined through a summation operation carried out on the particle array which has been masked by the condition iGP = iGC . In that way either additional particles can be seeded or particles with the largest values of |φP | can be deleted to adjust the actual particle count in cell iGC . This new algorithm allows for an OpenMP -parallelization and is vectorized. When computing the two-phase flow problem of Sect. 2.1 but now including the particle algorithm we measured a performance of 6.05 GFLOPs per CPU. For large scale computations employing grids of up to 44 · 106 points we found the HPLS-scheme to incur approximately 15 − 18% of the overall computational time. The greatest potential for further enhancing eﬃciency of the code is found in the solver for the pressure-Poisson equation. For an iterative solution of this equation we currently employ a preconditioned ConjugateGradient method since other strategies such as Multi-Grid appear not to be superior for the considered flows.

3 Simulations of Rising Gas Bubbles in Liquids To validate our HPLS-implementation we have computed air bubbles rising in water-glucose solutions with flow parameters set as in the experiments of Bhaga and Weber [1]. We considered two cases where the bubble volume was 9.3cm3 giving a diameter d = 2.61cm. The selected parameters for Case 1 are a Morton number of M o = 41.1, an Eotvos number of Eo = 116, and a Reynolds number of Re = 7.2. For Case 2 we chose M o = 0.103, Eo = 116, and Re = 42.2. For Case1 the computational domain consisted of 8.2 · 106 grid points resolving the bubble diameter d by 40 points. For Case 2 we used a refined grid of 27.6 · 106 points to be able to properly resolve the thin outer regions of the flat ellipsoidal bubble. The simulations were started with a spherical bubble being initially at rest. After an acceleration phase the bubbles reach their final shapes which are in good agreement with the experimental findings, see Fig. 2, 3, 4(b) and 4(c). The terminal rising velocities UT of the bubbles were measured with 0.19m/s

(a) Experiment

(b) Numerical simulation

Fig. 2. Bubble shape for Case 1 : Mo = 41.1, Eo = 116, and Re = 7.2

452

D. Gaudlitz, N.A. Adams

(a) Experiment

(b) Numerical simulation

Fig. 3. Bubble shape for Case 2 : Mo = 0.103, Eo = 116, and Re = 42.2 2

Bubble width w/d

w

h hs hw

1.6

1.2

0.8 -1 10

ww

(a) Deﬁnition of bubble and wake geometry.

103

0.8

Drag coefficient cD

Bubble height h/d

101 Re

0.6 0.4 0.2

102

(b) Dimensionless bubble width.

1

0 -1 10

100

Clift et al. Mei et al. Taylor, Acrivos

102

101 ⊗ ⊗⊗⊗⊗⊗⊗

100

101 Re

100 -1 10

102

(c) Dimensionless bubble height.

10

0

1

10 Re

10

2

10

3

(d) Bubble drag coeﬃcient.

Fig. 4. Comparison of experimental und numerical results. Values of Morton numbers in the experiments are: : 711; : 55.5; △: 4.17; ⋄:1.03; ▽: 0.108; ⊳: 5.48·10−3 ; ⊗: 1.64 · 10−3 . Numerical simulations: : Case 1; : Case 2

and 0.33m/s for Case 1 and Case 2, respectively. In Fig. 4(d) drag coeﬃcients as a function of the Reynolds number are shown, where the computational results have been obtained by cD =

4·g·d . 3 · UT2

(18)

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers

453

The solid line in Fig. 4(d) corresponds to the empirical drag law of Mei et al. [12] which has been confirmed to be valid for a large range of Reynolds numbers by Takagi and Matsumoto [21]. However, Mei’s drag law seems to be suited only for bubbles which remain almost spherical because it shows large deviations from the experimental and numerical results when bubbles become flat ellipsoids. The relation cD = 14.9/Re0.78 proposed by Clift et al. [5] is able to predict the drag coeﬃcient at low Reynolds numbers with suﬃcient accuracy. The best match with the presented data is obtained using the drag law of Taylor and Acrivos [22] cD = (16/Re) + 2. Due to the high liquid viscosity the Reynolds numbers are low and the bubbles develop a laminar closed toroidal wake. In Fig. 5(a), 5(b), and 5(c) geometry-parameters of the trailing wake as a function of the Reynolds number are shown. Again, the numerical results agree well with the experimental data.

3

2.4

⊕

2

Wake length hw / d

Wake width wW / d

2.8

⊗ ⊕

1.6

⊕ ⊕

1.2 0.8 0.4 0 0 10

101

⊕

2.5

⊗ ⊕

2 1.5

⊕

1

⊕

0.5 0 0 10

102

101

Re

102 Re

Stagnation ring position, hS / d

(a) Dimensionless width of bubble wake.

(b) Dimensionless length of bubble wake.

1.2 1

⊕

0.8 ⊗ ⊕

0.6 ⊕

0.4

⊕

0.2 0 0 10

101

102 Re

(c) Dimensionless position of the wake stagnation ring. Fig. 5. Comparison of experimental und numerical results. Values of Morton numbers in the experiments are: : 258; : 43.5; △: 4.41; ⋄:0.962; ▽: 0.144; ⊳: 4.58·10−3 ; ⊗ : 1.73 · 10−2 ; ⊕ : 6.5 · 10−2 . Numerical simulations: : Case 1; : Case 2

454

D. Gaudlitz, N.A. Adams

4 Summary In the present paper we have discussed approaches for an eﬃcient implementation of the HPLS-method for the computation of incompressible two-phase flows. The HPLS-method can be regarded as superior to pure level-set schemes because mass-conservation properties are improved significantly by means of marker-particle corrections. This allows for a decrease of resolution by the computational mesh while still preserving a high accuracy of the interface description throughout the simulation. For the computation of density and viscosity distributions and the surface tension force within the level-set part of the HPLS-scheme eﬃcient algorithms have been devised and computational work is confined to a region close to the interface. The correction step of the level-set function using the particles has been fully parallelized. A parallelized and partially vectorized adaptive reseeding algorithm for the particles has been developed. In summary, the described optimizations improved the performance of the HPLS-scheme considerably which consumes approximately 15 − 18% of the overall computational time in large scale computations. Furthermore, the HPLS-scheme has been successfully applied to two-phase flows. Due to the good agreement of the experimental and numerical results detailed in Sect. 3 this method can be recommended for the computation of more complex configurations, e.g. the computation of bubbles featuring an unsteady open wake and rising on zig-zagging or helical paths. Prior to large-

(a) Bubble rising path.

(b) Bubble shape and trailing vorticies. The red and blue color show two isocontours of ωy having the same magnitude but opposite sign.

Fig. 6. Preliminary simulation of a 3D bubble rising on a zig-zag path

On Implementing the Hybrid Particle-Level-Set Method on Supercomputers

455

scale computations of such flows a preliminary simulation of a flat ellipsoidal bubble rising on a zig-zag path has been done, see Fig. 6. The parameters for this case are chosen as given by Br¨ ucker [3], who performed detailed experimental investigations of bubble shapes and the bubble wakes. In Fig. 6(b) the bubble shape and its turbulent wake is shown at an instant in time during rise. Acknowledgements The work is supported by the German Research Council (DFG) within SFB 609 and under contract AD 186/7-1. The simulations were performed on the national super computer NEC SX8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number 2FLOWMAG/12785.

References 1. D. Bhaga and M.E. Weber. Bubbles in viscous liquids: shapes, wakes and velocities. J. Fluid Mech., 105:61–85, 1981. 2. J. U. Brackbill, D. B. Kothe, and C. Zemach. A continuum method for modeling surface tension. J. Comput. Phys., 100:335–354, 1992. 3. C. Br¨ ucker. Structure and dynamics of the wake of bubbles and its relevance for bubble interaction. Phys. Fluids, 11:1781–1796, 1999. 4. A. Chorin. A numerical method for solving viscous ﬂow problems. J. Comput. Phys., 12, 1967. 5. R. C. Clift, J. R. Grace, and M. E. Weber. Bubbles, Drops and Particles. Academic, New York, 1978. 6. D. Enright, R. Fedkiw, J. Ferziger, and I. Mitchell. A hybrid particle level set method for improved interface capturing. J. Comput. Phys., 183:83–116, 2002. 7. D. Gaudlitz and N.A. Adams. A hybrid particle-level-set method for two-phase ﬂows. In Proc. Appl. Math. Mech., pages 506–507. PAMM, 2004. 8. D. Gaudlitz and N.A. Adams. The hybrid particle-level-set method applied to two-phase ﬂows. In Proceedings of FEDSM2006. ASME, ASME Joint U.S.European Fluids Engineering Summer Meeting, Miami, FL, July 2006. 9. G. S. Jiang and D. Peng. Weighted eno schemes for hamilton-jacobi equations. SIAM J. Sci. Comput., 21:2126–2143, 2000. 10. G. S. Jiang and C. W. Shu. Eﬃcient implementation of weighted eno schemes. J. Comput. Phys., 126:202–242, 1996. 11. D. B. Kothe. Free Surface Flows, chapter Perspective on Eulerian ﬁnite volume methods for incompressible interfacial ﬂow, pages 267–331. Springer, 1998. 12. R. Mei, J. F. Klausner, and C. J. Lawrence. A note on the history force on a spherical bubble at ﬁnite reynolds number. Phys. Fluids, 6:418–420, 1994. 13. S. Osher and R. P. Fedkiw. Level set methods: An overview and some recent results. J. Comput. Phys., 169:463–502, 2001. 14. S. Osher and J. A. Sethian. Fronts propagating with curvature-dependent speed: Algorithms based on hamilton-jacobi formulations. J. Comput. Phys., 79:12–49, 1988.

456

D. Gaudlitz, N.A. Adams

15. D. Peng, B. Merriman, S. Osher, H. Zhao, and M. Kang. A pde-based fast local level set method. J. Comput. Phys., 155:410–438, 1999. 16. W. Rider and D. Kothe. A marker particle method for interface tracking. In Proceedings of the Sixth International Symposium on Computational Fluid Dynamics, 1995. 17. R. Scardovelli and S. Zaleski. Direct numerical simulation of free-surface and interfacial ﬂow. Annu. Rev. Fluid Mech., 31:567–603, 1999. 18. J. A. Sethian and P. Smereka. Level set methods for ﬂuid interfaces. Annu. Rev. Fluid Mech., 35:341–372, 2003. 19. M. Sussman and E. Fatemi. An eﬃcient, interface-preserving level set redistancing algorithm and its application to interfacial incompressible ﬂuid ﬂow. SIAM J. Sci. Comput., 20:1165–1191, 1999. 20. M. Sussman, P. Smereka, and S. Osher. A level set approach for computing solutions to incompressible two-phase ﬂows. J. Comput. Phys., 114:146–159, 1994. 21. S. Takagi and Y. Matsumoto. Force acting on a rising bubble in a quiescent ﬂuid. In Proc. ASME Summer Meeting on Numerical Methods for Multiphase Flow, San Diego, CA, pages 575–580. ASME, 1996. 22. T. D. Taylor and A. Acrivos. On the deformation and drag of a falling viscous drop at low reynolds number. J. Fluid Mech., 18:466–477, 1964. 23. S. O. Unverdi and G. Tryggvason. A front-tracking method for viscous, incompressible, multi-ﬂuid ﬂows. J. Comput. Phys., 100:25–37, 1992.

Lattice Boltzmann Simulations of Microemulsions and Binary Immiscible Fluids Under Shear Jens Harting1 and Giovanni Giupponi2 1 2

Institut f¨ ur Computerphysik, Pfaﬀenwaldring 27, 70569 Stuttgart, Germany Centre for Computational Science, Chemistry Department, University College London, 20 Gordon Street, London WC1H 0AJ, UK

Summary. Large scale lattice Boltzmann simulations are utilized to investigate spinodal decomposition and structuring eﬀects in binary immiscible and ternary amphiphilic ﬂuid mixtures under shear. We use a highly scalable parallel Fortran 90 code for the implementation of the simulation method and demonstrate that adding surfactant to a system of immiscible ﬂuid constituents can change the mixture’s properties profoundly: stable bicontinuous microemulsions form which undergo a transition from a sponge to a lamellar phase by applying a constant shear. Under oscillatory shear tubular structures can be observed.

1 Introduction A mixture of immiscible fluids and surfactant is often classified as a “complex fluid”. Such systems involve both hydrodynamic flow eﬀects and complex interactions between fluid particles. In general, the macroscopic flow is aﬀected by microscopic properties. Typical examples can be found in the food, cosmetic and chemical industries: a barbecue sauce containing large fractions of water and oil or fat would phase separate into its constituents without any additives, entering a not very appealing de-mixed state. Adding an emulsifier or surfactant helps to stabilize the sauce. These molecules are often called amphiphiles and, in their simplest form, are comprised of a hydrophilic (water-loving) head group and a hydrophobic (oil-loving) tail. The surfactant molecules self-assemble on the surface of oil droplets and reduce the surface tension. Thus, the droplets stabilize and remain suspended within the bulk water. Typical emulsifiers used by the food industry are egg yolk lecithin and proteins. In contrast to “complex fluids”, “simple fluids” like water or air usually can be described to a good degree of approximation by macroscopic quantities only, such as the density ρ(x), velocity v(x), and temperature T (x). They are governed by the Navier-Stokes equations [12].

458

J. Harting, G. Giupponi

As shown by Langevin, molecular dynamics, lattice gas, and lattice Boltzmann simulations, the temporal growth law for the size of oil and water domains in a binary system without amphiphiles follows a power law tα [4, 15] and crosses over to a logarithmic growth law (ln t)θ , where α, θ are fitting parameters and t is the time [10, 14, 21]. A further increase of the surfactant concentration can lead to growth which is well described by a stretched exponential form A − B exp(−CtD ), where capital letters denote fitting parameters [10, 14]. By adjusting temperature, fluid composition or pressure, amphiphiles can self-assemble and force the fluid mixture into a number of equilibrium structures or mesophases. These include lamellae and hexagonally packed cylinders, micellar, primitive, diamond, or gyroid cubic mesophases as well as sponge phases. In this report we focus on the sponge mesophase, which in the context of our simulations is called a bicontinuous microemulsion since it is formed by the amphiphilic stabilization of a phase-separating binary mixture, where the immiscible fluid constituents occur in equal proportions. Here, the oil and water phases interpenetrate and percolate and are separated by a monolayer of surfactant at the interface. Under shear, such complex fluids show pronounced rheological properties [20]. Often, transitions from isotropic to lamellar phases can be observed [2, 27]. If oscillatory shear is applied, a further transition to a tubular phase or a transition between diﬀerently oriented lamellar phases can occur [30, 37]. Within this project, we are interested in such transitions as will be demonstrated later in this paper. Our results are only given in a shortened version here since a more detailed description has been published elsewhere [16]. Our aim is to focus on the eﬀect of surfactant concentration on the length and time scales of arrested growth and on the changes in structural property induced by steady or oscillatory shear. Computationally, the problems we are interested in are too large and expensive to tackle with atomistic methods such as molecular dynamics, yet they require too much molecular detail for continuum Navier-Stokes approaches. Algorithms which work at an intermediate or “mesoscale” level of description in order to solve these problems have been developed within the last two decades. These include dissipative particle dynamics [11], lattice gas cellular automata [31], stochastic rotation dynamics [25], and the lattice Boltzmann method [1, 35]. In particular, the lattice Boltzmann method has been found highly useful for the simulation of complex fluid flows in a wide variety of systems. This algorithm is extremely well suited to be implemented on parallel computers, which permits very large systems to be simulated, reaching hitherto inaccessible physical regimes.

2 Simulation Method and Implementation A standard LB system involving multiple species is usually represented by a set of equations [35]

Simulations of Sheared Microemulsions and Binary Fluids

459

αeq 1 α α nα (x, t)), i = 0, 1, . . . , b , i (x + ci , t + 1) − ni (x, t) = − τα (ni (x, t) − ni (1) where nα (x, t) is the single-particle distribution function, indicating the deni sity of species α (for example, oil, water or amphiphile), having velocity ci , at site x on a D-dimensional lattice of coordination number b, at time-step t. The collision operator Ωiα represents the change in the single-particle distribution function due to the collisions. We choose a single relaxation time τα , ‘BGK’ form [3] for the collision operator. In the limit of low Mach numbers, the LB equations correspond to a solution of the Navier-Stokes equation for isothermal, quasi-incompressible fluid flow whose implementation can eﬃciently exploit parallel computers, as the dynamics at a point requires only information about quantities at nearest neighbour lattice sites. The local equilibrium distribution nαeq plays a fundamental role in the dynamics of the system as i shown by (1). In this study, we use a purely kinetic approach, for which the local equilibrium distribution nαeq (x, t) is derived by imposing certain restrici tions on the microscopic processes, such as explicit mass and total momentum conservation [6] ci · u (ci · u)2 u2 (ci · u)3 u2 (ci · u) αeq α ni = ζi n 1 + 2 + , (2) − 2+ − cs 2c4s 2cs 6c6s 2c4s

where u = u(x, t) is the macroscopic bulk velocity of the fluid, defined as nα (x, t)uα ≡ i nα i (x, t)ci , ζi are the coeﬃcients resulting from the velocity space discretization and cs is the speed of sound, both of which are determined by the choice of the lattice, which is D3Q19 in our implementation. Immiscibility of species α is introduced in the model following Shan and Chen [32, 33]. Only nearest neighbour interactions among the immiscible species are considered. These interactions are modelled as a self-consistently generated mean field body force Fα (x, t) ≡ −ψ α (x, t) gαα¯ ψ α¯ (x′ , t)(x′ − x) , (3) α ¯

x′

where ψ α (x, t) is the so-called eﬀective mass, which can have a general form α for modelling various types of fluids (we use ψ α = (1 − e−n ) [32]), and gαα ¯ is a force coupling constant whose magnitude controls the strength of the interaction between components α, α ¯ and is set positive to mimic repulsion. The dynamical eﬀect of the force is realized in the BGK collision operator by adding to the velocity u in the equilibrium distribution of (2) an increment δuα =

τ α Fα . nα

(4)

As described above, an amphiphile usually possesses two diﬀerent fragments, each having an aﬃnity for one of the two immiscible components. The addition of an amphiphile is implemented as in [5]. An average dipole vector

460

J. Harting, G. Giupponi

d(x, t) is introduced at each site x to represent the orientation of any amphiphile present there. The direction of this dipole vector is allowed to vary continuously and no information is specified for each velocity ci , for reasons of computational eﬃciency and simplicity. Full details of the model can be found in [5] and [28]. In order to inspect the rheological behaviour of multi-phase fluids, we have implemented Lees-Edwards boundary conditions, which reduce finite size effects if compared to moving solid walls [23]. This computationally convenient method imposes new positions and velocities on particles leaving the simulation box in the direction perpendicular to the imposed shear strain while leaving the other coordinates unchanged. Choosing z as the direction of shear and x as the direction of the velocity gradient, we have ⎧ ⎧ ⎨ u z + U , x > Nx ⎨ (z + Δz ) mod Nz , x > Nx , 0 ≤ x ≤ Nx , (5) , 0 ≤ x ≤ Nx u′z ≡ uz z ′ ≡ z mod Nz ⎩ ⎩ uz − U , x < 0 (z − Δz ) mod Nz , x < 0 where Δz ≡ U Δt, U is the shear velocity, uz is the z−component of u and Nx(z) is the system length in the x(z) direction. We also use an interpolation scheme suggested by Wagner and Pagonabarraga [36] as Δz is not generally a multiple of the lattice site. Consistent with the hypothesis of the LB model, we set the maximum shear velocity to U = 0.1 lattice units. For oscillatory shear, we set U (t) = U cos(ωt) , (6)

where ω/2π is the frequency of oscillation. To analyze the behaviour of the various simulations, we define the time dependent lateral domain size L(t) along direction i = x, y, z as

where

2π , Li (t) ≡ : 2 ki (t)

2 2 k S(k, t) ki (t) ≡ k i k S(k, t)

(7)

(8)

is the second order moment of the three-dimensional structure function S(k, t) ≡

1 ′ 2 |φ (t)| V k

(9)

with respect to the Cartesian component i, denotes the average in Fourier space, weighted by S(k, t) and V is the number of nodes of the lattice, φ′k (t) the Fourier transform of the fluctuations of the order parameter φ′ ≡ φ − φ, and ki is the ith component of the wave vector. A projection of the structure function allows us to compare simulation data to scattering patterns obtained in experiments. We obtain those projections by summing up S(k, t) in one of the Cartesian directions. For example, for the projection in the z-direction this leads to Sz (kx , ky , t) = kz S(k, t).

Simulations of Sheared Microemulsions and Binary Fluids

461

We use LB3D [17], a highly scalable parallel LB code, to implement the model. LB3D is written in Fortran 90 and designed to run on distributedmemory parallel computers, using MPI for communication. In each simulation, the fluid is discretized onto a cubic lattice, each lattice point containing information about the fluid in the corresponding region of space. Each lattice site requires about a kilobyte of memory per lattice site so that, for example, a simulation on a 1283 lattice would require around 2.2GB memory. The code runs at over 6 · 104 lattice site updates per second per CPU on a recent machine, and has been observed to have roughly linear scaling up to order 3 · 103 compute nodes. Larger simulations have not been possible so far due to the lack of access to a machine with a higer processor count. The largest simulation we performed used a 15363 lattice and was performed on the new AMD Opteron based cluster in Karlsruhe. There, it was not possible to use a larger lattice since the amount of memory per CPU is limited to 4GB and only 1024 processes are allowed within a single compute job. On the NEC SX8 in Stuttgart, typical system sizes were of the order of 128 × 128 × 512 lattice sites. The output from a simulation usually takes the form of a single floatingpoint number for each lattice site, representing, for example, the density of a particular fluid component at that site. Therefore, a density field snapshot from a 1283 system would produce output files of around 8MB. Writing data to disk is one of the bottlenecks in large scale simulations. If one simulates a 10243 system, each data file is 4GB in size. The situation gets even more critical when it comes to the files needed to restart a simulation. Then, the state of the full simulation lattice has to be written to disk requiring 0.5TB of disk space. LB3D is able to benefit from the parallel filesystems available on many large machines today, by using the MPI-IO based parallel HDF5 data format [19]. Our code is very robust regarding diﬀerent platforms or cluster interconnects: even with moderate inter-node bandwidths it achieves almost linear scaling for large processor counts with the only limitation being the available memory per node. The platforms our code has been successfully used on include various supercomputers like the NEC SX8, IBM pSeries, SGI Altix and Origin, Cray T3E, Compaq Alpha clusters, as well as low cost 32- and 64-bit Linux clusters. During the last year, a substantial eﬀort has been invested to improve the performance of LB3D and to optimize it for the simulation of binary fluid mixtures as well as flow in porous media. Further, we improved the performance on the SX8 in Stuttgart substantially by rearranging parts of the code and trying to increase the length of the loops. These changes were proposed by the HLRS support staﬀ. However, while the code scales very well with the number of processors used, the single CPU performance is still below what one could expect from a lattice Boltzmann implementation on a vector machine. The vector operation ratio is about 93%, but due to the inherent structure of our multiphase implementation, the average loop length is only between 20 and 30. Thus, the performance of our code stays below 1GFlop/s. For this reason, we are currently performing most of our simulations on the recently opened Opteron cluster XC2 in Karlsruhe.

462

J. Harting, G. Giupponi

Our code performs extremely well there and shows almost linear scaling to up to 1024 CPUs.

3 Results In non-sheared studies of spinodal decomposition it has been shown that large lattices are needed to overcome finite size eﬀects. We therefore choose 2563 for all non-sheared simulations to limit the influence of finite size eﬀects even further. For high shear rates, systems also have to be very extended in the direction of the applied shear because, if the system is too small, the domains interconnect across the z = 0 and z = Nz boundaries to form continuous lamellae in the direction of shear [17, 18]. Such artefacts need to be eliminated from our simulations. In this case, a good compromise to limit finite size eﬀects and to keep the computational expense as low as possible is a lattice size of 128x128x512 and this is used here. Mass and relaxation times are always set to unity, i.e. τ α = 1.0, mα = 1.0. We call the two immiscible fluids “red” and “blue” and set their initial densities to identical values, ρr = ρb . The initial average surfactant density ρs is varied between 0.0 and 0.7. The lattice is than randomly populated with constant initial total fluid densities ρtot = ρr + ρb + ρs = 1.6. This is in contrast to previous studies where only ρr + ρb was kept constant [14]. The coupling constant between “red” and “blue” species is set to gbr = 0.08, the coupling between an immiscible fluid and surfactant to gbs = −0.006 and the constant describing the strength of the surfactant-surfactant interaction is kept at gss = −0.003. All units in this paper are given in lattice units if not stated otherwise. Spinodal decomposition of a binary immiscible fluid mixture has been studied extensively in the past [15]. Domain size scaling was reported with a transition from diﬀusive behaviour to hydrodynamic viscous growth and the authors find very good agreement with the dynamical scaling hypothesis, recovering the expected universal behaviour of the structure function. Adding surfactant to a binary immiscible fluid mixture causes the surfactant molecules to self-assemble at the interface between the two species. The phase separation process slows down and for suﬃciently high surfactant concentrations, domain growth is arrested completely leading to a stable microemulsion. To depict the influence of the surfactant density on the phase separation process, Fig. 1 shows three volume rendered 2563 systems at surfactant densities 0.0 (left), 0.15 (center), and 0.3 (right). As in all figures throughout the paper, for better visibility only one of the immiscible fluid species is shown. Diﬀerent colours denote the interface and areas of high density of the rendered fluid. The surfactant particles are aligned at the interfaces and the second immiscible constituent fills the void space. After 30000 timesteps the phases have separated to a large extent when no surfactant is present (left). Running the simulation for even longer would result in two perfectly separated phases, each of them contained in a single domain only. If one adds some surfactant

Simulations of Sheared Microemulsions and Binary Fluids

x

463

y z

Fig. 1. Volume rendered ﬂuid densities of 2563 systems at t = 30000 for surfactant densities ρs = 0.00, 0.15, 0.30 (from left to right). Only one immiscible ﬂuid species is shown. Different colours denote the interface and areas of high density of the visualized ﬂuid. The phases have almost separated in the surfactantless case (left). Adding some surfactant (ρs = 0.15, center ) causes the domains to grow more slowly. For suﬃciently high amphiphile concentrations (ρs = 0.30, right) the growth process arrests and a stable bicontinuous microemulsion forms (see [16])

(ρs = 0.15, center), the domains grow more slowly, visualized by the smaller structures in the volume rendered image. For suﬃciently high amphiphile concentrations (ρs = 0.30, right) the growth process arrests leading to a stable bicontinuous microemulsion with small individual domains formed by the two immiscible fluids. The projected structure function Sz (kx , ky , t) (“scattering pattern”) is given in Fig. 2 for surfactant densities ρs = 0.00 (a) and 0.30 (b) at timestep t = 10000. As can be clearly seen in Fig. 2a), a strong peak occurs for small values of kx , ky depicting the occurrence of length scales which are of the order of the system size. For ρs = 0.30, however, the peaks are by a factor of 100 smaller and shifted to larger values of kx , ky . We find a volcano-like scattering pattern indicating the dominance of small length scales. Measurements of the lateral domain size L(t) allow us to fit our data with the corresponding growth laws. We find that for ρs smaller than 0.15 L(t) is best fit by a function proportional to tα . For ρs being 0.15 or 0.20, a logarithmic behaviour

a)

b)

Fig. 2. Projected structure function Sz (kx , ky , t) for (a) ρs = 0.00 and (b) 0.30 at timestep t = 10000 (see [16])

464

J. Harting, G. Giupponi

proportional to (ln t)θ is observed. Increasing ρs further results in L(t) being best described by a stretched exponential. These results correspond well with the findings in [14]. The dependence of the final domain size Lmax (ρs ) on the amount of surfactant is depicted in Fig. 3(a). It can be observed that the maximum domain size decreases linearly from 20.9 for ρs = 0.25 with increasing ρs until a threshold value is reached at ρs = 0.5, where Lmax (t) = 7.7. Then, Lmax (ρs ) decreases much more slowly and stays almost constant. The slope of the linear regime corresponds to -52.8. The behaviour of Lmax (ρs ) and tarrest (ρs ) is consistent with previous lattice gas [10, 24] and lattice Boltzmann studies [14]. The eﬀect of an existing final domain size can be explained as follows: adding surfactant to a binary fluid mixture causes the amphiphiles to minimize the free energy in the system by assembling at the interface between the two immiscible fluid species. An increase of surfactant concentration causes the interfacial area to be maximised in order to accommodate as much surfactant as possible. The increasing interfacial area causes the individual domains to become smaller and Lmax (ρs ) decreases. If the surfactant concentration becomes very high (ρs > 0.5 in our case), Lmax (ρs ) saturates due to the maximum possible interfacial area being reached and all available area being covered with surfactant molecules. More amphiphiles accumulating at the interface would lead to very steep and energetically unfavourable gradients of surfactant density in the system. Therefore, further amphiphiles have to reside within the bulk fluid phases forming micellar structures. In Fig. 3(b), the number of simulation timesteps tarrest (ρs ) needed to reach the final domain size is plotted. Since the time it takes for the system to relax to its equilibrium state directly depends on the final domain size, it is consistent with the data presented in Fig. 3(a) that a linear dependence of tarrest (ρs ) on the surfactant concentration can be observed. While for ρs = 0.25 7000 timesteps are needed to reach the maximum possible domain size, for ρs = 0.5 500 timesteps are suﬃcient. For ρs > 0.5, tarrest (ρs ) decreases much more slowly than for ρs < 0.5. The slope of tarrest (ρs ) in the linear regime is given by −26000. a)

25

b)

8000 7000

20

15

5000

s

tarrest(ρ )

s

Lmax(ρ )

6000

10

4000 3000 2000

5 1000 0 0

0.2

0.4

ρ

s

0.6

0.8

0 0

0.2

0.4

ρ

s

0.6

0.8

Fig. 3. (a) Maximum domain size Lmax (ρs ) and (b) time of arrest tarrest (ρs ) for various surfactant densities ρs (see [16])

Simulations of Sheared Microemulsions and Binary Fluids

465

Shearing binary immiscible fluid mixtures can change the evolution of domains and phase separation processes profoundly [2, 27, 39]. Most notably is the formation of a lamellar phase, i.e. elongated domains or lamellae form and align along the flow direction. Due to the anisotropy of the system, the time dependent domain size L(t) behaves diﬀerently for discerned coordinate axes in this case. Furthermore, modified growth exponents are expected due to the anisotropic eﬀects. Also, adding amphiphiles to a binary immiscible fluid under shear stabilizes the interface between the immiscible fluid species and hinders the domain growth. We study ternary systems under constant shear with the shear rate set to γ˙ = 1.56 × 10−3 and γ˙ = 3.12 × 10−3 , while ρs is varied between 0.0 and 0.4. Individual fluid domains occurring in the system are being elongated due to the shear and try to align with the shear gradient. For increasing ρs , the average domain size reduces due to the influence of the amphiphiles, thus causing the individual domains to become smaller. For high surfactant concentrations (ρs = 0.4) all three directions behave very similarly: domain growth comes to an end after short simulation times and the final domain size is between 10 and 15 lattice units in all directions, signaling the appearance of a stable microemulsion. Regular peaks occur in Lz (t) with less pronounced peaks in between them. These peaks can be explained as follows: for the stretching of domains, a certain amount of work against surface tension is needed. On macroscopic scales, the stress tensor does not vanish due to the viscoelastic response of the system. On the microscale, however, a breakup and recombination of domains can be observed [29]. These domains grow by diﬀusion and eventually join each other to form larger structures. If the internal stress becomes too large due to the shear induced deformation, they break up and start to form again. Assuming a large system with many independent domains growing and breaking incoherently, the only observable eﬀect might be a slowing down of the domain growth. In contrast, if the growth and breakup occur coherently as they do in our simulations, a periodicity in the measured time dependent domain size can be observed [7]. We have found that the frequency of domain breakup is independent of the surfactant concentration, while the height of the peaks decreases with increasing ρs . Figure 4 shows volume rendered examples of a simulated system with surfactant density ρs = 0.2 and a constant shear rate of γ˙ = 1.56 × 10−3 . The four snapshots are taken a diﬀerent times t = 1000 (upper left), t = 4000 (upper right), t = 6000 (lower left), and t = 10000 (lower right). It can be observed that at early stages of the simulation, the shape of individual domains does not show distinct features, while at t = 4000, slightly elongated domains start to occur which begin to align with the shear gradient. At t = 6000, these features are substantially more dominant and at late simulation times (t = 10000) the system is filled with elongated and thin lamellae consisting of one of the immiscible fluid species and which are almost parallel to the shear plane.

466

J. Harting, G. Giupponi

x

y z

Fig. 4. Volume rendered ﬂuid densities for surfactant density ρs = 0.2, a constant shear rate γ˙ = 1.56 × 10−3 and variable number of time steps t = 1000 (upper left), t = 4000 (upper right), t = 6000 (lower left), and t = 10000 (lower right) (see [16])

We have shown in the previous paragraphs that the dynamical scaling hypothesis does not hold for sheared ternary systems in three dimensions since we indeed find three individual length scales pointing out the transition from the sponge to the lamellar phase: while in the flow direction (z), L(t) is determined by the resultant length of the occurring lamellae, in the direction between the shear planes (x), the domains grow steadily and exhibit power law behaviour up to a maximum that depends on the surfactant concentration. In the y direction, domain growth is not hindered by shear. In fact, Ly (t) grows slightly faster than in the non-sheared case. Increasing the surfactant concentration has a strong impact on domain growth: starting at ρs = 0.3, Ly (t) and Lz (t) recover the behaviour of the case without shear, i.e. the length scales saturate. In the x direction, however, growth continues to up to Lx (t) = 26. This can be explained as follows: with increasing surfactant concentration, the final domain sizes become smaller, reducing the influence of the shear forces in the y and z directions. In the direction between the shear planes, however, an increase of Lx (t) can still be observed because the domains are still being elongated due to shear and try to align with the velocity profile. Thus, they are tilted and their size appears to be smaller than it actually is in z direction and larger in the x direction. Our findings are in agreement with Ginzburg-Landau and Langevin calculations [7, 8] as well as two-dimensional lattice-Boltzmann simulations of binary immiscible fluid mixtures as presented in [13, 34, 38]. However, to the best of our knowledge, there are no detailed theoretical studies of the dependence of domain growth properties on the surfactant concentration. The only known work utilizes a Ginzburg-Landau free-energy approach to study sheared microemulsions, but does not vary the amount of surfactant. In addition, the authors only cover two-dimensional systems and are thus unable to describe the behaviour of Ly (t) [9]. In the case of oscillatory shear, the morphology and the domain growth are altered significantly, although the average deformation is zero after each

Simulations of Sheared Microemulsions and Binary Fluids

467

period of shear. For example, it has been found experimentally for binary fluid mixtures that for very low oscillation frequencies domain growth can be interrupted [22], or domains can grow on much longer time scales than given by the oscillation frequency [26]. We apply two diﬀerent oscillation frequencies ω = 0.001 and ω = 0.01, where a single oscillation takes 6283 timesteps in the slow case and 628.3 timesteps in the fast case. Let us first consider the case with a lower oscillation frequency and lower shear rate, i.e. ω = 0.001 and γ˙ = 1.56 × 10−3 . In the case of oscillatory sheared systems, the individual fluid domains try to align with the velocity gradient as in the steadily sheared case. However, here domains are never able to reach a steady state and instead have to follow the oscillation of the planes. This is depicted in Fig. 5 which shows two typical examples from a simulation with ρs = 0.2. On the left hand side, a volume rendered snapshot is given at t = 2500. Here, the oscillating shear planes have just passed their reversal point. Close to the shear planes, the domains are aligned vertically because they have to be turned around in order to follow the changing direction of movement of the shear planes. In the bulk of the system, however, no preferred direction can be observed since the velocity gradient does not interpenetrate the whole system. At t = 10000, the shear planes are in a position just before their reversal point. Thus, the fluid mixture was accelerated for more than 2000 timesteps and the domains close to the shear boundary are well aligned in the direction of the flow. In the bulk, again no preferred direction can be observed. We observe the formation of tubular structures which are elongated in the y direction and show similar length scales in x and z direction. If the frequency is set to the higher value, the fluid is not able to follow the movement of the walls anymore and the influence of the shear on the growth behaviour becomes less pronounced, with the domains constantly growing as long as the amount of surfactant present allows it. The growth rates are comparable to the non-sheared case here and show identical growth laws as in the non-sheared case. The only diﬀerence is that the exponents are found to be smaller while Ly (t) grows slightly faster than Lx (t) and Lz (t) depicting the occurrence of tubular structures in the system. The z direction is the only component of the time dependent lateral domain size that diﬀers from the unsheared case, because strong oscillations start to appear due to the distortions caused by the moving boundaries.

x

y z

Fig. 5. Volume rendered ﬂuid densities for surfactant density ρs = 0.2 at t = 2500 (left) and t = 10000 (right). The shear rate is γ˙ = 1.56 × 10−3 and ω = 0.001 (see [16])

468

J. Harting, G. Giupponi

In the case of oscillatory shear we have shown the occurrence of tubular structures due to shear imposed anisotropic domain growth, the slowing down of the domain growth rate depending on the oscillation frequency, as well as that a microemulsion with high surfactant concentration stays unaﬀected by external shear forces.

4 Conclusion In this report we have presented our results from ternary amphiphilic latticeBoltzmann simulations performed on the NEX SX-8 at the HLRS. We have shown that our simulation code performs well on this machine and that we are able to investigate spinodal decomposition with and without shear. In addition, we have studied the influence of the surfactant concentration on the time dependent lateral domain size. Our results are already published in [16] and can be summarized as follows: we reproduced the well-known power law growth of domains in the case of binary immiscible fluids (spinodal decomposition) which crosses over to a logarithmic law and to a stretched exponential if one increases the surfactant concentration even further. For suﬃciently high surfactant concentrations, domain growth can come to an end and the system corresponds to a stable bicontinuous microemulsion. For sheared systems, we have found domain breakup phenomena and a transition from a sponge phase to a lamellar phase depending on the shear rate as well as the amphiphile concentration. Under oscillatory shear, tubular structures occur. For very fast oscillations (ω = 0.01), the system is not able to follow the external shear at all. Thus, it behaves similar to a non-sheared one. Acknowledgements We are grateful for the support of the HPC-Europa programme, funded under the European Commission’s Research Infrastructures activity, contract number RII3-CT-2003-506079 and the H¨ochstleistungsrechenzentrum Stuttgart for providing access to their NEC SX8. We would especially like to thank H. Berger, R. Keller, and P. Lammers for their technical support and P.V. Coveney for fruitful discussions.

References 1. R. Benzi, S. Succi, and M. Vergassola. The lattice Boltzmann equation: theory and applications. Phys. Rep., 222(3):145 – 197, 1992. 2. J. Berghausen, J. Zipfel, O. Diat, T. Narayanan, and W. Richtering. Lamellar phases under shear: Variation of the layer orientation across the couette gap. Phys. Chem. Chem. Phys., 2:3623, 2000.

Simulations of Sheared Microemulsions and Binary Fluids

469

3. P. L. Bhatnagar, E. P. Gross, and M. Krook. Model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems. Phys. Rev., 94(3):511–525, 1954. 4. A. J. Bray. Theory of phase-ordering kinetics. Adv. Phys., 43(3):357–459, 1994. 5. H. Chen, B. M. Boghosian, P. V. Coveney, and M. Nekovee. A ternary lattice Boltzmann model for amphiphilic ﬂuids. Proc. R. Soc. Lond. A, 456:2043–2047, 2000. 6. S. Chen, H. Chen, D. Mart´ınez, and W. Matthaeus. Lattice Boltzmann model for simulation of magnetohydrodynamics. Phys. Rev. Lett., 67(27):3776–3779, 1991. 7. F. Corberi, G. Gonnella, and A. Lamura. Phase separation of binary mixtures in shear ﬂow: a numerical study. Phys. Rev. E, 62(6):8064–8070, 2000. 8. F. Corberi, G. Gonnella, and A. Lamura. Ordering of the lamellar phase under shear ﬂow. Phys. Rev. E, 66:016114, 2002. 9. F. Corberi, G. Gonnella, and D. Suppa. Steady state of microemulsions in shear ﬂow. Phys. Rev. E, 63:040501(R), 2001. 10. A. N. Emerton, P. V. Coveney, and B. M. Boghosian. Lattice-gas simulations of domain growth, saturation and self-assembly in immiscible ﬂuids and microemulsions. Phys. Rev. E, 56(1):1286–1306, 1997. 11. P. Espa˜ nol and P. Warren. Statistical mechanics of dissipative particle dynamics. Europhys. Lett., 30(4):191–196, 1995. 12. T. E. Faber. Fluid Dynamics for Physicists. Cambridge University Press, 1995. 13. G. Gonnella, E. Orlandini, and J. M. Yeomans. Spinodal decomposition to a lamellar phase : Eﬀects of hydrodynamic ﬂow. Phys. Rev. Lett., 78(9):1695– 1698, 1997. 14. N. Gonz´ alez-Segredo and P. V. Coveney. Coarsening dynamics of ternary amphiphilic ﬂuids and the self-assembly of the gyroid and sponge mesophases: lattice-Boltzmann simulations. Phys. Rev. E, 69:061501, 2004. 15. N. Gonz´ alez-Segredo, M. Nekovee, and P. V. Coveney. Three-dimensional lattice-Boltzmann simulations of critical spinodal decomposition in binary immiscible ﬂuids. Phys. Rev. E, 67:046304, 2003. 16. J. Harting, G. Giupponi, and P. V. Coveney. Structural transitions and arrest of domain growth in sheared binary immiscible ﬂuids and microemulsions. Phys. Rev. E, in press, 2007. 17. J. Harting, M. Harvey, J. Chin, M. Venturoli, and P. V. Coveney. Large-scale lattice boltzmann simulations of complex ﬂuids: advances through the advent of computational grids. Phil. Trans. R. Soc. Lond. A, 363:1895–1915, 2005. 18. J. Harting, M. Venturoli, and P. V. Coveney. Large-scale grid-enabled latticeboltzmann simulations of complex ﬂuid ﬂow in porous media and under shear. Phil. Trans. R. Soc. Lond. A, 362:1703–1722, 2004. 19. 2003. HDF5 – a general purpose library and ﬁle format for storing scientiﬁc data, http://hdf.ncsa.uiuc.edu/HDF5. 20. R. A. L. Jones. Soft Condensed Matter. Oxford University Press, 2003. 21. T. Kawakatsu, K. Kawasaki, M. Furusaka, H. Obayashi, and T. Kanaya. Late stage dynamics of phase separation processes of binary mixtures containing surfactants. J. Comp. Phys., 99:8200, 1993. 22. A. H. Krall, J. V. Sengers, and K. Hamano. Experimental studies of the rheology of a simple liquid mixture during phase separation. Phys. Rev. E, 48:357–376, 1993.

470

J. Harting, G. Giupponi

23. A. Lees and S. Edwards. The computer study of transport processes under extreme conditions. J. Phys. C., 5(15):1921–1928, 1972. 24. P. J. Love, P. V. Coveney, and B. M. Boghosian. Threer-dimensional latticegas simulations of domain growth and self-assembly in binary immiscible and ternarfy amphiphilic ﬂuids. Phys. Rev. E, 64:021503, 2001. 25. A. Malevanets and R. Kapral. Continuous-velocity lattice-gas model for ﬂuid ﬂow. Europhys. Lett., 44(5):552–558, 1998. 26. K. Matsuzaka, H. Jinnai, T. Koga, and T. Hashimoto. Eﬀect of oscillatory shear deformation on demixing processes of polymer blends. Macromolecules, 30:1146–1152, 1997. 27. C. Meyer, S. Asnacios, C. Bourgaux, and M. Kleman. Rheology of lyotropic and thermotropic lamellar phases. Rheol. Acta, 39:223, 2000. 28. M. Nekovee, P. V. Coveney, H. Chen, and B. M. Boghosian. Lattice-Boltzmann model for interacting amphiphilic ﬂuids. Phys. Rev. E, 62:8282, 2000. 29. T. Ohta, H. Nozaki, and M. Doi. Computer simulations of domain growth under steady shear ﬂow. J. Comp. Phys., 93:2664–2675, 1990. 30. F. Qiu, H. Zhang, and Y. Yang. Oscillatory shear induced anisotropic domain growth and related rheological properties of binary mixtures. J. Comp. Phys., 109:1575–1583, 1998. 31. J.-P. Rivet and J. P. Boon. Lattice Gas Hydrodynamics. Cambridge University Press, 2001. 32. X. Shan and H. Chen. Lattice Boltzmann model for simulating ﬂows with multiple phases and components. Phys. Rev. E, 47(3):1815–1819, 1993. 33. X. Shan and H. Chen. Simulation of nonideal gases and liquid-gas phase transitions by the lattice Boltzmann equation. Phys. Rev. E, 49(4):2941–2948, 1994. 34. P. Stansell, K. Stratford, J. C. Desplat, R. Adhikari, and M. E. Cates. Nonequilibrium steady states in sheared binary ﬂuids. Phys. Rev. Lett., 96:085701, 2006. 35. S. Succi. The Lattice Boltzmann Equation for Fluid Dynamics and Beyond. Oxford University Press, 2001. 36. A. Wagner and I. Pagonabarraga. Lees-edwards boundary conditions for lattice Boltzmann. J. Stat. Phys., 107:521, 2002. 37. A. Xu, G. Gonnella, and A. Lamura. Phase separating binary ﬂuids under oscillatory shear. Phys. Rev. E, 67:056105, 2003. 38. A. Xu, G. Gonnella, and A. Lamura. Phase separation of incompressible binary ﬂuids with lattice Boltzmann methods. Physica A, 331:10–22, 2004. 39. J. Zipfel, J. Berghausen, G. Schmidt, P. Lindner, M. Tsianou, P. Alexandridis, and W. Richtering. Shear induced structures in lamellar phases of amphiphilic block copolymers. Phys. Chem. Chem. Phys., 1:3905, 1999.

Numerical Investigation of Hypersonic Intake Flows Martin Krause1 and Josef Ballmann2 1

2

Lehr- und Forschungsgebiet f¨ ur Mechanik (Mechanics Laboratory), RWTH Aachen University, Templergraben 64, 52062 Aachen, Germany [email protected] [email protected]

Summary. A numerical and experimental analysis of scramjet intake ﬂows has been initiated at RWTH Aachen University as part of the Research Training Group GRK 1095: “Aero-Thermodynamic Design of a Scramjet Engine for a Future Space Transportation System”. This report presents an overview of the ongoing work on the numerical simulations of air intake ﬂow using two diﬀerent, well validated Reynolds averaged Navier Stokes solvers. Several geometry concepts e.g. 2D intake, 3D intake using a single or double ramp conﬁguration were investigated. One example for the so-called 2D intake can be seen in Fig. 1 and for a 3D intake in Fig. 2. To analyze the eﬀects these diﬀerent geometries have on the ﬂow, especially on the separation bubble in the isolator inlet as well as on transition and eﬃciency, several numerical simulations (2D and 3D) were performed using a variety of turbulence models. Mostly the Spalart–Allmaras – one equation model and the so called SSG–Reynolds stress model by Speziale, Sakar and Gatski were used. The data obtained will be compared with experimental results. These experiments started in March 2007. It has to be said that not all results presented here were achieved using the NEC computing cluster. For comparison several calculations were conducted on the IBM Jump system of the J¨ ulich Research Centre and on the SUN cluster of RWTH Aachen University. At the end of this report we give comments on the computational performance.

1 Introduction The engine inlet of an air breathing hypersonic propulsion system mostly consists of several exterior compression ramps followed by a subsequent interior isolator/diﬀusor assembly (see Fig. 1, Fig. 2). Oblique shock waves without a final normal shock are performing the compression of the incoming flow from hypersonic to supersonic speed. Concerning flight conditions, two main problems of such a hypersonic intake are conspicuous. The first diﬃculty is, that the interaction of shock waves with thick hypersonic boundary layers causes large separation zones that are responsible for a loss of mass flow and

472

M. Krause, J. Ballmann

Fig. 1. Hypersonic 2D inlet model without sidewall

Fig. 2. Hypersonic 3D inlet model

several other eﬀects, like e.g. unsteady shock movement. Thereby the engine performance decreases. The separation zones are also responsible for high heat flux into the structure of the engine. The high total enthalpy of the flow yields severe aerodynamic heating, further enhanced by turbulent heat flux. In the current report the results for several test cases are presented. On the one hand the influence of geometry modifications on the flow within the separation regions will be discussed. On the other hand the influence of the used numerical model, which means the turbulence model, as well as the diﬀerence between a 2D and 3D simulation will be considered. Fig. 1 shows a configuration with sharp leading edges. Introducing leading edges with a defined radius will change the flow in several aspects, e.g. the positions of the front and the cowl shocks. There will be a curved bow shock on both edges, that generates large entropy layers. The boundary layer along the surface grows inside this entropy layer causing an increase in aerodynamic heating [1]. Furthermore, the shock detachment influences the captured mass flow and the flow conditions in the isolator intake. Therefore, the magnitude of leading edge radii will also be a matter of interest. Remarkable diﬀerences occur by using a 3D geometry with converging sidewalls. To get the same compression ratio the intake can be much shorter, because in a 3D intake the flow is compressed by the ramps and the sidewalls, instead of only ramp compression in a 2D intake. Thus at first glance the boundary layer is expected to be thinner before entering the isolator, so that more mass flow enters the combustion chamber, because of the lower displacement thickness. But the sidewalls create vortices, that lead to extra shocks and so causing additional Shock/Boundary Layer and shock/shock interactions, that enhance the separation regions and the thickness of the boundary layers. In real flight with hypersonic speed the intake walls will be strongly heated up to a certain thermodynamic equilibrium temperature. Related to this, ground testing should include windtunnel models with heated surface. Therefore, the first experimental configuration is a heated model to investigate the influence of the wall temperature change as well as diﬀerent nose radii to investigate their influence on the separation of the boundary layer. These experiments are done at the Shock Wave Laboratory (SWL) of RWTH Aachen University. where suction can be installed in

Numerical Investigation of Hypersonic Intake Flows

473

and behind the flow expansion region, too. The second configuration (investigated at DLR Cologne) is a non-heated model, where the influence of suction will be investigated separately. It is also possible to change the isolator geometry and the position of the cowl. These experiments are done within the GRK 1095 (Research Training Group “Aero-Thermodynamic Design of a Scramjet Engine for Future Space Transportation Systems”).

2 Physical Model 2.1 Conservation Equations The governing equations for high-speed turbulent flow are the unsteady Reynolds-averaged Navier-Stokes equations for compressible fluid flow in integral form ; c

∂ F − Fd n dS = 0 (1) U dV + ∂t V ∂V

where

˜ , ρ¯e˜tot ]T U = [ ρ¯ , ρ¯v

(2)

is the array of the mean values of the conserved quantities: density of mass, momentum density, and total energy density. The tilda and the bar over the variables denote the mean value of Reynolds-averaged and Favre-averaged variables, respectively. The quantity V denotes an arbitrary control volume with the closed surface ∂V and the outer normal n. The fluxes are splitted into the inviscid part ⎛ ⎞ ˜ ρ¯v ˜ ◦v ˜ + p¯ 1 ⎠ Fc = ⎝ ρ¯v ˜ (¯ v ρe˜tot + p¯) and the diﬀusive part ⎛

⎞ 0 ⎠, ¯ − ρv′′ ◦v′′ σ Fd = ⎝ 1 ′′ ′′ ′′ ′′ ′′ ′′ ′′ ′′ ˜σ ¯ + v σ − q − cp ρv T − 2 ρv v ◦v − v ˜ ρv ◦v v

where 1 is the unit tensor and ◦ denotes the dyadic product3 . The air is considered to be a perfect gas with constant ratio of specific heats, γ = 1.4, and a specific gas constant of R = 287 J/(kgK). Correspondingly the expression for the specific total energy reads: 1 ˜v ˜ +k . e˜tot = cv T¯ + v 2 3

(3)

Scalar Products of dyadics formed by two vectors a and b with a vector c are deﬁned as usual, i.e., a ◦ b c = a(bc), c a ◦ b = (ca)b.

474

M. Krause, J. Ballmann

The last term represents the turbulent kinetic energy k :=

1 ρv′′ v′′ . 2 ρ¯

(4)

For isotropic Newtonian fluids, the mean molecular shear stress tensor is a linear, homogeneous and isotropic function of the strain rate

¯ − 2μ ¯ 1, ¯ = 2¯ σ μS ¯ tr S 3

(5)

if bulk viscosity is neglected. The mean strain rate tensor is ¯ := 1 grad(¯ S v) + (grad(¯ v))T , (6) 2 and the molecular viscosity μ ¯=μ ¯(T¯) obeys Sutherland’s law. Similarly, the molecular heat flux is considered a linear, homogeneous, isotropic function of the temperature gradient ¯=− q

¯ cp μ grad(T¯ ), Pr

(7)

with the Prandtl number P r = 0.72. 2.2 Turbulence Closure To close the above system of partial diﬀerential equations, the Boussinesq hypothesis is used where the remaining correlations are modeled as functions of the gradients of the mean conservative quantities and turbulent transport coeﬃcients. The Reynolds stress tensor thus becomes ¯ − 2 ρ¯k 1 ¯ − 1 tr(S)) −ρv′′ ◦v′′ = 2μt (S 3 3

,

with the eddy viscosity μt , and the turbulent heat flux is cp μt cp ρv′′ T ′′ = − grad(T¯), P rt

(8)

(9)

with the turbulent Prandtl number P rt = 0.89. Finally, for hypersonic flows the molecular diﬀusion and the turbulent transport are modeled as functions of the gradient of the turbulent kinetic energy v′′ σ −

1 ′′ ′′ ′′ μt ρv v ◦v = (μ + ) grad(k), 2 P rk

(10)

with the model constant P rk = 2. The turbulent kinetic energy and the eddy viscosity are then obtained from the turbulence model. In case of laminar flow, both variables are set to zero to regain the original transport equations. As mentioned above, the following turbulence models were used for the numerical simulations: SA, LEA, LLR, SST, k-ω, SSG. For description of the models used, we refer to [2], [3] and [4].

Numerical Investigation of Hypersonic Intake Flows

475

3 Numerical Methods The computations have been performed using the DLR FLOWer code, Versions 116.+ and the QUADFLOW code developed in SFB 401. Both solvers use finite-volume formulations. The well-established FLOWer code is an explicit structured solver operating on hexahedral multi-block grids and contains diﬀerent flux formulations including central discretization and all important techniques for convergence accelerations [5]. QUADFLOW is a new unstructured adaptive, implicit solver with surface based data structure. Different upwind flux formulations are implemented and acceleration techniques as well. Multi-scale analysis and B-Spline techniques form the basis for automatic grid adaptation. Both solvers are capable of doing hypersonic, turbulent flow computations because upwind discretizations and advanced compressible turbulence models are implemented [6]. 3.1 Spatial Discretization and Time Integration The finite-volume discretization that is applied to (1) in both solvers, ensures a consistent approximation to the conservation laws. Starting from a formulation in general curvilinear coordinates ξ, η, ζ, the computational domain is divided into non-overlapping hexahedra which are in FLOWer bounded by plane quadrilateral surface elements. The integral formulation (1) is then applied to each cell (i, j, k) separately. In QUADFLOW these elements are curved instead of plane in order to ensure that new surface grid points introduced during adaptation are positioned on the true wetted surface. Standard central discretization schemes are used for diﬀusive terms in both codes. For the convective terms upwind discretization was applied in both codes to achieve the present hypersonic flow results. The AUSM (Advection Upstream Splitting Method) scheme was used. Higher-order accuracy and consistency with the central diﬀerences used for the diﬀusive terms is achieved by MUSCL (Monotonic Upstream Scheme for Conservation Laws) Extrapolation, and TVD (Total Variation Diminishing) property of the scheme is ensured by a modified van Albada limiter function in FLOWer. In QUADFLOW the Venkatakrishnan limiter was used. FLOWer solves the system of ordinary diﬀerential equations by an explicit five-stage Runge-Kutta timestepping scheme of fourth order in combination with diﬀerent convergence acceleration techniques, like multigrid and local time stepping for asymptotically steady-state solutions [8]. Additionally, because for inviscid non heatconducting stationary flow the total enthalpy is a constant throughout the flow field, its numerical deviation is applied as forcing function to accelerate convergence. For turbulent flow, the time integration of the turbulence equations is decoupled from the mean equations and the turbulence equations are solved using a Diagonal Dominant Alternating Direction Implicit (DDADI) scheme.

476

M. Krause, J. Ballmann

QUADFLOW uses a fully implicit Euler method for time integration and a Newton-Krylov-Method for the linear equation system. Local time stepping can also be applied in QUADFLOW. Implicity increases the numerical stability of turbulent flow simulations which is especially important since the low Reynolds number damping terms as well as the high grid cell aspect ratios near the wall make the system of turbulent conservation equations stiﬀ. Due to the CFL condition for explicit schemes in FLOWer, the CFL number of the multi-stage Runge-Kutta scheme has an upper limit of 4. Implicit residual smoothing allows to increase the explicit stability limit by a factor of 2 to 3 [8]. Due to the complete implicit formulation with use of the Newton-Krylov-Method QUADFLOW does not have such restricted limits. 3.2 Boundary Conditions At the inflow, outflow and other farfield boundaries, a locally one-dimensional inviscid flow normal to the boundary is assumed. The governing equations are linearized based on characteristic theory and the incoming and outgoing number of characteristics are determined for both solvers. For incoming characteristics, the state variables are corrected by freestream values using the linearized compatibility equations. Else the variables are extrapolated from the interior [8]. However, a certain diﬀerence between QUADFLOW and FLOWer exists in the formulation of the characteristic boundary conditions. In QUADFLOW the characteristics are formulated in space and time, in contrast FLOWer only uses space dependent formulations. For turbulent flow, the turbulent freestream values are determined by specifying the freestream turbulence intensity Tu∞ : k∞ = 0.667 · Tu∞ v∞ and ω∞ = k∞ /(0.001 · μ). This is done in both flow solvers. For steady inviscid flow, it is suﬃcient to set ˜ vn ˜ = 0 at slip surfaces. In the viscous case, the no-slip condition is enforced at solid walls by setting all velocity components to zero. Additionally, the turbulent kinetic energy and the normal pressure gradient are set to zero. The specific dissipation rate is set proportional to the wall shear stress and the surface roughness. The energy boundary condition is directly applied through the diﬀusive wall flux: either by driving to zero the contribution of the diﬀusive flux for adiabatic walls or by prescribing the wall temperature or wall heat flux when calculating the energy flux through wall faces. At the symmetry plane of the half configuration, the conservation variables are mirrored onto the ghost cells to ensure symmetry.

4 Results At first we introduce a coordinate system for a better understanding of the intake geometries. This is shown in Fig. 3 where x points in freestream flow direction, z is defined along the leading edge of the ramp and y is defined

Numerical Investigation of Hypersonic Intake Flows

477

Fig. 3. Sketch of intake SCR02

as the vector product of both. So far, there are several results available from 2D and 3D computations for a 2D intake geometry like in Fig. 1, which were performed using diﬀerent turbulence models. It has to be mentioned, that the 2D calculations were done on the SUN cluster of RWTH Aachen University and the JUMP cluster of the NIC Forschungszentrum J¨ ulich. For eﬃciency and computation time reasons all 3D computations were performed on the NEC SX8. At first several calculations for the 2D intake SCR02 (Fig. 3) of the DLR Cologne have been done using nearly all common turbulence models like Spalart–Allmaras (SA), Linear Explicit Algebraic Stress model (LEA), Local Linear Realizable (LLR), Shear Stress Transport (SST), Wilcox k − ω and the Speziale, Sakar and Gatski (SSG) Reynolds stress model to validate the used numerical methods. That was done to examine the influence of the applied model on the flow solution and to find out which model performs best for the flow problem investigated here. Simulations for two diﬀerent geometries of the SCR02 had been done. First one was without and second one with suction, that means a gap was introduced into the lower isolator wall, where the cowl shock is expected to hit the boundary layer, that flows into that gap, thus the separation vanishes in that region. To do the simulation, the undisturbed inflow conditions were assumed as follows for minus infinity: M∞ = 6.0, α = 0◦ , Rel = 10.543 · 106 [1/m] and T∞ = 59K, p∞ = 747P a. It was found that for this hypersonic flow problem, none of the applied two equation turbulence models yielded good results [9]. They overestimated the boundary layer thickness and the separation within the isolator of the scramjet. That led to a blockade of the inlet and finally ended in an unstable computation. While comparing the measurements with the numerical simulations it was found that only the Spalart–Allmaras model and the SSG Reynolds stress model yielded appropriate results. For 3D computations the SA model also overestimated the separation within the isolator such that the intake was blocked. Finally it can be stated for the tested turbulence models that only the SSG model provided adequate results (Fig. 4 and 5). In Fig. 4 the Mach contours without suction and in Fig. 5 with suction for the 2D intake simulation of the SCR02 can be seen. In both computations the SSG turbulence model was applied. It has to be mentioned that this configuration has sharp leading edges. Having

478

M. Krause, J. Ballmann Mach

0.2 0.6 1 1.4 1.8 2.2 2.6 3 3.4 3.8 4.2 4.6 5 5.4 5.8 0.2

y [m]

Mach = 6

cowl shock

reattachment shock (reflected)

Pressure Probes

nd

2 ramp shock 0.1 1 st ramp shock

separation bubble

0 0

0.2

0.4

x [m]

0.6

0.8

Fig. 4. Mach contours for scramjet intake SCR02, 2D simulation without suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K,p∞ = 747 Pa ) Mach

0.2 0.6 1 1.4 1.8 2.2 2.6 3 3.4 3.8 4.2 4.6 5 5.4 5.8 0.2

Mach = 6

y [m]

cowl shock reflected cowl shock

Pressure Probes

2 nd ramp shock 0.1 1 st ramp shock

bleed implementation

0 0

0.2

0.4

x [m]

0.6

0.8

Fig. 5. Mach contours for scramjet intake SCR02, 2D simulation with suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K, p∞ = 747 Pa)

a look on the isolator inlet flow it can be seen, that the strong cowl shock interacts on the opposite wall with the hypersonic boundary layer and the expansion fan, which is generated by the convex ramp shoulder, yielding a large separation bubble that blocks half of the overall isolator height. The Figs. 6, 7, 8 and 9 show the pressure distributions along the solid walls of the cowl and upper isolator wall as well as along the ramp. Two numerical simulations are compared with the experimental results. The first one was performed using the QUADFLOW code with the SA model in the Ashford version, the second one using the FLOWer code with the SSG Reynolds stress model. Both computational results fit very well to the measurements done by the DLR Cologne. In Figs. 6 and 8 one can see a computational result for an adaptive solution. That means, in this case the multi-scale analysis of QUADFLOW was used to adapt the grid. That led to a better result in comparison to the computation with QUADFLOW using a structured grid especially in the separation region of the inlet, where the cowl shock hits the lower isolator wall, but the computational eﬀort was nearly 5 times higher than on the non-refined grid. That is the price to be paid for the high resolution of shocks and layers. The question if one should apply the adaptation mode in QUADFLOW has to be answered by the required flow resolution for the particular application. In the

Numerical Investigation of Hypersonic Intake Flows

Fig. 6. Pressure distribution on cowl of SCR02, 2D simulation without suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K, p∞ = 747 Pa)

Cowl 4 SA Model, QUADFLOW SSG Model, FLOWer Measurement (Haeberle, DLR Cologne) SA Model QUADFLOW adaptive

cp

3

479

2

1

0 0.3

0.4

0.5 x [m]

0.6

0.7

0.8

Fig. 7. Pressure distribution on cowl of SCR02, 2D simulation with suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K, p∞ = 747 Pa)

Cowl 4

SA Model, QUADFLOW SSG Model, FLOWer Measurement (Haeberle DLR Cologne)

cp

3

2

1

0 0.4

x [m]

0.6

0.8

Fig. 8. Pressure distribution on ramp of SCR02, 2D simulation without suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K, p∞ = 747 Pa)

Ramp 4

SA Model, QUADFLOW SSG Model, FLOWer Measurement (Haeberle, DLR Cologne) SA Model QUADFLOW adaptive

cp

3

2

1

0 0.3

0.4

0.5 x [m]

0.6

0.7

0.8

present one very thick hypersonic boundary layers occur in conjunction with several shock structures along with separation zones that lead to a refinement of the whole computational domain. The advantage of getting high resolution including sharp shocks is therefore paid by the higher computational eﬀort. Figure 10 shows a 3D simulation of a 2D intake with sidewalls like shown in Fig. 1. It is found that the sidewall has great influence on the flow field.

480

M. Krause, J. Ballmann Fig. 9. Pressure distribution on ramp of SCR02, 2D simulation with suction (Twall = 300K, M∞ = 6.0, Rel = 10.543 · 106 [1/m], T∞ = 59 K, p∞ = 747 Pa)

Ramp 4

SA Model, QUADFLOW SSG Model, FLOWer Measurement (Haeberle DLR Cologne)

cp

3

2

1

0 0.4

x [m]

0.6

0.8

Fig. 10. Mach contours for scramjet intake investigated at Shock Wave Laboratory of RWTH (Twall = 300K, M∞ = 7.46, Rel = 3.3 · 106 [1/m], T∞ = 258 K)

The sidewall shocks are bending the ramp shocks in upstream direction leading to an increased loss in mass flow. The separation within the isolator is larger than in the 2D simulation, because the boundary layer of the sidewall increases it. As expected, vortices are generated in the corners between sidewall and ramps. These vortices also lead to shock bending and very high heat loads on the surface. As already mentioned, within the GRK 1095 a new intake will be designed. Therefore, several concepts are investigated. Purely 2D compression as seen in Fig. 1 with or without sidewall and with a convex curve or convex corner ramp to redirect the flow into horizontal direction. Also

Numerical Investigation of Hypersonic Intake Flows

481

several combinations of ramp angles for the double ramp configuration were investigated. In these investigations it was worked out that a convex sharp corner leads to a larger separation area then a convex continuously curved section of the intake with the consequence of a greater mass flow loss and higher heat loads on the walls. It was also found, that moderate ramp angles to compress and redirect the flow into the inlet lead to higher eﬃciency and more homogeneous flow. In this case the gradients in the flow field are smaller. As can be seen in Fig. 10 the sidewall was minimized such that it just catches the first ramp shock. This resulted in smaller sidewall boundary layers, thus led to less shock bending and sidewall compression, and this finally resulted in less mass flow loss and higher intake eﬃciency. To realize hypersonic combustion a necessary amount of mass flow has to enter the combustion chamber. Additionally, a certain area ratio between the inflow capture cross-section and the isolator cross-section has to be preserved (mostly around 6). To get the required compression ratio and mass flow by simultaneously having moderate ramp angles the intake has to be very long. Another diﬃculty appears while looking on the combustion chamber, which performs better when it has a more quadratic shape (height/width of max. 2), otherwise the 2D intake performs best when its height to width ratio is larger than 5. The shape of the combustion chamber cross-section is the bottle neck, so that this led to a very long and thin intake. The thinner the intake, the bigger the influence of the sidewall is. To avoid the deficiencies of simply ramp compression, studies of a 3D intake like shown in Fig. 2 are conducted. Here the sidewalls converge with an angle of 2.8◦ and contribute to the compression of the flow, resulting in a much shorter and wider intake (intake length/width: 2D ≈ 14; 3D ≈ 4). This intake was defined so that the averaged Mach number and temperature at the isolator exit were approximately the same as computed for the 2D intake without sidewall compression. In that case the sidewall started with the first ramp, and its leading edge is normal to the x-y plane and points in z direction. This led to thick boundary layers at the sidewalls and to big vortices generated in the corners. These vortices reduce the boundary layer thickness in some regions, because slow wall material is transported away from the walls. At other locations they lead to separation zones at the sidewalls and the ramp. These additional separations produce additional shock–shock interactions. That is why special attention must be paid to the sidewall compression and the eﬀects that occur by using it. So far only one single ramp configuration was investigated. Diﬀerent sidewall concepts will be studied. Beginning with diﬀerent sidewall angles, a larger angle leads to stronger shocks, but shorter intake which means smaller boundary layers leading to less compression, because the displacement thickness of the boundary layer is smaller. Furthermore, the “starting” angle of the sidewall will be varied (that means the leading edge of the sidewall lies in the x-y plane and has a certain angle to the y axis) as it was done by Goonko [10]. This will lead to smaller boundary layers. The problem here is that the leading edge of the sidewall has to be in

482

M. Krause, J. Ballmann Mach

0.2 0.6 1 1.4 1.8 2.2 2.6 3 3.4 3.8 4.2 4.6 5 5.4 5.8 6.2 6.6 7 7.4 7.8

z [m]

0.1

0 0.2

x [m]

0.4

Fig. 11. Mach contours for 3D scramjet intake with sidewall compression (Twall = 300K, M∞ = 8, Rel = 2.945 · 106 [1/m], T∞ = 226 K, p∞ = 1172 Pa) Table 1. Comparison of 2D and 3D intake values 2D with sidewall 3D 2.8◦ sidewall mass ﬂow possible [kg/s] 0.75355 0.69146 loss [%] 20.88 15.23 averaged temperature at exit[K] 1035 987 averaged contraction ratio at exit [bar] 28.5 42.7 averaged Mach number at exit 3.02 3.24 enthalpy loss [%] 8 4.8

a “V” shape for geometrical reasons [10]. To find a well suited shape several computations will be performed concerning this matter. Fig. 11 shows the Mach contours for a 3D intake like shown in Fig. 2 on its horizontal longitudinal mid plane of the isolator. The sidewall shocks, the separation at the sidewalls, the strong shock–shock interactions and thick boundary layers can be seen. The computations performed so far showed quite promising results for a 3D compression of the flow. Table 1 shows a comparison between the results (for same freestream flow conditions and isolator inlet cross-section) for a 2D intake (ramp angles of 7.5◦ and 19◦ , no sidewall compression, capture cross-section = 76 mm [width] × 228 mm [height]) and 3D (ramp angle 13.5◦ , 2.8◦ sidewall compression, capture cross-section = 106 mm [width] × 150 mm [height]) intake. Looking at these results the 3D intake seems to be the design of choice, but there are several disadvantages. Due to the strong shock interactions and vortices the flow field is very inhomogeneous. There are large gradients of the flow quantities. It is also not clear how this complex flow system will react on disturbances in the free-stream conditions. Because of the inhomogeneous flow, there are great diﬀerences in the heat loads on the surface of the structure. At the exit of the isolator are zones with very high temperature (around 1600 K) and zones with very low temperature (around 550 K) generated next to each other. The combustion chamber specialists will have to decide if that is favourable or modifications of the compression flow is required. Further investigations have to be done for the 3D intake to get a broader basis for decision by combustion chamber arguments. Fig. 12 and 13 compare the temperature profiles at the isolator exit for the both

Numerical Investigation of Hypersonic Intake Flows

483

T [K] 320

420

520

620

720

820

920 1020 1120 1220 1320 1420 1520

0.15 1600

1130

1190

1100

0.14

830

1030

y [m]

970 890 790

670

0.13

1170

1070

1365

1010

505

0.12 1010

1040

0.02

z [m]

0.04

0.06

0.08

Fig. 12. Temperature distribution at isolator exit cross-section for 3D intake (Twall = 300K, M∞ = 8.0, Rel = 2.945 · 106 [1/m], T∞ = 226 K, p∞ = 1172 Pa) T [K]

740

1220

0.22

840

0.21

y [m]

850 1080

0.2

940

1600

1380

0

0.01

0.02 z

[m]

0.03

1600 1550 1500 1450 1400 1350 1300 1250 1200 1150 1100 1050 1000 950 900 850 800 750 700 650 600 550 500 450 400 350

Fig. 13. Temperature distribution at isolator exit cross-section for 3D intake (Twall = 300K, M∞ = 8.0, Rel = 2.945 · 106 [1/m], T∞ = 226 K, p∞ = 1172 Pa)

intakes described in Table 1. Both 2D and 3D compression have their advantages, so it will be the future work to elaborate all these advantages and disadvantages of the diﬀerent geometry concepts to find a compromise for a most appropriate hypersonic intake design. 4.1 Transition One task will be to investigate transition from laminar to turbulent flow in hypersonic intakes. It is expected that the flow becomes turbulent on the first ramp. Additionally it may be possible that relaminarization takes place after the redirection and acceleration of the flow over the curved ramp section at the beginning of the isolator. To realize this the transition model by

484

M. Krause, J. Ballmann 0.01 laminar Blasius solution turbulent Blasius solution laminar QUADFLOW turbulent QUADFLOW Experiment transitional QUADFLOW Cor. Suluksna

0.008

cf

0.006

0.004

0.002

0 0

200000

400000

600000

Re_x

800000

1E+06

Fig. 14. Friction coeﬃcient for T3B ﬂat plate test case (Twall = 293K (M∞ =0.027394 , Rel = 6.09 · 105 [1/m], T∞ =293 K, ρ∞ = 1.225kg/m3 )

Menter/Langtry [11] was implemented in our in-house QUADFLOW code. Due to the fact, that some correlations of the model are not published yet, we use a correlation by Suluksna [12] and develop own ones. A first result for the standard T3B Flat Plate test case of Savill [13] can be seen in Fig. 14. It shows the friction coeﬃcient over the Reynolds number, calculated with streamwise coordinate.

5 Performance of the NEC SX8 in Comparison with Parallel Computer Systems As already mentioned not all simulations were carried out on the NEC cluster. Additional parallel computer systems like the SUN cluster of RWTH Aachen University and the Jump cluster of the Research Centre J¨ ulich were used. Having analyzed the computational eﬀort for simulations performed on all three systems for the same physical problem it can be asserted that the NEC SX-8 is the most eﬃcient one. In the following table computing times for two simulations are compared (2D/3D). Using FLOWer the averaged vector length was approximately 110 and the vector operation ratio was 98.2%. The overall performance was approximately 3.3 GFLOPS per second. It has to be mentioned that the waiting times on the NEC were longer as on the other computation systems, that reduces the factor a little. The hypersonic flow problem investigated here requires a great amount of computing resources, because a lot of iterations have to be done in each run to get a stationary result e.g. approximately 60000 iterations for one simulation.

Numerical Investigation of Hypersonic Intake Flows

485

Table 2. Performance of the diﬀerent computing systems SUN Jump NEC SX8 Iterations(2D)/(3D) 20000/10000 20000/10000 20000/10000 number of processors(2D/3D) 10/20 10/20 1/24 computing time in minutes(2D/3D) 1700/9000 800/4300 300/260 factor (NEC = 1)(2D/3D) 5.3/28 2.6/13.8 1/1

The 3D flow simulation will need very high storage as well. The use of the NEC SX8 system will be inevitable to achieve reliable design results in reasonable time. The QUADFLOW solver which is presently running on PC- and Workstation clusters like SUN and Jump will be transferred to NEC SX 8 as well. The flow computations in 3D will then be performed with both solvers FLOWer and QUADFLOW.

6 Conclusions For supersonic combustion with resulting positive engine thrust, the performance of the intake is essential. That is why a combined investigation of numerical analysis and experimental examination of scramjet intake geometries are conducted. The numerical investigation is of great importance, because one aim of the research done is designing a well-suited hypersonic intake for the given flight trajectory. The experimental work will allow to validate the computational results, so that the numerical simulations can be used to investigate the flow behaviour independently from experiments. In the future the task will be to create an appropriate hypersonic intake, therefore a lot of 3D numerical computations has to be done and a lot of computational power will be needed. So the use of the NEC SX8 system will be inevitable.

References 1. Anderson, J., Hypersonic and High Temperature Gas Dynamics, MacGraw–Hill, 1989. 2. Wilcox, D. C., “Turbulence Energy Equation Models,” Turbulence Modeling for CFD, Vol. 1, DCW Industries, Inc., La Canada, CA, 2nd ed., 1994, pp. 73–170. 3. Spalart, P. R. and Allmaras, S. R., “A One-Equation Turbulence Model for Aerodynamic Flows,” AIAA Paper 92–0439, January 1992. 4. Menter, F. R., “Two-Equation Eddy-Viscosity Turbulence Models for Engineering Applications,” AIAA Journal , Vol. 32, No. 8, Aug. 1994, pp. 1598–1605. 5. Kroll, N., Rossow, C.-C., Becker, K., and Thiele, F., “The MEGAFLOW Project,” Aerospace Science and Technology, Vol. 4, No. 4, 2000, pp. 223–237. 6. Reinartz, B. U., van Keuk, J., Coratekin, T., and Ballmann, J., “Computation of Wall Heat Fluxes in Hypersonic Inlet Flows,” AIAA Paper 02-0506, January 2002.

486

M. Krause, J. Ballmann

7. Bramkamp, F. D., Lamby, P., and M¨ uller, S., “An adaptive multiscale ﬁnite volume solver for unsteady and steady ﬂow computations,” Journal of Computational Physics, Vol. 197, 2004, pp. 460–490. 8. Radespiel, R., Rossow, C., and Swanson, R., “Eﬃcient Cell-Vertex Multigrid Scheme for the Three-Dimensional Navier-Stokes Equations,” AIAA Journal , Vol. 28, No. 8, 1990, pp. 1464–1472. 9. Krause, M., Reinartz, B., and Ballmann, J., “Numerical Computations for Designing a Scramjet Intake,” 25th ICAS Conference, 2006. 10. Goonko, Y. P., “Investigation of a Scramjet Model at Hypersonic Velocities and High Reynolds Numbers,” AIAA, Vol. 5273, 2002. 11. Menter, F. and Langtry, R., “A Correlation-Based Transition Model Using Local Variables Part 1 – Model Formulation,” Proceedings of ASME Turbo Expo, 2004. 12. Suluksna, K. and Juntasaro, E., “Assessment of Intermittency Transport Equations for Modeling Transition in Boundary Layers Subjected to Freestream Turbulence,” Whither Turbulence Prediction and Control (WTPC), 2006. 13. Savill, A., “Some recent progress in the turbulence modelling of by-pass transition,” Near – Wall Turbulent Flows, 1993, pp. 829.

Trimmed Simulation of a Complete Helicopter Configuration Using Fluid-Structure Coupling M. Dietz, M. Kessler, and E. Kr¨amer Institut f¨ ur Aerodynamik und Gasdynamik (IAG), Universit¨at Stuttgart, Pfaﬀenwaldring 21, 70550 Stuttgart, Germany

Summary. In this paper we present numerical simulation results of a complete helicopter conﬁguration. The CFD code FLOWer (DLR) is used for the simulation of the aerodynamics. For the main rotor an aeroelastic analysis is performed using weak ﬂuid-structure coupling between FLOWer and the ﬂight mechanics code HOST (Eurocopter). As a reference conﬁguration the complete helicopter conﬁguration investigated in the EU project GOAHEAD has been chosen.

1 Introduction Within the past years helicopter main rotor aerodynamics and aeroelasticity has been an extensive field of research at the Institut f¨ ur Aerodynamik und Gasdynamik (IAG). The Institute has gathered profound experience in fluid-structure coupling methodologies between Computational Fluid Dynamics (CFD) and Computational Structural Dynamics (CSD) for isolated main rotors. Both strong (time-accurate) and weak (or loose) coupling schemes have been implemented and applied to several test cases [1, 9, 10, 11, 12, 13]. Especially the weak coupling method has proven to be a powerful tool for the aeroelastic investigation of helicopter rotors as it allows for a quick trim of the rotor towards prescribed objectives with a comparatively low computational eﬀort. The trim of the rotor is absolutely mandatory in order to allow for a meaningful comparison of the numerical results to experimental or flight test data. As the next logical step towards a further improvement of the computational setup we have recently started to extend our investigations from isolated rotor computations towards the simulation of full helicopter configurations including fuselage, rotor head and tail rotor. At the current state of development the consideration of these additional components is restricted to the aerodynamic modelling by adding additional components to the CFD grid system. Fluid-structure coupling and trim is performed for the main rotor, i.e. the aerodynamic loads acting on the additional components are not yet taken

488

M. Dietz, M. Kessler, E. Kr¨ amer

into account for the coupling process. Nevertheless, the consideration of additional components within the aerodynamic simulation is a very important improvement. One can easily imagine that the flow around the rotor is directly aﬀected by e.g. the blocking eﬀect of the fuselage or wind tunnel walls. Thus, compared to an isolated rotor computation, the loads on the rotor change which in turn leads to a diﬀerent aeroelastic behaviour of the rotor. The results presented in this paper are part of the so-called blind test phase of the EU project GOAHEAD. In the frame of this project a complete helicopter will be investigated in the DNW wind tunnel. The project is intended to generate a comprehensive experimental database especially conceived to validate CFD codes for helicopter related applications. Wind tunnel experiments are going to be carried out in autumn 2007. Additional information on the GOAHEAD project can be found in [15]. In the following sections we will present the numerical methods, the weak coupling scheme between FLOWer and HOST and the results obtained.

2 Mathematical Formulation and Numerical Scheme 2.1 Governing Flow and Structure Models Aerodynamics. FLOWer solves the three-dimensional, unsteady Reynoldsaveraged Navier-Stokes equations (RANS) in integral form in the subsonic, transonic and supersonic flow regime. The equations are derived for a moving frame of reference and are given by 1 ∂Fv ∂Gv ∂Q ∂E ∂F ∂G ∂Ev + + + − + + =R. (1) ∂τ ∂ξ ∂η ∂ζ Re0 ∂ξ ∂η ∂ζ Q represents the solution vector containing the conservative variables. Centrifugal and coriolis accelerations are included in the source term R and Re0 is the reference Reynolds number evolving from non–dimensionalization. The parabolic–hyperbolic system of partial diﬀerential equations can be closed by assuming a thermally and calorically perfect gas as well as Newtonian fluid properties. Turbulence can be modelled either by algebraic or by transport equation models. The numerical procedure is based on structured meshes. The spatial discretization uses a central cell-vertex, cell-centered or an AUSM (Advection Upstream Splitting Method) finite volume formulation. Dissipative terms are explicitly added in order to damp high frequency oscillations and to allow sufficiently sharp resolutions of shock waves in the flow field. On smooth meshes, the scheme is formally of second order in space. The time integration is carried out by an explicit Runge–Kutta scheme featuring convergence acceleration by local time stepping and implicit residual smoothing. The solution procedure is embedded into a sophisticated multigrid algorithm, which allows standard

Simulation of a Complete Helicopter Conﬁguration

489

single grid computations as well as successive grid refinement. Unsteady calculations are carried out using the the implicit Dual Time Stepping Scheme which reduces the solution of a physical time step to a steady–state solution in pseudo time. This approach is very eﬀective as all convergence acceleration methods mentioned above can be used. The code is written in a flexible block structured form enabling treatment of complex aerodynamic configurations with any mesh topology. It is fully portable for either scalar or vector architectures on sequential and parallel computers. FLOWer is capable of calculating flows on moving grids (arbitrary translatory and rotatory motion). For this purpose the RANS equations are transformed into a body fixed rotating and translating frame of reference. Furthermore the Arbitrary-Lagrangian-Eulerian (ALE) method allows the usage of flexible meshes which is essential in the context of fluid-structure coupling. Arbitrary relative motion of grid blocks is made possible by the Chimera technique of overlapping grids. Additional information on the code can be found in References [16, 17]. Structure Dynamics. The Eurocopter flight mechanics tool HOST [18] represents a computational environment for simulation and stability analysis of the complete helicopter system. It enables the study of single helicopter components like isolated rotors as well as complete configurations with related substructures. As a general purpose flight mechanics tool, HOST is capable of trimming the rotor based on a lifting line method with 2D airfoil tables. The elastic blade model in HOST considers the blade as a quasi one– dimensional Euler-Bernoulli beam. It allows for deflections in flap and lag direction and elastic torsion along the blade axis. In addition to the assumption of a linear material law, tension elongation and shear deformation are neglected. However, possible oﬀsets between the local cross-sectional center of gravity, tension center and shear center are accounted for, thus coupling bending and torsional degrees of freedom. The blade model is based on a geometrically non-linear formulation, connecting rigid segments through virtual joints. At each joint, elastic rotations are permitted around the lag, flap and torsion axes. Since the use of these rotations as degrees of freedom would yield a rather large system of equations, the number of rotations is reduced by a modal Rayleigh-Ritz approach. A limited set of mode-like deformation shapes together with their weighting factors are used to yield a deformation description. Therefore, any degree of freedom can be expressed as h(r, ψ) =

n i=1

¯ i (r) qi (ψ) · h

(2)

490

M. Dietz, M. Kessler, E. Kr¨ amer

where n is the number of modes, qi the generalized coordinate of mode i ¯ i is the modal shape (a function of (a function of the azimuth angle ψ), and h the radial position r). 2.2 Weak Coupling Strategy The idea of the weak coupling scheme is as follows: HOST uses CFD loads to correct its internal 2D aerodynamics and re-trims the rotor. The blade dynamic response is introduced into the CFD calculation in order to obtain updated aerodynamic loads. This cycle is repeated until the CFD loads match the blade dynamic response evoked by them. A criterion for this converged state is given by the change of the three free control inputs, namely the collective and cyclic pitch angles, with respect to the preceding cycle. Convergence has been reached after the changes in these free control angles have fallen below an imposed limit. The specific steps of the coupling procedure are thus given as follows: 1. HOST determines an initial trim of the rotor based on its internal 2D aerodynamics derived from airfoil tables. The complete blade dynamic response for a given azimuth angle is fully described by the modal base and the related generalized coordinates. 2. The blade dynamic response is taken into account in the succeeding CFD calculation by the reconstruction of the azimuth angle dependent blade deformation from the modal base and the respective grid deformation of the blade grid. 3. The CFD calculation determines the 3D blade loads in the rotating rotor hub system (Fx [N/m], Fy [N/m], Fz [N/m], Mx [N m/m], My [N m/m], Mz [N m/m]) for every azimuth angle and radial section of the blade. 4. For the next trim HOST uses a load given by n−1 n−1 n n = F¯2D + F¯3D − F¯2D F¯HOST

(3)

n F¯2D represents the free parameter for the actual HOST trim. A new dynamic blade response is obtained which is expressed by an update of the generalized coordinates. 5. Steps (2) to (4) are repeated until convergence has been reached, i.e. when the diﬀerence n−1 n ΔF¯ n = F¯2D − F¯2D −→ 0 (4)

tends to zero and the trim-loads depend only on the 3D CFD aerodynamics. The actual trim objectives may be defined in diﬀerent ways. As three input parameters (collective and cyclic pitch angles) are set free, three independent output parameters may be predefined. The most common way is to define the time-averaged global rotor thrust, the global rotor pitching moment and the global rotor rolling moment. These trim objectives correspond almost

Simulation of a Complete Helicopter Conﬁguration

491

directly to the input control angles, i.e. cross-correlations are small. For the GOAHEAD project it was decided to apply a pure force trim, i.e. to predefine the rotor vertical, lateral and longitudinal force. Hence these trim objectives will also be used for the numerical simulation. The weak coupling method enforces the periodicity of the solution as on the one hand the blade dynamics are provided to the CFD method as Fourier series of the modal base and on the other hand a Fourier decomposition of the CFD loads is performed before providing them to the trim module of the CSD method. It is mandatory that the updated CFD loads for each successive trim are periodic with respect to the azimuth angle. After the CFD calculation has been restarted from the previous run, a certain number of time steps (i.e. a certain azimuth angle range) is necessary until the perturbation introduced by the updated set of generalized coordinates has been damped down and a periodic answer is obtained again. As mentioned earlier the weak coupling scheme is applied to the full helicopter configuration in exactly the same way as it is applied to an isolated rotor computation. Thus a 1/rev periodicity is enforced for every individual blade which is equivalent to a 4/rev periodicity of the complete rotor in case of a four-bladed rotor. Due to interference eﬀects with other components, especially fuselage or tail rotor, the aerodynamic answer of a rotor blade does not necessarily need to be strictly 1/rev, as other lower-frequency eﬀects (e.g. a low-frequency vortex shedding from the fuselage) might be superimposed. This is not taken into account by the current trim method. However, the occurance of such eﬀects depends on the flight case and its influence on the main rotor can generally be considered to be small. It can be stated that the current method accounts for all major interference eﬀects, especially the dominant blocking eﬀect of the fuselage. As seen from the above explanations the load exchange between CFD and CSD is restricted to the main rotor. Thus only the time-averaged loads of the main rotor can be trimmed towards prescribed trim objectives. This is at the same time the most common procedure applied to experiments: The experimental rotor is trimmed towards a set of prescribed time-averaged loads. Thus, these prescribed loads can be directly defined as the trim objective of the numerical simulation. If numerical results are supposed to be compared to free flight data, the main rotor loads either need to be directly measured or are obtained from a comprehensive code (like HOST in stand-alone mode) from the helicopter flight condition. In future a direct free flight trim of the full helicopter is envisaged. This means the time-averaged loads (three force components and three moment components) around the helicopter’s center of gravity need to be trimmed towards zero to allow for a steady flight condition. However, this will require an extension of the coupling interface and the trim module in HOST.

492

M. Dietz, M. Kessler, E. Kr¨ amer

2.3 Grid Deformation Tool In order to account for the blade dynamics in the CFD computation the blade grids need to be deformed according to the surface deformation provided by the structure solver. As this grid deformation process has to be carried out at the beginning of each physical time step it should perform as fast as possible. For this purpose a very quick and robust algebraic grid deformation tool utilizing Hermite polynomials has been developed at the IAG and integrated into FLOWer [1, 5]. The deformation tool works as follows: In the first step the deformed blade surface is determined by bending and twisting the blade quarter–chord line. The reconstructed surface description contains both the elastic blade deformation and all rigid body articulations. In order to minimize the amount of grid deformation and thus maintain a high level of grid quality, the entire virgin grid is rotated into the root–tip secant of the deformed quarter–chord line prior to the actual deformation process. In the second step the grid coordinates are updated by continuously reducing the deformation to zero towards the outer block boundary using transfinite interpolation. The grid deformation tool has been originally designed for monoblock blade grids. Recently this drawback has been eliminated as the deformation tool has been extended towards the treatment of multiblock grids. The diﬀerent blocks of the blade grid may now be distributed onto diﬀerent processes, thus allowing for an eﬀective parallelization of the flow computation.

3 Results 3.1 Flight Condition and Computational Setup The GOAHEAD test setup consists of the 7AD model main rotor (2.1 meter radius) including rotor head, the Bo105 model tail rotor and a NH90 fuselage. The cofiguration will be mounted on a pivoting sting in the closed test section of the DNW Large Low Speed Facility. Diﬀerent flight conditions covering the typical helicopter flight regime will be measured in the forthcoming measurement campaign. The flight conditions include Low-Speed Flight, Cruise Flight, High-Speed Flight and a Highly Loaded Rotor test case. The IAG contributes to the project with the coupled and trimmed CFDCSD simulation of the Low-Speed Pitch-Up case. This flight condition occurs during transition of the helicopter from hover into forward flight and is characterized by an impingement of the main rotor wake with the horizontal stabilizer, leading to a change in the overall pitching moment. As for all GOAHEAD test cases a pure force trim of the main rotor will be performed. The flight condition and the trim objectives are given in Table 1. As stated earlier the rotor is trimmed by adapting the collective and cyclic pitch control. The pitch angle of the configuration, and thus the rotor shaft angle, is held fixed at a predefined value.

Simulation of a Complete Helicopter Conﬁguration

493

Table 1. Flight condition and trim objective Rotor RPM Blade Tip Mach Number Free Stream Mach Number Fuselage Pitch Angle Rotor Shaft Angle Rotor Vertical Force (Fx ) Rotor Lateral Force (Fy ) Rotor Longitudinal Force (Fz )

954 0.617 0.059 +5◦ 0◦ 4500N (upward) 0 -44N (forward)

Table 2. Grid data Number of structures Number of blocks No. of cells main rotor No. of cells fuselage No. of cells background No. of cells tail rotor No. of cells rotor head No. of cells strut Total number of cells

Full Setup 10 169 4 x 811,200 8,694,848 265,728 2 x 335,488 830,208 476,672 14,183,232

Reduced Setup 6 143 4 x 811,200 561,280 220,928 – – – 4,027,008

Fig. 1. Full CFD grid conﬁguration

The CFD grid setup includes all components present in the experimental setup: Main rotor including rotor head, tail rotor, fuselage, sting and wind tunnel. The expensive grid generation process has been carried out at DLR. The full computational setup consists of 10 Chimera grid structures and 169

494

M. Dietz, M. Kessler, E. Kr¨ amer

blocks. The overall number of grid cells is approximately 14 million. For testing purposes a reduced grid setup consisting of main rotor, wind tunnel and a coarsened fuselage mesh has been created. The overall number of cells of this reduced setup is approximately 4 million. The grid data are summarized in Table 2. Figure 1 shows the full grid configuration. All CFD computations have been performed with an azimuthal time step of 2◦ for the main rotor, which is equivalent to an azimuthal time step of 10◦ for the tail rotor. The computations were run fully turbulent using the kω turbulence model according to Wilcox. 3.2 Trim Convergence As stated earlier, the blocking eﬀect of the fuselage is supposed to lead to significant changes in the aerodynamic and aeroelastic rotor solution compared to an isolated rotor computation. It was assumed that this eﬀect is dominant compared with the interference eﬀects between main rotor, rotor head and tail rotor. Hence it seemed reasonable to perform a preliminary trim using the reduced grid setup given in Table 2. The idea was to obtain a trimmed state close to the final trimmed state of the full configuration, and thus to reduce the number of trim cycles required for the full configuration. Figure 2 shows the convergence of the control angles θ0 , θC , and θS over all trim iterations performed. Note, that the dashed line (trim iterations 0 to 5) denotes the reduced grid setup and the solid line denotes the full grid setup (trim iterations 6 to 10). It can be clearly seen, that a trimmed solution for the reduced grid setup has been obtained after five trims. This trimmed state was assumed to be fairly close to the final trimmed state of the complete configuration. Thus, at trim iteration five we switched from the reduced to the full grid setup. Figure 2 shows that another five trim iterations are required in order to again obtain convergence – now for the full configuration. The comparison of the initial HOST trim (0th trim, no CFD) with the trimmed state of the reduced CFD grid setup (trim 5) and the trimmed state of the full configuration (trim 10) clearly shows that the initial assumption stated above did not prove true: The control angles of the final trimmed state of the full configuration show a significant deviation from the trimmed state of the reduced setup, comparable to the deviation from the initial HOST stand-alone trim. Thus, it can be stated that the flow field around the rotor is significantly influenced not only by the blocking eﬀect of the fuselage, but also by the interference eﬀects caused by the rotor head and the tail rotor. A detailed study of these interference eﬀects will be subject to further investigations. Figure 3 shows the distribution of the unsteady rotor forces predicted by CFD versus the azimuth angle. The distribution for the reduced setup (trim iterations 0 – 5) is shown on the left side. The distribution for the full setup (iterations 6 – 10) is given on the right side. Each re-trim is marked oﬀ with respect to the preceding trim by the line type change from solid to dash. It can be seen that the disturbance introduced by the update of the blade

Simulation of a Complete Helicopter Conﬁguration 10

3

9.8

2.5

θ0

θC

9.6 9.4

FLOWer/HOST reduced setup FLOWer/HOST full configuration

0

1

2

3

4

5

Trim

6

7

FLOWer/HOST reduced setup FLOWer/HOST full configuration

2

1.5

9.2 9

495

8

9

1

10

0

(a) Collective Pitch

1

2

3

4

5

Trim

6

7

8

9

10

(b) Lateral Cyclic Pitch

3

θC

2.5 FLOWer/HOST reduced setup FLOWer/HOST full configuration

2

1.5

1

0

1

2

3

4

5

Trim

6

7

8

9

10

(c) Longitudinal Cyclic Pitch Fig. 2. Convergence of the control angles

6000

7000

5000

100

50

0

3500

Fx[N]

Fx [N]

4000

Fy [N], Fz [N]

6000

50

4500

0

5500 -50 5000

-100

4500

3000 -50

2500 2000

Fx [N] Fy [N] Fz [N]

6500

0

2

4

6

8

Revolutions

10

12

(a) Reduced setup

14

4000 3500

Fy[N], Fz[N]

Fx[N] Fy[N] Fz[N]

5500

-150

0

1

2

3

4

5

6

7

Revolutions

8

9

10

11

12

(b) Full setup

Fig. 3. CFD unsteady rotor forces

dynamic response decreases from each trim cycle to the next as the procedure converges towards the trimmed state. For both configurations convergence is reached after five re-trims and the predefined rotor forces are obtained with the required accuracy.

496

M. Dietz, M. Kessler, E. Kr¨ amer

3.3 Aerodynamics and Blade Dynamics In this section sample results for aerodynamics and blade dynamics are given. As the results lack the experimental comparison (the GOAHEAD measurements have not yet been performed) no final conclusions can be made at this point of time. However all results obtained during this blind test phase look reliable and are in line to what has been expected for this flight case. Figures 4 and 5 show results for the blade dynamics. Figure 4 shows the blade tip flap deflection versus azimuth, whereas the elastic blade tip torsion amplitude versus azimuth is given in Fig. 5. The blade tip flap deflection predicted by the coupled CFD/CSD analysis does not show significant discrepancies from the HOST stand-alone computation. However, a clear 4/rev excitation in elastic torsion is introduced by CFD which is not observed in the HOST stand-alone solution. As the HOST-internal model is restricted to the isolated rotor the 4/rev excitation might be caused by interference eﬀects with the fuselage and the rotor head.

Fig. 4. Blade tip ﬂap deﬂection versus azimuth

Fig. 5. Elastic blade tip torsion versus azimuth

Simulation of a Complete Helicopter Conﬁguration

497

The distributions of the thrust coeﬃcient Ct and the sectional pitching moment coeﬀcient Cm M a2 on the rotor disk are shown in Fig. 6 and 7. As expected Ct reveals a fairly equal distribution between advancing and retreating blade side, consistent to the specified trim objective. The most significant features of the Cm M a2 -distribution can be seen at the outboard region of the blade starting at around r/R = 0.75. Here, the twist distribution of the 7AD blade changes from nose-down to nose-up leading to a change in the pitching moment behaviour. Figure 8 shows the pressure distribution in the symmetry plane of the fuselage for diﬀerent rotor azimuth angles. It can clearly be seen, that the rotor downwash impinging on the upper side of the fuselage during a blade passage (i.e. around ψ = 0◦ , ψ = 90◦ , ψ = 180◦ and ψ = 270◦ ) leads to a pressure rise on the upper side of the fuselage. The distinct peaks in the pressure distribution at rear end of the fuselage are caused by the vertical fin. The fuselage is not strictly symmetrical, but the fin is slightly tilted to starboard. Hence the slice shown is not located in the symmetry plane of the fin and intersects with the complex geometry of the tail rotor gear box fairing. Finally Fig. 9 shows an instantaneous snapshot of the 3D flow field which has been obtained for the complete configuration. The vortex wake structure is indicated as semi-transparent iso-surfaces using the λ2 vortex criterion [19]. Looking at the area above the tail boom close to the vertical fin, it can be

Fig. 6. Distribution of sectional thrust coeﬃcient

Fig. 7. Distribution of sectional pitching moment coeﬃcient

498

M. Dietz, M. Kessler, E. Kr¨ amer

Fig. 8. Fuselage pressure distribution

Fig. 9. Instantaneous snapshot of the 3D ﬂow ﬁeld

seen that the main rotor tip vortices are chopped up by the tail rotor. The slice visible at the reverse side of the fuselage uses λ2 as the contour variable and gives an impression on the wake trajectory. As desired for the Low Speed Pitch-Up test case the horizontal stabilizer is hit by the main rotor wake which can be seen from the fact that the wake shed from the front part of the rotor disk passes below the horizontal stabiliter and the wake shead from the rear part of the rotor disk passes above the stabilizer.

Simulation of a Complete Helicopter Conﬁguration

499

4 Computational Performance All CFD computations presented in this paper were performed on the NEC SX-8, either as single-node computations or as multi-node computations on three nodes. In contrast to earlier computations the multiblock grid deformation tool allowed for an eﬀective parallelization of the computation. The computational performance data are given in Table 3. It can be seen that both performance and vector operation ratio have decreased compared to our earlier isolated rotor computations. This is owed by the complexity of the configuration and cannot be avoided. As seen from Sect. 3.1 the Chimera setup is extremely complex. At the beginning of every physical time step the Chimera connectivities have to be set, involving a costly search and interpolation process which is only partially vectorized. Furthermore the complex geometry leads to a high number of comparatively small blocks leading to a decrease of the average vector length and thus to an overall performance decrease compared to an isolated rotor computation. However, one has to emphasize that running FLOWer computations with such a level of complexity within a reasonable time is actually only made possible by High-Performance Computing on the SX-8 which is thus the ideal platform for this purpose. Table 3. Computational performance Single Node Multi Node Platform SX8 SX8 Number of nodes 1 3 Number of CPUs 8 24 Number of blocks 169 169 Number of cells 14,183,232 14,183,232 GFLOPS 14.2 36.6 Vector operation ratio 96.0% 96.0% Wall clock time per rotor revolution 15.5h 7.0h Required memory approx. 2.0 kB/cell approx. 2.0 kB/cell

5 Conclusions and Outlook We have shown results of a trimmed numerical simulation of a complete helicopter configuration using weak CFD/CSD fluid-structure coupling. The weak fluid-structure coupling approach has proven to be well suited for the coupled and trimmed simulation of complete helicopter configurations. First results show that the control angles required to obtain the predefined state are significantly influenced by the level of aerodynamic modelling. Considerable differences in the control angles have been noticed between a simplified grid

500

M. Dietz, M. Kessler, E. Kr¨ amer

setup only including main rotor, fuselage and wind tunnel, and the full setup including all major experimental components. As soon as the experimantal data of the GOAHEAD experimental campaign will be available we will start an extensive comparison of our numerical results. Finally the computations will be repeated in the frame of the GOAHEAD post test phase using the exact wind tunnel conditions (free stream conditions, free stream turbulence, etc.) from the measurements. Acknowledgements This work has been funded by the European Union. The authors would like to thank the GOAHEAD partners for their support and the excellent cooperation. Furthermore we would like to thank the system administrators of HLRS for their technical support.

References 1. Altmikus, A. R. M.: Nichtlineare Simulation der Str¨ omungs-Struktur-Wechselwirkung am Hubschrauber. Dissertation, Universit¨ at Stuttgart, ISBN 3-18346607-4, 2004. 2. Pomin, H.: Simulation der Aerodynamik elastischer Rotorbl¨ atter mit einem Navier-Stokes-Verfahren. Dissertation, Universit¨ at Stuttgart, ISBN 3-83222276-6, 2003. 3. Buchtala, B.: Gekoppelte Berechnung der Dynamik und Aerodynamik von Drehfl¨ uglern. Dissertation, Universit¨ at Stuttgart, ISBN 3-8265-9732-X, 2002. 4. Wehr, D.: Untersuchungen zum Wirbeltransport bei der Simulation der instation¨ aren Umstr¨ omung von Mehrblattrotoren mittels der Euler–Gleichungen. Dissertation, Universit¨ at Stuttgart, ISBN 3-89722-285-X, 1999. 5. Hierholz, K.: Ein numerisches Verfahren zur Simulation der Str¨ omungs– Struktur–Kopplung am Hubschrauberrotor. Dissertation, Universit¨ at Stuttgart, ISBN 3-18-337507-9, 1999. 6. Stangl, R.: Ein Euler–Verfahren zur Berechnung der Str¨ omung um einen Hubschrauber im Vorw¨ artsflug. Dissertation, Universit¨ at Stuttgart, ISBN 3-89675141-7, 1996. 7. Wagner, S.: On the Numerical Prediction of Rotor Wakes Using Linear and Non–Linear Methods. AIAA–Paper 2000-0111, January 2000. 8. Wagner, S.: Flow Phenomena on Rotary Wing Systems and their Modeling. ZAMM 79 (1999) 12, pp. 795–820, 1999. 9. Altmikus, A. R. M., Wagner, S.: On the timewise accuracy of staggered aeroelastic simulations of rotary wings. AHS Aerodynamics, Acoustics, and Test and Evaluations Technical Specialist Meeting, San Francisco, CA, 2002. 10. Altmikus, A. R. M., Wagner, S., Beaumier, P., Servera, G.: A comparison: Weak versus strong modular coupling for trimmed aeroelastic rotor simulations. American Helicopter Society 58th Annual Forum, San Francisco, CA, 2002. 11. Dietz, M., Kr¨ amer, E., Wagner, S., Altmikus, A. R. M.: Weak coupling for active advanced rotors. Proceedings of the 31st European Rotorcraft Forum, Florence, Italy, 2005.

Simulation of a Complete Helicopter Conﬁguration

501

12. Dietz, M., Kessler, M., Kr¨ amer, E.: Aeroelastic Simulations of Isolated Rotors Using Weak Fluid-Structure Coupling. High Performance Computing in Science and Engineering, pp. 407–420, Springer Verlag, 2006. 13. Dietz, M., Kr¨ amer, E., Wagner, S.: Tip Vortex Conservation on a Main Rotor in Slow Descent Flight Using Vortex-Adapted Chimera Grids. Proceedings of the AIAA 24th Applied Aerodynamics Conference, San Francisco, June 2006. 14. Servera, G., Beaumier, P., Costes, M.: A weak coupling method between the dynamics code HOST and the 3D unsteady Euler code WAVES. Proceedings of the 26th European Rotorcraft Forum, The Hague, The Netherlands, 2000. 15. Pahlke, K.: The GOAHEAD Project. Proceedings of the 33rd European Rotorcraft Forum, Kazan, Russia, September 2007. 16. Kroll, N., Eisfeld, B., Bleecke, H.M.: The Navier-Stokes Code FLOWer. Notes on Numerical Fluid Mechanics, Vol. 71, Vieweg, 1999, pp. 58–71. 17. Schwarz, T.: The Overlapping Grid Technique for the Time-Accurate Simulation of Rotorcraft Flows. 31st European Rotorcraft Forum, Florence, Italy, September 2005.. 18. Benoit, B., Dequin, A-M., Kampa, K., Gr¨ unhagen, W.v., Basset, P-M., Gimonet, B.: HOST: A General Helicopter Simulation Tool for Germany and France. American Helicopter Society, 56th Annual Forum, Virginia Beach, Virginia, May 2000. 19. Jeong, J., Hussain, F.: On the Indentification of a Vortex. Journal of Fluid Mechanics, Vol. 285, pp. 69–94, 1995.

FEAST: Development of HPC Technologies for FEM Applications C. Becker, S. Buijssen, and S. Turek Institute for Applied Mathematics and Numerics, University of Dortmund, 44227 Dortmund, Germany [email protected] Summary. Modern processors reach their performance speedup not merely by increasing clock frequency, but to a greater extend by fundamental changes and extensions of the processor architecture itself. These extensions require the application developer to adapt programming techniques to exploit the existing performance potential. Otherwise the situation may arise that the processor becomes nominally faster, but the application doesn’t run faster [3, 4]. A limiting factor for computations is memory access. There is an ever increasing discrepancy between CPU cycle time and main storage access time. Fetching data is expensive in terms of CPU being idle. To narrow the gap between smaller CPU cycle times and possible access times of main storage in general, a rapid access temporary storage between CPU and main storage was introduced, the so-called cache. The basic idea of a cache is to store data following the locality of reference principle. Latency is reduced if a subsequently requested datum is found in the faster cache instead of having to transfer it from slow main storage. Given a suﬃcient locality of the data, i.e. the data of preceding accesses is still cached, the number of accesses to the cache will exceed those to slow main storage. Throughput can be increased signiﬁcantly this way. Access to main storage will not be faster with any access sample automatically, but only if the program uses mainly data being already in the cache. This requires appropriate adjustments being made to the applications [2].

1 High Performance Linear Algebra In the following, matrix vector multiplication techniques shall be examined for two common matrix storage formats (CSR and DIA). The CSR format stores the non-zero matrix entries line by line. An indirect access vector is needed though, which holds the positions within a matrix line. In contrast to CSR the DIA format stores complete diagonals. Memory consumption is higher, because zero entries are stored too. But an additional access vector is not required.

504

C. Becker, S. Buijssen, S. Turek

The matrix vector multiplication is one of the numerical simulation workhorses and its eﬃciency has a substantial influence on the entire simulation’s total eﬃciency. In the context of this study diﬀerent numbering strategies are examined [6–8]. The test configurations whose results are shown in Table 1 are: • SP-ID: Test of the matrix vector multiplication for the CSR format with identical indexing. This corresponds to a line by line numbering, in order to examine the influence of indirect addressing. • SP-MOD: Test of the matrix vector multiplication for the CSR format with a √ moderate indexing, the indices vary within the range of O(M ) with M = N . This corresponds to a semi-adaptive refinement of the grid without local refinement. • SP-STO: Test of the matrix vector multiplication for the CSR format with a stochastic indexing, the indices vary within the range of O(N ). This corresponds to a fully adaptive refinement where neighbouring elements may have very diﬀerent indices. Table 1. MFLOP/s of the diﬀerent matrix vector multiplication variants. Real peak performance is given for vector architectures, for non-vector machines LINPACK performance system

#unknowns

SP-ID

SPMOD

SP-STO BAND-V BAND-C

Opt250 3.95 GFlop/s Peak

4,225 66,049 1,050,625

508 402 408

511 390 385

508 212 71

743 253 238

1,254 570 356

Opt852 4.28 GFlop/s Peak

4,225 66,049 1,050,625

557 402 412

557 395 391

561 223 75

803 254 253

1,361 581 356

NEC-SX6 8 GFlop/s Peak

4,225 66,049 1,050,625

27 198 428

31 200 485

28 200 469

2,721 2,413 2,495

3,568 3,392 3,482

NEC-SX8 16 GFlop/s Peak

4,225 66,049 1,050,625

40 332 710

41 327 705

38 312 711

4,865 5,020 4,830

7,298 6,972 6,963

Power4 4.24 GFlop/s Peak

4,225 66,049 1,050,625

550 445 342

546 439 292

539 259 52

613 413 313

1,316 1,120 504

FEAST: Development of HPC Technologies for FEM Applications

• •

505

BAND-V: Test of the matrix vector multiplication for the DIA format with variable matrix entries. BAND-C: Test of the matrix vector multiplication for the DIA format with constant matrix entries.

The comparison of architectures shows clear diﬀerences in the behaviour between vector and non-vector machines. The non-vector architectures Opteron and Power4 are clearly dependent on the numbering of unknowns and the length of vectors when using the CSR format. The longer the vectors the worse the rates will be. In the worst case the obtained rates decrease as low as to 50 MFlop/s. The diagonal oriented matrix vector multiplication supplies higher performance data, in particular multiplication with constant matrix entries. Increasing vector lengths lower the performance of bandwise multiplication to the disappointing one of sparse matrix vector multiplication. Vector machines exhibit a completely diﬀerent behaviour. Sparse multiplication gives very bad results for small vector lengths. Increasing the vector length leads to better results, independent of the kind of numbering. Both eﬀects are contrary to the behaviour of non-vector machines. The diagonal oriented matrix vector multiplication supplies much higher rates. Multiplication with constant entries reaches nearly half of the peak performance. These results are independent of the vector length. Discretisations with finite diﬀerence methods or on tensor product meshes with isotropic operators produce very regular matrix structures, e.g. the 9 point stencil in case of the Poisson operator. The matrix entries are arranged in diagonals. Substantially important is the observation that matrix diagonals have constant entries for some operators if the underlying tensor product mesh is orthogonal. For these meshes a complete discretisation can be described with merely 9 floating point values. Furthermore, the storage of the diagonals as sequential vector allows to renounce indirect addressing. This enables faster processing.

I+M-1

I+M

I+M+1

I-1

I

I+1

UU UD

I-M-1

I-M

UL

I-M+1

DU LL LD

Fig. 1. Window for MV multiplication

LU

DL

DD

506

C. Becker, S. Buijssen, S. Turek

In case of the Poisson √ operator, the matrix consists of blocks of tridiagonal submatrices of size M = N which are coupled with one another. The idea is to not work on the complete diagonal at once, but on slices of it only (so-called ‘windows’) as illustrated in Fig. 1. By this policy a greater part of the result vector x can be held in cache reducing the rate of cache misses. The results of this method, in each case for variable (-V) and constant (-C) matrix entries, are shown in Table 2. The Sparse Banded BLAS variants (SBB) clearly show a better eﬃciency than the diagonal variants. With increasing vector length, the vector processors are more eﬃciently utilised and reach in case of constant entries even the peak performance of the processor. For non-vector machines this is not the Table 2. MFLOP/s rates for Sparse Banded BLAS MV variants. Abbreviations are: SBB-V: Sparse Banded BLAS MV multiplication with variable matrix entries, SBBC: Sparse Banded BLAS MV multiplication with constant matrix entries, BAND-V: diagonal oriented MV multiplication with variable matrix entries, BAND-C: diagonal oriented MV multiplication with constant matrix entries system

#unknowns

SBB-V

SBB-C

BAND-V BAND-C

Opt250 3.95 GFlop/s Peak

4,225 66,049 1,050,625

1,605 660 591

2,890 2,353 1,774

743 253 238

1,254 570 356

Opt852 4.28 GFlop/s Peak

4,225 66,049 1,050,625

1,805 660 591

3,334 2,353 1,774

803 254 253

1,361 581 356

NEC-SX6 8 GFlop/s Peak

4,225 66,049 1,050,625

3,212 3,588 3,550

4,335 5,786 5,630

2,721 2,413 2,495

3,568 3,392 3,482

NEC-SX8 16 GFlop/s Peak

4,225 66,049 1,050,625

6,433 8,392 7,989

6,935 15,062 15,970

4,865 5,020 4,830

7,298 6,972 6,963

Power4 4.24 GFlop/s Peak

4,225 66,049 1,050,625

1,806 896 347

2,709 2,509 1,774

613 413 313

1,316 1,120 504

FEAST: Development of HPC Technologies for FEM Applications

507

case, but the performance is still superior to bandwise operations. Due to the small number of operations which are performed per datum the bandwidth to main storage remains the limiting factor.

2 Scalable Recursive Clustering Another central aspect of the FEAST (Finite Element Analysis & Solution Tools) project is the analysis of solution schemes for linear equation systems with a large number of unknowns. Multi grid schemes belong to the most powerful tools for solving huge systems of linear equations. They have a set of pro and cons (see the work of Kilian [5]). Kilian introduces a method called ScaRC (Scalable Recursive Clustering) which combines the advantages of multi grid, domain decomposition methods and Krylov space methods and avoids their disadvantages as far as possible. In general, serial multi grid schemes with point-wise smoothing have a very good numerical convergence rate, while domain decomposition methods reach a high degree of data locality, which is important for proper utilisation of computing resources. The application of a ScaRC smoothing step consists of the following tasks: • • •

calculate the global defect, perform a local multi grid method with the restricted defect as right hand side of the local problem, perform global defect correction, comprising the local problems’ solutions.

Multi layer ScaRC works in such a way that the smoothing operation of one hierarchical layer consists of a multi grid sweep on a lower hierarchy level. The smoother for this inner multi grid is again a multi grid method building a recursive scheme. The number of global iteration steps is reduced, resulting in a reduction of communication time and, hence, achieving a higher total eﬃciency. The ScaRC layer hierarchy consists of the following levels: •

•

•

matrix block layer: Atomic unit of the grid structure is a matrix block. It is either of tensor product structure (a so-called ‘macro’, being a patch of elements) or a group of macros. Characteristic property of a matrix block is that only on this level the system matrix can directly be accessed. parallel block layer: A parallel block consists of matrix blocks which are assigned to the same processing unit. Motivation for this data structuring is the assumption that data exchange between entities of the same processing unit is less expensive than between diﬀerent processing units. domain block layer: This level comprises all matrix blocks of the complete domain. All global operations (global defect calculation, global norm calculation etc.) are defined on this level.

508

C. Becker, S. Buijssen, S. Turek

Figure 2 shows a ‘typical’ grid containing the following prototypical macro shapes: 1. 2. 3. 4.

isotropic macro with isotropic refinement, isotropic macro with anisotropic refinement, anisotropic macro with isotropic refinement, anisotropic macro with anisotropic refinement.

Kilian examines the ScaRC smoother on the basis of a numerical analysis [5] and revealed a single weak point of the method: convergence rates deteriorate for high degrees of anisotropy on macro level. In order to evade this dependency the following enhancements shall be examined: • Introduction of an additional hierarchy layer beside the local and global level: Existing anisotropies can be better ‘hidden’ locally, while the processing units should have more work to carry out, in order to reduce the number of global iterations. Remember that a global iteration also means global communication, which would impair the eﬃciency rate.

Anisotropic macro with anisotropic refinement

Anisotropic macro with isotropic refinement

Anisotropic macro with isotropic refinement

Anisotropic macro with anisotropic refinement

Isotropic macro with anisotropic refinement

Anisotropic macro with isotropic refinement

Fig. 2. Several prototypical macro shapes for an example conﬁguration

FEAST: Development of HPC Technologies for FEM Applications

• •

509

Clustering several macros into a single matrix block sharing one matrix: Macros with a high anisotropic degree can be hidden better. Hence, more powerful local smoothers like ILU can be applied. Use of diﬀerent local smoothers within one ScaRC smoothing step (multishape): Typically, the macros of a domain diﬀer in shape, e.g. there are isotropic and anisotropic ones. Instead of using for all macros the same smoother, every macro will see the most suitable smoother being used. Suitability means being able to solve the problem at minimal cost with respect to computing time and memory consumption.

The use of all three levels is not mandatory. It is feasible to omit one hierarchical layer. Such a two level scheme works either for the case DB-PB (global iteration on domain block layer, local iteration on parallel block layer) or DB-MB (global iteration on domain block layer, local iteration on matrix block layer). To examine this extended ScaRC method, a set of calculations for diﬀerent test configurations are performed. These configurations diﬀer with respect to refinement level of the domain, the number of macro elements, local and global anisotropy degrees and number and kind of domain partitions. As test problem the Poisson problem will be considered: −Δu = f on a domain Ω with Dirichlet boundary conditions. This equation serves as prototype for elliptical diﬀerential equations of second order. Equations of this type play a key role in the numerical simulation of fluid and astrophysical problems. For the evaluation of the ScaRC scheme both the numerical eﬃciency represented by the convergence rate and the computing eﬃciency in terms of the absolute runtime are considered. ScaRC is a very modular solution scheme, the following solver schemes can be characterised as follows: • • • •

number of layers: 2 or 3, hierarchical layer: definition of hierarchical layer the ScaRC smoothing is to be performed: domain block (DB), parallel block (PB) or matrix block (MB), solver: multigrid (V, F or W cycle) or CG scheme, local smoothers: Jacobi, Tri-Gauss-Seidel, ADI-Tri-Gauss-Seidel, ILU or multishape. In the test scenarios the following parameters are examined:

• • •

dependency on the element size h, dependency on the number of macro elements, dependency on local anisotropy on global and local level. Tables 3 through 5 contain some results of the test calculations.

510

C. Becker, S. Buijssen, S. Turek

Table 3. Aggregated results of the eﬃciency of ScaRC solvers with respect to element size h, number of iterations (in parenthesis) and runtime in seconds are given (parallel conﬁguration with 16 partitions on a B8×8 topology, see [1]) algorithm 4,225 MG (46) 0.73 2-level-ScaRC DB-PB type V (24) 0.62 2-level-ScaRC DB-PB type F (12) 0.57 2-level-CG-ScaRC DB-PB type V (7) 0.37 2-level-CG-ScaRC DB-PB type F (4) 0.42 2-level-CG-ScaRC DB-MB (9) 0.40 3-level-CG-ScaRC (4) 1.49 2-level-CG-ScaRC DB-MB-MS (8) 0.40 2-level-CG-ScaRC DB-MB-CL (2) 1.91

16,641 (74) 2.05 (32) 2.01 (16) 2.70 (7) 0.85 (4) 1.36 (11) 0.93 (5) 8.79 (11) 0.99 (2) 6.66

n 66,049 (115) 5.19 (43) 5.14 (21) 6.83 (12) 3.14 (5) 3.15 (14) 1.55 (5) 20.24 (14) 1.79 (2) 12.77

263,169 (160) 12.02 (51) 19.21 (25) 30.32 (10) 7.11 (6) 12.09 (15) 3.66 (6) 90.88 (16) 4.09 (1) 20.68

Table 4. Aggregated results of the eﬃciency of ScaRC solvers with respect to number of macro elements, number of iterations (in parenthesis) and runtime in seconds are given (parallel conﬁguration with 16 partitions for a = 0.01, see [1]) algorithm

macro decomposition [a] [a] [a] A4×4 A8×8 A16×16 MG (37) 2.06 (42) 3.24 (44) 6.28 2-level-ScaRC DB-PB type V (56) 3.20 (54) 6.04 (51) 29.16 2-level-ScaRC DB-PB type F (28) 4.29 (26) 10.95 (24) 48.88 2-level-CG-ScaRC DB-PB type V (6) 0.59 (6) 2.66 (5) 14.40 2-level-CG-ScaRC DB-PB type F (4) 1.29 (4) 4.48 (4) 24.16 2-level-CG-ScaRC DB-MB (6) 0.47 (7) 1.38 (8) 5.83 3-level-CG-ScaRC (4) 5.11 (4) 31.46 (3) 128.25 2-level-CG-ScaRC DB-MB-MS (6) 0.55 (7) 1.83 (8) 7.72 2-level-CG-ScaRC DB-MB-CL (4) 1.20 (6) 13.60 (5) 48.43 Table 5. Aggregated results of the eﬃciency of ScaRC solvers with respect to local anisotropy on element and macro level, number of iterations (in parenthesis) and runtime in seconds are given (parallel conﬁguration with 16 partitions on a B8×8 topology, see [1]) algorithm

element anisotropy isotropic low high aniso. aniso. MG (160) 11.57 (80) 6.26 (4) 0.38 2-level-ScaRC DB-PB type V (51) 18.89 (33) 12.86 (8) 1.91 2-level-ScaRC DB-PB type F (25) 30.98 (16) 16.52 (4) 3.21 2-level-CG-ScaRC DB-PB type V (10) 7.48 (9) 6.58 (12) 3.30 2-level-CG-ScaRC DB-PB type F (6) 12.02 (6) 11.26 (6) 8.61 2-level-CG-ScaRC DB-MB (15) 2.32 (16) 2.87 (14) 3.42 3-level-CG-ScaRC (6) 93.88 (7) 112.37 (9) 114.65 2-level-CG-ScaRC DB-MB-MS (16) 3.45 (14) 2.94 (13) 3.96 2-level-CG-ScaRC DB-MB-CL (18) 39.08 (14) 30.01 (14) 28.65

FEAST: Development of HPC Technologies for FEM Applications

511

Summary of results: • • • • •

For isotropic configurations a standard multi grid scheme is suﬃcient. For configurations with strong anisotropies and a high degree of parallelism the variants 2-level-CG-ScaRC DB-MB and 2-level-CG-ScaRC DBPB type V achieve the results with best total eﬃciency. The results for the multishape smoother are not substantially worse compared to those of ADI-TriGS, but at lower storage costs. The 3-level-CG-ScaRC has the best numerical eﬃciency. But due to the very high expenditure of a single smoothing step, the total eﬃciency is worse. Using more processes does not reduce run time. Clustering of macros does not lead to a better total eﬃciency. The numerical eﬃciency is improved, but usually at the price of a worse total eﬃciency.

3 FEAST In the following section, a ScaRC algorithm is applied to diﬀerent topologies with diﬀerent partition sizes, in order to examine the eﬃciency of the ScaRC scheme for massive parallel applications. The calculations are performed on three architectures Power4, NEC and Opteron cluster with the help of the FEAST library. A 2-level-ScaRC DB-PB type V solver is used for all configurations except for the unit square configuration. For the latter configuration a common multigrid scheme with Jacobi smoothing is used. Partition sizes are chosen based on the FEAST design mark to use a n + 1 architecture for its parallel processes: • •

n slave processes perform the arithmetic work, one master process performs global tasks such as synchronisation, global coarse grid solution etc.

The Power4 and NEC architecture are SMP systems, containing 32 and 8 CPUs, respectively, per node. Due to this technical reason the number of processes per calculation is chosen as multiple of 8. The used partition sizes are: 7, 15, 31, 63, 127 and 255. The Opteron cluster does not have this constraint, but in order to compare the results, the same partition sizes for this architecture are used. Topology: unit square Figure 3 shows the configuration which is to be tested. The unit square represents a simple grid structure on which the eﬃciency behaviour can be tested for an ‘optimal’ problem topology. The Tables 6 and 7 show the result of the unit square topology tests. For the Power4 architecture the eﬃciency rates per CPU in comparison to the

512

C. Becker, S. Buijssen, S. Turek Fig. 3. Unit square topology with 256 macros

Table 6. Eﬃcency tests for unit square topology, overall runtime and eﬃciency are given (convergence rate ρ = 0.03) level #unknowns

arch.

7p 15p 31p 63p sec MF/s sec MF/s sec MF/s sec MF/s

8

16,785,409

Power4 15.8 1,675 11.6 2,289 11.5 2,302 22.3 1,194 NEC 23.7 1,073 12.6 1,959 10.2 2,415 6.3 3,932 Opteron 6.6 3,719 4.5 5,513 5.0 4,965 6.5 3,814

9

67,125,249

Power4 - 38.7 2,720 27.4 3,844 34.0 3,102 NEC 34.0 2,875 18.7 5,223 15.0 6,519 8.8 11,067 Opteron 20.7 4,714 19.5 5,042 13.6 7,198 9.1 10,786

10 268,468,225

Power4 - 77.5 5,407 NEC 55.3 7,034 30.6 12,738 24.1 16,159 13.2 29,425 Opteron - 43.6 8,935 24.5 15,833

expected eﬃciency rate of ∼350 MFlop/s per CPU are low. Equally the local speedup rates are only in the range < 1.41. A degradation of the runtime with increasing number of processors partly occurs. The eﬃciency rate with rising problem size improves, even if only up to a rate of ∼100 MFlop/s. The NEC architecture scales better for this topology, the speedup rates vary within the range of 1.2 to 1.8. The eﬃciency rate is proportional to the problem size (up to 1 GFlop/s per CPU). This high eﬃciency rate is credited to the local smoother used, which fits well the vector architecture of the NEC. This value is still clearly lower than the expected eﬃciency value of 8 GFlop/s per CPU (see Sparse Banded BLAS tests). The dependency on the local problem size is obvious. The eﬃciency per CPU rises from 153 MFlop/s to over 1,000 MFlop/s (for 7 tasks). With small problem sizes the non–vector program sections (communication, synchronisation etc.) dominate the calculation. Opteron architecture reaches speedup rates up to 1.78, but at the worst

FEAST: Development of HPC Technologies for FEM Applications

513

Table 7. Eﬃcency tests for unit square topology, speedup to previous partitioning and eﬃciency per CPU are given level #unknowns

arch.

7p 15p 31p 63p speedup MF/s speedup MF/s speedup MF/s speedup MF/s

8

16,785,409

Power4 NEC Opteron

1.00 1.00 1.00

239 153 531

1.36 1.88 1.47

153 131 368

1.00 1.24 0.90

74 78 160

0.52 1.62 0.77

19 62 61

9

67,125,249

Power4 NEC Opteron

1.00 1.00

411 673

1.00 1.82 1.69

181 348 336

1.41 1.25 1.43

124 210 232

0.81 1.70 1.49

49 176 171

10 268,468,225

Power4 NEC Opteron

1.00 1,005 -

1.81 -

849 -

1.27 1.00

521 288

1.00 1.83 1.78

89 467 253

smaller than 1. The latter means that runtime increases with more processors being used. The local eﬃciency rate is for small processor numbers at a high value, but the more processors are involved, the more the eﬃciency decreases. The communication time becomes the dominating issue. Still, eﬃciency rate increases with problem size. Topologie: Tux Figure 4 shows the configuration of this test. The topology contains a complex boundary with an automatically generated coarse grid.

Fig. 4. Tux topology with 1,377 macros

514

C. Becker, S. Buijssen, S. Turek

The Tables 8 and 9 outline the results for this topology. The Power4 architecture reaches speedup rates between 1.16 and 1.65, the local eﬃciency never exceeds 100 MFlop/s per CPU. This configuration could not be computed on NEC architecture because of invalid code generated by the compiler. This error crashes the direct coarse grid solver, so only a distributed CG method as coarse grid solver could be used for the Tux configuration. Since this method is less eﬃcient compared to a direct solver, the absolute runtimes can not be compared to the results of the other architectures. The speedup rates vary within the range of 1.63 to 2. The initial eﬃciency rates are very low, they reach only a value of ∼45 MFlop/s. They improve with increasing problem size to over 350 MFlop/s for the partition with 63 processes. For the partition with 127 processes the rate drops again to ∼280 MFlop/s. The speedup rates Table 8. Eﬃciency tests for Tux topology, overall runtime and eﬃciency are given level #unknowns

arch.

ρ

31p 63p sec MF/s sec MF/s

127p sec MF/s

255p sec MF/s

8

90,317,056

Power4 281.5 3,178 170.4 5,252 116.4 7,698 100.4 10,306 NEC 0.247 297.0 2,835 156.7 5,376 91.9 9,167 Opteron 120.5 7,256 65.2 13,412 40.5 21,586 -

9

361,120,256

Power4 - 269.6 11,912 202.3 17,408 NEC 0.244 454.6 6,621 222.8 13,511 128.0 23,519 Opteron 529.0 5,934 192.6 16,299 104.7 29,974 -

10 1,444,185,088

Power4 NEC 0.214 Opteron

-

- 595.0 22,328 364.9 36,415 - 615.6 25,637

-

-

Table 9. Eﬃciency tests for Tux topology, speedup to previous partitioning and eﬃciency per CPU are given level #unknowns

arch.

31p 63p 127p 255p speedup MF/s speedup MF/s speedup MF/s speedup MF/s

8

90,317,056

Power4 NEC Opteron

1.00 1.00 1.00

103 91 234

1.65 1.89 1.85

83 85 213

1.46 1.70 1.61

61 72 170

1.16 -

40 -

9

361,120,256

Power4 NEC Opteron

1.00 1.00

214 191

2.04 2.75

215 259

1.00 1.74 1.84

94 185 236

1.33 -

68 -

10 1,444,185,088

Power4 NEC Opteron

-

-

1.00 -

354 -

1.63 1.00

287 201

-

-

FEAST: Development of HPC Technologies for FEM Applications

515

of the Opteron architecture are in a range of 1.61 to 2.75. The local eﬃciency reaches at first ∼200 MFlops/s. With rising problem sizes the local eﬃciency varies within a moderate range.

4 Discussion The relatively large fluctuations of the scaling factor are partially based on the structure of the ScaRC solver. The smoother used is of type 2-level-ScaRC DB-PB type V, this scheme uses as smoother a local method which is defined on parallel block level. For each new partitioning the composition of the parallel blocks changes and concomitantly the convergence behaviour of the assigned local smoother. A new partitioning may need more local steps compared to a preceding partitioning resulting in a larger runtime. This means a degradation of the scaling factor. The results for the Power4 architecture regarding local eﬃciency and scaling are disappointing. This architecture does not reach the eﬃciency rates expected from the Sparse Banded BLAS test results. The scaling is relatively bad compared to other architectures. Assumably, the communication subsystem is to be blamed for this behaviour. Contrastingly, for the same problem configurations the results are substantially better for the Opteron architecture and lie within the range of expectations. A certain loss in eﬃciency, which is caused particularly by communication of the processes, can not be avoided. Note that the eﬃciency rate with rising problem size remains almost constant, whereas applications which use the sparse standard matrix vector multiplication (see Sparse Banded BLAS tests) show a clear eﬃciency loss with rising problem size. The results for the NEC are satisfying concerning the scaling. The local eﬃciency is satisfying for the smoother used. A smoother better adapted to the architecture should give even better results. The NEC architecture shows a clear dependency on the problem size. For small problem sizes the nonvector instructions dominate runtime, whereas only for greater vector lengths the vector instructions outweigh improving the total eﬃciency rate clearly. Acknowledgements Simulations on NEC have been performed on the system at HLRS Stuttgart. Simulations on the Power4 architecture were done on the JUMP system at Forschungszentrum J¨ ulich. The Opteron cluster used is LiDO, located at the HRZ of the University of Dortmund. The authors would like to acknowledge valuable help and support of the administrative staﬀ at the sites mentioned.

516

C. Becker, S. Buijssen, S. Turek

References [1] Becker, Ch.: Strategien und Methoden zur Ausnutzung der High-PerformanceRessourcen moderner Rechnerarchitekturen f¨ ur Finite-Element-Simulationen und ihre Realisierung in FEAST (Finite Element Analysis & Solution Tools). Logos Verlag, Berlin, Universit¨ at Dortmund, Dissertation, Juli 2007 ¨ de, [2] Douglas, C. C. ; Haase, G. ; Hu, J. ; Karl, W. ; Kowarschik, M. ; Ru U. ; Weiss, C.: Portable memory hierarchy techniques for PDE solvers: Part I. In: SIAM News 33 (2000), Nr. 5, S. 1, 8–9 [3] Gropp, W. D. ; Kaushik, D. K. ; Keyes, D. E. ; Smith, B. F.: High Performance Parallel Implicit CFD. In: Parallel Computing 27 (2001), S. 337–362 [4] Keyes, D. E.: Four Horizons for Enhancing the Performance of Parallel Simulations based on Partial Diﬀerential Equations. In: Europar 2000 Parallel Processing Bd. 1900, 2000, S. 1–17 [5] Kilian, S.: Ein verallgemeinertes Gebietszerlegungs-/Mehrgitterkonzept auf Parallelrechnern. Logos Verlag, Berlin, Universit¨ at Dortmund, Dissertation, September 2001 [6] Turek, S. ; Becker, Ch. ; Kilian, S.: Some concepts of the software package FEAST. In: Palma, J. M. (Hrsg.) ; Dongarra, J. J. (Hrsg.) ; Hernandes, V. (Hrsg.): VECPAR‘98 - Third International Conference for Vector and Parallel Processing Bd. 1573, Springer, Berlin, 1998 [7] Turek, S. ; Becker, Ch. ; Kilian, S.: Hardware-oriented numerics and concepts for PDE software. In: Future Generation Computer Systems 22 (2006), Nr. 1-2, S. 217–238 [8] Turek, S. ; Runge, A. ; Becker, Ch.: The FEAST Indices - Realistic evaluation of modern software components and processor technologies. In: Computers and Mathematics with Applications 41 (2001), S. 1431–1464

Global Long-Term MIPAS Processing M. Kiefer, U. Grabowski, and H. Fischer Institut f¨ ur Meteorologie und Klimaforschung, Universit¨ at Karlsruhe/Forschungszentrum Karlsruhe, Karlsruhe, Germany {michael.kiefer,udo.grabowski,herbert.fischer}@imk.fzk.de

Summary. The Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) is a Fourier transform mid-infrared limb scanning high resolution spectrometer for measurement of more than 30 atmospheric trace species related to atmospheric chemistry and global change. At the Institute for Meteorology and Climate Research (IMK), measured radiance spectra are used for retrieval of altitude-resolved proﬁles of abundances of diﬀerent trace species of the atmosphere (H2 O, O3 , N2 O, CH4 , NO2 , HNO3 , NO, CO, CFC-11, CFC-12, HCFC-22, CFC-113, HCFC-142b, H2 O2 , HDO, ClONO2 , N2 O5 , HO2 NO2 , HOCl, ClO, C2 H6 , SF6 , NH3 , OCS, HCN, HCOOH, PAN, acetone, CH3 CN, and others). These 4-D trace gas distributions are used for the assessment of (a) stratospheric ozone chemistry (b) stratospheric cloud physics and heterogeneous chemistry (c) tropospheric stratospheric exchange (d) intercontinental transport of pollutants in the upper troposphere (e) mesospheric stratospheric exchange (f) eﬀects of solar proton events on stratospheric chemistry. While data analysis strategy developed at IMK over the last ﬁfteen years has proven successful for atmospheric case studies of limited spatial and temporal coverage, numerous research topics require either a complete global data set, or the retrieval of many diﬀerent species, or both. These requirements cannot be fulﬁlled by IMK’s limited computational resources. The opportunity to process major parts of the data on the XC supercomputers therefore oﬀers a unique chance to improve not only the quantity of processed data but also the quality, because in the same time more species can be processed, which leads to a more thorough picture of middle atmosphere chemistry. After the successful transfer of the core processing tools to the XC1/XC2 several projects have already been processed partly on these supercomputers. Examples of projects are given and the process of tool transfer and adaptation is discussed as well as the current performance and remaining problems and potential sources for further optimization.

520

M. Kiefer, U. Grabowski, H. Fischer

1 Introduction For many questions of atmospheric research global distributions of trace species resolved in space and time play a key-role. These data can best be obtained with remote sensing techniques. The Institut f¨ ur Meteorologie und Klimaforschung (IMK) has a long standing experience in developing, operating and characterizing various remote sensing instruments and scientific data analysis. A major Earth observation remote sensing project is the space-borne MIPAS instrument which originally has been proposed by IMK.

2 The MIPAS/Envisat Mission The Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) is a space-borne cooled Fourier transform spectrometer which operates in the mid-infrared spectral region from 865 to 2410 cm−1 (14.6 to 4.15 µm). The spectral resolution is 0.05 cm−1 . It measures thermal emission spectra of the Earth’s limb, whereby variation of the tangent altitude provides altituderesolved information [1, 2]. MIPAS is part of the core-payload of the Envisat research satellite. Envisat is a sun-synchronous polar orbiter which passes the equator at 10:00 am local time in southward direction 14 times a day. The MIPAS mission intends to improve the understanding of the composition of the Earth’s atmosphere by measurement of 4D distributions of more than 30 trace species relevant to atmospheric chemistry and climate change. Operation of the satellite and the instrument, level-1b (L1b) data processing (Fourier transformation, phase correction, calibration) and generation of some limited level-2 (L2) data product (distributions of temperature and easy-to-analyse trace species, namely H2 O, O3 , CH4 , N2 O, HNO3 , and NO2 ) is done operationally under responsibility of the European Space Agency, while retrieval of further species (ClONO2 , ClO, HOCl, (H)CFCs, NO, N2 O5 , CO, C2 H6 and many others) is left to scientific institutions. MIPAS was operational in its original, high spectral resolution specification from 1 March 2002 to 26 March 2004 and provided about 1000 limb scans a day, each consisting of 17 times 60000 spectral radiance measurements. In summary 4 TB of data are available for scientific analysis from this period. Due to an instrument problem the measurements could not be resumed before the beginning of 2005. The spectral resolution is reduced to about 40% of the initial value. MIPAS is operated with approximately 50% duty cycle in this reduced resolution mode since then. Since the instrument’s condition seems to be good, some further years of MIPAS operation can be expected.

Global Long-Term MIPAS Processing

521

3 Data Analysis Retrieval of atmospheric constituents abundances requires the inverse solution of the radiative transfer equation. Measured radiances are compared to radiative transfer calculations and residuals are minimized in a least squares sense by adjustment of the constituents abundances which are fed into a forward model. The Retrieval Control Program (RCP), together with the forward model, is the core of the processing chain. The forward model, the Karlsruhe Optimized and Precise Radiative Transfer Algorithm (KOPRA) [4] is a computationally optimized line-by-line model which simulates radiative transfer through the Earth’s atmosphere under consideration of all relevant physics: the spectral transitions of all involved molecules, atmospheric refraction, line-coupling and non-local thermodynamic equilibrium. Along with the spectral radiances, KOPRA also provides the Jacobian matrices, i.e. the sensitivities of the spectral radiances to changes in the atmospheric state parameters. Except for special applications, horizontal homogeneity of the atmosphere is assumed. Atmospheric state parameters are retrieved by constrained multi-parameter non-linear least-squares fitting of simulated and measured spectral radiances. The inversion is regularized with a Tikhonov-type constraint which minimizes the first order finite diﬀerences of adjacent profile values [5]. Instead of simultaneous retrieval of all target parameters from a limb sequence, the retrieval is decomposed in a sequence of analysis steps. First, spectral shift is corrected, then the instrument pointing is corrected along with a retrieval of temperature, using CO2 transitions. Then the dominant emitters in the infrared spectrum are analyzed one after the other (H2 O, O3 , HNO3 ...), each in a dedicated spectral region where the spectrum contains maximum information on the target species but least interference by non-target species. Finally, the minor contributors are analyzed, whereby pre-determined information on the major contributors is used. The analysis is done limb sequence by limb sequence, while typically several thousands of limb sequences are collected in one run. The inner loop is over the limb sequences, while the outer loop is over the target parameters.

4 Examples of Scientific Projects 4.1 The ACHEDYRE Project The ACHEDYRE project (Advances in atmospheric CHemistry and DYnamics REsearch by development of coupled chemistry-dynamics data assimilation models), funded by the European Space Agency and coordinated by Environment Canada, aims at a detailed study of the possible benefits and diﬃculties associated with the assimilation of chemical and dynamical observations into the same data assimilation system. The employed data assimilation system is

522

M. Kiefer, U. Grabowski, H. Fischer

an extension of the system used operationally by the Meteorological Service of Canada. The observations of stratospheric chemistry come mainly from the MIPAS instrument aboard Envisat. Two level 2 datasets are assimilated: the ESA oﬄine dataset delivered by the operational algorithm, and the scienceoriented dataset obtained by the IMK algorithm. The latter dataset was delivered by IMK specifically for this project. Compared with the ESA dataset, it has a much more detailed characterization of error budgets and vertical resolution, contains ClONO2 retrievals which are extremely useful for ozone depletion studies, and compares better with output from the free-running model, especially for temperature and ozone (pers. comm. S. Chabrillat). At IMK three data sets were processed: The first one covers the time from 11th to 25th of August 2003 (14 days), the second covers 26th of August to 9th of September 2003, and the last one 3rd to 9th of October 2003. Since contiguous data was requested every single day of the time periods was processed. For two orbits, 7600 and 8415, which are approximately two months apart, results for temperature and ClONO2 are shown in Figs. 1 and 2. Chlorine nitrate (ClONO2 ) is a major temporary reservoir gas of chlorine in the atmoT,KANADA_all, orbit 7600, 20030814, 0018 - 0157 UT 80

K

Altitude [km]

280 60

260 240

40

220 20

200 180 0

100 200 relative Latitude [deg]

300

Altitude [km]

ClONO2,KANADA_all, orbit 7600, 20030814, 0018 - 0157 UT ppmv 50

0.0025

40

0.0020 0.0015

30

0.0010 20 0.0005 10

0.0000 0

100 200 relative Latitude [deg]

300

Fig. 1. Temperature (top) and chlorine nitrate volume mixing ratio distributions over height and latitude for orbit 7600 (14th of August 2003)

Global Long-Term MIPAS Processing T,KANADA_all, orbit 8415, 20031009, 2246 - 0025 UT 80

523

K

Altitude [km]

280 60

260 240

40

220 20

200 180 0

100 200 relative Latitude [deg]

300

Altitude [km]

ClONO2,KANADA_all, orbit 8415, 20031009, 2246 - 0025 UT ppmv 50

0.0025

40

0.0020 0.0015

30

0.0010 20 0.0005 10

0.0000 0

100 200 relative Latitude [deg]

300

Fig. 2. Same as Fig. 1 but for orbit 8415 (9th of October) i.e. approximately two months later. The relative latitude has its zero set to the equator on the night side (10:00 pm). The north pole is at 90 degrees while the south pole is reached at 270 degrees. Note that the vertical scales of the two plots are diﬀerent

sphere and therefore plays an important role in the process of ozone depletion during the polar winter (see e.g. [3]). From both figures it is well visible that there is a very cold region at the south pole below 30 km. This region belongs to the so-called polar vortex, which is the region where most of the ozone destruction takes place. The time series of such data allows, e.g., to gain insight into the detailed processes which are involved. 4.2 The QUALM Project The general objective of this project is to contribute to the assessment of the global impact of pollutants’ emissions from biomass burning sites and megacities in terms of uplifting to the upper troposphere, outflow from emission regions, intercontinental transport, the impact on upper tropospheric chemistry, and intrusion into the lower stratosphere. The observational data base will be formed by the following trace species: O3 , CO, CH4 , NH3 , C2 H6 , C2 H2 , PAN, HNO3 , HNO4 , H2 CO, N2 O, methanol, formic acid, acetone, HCN, etc..

524

M. Kiefer, U. Grabowski, H. Fischer

While some of these data products are already available, others require dedicated processing as part of this project. The data base will allow to • contribute to the detection of emission sources; • contribute to the detection of pollution plumes in the upper troposphere; • distinguish megacity origin from other origin by the composition of the plume; • study aging (chemical transformation) within the plumes; • study transport ways from the upper troposphere to the lower stratosphere. For this project a longer time span without major gaps and more minor species is necessary, compared to the ACHEDYRE project. Since computation time still is prohibitive (at least if data is expected to be available within a finite time), a coverage of approximately 3 days per week in the time range from September to November 2003 is realized initially. If analysis of results will indicate that the temporal resolution is not suﬃcient, further dates will be processed to fill the gaps.

5 Computational Considerations 5.1 Overview The retrieval of atmospheric trace constituents is performed for each geolocation separately. A geolocation is determined by the mean values of the geographic coordinates and of the times of the spectral radiance measurements of the corresponding limb scans. ESA delivers L1b data as binary files with one file containing spectra plus additional information for all geolocations of one Envisat orbit. The computational steps consist of: • preprocessing with single steps performed by 8 diﬀerent programs; always performed at IMK • retrieval (core processing step) with the Retrieval Control Program (RCP); performed either at IMK or on the XC supercomputer • postprocessing with a diagnostics program; performed either at IMK or on the XC. 5.2 Preprocessing As already outlined in Sect. 3 the input needed for the retrieval of temperature and trace gases depends on the specific retrieval target. A dedicated preprocessing tool is used to generate the retrieval setup individually. There are the following steps performed by this tool:

Global Long-Term MIPAS Processing

• •

• •

• • • •

525

Temperature and pressure profiles from ECMWF analysis data are extracted. These are needed as initial guess and a-priori in the temperature and LOS retrievals. Valid spectra are selected. In this step all spectra are rejected if they do not match certain criteria, e.g. no corruption marks were assigned in the level-0 processing by ESA, suﬃciently low noise values, no clouds in the corresponding field of view, etc.. The directory tree for the processing is built up. Initial guess and a-priori profiles for the retrieval target and profiles of interfering species are collected. If there are already retrieval results, which are generally considered to be superior in quality over climatology values, they are ingested from the database of results of preceding retrieval results. An adequate occupation matrix is determined and selected from the corresponding database. The occupation matrix contains altitude specific information about the spectral ranges best suited to perform the retrieval. The input file for the forward model is generated. Input files for the retrieval control program are generated. Some regularization matrices are generated. These are necessary because in the IMK implementation, with the retrieval performed on an over-sampling height grid, the inversion is an underdetermined problem.

Previously, the spectral covariance matrix was generated by the preprocessing tool. Since the matrix is huge (approximately 5 MB per geolocation) for typical retrievals, and had the lion’s share (98%) of the data to be transferred to the XC, the matrix generation has been built into the retrieval control program. During the preparatory steps, extensive use of large databases of profiles, spectral data, and occupation matrices is made. This prevents the preprocessing tool to be run on the XC computer. Therefore, all preparation work is done at the IMK facilities. The preparatory steps are performed such that the resulting data/directory structure and files are suited for either retrieval processing on the IMK facilities or on the XC with the little exception that processing on the XC requires the change of some paths in the input. 5.3 Adaptation of the Core Processing Tools to the XC Supercomputers and Optimization Overview The main processing tool, the Retrieval Control Program (RCP), together with the forward model KOPRA, which is linked as a library, had to be adapted to the new environment. The program is written in full featured Fortran 95 and has to run on several diﬀerent operating systems and compilers,

526

M. Kiefer, U. Grabowski, H. Fischer

so as little change as necessary was the target of the adaptation task. A special Makefile/Dependency Generator hides the diﬀerences of the OS and compilers from the user. Since the XC environment mandates the use of MPI, and the RCP (especially its main part, the forward model) is an inherently sequential program, and several of the other environments do not support MPI, we decided to implement MPI on the highest level, that is, parallelizing on geolocations. This only required to change the main control routine to a subroutine and writing a new main driver controlling the distribution of a given group of geolocations to several processors. Since then all processors are working at full load, we did not activate OpenMP, which would have implied additional overhead with no eﬀective gain in processing power. With this implementation, we have almost no communication at all between processors or nodes, since all data processed is local to the geolocation. Nevertheless, several obstacles in the XC environment complicated this task. Compilation and Optimization The RCP/KOPRA routines consists of aprox. 150 000 lines of code which heavily uses the new Fortran 95 features like modules, pointers, allocatable arrays, array syntax, etc.. Only a small subset of marketed compilers are able to compile this (standard conforming!) code correctly, not to speak about optimizing it. From the available compilers on the XCs, only the Intel 9.1 compiler (which was delivered in September 2006, 6 months after project approval), was able to at least compile without flaws. Still this version and followups have problems with memory usage, optimization, and compile time. The time to compile the forward model is 14 hours, and it uses over 3.5 GB of main memory for most of the modules, compiling only module by module, not the whole project at once (which would be impossible). Despite of this time, the most processing time consuming modules cannot be optimized due to internal resource limitations of the Intel compiler. This problem had already been reported to the manufacturer three years ago at another try with an early version of the Intel compiler on another platform, but could not be resolved (and probably cannot be resolved) due to a principal problem with the internal compiler architecture. Compared to this special brand of compiler, the Compaq Alpha Tru64Unix compiler used on our inhouse system compiles the same code with highest optimization in less then 10 minutes using only 150 MB main memory, and compiles it correctly and best optimized for the given platform. Nevertheless, the resulting processing time on the XC1 for one geolocation was exactly comparable to our processors (64bit Alpha EV68 833 MHz), and the specifications of 530 MFlops for the floating point parts of the program (inferred from the total runtime of the modules

Global Long-Term MIPAS Processing

527

in comparison to profiled code on the Alphas, since compilation with profiling is not possible on the XCs due to compiler bugs) where reached on both systems. Since it was already decided to retire the XC1 platform, we did not bother with further optimization workarounds for the Intel compiler on the XC2 (since first tests showed just the same problems as on the XC1), but decided to go with an imported Sun amd64 Linux StudioExpress compiler, which already delivered good results on our new inhouse Solaris 10 amd64 Opteron machines. The new compiler is able to compile the whole toolchain in about 20 minutes with high optimization (although profiling currently does not work also) with less than 1 GB main memory consumption, and the code now runs more than twice the speed compared to the XC1, using the internal Sun performance libraries for BLAS and Lapack, not the ACML from AMD, which crashes in one particular routine due to unknown reasons (but the time consumed in these libraries is only a small fraction of our total runtime). We therefore can recommend that compiler for general systemwide use. Subproject Turnaround Time, Runtime, and Job Limits The maximum number of jobs in queue per user (6 on the XC1), together with the maximum running time per job of approx. 4 days and the maximum number of 32 CPUs practically attainable at the same time on the XC1 limited the eﬀective throughput, compared to the approved CPU time of 1.54 Mhours/4 years. The full time CPU allocation needed for the project to consume that amount is 1056 hours/day or 44 processors, not counted downtime, transfer and postprocessing time, etc. The problem was leveraged with the commissioning of the XC2, which (currently) accounts 320 processors with 10 jobs in parallel for our project, which shifts the bottleneck for the turnaround time for one subproject to somewhere else (see Sect. 5.4). A typical subproject for one target gas consists of a set of about 25000 geolocations. Since the maximum runtime divided by the runtime per geolocation multiplied by the amount of available cores and maximum allowed number of jobs in the queue is smaller for several targets than what must be handled to process the whole subproject, we had to develop our own job distributor to work around these (rather artificial, and, from a technical viewpoint, unnecessary) job limits to avoid the need for manual control (which would not be possible, e.g., at weekends). This scheduler distributes the geolocations evenly on the available jobs (so that they all end approximately at the same time), accumulating the maximum amount of individual MPI runs per job to fill up the maximum runtime based on the expected maximum runtime per target gas. It would be strongly advisable to lift the

528

M. Kiefer, U. Grabowski, H. Fischer

restriction of having a limited number of queued jobs, since the system scheduler already takes care of the jobs that can be started per user at the same time, and since the user anyway will queue jobs until the work which has to be done is actually done, nothing is gained by having this restriction. It only burdens the user with additional, really unnecessary management tasks and has no eﬀect on the system load. As an example, on our inhouse Alpha Cluster (32 cores), we have, typically, 25000–40000 jobs in the queue, which are automatically processed in 1–2 weeks without any need for user interaction. Although the total amount of actually available CPU time is much higher on the XC2, the constraints due to memory limitations are harder to handle since the new blades only have 4 GB available per core, compared to 6 GB per core on the XC1. Our memory usage for the main production line ranges from 1 GB to about 8 GB, depending on the target gas processed, which narrows the set of gases which can be computed with all cores loaded more on the XC2 than on the XC1, although a higher amount of memory had to be allocated (400 MB) exclusively for the MPI communication on the XC1 (without actually using it) to simply start the program than on the XC2. Since allocatable arrays in Fortran are typically allocated on the stack, a significant increase of stack space (around 1 GB) is also necessary to successfully run the code. On the XC1 there were 8 core machines to handle larger jobs, which we are missing on the XC2, so a couple of memory intensive jobs now have to be processed on our own system again. It would be advisable to additionally provide a smaller set of machines which have at least 8 GB available memory per core, which could be separated from the others by a special queue. Table 1 gives an overview on the average expected runtimes and the physical and virtual memory usage, respectively, for selected species. Table 1. Average expected runtimes and the physical and virtual memory usage, respectively, for selected species (only XC2) target N2 O, CH4 ClO ClONO2 H2 O HNO3 N 2 O5 HOCl

expected runtime/ geolocation [min] 7 10 2 7 7 7 55

memory usage phys [MB] virt [MB] 1916 1874 510 1408 1412 1331 2112

8817 8735 7361 3114 8290 8209 4354

Global Long-Term MIPAS Processing

529

5.4 Postprocessing Besides the result data RCP generates some further data files, which are necessary to perform a diagnostics of the result. After the diagnostics program has finished most of the files can be removed. In an early version of the processing all results and associated data files were transferred back to IMK and the diagnostics software was run there. Some of the data files, e.g. the full error covariance matrix, are very large, and this led to a serious performance degradation of the entire processing due to long transfer times. In a next step, the diagnostics tool was adapted to the XC2. This much reduces the size of the files to be transferred. Unfortunately, it turned out that the gain was marginal at best, because the diagnostics tool showed a bad performance on the XC2. The reason is that file operations (like e.g. the UNIX command ls) seem to be very ineﬃcient there. An optimization with respect to this aspect resulted in a three times faster program, simply by collecting and reducing file system access operations in the code. Currently further optimization of the whole postprocessing is underway, since it meanwhile is a major part of the whole process (see 5.3). The current status is shown in Figs. 3 and 4 in terms of cumulative times per orbit. Per orbit there is a set of 4 time periods, namely: (1) runtime of diagnostics program, (2) time for building and compressing one archive file of all result and diagnostics files of the orbit, (3) time for transfer of this archive file from XC back to IMK computers, (4) time for decompressing and inflating of the archive file on the IMK computers. Shown are the cumulative durations of the sub-processes (1)–(4) at diﬀerent start times of the postprocessing. The two figures show that there obviously are good times and bad times for postprocessing. Orbits with N2 O5 data of the QUALM project, with compressed archive file sizes of approx. 41 MB per orbit, took about 6–8 minutes for all postprocessing steps (see Fig. 3). The single times are divided roughly as follows: half the time for diagnostics generation, one quarter of the time for building and compressing of the archive file, one quarter for transfer to IMK and archive inflation. On the other hand, the postprocessing of ClONO2 of the QUALM project (compressed archive files have sizes of 19.5 MB) took 6–250 minutes, with many orbits needing more than 15 minutes. It can be clearly seen that the major contributors are steps (1) and (2), both performed on the XC2.

530

M. Kiefer, U. Grabowski, H. Fischer

Fig. 3. Times per orbit, as explained in detail in the text, for N2O5 of the QUALM project. Shown are two out of three transfer logs (the whole project was split into three parts). The ﬁles have a size of approx. 41 MB per orbit

Global Long-Term MIPAS Processing

531

Fig. 4. Times per orbit, as explained in detail in the text, for ClONO2 of one half of the QUALM project. Both plots show the same data but with diﬀerent ordinate scaling. The ﬁles have a size of approx. 19.5 MB per orbit

532

M. Kiefer, U. Grabowski, H. Fischer

6 Conclusions and Outlook After an initial delay due to compiler problems, the XC supercomputer is used since July of 2006 for MIPAS data processing. The first project, which was partly processed at the XC1, was part of a monitoring data set which covers dates from October 2003 through March 2004. The next data set which could be processed successfully on the XC1 was the contribution of IMK to the ACHEDYRE project (see Sect. 4.1). After the installation of the new XC2, IMK started to process data for the QUALM project (see Sect. 4.2). The fact that the XC2 features much more nodes than the XC1 led to a significant performance increase for MIPAS data processing. Currently, the postprocessing of the data takes the greatest part of the project turnaround time. Some preparations for more eﬃcient postprocessing are already implemented (see Sect. 5.4). Further significant improvements will be made operational within the next months. After implementation and testing of the optimization steps, there will be set up a strategy for the processing of as much MIPAS data as possible. The details will have to be determined on base of the availability and anticipated throughput of both, IMK facilities (necessary for at least the preprocessing) and XC2 supercomputer.

References 1. Endemann, M. (1999). MIPAS instrument concept and performance. In Proceedings of European Symposium on Atmospheric Measurements from Space, Noordwijk, Netherlands, 18–22 January, pages 29–43. European Space Agency, ESTEC, Noordwijk, The Netherlands. 2. Fischer, H., Blom, C., Oelhaf, H., Carli, B., Carlotti, M., Delbouille, L., Ehhalt, D., Flaud, J.-M., Isaksen, I., L´opez-Puertas, M., McElroy, C. T., and Zander, R. (2000). Envisat-MIPAS, an instrument for atmospheric chemistry and climate research. European Space Agency-Report SP-1229, C. Readings and R.A. Harris (eds.), ESA Publications Division, ESTEC, P. O. Box 299, 2200 AG Noordwijk, The Netherlands. 3. Solomon, S. (1999). Stratospheric ozone depletion: A review of concepts and history. Rev. Geophys., 37(23):275–315. 4. Stiller, G. P., editor (2000). The Karlsruhe Optimized and Precise Radiative Transfer Algorithm (KOPRA), volume FZKA 6487 of Wissenschaftliche Berichte. Forschungszentrum Karlsruhe. 5. von Clarmann, T., Glatthor, N., Grabowski, U., H¨opfner, M., Kellmann, S., Kiefer, M., Linden, A., Mengistu Tsidu, G., Milz, M., Steck, T., Stiller, G. P., Wang, D. Y., Fischer, H., Funke, B., Gil-L´opez, S., and L´ opez-Puertas, M. (2003). Retrieval of temperature and tangent altitude pointing from limb emission spectra recorded from space by the Michelson Interferometer for Passive Atmospheric Sounding (MIPAS). J. Geophys. Res., 108(D23).

Modelling the Regional Climate of Southwest Germany: Sensitivity to Simulation Setup C. Meissner and G. Schaedler Institute for Meteorology and Climate Research, Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany [email protected]

1 Introduction Already a quick glance at regional climatic maps, for instance the ones prepared for the State of Baden-Wuerttemberg [1], shows that near surface temperature, wind and precipitation can vary strongly over distances of about ten kilometres and less. In order to assess the regional impact of climatic change, we therefore need information at such small spatial scales. The only way to obtain information on the future climate is by using global climate models. Their resolution, being in the order of more than 100 km even for the most recent models (the presently used ECHAM5 model at the Max-Planck-Institut for Meteorology has a resolution of about 150 km in our latitudes), however, is much too coarse for impact studies, lest for detailed planning purposes. At such resolutions, terrain height is smoothed resulting in over-/ underestimation of valley/mountain heights (e.g. in the Black Forest) by several 100 m; this can result in large errors for quantities which are closely related to terrain height, like temperature, precipitation and wind. Furthermore, subgrid-scale features like urban areas, valleys and mountains which can have considerable influence on regional climate are not taken into account. The same is true for subgrid-scale climatically relevant processes, like convective summer precipitation (thunderstorms), which have to be parameterised. This problem is similar to the closure problem in turbulence, perhaps with the diﬀerence that the cloud physical processes involved are less known and the parameterisations are even more uncertain. For these reasons, simulations with a much finer resolution than global models oﬀer are necessary. To keep such simulations feasible, we apply here the so-called dynamical downscaling, where the global model data are used to provide boundary and initial conditions for the high resolution regional model. Up to now, standard resolutions in regional modelling were in the order of 20 to 50 km; recently, runs covering Central Europe at 10 km resolution have been finished. In the work presented here, we use resolutions of 7 km and 14 km for Southern Germany. As already mentioned, the ultimate goal

534

C. Meissner, G. Schaedler

of most climate studies are prognoses of the future climate and subsequent impact studies. However, to have confidence in such prognoses, it is necessary to see how well the climate of the recent past can be reproduced by the model and to attach error bars to the results. There are diﬀerent types of sources for diﬀerences between model results. First, external factors which are not under the control of the regional model: it is known that regional model results depend very much on the driving data (see for example the contribution by D. Gyalistras [2]), so that errors in the global model predictions are transferred (and sometimes even amplified) to the regional scale. Similarly, there may be errors in the terrain, soil and vegetation databases used by the model. Second, there are model to model diﬀerences due to diﬀerent physical and numerical model structures and, third, there are diﬀerences within simulations using the same model due to domain choice, horizontal and vertical resolution, time step, selected physical parameterisations etc. Some issues concerning regional modelling are discussed in more detail in [3]. Besides factors likely to produce unsatisfying results, like too coarse resolution or systematic errors in the driving data, some factors may produce perhaps better results for one variable and worse results for the other, so that no single combination can be considered the best; therefore simulation results should be presented together with a quality assessment for the variables involved. This means, basically, that it is necessary to perform ensemble simulations and to evaluate the results statistically. Such ensembles can be built by using diﬀerent driving data, by perturbing the initial and/or boundary data and by using diﬀerent parameterisations. Obviously, this way to assess the quality of climate simulations on time scales of decades requires the computational power of centres like HLRS. As a first step towards such ensemble simulations, we studied the sensitivity of the climate simulation results with respect to the factors just mentioned. The model we used was the climate version CLM of the operational weather forecast model (LM) of the German Weather Service (DWD). In Sect. 4, we will present some results concerning the impact of driving data, grid size and model initialisation and comparisons with observations. Up to now, such studies on the climatic time scale are quite scarce. A quantity of prominent interest is the temperature trend; in Sect. 4, we present a comparison of observed and modelled trends during the last decade of the 20th century. In Sects. 2 and 3, we will describe the CLM regional climate model and the setup of the numerical experiments. Section 5 will give some information on the performance of the model on the NEC SX-8. Finally, we will present our conclusions and an outlook.

2 Model History and Description Since December 1999, LM (Lokal-Modell) is the operational forecast model of the German Weather Service (DWD). Around 2001, it was suggested to use

Modelling the Regional Climate of Southwest Germany

535

this non-hydrostatic model also for regional climate forecasts, other regional climate models used at the time in Germany being mainly the hydrostatic (and hence limited in resolution) model REMO (also derived from a previous weather forecast model of the DWD) and the non-hydrostatic model MM5. A consortium was set up to foster the development of the CLM (Climate LM). For some years the development of CLM lagged behind the development of LM, the former being based on older versions of LM. In 2006, an eﬀort was made to synchronise the two models with the aid of DWD, resulting in the common version 3.21, which can be used in weather forecast as well as in climate mode. More information about the model, its structure and an evaluation of long-term simulations can be found in the COSMO Newsletters (http://www.cosmo-model.org), especially No. 2 and No. 6. The CLM model is a three-dimensional non-hydrostatic model using the unfiltered prognostic equations for the three wind components, pressure, temperature and the gaseous, liquid and solid water phase [4]. The soil layer is described by prognostic equations for soil temperature and soil water content. The atmospheric equations are completed by the ideal gas equation and parameterisations for radiation, cloud and precipitation processes, turbulence, soil processes and the exchange of energy and water between the Earth′ s surface and the atmosphere. For most of these processes, several alternative parameterisations can be chosen (e.g. turbulent fluxes from flux-gradient relationships or from prognostic calculation of TKE). The model has its own databases for external data like orography, land-sea mask, soil type and vegetation cover. Numerically, the prognostic equations can be solved either by leapfrog or Runge-Kutta methods with operator splitting for the fast (sound waves) and slow processes. Standard resolutions are 14, 7, and 2.8 km with time steps between 60 s and 7 s. An interesting feature of the model is its rotated coordinate system: typical applications of the model cover areas with size of a few million square kilometres (e.g. the whole of Europe), which makes it necessary to take the curvature of the Earth into account. Therefore, the model equations are written in spherical coordinates. However, when using spherical coordinates, two problems arise: one is the “pole problem”, which means that the geographical poles represent a singularity due to the convergence of the meridians and that therefore special measures have to be taken when a geographical pole lies within the simulation domain; more often however, the second problem, also connected with the convergence of the meridians, is encountered, namely varying horizontal resolution with latitude away from the equator. An elegant way to avoid both problems is realised in CLM: the rotated grid. The computational spherical coordinate system is rotated in such a way that the intersection of the equator and the prime meridian of the new system passes through the centre of the simulation domain, thus avoiding the pole problem and providing minimal convergence of the meridians at the same time. The necessary coordinate transformations are performed during pre- and postprocessing.

536

C. Meissner, G. Schaedler

The model is written in Fortran 90, making extensive use of the modular structures provided by this language. Code parallelisation is done via MPI (message passing interface) on distributed memory machines using horizontal domain decomposition with a 2-grid halo. The model is running on various platforms, including Windows PCs, LINUX clusters, vector computers like NEC SX and Fujitsu VPP series as well as MPP machines like IBM-SP and SGI ORIGIN.

3 Simulation Setup and the Downscaling Chain As already mentioned, there are several methods to “downscale” large scale information to the regional scale. We use here the so-called dynamical downscaling, which is basically a nesting of the regional model into large-scale (observational or simulated) data. This means that the model is given largescale information at its boundaries at regular time intervals. At the lateral and top atmospheric boundaries, this information consists of prognostic meteorological data like pressure, wind, temperature and humidity; the lower boundary is the bottom of the soil layer in about 15 m depth, where constant (“climatological”) values of temperature and soil moisture are prescribed. All other quantities in the atmosphere and the soil layer are calculated by the model. All prognostic fields are initialised once using interpolated large-scale data, and the updating of the prognostic atmospheric boundary data takes place every 6 hours (with linear interpolation in between). Soil temperature and moisture are not updated externally. Presently, interfaces to the large scale driving data sets presented in Table 1 are available. Except for the grid size sensitivity studies, the simulations presented here were driven by ERA-40 and NCEP reanalyses. Reanalysis data are results of simulations with global models in which the model results are continuously corrected by observational data (data assimilation). One could speak of an optimal, physically consistent interpolation of observational data onto a regular grid by using a meteorological model. This has the advantage that this data set is close to reality, making it possible to assess the capability of the Table 1. Available large scale driving data sets for CLM. ECHAM5 is the global model operated by the Max-Planck-Institut for Meteorology, Hamburg, and GME is DWD′ s global weather forecast model Name

Approx. Time covered Type resolution (km)

Suitable for prognoses of future climate

ERA-40 NCEP ECHAM5 GME

125 210 150 60

No No Yes No

1950–2002 1948–present 1960–2100 1999–present

Reanalyses Reanalyses Climate runs Analyses

Modelling the Regional Climate of Southwest Germany

537

Fig. 1. Simulation domain for the 50 km runs (black rectangle). The red rectangle in the picture indicates the domain for the nested runs with 7 and 14 km. The blue rectangle in the picture indicates the domain of investigation (in the following often called “small domain”)

regional model to simulate the climate right if driven with adequate data. Nevertheless, diﬀerent reanalyses show biases compared to observations and the results depend on the weather prediction model used. To detect the impact of these diﬀerences on the CLM results, it is necessary to use more than one data set to drive the CLM. The most commonly used reanalysis data sets are the ERA-40 [5] and NCEP-DOE AMIP-II [6] reanalysis data which we also have used for this comparison. We use here a double nesting technique, first from the reanalyses to a 50 km grid using CLM, and then from this grid to a 14 km resp. 7 km grid, again using CLM. The domain we used for our simulations covers all of Europe without Scandinavia for the 50 km runs and Central Europe for the high resolution (7 km resp. 14 km) runs, and our main region of interest is Southwestern Germany (blue frame in Fig. 1). Due to the influence of the Alps it is not possible to restrict the high resolution simulations to this region; rather, we have to simulate a larger region which includes the alpine region (red frame in Fig. 1). The simulations were run from 1988 until 2002 with 60 × 81 grid points in the horizontal for the 50 km runs and from 1999 until 2001 with 176 × 128 / 88 × 64 grid points for the 7 km / 14 km runs. The number of vertical levels was 35 in both runs. Output fields including precipitation, temperature, soil moisture, wind and energy balance components are written hourly. Depending on the resolution, this amounts to about 1 MB of data per write process.

4 Results In this section, we will a) discuss the impact of driving data, grid size (7 km vs. 14 km) and initialisation on the simulation results, and b) present a comparison of observed and modelled trends of the near surface temperature between 1991 and 2000. The observational data were kindly provided by DWD and the Potsdam Institut fuer Klimafolgenforschung (PIK).

538

C. Meissner, G. Schaedler

4.1 Inﬂuence of Driving Data Using the double nesting technique described above, there are three possible sources of errors in the runs of the small model domain (7 km and 14 km): 1. The original reanalysis data sets can diﬀer from the observational data and can cause spurious results of the 50 km simulations 2. The CLM simulation with 50 km can produce unrealistic results even if the driving reanalysis data is correct and therefore we get spurious driving data for the small model domain 3. The 50 km simulations give realistic driving data for the small model domain, but the small domain simulations give spurious results The most comprehensive comparison would be a comparison of the threedimensional meteorological fields, but there are not enough observations for such comparisons. Therefore usually climatologically relevant derived variables like near surface temperature, humidity and precipitation are compared. In the following, we will focus on a comparison of near surface temperature (this is defined as the temperature at 2 m height) and precipitation. More detailed studies for precipitation are under way. Since the domain for the 50 km run has its boundaries over the Atlantic Ocean in the west and over North Africa in the south where observations are quite scarce, a comparison with ground based measurements is not possible in these areas. Therefore, we restrict the comparison to the land areas and compare with the data set from Willmott and Matsuura version 1.02 (2001). Figure 2 (top) shows the diﬀerences in the near surface temperature between the two reanalysis data sets. In the small model domain we get diﬀerences up to 1 K for the annual mean near surface temperature. A comparison with measurements (Fig. 2, bottom) shows diﬀerences up to 2 K in the small model domain, but we should keep in mind that we compare data sets with diﬀerent horizontal resolutions; also, some diﬀerences in temperature can be explained by diﬀerent terrain heights. Nevertheless, it can be stated that there are diﬀerences between the reanalysis data which can lead to diﬀerent simulation results. As for the second source of errors mentioned above, Fig. 3 shows the temperature diﬀerences between the measurements and the simulations with 50 km resolution driven by ERA-40 (bottom left) and NCEP (bottom right) data. Large negative diﬀerences can be seen over major mountain ranges like the Alps. This may be due to the diﬀerences in terrain height between the real and the model topographies. Comparing the 50 km simulation results driven by ERA-40 and NCEP (Fig. 3, top), we can see that the temperature diﬀerence is in the order of 0.5 K in our region of interest. That means that for the small model domain the diﬀerences between driving data and observations on the one hand and between the reanalyses themselves are of similar magnitude. The ERA-40 simulation is colder and produces less rain (not shown) in the small model domain than the simulation with NCEP. We can therefore state that the driving data sets for the small model domain which are derived

Modelling the Regional Climate of Southwest Germany

539

Fig. 2. Annual mean near surface temperature diﬀerence averaged over the period 1990–1999: diﬀerence between NCEP and ERA40 reanalysis (top), ERA-40 and measurements (bottom, left) and NCEP and measurements (bottom, right). The red rectangle indicates the area of investigation

Fig. 3. Annual mean near surface temperature diﬀerence averaged over the period 1990–1999: diﬀerence between 50 km simulations with NCEP and ERA40 (top), ERA-40 and measurements (bottom, left) and NCEP and measurements (bottom, right)

540

C. Meissner, G. Schaedler

Fig. 4. Comparison of annual mean near surface temperatures for the 7 km run driven with ERA-40 and NCEP reanalysis data and observations for 2001

from the 50 km resolution run already show diﬀerences to the measurement data and we therefore expect the 7 km runs to show similar diﬀerences, too. At the 7 km resolution, we can use data from observation stations for comparison. Fig. 4.1 shows a comparison of the annual mean temperatures between simulations driven by the two reanalysis data sets and observations. Since the simulations are very time consuming, only the year 2001 has been simulated up to now with a spin up period of 2 years for both reanalysis data sets. The absolute diﬀerences between the observations and the 7 km simulations increase compared to the diﬀerences between the measurements and the 50 km simulations. Diﬀerences in temperature can now be more than 2 K; although these diﬀerences may partly be due to the higher local variability of the observations, a systematic underestimation by the simulations can be noted. The diﬀerences between the 7 km simulations driven by ERA-40 and NCEP has a lower range as in the 50 km runs and a diﬀerent sign. For the year

Fig. 5. Comparison of the 7 km runs driven with ERA-40 and NCEP reanalysis data and measurements for monthly mean near surface temperature of the year 2001

Modelling the Regional Climate of Southwest Germany

541

2001 the ERA40 annual mean temperature is warmer then the NCEP temperature. The diﬀerences between the two simulations are lower than 0.3 K. 15 years runs with 7 km resolution to be finished soon will show if these differences occur also over the decade. Looking at the annual cycle (Fig. 5), we see that the typical characteristics are captured well; the largest diﬀerences to the measurements occur in winter and early summer. There is also a close connection between temperature and precipitation; since precipitation is overestimated (not shown here), the near surface temperature is underestimated by the 7 km simulations as compared to the observations. 4.2 Inﬂuence of Grid Size Studies on the influence of grid size have been performed using DWD′ s global model GME analysis data as driving data for the 7 km model domain. The advantage of the use of GME data is the higher resolution of the global model and therefore the possibility to drive the 7 km domain without nesting. Because the availability of suitable GME data is restricted to the year 2000 and later, we made the comparisons for the year 2001. The diﬀerences in near surface temperature are quite small, but there are considerable diﬀerences in precipitation between the simulations with diﬀerent grid sizes (Fig. 6). The 7 km results agree much better with the measurements than the 14 km results do. For the calculation of the water balance and the hydrological cycle, the correct simulation of precipitation is very important. Therefore we suggest taking 7 km grid size resolution for further studies. However, further studies with nested simulations and for longer time scales have to be carried out. 4.3 Inﬂuence of Initialisation The third aspect which aﬀects the model results besides driving data and model resolution is the initialisation of the model. The atmospheric initialisation is not as important as the soil initialisation because the memory of the soil lasts much longer. The soil initialisation can therefore exert an influence on the simulations over many years. The initialisation of the soil temperature and the soil water content which is taken over from the reanalysis data is rather inadequate. The driving data often contain less soil layers than the CLM model and also the soil types which influence water and heat conductivity can diﬀer between driving model and regional model. This implies that the profile of soil water content and soil temperature taken over from the driving data can diﬀer considerably from the profile which would be produced by CLM itself for that grid point. In literature we can find several initialisation methods which are recommended for soil temperature and water initialisation. One method is to use climatological values [7]. Such a data set was provided by the ETH Zuerich for the 50 km grid, where the CLM was run with ERA-40 reanalysis data over a period of 40 years. From this data the climatological mean for every day of the year was calculated. To initialize our simulation we extracted

542

C. Meissner, G. Schaedler

Fig. 6. Comparison of modelled annual precipitation (left) and temperature (right) for the year 2001 with 14 km (top) and 7 km (middle) grid size and measurement data for Germany from the DWD measurement stations (bottom)

the climatological values at our initialisation day and replaced the ERA-40 values by them. Since the soil is expected to need some years for adaptation we conducted two simulations from 1988 to 2001: one with the original ERA40 initialisation and one with the climatological initialisation. As result we can state that the soil has nearly reached a steady state in both simulations after two years when the two simulations show only small diﬀerences in the upper soil levels (Fig. 7). Due to the fixed lower temperature the diﬀerences in the deep soil layers do not vanish. It is not possible to determine which of the initialisations gives better results because the diﬀerences in annual mean near surface temperature are smaller than measurement accuracy (Fig. 8). Nevertheless we recommend using the climatological values for initialisation for consistency reasons. 4.4 Comparison of Observed and Modelled Trends of the Near Surface Temperature Between 1991 and 2000 In this section we compare observed and modelled trends of the near surface temperature for the decade 1991 to 2000 in Southwestern Germany. Figure 9

Modelling the Regional Climate of Southwest Germany

543

Fig. 7. Root Mean Square Diﬀerence of soil temperature (top) and soil moisture (bottom) between simulation with original initialisation and climatological initialisation for diﬀerent soil depth

Fig. 8. Diﬀerences of annual mean surface temperature for the year 1989 (left) and 2001 (right) between simulation with ERA-40 and climatological soil initialisation

shows a comparison of the modelled and the simulated trends of the near surface temperature for this decade. During this time period, a marked increase of the surface temperature could be noted at many sites. Due to the shortness of this time interval, however, the significance of these trends is lower than

544

C. Meissner, G. Schaedler

Fig. 9. Modelled (left) and observed (right) trends of near surface temperature between 1991 and 2000

for the trends between 1931 and 2000 [8]. The modelled trends presented here were computed using the 7 km domain with ERA-40 driving data. Figure 9 shows the modelled trends (left) together with the observed trends (right) as derived from observations. The trends were calculated from annual average temperatures using the least squares method, their significance was calculated with the Mann-Kendall test [9]. The modelled trends represent grid-point values, whereas the observed trends are derived from point (i.e. station) data. Ideally, either grid-point data should be mapped onto the model grid, or the model data should be interpolated to the stations. We found, however, that there are only small diﬀerences between this method and the direct comparison we present here, probably because temperature is quite a “smooth” variable and because of the relatively high model resolution. The orders of magnitude of the observed and the modelled trends agree quite well, both of them being mainly in the range between 0.1 and 0.9 K/decade. This is the same magnitude as for the time span between 1931 and 2000 and shows that a considerable part of the warming took place during the last decade of the 20th century. Observations and model agree also in that relatively strong warming occurred in the Rhine valley, the western slopes of the Black Forest and in the Kraichgau region, whereas trends and/or significances are smaller in the eastern parts of the Black Forest and extended portions of the Suebian Jura. On the other hand, there are also some diﬀerences: some of the observed warming trends in the east of the state of Baden-Wuerttemberg are not reproduced by the model and modelled trends are – to a varying degree – smaller than the observed ones. Since smaller trends result also in smaller significance, the significance of the modelled is smaller than the significance of the observations. It remains to be seen what impact ensemble simulations and simulations over longer time spans will have on the results.

Modelling the Regional Climate of Southwest Germany

545

5 Performance of the Model on the SX-8 During a typical run (not on the SX-8), roughly 95 percent of CPU time are spent for calculations, 4 percent for communication and 1 percent for input/output. For the calculations, about 63 percent of the time are spent for the dynamics, 31 percent for the various parameterisations and about 6 percent for other tasks [10]. Before using the NEC SX-8 we had run our simulations on an IBM Power 4 system. Between the two machines we had a speed up factor of 4 for the 7 km runs. Besides the speed up factor, the possibility to perform multiple simulations simultaneously on the NEC reduced the waiting time considerably. Porting the model to the NEC SX-8 at HLRS required adaptation of the run script to the HLRS environment, compilation of grib and netcdf libraries for the data input and changing of some source code from MPI 1 to MPI 2. Due to the small model domain with relatively few grid points there was no gain in running the code on several nodes. The results of our test simulations to decide about the number of CPUs are shown in Table 2. We obtained the best results by using 2 CPUs per simulation. Table 2. Performance of CLM with diﬀerent numbers of CPUs for one day of simulation time NPX NPY CPUs/24 h for 7 km run (176 × 128 × 35 grid points) 1 1 1 2 1 2

1 2 4 2 8 4

104 min 54 min 56 min 64 min 65 min 73 min

30 s 15 s 20 s 30 s 30 s

6 Conclusion and Outlook The aim of the study presented here was to identify a suitable model configuration for high resolution climate simulations. As an appropriate model configuration we identified a model setup with 7 km grid size, climatological soil initialisation and ERA-40 reanalysis data as driving data. With this configuration we will now perform ensemble simulations for the small simulation region. The ensemble will be created by using diﬀerent physical parameterisations available in the CLM. With the results of this ensemble it will be possible to quantify the uncertainty of the model results.

546

C. Meissner, G. Schaedler

7 Data Sets NCEP/DOE 2 Reanalysis data provided by the NOAA/OAR/ESRL PSD, Boulder, Colorado, USA, from http://www.cdc.noaa.gov ERA-40 data provided by the European Centre for Medium-Range Weather Forecasts, from http://www.ecmwf.int Willmott and Matsuura (2001): Terrestrial Air Temperature and Precipitation: Monthly and Annual Time Series (1950–1999), from http://climate.geog.udel.edu/climate Temperature and precipitation data from climate stations from DWD Climatological Soil Moisture and Soil Temperature Data Sets provided by Daniel Luethi from ETH Zuerich.

References 1. LUBW: Klimaatlas Baden-Wuerttemberg. Landesanstalt fuer Umwelt, Messung und Naturschutz Baden-Wuerttemberg (2006) 2. Wanner, H., Gyalistras, D., Luterbacher, J., Rickli, R., Salvisberg, E., Schmutz, Ch.: Klimawandel im Schweizer Alpenraum. Zuerich, VdF, 285 p. (2000) 3. Georgi, F., Mearns, L. O.: Approaches to the Simulation of Regional Climate Change: A Review. Reviews of Geophysics, 29, 191–216 (1991) 4. Doms, G., Foerstner, J., Heise, E., Herzog, H.-J., Raschendorfer, M., Schrodin, R., Reinhardt, T., Vogel, G.: A description of the nonhydrostatic regional model LM Part II: Physical parameterizations. Deutscher Wetterdienst, Oﬀenbach (2005) 5. Simmons, A.J. , Gibson, J.K.: The ERA-40 Project Plan. ERA-40 Project Report Series 1, 62 p. (2000) 6. Kanamitsu, M., Ebisuzaki, W., Woollen, J., Yang, S.-K., Hnilo, J.J., Fiorino, M., Potter, G.L.: NCEP-DOE AMIP-II Reanalysis (R-2). Bull. of the Amer. Met. Soc., 77, 3, 1631–1643 (2002) 7. Rodell, M., Houser, P. R., Berg, A. A., Famiglietti, J. S.: Evaluation of 10 methods for initializing a land surface model. J. Hydrometeorol., 6, 146–155 (2005) 8. KLIWA: Langzeitverhalten der Lufttemperatur in Baden-Wuerttemberg und Bayern. KLIWA-Berichte, Heft 5, 76 p. (2005) 9. Rapp, J., Schoenwiese, C.-D.: Atlas der Niederschlags und Temperaturtrends in Deutschland 1891-1990. Frankfurter geowissenschaftliche Arbeiten, Serie B, Band 5, 255 p. (1996) 10. Schaettler, U.: Nutzung moderner Hoechstleistungsrechner fuer die NWV. promet, 28, Nr.1/2, 33–39 (2002)

OpenMP Parallelization of the METRAS Meteorology Model: Application to the America’s Cup Werner Augustin1 , Vincent Heuveline2 , G¨ unter Meschkat3 , K. Heinke 4 5 Schl¨ unzen , and Guido Schroeder 1

2

3

4

5

Steinbuch Centre for Computing, Universit¨ at Karlsruhe [email protected] Steinbuch Centre for Computing, Universit¨ at Karlsruhe [email protected] T-Systems, Project Team Shosholoza [email protected] Meteorological Institute, ZMAW, University of Hamburg [email protected] Meteorological Institute, ZMAW, University of Hamburg [email protected]

Summary. We describe the parallelization of the meteorology model METRAS (MEsoscale TRAnsport and Stream) in the context of the America’s Cup 2007 for the South African sailing yacht Shosholoza. METRAS is a community model of the atmosphere whose development is coordinated at the Meteorological Institute, ZMAW, University of Hamburg. The parallelization which is based OpenMP was done at the Steinbuch Centre for Computing (SCC) of the University of Karlsruhe and took advantage of the speciﬁc features of the Itanium-2 processors available on the local parallel computer HP XC6000. In this paper, we report on the parallelization of the meteorology model METRAS as well as describe how this parallelized version is being used in the highly challenging context of the America’s Cup. Key words: high performance computing, parallelization, OpenMP, meteorology model, America’s Cup

1 Introduction The America’s Cup is one of the most prestigious regatta and match races in the sport of sailing. This competition attracts top sailors and yacht designers not only because of its long history but also due to the fact that innovative technologies have always played a key role in the framework of this competition. It is not surprising that numerical simulation both for the optimization

548

W. Augustin et al.

of the yacht design and for the weather prediction play nowadays a key role in the context of the America’s Cup. Accurate prediction that allows to take into account the weather conditions to chart the best course is a sine qua non condition to win in the America’s Cup. Having accurate, high resolution forecasts as well as reports of current wind and frontal conditions provides a tactical and strategic advantage on a race day. The concern of this paper is twofold. First, we report on the parallelization of the meteorology model METRAS (MEsoscale TRAnsport and Stream) by means of the parallelization paradigm OpenMP. Second, we shall describe on how this parallelized version is being used in the highly challenging context of the America’s Cup. METRAS is a mesoscale meteorology simulation model originally developed at the Meteorological Institute, ZMAW at the University of Hamburg (see [1] for more details). Especially due to its physical adjustment to high resolution, this model was chosen as weather forecast model by the South African sailing team Shosholoza [3] for the America’s Cup 2007 in Valencia. Due to the high computational demands a process optimization and parallelization of the code was essential for this application. The parallelization was done at the Steinbuch Centre for Computing (SCC) of the University of Karlsruhe using the local HP XC6000 Itanium-2 high-performance parallel computer [4]. This paper is organized as follows. Section 2 is dedicated to the description of the meteorology model METRAS with an emphasize on coding issues. In Sect. 3 we describe the considered parallelization techniques. We motivate diﬀerent parallelization strategies and present the eﬃciency results obtained in the context of the America’s Cup. The description of the overall scenario for the America’s Cup where the developed technology has been applied is the object of Sect. 4.

2 Meteorology Model The meteorology model METRAS is applied to investigate atmospheric phenomena in diﬀerent climate regimes, as the Arctic [21], mid latitudes [15] or sub-tropics (Sheng et al., 2000 [13]). It was tested for being applicable over steep terrain ( [10]) as well as for inland regions [11]. METRAS is coupled to a sea ice model to calculate air-sea interaction with a high temporal and spatial resolution ( [7]). The model also includes a pollen transport module [18] and is part of the model system M-SYS that is applied to pollutant transport and deposition studies in conjunction with a chemistry model [16, 19]. 2.1 METRAS METRAS [14] is based on the equations for conservation of momentum, mass and energy. It solves three conservation equations for the wind components,

OpenMP Parallelization of the Meteorology METRAS Model

549

conservation equations for potential temperature, water vapour, cloud liquid water content and rain water content as well as for concentrations (if selected). All equations are solved in flux form. METRAS applies the Boussinesq and anelastic approximation, leading to a diagnostic Poisson equation for pressure derived from the continuity equation. The thermodynamic quantities are also linked by the linearized perfect gas law. A multitude of parameterisations and numerical schemes is implemented and can be used by selecting the corresponding scheme. Here we present the model in the combination used for the Americas Cup wind forecast simulations. For these a first order turbulence closure with counter-gradient term for the unstable atmosphere was selected [9]. The surface fluxes were calculated using a flux aggregation with blending height method [12]. To calculate surface temperature and humidity in dependence of time, surface energy and humidity budget equations were solved. Cloud processes were considered, including condensation, evaporation, autoconversion aggregation and sedimentation [8]. Absorption and reflection of long- and shortwave radiation by clouds were solved with a two-stream approximation scheme. The model is discretized in terrain-following coordinates using a staggered Arakawa-C-Grid. While a uniform grid of 1 km was used in horizontal directions, a non-uniform vertical grid was applied. The lowest level was placed at 10 m above ground and the model top at about 11 km. Grid sizes increased from 120 m above ground by max. 20% (from grid size 20 m to grid size 1000 m). For the Americas Cup wind forecasts central diﬀerences in space and Adams Bashforth time integration were applied for solving advection and diﬀusion in the momentum equations. This scheme was shown to deliver good model results for horizontally uniform grids without using too much computer resources [17]. If vertical diﬀusion lead to a too strong constraint on the time step, an implicit Crank Nicholson scheme was applied that is unconditionally stable. However, to ensure physical reliability of the model results the time step needs to be limited on the basis of physical processes. The Poisson equation for solving pressure was solved with Bi-CGSTAB [20] applying an incomplete LU-factorization (ILU(0)) as preconditioner. 2.2 Process Optimization and Coding Issues Diﬀerent processes in the model act on diﬀerent time scales and change their characteristic times during the day and in dependent of place. Lowest time steps were mostly caused by the vertical advection (at nighttime and over land) or the vertical diﬀusion (for unstable atmosphere), but in cases with heavy precipitation the gravitational settling of large rain droplets resulted in small time steps. However, the general structure of a mesoscale meteorological situation varies with a larger time scale than the time step demanded for stability of the numerical schemes. For example, the radiation fluxes and diﬀusion coeﬃcients depend on the typical lifetime of a cloud (∼ minutes) and the boundary structure (∼ minutes), respectively. Therefore, the process

550

W. Augustin et al.

eﬀects are updated only once per minute. Numerical experiments have been performed to investigate the eﬀect of the update interval on the model performance for the Valencia region. These experiments have shown that an update period of a minute is an optimal compromise between high accuracy and low CPU time. The time step is usually much smaller than one minute (seconds). Some details on the code are given in the following: • approx. 79000 lines of Fortran 90 (almost half of it comments) • 393 program units, each in a single file • developed for a vector computer, though the model is also running on conventional PCs • first prototype available in March 1990, Fortran 90 prototype 1999, OpenMP prototype 2006 • very regular structure, no gotos, no complicated command structure • style guide documenting naming conventions • a lot of data structures, all of them arrays, most of them over the whole simulation domain • no automatic test bed, but benchmark tests available

3 Parallelization The parallelization of METRAS started in September 2006. A running parallel version was needed for the first races of the America’s Cup on 25th of April 2007, with the test runs to be started already in December 2006. Considering the large size of the software library METRAS and the very tight time schedule, specific concepts for the parallelization had to be considered. This section is dedicated to the description of the possible strategies in that highly challenging context and address topics related to the considered parallelization paradigm as well as associated computer architecture issues. Eﬃciency results obtained in the operational phase of the America’s Cup are presented. 3.1 Concepts The parallelization of a software library of the size of METRAS is a highly non trivial task. The analysis of diﬀerent strategies and their implications were the object of the first phase of the associated project. Three parallelization strategies have been evaluated: big-bang strategy with MPI, incremental development with OpenMP, modular big-bang strategy. Their respective advantages and possible drawbacks are depicted below. • Big-bang strategy with MPI This strategy consists of a global redesign of the library towards a parallelization on distributed architecture by means of the Message Passing Interface (MPI). A main diﬃculty associated to this approach is that new

OpenMP Parallelization of the Meteorology METRAS Model

•

•

551

data structures supporting domain decomposition and partitioning features are needed for the data containers. The modification of these containers would lead, however, to a change of almost all routines of METRAS especially for the solver interfaces. The main drawback is therefore that the software would not be operational in the migration phase and debugging would become tedious due to the non locality of the implemented changes both in time and in the library. The benefit of this strategy would be that the parallelized version could take advantage of the versatile compute power capabilities of MIMD platforms. Incremental development with OpenMP This approach consists in a thread-based parallelization using OpenMP. The main advantages of this approach are that the sequential data structures do not need to be modified and that the parallelization can occur step by step along the library. Especially, the software is always operational during the migration phase and one can take directly advantage of the already available new features. Debugging can occur locally by means of comparison with the sequential version. A main drawback of this approach is that the attainable speed up is in practice limited mainly due to the purely shared memory character of this approach. Modular big-bang strategy with MPI This approach relies on the previously described big-bang strategy with MPI restricted to subsets of dedicated routines of METRAS. This strategy would lead to diﬀerent operational releases during the migration phase. It happens, however, that the underlying coupling of the data structure with the solvers is such intricate in METRAS that this strategy has to be rejected in that context.

Based on these arguments, the second strategy relying on an incremental development with OpenMP has been used in this project. 3.2 OpenMP Based Parallelization OpenMP (Open Multi-Processing) is an application programming interface for portable shared memory multiprocessing programming (see e.g. [6] and references therein). The main design goal was to keep the sequential functionality and the source code structure of the program while providing flexible but simple ways to annotate the program that specific parts can be run in parallel. OpenMP is characterized by a rather simple semantic which makes it diﬃcult to capture the complexity of the underlying hardware. •

Synchronization overhead Frequent change of control flow between one sequential main task and parallel parts can reduce the speedup. Though there are many separate parallel loops because of the many diﬀerent data structures which are computed during one global time step this was not a severe problem. The

552

W. Augustin et al.

EPCC OpenMP Microbenchmarks [2] showed an overhead per loop construct of about 5 microseconds, with up to 25 microseconds when using a reduction operation. This was tolerable considering that the runtime of most subroutines in METRAS ranged between at least ten milliseconds and around one second and these subroutines usually involved 5 up to 10 consecutive loops. • shared access to common data Due to the very regular data structures which allowed domain decomposition this was not an issue in METRAS. • shared memory bandwidth This was the most severe problem because all data structures were larger than the level 3 cache. Often the resulting bottleneck could be reduced by restructuring or fusing loops together. When parallelizing incrementally a sequential application one is tempted to parallelize the main calculations which run for 99 percent of the time and to leave the initialization part running sequentially. Unfortunately this can lead to severe performance problems. The OpenMP programming paradigm only supports uniform memory access (UMA) i.e. the time to access specific parts of the memory are assumed to be equal. This is in contrast to the actual hardware shared memory compute nodes. They often have non-uniform memory access (NUMA), the memory is partitioned into diﬀerent parts which are local to a specific processor. Unfortunately these diﬀerent partitions are not considered in current programming environments, where one cannot specify to which processor the result of a memory allocation should be local. Even worse, because of the shared memory paradigm, one usually has only one memory allocation command per data structure, whose diﬀerent parts are later accessed by diﬀerent processors. It is important to note that an allocation command on an operating system like Linux does not actually allocate virtual or even physical memory but address space. The allocation to virtual memory pages takes place when the memory is accessed the first time causing a page fault. Therefore, it is crucial that the first memory access on the data happens in the same pattern the processors will have during the rest of the calculations. The worst case occurs when the initialization is left sequentially and all the allocated memory is the root processor’s local memory. The 16-way nodes of the used parallel computer XC6000 (see Sect. 3.3 for more details) allow to configure the machine either as NUMA or ’interleaved’ i.e. memory is assigned round-robin to the diﬀerent local memory banks resulting in a uniform, load-balanced access. In either configurations 4 processors share the same local memory – so there are eﬀectively only two diﬀerent local memories. Using the machine in NUMA mode without parallel initialization resulted in an overall performance degradation of about 20%. Quite surprisingly there was no significant diﬀerence in performance between ’interleaved’ memory access and NUMA access with parallel initialization.

OpenMP Parallelization of the Meteorology METRAS Model

553

3.3 Architecture and Itanium Speciﬁc Issues For all computations the used parallel computer is an HP Integrity RX8620 16-way Itanium-2 server which is partitioned into two 8-way nodes. These nodes are part of a larger XC6000 installation at the Steinbuch Centre for Computing at the University Karlsruhe with another 100 2-way Itanium-2 nodes. The processors of the so called “fat” nodes run at 1.6 MHz and have 64 GB memory and 6 MB level 3 cache. The METRAS code has been in its sequential version highly optimized for a NEC vector processor i.e. in a context where the trade-oﬀ between bandwidth and computing power was diﬀerent, the main emphasis being put on regularity of the loops and reuse of intermediate results; data locality was not an issue. A typical property of vector processors is that they have a very good memory bandwidth if they can calculate with vectors. Therefore, dependencies between consecutive results inside inner loops should be avoided by any cost because they prevent this vectorization. On the other hand cachebased processors like the Itanium can profit from re-use of consecutive results and especially operands inside a loop – at least if the time to re-use is long enough to finish the operation. One of the main design features of the Itanium architecture is explicity in the sense that the compiler has to be able to analyze the code statically and create explicit code for instruction parallelism, data prefetching, instruction predication, loop optimization through software pipeling etc.. In that respect, similar to a vector processor, the Itanium processor performs best when it has large regular problems with additionally at least some cache re-use. Conditional instructions or even subroutine calls in inner loops can impede prefetching. In the software implementation special care has been given to use this specific features of the Itanium architecture. In that framework and for this implementation the two first authors have obtained the finalist prize of the International Itanium Contest 2007. 3.4 Eﬃciency Results In this section we present the prototypical eﬃciency results obtained in the framework of the America’s Cup. The Table 1 shows the speedup of the most important subroutines used in METRAS. It clearly demonstrates the worst opponent of parallelization: Amdahl’s Law. Though quite a lot of these subroutines show a good speedup, only few others can lower the overall speedup considerably, their percentage of the total runtime increasing noticeably. Further investigations were done using results from hardware counters. Table 2 shows the diﬀerent ratios between floating point operations, CPU cycles and load instructions for some selected subroutines. Generally, a higher ratio of floating point operations to load instructions gives a higher speedup, showing that the calculations are memory bound. Another aspect are the sometimes large diﬀerences between the load instructions per cycle ratios of

554

W. Augustin et al. Table 1. Speedups of most important subroutines (on 8 processors)

sequential parallel speedup runtime runtime description [%] [%] se escalar 5.7 49.0 32.7 scalar quantities orad 5.0 1.4 1.1 radiation se phad 5.8 21.2 13.9 advection and diffusion odipvi 6.3 12.1 7.2 implicit vertical diffusion of scalar quantities oaduph 6.3 5.8 3.5 advection upstream oevap 5.6 5.8 3.9 evaporation of rain drops osedi 7.8 2.7 1.3 sedimentation rate rain water okond 7.3 4.4 2.3 condensation rate ouitra 4.8 1.8 1.4 coordinate transformation of wind oakkr 6.1 1.6 1.0 accretion of raindrops oqbsur 12.8 1.2 0.4 surface humidity others 4.6 9.0 7.5 se ewical 3.0 45.0 56.6 vector quantities se p2 1.7 14.7 33.4 air-pressure solver se pilmat 1.0 2.0 8.2 preconditioner (running sequentially) se piumat 1.0 2.2 8.8 preconditioner (running sequentially) se plmat 2.7 1.9 2.6 matrix iteration odif 5.3 16.3 11.7 vertical diffusion of wind oadvf 7.6 5.0 2.5 advection terms of equation of motion others 3.7 9.0 9.1 others 2.1 6.0 10.6 total 3.8 100.0 100.0 subroutine

Table 2. relation of diﬀerent hardware counters (ﬂoating point operations, cpu cycles, load instructions) and the resulting speedup

odifhe oad0ph okond onscal

sequential fp/cyc fp/ld ld/cyc 1.97 6.56 0.30 1.60 2.14 0.75 0.27 6.52 0.04 0.29 1.95 0.15

8 processors speedup fp/cyc fp/ld ld/cyc 1.76 6.35 0.28 7.1 0.80 2.12 0.38 4.2 0.22 4.98 0.04 7.3 0.13 1.81 0.07 3.7

diﬀerent routines. The okond subroutine has an if-construct in the innermost loop which could not be transformed eﬃciently into instructions with predication therefore giving a high branch misprediction penalty. On the other hand there are simple, regular and straightforward subroutines like onscal which nevertheless show a bad performance without any obvious reason.

4 Scenario for the America’s Cup The America’s Cup is the oldest trophy in sailing sport and gives name to one of the most important sailing competition events. The important match races are two boat races. Very high eﬀorts are done by the competing teams to win the first leg (beat) of a race. Wind shifts of a few degrees after the

OpenMP Parallelization of the Meteorology METRAS Model

555

start can bring a competitor immediately in front. Therefore each sailing team has a weather team which does every eﬀort to forecast wind shifts and speed diﬀerences over the race course. A better forecast can make a sailing team win a race. A typical race course length is 2 to 3 nautical miles and typical race duration is 60 minutes. The South African sailing team Shosholoza chose the high resolution model METRAS for getting a high resolution forecast in space and time and for using the results in strategy at each race. A precise prediction of the wind fields during the time of the races often decides between victory and defeat. This prediction cannot be derived or interpolated from local measurements or standard weather forecasts. To achieve the necessary precision a high resolution model of the atmosphere is required, which can calculate wind direction changes of a few degrees and speed diﬀerences of at least 1 knot in the racing area of about 5×5 km2 during a time period from 2 to 6 pm local summer time. For Team Shosholoza this was achieved with the non-hydrostatic high-resolution mesoscale model METRAS described in Sect. 2. The diﬀerences in the ground heating is taken into account as well as the shading of the ground by mountains, heterogeneous development of boundary layers and the influence of small scale heterogeneities on the wind fields. An exact simulation of the meteorological processes above the land is essential for a good prediction of the sea breeze in the bay of Valencia. 4.1 Conﬁguration and Setup From the middle of February 2007 untill the middle of May 2007 two 8processor SMP nodes of the HP XC6000 supercomputer at the Steinbuch Centre for Computing (SCC) in Karlsruhe were used exclusively for the wind forecast for Team Shosholoza at the America’s Cup this year in Valencia. G¨ unter Meschkat was the chief meteorologist on site in Valencia and was supported by Prof. Heinke Schl¨ unzen, Guido Schr¨ oder and their team at the Meteorological Institute, ZMAW, University of Hamburg. The IT support for the numerical simulation was given by SCC in Karlsruhe. Every day two simulation runs were started at 6:00 UTC using the current weather prediction of the Spanish meteorological service as boundary conditions in two diﬀerent versions. Figure 1 shows the nesting of the METRAS area with boundaries. The results – mainly wind diagrams like the one shown in Fig. 3 – could be reviewed together with the sailing crew till about noon (dock oﬀ time). Races started around 2 pm. Till 5 minutes before race start communication to the sailing team was allowed by rules. Therefore UMTS based radio connections were set up. By this line voice communication from the weather team to the sailing team and downloads via a VPN based connection between the computers on the race yacht RSA 83 to the computers of the weather team at the Shosholoza base were possible. Via this line the METRAS forecast results about wind shifts and wind speed diﬀerences were communicated to

556

W. Augustin et al.

Fig. 1. Nesting of METRAS into low resolution forecast

Fig. 2. Communication schedule

the sailing team in a special format. These results were used by the strategist and the navigator of the sailing team. 5 minutes before start all communication equipment had to go oﬀ the race yacht. 4.2 Results Figure 3 shows the horizontal wind field in knots at 10 m above ground. The time is given in UTC, the simulated grid had a resolution of 1 km, the area a size of 151 km × 179 km in the horizontal and a vertical extension of 11 km (35 grid points). For the case presented, the large-scale synoptic situation is characterized by a north-eastern wind and changes through local, thermal caused and orography influenced flow fields. A sea breeze circulation develops,

OpenMP Parallelization of the Meteorology METRAS Model

557

Fig. 3. Wind prediction in the bay of Valencia

having little intensity at about 10 am and intensifies during the day, thereby expanding oﬀshore and inland until the evening. This causes a south-easternly wind at the race course of the America’s Cup. The course extends about 5 km and is located approximately at the point of origin of the figures. At 12

558

W. Augustin et al.

UTC (corresponds to 2 pm local summer time), i.e. only a short time before the start of the race, the sea breeze front, characterized by a strong updraft zone and a convergence zone at ground level, is located approximately 20 km inland. It continues to move inland, leaving a zone of intense sea wind over the race course during the rest of the afternoon. The situation mentioned above is a good example of the individual development of the sea breeze which develops diﬀerently every day at this time of the year.

5 Conclusions The OpenMP parallelization of the meteorology model METRAS on the Itanium-2 based parallel computer HP XC 6000 was presented. The advantages and drawbacks of the proposed approach in the context of the numerical simulations and tight schedule dictated by the constraints of the America’s Cup were discussed. On the 8-way nodes a speedup of 3.8 (resp. 5.1 without pressure solver) has been achieved. Additionally, an optimization of physical processes like ratiation and turbulence update lead to a speedup of approximately a factor of two. The number of iterations for the inherently sequential pressure solver was reduced to a physically justifiable minimum. Based on this and the parallelization, METRAS is now about a factor of 1.8 up to 2.3 faster than real time for the America’s Cup set-up. In addition, the process optimisation has to be deeper investigated to generalize the currently case-specific adjustment. Future work will mainly cover hybrid parallelization where both MPI and OpenMP will be combined for the parallelization of METRAS. Acknowledgements The financial support of this project by the Ministry of Science, Research and the Arts of the State of Baden-W¨ urttemberg in the framework of the High Performance Computing Competence Center Baden-W¨ urttemberg HKZ-BW is greatly acknowledged. The two first authors want to thank Prof. Dr. R. Lohner, Dr. H. Obermaier from the Steinbuch Centre for Computing for their continuous support and many valuable discussions.

References 1. http://www.mi.uni-hamburg.de/Mesoscale_Model_METRAS.359.0.html 2. http://www2.epcc.ed.ac.uk/computing/research_activities/ openmpbench/openmp_index.html 3. http://www.team-shosholoza.com/ 4. http://www.rz.uni-karlsruhe.de/ssck/hpxc.php 5. http://www.rz.uni-karlsruhe.de/ssck/hpxc4000.php 6. R. Chandra, R. Menon, L. Dagum, D. Kohr, D. Maydan and J. McDonald (2001): Parallel Programming in OpenMP. Morgan Kaufmann.

OpenMP Parallelization of the Meteorology METRAS Model

559

7. S. Dierer, K. Schl¨ unzen, G. Birnbaum, B. Br¨ ummer and G. M¨ uller (2005): Atmosphere-sea ice interactions during cyclone passage investigated by using model simulations and measurements. Mon. Wea. Rev., 133, 3678–3692. 8. E. Kessler, E. (1969): On distribution and continuity of water substance in atmospheric circulations. Mereorol. Monogr., American Meteorological Society, 84 pp. 9. C. L¨ upkes, C. and K. H. Schl¨ unzen (1996): Modelling the Arctic convective boundary-layer with diﬀerent turbulence parameterizations. Boundary-Layer Meteorol., 79, 107–130. 10. U. Niemeier and K. H. Schl¨ unzen (1993): Modelling steep terrain inﬂuences on ﬂow patterns at the isle of Helgoland. Beitr. Phys. Atmosph., 66, 45–62. 11. E. Renner and A. Munzenberg (2003): Impact of biogenic terpene emissions from Brassica napus on tropospheric ozone over Saxony (Germany) – Numerical investigation. Environmental Science and Pollution Research, 10 (3). 12. K. von Salzen, M. Claussen and K. H. Schl¨ unzen (1996): Application of the concept of blending height to the calculation of surface ﬂuxes in a mesoscale model Meteorol. Zeitschrift, N.F. 5, 60–66. 13. L. Sheng, K.H. Schl¨ unzen and Z. Wu (2000): Three-dimensional numerical simulation of the mesoscale wind structure over Shandong peninsula. Acta Meteorol. Sinica, 1, 97 – 107. 14. K. H. Schl¨ unzen (1990): Numerical studies on the inland penetration of sea breeze fronts at a coastline with tidally ﬂooded mudﬂats. Beitr. Phys. Atmosph., 63, 243–256. 15. K.H. Schl¨ unzen and J.J. Katzfey (2003) : Relevance of sub-grid-scale land-use eﬀects for mesoscale models. Tellus, 55A, 232–246. 16. K. H. Schl¨ unzen and E. M. I. Meyer (2007): Impacts of meteorological situations and chemical reactions on daily dry deposition of nitrogen into the southern north sea. atmospheric environment. Atmos. Environ., 41-2, 289–302. 17. G. Schr¨ oder, K. H. Schl¨ unzen and F. Schimmel (2006): Use of (weighted) essentially non-oscillating advection shemes in the mesoscale model. Quarterly Journal Roy. Met. Soc., 132, 1509–1526; DOI: 10.1256/j.04.191. 18. S. Schueler and K. H. Schl¨ unzen (2006): Modeling of oak pollen dispersal on the landscape level with a mesoscale atmospheric model. Environ. Mod. Assess., 11, 179–194. 19. A. Trukenm¨ uller, D. Grawe and K. H. Schl¨ unzen (2004): A model system for the assessment of ambient air quality conforming to ec directives. Meteorol. Z., 13, 387–394. 20. H. A. Van der Vorst (1992): Bi-CGSTAB: A fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems. SIAM J. Sci. Statist. Comput., 13, 631–644. 21. T. Vihma, J. Hartmann and C. L¨ upkes (2003): A case study of an on-ice air ﬂow over the Arctic marginal sea ice zone. Boundary Layer Meteorol., 107, 189–217

Adaptive Analysis of Bifurcation Points of Shell Structures E. Ewert and K. Schweizerhof Institut f¨ ur Mechanik, Universit¨ at Karlsruhe (TH), Kaiserstraße 12, D-76131 Karlsruhe, Germany [email protected]

1 Introduction In recent years many procedures for stability investigations based on the Finite Element Method have been developed to compute the so-called stability points of structures in order to judge the stability. However, in engineering practice predominantly knock-down factors based on experiments are used for the design loads. The reason for such a procedure is the hard to quantify sensitivity of shell structures – especially of cylindrical shells – against perturbations like geometrical and loading imperfections or imperfections in boundary conditions. The standard procedure as also proposed in design rules is based on the computation of the limit load taking into account the modification of the bifurcation load resp. the snap-through load due to geometrical imperfections. Thus, the eigenvalueproblem for stability points have to be computed very accurately. In the present contribution an adaptive h-refinement procedure is taken for the solution using low order shell elements. The algorithm is partially based on the well-known a-posteriori error estimator of Zienkiewicz and Zhu [Zie92] with the stresses computed using the eigenvectors instead of displacement vectors, as e.g. proposed by Stein et al. [Stein94].

2 Static Stability Criteria After discretization with finite elements analysis the following nonlinear system of equations must be solved to compute states of equilibrium r(u) − λp = 0.

(1)

Herein r denotes the vector of internal forces depending nonlinearly on the displacements, p the load vector and λ the load multiplier. For a Newton type solution within an incremental iterative procedure, the linearization of system

564

E. Ewert, K. Schweizerhof

of equations (1) leads to K T (ui )Δui+1 = λp − r(ui )

with ui+1 = ui + Δui+1 .

(2)

Herein K T denotes the tangent stiﬀness matrix in general nonlinearly depending on the current displacement state and Δui+1 is the vector of the displacement increments for the next iteration step. Within the incremental iterative solution process involving an update of the displacement vector the iterations are performed until a certain tolerance is met and convergence is achieved. The tangent stiﬀness matrix K T , which is obtained by linearization of r(u), can be used in stability investigations of converged states of equilibrium. As it is well known for conservative systems as considered here, a state of equilibrium is stable, if K T is positive definite, i.e. all eigenvalues μ from the standard eigenvalue problem (K T − μI) Ψ = 0

(3)

are larger than zero. Another commonly used stability criterion is based on the determinant of K T . As the transition from stable to unstable equilibrium states is characterized by zero eigenvalues μ = 0, the characteristic equation of (3) gets (4) det(K T ) = 0. Using e.g. these criteria singular points can be determined.

3 Computation of Stability Points for a Geometrically Perfect Cylinder Under Axial Compression First of all mesh convergence studies are performed for a quarter of a perfect cylinder under axial compression using uniform mesh refinement (Fig. 1). The results achieved with the quarter of a cylinder have been verified for few meshes using a model of the whole cylinder, which required very large memory, see Sect. 4. As boundary conditions the upper and lower edges are hinged allowing displacements in axial direction. At both edges a defined load is applied in axial direction. In this case the analytical solution can be written as Fcr 2πEt2 α= = 0.843 with Fcr,cl = : . (5) Fcr,cl 3(1 − ν 2 )

with the normalized critical load α for the given boundary conditions, the critical load Fcr and the classical critical load Fcr,cl , see Yamaki [Yam84]. Two diﬀerent shell elements are used in the FE model: a bilinear shell element with ANS for the transversal shear strains according to Dvorkin and Bathe [Dvor89] and a biquadratic shell element MITC9 according to Bucalem and Bathe [Buca93]. The results of 4 refinement steps are given in Fig. 2a). The

Adaptive Analysis of Bifurcation Points of Shell Structures t symm. bc’s

y,v

R

Geometry: R = 625 mm H = 966 mm t = 0.56 mm

90° f

6 z,w * HH jx,u

565

symm. bc’s

Material (steel): E = 2.0 · 105 N/mm2 ν = 0.3

H

bc’s for upper and lower edges: u = w = 0; v = 0

f

Fig. 1. Quarter of perfect cylinder under axial compression, geometry, material and boundary conditions a) uniform reﬁnement only 1.2

1.2

biquadratic bilinear

1.15 normalized load α

b) uniform and adaptive reﬁnement

1.1

1.1

1.05

1.05

1

1

0.95

0.95

0.9

0.9

0.85

0.85

0.8

l1

10

20 30 40 50 60 70 degrees of freedom [*1000]

80

l2

q1

l3 q2

0.8 0

bilinear, uniform bilinear, adaptiv biquadr., uniform biquadr., adaptiv

1.15

0

10

q3

20 30 40 50 60 70 degrees of freedom [*1000]

80

Fig. 2. Static buckling load levels (singular points) of a perfect cylinder modeled with bilinear and biquadratic elements; normalized to classical critical load

convergence diagram for the normalized buckling load α shows a much better convergence for the biquadratic elements than for bilinear elements. Even for a very fine mesh with 120 × 120 elements and 72240 degrees of freedom (Ndof ) the deviation of the buckling load computed with bilinear elements from the analytical solution is still about 3% in this simple example. The reason is the much smoother approximation of the curved geometry using biquadratic elements. Though diﬀerent eigenvectors are achieved for the starting meshes, both models converge to the same final buckling mode, see Fig. 3. An adaptive h-refinement procedure might improve the convergence behaviour of a FE solution. In the present contribution a procedure partially based on the well-known a-posteriori error estimator of Zienkiewicz and Zhu [Zie92] is used. The exact error distribution ||e|| is defined by 2 (σ − σ h ) : (ǫ − ǫh )dΩ (6) ||e|| = Ω0

566

E. Ewert, K. Schweizerhof bilinear

biquadratic

Fig. 3. First eigenvectors at singular points for each coarse (Ndof = 2040) and each ﬁne (Ndof = 72240) meshes; uniform reﬁnement with bilinear and biquadratic elements

with exact values of stresses σ resp. strains ǫ. The stresses σ h resp. strains ǫh are the discrete values the FE solution. As the exact values are unknown the expression (6) is approximated by 2 (σ ∗ − σ h ) : (ǫ∗ − ǫh )dΩ. (7) ||eh || = Ω0

The values denoted with “*” are called recovered values and are computed by a least-square fit on element patches using the so-called superconvergent patch recovery procedure, see Zienkiewicz and Zhu [Zie92]. Using the recovered and the discrete values the error distribution can be computed and used for mesh refinement. The values resp. the error distribution is computed here using the lowest eigenvector of system of equations (3) at the singular point, similar to the proposal of Stein et al. [Stein94]. The following procedure is followed in an adaptive computation of singular points and buckling modes: starting with an initial mesh, e.g. 20×20 elements, singular points are computed using a nonlinear computation monitoring the eigenvalues of interest in system of equations (3) in combination with the bisection procedure. The first eigenvalue at this point is then obtained using the standard eigenvalue problem (3). Finally a refined mesh is achieved using the error distribution, which is computed as described before. Then the next refinement step is performed for the refined mesh starting with the computation of a new singular point. The results of an adaptive analysis using bilinear

Adaptive Analysis of Bifurcation Points of Shell Structures

567

elements show, that the convergence behaviour concerning the singular point could be improved, see Fig. 2b). In contrast to this the found buckling modes do not converge to the target mode of the uniformly refined mesh, see Fig. 4. This is due to transition elements, which appear in adaptive meshes and are often considerably distorted for curved shells compared to standard element form. Therefore they often cause artificial geometrical imperfections leading to buckling modes, which are partially totally diﬀerent from the correct modes. Modeling the cylinder with biquadratic elements these imperfections introduced by imperfect geometry approximation are small enough and do not influence the computation of eigenmodes at singular points distinctly, see Fig. 5. The sensitivity concerning stability depends strongly on the choice of the perturbation patterns, see Ewert et al. [Ewe05]. As the perturbation patterns are often chosen aﬃne to the eigenvectors, the latter should be computed as accurately as possible for following sensitivity investigations. l1

l2

l3

Fig. 4. First eigenvectors at singular points for reﬁnement steps l1 (Ndof = 8080), l2 (Ndof = 25834) and l3 (Ndof = 65315) indicated in Fig. 2; adaptive reﬁnement using bilinear elements q1

q2

q3

Fig. 5. First eigenvectors at singular points for reﬁnement steps q1 (Ndof = 8080), q2 (Ndof = 19698) and q3 (Ndof = 52442) indicated in Fig. 2; adaptive reﬁnement using biquadratic elements

568

E. Ewert, K. Schweizerhof

4 Computational Aspects For the investigations described above the own FE-code named FEAP-MeKA is used. This code is written in FORTRAN 77 (compiled with the Intel compiler) and is based on the open-source FE-code named FEAP (Finite Element Ananlysis Program) from the University of Berkeley. Below some examples for computational time and memory requirement of calculations performed on a single processor are given. • Bisection for a quarter of a perfect cylinder with 72000 dof (120 × 120 bilin. el.): – comp. time: 5 h 6 m 23 s (accuracy 1 · 10−5 ) – comp. time: 2 h 15 m 11 s (accuracy 1 · 10−2 ) – physical memory: 668 MB – virtual memory: 5488 MB • Direct computation of singular point for a whole perfect cylinder with 71761 dof (60 × 240 biquad. el.): – comp. time: 21 m 17 s (accuracy 1 · 10−5 ) – physical memory: 2001 MB – virtual memory: 5488 MB • Bisection for a whole perfect cylinder with 287521 dof (120 × 480 bilin. el.) – comp. time: more than 3 days – comp. time for a single iteration: 28 m 9 s – physical memory: 5289 MB – virtual memory: 5488 MB Investigations with the parallelized version of FEAP-MeKA lead to the speedup diagram in Fig. 6, which is typical for the investigated problems. 0.045

-1

(computational time) [1/s]

0.04

0.035

0.03

0.025

0.02

0.015 2

3

4

5

6

7

8

number of processors

Fig. 6. Speed-up diagramm for the parallelized FEAP-MeKA; typical run for buckling analysis

Adaptive Analysis of Bifurcation Points of Shell Structures

569

References [Buca93] Bucalem ML, Bathe KJ.: Higher-order MITC general shell elements. Int. J. Num. Meth. Eng. 36: 3729–3754 (1993) [Dvor89] Dvorkin EN, Bathe KJ: A continuum mechanics based four node shell element for general nonlinear analysis. Eng. Comp.1: 77–88 (1989) [Yam84] Yamaki N.: Elastic stability of circular cylindrical shells. North-Holland series in applied mathematics and mechanics,27, (1984) [Zie92] Zienkiewicz O and Zhu J.: The superconvergent patch recovery and a posteriori error estimates. Part 1: The recovery technique. Part 2: Error estimates and adaptivity. Int. J. Num. Meth. Eng. 33: 1131–1382 (1992) [Ewe05] Ewert E, Schweizerhof K, Vielsack P.: Measures to judge the sensitivity of thin-walled shells concerning stability under diﬀerent loading conditions. Int. J. Comp. Mech. 37: 507–522 (2006) [Stein94] Stein, E, Seifert, B, Ohnimus, S, Carstensen, C. Adaptive Finite Element Analysis of Geometrically Non-linear Plates and Shells, Especially Buckling. Int. J. Num. Meth. Eng., 37-15: 2631–2656 (1994)

Molecular Modeling of Hydrogen Bonding Fluids: Formic Acid and Ethanol + R227ea T. Schnabel, B. Eckl, Y.-L. Huang, J. Vrabec, and H. Hasse Institut f¨ ur Technische Thermodynamik und Thermische Verfahrenstechnik, Universit¨ at Stuttgart, D-70550 Stuttgart, Germany [email protected]

1 Introduction Currently, molecular modeling and simulation gains importance for the prediction of thermophysical properties of pure fluids and mixtures, both in research and industry. This is due to several reasons: Firstly, the predictive power of molecular models allows for results with technically interesting accuracies over wide range of state points and makes it superior to classical methods. Secondly, a given molecular model provides access to the full variety of thermophysical properties, such as thermal, caloric, transport or phase equilibrium data. Finally, through the advent of cheaply available powerful computing infrastructure, reasonable execution times for molecular simulations can be achieved. Molecular modeling and simulation are based on statistical thermodynamics which links the intermolecular interactions to the macroscopic thermophysical properties. This sound physical background also supports the increasing acceptance compared to phenomenological modeling. Modeling thermophysical properties of hydrogen bonding systems remains a challenge. Phenomenological models often fail to describe the interplay between the energetics of hydrogen bonding and its structural eﬀects. Molecular force field models, however, are much better suited for solving that task as the explicitly consider this interplay. Most of the presently available molecular models use crude assumptions for the description of hydrogen bonding which can, for instance, be simply modeled by point charges superimposed to Lennard-Jones (LJ) potentials. One benefit of this simple modeling approach for hydrogen bonding is the small number of adjustable model parameters. Furthermore, the approach is compatible with numerous LJ based models from the literature and it can be combined to model mixtures. This simple modeling approach emerged to be fruitful in many ways, although most of the molecular models proposed in the literature lack in the quantitative description of thermophysical properties. The aim of this project is to tackle this problem and to show that a sound modeling and parameterization does indeed yield quantitatively correct results.

574

T. Schnabel et al.

Molecular models which accurately describe vapor-liquid equilibria over the full temperature range exhibit good predictive power throughout the whole fluid region. The molecular model for ethanol developed in the first period of the MMHBF project [NJR05], has these characteristics and excellently performed in the prediction of vapor-liquid equilibrium properties of mixtures, i.e. Henry’s law constants [SVH05, SVH06]. Following this, the ethanol model was applied in the present period of the MMHBF project to participate in a competition, organized by the Industrial Fluid Properties Simulation Collective [IFPSC], to predict the vapor-liquid coexistence curves of ethanol + 1,1,1,2,3,3,3-Heptafluoropropane (R227ea). To solve this problem, a new molecular model for the refrigerant R227ea was developed which does not form hydrogen bonds with itself but shows solvent-solute eﬀects due to weak interspecies hydrogen bonding with ethanol [EHVH07]. The chosen modeling strategy for hydrogen bonding turned out to be successful in this international competition. To study the modeling approach for hydrogen bonding also for other strongly associating fluids, a new molecular model for formic acid [SCVLH07] was developed in the present period of the MMHBF project with the aim to yield accurate vapor-liquid equilibria. Formic acid is the simplest carboxylic molecule and has exceptional thermophysical properties due to its strong ability to act both as hydrogen bond donor and acceptor. Since both hydrogen atoms of formic acid can act as proton donors and both oxygen atoms provide proton acceptance, four unlike hydrogen bond types yield the basis for a complex self-association which is the reason for its exceptional thermophysical behavior. This report is organized as follows: Firstly, the new molecular models for formic acid and R227ea are introduced. Subsequently, the vapor-liquid coexistence curve of ethanol + R227ea is discussed. The improvements regarding execution time on NEC SX-8 of our molecular simulation program ms2 used in the present project period complete this report.

2 Molecular Model for Formic Acid A new formic acid model [SCVLH07] was developed with the aim to yield a favorable description of the vapor-liquid equilibrium at relatively low computational cost. It neglects the internal degrees of freedom and uses anisotropic united atom LJ sites accounting for repulsive and dispersive interactions. Point charges were used to model both polarity and hydrogen bonding of formic acid. In total, three LJ sites and four point charge sites at five positions were chosen. Thus, the potential energy uij between two formic acid molecules i and j is given by 12 6 5 5 qia qjb σab σab uij (rijab ) = + 4ǫab , (1) − rijab rijab 4πε0 rijab a=1 b=1

Molecular Modeling of Hydrogen Bonding Fluids

575

where a is the site index of molecule i and b the site index of molecule j, respectively. The site-site distance between molecules i and j is denoted by rijab . σab and ǫab are the LJ size and energy parameters, qia and qjb are the point charges located at the sites a and b on the molecules i and j, respectively. Finally, ε0 is the permittivity of vacuum. The interaction between unlike LJ sites is defined by the Lorentz-Berthelot combining rule [Lor81, Ber98] σaa + σbb , 2 √ = ǫaa ǫbb .

σab =

(2)

ǫab

(3)

Figure 1 depicts the positions of the interaction sites of the present molecular model and Table 1 gives the model parameters according to (1). To avoid unphysical interactions due to the superposition of point charge and LJ potentials at very small site-site distances, which primarily may occur in Monte Carlo simulations, internal hard-sphere cutoﬀs at the point charge sites were employed. Such internal hard-sphere cutoﬀs were also used, e.g., by L´ısal et al. [LSN01] for vapor-liquid equilibrium simulations of hydrogen bonding components. However, their internal hard-sphere cutoﬀs were located at the LJ sites, hence, the cutoﬀs chosen by L´ısal et al. [LSN01] were considerably larger than the present ones, which are located directly at the point charge sites. Such an approach was previously proposed by M¨ oller and Fischer [MF94] and Stoll [Sto05]. The diameter of an internal hard-sphere located at site a is c c termed σaa in the following. The internal hard-sphere diameter σab between

Fig. 1. Geometry of the present formic acid model: Si indicates the model interaction site i and Nj the nucleus position of atom j of the cis-conformer obtained from quantum chemical calculations. Note that all sites of the present model are within a plane

576

T. Schnabel et al.

Table 1. LJ, point charge, internal hard-sphere and geometry parameters of the present formic acid model, cf. (1) and Fig. 1; the electronic charge is e = 1.6021 . . . · 10−19 C, Boltzmann’s constant is denoted by kB Site SO SCH SH(C) SOH SH(O)

σaa ˚ A 2.9953 3.2335 0 3.1496 0

ǫaa /kB K 96.696 59.993 0 85.053 0

qia e −0.42186 0 +0.29364 −0.31574 +0.44396

c σaa ˚ A 0.2995 0 0.3234 0.3150 0.3150

h1 h2 h3 h4 ˚ ˚ ˚ ˚ A A A A 1.21473 0.12650 1.38988 0.98023 γ1 γ2 γ3 deg deg deg 125.545 120.255 110.804

unlike point charges is set to the arithmetic mean of the like internal hardsphere cutoﬀ diameters. The like internal hard-sphere diameters of the present formic acid model are included in Table 1. Vapor-liquid equilibrium results of the present molecular model for formic acid are compared to experimental data [DIPPR05] in Figs. 2 to 4. These figures also include the results of Chialvo et al. [CKN05] who used the molecular model of Jedlovszky and Turi [JT97, JT99]. The results of Chialvo et al. [CKN05] had to be extracted from the published graphs since the numerical data are not accessible. For the saturated densities in Fig. 2, also Gibbs ensemble Monte Carlo results for the model of Jedlovszky and Turi [JT97, JT99] are available from Min´ ary et al. [MJMT00]. Simulation results were tested for thermodynamic consistency with the Clausius-Clapeyron equation and consistency is given for the present model in the scope of the simulation uncertainties. The present bubble densities agree well with the experiment, the mean unsigned error is only 0.8%. Note that the bubble density of the present model at 290 K shown in Fig. 2 was estimated by a N pT simulation at vanishing pressure. However, the agreement of the dew densities with a mean unsigned error of 29% is poor. The “experimental” dew densities were obtained from the Clausius-Clapeyron equation using experimental vapor pressure, bubble density and heat of vaporization [DIPPR05]. These “experimental” dew densities are in good agreement with dew densities evaluated with the dimerization constant for formic acid proposed by B¨ uttner and Maurer [BM83] up to 400 K. Figure 2 indicates that the new molecular model is significantly more

Molecular Modeling of Hydrogen Bonding Fluids

577

Fig. 2. Saturated densities of formic acid: bullet, present simulation results; full triangle down, simulation results of Chialvo et al. [CKN05] using the model of Jedlovszky and Turi [JT97, JT99]; open square, simulation results of Min´ ary et al. [MJMT00] using the model of Jedlovszky and Turi [JT97, JT99]; line, experimental data [DIPPR05]

Fig. 3. Vapor pressure of formic acid: bullet, present simulation results; full triangle down, simulation results of Chialvo et al. [CKN05] using the model of Jedlovszky and Turi [JT97, JT99]; line, experimental data [DIPPR05]

precise regarding the saturated densities than the model of Jedlovszky and Turi [JT97, JT99]. The new molecular model is also much more accurate for the vapor pressure and the heat of vaporization, cf. Fig. 3 and 4. The vapor pressure is described by the new molecular model with a mean unsigned error of 5.1%. The heat of vaporization shows a reasonable temperature dependence, however, significant deviations are present, where the mean unsigned error is 20%.

578

T. Schnabel et al.

Fig. 4. Heat of vaporization of formic acid: bullet, present simulation results; full triangle down, simulation results of Chialvo et al. [CKN05] using the model of Jedlovszky and Turi [JT97, JT99]; line, experimental data [DIPPR05]

It can be speculated that this is due to the inability of the present model to yield accurate vapor configurations which could also be the cause for the poor dew densities.

3 Molecular Model for R227ea The new molecular model for R227ea (CF3 -CHF-CF3 ) [EHVH07] was developed based on quantum chemical calculations and optimized using experimental vapor pressure and bubble density. As the complexity of a molecular model determines the required computing time in molecular simulation, it was attempted to find an eﬃcient solution balancing accuracy and simplicity. A rigid model with ten LJ sites plus one point dipole and one point quadrupole was chosen. The assumption of rigidity is reasonable due to the compact shape of the molecule, where only minor conformational changes are present. Neglecting the explicit repulsive and dispersive contribution of the hydrogen atom, the number of LJ sites was determined by the number of the remaining atoms (11 total −1 hydrogen) here. The geometric and electrostatic parameters of the molecular model were taken directly from quantum chemical calculations. For this purpose, initially a geometry optimization was performed with the GAMESS (US) package [SBB93], employing the Hartree-Fock method and the basis set 6-31G. For the quantum chemical calculations, the symmetry of the molecule was exploited and only half of it was regarded. A LJ site was located exactly at all resulting nuclei positions, except for the hydrogen atom. The C-H group was modelled by a single LJ site, i.e. the united-atom approach was used. The coordinates of all ten LJ sites are given in Table 2.

Molecular Modeling of Hydrogen Bonding Fluids

579

Table 2. Coordinates of the LJ sites in principal axes of the new molecular model for R227ea. Bold characters indicate represented atoms Site C(1) C(2)-H C(3) F-C(1) F-C(1) F-C(1) F-C(2) F-C(3) F-C(3) F-C(3)

x ˚ A 0.143398 –0.402114 0.143398 –0.004171 –0.473866 1.456004 –1.799824 –0.004171 –0.473866 1.456004

y ˚ A –0.009483 –1.009569 –0.009483 1.321357 –0.562621 –0.303824 –0.265525 1.321357 –0.562621 –0.303824

z ˚ A –1.278946 0.000000 1.278946 –1.351360 –2.339021 –1.348218 0.000000 1.351360 2.339021 1.348218

Table 3. Orientations and moments of the electrostatic interaction sites of the new molecular model for R227ea. Orientations are deﬁned in standard Euler angles where ϕ is the azimuthal angle with respect to the x–z plane and θ is the inclination angle with respect to the z axis Site Dipole Quadrupole

ϕ deg

θ deg

μ or Q

278.48 8.48

90 90

2.209 D 6.440 B

For the parameterization of the electrostatic interactions another quantum chemical calculation was carried out on top of the optimized geometry. To account for the electron correlation, the perturbation theory Møller-Plesset level 2 was used with the polarizable 6-311G** basis set. Molecular electrostatics are significantly influenced by the molecular environment, hence liquid phase dipole moments diﬀer from those of isolated molecules in the vacuum or in the gaseous state. As the aim was to develop a molecular model with state independent parameters and the liquid phase is more dominated by the intermolecular interactions than the gas phase, it was tried to mimic the liquid phase in the quantum chemical calculation. This was obtained by the COSMO (COnducter-like Screening MOdel) method [BK97], i.e. placing the molecule in a dielectric cavity with a dielectric constant of εr = 20. Note that the resulting polarities depend only weakly on the dielectric constant in this range. On the basis of the obtained electron density distribution, the electrostatic interactions were reduced to one point dipole and one point quadrupole both located in the center of mass of the molecule. The orientations and moments of these two polarities are given in Table 3. The parameters of the LJ sites were optimized to fit correlations of experimental saturated liquid density and vapor pressure of pure R227ea [DIPPR05]

580

T. Schnabel et al.

in the range from 280 to 355 K. Three diﬀerent LJ parameters sets were used: one for the seven fluorine atom sites, one for the united-atom C-H group site, and one for the remaining two carbon atom sites. As a starting set for optimization, LJ parameters for the carbon sites were taken from unpublished work on carbon dioxide, for the C-H site from Bourasseau et al. [BUBF02], and for the fluorine sites from a model for its bonded molecular form F2 [VSH01]. It was suﬃcient to adjust only the two fluorine LJ parameters, where the optimized parameters show almost no change in the size parameter σ and about +5% in the energy parameter ε. Vapor-liquid equilibria of the new R227ea model are given together with experimental data [DIPPR05] in Figs. 5 to 7. The agreement between the molecular model and the experimental data is good. The mean unsigned errors in vapor pressure, bubble density and heat of vaporization are 1.1, 1.0, and 7.3%, respectively, in the temperature range from 280 to 355 K, which is about 75 to 95% of the critical temperature. Table 4. Parameters of the Lennard-Jones sites of the new molecular model for R227ea Site C C-H F

σ ˚ A 2.81 3.36 2.826

ε/kB K 10.64 51.00 54.79

Fig. 5. Saturated densities of R227ea: bullet, present simulation data; line, experimental data [DIPPR05]; full triangle down, critical point derived from simulated data; open triangle up, experimental critical point [DIPPR05]; dashed line, rectilinear diameter derived from [DIPPR05]

Molecular Modeling of Hydrogen Bonding Fluids

581

Fig. 6. Vapor pressure of R227ea: bullet, present simulation data; line, experimental data [DIPPR05]; full triangle down, critical point derived from simulated data; open triangle up, experimental critical point [DIPPR05]

Fig. 7. Heat of vaporization of R227ea: bullet, present simulation data; line, experimental data [DIPPR05]

582

T. Schnabel et al.

Fig. 8. Vapor-liquid equilibria of ethanol + R227ea at 343.13 K: bullet, present simulation data; cross, experimental bubble points [KS06]; line and dashed line, Peng-Robinson equation of state with kij = −0.03 and −0.08, respectively

4 Vapor-Liquid Coexistence Curve of Ethanol + R227ea Vapor-liquid coexistence curves for the mixture ethanol + R227ea were predicted by simulation at 343.13 K throughout the full composition range [EHVH07]. Figure 8 presents this data graphically and compares it to pure substance data [DIPPR05] and the Peng-Robinson equation of state [RPC85]. Recommended experimental data were published after the close of the Industrial Fluid Properties Simulation Collective competition on the Internet [KS06]. These data were added to Fig. 8. Except the result at xR227ea = 0.274 mol/mol, which also has a very large statistical uncertainty, all simulation results at 343.13 K agree with the experimental values within the statistical uncertainty. For the isotherm 283.17 K, experimental vapor-liquid coexistence data are available to which the binary parameter kij for the Peng-Robinson equation of state was adjusted. It can be seen that the Peng-Robinson equation of state adjusted at low R227ea mole fractions at 283.17 K (kij = −0.08) is too low throughout. On the other hand, adjusting the Peng-Robinson equation of state to simulation data at low R227ea mole fractions at 343.13 K (kij = −0.03) yields the qualitatively wrong shape of the bubble line. Hence, it can be concluded that this classical approach fails while the molecular approach yields accurate predictions.

Molecular Modeling of Hydrogen Bonding Fluids

583

5 Computing Performance All vapor-liquid equilibrium simulations in Sects. 2 to 4 were carried out on the NEC SX-8 and the NEC Xeon EM64T (Cacau) with the MPI based molecular simulation program ms2 developed in our group. The parallelization of the molecular dynamics part of ms2 is based on Plimptons particle-based decomposition algorithm [Pli94]. The previous performance of ms2 was slowed down by the poor vectorization ratio of two subroutines, cf. [NJR06], which transform the interaction site coordinates from the body fixed to the space fixed system and vice versa. The performance of these two subroutines was significantly improved by changing the sequence of a double loop, so that the inner loop within these subroutines is now the longer loop. This problem had to be solved manually, while the compiler of NEC SX-5 apparently optimized this double loop automatically. An additional improvement of the ms2 code was obtained for the calculation of the chemical potential with Widom’s test particle method [Wid63] by shifting an “if clause” out of the core loop by introducing a logical variable which is set within the core loop. The contribution of a ghost particle to the chemical potential is set to zero if the ghost particle overlaps with a real particle within the shielding radius of the electrostatic interaction site. The flow trace analysis of a parallel one node job on the NEC SX-8 for a mixture state point from Sect. 4 obtained with the optimized ms2 code is given by Table 5. This table exhibits that the vector ratio of the two preTable 5. Flow trace on one node of the NEC SX-8 for a parallel molecular dynamics simulation run EXCL. MFLOPS V.OP AVER. I-CACHE O-CACHE TIME [%] RATIO V.LEN MISS MISS ms2 potential.tpotljlj chemicalpotential 49.9 4436.2 99.84 205.7 0.0642 0.6465 ms2 potential.tpotljlj force 12.1 4749.9 99.82 224.0 0.8410 3.3339 ms2 potential.tpotcc chemicalpotential 9.5 4725.8 99.89 241.9 0.0135 0.0425 ms2 potential.tpotcc force 7.9 5478.0 99.88 242.5 0.0218 1.1883 .. .. .. .. .. .. . . . . . . ms2 component.tcomponent mol2atom 0.1 1453.5 96.27 210.4 0.2349 0.2722 ms2 component.tcomponent atom2mol 0.1 1114.6 96.50 212.6 0.4444 0.3115 .. .. .. .. .. .. . . . . . . total 100.0 4314.1 99.70 216.4 2.8214 88.6945

584

T. Schnabel et al.

viously limiting subroutines labeled ms2 component.tcomponent mol2atom and ms2 component.tcomponent atom2mol improved to vectorization ratios of 96.27% and 96.50%, respectively. Table 5 also presents the excellent performance of the subroutine ms2 potential.tpotcc chemicalpotential for the calculation of the chemical potential of molecular models composed of point charges like the molecular model of ethanol. Since the diameter of the cut-oﬀ sphere is close to the simulation box length and a list of interacting particles is obtained before the calculation of the interaction, the overall vectorization ratio and sustained performance is 99.7% and 4.3 GFLOPS. Present simulation results [EHVH07, SCVLH07] were carried out on the NEC SX-8 using one node with 8 CPUs, where a pure component job lasted typically 5 hours and a mixture job typically 8 hours, respectively. On Cacau, typically 32 CPUs were used, where such jobs needed approximately the same time as on 8 CPUs of NEC SX-8.

References [BK97]

Baldridge, K., Klamt, A.: First principles implementation of solvent eﬀects without outlying charge error. J. Chem. Phys., 106, 6622 (1997). [Ber98] Berthelot, D.: Sur le M´elange des Gaz. Comptes Rendus de l’Acad´emie des Sciences Paris, 126, 1703 (1889). [BM83] B¨ uttner, R. and Maurer, G.: Dimerization of some organic acids in the gas phase. Ber. Bunsenges. Phys. Chem., 87, 877 (1983). [BUBF02] Bourasseau, E., Ungerer, P., Boutin, A., Fuchs, A.H.: Monte Carlo simulation of branched alkanes and long chain n-alkanes with anisotropic united atoms intermolecular potential. Mol. Sim., 28, 317 (2002). [CKN05] Chialvo, A.A., Kettler, M. and Nezbeda, I.: Eﬀect of the Range of Interactions on the Properties of Fluids. 2. Structure and Phase Behavior of Acetonitrile, Hydrogen Fluoride, and Formic Acid. J. Phys. Chem. B, 109, 9736 (2005). [DIPPR05] DIPPR Project 801 - Full Version. Design Institute for Physical Property Data/AIChE, 2005. [EHVH07] Eckl, B., Huang, Y.-L., Vrabec, J. and Hasse, H.: Vapor pressure of R227ea + Ethanol at 343.17 K. Fluid Phase Equilib., submitted, (2007). [IFPSC] Industrial Fluid Properties Simulation Collective, http://www.ifpsc.org. [JT97] Jedlovszky, P. and Turi, L.: A New Five-Site Pair Potential for Formic Acid in Liquid Simulations. J. Phys. Chem. A, 101, 2662 (1997). [JT99] Jedlovszky, P. and Turi, L.: Erratum to “A New Five-Site Pair Potential for Formic Acid in Liquid Simulations”. J. Phys. Chem. A, 103, 3796 (1999). [KS06] Kao, C.-P.C., Schiller, M.: http://www.ifpsc.org/ﬁles/VLEBenchmark2006.pdf. [LSN01] L´ısal, M., Smith, W.R., Nezbeda, I.: Accurate vapour-liquid equilibrium calculations for complex systems using the reaction Gibbs ensemble Monte Carlo simulation method. Fluid Phase Equilib., 181, 127 (2001).

Molecular Modeling of Hydrogen Bonding Fluids

585

¨ Lorentz, H. A.: Uber die Anwendung des Satzes vom Virial in der kinetischen Theorie der Gase. Annalen der Physik, 12, 127 (1881). [MJMT00] Min´ ary, P., Jedlovszky, P., Mezei, M. and Turi, L.: A Comprehensive Liquid Simulation Study of Neat Formic Acid. J. Phys. Chem. B, 104, 8287 (2000). [MF94] M¨ oller, D., Fischer, J.: Determination of an eﬀective intermolecular potential for carbon dioxide using vapor-liquid phase equilibria from N pT +test particle simulations. Fluid Phase Equilib., 100, 35 (1994). [NJR05] Nagel, W.E., J¨ ager, W., Resch, M.: High Performance Computing in Science and Engineering ’05. Springer, Berlin (2005). [NJR06] Nagel, W.E., J¨ ager, W., Resch, M.: High Performance Computing on Vector Systems ’06. Springer, Berlin (2006). [Pli94] Plimpton, S.: Fast Parallel Algorithms for Short-Range Molecular Dynamics. J. Comp. Phys., 117, 1 (1994). [RPC85] Robinson, D.B., Peng, D.Y. and Chung, S.Y.K.: The Development of the Peng-Robinson Equation and Its Application to Phase Equilibrium in a System Containing Methanol. Fluid Phase Equilib., 24, 25 (1985). [RGBK05] Roszak, S., Gee, R.H., Balasubramanian, K., Fried, L.E.: New theoretical insight into the interactions and properties of formic acid: Development of a quantum-based pair potential for formic acid. J. Chem. Phys., 123, 144702 (2005). [SBB93] Schmidt, M.W., Baldridge, M.W., Boatz, J.A., et al.: General atomic and molecular electronic structure system. J. Comput. Chem., 14, 1347 (1993). [SVH05] Schnabel, T., Vrabec, J., Hasse, H.: Henry’s law constants of methane, nitrogen, oxygen and carbon dioxide in ethanol from 273 to 498 K. Fluid Phase Equilib., 233, 134 (2005). [SVH06] Schnabel, T., Vrabec, J., Hasse, H.: Erratum to “Henry’s law constants of methane, nitrogen, oxygen and carbon dioxide in ethanol from 273 to 498 K”. Fluid Phase Equilib., 239, 125 (2006). [SCVLH07] Schnabel, T., Cortada, M., Vrabec, J., Lago, S., Hasse, H.: Molecular Model for Formic Acid adjusted to Vapor-Liquid Equilibria. Chem. Phys. Lett., 435, 268 (2007). [Sto05] Stoll, J.: Molecular Models for the Prediction of Thermophysical Properties of Pure Fluids and Mixtures, Fortschritt-Berichte VDI, Reihe 3, 836, VDI Verlag, D¨ usseldorf, (2005). [VSH01] Vrabec, J., Stoll, S., Hasse, H.: A set of molecular models for symmetric quadrupolar ﬂuids. J. Phys. Chem. B, 105, 12126 (2001). [Wid63] Widom, B.: Some topics in the theory of ﬂuids. J. Chem. Phys., 39, 2808 (1963).

[Lor81]

Modeling Elastic and Poroelastic Wave Propagation in Complex Geological Structures Fabian Wenzlau, Tian Xia, and Tobias M. M¨ uller Geophysikalisches Institut, Universit¨at Karlsruhe (TH), Hertzstraße 16, 76187 Karlsruhe, Germany [email protected] Summary. The description of wave propagation in complex geological structures is of fundamental importance for earth sciences, in particular solid earth geophysics. Both, modern seismology and exploration geophysics increasingly make use of numerical procedures to simulate waves within the earth crust, the mantle and hydrocarbon reservoirs. It is the main goal of rock physics research to establish relationships between seismic waves and the properties of the subsurface. For this purpose, elastic and coupled poroelastic wave equations are solved using an explicit, high-order, ﬁnitediﬀerence (FD) algorithm. The results are used for the veriﬁcation of theoretical scattering attenuation estimates in heterogeneous, eﬀectively anisotropic media and for the simulation of waves in ﬂuid-saturated porous rocks. It is shown that by using modiﬁed FD operators, the stability of the scheme can be signiﬁcantly increased when high material contrast are present within the medium. An improved understanding of wave dispersion and attenuation in heterogeneous rocks with partial saturation provides a basis for further rock physics applications.

1 Analytical and Numerical Analysis of Scattering Attenuation in Random Media For realistic, i.e. quantitative modeling wave amplitudes in engineering seismology and for amplitude studies in reflection seismics it is necessary to take into account heterogeneities in the earth crust. In random heterogeneous media, wave field energy is transferred from the vicinity of the first arrival to later arriving signals, i.e. from the primary to the seismic coda. This eﬀect is related to the randomness of wave field attributes and causes scattering attenuation. Using numerical experiments in 2-D elastic, anisotropic, random media we investigate the amplitude fluctuation of plane waves and quantify the attenuation behavior. The results are then compared to theoretical estimates.

588

F. Wenzlau, T. Xia, T.M. M¨ uller

1.1 Scattering Theory We are looking for solutions to the stochastic wave equation c20 ∂ 2 u(x, t) Δu(x, t) = 2 ∂t 1 + 2n(x)

(1)

with a heterogeneous, random field of propagation velocity c. The wavefield is then characterized by its statistical properties. For theoretical estimation of wave scattering we use the logarithmic amplitude variance of the wavefield χ = ln |A/A0 |. Scattering coeﬃcients and quality factor can then directly be related to the mean and the variance of χ [MS04]: α = σχ2 /L and Q−1 = 2α/k, with the propagation distance L and the wavenumber k. There are theoretical approaches that describe amplitude fluctuations of seismic waves in random media. By combining theories for backscattering in layered media and diﬀraction/refraction in 2-D random media, a hybrid model can be obtained, see Fig. 1. If the medium heterogeneities are not isotropic, the scattering depends considerably on the angle of incidence. Backscattering in Layered Media Amplitude variance σχ2 rises linearly with propagation distance L and can be estimated as √

π 2 ax L k 2 exp −k 2 a2x cos2 φ , (2) = σn2 σχ,1D 2

where σn2 is the variance of the medium heterogeneities and ax is correlation length and φ is the angle of incidence. This estimation is obtained on the basis of the generalized O’Doherty-Anstey formalism [SH99].

Fig. 1. Backscattering and random diﬀraction in layered and in 2-D random media. How do anisotropic heterogeneities behave? How large is the (anisotropic) scattering attenuation?

Modeling Elastic and Poroelastic Wave Propagation

589

Scattering Due to Diﬀraction and Refraction Using the Rytov transformation, the amplitude variance can be estimated in the case of weak scattering in 2-D media [MS01, MS04]. It rises faster than linear with propagation distance: √ 9: 1 π aξ 3 3 2 σχ,diff = σn2 1 + 4D2 − 1 (3) k aη D 1 − √ 4 aη 2D mit

D = 2L/(ka2η ) <9 cos2 φ a2x + sin2 φ a2y aξ = ax ay aη =

9 sin2 φ a2x + cos2 φ a2y .

Here, The quantities aξ and aη are eﬀective longitudinal and lateral correlation lengths. The ratio of correlation lengths γ = ax /ay provides an estimate of the degree of anisotropy. Again, φ is the angle of incidence, see Fig. 1. If the anisotropy is very strong, diﬀraction for small angles is negligible and backscattering dominates. In the case of weak anisotropy the relationship is vice versa. 1.2 Numerical Experiments In order to verify the theoretical estimates of wavefield scattering, numerical experiments of elastic wave propagation in random media are required, see Fig. 2. Using a higher-order FD solver [Boh02, Coh02, MSS01], we simulate plane compressional waves. The wave field is recorded at 21 geophone lines along the propagation path, examples of recorded synthetic wavefields are given in Fig. 3. After tapering and transformation of the seismograms into the frequency domain we evaluate the amplitude fluctuation along each geophone line, i.e. for diﬀerent propagation distances. We consider only the contribution of the dominant frequency.

1000

a = 45m

y [m]

y [m]

1000

500

0 0

500

1000 x [m]

1500

2000

φ = 30° 500

0 0

500

1000 x [m]

1500

2000

Fig. 2. Examples of isotropic and anisotropic velocity models used for numerical scattering experiments

F. Wenzlau, T. Xia, T.M. M¨ uller

25 50 75 100 125 150

γ=1:3, φ=45°

Time [ms]

Time [ms]

γ=1:3, φ=0° 25 50 75 100 125 150

γ=1:3, φ=90°

Time [ms]

590

25 50 75 100 125 150

Fig. 3. Synthetic seismograms of plane compressional waves after passing a heterogeneous eﬀectively anisotropic medium. With increasing angle of incidence φ diﬀraction is becoming dominant

Since the Rytov approximation is a high frequency approximation, the wavelength has to be smaller than the medium correlation length and for one wavelength, at least 12 gridpoints are required in order to prevent numerical dispersion. Furthermore, a suﬃciently high number of heterogeneities is required for the statistical evaluation to be representative. This results into model sizes of about 2150 × 8192 = 17.6 Mio. gridpoints and with a time step size of 0.2 ms a calculation of about 5000 timesteps is needed. On four two-way nodes of the HP-XC6000 linux cluster, the computing time for one simulation is about 45 minutes. Results of the numerical experiments are shown in Fig. 4. For varying angle of incidence and for two diﬀerent degrees of anisotropy, the log-amplitude mean and variance are plotted as a function of propagation distance. In this 2-D anisotropic case and for small angles of incidence, the amount of amplitude variance can be quite well described by a combination of theoretical models for backscattering and scattering attenuation due to diﬀraction. If the angle becomes larger, the initially plane wavefront is strongly perturbed and is becomes more and more diﬃcult to extract the seismic primary from the whole wavefield. Hence, an evaluation of amplitude variance is not possible anymore for large angles of incidence. Furthermore, the application of the scattering theory is restricted to the regime of weak scattering, i.e. to short propagation distances. This condition is fulfilled if aξ L σn2 (kaη )2 < 1. (4) aη aη For larger propagation distances, the amplitude variances saturate, see Fig. 5. 1.3 Conclusions of the First Part In random media with an anisotropic correlation function, diﬀraction and backscattering act together and scatter the seismic wavefield. In the case of strong anisotropy and for layered media, the eﬀect of backscattering dominates for small angles of incidence or in other words if the wave hits the

Modeling Elastic and Poroelastic Wave Propagation γ=1:3 φ=0°

γ=1:12 φ=0°

σχ

σ2χ

0.2

2

0.2 0.15

0.15

0.1

0.1

0.05

0.05

0

0

200

0

400

0

γ=1:3 φ=15°

200

400

γ=1:12 φ=15°

σχ

σ2χ

0.2

2

0.2 0.15

0.15

0.1

0.1

0.05

0.05

0

0

200

0

400

0

γ=1:3 φ=30°

200

400

γ=1:12 φ=30°

σχ

σ2χ

0.2

2

0.2 0.15

0.15

0.1

0.1

0.05

0.05

0

0

200

0

400

0

γ=1:3 φ=45°

400

Log−amp variance σ2χ

2

0.2

0.15 0.1 0.05 0

200

γ=1:12 φ=45°

0.2 Log−amp variance σχ

591

0

200 400 Distance L [m]

0.15 0.1 0.05 0

0

200 400 Distance L [m]

Fig. 4. Log-amplitude variance of diﬀerent angles of incidence φ and diﬀerent degrees of anisotropy γ. Black lines are theoretical estimations, red lines show the contribution due to backscattering, blue dashed lines show the sum of both eﬀects. The dots are mean values −χ obtained from numerical computations and crosses are the corresponding variances

592

F. Wenzlau, T. Xia, T.M. M¨ uller γ = 1:3, φ=0°

γ = 1:3, φ = 90° χ

Log−amp variance σ

Log−amp variance σ2

0.5

2 χ

0.5 0.4 0.3 0.2 0.1 0 −500

0

500 1000 1500 2000 Distance L [m]

0.4 0.3 0.2 0.1 0 −500

0

500 1000 1500 2000 Distance L [m]

Fig. 5. Behavior of log-amplitude variance for large propagation distances. Symbols and lines as in Fig. 4 but the red dashed line shows the theoretical saturation value that is attained in the long range limit

heterogeneities perpendicular to the long correlation axis. Diﬀraction occurs dominantly in the case of large angles of incidence, i.e. if the wave propagates along the long axis of the heterogeneity. Backscattering does not occur in isotropic and weak anisotropic media. Using a superposition of theoretical estimates for diﬀraction and backscattering, the scattering attenuation in random media can be estimated for arbitrary angles of incidence within the weak fluctuation regime.

2 Simulation of Waves in Poroelastic Structures Numerical modeling of seismic waves in heterogeneous porous reservoir rocks is an important tool for the interpretation of seismic surveys in reservoir engineering. Furthermore, there are various theoretical studies that derive eﬀective elastic moduli and seismic attributes like wave attenuation and velocity dispersion from complex rock properties, involving patchy-saturation and fractured media. In order to confirm and further develop rock physics theories for reservoir rocks, accurate numerical modeling tools are required. Usually, for geophysical applications, the size a of the considered cracks or fluid patches, respectively, is mesoscopic, i.e. the structures are very large when compared to the typical dimensions of the pore space but still much smaller than the seismic wavelength λ, see Fig. 6. It is well known that mesoscale eﬀects can significantly aﬀect wave propagation in porous reservoirs [Car01]. Here, we present a 2-D velocity-stress finite-diﬀerences (FD) scheme to simulate waves within poroelastic media. The mathematical background is provided by Biot theory of poroelasticity that describes the interplay between particle motion, stress and pore pressure by means of two coupled wave equations [Bio62]. The Biot equations are discretized by a FD scheme that is second-order in time and contains higher-order spatial derivative operators. Our scheme has enhanced stability properties compared to standard FD

Modeling Elastic and Poroelastic Wave Propagation

593

Fig. 6. Mesoscale structures in poroelastic media. They can be investigated using numerical FD simulations

schemes, especially in the presence of very high material contrasts. This is due to the use of rotated spatial derivative operators. 2.1 Biot Equations A description of wave- and pore pressure fields in porous rock has been established by M. A. Biot in the middle of the past century [Bio56, Bio62]. We denote by ui the displacements of the solid frame, by wi the relative displacements, defined as φ(uf − u), where porosity φ is the volume fraction of the pore space. Then, using tensor notation and neglecting source terms, Biot’s equations for a fluid saturated porous medium are given by ¨ i + ρf w ¨i = (λc + μ) uj,ji + μ ui,jj + αM wj,ji ρb u f m ¨i + ρ w ¨i + b w˙ i = αM uj,ji + M wj,ji . ρ u

(5) (6)

The bulk density ρb is determined from the density of the solid grains ρs and that of the pore fluid ρf by φρf + (1 − φ)ρs . The eﬀective fluid density ρm equals ν/φρf , where ν is the tortuosity, a parameter characterizing the pore geometry. In an isotropic porous medium, we assign by Ks and Kf the bulk moduli of the solid and fluid phases, Kdry and μ are the bulk and shear moduli of the dry porous matrix. Then the poroelastic coeﬃcient of eﬀective stress α is defined by 1 − Kdry /Ks , the coupling modulus M by [(α − φ)/Ks + φ/Kf ]−1 and the Lam´e parameter λc of the saturated matrix is given by Kdry − 2/3μ + α2 M . Finally, b is called coeﬃcient of friction i.e. the ratio of fluid viscosity and rock permeability η/κ. Together with suitable boundary conditions, equations (5, 6) describe wave propagation in heterogeneous porous solids. 2.2 Finite-Diﬀerence Implementation Velocity-Stress Formulation and Temporal Discretization For the discretization of the equations we use the velocity-stress formulation, an approach proposed for elastic wave equations by Virieux [Vir86]. Considering particle velocities instead of displacements, i.e. setting vi = u˙ i and

594

F. Wenzlau, T. Xia, T.M. M¨ uller

qi = w˙ i , Biot equations can be rewritten as a set of four evolution equations of first order for the field variables vi , qi , τij and p, where τi j is the total stress and p is pore pressure. Time discretization is based on a second order staggered scheme with vi ((n + 1/2)Δt) = v n+1/2 for the velocities and pi (nΔt) = pn for stress and pressure. Using the discrete time diﬀerential operator Dt φn = 1/Δt(φn+1/2 − φn−1/2 ) we write n+1/2

vi

n+1/2 qi

= =

τij n+1 =

p

n+1

=

n−1/2

vi

n−1/2 D qi

+ A Dj τijn + B Di pn + C q¯in +

E Dj τijn

+ F Di p

n+1/2

τijn + G (Dj vi

n+1/2

+ Di vj

(7) (8)

)+

n+1/2 n+1/2 + I Di qi ) δij + (H Di vi n+1/2 n+1/2 n p − I Di vi + J Di qi ,

(9) (10)

where Di denotes the discrete spatial derivative operator and q¯in is the time n−1/2 n−1/2 averaged relative velocity (qi + qi )/2. With the abbreviation ρ¯2 = b m f f ρ ρ − ρ ρ , the 10 update coeﬃcients coeﬃcients are ⎫ F = −(ρb Δt)/(¯ ρ2 + ρb bΔt) A = (ρm Δt)/ρ¯2 ⎪ ⎪ ⎪ ⎪ ⎪ f 2 ⎪ B = (ρ Δt)/ρ¯ G = μΔt ⎪ ⎬ f 2 (11) C = (ρ bΔt)/(¯ ρ ) H = λc Δt ⎪ ⎪ ⎪ 2 b 2 b ⎪ D = (2ρ¯ − ρ bΔt)/(2ρ¯ + ρ bΔt) I = αM Δt ⎪ ⎪ ⎪ ⎭ f 2 b E = −(ρ Δt)/(¯ ρ + ρ bΔt) J = −M Δt. The proposed scheme is similar to that proposed by Sheen [STBO06]. Spatial Discretization The fields are spatially discretized according to u(x = iΔx, y = jΔy) = uji . As in the case of temporal discretization, velocities and stresses are stored on staggered grids in order to improve the accuracy of the spatial diﬀerentiation. Two diﬀerent staggering methods are implemented: the standard staggered grid [see e.g. Vir86, Lev88] and the rotated staggered grid [SGS00]. Standard diﬀerential operators approximate derivatives along the the horizontal or vertical axis, while rotated diﬀerential operators approximate derivatives first along diagonal lines and in a second step, the derivatives along the cartesian axes are computed by a linear mapping (see Fig. 7). Using rotated operators for computing derivatives entails an increase of computational cost but the additional mapping of the derivatives consists of one single floating point operation only, regardless of the order of the FD approximation. By applying the rotated grid approach we improve the stability compared to schemes based on the standard grid approach. Following this approach, high material contrast are modelled more appropriately.

Modeling Elastic and Poroelastic Wave Propagation

595

Fig. 7. Stencils of ﬁnite diﬀerence operators. Left: Standard staggered grid. Right: Rotated staggered grid with all velocity components stored on the same gridpoints

Source Implementation and Boundary Conditions The poroelastic formulation involves two distinct phases present within a porous medium and forces may act on each of them. A body force source acting on the solid frame is available as well as pressure sources acting on either the bulk material, the pore fluid or a combination of both. By default, at all four boundaries of the rectangular domain, an absorbing layer is applied in order to reduce reflections from the boundaries back into the medium. Within this layer, wave motion experiences an exponential damping. Alternatively, a traction-free surface condition or periodic boundaries are used. Parallelization The code is parallelized using the domain decomposition technique and information is exchanged between adjacent cells via the message passing interface MPI [Boh02], see Fig. 8.

Fig. 8. Schematic view of the parallelization using domain decomposition. After [Boh02]

596

F. Wenzlau, T. Xia, T.M. M¨ uller

2.3 Numerical Experiments We give three examples to demonstrate the range of applicability of our numerical scheme and to indicate particular challenges of modeling coupled wave equations. In the first two numerical examples, the source is a Ricker wavelet with center frequency 25kHz and material properties are given in Table 1. We choose a spatial grid spacing of Δx = 1mm, the model size is 800 × 800 = 640000 gridpoints. Time step is chosen optimally to ensure stability, in our case Δt = 20ns. For simulation times of 200μs 10000 timesteps are required. On four nodes, the computation time is about 30 minutes. A third example illustrates that the full solutions of poroelastic wave equations not only contains propagating waves, but at low frequencies also diﬀusion processes. Since diﬀusion and propagation occur on very diﬀerent time scales, the numerical problem becomes stiﬀ [CQG95]. Table 1. Material properties used in the numerical examples

Ks (GP a) Kdry (GP a) μ(GP a) φ ν κ(mD) ρs (t/m3 )

sandstone

crack

35 5 11 0.2 2.0 1000 2.65

0 0 0 1.0 1.01 ∞ 0.0

Kf (GP a) ρf (t/m3 ) η(mP as)

water

gas

2.25 1.0 1.0

0.00014 0.001 0.02

Gas-Water Contact in Partially Saturated Sandstone It is well known that reservoir rocks show patchy saturation in the presence of two or more fluid phases, usually liquid and gaseous. Up to date, there was no accurate modeling technique that allows to determine relationships between saturation and P -wave velocity [HPC03]. In this example we show 2D snapshots and seismograms from quasi 1-D wave propagation experiments within a medium having a gas saturated inclusion, embedded in water saturated sandstone. Due to the low bulk modulus of the gaseous phase, no pore pressure is built up within the dry inclusion during the passage of the incident wave, resulting in a pore pressure gradient that drives fluid flow across the interface. A slow Ps -wave arises that is strongly dissipated, see snapshots in Fig. 9.

Modeling Elastic and Poroelastic Wave Propagation 120 µs

0

0

100

100

200

dry

y (mm)

y (mm)

80 µs

597

300

200 300

wet 400

400

50 125 200 275 350 x (mm) 160 µs

50 125 200 275 350 x (mm) 200 µs

0

0

100

100 y (mm)

y (mm)

PfPf

200 300

PfPs

200 300

P

f

400

50 125 200 275 350 x (mm)

400

50 125 200 275 350 x (mm)

Fig. 9. Pore pressure ﬁeld snapshots associated with a plane wave in the vicinity of a circular gas inclusion. Note the real high contrast in bulk moduli of the two ﬂuid phases, given in Table 1

Cracks in Porous Media The influence of cracks on eﬀective poroelastic parameters is a current research topic in rock physics. Mechanically, cracks are very thin, soft and highly permeable heterogeneities within a porous rock matrix. They are characterized by their thickness and shape as well as their distribution in the rock matrix. The presence of cracks and their geometry cause characteristic frequencydependent dispersion and attenuation [BMG06] of seismic waves. From a numerical point of view, the modeling of high contrasts in poroelastic parameters between cracks and the surrounding matrix are challenging. Using the rotated staggered grid approach, stable solutions are obtained, provided that the time step is chosen adequately. Figure 10 shows the simulation of a plane wave interacting with an elliptically shaped crack in wet porous sandstone. The results confirm that high-contrast FD modeling is a valuable tool to simulate wave propagation in cracked poroelastic media. Coupled Diﬀusion and Propagation One of the main results of Biot’s theory is the existence of a slow compressional wave mode in addition to the compressional and shear wave modes in

598

F. Wenzlau, T. Xia, T.M. M¨ uller 120 µs 0

100

100 crack

200 300 400

y (mm)

y (mm)

80 µs 0

200 300

wet sandstone

400

50 125 200 275 350 x (mm)

50 125 200 275 350 x (mm) 200 µs

0

100

100 y (mm)

y (mm)

160 µs 0

200 300 400

200 300

50 125 200 275 350 x (mm)

400

PfPf PfPs Pf 50 125 200 275 350 x (mm)

Fig. 10. Same as Fig. 9, but with a medium containing a mechanically soft crack. During compression, ﬂuid is squeezed out from the crack into the surrounding matrix, resulting in the excitation of slow Ps -waves at the crack-matrix interface, as in the case of the gas-water contact

elastic solids. At very high frequencies this slow Ps -wave has the character of a propagating wave mode, while at low frequencies, it shows diﬀusive behavior. Both regimes are separated by a critical relaxation frequency ωc = φη/(νρf κ), where φ is the medium porosity, ν the tortuosity of the pore space, κ is the permeability. ρf and η are fluid density and viscosity, respectively. From the numerical point of view, the poroelastic wave equations become stiﬀ for low frequencies, i.e. they possess solutions on very diﬀerent temporal and spatial scales [CQG95]. The very slow diﬀusion process – usually occurring across interfaces within the medium – needs a very fine grid spacing to be accurately resolved, while a very short time step is required to ensure that propagating waves are well-sampled. The stiﬀness of the equations therefore entails high computational cost. However, a consideration of coupled diﬀusion and propagation processes is necessary to describe frequency-dependent eﬀects in porous rocks. The example given in Fig. 11 shows that potentially, numerical implementations of Biot’s equations describe not only propagating waves but contain also completely the low-frequency diﬀusive behavior of the slow P -wave. For that purpose, a fluid injection source, i.e. a step function in pore pressure was

Modeling Elastic and Poroelastic Wave Propagation time = 0.2 ω−1

time = 2 ω−1

time = 20 ω−1

c

c

1

1

0.8

0.8

0.8

0.6 0.4 0.2 0

pore pressure

1 pore pressure

pore pressure

c

0.6 0.4 0.2 0

0

0.01 0.02 distance

0.03

599

numerical diffusion inertial

0.6 0.4 0.2 0

0

0.01 0.02 distance

0.03

0

0.01 0.02 distance

0.03

Fig. 11. Pore pressure responses as a function of distance for a step loading at x = 0. The numerical solution of fully dynamical Biot’s equation is compared with the limiting cases of a purely inertial wavefront (x = vPs t) and a the solution of the diﬀusion equation that is a complementary error function [Rud86]

used to excite a homogeneous wet sandstone model. By varying the medium permeability κ and therefore ωc , the spatiotemporal response is either more of inertial type with a propagating wave front or of diﬀusive type. The theoretical diﬀusion solution in Fig. 11 was taken from [Rud86]. 2.4 Conclusions of the Second Part A velocity-stress finite-diﬀerences scheme has been implemented to simulate wave propagation in heterogeneous, poroelastic structures. It solves Biot’s equations of poroelasticity for a wide range of frequencies. The scheme is second order in time and includes higher-order spatial diﬀerentiation operators both on standard and rotated staggered grids. The rotated grid approach is meant to improve the stability of the algorithm when high material contrasts are present within the medium. This is demonstrated by a series of numerical examples that show poroelastic eﬀects occurring especially on the scale of several centimeters and that are therefore mesoscale for the seismic frequency range. One particularity of the poroelastic wave equations is their stiﬀ behavior at very low frequencies, resulting in an additional computational demand when compared to elastic or viscoelastic wave propagation codes. The results are promising and show that this numerical algorithm may become a powerful tool for investigating waves within complex poroelastic formations. Fields of application are the simulation of waves within layered structures, quantification of the eﬀect of cracks and fractures on the wavefield, as well as investigating seismic signatures of patchy-saturation and of so-called double-porosity media. Acknowledgements The poroelastic FD program presented here has been developed on the basis of a parallel 3-D viscoelastic algorithm and the author Thomas Bohlen is

600

F. Wenzlau, T. Xia, T.M. M¨ uller

acknowledged for providing the code. The simulations were performed on the computer cluster HP-XC6000 at the High Performance Computing Center Stuttgart (HLRS) under the grant number Poromod. We also thank the DFG Emmy-Noether programme and the sponsors of the PHASE consortium for supporting the research presented in this paper.

References [Bio56] Maurice A. Biot. Theory of propagation of elastic waves in a ﬂuidsaturated porous solid. I. Low frequency range. J. Acoust. Soc. Am., 28(2):168–178, March 1956. [Bio62] Maurice A. Biot. Mechanics of deformation and acoustic propagation in porous media. J. Appl. Phys., 33(4):1482–1498, April 1962. [BMG06] Miroslav Brajanovski, Tobias M. M¨ uller, and Boris Gurevich. Characteristic frequencies of seismic attenuation due to wave-induced ﬂuid ﬂow in fractured porous media. Geophys. J. Int., 166:574–578, 2006. [Boh02] Thomas Bohlen. Parallel 3-D viscoelastic ﬁnite diﬀerence seismic modelling. Computers & Geosciences, 28:887–899, 2002. [Car01] Jos´e M. Carcione. Wave fields in real media. Pergamon, 2001. [Coh02] Gary C. Cohen. Higher-Order Numerical Methods for Transient Wave Equations. Springer, Berlin, Heidelberg, New York, 2002. ISBN 3-54041598-X. [CQG95] Jos´e M. Carcione and Gerardo Quiroga-Goode. Some aspects of the physics and numerical modeling of Biot compressional wave. J. Comp. Acoust., 4:261–280, 1995. [HPC03] Hans B. Helle, Nam H. Pham, and Jos´e M. Carcione. Velocity and attenuation in partially saturated rocks: poroelastic numerical experiments. Geophys. Prosp., 51:551–566, 2003. [Lev88] Alan R. Levander. Fourth-order ﬁnite-diﬀerence PS-V seismograms. Geophysics, 53(11):1425–1436, November 1988. [MS01] Tobias M. M¨ uller and Serge A. Shapiro. Most probable seismic pulses in single realizations of 2-d and 3-d random media. Geophys. J. Int., 144:83–95, 2001. [MS04] Tobias M. M¨ uller and Serge A. Shapiro. Scattering attenuation in randomly layered structures with ﬁnite lateral extent: A hybrid Q model. Geophysics, 69:1530–1534, 2004. [MSS01] Tobias M. M¨ uller, Christoph Sick, and Serge A. Shapiro. Wave propagation in heterogeneous media. part 2: Attenuation of seismic waves due to scattering. In E. Krause and W. J¨ ager, editors, High Performance Computing in Science and Engineering ’01, Berlin, 2001. Springer Verlag. [Rud86] John W. Rudnicki. Fluid mass sources and point forces in linear elastic diﬀusive solids. Mechanics of Materials, 5:383–393, 1986. [SGS00] Erik H. Saenger, Norbert Gold, and Serge A. Shapiro. Modeling the propagation of elastic waves using a modiﬁed ﬁnite-diﬀerence grid. Wave Motion, 31:77–92, 2000.

Modeling Elastic and Poroelastic Wave Propagation

601

[SH99] Serge A. Shapiro and Peter Hubral. Elastic waves in random media. Springer Verlag, Heidelberg, 1999. [STBO06] Dong-Hoon Sheen, Kagan Tuncay, Chang-Eob Baag, and Peter J. Ortoleva. Parallel implementation of a velocity-stress staggered-grid ﬁnitediﬀerences method for 2-D poroelastic wave propagation. Computers & Geosciences, 32:1182–1191, 2006. [Vir86] Jean Virieux. P-SV wave propagation in heterogeneous media: Velocitystress ﬁnite-diﬀerence method. Geophysics, 51(4):889–901, April 1986.

Whole-Mantle Convection, Continent Generation, and Preservation of Geochemical Heterogeneity Uwe Walzer1 , Roland Hendel1 , and John Baumgardner2 1

2

Institut f¨ ur Geowissenschaften, Friedrich-Schiller-Universit¨ at, Burgweg 11, 07749 Jena, Germany [email protected] Dept. Earth Planet. Science, University of California, Berkeley, CA 94720, USA

Summary. The focus of this paper is numerical modeling of crust-mantle diﬀerentiation. We begin by surveying the observational constraints of this process. The present-time distribution of incompatible elements are described and discussed. The mentioned diﬀerentiation causes formation and growth of continents and, as a complement, the generation and increase of the depleted MORB mantle (DMM). Here, we present a solution of this problem by an integrated theory that also includes the thermal solid-state convection in a 3-D compressible spherical-shell mantle heated from within and slightly from below. The conservation of mass, momentum, energy, angular momentum, and of four sums of the number of atoms of the pairs 238 U206 Pb, 235 U-207 Pb, 232 Th-208 Pb, 40 K-40 Ar is guaranteed by the used equations. The pressure- and temperature-dependent viscosity is supplemented by a viscoplastic yield stress, σy . No restrictions are supposed regarding number, size, form and distribution of continents. Only oceanic plateaus touching a continent have to be united with this continent. This mimics the accretion of terranes. The numerical results are an episodic growth of the total mass of the continents and acceptable courses of the curves of the laterally averaged surface heat ﬂow, qob, the Urey number, U r, and the Rayleigh number, Ra. In spite of more than 4500 Ma of solid-state mantle convection, we typically obtain separate, although not simply connected geochemical mantle reservoirs. None of the reservoirs is free of mixing. This is a big step towards a reconciliation of the stirring problem. As expected, DMM strongly predominates immediately beneath the continents and the oceanic lithosphere. Apart from that, the result is a marble-cake mantle but DMM prevails in the upper half of the mantle. We ﬁnd Earth-like continent distributions in a central part of Ra-σy plot obtained by a comprehensive variation of parameters. There are also Ra-σy areas with small deviations of the calculated total continental volume from the observed value, with acceptable values of U r and with realistic surface heat ﬂow. It is remarkable that all of these diﬀerent acceptable Ra-σy regions share a common overlap area. We compare the observed present-time topography spectrum and the theoretical ﬂow 2 . spectrum n1/2 × (n + 1)1/2 × vn,pol

604

U. Walzer, R. Hendel, J. Baumgardner

1 Introduction: Whole-Mantle Convection and Geochemistry Solid state convection currently explains the thermal evolution of the Earth and the plate-tectonic regime in a satisfying manner (Schubert et al. [94]). How well the Earth’s chemical evolution is accounted for has not been so clear. Chemical diﬀerentiation alters the distribution of radioactive elements and volatiles and generates geochemical heterogeneity. On the other hand, convection diminishes and annihilates chemical heterogeneities by stirring in low-viscosity regions. Convection recycles the oceanic crust and transports and mixes mantle lithosphere into the deeper mantle. Mixing is suppressed and chemical signatures tend to be preserved, on the other hand, where mantle viscosity is high. Generation, preservation and destruction of heterogeneities are obviously influenced by the interplay between chemical diﬀerentiation, convection and secular cooling of the Earth’s mantle. Only the present existence of remnants of the primordial mantle is a controversial question. It is evident that the chemical evolution of the mantle cannot be modeled without accounting for the convective process and the mixing it generates. On the other hand, the fact that chemical diﬀerentiation causes the distribution of heat-producing elements in the mantle to be non-uniform, no doubt, influences convection. This paper seeks to account for both diﬀerentiation and convecting/mixing together. We check our overall model against reality by comparing the evolution of continents, oceanic plateaus and of the MORB source that we obtain in our models with observations. Modern seismic techniques document convincingly that subducted oceanic lithosphere penetrates into the lower mantle. Other seismic methods give strong support to the conclusion that at least some hotspots rise upward and penetrate the 660-km phase boundary (Van der Hilst et al. [109], Grand et al. [39]). Moreover, diﬀerential travel times may be used to determine the vertical distance between the olivine-wadsleyite discontinuity and the ringwoodite-perovskite discontinuity and its lateral variation. From this it has been deduced that at least some hotspots penetrate the 660-km boundary (Bina [14]). At the corresponding places, a thinning of the transition zone thickness has been observed as expected from corresponding ClausiusClapeyron slopes (Schubert et al. [94], chapter 4.6.2). It is evident that the topography of the mentioned seismic discontinuities can be explained by thermal perturbations of phase transformations in a peridotite mantle (Helffrich and Wood [44]). There are also direct evidences of plumes penetrating the 660-km discontinuity by seismic tomography. These studies demonstrate that the transition zone is not impermeable either for slabs or for plumes. Tomography suggests that some plumes originate in the D′′ layer. Other plumes probably originate from shallower depths, e.g., near the 660-km discontinuity (Courtillot et al. [29], Montelli et al. [75]).

Mantle Convection, Continent Generation and Geochemical Heterogeneity

605

Some have proposed that the bulk compositions of the upper and the lower mantle are diﬀerent, especially in the Mg/Fe and Mg/Si ratios, and that the 660-km discontinuity is not only a phase transition from ringwoodite to perovskite plus magnesiow¨ ustite and from majorite garnet to Al-bearing Mg-rich perovskite (Hirose [46]) but also a discontinuity in the major-element abundances. Jackson and Rigden [55] showed, however, that the essential features of the seismological models can be well explained by the phase transitions of a pyrolite model, as a special case, and that, in the general case, the current knowledge of the seismic structure of the mantle is consistent with a uniform major-element composition of the mantle with the probable exception of the D′′ layer. From a geochemical point of view, Palme and O’Neill [80] concluded that, concerning the major elements, the whole Earth’s mantle is compositionally uniform if we consider averages of large volumes. If the abundances, normalized to Ti and CI, are drawn versus equidistant locations of the carbonaceous chondrites CI, CM, CO, CK, CV and the Earth’s mantle, then constant-value lines result for Al and Ca and monotonously decreasing curves arise from this procedure for other major and moderately volatile elements. So, the Earth’s mantle may be considered as an extension of the carbonaceous chondrite trend. Only the Cr content is too low in the present mantle as a result of Cr partitioning into the outer core. Bina [14] also concluded that the seismic arguments for a chemical boundary of the major elements at the 660km discontinuity have faded. An isochemical mantle is consistent with seismic results. However, this conclusion applies only for the major elements since it is these elements that constrain the bulk physical parameters such as compressional velocity and density which can be computed for a perfect lattice. Major elements are the essential constituents of rock-forming minerals. Trace elements often act only as tracers (Hofmann [50]). However, other important physical quantities such as shear viscosity, seismic shear velocity, electrical conductivity, etc. depend on the concentrations of Schottky holes, Frenkel defects and dislocations which, in turn, strongly depend on small amounts of volatiles. Also small concentrations of U, Th and K influence the heating rate and hence the buoyancy that drives convection. To summarize this introduction, we conclude that there is strong evidence for whole-mantle convection. In regard to the major elements, averaging over large volumes, current evidence points to a chemically homogeneous mantle except for the lithosphere and its counterpart at CMB, the D′′ layer. It is quite possible that this simple picture may have to be modified somewhat since Trampert et al. [106] showed that compositional variation aﬀects the seismic velocities, vp and vs , not only in the D′′ layer but also in most of the lowermost mantle.

606

U. Walzer, R. Hendel, J. Baumgardner

2 Observational Constraints 2.1 The Evolution of Continents, the MORB Source and Other Possible Geochemical Reservoirs Geochemical Preliminary Examinations The continental crust (CC) (Hofmann [48], McCulloch and Bennett [71], Rudnick and Gao [91]) is extraordinarily enriched in incompatible elements compared to the bulk silicate earth (BSE). Therefore it is to be expected that there are one or more regions in the mantle that are depleted in incompatible elements in a complementary manner. Mid-oceanic ridge basalt (MORB) is supplied by a source region that has to be situated immediately below the oceanic lithosphere since, independently where oceanic plates are spreading apart, MORB is surprisingly homogeneous in its trace element composition. This source region appears to be very well stirred in comparison to that of ocean-island basalts (OIB). All`egre and Lewin [4] emphasize the homogeneity of MORB. Let R/RA denote the mass ratio 3 He/4 He of rock divided by the present-day 3 He/4 He value of the atmosphere. Barfod et al. [8] showed that the R/RA values of ridge basalts are around 8 with a small standard deviation. HIMU-like OIBs have lower R/RA values with greater standard deviations. Nearly all other OIBs have augmented R/RA ratios with large standard deviations. However, Hofmann [50] pointed out that also the depleted MORB mantle (DMM) is isotopically not entirely uniform. Hofmann [50] reported a fundamental observation: The abundances of the elements Rb, Pb, U, Th, K, Ba, La, Nb, Sr, Na, Yb, Al, Ca, Si, Sc, Mn, Co and Mg, normalized by their BSE abundances (McDonough and Sun [73]) have been plotted versus their degree of compatibility for three reservoirs: continental crust (Rudnick and Fountain [90]), MORB (Su [99]), and DMM. If the generation of CC would be a one-act diﬀerentiation process then the result could be easily explained as follows: The more incompatible an element is the stronger it is enriched in CC since it rose within the melt to this place. DMM is the complementary mantle part where the abundances of the most incompatible elements have the lowest values. If this depleted mantle region is again partly melted, the diﬀerentiation product is a typically oceanic crust (MORB) with an observed normalized concentration curve of the mentioned elements that is between the CC and the DMM curve. So compared to DMM, the MORB curve is also somewhat enriched. However, the formation of CC is not a single-stage process, but a multistage process. The enrichment of CC happens by three kinds of processes: (a) partial melting in the uppermost parts of the mantle and ascent into the oceanic crust, (b) dehydration and decarbonation in subduction zones with metasomatic transport (Hofmann, [50]), and (c) generation of plateau basalts. The source rocks for plateau basalts are relatively enriched parts of the mantle that penetrated into the uppermost mantle layer that has now usually

Mantle Convection, Continent Generation and Geochemical Heterogeneity

607

a DMM composition. Plateau basalts are divided into three classes: continental flood basalts, oceanic plateaus and ocean basin flood basalts (Coffin and Eldholm [25]). The latter two classes of large igneous provinces (LIP) are driven near the continental margin by the conveyor-belt-like oceanic lithosphere and accreted to the continent in the zones of andesitic volcanism. Generation of LIPs is episodically distributed over the time axis of the Earth’s history. LIP volcanism was dominant during the Alpidic orogenesis, in contrast to the MORB volcanism in the orogenetically somewhat more quiet period of today. On other terrestrial planets of our solar system, LIP volcanism appears to be the dominant form. A terrestrial example is the Ontong Java Plateau. Its basalt developed from a 30% melting of a peridotite source. It can only be achieved by decompression of hot material with a potential temperature > 1500 ◦C beneath a thin oceanic lithosphere. As expected, the Ontong Java Plateau basalt is enriched in U, Th and K in comparison to Pacific MORB. It is also isotopically and chemically distinct from it (Fitton et al. [34]). Furthermore, the present-day proportion of DMM of the mass of the mantle is not exactly known since it depends on which element is used and on its assumed abundance in CC and DMM. If samarium and neodymium are not so extremely enriched in CC, then smaller volumes of DMM would be suﬃcient. 50% of depleted mantle is derived from Cs, Rb, Th and K. Bennett [10] estimated that between 30% and 60% of the mantle is depleted if an intermediate degree of depletion of DMM is assumed. Hofmann [50] deduced a depleted reservoir of the mantle between 30% and 80%. Van Keken et al. [110], however, itemized some strong arguments against the opinion that the rest of the mantle is simply a BSE reservoir. Hofmann et al. [51] investigated the mass ratio Nb/U of MORB and IOB and found a very similar average. The same conclusion applies for Ce/Pb. Rudnick and Fountain [90] derived Nb/U = 47 ± 11 for MORBs, Nb/U = 52 ± 15 for OIBs but Nb/U = 8 for CC. The trace element ratios Ce/Pb, Nb/U, and Nd/U proved to be nearly identical for MORB and OIB and to be nonprimitive, i.e., there is no correspondence with BSE. Evidently, it is not possible to derive the BSE abundances simply from the MORB and OIB abundances. Some authors concluded from the observations that there is no present-day primordial material at all in the mantle. However, Hofmann [50] emphasized that only mass ratios of similarly incompatible pairs of elements are suitable for addressing this issue. When the isotope ratio of a single element is plotted against the isotope ratio of another element, for many basaltic rocks from around the world, including OIBs and MORBs, they tend to scatter into distinct mixing lines. At each end of such a line is an extremal case. If, for example, the present-day mass ratio of 206 Pb/204 Pb is plotted versus 87 Sr/86 Sr then four distinctly separated mixing lines typically appear, the ends of which are called HIMU, DMM, EM1 and EM2 (cf. Hofmann [50], Fig. 15). HIMU stands for high μ where μ = (238 U/204 P b)τ =0 (Houtermans [53], Zindler and Hart [124]). τ is the age. One interpretation is that these end compositions represent only extremes

608

U. Walzer, R. Hendel, J. Baumgardner

of a continuum of mixtures of isotopic compositions that are distributed on various spatial scales throughout the mantle. The other opinion is that these end-member compositions represent separate distinct reservoirs in diﬀerent regions of the mantle. However, the observed compositions are not evenly distributed in 206 Pb/204 Pb–87 Sr/86 Sr space. Instead, four linear trends dominate. Not only this observation but also the following hint by Hofmann [49] argues for the second option, namely that nearly pure HIMU basalts are found not only in a single ocean island group like the Cook-Austral chain but also on St. Helena on the other side of the globe. The latter argument is not invalidated by the fact that Mangaia, an individual island of the Cook chain, is distinctly diﬀerent from the observed HIMU of the other Cook-Austral islands and St. Helena (Stracke et al. [98]). HIMU has not been observed in MORB and is rare in OIB. It could represent ancient recycled oceanic crust (Stracke et al. [98]). EM1 and EM2 are more enriched in very incompatible elements compared to HIMU. EM2 shows maximum 87 Sr/86 Sr values at nearly constant 206 Pb/204 Pb≈19 whereas EM1 forms a distinctly separated line beneath the EM2 line and is distinctly separated regarding 206 Pb/204 Pb. The usual explanation for EM1 is that it is generated by recycling of oceanic crust plus portions of lower CC or subcontinental lithosphere or ancient pelagic sediments. EM2 is customarily explained by recycling of oceanic crust with portions of the upper CC (Willbold and Stracke [116]). The latter authors deny the distinction between EM1 and EM2. They propose a flowing transition and explain the common EM by subduction of oceanic crust with variable proportions of lower and upper CC. If all mixing arrays are plotted, e.g. in a three-dimensional 206 Pb/204 Pb– 87 Sr/86 Sr–143 Nd/144 Nd diagram then all mixing lines aim at a small volume called FOZO (or focal zone) according to Hart et al. [42]. There are also related proposals: PREMA (W¨ orner et al. [118], Zindler and Hart [124]), PHEM (Farley et al. [33]) and C (Hanan and Graham [41]). FOZO is significantly more radiogenic in lead isotopes than DMM, moderately more radiogenic in Sr and less radiogenic in Nd and Hf (Hofmann [50]). Furthermore, FOZO has a higher 208 Pb/206 Pb ratio than HIMU. Although FOZO is evidently produced by subduction and is by no means primordial, FOZO can play the part of the rich principal reservoir in our present convection-diﬀerentiation model, rich in U, Th and K. There would be a broad mixing zone between the FOZO and the DMM reservoir. FOZO would be the main source in the OIBs while EM1, EM2 and HIMU represent contributions from minor reservoirs. The latter ones have not to be joined to one region each. The diﬀerent contributions of the various minor reservoirs generate the large isotopic and chemical standard deviations of OIBs, large in comparison to that of MORBs. Stracke and Hofmann [98] redefined a new FOZO that is similar to the traditional FOZO according to Hart et al. [42]. They propose that this new FOZO could be a ubiquitously dispersed, small-scale component in the entire mantle. We remark that, according to our present dynamical model, the percentage of FOZO in the upper half of the mantle should be less than in the lower half.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

609

This result corresponds with Wilson’s and Spencer’s [117] conclusion that FOZO is the characteristic signal of lower mantle plumes. Geochemical models that totally abandon the assumption of BSE remnants in the present-day mantle are unable to explain the observation that the flux of 3 He is unidirectional, that, e.g., Reykjanes Ridge has a 3 He content 15 times higher than that of DMM and that the averaged 3 He concentration in the plume sources is 4 times higher than that of DMM. Hilton and Porcelli [45] are convinced that at present 1039-2270 mol·a−1 primordial 3 He leaves the Earth. Trieloff and Kunz [107] systematically discuss the problem of noble gases in the Earth’s mantle. One proposal for the source of primordial noble gases has been the Earth’s core (Tolstikhin and Marty [104], Porcelli and Halliday [85]). This proposal is unconvincing since the viscosity of the outer core is between 1 and 100 Pa · s and it circulates with velocities between 10 and 30 km/a. So, each volume element of the outer core has been in frequent contact with the CMB during the Earth’s evolution: If the mantle is unable to retain its 3 He, then the outer core cannot retain its inventory either. But for the dynamical theory, presented in this paper, it is irrelevant where exactly the 3 He source region is situated. It is only important that there are regions within the present mantle which have higher abundances of U, Th and K than occur in the DMM. MORBs and OIBs are chemically distinct and their standard deviations are diﬀerent. So, it is improbable that both of them originate from the same quasi-homogeneous DMM. Moreover, the present-day heat output of the mantle is 36 TW: it is not possible to produce such a large amount of heat from a mantle that is entirely a DMM reservoir (Bercovici and Karato [12]). A more detailed argument is as follows: The Earth’s present-day heat loss is 44 TW (Pollak et al. [83]). 4.89.6 TW of it are produced by the CC (Taylor and McLennon [102], Rudnick and Fountain [90]), Porcelli and Ballentine [84]). The diﬀerent numbers stem from diﬀerent assumptions on the average CC abundances of radioactive elements. The contribution of the Earth’s core is between 3 and 7 TW (Buffet et al. [18]. If the DMM abundances would be by a factor of 1/2.6 lower than the BSE abundances according to Jochum et al. [56] then a whole-mantle DMM would generate only 7.2 TW (Porcelli and Ballentine [84]). The contribution of secular cooling is between 21.8 TW and 17.8 TW. If the terms of CC, the core and secular cooling are subtracted from the observed 44 TW then values between 9.6 and 14.4 TW remain. The latter numbers exceed the 7.2 TW of a hypothetical mantle that is comprised entirely of pure DMM. Therefore, the mantle must contain at least one reservoir that is enriched in U, Th and K. Continental Crust: Massive Early Formation or Gradual or Episodic Growth? The isotopic compositions of lunar rocks (Norman et al. [77]) and of SNC meteorites (Brandon et al. [17], Nyquist et al. [78], Bogard et al. [15]) show rapid chemical diﬀerentiation of Moon and Mars within the first 200 Ma of

610

U. Walzer, R. Hendel, J. Baumgardner

their existence. Therefore it is highly probable that not only did an iron core form early but a silicate crust did as well. It has been argued that not only the Moon but also the Earth posessed a magma ocean early in their histories (Stevenson [97], Tonks and Melosh [105]). In the case of the Earth, it is not clear whether all or only part of the mantle was melted. The generation of the metallic cores likely occured within the first 30 Ma associated with the decay of short-lived isotopes (Cameron [23], Kleine et al. [65]). It was a controversial question whether the total mass of the Earth’s crust continued to grow during its later evolution taking no account of the mentioned early stage. Armstrong [6] and Bowring and Housh [16] advocated that the full amount of current mass of CC formed very early, before an age of 4 Ga. They assumed that CC has only been recycled since that time. On the other hand, there is strong evidence that juvenile CC has indeed been formed during the Earth’s subsequent evolution. Therefore, models involving episodic or continuous continental growth have been proposed (Taylor and McLennon [102], McCulloch and Bennett [71, 72], Condie [28], Bennett [10]). The 147 Sm–143 Nd isotopic system and the 176 Lu–176 Hf decay both suggest that chemical diﬀerentiation of CC has persisted over the whole of geological history in addition of a pulse of diﬀerentiation during the Earth’s earliest history (Bennett [10]). The of Sm/Nd ratio was not altered during the accretion of the Earth since both elements are refractory. Neither was this ratio modified during core formation since both elements are lithophile. Both conclusions also apply for Lu/Hf. Therefore, we may conclude that these two ratios in BSE are chondritic. However, the quantity εN d as a function of time displays an ongoing chemical evolution of DMM distributed over the ≥ 4.49 × 109 a history of the solid silicate mantle. Here εN d = whole 143 ( N d/144 N d(t)sample )/(143 N d/144 N d(t)BSE ) − 1 × 104 with t the crystallization age. εN d of the depleted mantle appears to have increased nonuniformly, probably episodically, and reaches its maximum value of εN d = 10 ± 2 for the present epoch (Hofmann [50]). Observed isotopic compositions of Nd point strongly to complex processes of depletion and crustal recycling. Similarly, the increase of εHf of DMM leads to the same conclusion. Here εHf = (176 Hf /177 Hf(t)sample )/(176 Hf /177 Hf(t)BSE ) − 1 × 104 . The quantity εHf increases non-uniformly and reaches its present value of εHf = 16±4 (Salters and White [92], Vervoort and Blichert-Toft [111], Amelin et al. [5], Bennett [10]). A similar non-uniform evolution can be shown for the 187 Os/188 Os ratio using mantle-derived samples. Condie [28] further demonstrated a progressive increase in the Nb/Th ratio for the depleted mantle throughout the Earth’s history. If we now consider the problem of CC evolution as it relates to the mantle components DMM, FOZO, HIMU, DM1 and DM2, discussed in Sect. 2.1. then we must infer that these components developed by diﬀerent diﬀerentiation processes, subduction and convective stirring. Also if we view the problem from this perspective it seems improbable that CC formed exclusively during the Earth’s initial history. Subduction continuously entrains heterogeneous material that subsequently sinks to the bottom

Mantle Convection, Continent Generation and Geochemical Heterogeneity

611

of the mantle. The composition of this basal layer almost certainly changes with time (Davies [31], Gurnis [40], Coltice and Ricard [26], Albar`ede and Van der Hilst [2]). Arndt [7] provides a review of similar and alternative views of the mantle’s chemical evolution. 2.2 Further Observational Constraints In contrast with the other terrestrial planets, the Earth has a pronounced double-peaked hypsometric curve. This relatively sharp division of the surface of the solid Earth into continents and ocean basins reflects a contrast in chemical composition. The thickness of the oceanic crust is only 0–7 km whereas the continental crust is distinctly thicker. Its thickness depends on its age: Archaic CC, older than 2.5 × 109 a, has an average thickness of 41 km whereas Early Proterozoic CC that is older than 1.6 × 109 a has an average thickness of 43 km. Late Proterozoic and Phanerozoic CC has a mean thickness of 35 km. The continental lithospheric mantle, attached to the CC, has essentially the same age. Isotopic investigations of the continental lithospheric mantle show that it has been isolated from the convecting mantle since the corresponding time of CC formation (Kramers [66], Pearson et al. [82]). There are, of course, alterations due to metasomatism (Alard et al. [1], Burton et al. [22]). The oceanic lithosphere moves in a piecewise plate-like manner and subducts. That is why there is no oceanic plate older than Upper Jurassic. Therefore, the upper CC is the only extensively accessible record of information on the main part of the Earth’s history. Only relatively small parts of the continent record have been removed by subduction or by delamination of the continental lithospheric mantle. Reymer and Schubert [87] summarized continental crustal growth curves of diﬀerent authors and presented their own continuous growth curve. Taylor and McLennon [102] emphasized what they recognize as major episodes of crustal growth. Condie [28] also emphasizes the episodicity of this process. O’Nions and Tolstikhin [79] show that convective avalanches could be responsible for episodic crustal growth. It is plausible that episodicity in convection indirectly causes the episodes of growth of CC. Yuen and Malevsky [123] and Yuen et al. [122] pointed out that mantle convection can operate in the hard turbulence regime at early stages of planetary thermal evolution and subside to a present-day state of soft turbulence. It is also to be expected that the rate of chemical diﬀerentiation depends directly on mantle convection. So, if convection displays episodes of vigor the juvenile contributions to the continents should also be episodic. Finally, we mention the process of the intracrustal diﬀerentiation that generates two chemically heterogeneous reservoirs with distinct systematic diﬀerences in their abundances of incompatible elements, namely, the upper and lower CC (Rudnick and Gao [91]). Table B.1 of Walzer et al. [113] specifies essential diﬀerences between the Earth and terrestrial planets.

612

U. Walzer, R. Hendel, J. Baumgardner

3 Model 3.1 Balance of Mass, Momentum and Energy We use a numerical strategy for modeling the diﬀerentiation and mixing processes that have operated during our planet’s history. We solve the diﬀerential equations of infinite Prandtl-number convection using a three-dimensional finite-element spherical-shell method. These express the conservation of mass, momentum, and energy. The mass balance ∂ρ + ▽ · (ρv) = 0 ∂t

(1)

with the anelastic-liquid approximation simplifies to 1 ▽ · v = − v · ▽ρ ρ where ρ is density, t time, and v is velocity. The conservation of momentum can be written as ∂ ∂v ρ + v · ▽v = − ▽ P + ρg + τik ∂t ∂xk

(2)

(3)

where P is the pressure, g is the gravity acceleration, and τik is the deviatoric stress tensor. For spherical symmetry, we have g = −ger and the hydrostatic pressure gradient may be written as −

∂P = ρg ∂r

(4)

∂P

By definition KS = −V ∂V and VV0 = ρρ0 , where KS is the adiabatic bulk S modulus, V volume, S entropy, r the radial distance from the Earth’s center. Hence ∂r ∂P ∂P =ρ (5) KS = ρ ∂ρ S ∂r S ∂ρ S Substituting (4) into (5) we obtain −ρ2 g ∂ρ = ∂r S KS

(6)

Upon neglecting horizontal spatial variations in ρ, (2) and (6) yield ρgvr 1 ∂ρ 1 = ▽ · v = − v · ▽ρ ∼ = − vr ρ ρ ∂r KS

(7)

It is well-known that KS =

cp KT = (1 + αγth T )KT cv

(8)

Mantle Convection, Continent Generation and Geochemical Heterogeneity

613

where KT is the isothermal bulk modulus, cp the specific heat at constant pressure, cv the specific heat at constant volume, α the coeﬃcient of thermal expansion, γth the thermodynamic Gr¨ uneisen parameter and T the absolute temperature. Equation (3) can be rewritten as ρ

∂σki dvi = ρgi + dt ∂xk

(9)

Using this equation, the energy balance can be expressed as follows ρ

∂qi du + = Q + σik ε˙ik dt ∂xi

(10)

where u is the specific internal energy, Q is the heat generation rate per unit volume; vi , gi , qi , xi , σik , ε˙ik are the components of velocity, gravity acceleration, heat flow density, location vector, stress tensor and strain-rate tensor, respectively. Another formulation of Eq. (10) is ∂ + v · ▽ u = ▽ · (k ▽ T ) + Q − P ▽ ·v + 2WD (11) ρ ∂t where 2WD = σik ε˙ik + P ▽ ·v and qk = −k

(12)

∂T ∂xk

(13)

The quantity k denotes the thermal conductivity. Using du = T ds − P dv and du = T

∂s ∂T

dT + T

P

∂s ∂P

(14)

T

dP − P dv

(15)

we eliminate the specific internal energy in Eq. (11) and obtain the equation ρcp since cp = T

dP dT = ▽ · (k ▽ T ) + Q + αT + 2WD dt dt

∂s ∂T

P

and

∂s ∂P

T

=−

∂v ∂T

P

= −vα

(16)

(17)

Here s signifies the specific entropy, v the specific volume, cp the specific heat at constant pressure and α the coeﬃcient of thermal expansion.

614

U. Walzer, R. Hendel, J. Baumgardner

Next, a less well known version of the energy balance is presented: (11) is equivalent to dv du ∂vi ρ +P = τik + ▽ · (k ▽ T ) + Q (18) dt dt ∂xk 1 ρ

because of (2) and

= v. Inserting (14) into (18), we obtain

ρT

∂ k T +Q ∂xj

ds ∂ ∂vi = τik + dt ∂xk ∂xj

(19)

On the other hand, ds = and

This implies

∂s ∂T

v

dT +

cv , = T

∂s ∂T

v

∂s ∂v

∂s ∂v

dv

(20)

= αKT

(21)

T

T

1 T ds = cv dT + αKT T d ρ

or

T ds = cv dT − where γth =

cv γT dρ ρ

(22)

(23)

αKT cv ρ

(24)

stands for the thermodynamic Gr¨ uneisen parameter. Inserting (23) into (19) we obtain ∂ ∂ dρ dT ∂vi ρcv − cv γT = τik k + T +Q dt dt ∂xk ∂xj ∂xj

(25)

From (1) and (25) ρcv

∂ ∂vj ∂vi dT = −ρcv γT + τik + dt ∂xj ∂xk ∂xj

∂ k T +Q ∂xj

(26)

or ∂vj 1 ∂T ∂ = −vj T − γT + dt ∂xj ∂xj ρcv

∂ ∂ ∂vi τik k + T +Q ∂xk ∂xj ∂xj

(27)

or ∂(T vj ) ∂vj 1 ∂T =− − (γ − 1)T + ∂t ∂xj ∂xj ρcv

∂ ∂ ∂vi τik k + T + Q (28) ∂xk ∂xj ∂xj

Mantle Convection, Continent Generation and Geochemical Heterogeneity

615

This is an alternative formula for the energy conservation. Although cv appears in (28), the latter expression is equivalent to (16) where cp is used. The deviatoric stress tensor can be expressed by ∂vk 2 ∂vj ∂vi τik = η (29) + − δik ∂xk ∂xi 3 ∂xj in the (3) and (28), where η denotes the viscosity. As an equation of state we take

ρ = ρr 1 − α(T − Tr ) + KT−1 (P − Pr ) +

2

Γk Δρk /ρr

k=1

(30)

where the index r refers to the adiabatic reference state, Δρk /ρr or fak denotes the non-dimensional density jump for the kth mineral phase transition. Γk is a measure of the relative fraction of the heavier phase where πk 1 Γk = 2 1 + tanh dk with πk = P − P0k − γk T describing the excess pressure πk . The quantity P0k is the transition pressure for vanishing temperature T . A non-dimensional transition width is denoted by dk . The quantity γk represents the Clausius-Clapeyron slope for the kth phase transition. Γk and πk have been introduced by Richter [89] and Christensen and Yuen [24]. Because of the very high Prandtl number, the left-hand side of (3) vanishes. Hence, we use the following version of the equation of conservation of momentum. ∂ ∂ 0=− (P − Pr ) + (ρ − ρr )gi (r) + τik (31) ∂xi ∂xk The final version of the equation of conservation of mass is 0=

∂ ρvj ∂xj

(32)

which stems from (2). The Equations (28), (30), (31) and (32) are a system of six scalar equations that we use to determine six scalar unknown functions, namely T , ρ, P and the three components of vi . 3.2 Viscosity and Phase Transitions The viscosity law of this paper is presented as follows. Based on experimental results of Karato and Li [58], Karato and Wu [60] and Li et al. [68], a Newtonian solid-state creep is assumed for the Earth’s mantle. The shear viscosity, η, is calculated by η(r, θ, φ, t) = 10rn ·

1 1 − (33) · η3 (r) · exp ct · Tm T Tav exp(c Tm /Tst )

exp(c Tm /Tav )

616

U. Walzer, R. Hendel, J. Baumgardner

where r is the radius, θ the colatitude, φ the longitude, t the time, rn the viscosity-level parameter, Tm the melting temperature, Tav the laterally averaged temperature, Tst the initial temperature profile, T the temperature as a function of r, θ, φ, t. The quantity η3 (r) is the viscosity profile at the initial temperature and for rn = 0. So, η3 (r) describes the dependence of the viscosity on pressure and on the mineral phase boundaries. The derivation of η3 (r) is to be found in Walzer et al. [113]. The quantity rn has been used for a stepwise shift of the viscosity profile to vary the averaged Rayleigh number from run to run. The second factor of the right-hand side of (33) describes the increase of the viscosity profile with the cooling of the Earth. For MgSiO3 perovskite we should insert c = 14, for MgO w¨ ustite c = 10 according to Yamazaki and Karato [120]. So, the lower-mantle c should be somewhere between these two values. For numerical reasons, we are able to use only c = 7. In the lateral-variability term, we inserted ct = 1. For the uppermost 285 km of the mantle (plus crust), an eﬀective viscosity, ηef f , was implemented where $ σy % (34) ηef f = min η(P, T ), 2ε˙

The pressure is denoted by P , the second invariant of the strain-rate tensor by ε. ˙ The quantity σy is a viscoplastic yield stress. The viscosity profile of the present paper (see Fig. 2) displays a highviscosity lithosphere. Beneath of it is a low-viscosity asthenosphere down to the 410-km phase boundary. The transition zone is highly viscous, at least between 520 and 660 km depth. This model is corroborated by the fact that downgoing slabs extending down to the asthenosphere only show extensional fault-plane solutions. If a slab enters the transition zone then compressional focal mechanisms are observed (Isacks and Molnar [54]), also in the case that the 660-km discontinuity is not touched. These observations cannot be explained by the olivine-wadsleyite or the wadsleyite-ringwoodite phase transition since the Clausius-Clapeyron slopes for both are positive and the phase boundary distortion enhances the cold downflow in these cases. The increase of the number of seismic events per 20 km Bin beneath of 520 km depth (Kirby et al. [64]) can be explained by a viscosity increase. Only if the slab reaches the 660-km phase boundary the corresponding negative Clausius-Clapeyron slope can contribute to the observed compressional fault-plane solutions. If the slab penetrates the 660-km phase boundary then the latter is deflected somewhat downward. An earthquake has never been observed below that deflection. Beneath of that, the slab is only detectable by elevated densities and seismic compressional and shear velocities. Therefore it is reasonable to infer a low-viscosity layer in the uppermost part of the lower mantle (Kido and Yuen [62]). This proposal of a high-viscosity transition layer, that is embedded between two low-viscosity layers, is consistent with the proposition that the transition zone is composed mostly of garnet and spinel (Meade and Jeanloz [74], Karato et al. [59], Karato [57], All`egre [3]). If there are no further phase transitions in the lower mantle, except near the D′′ layer (Matyska and

Mantle Convection, Continent Generation and Geochemical Heterogeneity

617

Yuen [70]), then the viscosity must rise considerably as a function of depth because of the pressure dependence of the activation enthalpy of the prevailing creeping mechanism for regions where the temperature gradient is near to adiabatic. This implies a thick high-viscosity central layer in the lower mantle. We infer a strong temperature gradient of the D′′ layer which causes a strong decrease of viscosity in the near neighborhood above the CMB. In our derivation of η3 (r), however, we did not make use of the above arguments. They serve only as a corroboration. The alternative systematic derivation is described by Walzer et al. [113]. We start from a self-consistent theory using the Helmholtz free energy, the Birch-Murnaghan equation of state, the free-volume Gr¨ uneisen parameter and Gilvarry’s [36] formulation of Lindemann’s law. The viscosity is calculated as a function of melting temperature obtained from Lindemann’s law. We use pressure, P , bulk modulus, K, and ∂K/∂P from the seismic model PREM (Dziewonski and Anderson [32]) to obtain the relative variation in radial viscosity distribution. To determine the Table 1. Model parameters Parameter Description

Value

rmin rmax

3.480 × 106 6.371 × 106

m m

288

K

4.10 × 105

m

6.60 × 105

m

+1.6 × 106

Pa·K−1

−2.5 × 106

Pa·K−1

h1 h2 γ1 γ2 fa1 fa2

d1 d2

ct k nr + 1

Inner radius of spherical shell Outer radius of spherical shell Temperature at the outer shell boundary Depth of the exothermic phase boundary Depth of the endothermic phase boundary Clapeyron slope for the olivine-wadsleyite transition Clapeyron slope for the ringwoodite-perovskite transition Non-dimensional density jump for the olivine-wadsleyite transition Non-dimensional density jump for the ringwoodite-perovskite transition Begin of the thermal evolution of the solid Earth’s silicate mantle Non-dimensional transition width for the olivine-wadsleyite transition Non-dimensional transition width for the ringwoodite-perovskite transition Begin of the radioactive decay Factor of the lateral viscosity variation Thermal conductivity Number of radial levels Number of gridpoints

0.0547 0.0848 4.490 × 109

a

0.05 0.05 4.565 × 109 a 1 12 W·m−1 ·K−1 33 1.351746 × 106

618

U. Walzer, R. Hendel, J. Baumgardner

absolute scale of the viscosity profile, we utilize the standard postglacial-uplift viscosity of the asthenosphere below the continental lithosphere. Our η3 (r) profile is supported by several recent studies. New inversion investigations for mantle viscosity profiles reveal an acceptable resolution down to 1200 km depth. For greater depths, models based on solid-state physics ˇ seem to be more reliable. Kido and Cadek [61] and Kido et al. [63] found two low-viscosity layers below all three oceans. The first layer is between the lithosphere and 410 km depth. The second one is between 660 and about 1000 km depth. Panasyuk and Hager [81] made a joint inversion of the geoid and the dynamic topography. They found three families of solutions for the radial viscosity profile that are diﬀerent regarding the position of the lowest-viscosity region: (a) directly beneath the lithosphere, (b) just above 400 km depth or (c) just above 670 km depth. The results of Forte and Mitrovica [35] show even more similarity with our profile η3 (r). Their viscous-flow models, based on two seismic models of three-dimensional mantle structure, revealed two viscosity maxima at about 800 and about 2000 km depth. This is similar to our model for η3 (r) that has also two maxima in the interior although is has been derived by a completely diﬀerent method. Cserepes et al. [30] investigated the eﬀects of similar viscosity profiles on Cartesian 3-D mantle convection in a box. In our dynamical model, we take into account the full eﬀect of phase boundary distortion of the olivine-wadsleyite and of the ringwoodite-perovskite phase boundary. The input parameters that define these phase transitions are provided in Table 1. 3.3 Numerical Method and Implementation In our models we include the full pressure dependence and the full radial temperature dependence of viscosity. For numerical reasons, however, we are able to treat only a part of the lateral temperature dependence of the viscosity. At the mineral phase boundaries in the interior of the Earth’s mantle, there are not only discontinuities of the seismic velocities and of the density but also jumps of activation volumes, activation energies and, therefore, of activation enthalpies. Since the viscosity depends exponentially on the activation enthalpy of the prevailing creeping process, the conclusion is inescapable that there are considerable viscosity jumps at the upper and lower surfaces of the transition zone. These jumps cause numerical problems in the solution of the balance equations. The problems have been solved. Nevertheless, our group is searching for more eﬀective solutions of the numerical jump problem. The minor discontinuity at a depth of 520 km has been neglected. We treat the mantle as a thick spherical shell. The grid for this domain is constructed by projection of the edges of a regular icosahedron onto concentric spherical shell surfaces with diﬀerent radial distances from the center. These surfaces subdivide the mantle into thin shells. A first step of grid refinement consists of bisecting the edges of the resulting spherical triangles into equal parts. Connecting the new points with great circles, we obtain four smaller

Mantle Convection, Continent Generation and Geochemical Heterogeneity

619

triangles from each starting triangle. The process can be repeated by successive steps to obtain a grid with the desired horizontal resolution. We replicate the resulting almost uniform triangular grid at diﬀerent radii to generate the 3D grid for a spherical shell. We can use diﬀerent formulae for the distribution of the radial distances of the spherical grid surfaces. In this paper, we used exclusively a radially nearly equidistant grid with a superposed sinoidal half-wave length to refine the grid near the upper and lower boundaries of the spherical shell. The grid is non-adaptive. The Navier-Stokes equations as well as pressure and creeping velocity are discretized using finite elements. We apply piecewise linear basis functions for the creeping velocity and either piecewise constant or piecewise linear basis functions for the pressure. We solve the equations for pressure and velocity simultaneously by a Schur-complement conjugate-gradient iteration (Ramage and Wathen [86]). This is a further development of an Uzawa algorithm. We solve the energy equation using an iterative multidimensional positive-definite advection-transport algorithm with explicit time steps (Bunge and Baumgardner [20]). Within the Ramage-Wathen procedure, the resulting equation systems are solved by a multigrid procedure that utilizes radial line Jacobi smoothing. In the multigrid procedure, prolongation and restriction are handled in a matrix-dependent manner. In this way, it is possible to handle the strong variations and jumps of the coeﬃcients associated with the strong viscosity gradients (Yang [121]). For the formulation of chemical diﬀerentiation, we modified a tracer module developed by Dave Stegman. This module contains a second-order Runge-Kutta procedure to move the tracer particles in the velocity field. Each tracer carries the abundances of the radionuclides. In this sense, tracers are active attributes which determine the heat production rate per unit volume that varies with time and position. [30] The FORTRAN code is parallelized by domain decomposition and explicit message passing (MPI) (Bunge [19]). For the most runs, we used a mesh of 1351746 nodes. For some runs, we used a mesh of 10649730 nodes in order to check the convergence of the lower resolution runs. We found hardly any discernable diﬀerences (< 0.5%) for the Rayleigh number, the Nusselt number, the Urey number and the laterally averaged surface heat flow as a function of time. The calculations were performed on 32 processors of a Cray Strider Opteron cluster. The code was benchmarked for constant viscosity convection by Bunge et al. [21] with data of Glatzmaier [37] for Nusselt numbers, peak velocities, and peak temperatures. The result is a good agreement with deviations ≤1.5%. 3.4 Heating, Initial and Boundary Conditions, and Chemical Diﬀerentiation We assume the Earth’s mantle is heated mostly from within. This internal heating is expressed by the heat production density Q in (28) that is measured in W · m−3 .

620

U. Walzer, R. Hendel, J. Baumgardner

Q =H ·ρ

(35)

where H is the specific heat production with H=

4

aμν aif ν H0ν exp(−t/τν )

(36)

ν=1

Table 2 presents the parameter data we use for the four major heat-producing isotopes. Here, ν stands for the radionuclide in the formulae, τν represents the decay time or the 1/e life, Hoν denotes the specific heat production of the νth radionuclide 4.565 × 109 years ago, aif ν is the isotope abundance factor. We represent the distribution of radionuclides in the mantle by tracers. Each tracer is associated with a specific geochemical principal reservoir. Because of mixing, the boundaries of these reservoirs become blurred with time. In principle, even a total mixing and homogenization of the mantle is possible if the dynamic system of the mantle allows this process. Each tracer is identified by a tracer index. The reservoir concentrations of elements are given by Table 3. The lower five elements of Table 3 serve only for the computation of concentration maps but not for the calculation of heating energy either because the contributions of these elements are too low or because they are daughter nuclides. Since the relative masses of HIMU, EM1 and EM2 are small they have been neglected in the calculated model, S3, of this paper. Our Table 2. Data of the major heat-producing isotopes 40

Isotope

K

232

Th

235

U

238

U

ν 1 2 3 4 τν [Ma] 2015.3 20212.2 1015.4 6446.2 H0ν [W kg −1 ] 0.272×10−3 0.0330×10−3 47.89×10−3 0.1905×10−3 aif ν 0.000119 1 0.0071 0.9928 Table 3. The abundances aµν of the major heat-producing elements Reservoir

Primordial Oceanic Continental Depleted MORB mantle (ppm) crust [MORB] (ppm) crust (ppm) mantle (ppm)

tracer index

(1)

(2)

(3)

(4)

element U Th K

0.0203 0.0853 250.

0.047 0.12 600.

0.94 4.7 9460.

0.0066 0.017 110.

Pb Sm Nd Rb Sr

0.1382 0.4404 1.354 0.635 21.0

0.30 2.63 7.3 0.56 90.

7.0 4.62 25.5 35.5 310.

0.035 0.378 0.992 0.112 16.6

Mantle Convection, Continent Generation and Geochemical Heterogeneity

621

model mantle starts with a uniform distribution of exclusively type-1 tracers, i.e., we start with a pure BSE mantle. If the modeled temperature, T , approaches the melting temperature, Tm , in a certain volume then chemical diﬀerentiation takes place. Plateau basalts quickly rise to form the plateaus as a terrane or preliminary form of the continental crust (CC) leaving behind depleted MORB mantle (DMM). The numerical conditions for this simplified chemical diﬀerentiation process will be given below. We do not use a detailed melt extraction equation system like the 2-D code of Schmeling [93] since, for a 3-D spherical-shell code, this would require more computational resources than we currently have available. We chose McCulloch and Bennett [71] reservoir abundances for our models because of the good internal compatibility of this geochemical model. These abundances are similar to those proposed by other investigators. Heier [43], Taylor and McLennon [102], Hofmann [48], McCulloch and Bennett [71], Wedepohl [115] and Rudnick and Fountain [90] have proposed values for the continental-crust K:U ratio of 10777, 10020, 10000, 10064, 10020, 11092, respectively. The spherical shell of our present model has free-slip and impermeable boundary conditions for both the Earth’s surface and CMB. The upper surface is isothermal at 288 K. The CMB is also isothermal spatially, but not with respect to time. Applying a cooling core-mantle evolution model (Steinbach et al. [96]), we adjust the CMB temperature, Tc , after each time step according to the heat flow through the CMB. We assume a homogeneous core in thermodynamic equilibrium similar to the approaches of Steinbach and Yuen [95] and Honda and Iwase [52]. Prior to this work, our modeling eﬀorts relating to the problem of integrated convection-fractionation were restricted to two dimensions (Walzer and Hendel [112], Walzer et al. [114]). We here describe two tracer methods for our 3-D compressible spherical-shell models. The first method has the advantage of being simple and readily comprehensible. The second method avoids certain deficiencies of the first one. The first method: We assign a 3-D cell to each node in the icosahedral grid with 1351746 nodes. There are Type-1 tracers, Type-3 tracers and Type-4 tracers with the abundances given in Table 3. At the beginning of the evolution of the model, the shell contains exclusively Type-1 tracers. Each cell starts with eight tracers or 64 tracers, respectively. The tracers are carried along by the velocity field. The element concentration of a node is determined by the average aμν , of the abundances of the elements carried by the tracers in the cell associated with the node. A local tracer refresh (LTR) is applied if a cell has fewer than four (or 32) or more than twelve (or 96) tracers. Tracers are redistributed from or to the neighboring cells, respectively, using fixed prescriptions. This procedure is to prevent a cell becoming empty of tracers and therefore having aμν become indeterminate. If the conditions for partial melting are fulfilled in a suﬃciently large volume then the Type-1 tracers in that volume are converted to Type-4 tracers corresponding to DMM to mimic

622

U. Walzer, R. Hendel, J. Baumgardner

the depletion. A greater number of changed Type-1 tracers are necessary to produce one new Type-3 tracer (corresponding to CC) from a Type-1 tracer near the surface above a region of diﬀerentiation, since the continental Type-3 tracers have considerably higher abundances of incompatible elements. The ratio z3∗ is given by A (1) (4) z3∗ = a(3) a(1) (37) μν − aμν μν − aμν z3 = round (z3∗ )

(38)

For uranium, z3∗ = 67.131387 based on the values of Table 3, and therefore z3 = 67. The same integer is derived for thorium and potassium. So, 67 Type-1 tracers from the asthenosphere are necessary to generate one Type-3 tracer in the lithosphere by transformation of one Type-1 tracer at the corresponding place. If a cell (a) has more than 50% Type-1 tracers and is, therefore, fertile and (b) has fulfilled the condition T > f3 · Tm in its grid point where f3 is a fixed parameter with 0 < f3 ≤ 1 and (c) has at least five neighboring cells with common boundary surfaces that also fulfill (a) and (b) then this cell is called Type-A cell. If a cluster of simply connected Type-A cells has nthr Type-1 tracers then the tracers are instantaneously changed in Type-4 tracers. This does not concern all of these tracers but (nthr + nn · z3 ) of them where nn is an integer. Here thr stands for threshold. The center of gravity of the cluster is projected to the top surface of the shell. The corresponding point at the surface is called P ′ . A number of (nthr /z3 + nn ) Type-1 tracers nearest to P ′ and not deeper than 65 km are changed to Type-3 tracers. This corresponds to oceanic plateaus. All Type-3 tracers are unsinkable and move with the horizontal component of the velocity field. This rule mimics the tendency of the continents to resist subduction. If two Type-3 tracers approach each other nearer than a small distance din then they are connected to a continent. If an unconnected Type-3 tracer approaches to a continent nearer then din then it will be connected with the continent. Tracers that are connected to form a continent move with a common angular velocity, ω, associated with that continent around the center of the shell. This quantity ω is calculated as the vector sum of the single angular velocities derived from the horizontal component of the undisturbed nodal velocity. Hence, the continent moves as a plate-like thin shell across the upper surface. In our present model, oceanic plates develop without any such construction simply as a result of the yield stress and of the existence of an asthenosphere. The second method: The second method is a translation of the ideas of the first method to the Stegman code with some improvements. For the starting distribution of the tracers, a cell is attributed to each node. Tracers are initially distributed in an almost uniform manner about each grid point, with eight (or 64) tracers per grid-point cell, except for the grid-point cells on the top and bottom shell boundaries, which have four (or 32). A new feature of this second method is that each particle carries its initial mass as one of its attributes. The sum of the individual particle masses is equal to the total mass of the

Mantle Convection, Continent Generation and Geochemical Heterogeneity

623

mantle. If 4 tracers are regularly distributed to each half-cell then the mass, mip, of a tracer, ip, is calculated by mip = 1/4 ∗ (volume of the half -cell) ∗ (density of the node)

(39)

The mass, Mmantle , of the whole mantle results from Mmantle =

npm

mip

(40)

ip=1

where npm is the total number of tracers in the mantle. A memory cell contains all tracers that are attributed to a node. Its base is a spherical hexagon or pentagon the corners of which are in the triangle centers of the triangular distribution around a node. Its altitude is again between the grid spherical surfaces ir and (ir + 1). Combining and splitting: Material from the top boundary layer that sinks to the CMB experiences a density increase and an increase of the tracer number per volume up to a factor two. However, the cell volume is diminished by a factor four during a movement from the surface to the CMB. So, a reduction of the number of tracers per memory cell by a factor two is to be expected. If the tracer number falls below four (or 32) then each tracer of this cell is split into two tracers. The mass of such a tracer is distributed equally to the child tracers. The reverse process will occur during upwelling. Overcrowding of the memory cells can take place. For technical reasons, we limit the number of tracers per cell to 12 (or 96). Each tracer coming into the cell beyond this limit is combined with one of the other tracers according to their order in memory. The masses are added. The location of the new tracer is the center of gravity of the two annihilated tracers. Only tracers of the same type can be combined. If an excess tracer enters the cell with a type not present in the cell, then two tracers of the most abundant type are united, the first two in the storage sequence. Splitting or combining does not alter the cell mass, M c, nor the sum of the tracer masses, mipc, present in the cell. The base of an interpolation cell is a spherical hexagon or pentagon the corners of which are the lateral neighboring nodes. The upper and lower base is determined by the upper and lower neighboring grid spherical surfaces. All tracers inside the interpolation cell contribute to the interpolation of tracer attributes, e.g., elemental abundance, to the node. The nearer the tracer is to the grid point, the larger is the weighting factor. The lateral weighting factor, wl, is simply the barycentric coordinate of the tracer when the tracer and node are both radially projected onto the same spherical surface: wl = (α|β|γ)(ip)

(41)

624

U. Walzer, R. Hendel, J. Baumgardner

The radial weighting factor is given by (r(ir + 1) − rip) (r(ir + 1) − r(ir)) (rip − r(ir − 1)) wr = (r(ir) − r(ir − 1))

if rip > r(ir)

wr =

if rip ≤ r(ir)

(42)

where r(ir + 1|ir − 1) are the neighboring grid spherical surfaces of the radius, r(ir), of the node and rip is the radius of the tracer. The total weighting factor is the product of these two factors. wip = wr ∗ wl

(43)

The weighted mass, wmip, of a tracer is wmip = wip ∗ mip

(44)

The mass, wM c, of an interpolation cell can be derived by a weighted integration over the mass continuum of the cell. This has to be done in such a way that wM c(interpolation cell) = M c(memory cell) (45) The total mass balance is not violated by the weighting procedure: npc nc

wmipc =

n3c nip3c nc

wmip3c =

c=1 i3c=1 ip3c=1

c=1 ip=1

=

n3 nip3

i3=1 ip3=1

=

n3 nip3

i3=1 ip3=1

n3 nip3

6

wmip3(node) =

i3=1 ip3=1 node=1

(wr + (1 − wr)) ∗ (α + β + γ) ∗ mip3 = mip3 =

np

mip

(46)

ip=1

where n is a number, c the interpolation cell, p a tracer, i counting index, 3 triangular cell, w weighted, m mass, node the counting index for the nodes at the boundaries of a triangular cell, wr the radial weighting factor of a tracer; α, β, γ are the barycentric coordinates from the three corner points of the basis of a triangular cell, so that α + β + γ = 1. Diminution of tracer mass can be observed in the spreading zones which is not induced by density diﬀerences. In other areas, a compaction of tracer mass is to be expected: npc wM c = wmipc (47) ip=1

Mantle Convection, Continent Generation and Geochemical Heterogeneity

625

The tracer mass ratio, Gmc, of the cells deviates from the obligated value 1 : ⎞ ⎛ npc wmipc⎠ /wM c = 1 (48) Gmc = ⎝ ip=1

This formula describes a distortion of the tracer representation. A local tracer mass refresh has been introduced to reduce this discrepancy of (48). At least, a deviation dGmcmax = max.permissible|Gmc − 1|

(49)

(Gmc − 1) > dGmcmax

(50)

is allowed. If applies in a cell, c, then distribute tracer mass of c to the neighboring cells in such a way that (Gmc − 1) = dGmcmax (51) The tracer mass dwM c =

npc

ip=1

wmipc − wM c ∗ (1 + dGmcmax)

(52)

has to be distributed to those neighboring cells, cn, that have a common edge with the cell, c, at least and that fulfill the condition dwM cn =

npcn ip=1

wmipcn − wM cn ∗ (1 + dGmcmax) < 0

(53)

If the neighboring cells, cn, have the capacity to hold the excess tracer mass of c, i.e., if

ncn dwM cn (54) dwM c < − cn=1

then dwM c is to be distributed to the cn. The proportionality of the diﬀerent tracer types is ensured. If (1 − Gmc) > dGmcmax

(55)

applies in c then remove tracer mass from cn in an analogous way so that (1 − Gmc) = dGmcmax.

(56)

626

U. Walzer, R. Hendel, J. Baumgardner

4 Results and Discussion of the Figures 4.1 Thermal and Chemical Evolution Using a Reference Run We begin by presenting what we call our reference run 808B. It is representative of the results we obtain in a moderately extensive region of Rayleigh number – yield stress parameter space. Our chosen reference run is defined by a viscoplastic yield stress σy = 115 MPa and a viscosity-level parameter rn = −0.65. Run 808B starts with eight tracers per grid-point cell. Now, we present the Figures, in each case immediately followed by the corresponding discussion. In Fig. 1, the laterally averaged temperature for the geological present time as a function of depth is represented by a solid line. This curve lies closer to the geotherm of a parameterized whole-mantle convection model than to the corresponding layered-convection temperature. This is understandable since the results of the present model, S3, show whole-mantle convection. However, the flow is somewhat impeded by the high-viscosity transition zone and by the endothermic 660-km phase boundary. Therefore, the temperature is slightly augmented, especially immediately beneath the 660-km boundary.

Fig. 1. The laterally averaged temperature of the geological present time (solid curve) as a function of depth for the reference run with a viscoplastic yield stress, σy = 115 MPa, and a viscosity level parameter, rn = −0.65. Cf. (33) and (34). A range of realistic mantle geotherms using parameterized models of the mantle’s thermal history given by Schubert et al. [94] is depicted for comparison. Label a and b signify geotherms of whole-mantle and partially layered convection, respectively. The dotted line denotes a mid-oceanic ridge geotherm

Mantle Convection, Continent Generation and Geochemical Heterogeneity

627

Figure 2 displays the laterally averaged present-day viscosity. Its derivation and discussion is given by Sect. 3.2. Figure 3 shows the time dependence of some spatially integrated quantities in our reference run. The evolution of the laterally averaged heat flow at the Earth’s surface is depicted in the first panel. The curve reaches a realistic value for the present time: The observed mean global heat flow has been estimated to be 87 mW/m2 (Pollak et al. [83]). The second panel exhibits the growth rate of continental mass as a function of time. It mimics observational indications that global magmatism and orogenesis are intrinsically episodic (Worsley et al. [119], Nance et al. [76], Hoffman [47], Titley [103], Lister et al. [69], Condie [28]). The third panel of Fig. 3 demonstrates the time dependence of Ror, the ratio of surface heat outflow to the mantle’s radiogenic heat production which is the reciprocal value of the Urey number. Parameterized models show roughly similar curves except for medium-large and smaller fluctuations. A pattern of general decrease and some fluctuations in the Rayleigh number are indicated in the fourth panel. The chemical heterogeneity of incompatible elements in a run with 64 tracers per grid-point cell for present time is shown by Fig. 4. It is remarkable that in spite of 4500 Ma of solid-state mantle convection chemical reservoirs continue to persist. This paper therefore represents a possible way to reconcile the geochemical and geophysical constraints. Heterogeneities are diminished only by stirring (Gottschaldt et al. [38]). Diﬀuse mixing is negligible. However, in our model there are no pure unblended reservoirs, and this may also be true of the Earth’s mantle. DMM predominates immediately below the continents (red) and beneath the oceanic lithosphere. This is a realistic fea-

Fig. 2. The laterally averaged shear viscosity of the reference run as a function of depth for the present geological time

628

U. Walzer, R. Hendel, J. Baumgardner

Fig. 3. Time evolution of some spatially integrated quantities from the reference run. (a) The laterally averaged surface heat ﬂow, qob. (b) The juvenile contributions to the total mass of the continents. The genuine increase of continental material is expressed as converted Type-3 tracer mass per Ma. (c) The reciprocal value of the Urey number. Ror represents the r atio of the surface heat outﬂow to the mantle’s r adiogenic heat production rate. (d) The Rayleigh number as a function of age

Mantle Convection, Continent Generation and Geochemical Heterogeneity

629

Fig. 4. This equatorial section shows the present-time state of the chemical evolution of the Earth’s mantle as computed in a companion run, 808C, of run 808. Run 808C has 64 tracers per grid-point cell at the beginning. Strongly depleted parts of the mantle which include more than 50% of depleted MORB mantle are represented by yellow areas. Less depleted and rich parts of the mantle are depicted by orange colors. Rich refers to a high abundance of incompatible elements. Continents are signiﬁed in red. Black dots stand for oceanic plateaus

ture of the model since where the real oceanic lithosphere is rifted, MORB magma is formed by decompression melting. The MORB source (DMM) is not only depleted in incompatible elements but also relatively homogenized. It is homogenized not only with respect to its major geochemical components (SiO2 , MgO, FeO, Al2 O3 , CaO) (Palme and O’Neill [80]) but also with respect to isotope ratios 87 Sr/86 Sr, 143 Nd/144 Nd, 206 Pb/204 Pb, 207 Pb/204 Pb and 208 Pb/204 Pb. As a consequence, the standard deviation of these isotope ratios and of the major element compositions is small for MORBs in comparison to OIBs (All`egre and Levin [4]) although Hofmann [50] has modified this conclusion somewhat. Figure 4 shows a marble-cake mantle as it was suggested by Coltice and Ricard [27] and Becker et al. [9] but reversed in terms of its pattern. It is the depleted regions in our model that are disconnected and distributed like raisins. Furthermore the present model, S3, does not present diﬃculties with the buoyancy since the present chemical diﬀerences refer to the incompatible elements and not to the geochemical major components. It is remarkable that we did not obtain simply connected volumes for any geo-

630

U. Walzer, R. Hendel, J. Baumgardner

chemical reservoir. Nevertheless, the depleted volumes tend to be in the upper parts of the mantle. This is not amazing since chemical diﬀerentiation takes place just beneath the lithosphere and the low viscosity of the asthenosphere promotes mixing and lateral migration of DMM. Figure 5 shows the present-time distribution of continents (red) of our reference run. The oceanic plateaus (black dots) are carried along by the self-consistently generated, moving oceanic lithosphere. If the plateaus touch a continent they join with it. This is the only additional implementation. Neither number nor form nor size of the continents is prescribed. The configuration results simply from the numerical solution of the system of equations and the initial and boundary conditions. At first, the comparison with the observed present-day continents was carried out simply visually. Then we decided to represent both topographies, the observed one and the theoretical one, in terms of spherical harmonics m −1 {Am · (2n + 1)1/2 · 2−1/2 · [(n − m)!]1/2 · [(n + m)!]−1/2 · n or Bn } = π ⎡ π ⎤ 2π · {cos mφ or sin mφ} · ⎣ f (θ, φ) · Pn,m (cos θ) · sin θ · dθ⎦ dφ, (57) 0

0

Fig. 5. The distribution of red continents and black oceanic plateaus at the Earth’s surface for the geological present time according to the reference run with yield stress σy = 115 MPa and viscosity-level parameter rn = −0.65. Arrows denote velocity. The oceanic lithosphere is denoted in yellow. There are no prescriptions concerning number, size or form of continents in the present model

Mantle Convection, Continent Generation and Geochemical Heterogeneity

631

respectively, where f (θ, φ) is topographic height. While the individual coefm ficients Am n or Bn depend on the position of the pole of the grid (θ, φ), the ∗ quantity hn is orientation-independent: h∗n

=n

1/2

1/2

· (n + 1)

·2

−1

·

n

m=0

2 (Am n)

+

2 (Bnm )

% 1/2

(58)

We are not aware of other papers on spherical-shell mantle convection with continents that evolve due to physical laws and that are not simply put onto the surface. Figure 6 reveals the plate-like motions of the lithospheric patches at the surface. This kind of motion arises because of the viscosity law that includes yield stress. It has nothing to do with the tracers. It arises in similar models without tracers. The colors represent the logarithm of the viscosity in Pa · s. Figure 7 exhibits the present-time temperature on an equal-area projection of a spherical surface at 134.8 km depth. The blue zones corresponding to cold, subducting rock correlate with convergent zones at the surface.

Fig. 6. The surface distribution of log viscosity (Pa·s) on an equal-area projection for the geological present time for the reference run. The velocity ﬁeld displays platelike character. Elongated high strain-rate zones lead to reduced viscosity because of viscoplastic yielding

632

U. Walzer, R. Hendel, J. Baumgardner

Fig. 7. Equal-area projection with the temperature distribution (colors) and the velocities (arrows) for the geological present for the reference run at a depth of 134.8 km. The narrow blue sheet-like subducting zones are evident also at greater depths. The slab-like features are narrow in comparison with the much broader upwellings

4.2 Variation of Parameters: The Evolution of Continents We varied the parameters Ra and σy to investigate the region in which we obtain Earth-like results and to find other regions with diﬀerent mechanisms. A multitude of runs were performed to convince us that the selected reference run is by no means exceptional but representative of a notable portion of the parameter space. We find that the general character of our results does not deviate too far from that of the real Earth. We compare the number, size, form and distribution of the calculated continents with the continent configuration of the present Earth. Earth-like continent solutions are shown by little black disks in the center of the Ra-σy plot of Fig. 8. Ra denotes the temporal average of the Rayleigh number of a given run. Figures 9 and 10 display presenttime continent distributions from two other runs, with σy = 130 MPa and σy = 115 MPa, respectively, and for rn = −0.6 in both cases. We performed further studies to attempt to refine the Earth-like Ra-σy area. Figure 11 describes a quantitative measure of the deviation of the calculated presenttime continental area from the observed one. Favorable agreement occurs in

Mantle Convection, Continent Generation and Geochemical Heterogeneity

633

Fig. 8. The types of continental distribution as a function of yield stress, σy , and of temporally averaged Rayleigh number, Ra. Each symbol of the plot denotes one run. Little black disks with a white center signify Earth-like distributions of the continents where the size of the disk is a measure of quality. Five-pointed stars stand for distributions with an unrealistic multitude of tiny continents. White circles represent runs with reticularly connected, narrow stripe-like continents

Fig. 9. Equal-area projection with the distribution of continents (red ) and oceanic plateaus (black dots) for the geological present of a run with yield stress σy = 130 MPa and viscosity-level parameter rn = −0.6. Yellow color stands for the oceanic lithosphere

634

U. Walzer, R. Hendel, J. Baumgardner

Fig. 10. The distribution of continents (red ) and oceanic plateaus (black dots) for the geological present of a run with yield stress σy = 115 MPa and viscosity-level parameter rn = −0.6. The oceanic lithosphere is signiﬁed by yellow color

Fig. 11. A classiﬁcation of the runs with respect to the diﬀerence of observed surface percentage of continents (= 40.35%) minus calculated surface percentage of continents. This diﬀerence, dc , is plotted as a function of yield stress, σy , and of the time average of the Rayleigh number, Ra. Little black disks denote slight deviations, namely −4.5 ≤ dc < 4.5 percent. White circles stand for 4.5 ≤ dc < 13.5. Plus signs signify 13.5 ≤ dc < 22.5. White triangles represent runs with 22.5 ≤ dc < 31.5. White diamonds denote 31.5 ≤ dc < 40.5.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

635

the center of the Ra-σy area. Favorable means that Earth-like solutions can be found in both Figs. 8 and 11 in the common part of the Ra-σy field. 4.3 Variation of Parameters: Plateness of Oceanic Lithospheric Pieces and Other Features A classification of runs in terms of the planforms of flow near the surface is presented by Fig. 12. Black disks denote plate-like solutions. An overlap set of the black disks is observed with the black disks of Figs. 8 and 11. Figure 13 shows the distribution of classes of Urey numbers as a function of yield stress, σy , and time average of the Rayleigh number, Ra. Runs with realistic Urey numbers are pictured by black disks. For a comparison of the present-time laterally averaged heat flows, qob(now), of the runs, it is important to filter away the random fluctuations. A simple method to do so is to replace the calculated values of qob(now) by qob∗ where qob∗ = mean[qob(now)/qob(time av)] ∗ qob(time av)

(59)

The expression time av denotes the time average of one run, mean stands for the average of all runs of the plot. Figure 14 demonstrates the distribution of the filtered present-time surface average of the heat flow, qob∗ , in an rn − σy diagram. Realistic values are again denoted by black disks. A partial covering with the favorable field of continent distribution of Fig. 5 is established.

Fig. 12. The types of lithospheric movements as a function of yield stress, σy , and time average of the Rayleigh number, Ra. Plate-like solutions with narrow subducting zones are depicted by little black disks. Its surface area is a measure of plateness. White circles represent runs with broad downwellings and minor plateness. White five-pointed stars denote unrealistic runs with local subduction only. Asterisks stand for rather complex planforms with lots of small but not narrow downwellings

636

U. Walzer, R. Hendel, J. Baumgardner

Fig. 13. The time average of the Urey number, U r is plotted in a diagram the abscissa of which is the yield stress, σy , and the ordinate is the time average of the Rayleigh number, Ra. Asterisks represent runs with U r ≤ 0.59. White squares stand for 0.59 < U r ≤ 0.625. Little black disks denote runs with 0.625 < U r ≤ 0.67. White circles depict runs with 0.67 < U r ≤ 0.71. Finally, plus signs signify runs with 0.71 < U r

Fig. 14. The symbols represent classes of the non-random values, qob∗ , of the present-time surface average of the heat ﬂow of the runs in a rn -σy plot where qob∗ is calculated using (59). The following numbers are given in mW/m2 . Asterisks signify runs with 97 ≤ qob∗ . White squares depict runs with 89 ≤ qob∗ < 97, little black disks stand for 81 ≤ qob∗ < 89. White circles denote runs with 77 ≤ qob∗ < 81, plus signs represent the range qob∗ < 77

Mantle Convection, Continent Generation and Geochemical Heterogeneity

637

Fig. 15. A comparison of the orientation-independent quantities h∗n of the total ﬂow of the Earth, T , and of the bathymetry, S, with the theoretical ﬂow spectrum 2 of the reference run (lower curve). The observational curves n1/2 ∗(n+1)1/2 ∗vn,pol T and S have been calculated from the topography of the global JGP95E Digital Elevation Model of Lemoine et al. [67], chapter 2

Figure 15 shows the present-time theoretical flow spectrum n1/2 × (n + 2 × vn,pol of the reference run (lower curve) in comparison with the 1) spectra of the total observed topography, T , and of the observed sea-floor topography, S, of the global JGP95E Digital Elevation Model (Lemoine et al. [67]). It would be senseless to compare the diﬀerent sets of coeﬃcients Am n and Bnm of (57) since they depend on the position of the pole of the coordinate system. The quantity h∗n of (58) is, however, independent on the orientation of the pole. The comparison of the theoretical spectrum h∗n (n) with that of T shows a coincidence of the maxima up to n = 17. A correspondence for higher values of n is not to be expected because of the simplicity of the model. The perpendicular auxiliary lines are therefore only in five-unit distances for the higher-n region. 1/2

5 Conclusions The main subject of this paper is a combined segregation-convection theory in a 3-D compressible spherical-shell mantle. It is a step toward a reconciliation of seemingly contradictory geochemical and geophysical findings and a preliminary answer to three questions: (a) Did the diﬀerentiation of the mass of the continental crust (CC) take place predominantly at the beginning of the

638

U. Walzer, R. Hendel, J. Baumgardner

Earth’s evolution similar to the cases of the Moon and Mars in which chemical segregation occured in the first 200 Ma, or have there been other modes of crustal production that continue to add juvenile crust in batches possibly connected with episodic orogenesis? (b) How can diﬀerent geochemical reservoirs be maintained in spite of persisting whole-mantle convection? (c) Why is DMM more homogeneous than other reservoirs? Our modeling suggests the following simplified answers: (a) Similar to the cases of the Moon and Mars, part of the Earth’s crust was probably also formed from a magma ocean, whether also CC was formed at this point is unknown. Nevertheless, since the mantle has been solid, our model indicates there have been episodes of CC growth comparable to magmatic and tectonic episodes in the Earth’s history (cf. Fig. 3, second panel). (b) The essential cause for the long-term conservation of complex mantle reservoirs less depleted than DMM is a high-viscosity zone in the central part of the lower mantle. Furthermore, the endothermal 660-km phase boundary and a possible highviscosity transition layer also retard the stirring. (c) DMM is produced in the conventional asthenosphere and is distributed by convection also to other parts of the mantle. Since the asthenosphere has the lowest viscosity, the stirring is most eﬀective there. Moreover, the Figs. 4, 5, 8, 9, 10, 11 and 15 show that our model, S3, generates convincing present-time distributions of continents. Although the problem of oceanic lithospheric plate generation is not the focus of this paper as in Trompert and Hansen [108], Tackley [100, 101], Richards et al. [88], Bercovici and Karato [11], Walzer et al. [113] and Bercovici and Ricard [13], we want to remark that also S3 shows good plate-like solutions (cf. Fig. 12). Other conclusions that we do not want to repeat here can be found in the Abstract. Acknowledgements We gratefully acknowledge the help of Dave Stegman. He provided us with his particle code and discussed some problems with us. This work was partly supported by the Deutsche Forschungsgemeinschaft under grant WA 1035/5-3. We kindly acknowledge the use of supercomputing facilities at HLRS Stuttgart and NIC J¨ ulich. The major part of the simulations was performed on the Cray Strider Opteron cluster at the High Performance Computing Center (HLRS) under the grant number sphshell /12714.

References 1. O. Alard, W.L. Griﬃn, J.P. Lorand, S.E. Jackson, and S.Y. O’Reilly. Nonchondritic distribution of the highly siderophile elements in mantle sulphides. Nature, 407:891–894, 2000.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

639

2. F. Albar`ede and R. van der Hilst. Zoned mantle convection. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 360:2569–2592, 2002. 3. C.J. All`egre. The evolution of mantle mixing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 360:2411– 2431, 2002. 4. C.J. All`egre and E. Levin. Isotopic systems and stirring times of the Earth’s mantle. Earth and Planetary Science Letters, 136:629–646, 1995. 5. Y. Amelin, D.C. Lee, and A.N. Halliday. Early-middle Archean crustal evolution from Lu-Hf and U-Pb isotopic studies of single zircon grains. Geochimica et Cosmochimica Acta, 64:4205–4225, 2000. 6. R.L. Armstrong. The persistent myth of crustal growth. Austral. J. Earth Sci., 38:613–640, 1991. 7. N. Arndt. Crustal growth rates. In K. Eriksson, W. Altermann, D.R. Nelson, W.U. Mueller, and O. Catuneanu, editors, The Precambrian Earth: Tempos and Events (Developments in Precambrian Geology, Vol. 12), pages 155–157. Elsevier, Amsterdam, 2004. 8. D.N. Barfod, C.J. Ballentine, A.N. Halliday, and J.G. Fitton. Noble gases in the Cameroon line and the He, Ne, and Ar isotopic compositions of HIMU mantle. Journal of Geophysical Research, 104:29509–29527, 1999. 9. T.W. Becker, J.B. Kellogg, and R.J. O’Connell. Thermal constraints on the survival of primitive blobs in the lower mantle. Earth and Planetary Science Letters, 171:351–365, 1999. 10. V.C. Bennett. Compositional evolution of the mantle. In R.W. Carlson, editor, Treatise on Geochemistry, Vol.2: The Mantle and the Core, pages 493–519. Elsevier, Amsterdam, 2003. 11. D. Bercovici and S. Karato. Theoretical analysis of shear localisation in the lithosphere. In S. Karato and H. Wenk, editors, Plastic Deformation of Minerals and Rocks (Rev. Mineral. Geochem.), volume 51, pages 387–420. Mineral. Soc. of Am., Washington, D.C., 2002. 12. D. Bercovici and S.-I. Karato. Whole-mantle convection and the transitionzone water ﬁlter. Nature, 425:39–44, 2003. 13. D. Bercovici and Y. Ricard. Tectonic plate generation and two-phase damage: Void growth versus grain size reduction. Journal of Geophysical Research, 110:B03401, 2005. 14. C.R. Bina. Seismological constraints upon mantle composition. In R.W. Carlson, editor, Treatise on Geochemistry, Vol.2: The Mantle and the Core, pages 39–59. Elsevier, Amsterdam, 2003. 15. D.D. Bogard, R.N. Clayton, K. Marti, T. Owen, and G. Turner. Chronology and evolution of Mars. Space Sci. Rev., 96:425–458, 2001. 16. S.A. Bowring and T. Housh. The Earth’s early evolution. Science, 269:1535– 1540, 1995. 17. A.D. Brandon, R.J. Walker, J.W. Morgan, and G.G. Goles. Re-Os isotopic evidence for early diﬀerentiation of the Martian mantle. Geochimica et Cosmochimica Acta, 64:4083–4095, 2000. 18. B.A. Buﬀett, H.E. Huppert, J.R. Lister, and A.W. Woods. On the thermal evolution of the Earth’s core. Journal of Geophysical Research, 101:7989–8006, 1996. 19. H.-P. Bunge. Global mantle convection models. PhD thesis, UC Berkeley, 1996.

640

U. Walzer, R. Hendel, J. Baumgardner

20. H.-P. Bunge and J.R. Baumgardner. Mantle convection modelling on parallel virtual machines. Computers in Physics, 9:207–215, 1995. 21. H.-P. Bunge, M.A. Richards, and J.R. Baumgardner. A sensitivity study of three-dimensional spherical mantle convection at 108 Rayleigh number: Eﬀects of depth-dependent viscosity, heating mode and an endothermic phase change. Journal of Geophysical Research, 102:11991–12007, 1997. 22. K.W. Burton, P. Schiano, J.-L. Birck, C.J. All`egre, M. Rehk¨ amper, A.N. Halliday, and J.B. Dawson. The distribution and behavior of rhenium and osmium amongst mantle minerals and the age of the lithospheric mantle beneath Tanzania. Earth and Planetary Science Letters, 183:93–106, 2000. 23. A.G.W. Cameron. The ﬁrst ten million years in the solar nebula. Meteoritics, 30:133–161, 1995. 24. U.R. Christensen and D.A. Yuen. Layered convection induced by phase transitions. Journal of Geophysical Research, 90:10291–10300, 1985. 25. M.F. Coﬃn and O. Eldholm. Large igneous provinces: Crustal structure, dimensions and external consequences. Reviews of Geophysics, 32:1–36, 1994. 26. N. Coltice and Y. Ricard. Geochemical observations and one layer mantle convection. Earth and Planetary Science Letters, 174:125–137, 1999. 27. N. Coltice and Y. Ricard. On the origin of noble gases in mantle plumes. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 360:2633–2648, 2002. 28. K.C. Condie. Incompatible element ratio in oceanic basalts and komatiites: tracking deep mantle sources and continental growth rates with time. Geochemistry, Geophysics, Geosystems, 4:1005, 1 2003. 29. V. Courtillot, A. Davaille, J. Besse, and J. Stock. Three distinct types of hot spots into the Earth’s mantle. Earth and Planetary Science Letters, 205:295– 308, 2003. 30. L. Cserepes, D.A. Yuen, and B.A. Schroeder. Eﬀect of the mid-mantle viscosity and phase-transition structure on 3D mantle convection. Physics of the Earth and Planetary Interiors, 118:135–148, 2000. 31. G.F. Davies. Geophysical and isotopic constraints on mantle convection: An interim synthesis. Journal of Geophysical Research, 89:6017–6040, 1984. 32. A.M. Dziewonski and D.L. Anderson. Preliminary reference Earth model. Physics of the Earth and Planetary Interiors, 25:297–356, 1981. 33. K.A. Farley, J.H. Natland, and H. Craig. Binary mixing of enriched and undegassed (primitive?) mantle components (He, Sr, Nd, Pb) in Samoan lavas. Earth and Planetary Science Letters, 111:183–199, 1992. 34. G. Fitton, J. Mahoney, P. Wallace, and A. Saunders, editors. Origin and Evolution of the Ontong Java Plateau, volume 229 of GSL Special Publications. Geological Society, London, 2004. 35. A.M. Forte and J.X. Mitrovica. Deep-mantle high-viscosity ﬂow and thermochemical structure inferred from seismic and geodynamic data. Nature, 410:1049–1056, 2001. 36. J.J. Gilvarry. The Lindemann and Gr¨ uneisen laws. Phys. Rev., 102:307–316, 1956. 37. G.A. Glatzmaier. Numerical simulations of mantle convection: Timedependent, three-dimensional, compressible, spherical shell. Geophys. Astrophys. Fluid Dyn., 43:223–264, 1988.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

641

38. K.-D. Gottschaldt, U. Walzer, R.F. Hendel, D.R. Stegman, J.R. Baumgardner, and H.-B. M¨ uhlhaus. Stirring in 3-d spherical models of convection in the Earth’s mantle. Philosophical Magazine, 86:3175–3204, 2006. 39. S.P. Grand, R.D. van der Hilst, and S. Widiyantoro. Global seismic tomography: A snapshot of convection in the Earth. GSA Today, 7:1–7, 1997. 40. M. Gurnis. The eﬀects of chemical density diﬀerences on convective mixing in the Earth’s mantle. Journal of Geophysical Research, 91:11407–11419, 1986. 41. B.B. Hanan and D.W. Graham. Lead and helium isotope evidence from oceanic basalts for a common deep source of mantle plumes. Science, 272:991–995, 1996. 42. S.R. Hart, E.H. Hauri, L.A. Oschmann, and J.A. Whitehead. Mantle plumes and entrainment: Isotopic evidence. Science, 256:517–520, 1992. 43. K.S. Heier. The distribution and redistribution of heat-producing elements in the continents. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 288:393–400, 1978. 44. G.R. Helﬀrich and B.J. Wood. The Earth’s mantle. Nature, 412:501–507, 2001. 45. D.R. Hilton and D. Porcelli. Noble gases as mantle tracers. In R.W. Carlson, editor, Treatise on Geochemistry, Vol.2: The Mantle and the Core, pages 277– 318. Elsevier, Amsterdam, 2003. 46. K. Hirose. Phase transitions in pyrolitic mantle around 670-km depth: Implications for upwelling of plumes from the lower mantle. Journal of Geophysical Research, 107(B4):2078, 2002. 47. P.F. Hoﬀman. Speculations on Laurentia’s ﬁrst gigayear (2.0 to 1.0 Ga). Geology, 17:135–138, 1989. 48. A.W. Hofmann. Chemical diﬀerentiation of the Earth: The relationship between mantle, continental crust and oceanic crust. Earth and Planetary Science Letters, 90:297–314, 1988. 49. A.W. Hofmann. Mantle geochemistry: The message from oceanic volcanism. Nature, 385:219–229, 1997. 50. A.W. Hofmann. Sampling mantle heterogeneity through oceanic basalts: Isotopes and trace elements. In R.W. Carlson, editor, Treatise on Geochemistry, Vol.2: The Mantle and the Core, pages 61–101. Elsevier, Amsterdam, 2003. 51. A.W. Hofmann, K.-P. Jochum, M. Seufert, and W.M. White. Nb and Pb in oceanic basalts: New constraints on mantle evolution. Earth and Planetary Science Letters, 79:33–45, 1986. 52. S. Honda and Y. Iwase. Comparison of the dynamic and parameterized models of mantle convection including core cooling. Earth and Planetary Science Letters, 139:133–145, 1996. 53. F.G. Houtermans. Determination of the age of the Earth from the isotopic composition of meteoritic lead. Nuovo Cimento, 10:1623–1633, 1953. 54. B.L. Isacks and P.M. Molnar. Distribution of stresses in the descending lithosphere from a global survey of focal-mechanism solutions of mantle earthquakes. Reviews of Geophysics and Space Physics, 9:103–174, 1971. 55. I. Jackson and S.M. Rigden. Composition and temperature of the Earth’s mantle: Seismological models interpreted through experimental studies of Earth materials. In I. Jackson, editor, The Earth’s Mantle. Composition, Structure and Evolution, pages 405–460. Cambridge University Press, Cambridge, UK, 1998.

642

U. Walzer, R. Hendel, J. Baumgardner

56. K.P. Jochum, A.W. Hofmann, E. Ito, H.M. Seufert, and W.M. White. K, U and Th in mid-ocean ridge basalt glasses and heat production, K/U and K/Rb in the mantle. Nature, 306:431–436, 1983. 57. S.-I. Karato. Phase transformations and rheological properties of mantle minerals. In D.J. Crossley, editor, Earth’s Deep Interior, pages 223–272. Gordon and Breach Sci. Publ., 1997. 58. S.-I. Karato and P. Li. Diﬀusion creep in perovskite: implications for the rheology of the lower mantle. Science, 255:1238–1240, 1992. 59. S.-I. Karato, Z. Wang, B. Liu, and K. Fujino. Plastic deformation of garnets: systematics and implications for the rheology of the mantle transition zone. Earth and Planetary Science Letters, 130:13–30, 1995. 60. S.-I. Karato and P. Wu. Rheology of the upper mantle: a synthesis. Science, 260:771–778, 1993. ˇ 61. M. Kido and O. Cadek. Inferences of viscosity from the oceanic geoid: indication of a low viscosity zone below the 660-km discontinuity. Earth and Planetary Science Letters, 151:125–137, 1997. 62. M. Kido and D.A. Yuen. The role played by a low viscosity zone under a 660 km discontinuity in regional mantle layering. Earth and Planetary Science Letters, 181:573–583, 2000. ˇ 63. M. Kido, D.A. Yuen, O. Cadek, and T. Nakakuki. Mantle viscosity derived by genetic algorithm using oceanic geoid and seismic tomography for whole-mantle versus blocked-ﬂow situations. Physics of the Earth and Planetary Interiors, 107:307–326, 1998. 64. S.H. Kirby, S. Stein, E.A. Okal, and D.C. Rubie. Metastable mantle phase transformations and deep earthquakes in subducting oceanic lithosphere. Reviews of Geophysics, 34:261–306, 1996. 65. T. Kleine, C. Muenker, K. Mezger, and H. Palme. Rapid accretion and early core formation on asteroids and the terrestrial planets from Hf-W chronometry. Nature, 418:952–955, 2002. 66. J.D. Kramers. Lead and strontium isotopes in Cretaceous kimberlites and mantle-derived xenoliths from Southern Africa. Earth and Planetary Science Letters, 34:419–431, 1977. 67. F.G. Lemoine, S.C. Kenyon, J.K. Factor, R.G. Trimmer, N.K. Pavlis, D.S. Chinn, C.M. Cox, S.M. Klosko, S.B. Luthcke, M.H. Torrence, Y.M. Wang, R.G. Williamson, E.C. Pavlis, R.H. Rapp, and T.R. Olson. The Development of the Joint NASA GSFC and the NIMA Geopotential Model EGM96. NASA Goddard Space Flight Center, NASA TP/-1998-206861:575 pp., 1998. 68. P. Li, S.-I. Karato, and Z. Wang. High-temperature creep in ﬁne-grained polycristalline CaTiO3 , an analogue material of (Mg, Fe)SiO3 perovskite. Physics of the Earth and Planetary Interiors, 95:19–36, 1996. 69. G.S. Lister, M.A. Forster, and T.J. Rawling. Episodicity during orogenesis. In J.A. Miller, R.E. Holdsworth, I.S. Buick, and M. Hand, editors, Continental Reactivating and Reworking, volume 184 of GSL Special Publications, pages 89–113. Geological Society, London, 2001. 70. C. Matyska and D.A. Yuen. Lower mantle dynamics with the postperovskite phase change, radiative thermal conductivity, temperature- and depth-dependent viscosity. Physics of the Earth and Planetary Interiors, 154:196–207, 2006.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

643

71. M.T. McCulloch and V.C. Bennett. Progressive growth of the Earth’s continental crust and depleted mantle: Geochemical constraints. Geochimica et Cosmochimica Acta, 58:4717–4738, 1994. 72. M.T. McCulloch and V.C. Bennett. Early diﬀerentiation of the Earth: An isotopic perspective. In I. Jackson, editor, The Earth’s Mantle. Composition, Structure and Evolution, pages 127–158. Cambridge University Press, Cambridge, UK, 1998. 73. W.F. McDonough and S.-S. Sun. The composition of the Earth. Chem. Geol., 120:223–253, 1995. 74. C. Meade and R. Jeanloz. The strength of mantle silicates at high pressures and room temperature: implications for the viscosity of the mantle. Nature, 348:533–535, 1990. 75. R. Montelli, G. Nolet, F.A. Dahlen, G. Masters, E.R. Engdahl, and S.-H. Hung. Finite-frequency tomography reveals a variety of plumes in the mantle. Science, 303:338–343, 2004. 76. R.D. Nance, T.R. Worsley, and J.B. Moody. Post-archean biogeochemical cycles and long-term episodicity in tectonic processes. Geology, 14:514–518, 1986. 77. M. Norman, L. Borg, L. Nyquist, and D. Bogard. Chonology, geochemistry and petrology of a ferroan noritic anorthosite clast from Descartes breccia 67215: clues to the age, origin, structure and impact history of the lunar crust. Meteorit. Planet. Sci., 38:645–661, 2003. 78. L.E. Nyquist, D.D. Bogard, C.-Y. Shih, A. Greshake, D. St¨ oﬀler, and O. Eugster. Ages and geological histories of Martian meteorites. Space Sci. Rev., 96:105–164, 2001. 79. R.K. O’Nions and I.N. Tolstikhin. Limits on the mass ﬂux between lower and upper mantle and stability of layering. Earth and Planetary Science Letters, 139:213–222, 1996. 80. H. Palme and H.S.C. O’Neill. Cosmochemical estimates of mantle composition. In R.W. Carlson, editor, Treatise on Geochemistry, Vol.2: The Mantle and the Core, pages 1–38. Elsevier, Amsterdam, 2003. 81. S.V. Panasyuk and B.H. Hager. Inversion for mantle viscosity proﬁles constrained by dynamic topography and the geoid and their estimated errors. Geophysical Journal International, 143:821–836, 2000. 82. N.J. Pearson, O. Alard, W.L. Griﬃn, S.E. Jackson, and S.Y. O’Reilly. In situ measurement of Re-Os isotopes in mantle sulﬁdes by laser ablation multicollector inductively coupled plasma mass spectrometry: analytical methods and preliminary results. Geochimica et Cosmochimica Acta, 66:1037–1050, 2002. 83. H.N. Pollak, S.J. Hurter, and J.R. Johnson. Heat ﬂow from the Earth’s interior: analysis of the global data set. Reviews of Geophysics, 31:267–280, 1993. 84. D. Porcelli and C.J. Ballentine. Models for the distribution of terrestrial noble gases and evolution of the atmosphere. In D. Porcelli, C.J. Ballentine, and R. Wieler, editors, Noble Gases in Geochemistry and Cosmochemistry (Rev. Mineral. Geochem.), volume 47, pages 411–480. Mineral. Soc. of Am., Washington, D.C., 2002. 85. D. Porcelli and A.N. Halliday. The core as a possible source of mantle helium. Earth and Planetary Science Letters, 192:45–56, 2001. 86. A. Ramage and A.J. Wathen. Iterative solution techniques for ﬁnite element discretizations of ﬂuid ﬂow problems. In Proceedings of the Copper Mountain Conference on Iterative Methods, volume 1, Copper Mountain, Colorado, 1992.

644

U. Walzer, R. Hendel, J. Baumgardner

87. A. Reymer and G. Schubert. Phanerozoic addition rates to the continental crust and crustal growth. Tectonics, 3:63–77, 1984. 88. M.A. Richards, W.-S. Yang, J.R. Baumgardner, and H.-P. Bunge. Role of a low-viscosity zone in stabilizing plate tectonics: Implications for comparative terrestrial planetology. Geochemistry, Geophysics, Geosystems, 3:1040, 2001. 89. F.M. Richter. Finite amplitude convection through a phase boundary. Geophys. J. R. Astron. Soc., 35:265–276, 1973. 90. R.L. Rudnick and D.M. Fountain. Nature and composition of the continental crust: A lower crustal perspective. Reviews of Geophysics, 33:267–309, 1995. 91. R.L. Rudnick and S. Gao. Composition of the continental crust. In R.L. Rudnick, editor, Treatise on Geochemistry, Vol.3: The Crust, pages 1–64. Elsevier, Amsterdam, 2003. 92. V.J.M. Salters and W.M. White. Hf isotope constraints on mantle evolution. Chem. Geol., 145:447–460, 1998. 93. H. Schmeling. A model of episodic melt extraction for plumes. Journal of Geophysical Research, 111:B03202, 2006. 94. G. Schubert, D.L. Turcotte, and T.R. Olson. Mantle convection in the Earth and Planets. Cambridge University Press, Cambridge, UK, 2001. 95. V. Steinbach and D.A. Yuen. Eﬀects of depth-dependent properties on thermal anomalies produced in ﬂush instabilities from phase transitions. Physics of the Earth and Planetary Interiors, 86:165–183, 1994. 96. V. Steinbach, D.A. Yuen, and W.L. Zhao. Instabilities from phase transitions and the timescales of mantle thermal convection. Geophysical Research Letters, 20:1119–1122, 1993. 97. D.J. Stevenson. Origin of the moon: the collision hypothesis. Annu. Rev. Earth Sci., 15:271–315, 1987. 98. A. Stracke, A.W. Hofmann, and S.R. Hart. FOZO, HIMU and the rest of the mantle zoo. Geochemistry, Geophysics, Geosystems, 6:Q05007, 2005. 99. Y.J. Su. Mid-ocean ridge basalt trace element systematics: Constraints from database management, ICP-MS analyses, global data compilation and petrologic modeling. PhD thesis, Columbia University, 2002. 100. P. Tackley. The quest for self-consistent generation of plate tectonics in mantle convection models. In M.A. Richards, R. Gordon, and R. van der Hilst, editors, History and Dynamics of Global Plate Motions, Geophysical Mograph Series, Vol. 121, pages 47–72. American Geophysical Union, Washington, D.C., 2000. 101. P.J. Tackley. Self-consistent generation of tectonic plates in time-dependent, three-dimensional mantle convection simulations. Geochemistry, Geophysics, Geosystems, 1:No. 8, 2000. 102. S.R. Taylor and S.M. McLennon. The Continental Crust: Its Composition and Evolution. Blackwell Scientiﬁc, Oxford, 1985. 103. S.R. Titley. Relationship of stratabound ores with tectonic cycles of the phanerozoic and proterozoic. Precambrian Research, 61:295–322, 1993. 104. I. Tolstikhin and B. Marty. The evolution of terrestrial volatiles: A view from helium, neon, argon and nitrogen isotope modelling. Chem. Geol., 147:27–52, 1998. 105. W.B. Tonks and H.J. Melosh. Magma ocean formation due to giant impacts. Journal of Geophysical Research, 98:5319–5333, 1993. 106. J. Trampert, F. Deschamps, J. Resovsky, and D.A. Yuen. Probabilistic tomography maps chemical heterogeneities throughout the lower mantle. Science, 306:853–856, 2004.

Mantle Convection, Continent Generation and Geochemical Heterogeneity

645

107. M. Trieloﬀ and J. Kunz. Isotope systematics of noble gases in the Earth’s mantle: Possible sources of primordial isotopes and impliﬁcations for mantle structure. Physics of the Earth and Planetary Interiors, 148:13–38, 2005. 108. R. Trompert and U. Hansen. Mantle convection simulations with rheologies that generate plate-like behavior. Nature, 395:686–689, 1998. 109. R. van der Hilst, S. Widiyantoro, and E.R. Engdahl. Evidence for deep mantle circulation from global tomography. Nature, 386:578–584, 1997. 110. P.E. van Keken, E.H. Hauri, and C.J. Ballentine. Mantle mixing: The generation, preservation and destruction of chemical heterogeneity. Annu. Rev. Earth Planet Sci., 30:493–525, 2002. 111. J.D. Vervoort and J. Blichert-Toft. Evolution of the depleted mantle: Hf isotope evidence from juvenile rocks through time. Geochimica et Cosmochimica Acta, 63:533–556, 1999. 112. U. Walzer and R. Hendel. A new convection-fractionation model for the evolution of the principal geochemical reservoirs of the Earth’s mantle. Physics of the Earth and Planetary Interiors, 112:211–256, 1999. 113. U. Walzer, R. Hendel, and J. Baumgardner. The eﬀects of a variation of the radial viscosity proﬁle on mantle evolution. Tectonophysics, 384:55–90, 2004. 114. U. Walzer, R. Hendel, and J. Baumgardner. Toward a thermochemical model of the evolution of the Earth’s mantle. In E. Krause, W. J¨ ager, and M. Resch, editors, High Performance Computing in Science and Engineering ’04, pages 395–454. Springer, Berlin, 2004. 115. K.H. Wedepohl. The composition of the continental crust. Geochimica et Cosmochimica Acta, 59:1217–1232, 1995. 116. M. Willbold and A. Stracke. Trace element composition of mantle endmembers: Implications for recycling of oceanic and upper and lower continental crust. Geochemistry, Geophysics, Geosystems, 7:Q04004, 2006. 117. M. Wilson and E.A. Spencer. The origin and evolution of the FOZO/PREMA and HIMU mantle components - the carbonatite perspective. http://www.geo.uw.edu.pl/ERASMUS/ﬁles/wilson3.ppt, 2003. 118. G. W¨ orner, A. Zindler, H. Staudigel, and H.U. Schmincke. Sr, Nd and Pb isotope geochemistry of Tertiary and Quarternary alkaline volcanics from West Germany. Earth and Planetary Science Letters, 79:107–119, 1986. 119. T.R. Worsley, R.D. Nance, and J.B. Moody. Tectonic cycles and the history of the Earth’s biogeochemical and paleoceanographic record. Paleoceanography, 1:233–263, 1986. 120. D. Yamazaki and S.-I. Karato. Some mineral physics constraints on the rheology and geothermal structure of the Earth’s lower mantle. American Mineralogist, 86:385–391, 2001. 121. W.-S. Yang. Variable viscosity thermal convection at infinite Prandtl number in a thick spherical shell. PhD thesis, University of Illinois, Urbana-Champaign, 1997. 122. D.A. Yuen, U. Hansen, W. Zhao, A.P. Vincent, and A.V. Malevsky. Hard turbulent thermal convection and thermal evolution of the mantle. Journal of Geophysical Research, 98:5355–5373, 1993. 123. D.A. Yuen and A.V. Malevsky. Strongly chaotic Newtonian and nonNewtonian mantle convection. In D.A. Yuen, editor, Chaotic Processes in the Geological Sciences, pages 71–88. Springer, New York, 1992. 124. A. Zindler and S. Hart. Chemical geodynamics. Annu. Rev. Earth Planetary Sci., 14:493–571, 1986.

Numerical Simulation of Human Radiation Heat Transfer Using a Mathematical Model of Human Physiology and Computational Fluid Dynamics (CFD) Rehan Yousaf1,2,3 , Dusan Fiala1,3,4 , and Andreas Wagner1 1

2

3 4

Division for Building Physics and Building Services (fbta), University of Karlsruhe, Germany {rehan.yousaf,dusan.fiala,andreas.wagner}@fbta.uni-karlsruhe.de Ventilation and Safety Division, Poyry Infra AG, Switzerland [email protected] Institute of Energy and Sustainable Development, De Montfort University, UK Institute of Construction Economics, University of Stuttgart [email protected]

Summary. Human-environment thermal interactions play an important role in numerous areas of human endeavour from the safety of ﬁre-ﬁghters, thermal comfort in buildings to energy eﬃcient planning of heating and ventilation systems. The thermal interactions that occur between humans and their immediate environments are however very complex and thus diﬃcult to predict. The project currently being carried out at Building Sciences Group (fbta) at the University of Karlsruhe started in March 2006. This project is part of a four year research project in collaboration with the Institute of Energy and Sustainable Development (IESD) at the De Montfort University in Leicester, UK and the Technical University of Denmark. The focus is on the complex human-environment interactions in naturally ventilated buildings to predict human thermal comfort. To correctly predict the thermo-physiological interactions of humans with their surroundings that dictate the perception of thermal comfort, an advanced and widely validated cybernetic multi-segmental thermophysiological model namely “IESD Fiala Model” is employed. The model is used to provide the necessary physiological data as boundary conditions for Computational Fluid Dynamics (CFD) simulations incorporating a detailed geometry model of the human body and the surrounding space. CFD techniques can predict in detail the spatial variations of air temperature, air speed, turbulence intensity, radiation eﬀects on heat transfer, moisture and pollutant concentrations. At the moment a number of CFD codes exist, but due to time constraints only ANSYS CFX Version 10.0 [1] is used in this project. ANSYS CFX CFD code is being widely used and validated for modelling heat and mass transfer in indoor spaces. The need to simulate in detail the personal exposures to indoor climates and contaminant movement, detailed geometry models of the human body commonly known as “Computer Simulated Person (CSP)”, appropriate for numerical studies are needed. The problem of cre-

648

R. Yousaf, D. Fiala, A. Wagner

ating and modelling accurate 3D CSP’s has been resolved by the availability of specialist software tools like Poser [2].

1 Objectives and Methodology The aim of the project is to develop a validated simulation system capable of predicting in detail the impact of buildings on their occupants, their comfort and the eﬀect of human occupants on the natural ventilation regime. The research aim will be achieved through the following objectives: • Modelling surface geometry of the human body and meshing it for the use in wide range of applications, from detailed human body resolution for health and safety research to simplified human body resolution for simulation of a number of humans in large indoor and outdoor spaces. • Establishing techniques for modelling local radiative and convective heat transfer phenomena at surfaces of the human body. • Establishing techniques for modelling heat loss due to evaporation and respiration. • Developing computational interfaces for iterative data exchange between ANSYS CFX and IESD-Fiala models, thereby coupling the heat and mass transfer phenomena taking place in and around the human body. • Validating the complete coupled system using data obtained from experimental investigations. • Demonstrating the application of the simulation system for an existing naturally ventilated building. • Providing generic guidelines on CFD modelling of human-environment interactions, including the integration of simulation subsystems, its accuracy and limitations. This report focuses on geometry and mesh generation for human body, comparison and validation of radiation models available in ANSYS CFX, comparison and validation of factors aﬀecting human radiative heat transfer and human thermal comfort.

2 Modeling Geometry and Meshing Detailed models of the human body surfaces were developed using one of the specialist software “Poser 6.0 [2]” which enabled the creation of realistic 3D human body geometries for various postures and clothing ensembles. Some example geometries are shown in the Figs. 1 to 3. Meshing is the process of building up the calculation domain for resolving flow physics. It is one of the most time consuming and challenging tasks when it comes to numerical simulations. A number of commercial tools are available for meshing both general and application focused geometries. Once the human geometries are

TC-CFD

Fig. 1. 3D standing nude man modelled in Poser

649

Fig. 2. 3D standing clothed man modelled in Poser

Fig. 3. 3D seated clothed man modelled in Poser

generated using the Poser software, they are exported to an advanced new generation CAD cleaner and mesh generator software ANSYS ICEM CFD [3]. Some of the geometries meshed for standing postures are shown in Fig. 4. Best possible calculation cells for resolving heat and mass transfer phenomena are regular shaped hexahedral cells which, due to their geometry characteristics, provide a regular calculation domain. This however is not a trivial task due to the complexity of geometry involved. For this reason, other geometrical

650

R. Yousaf, D. Fiala, A. Wagner

Fig. 4. Examples of meshed geometries using ANSYS ICEM CFD

shape cells like tetrahedral and pyramids are needed to form a calculation domain. A number of various algorithms provided by ANSYS ICEM CFD e.g. patch based, patch independent, block structure, delauny approach [3] etc were studied during the course of this project. In the first year of the project run, focus was put on high quality surface and volume meshes. One such advanced technique employed is “hybrid meshing” [3]. Hybrid meshing combines the advantages of hexahedral and polyhedral meshing techniques, which, when intelligently used, can minimize the number of cells to a large extent. Figure 5 is a representation of one of such hybrid meshes used for radiation heat transfer studies. The human body is surrounded by tetrahedral (blue) in the near region and by hexahedral (pink) in the far region. The two regions (mesh types) are merged together by using transitional pyramids.

3 Modelling Radiation Radiative heat transfer is dependent on a number of factors such as the wavelength characteristics of the radiation, surface properties of the medium and materials, the shape of the radiating objects as well as their orientations. In ANSYS CFX, radiation heat transfer is numerically modelled using a generic, first order integro-diﬀerential equation of spectral radiation intensity which is a function of object’s position and direction. This equation is known as “Radiation Transport Equation (RTE)” and is given as follows [4]: dIν(r,s) Ksν = −(Kav + Ksv )Iν(r,s) + Kav Ib(ν,T ) + dIν(r,s) φ(S.S ′ )dΩ (1) ds 4π 4π

TC-CFD

651

Fig. 5. Example of a hybrid mesh with blue zone meshed with tetrahedral, pink cells with hexahedral and the junction between the two with pyramid shaped cells

Where r is the position vector s is the direction vector s′ is the path length [m] Ka is the absorption coeﬃcient [m-1] Ks is the Scattering coeﬃcient [m-1] lb is the Blackbody emission intensity [W/(m2 Sr)] I is the spectral radiation intensity depending on position and direction [W/(m2 Sr)] T is the local absolute temperature[K] Ω is the solid angle[Sr] φ is the in-scattering phase function [Sr−1 ] The solution of RTE is iterative and is highly dependent on the three spatial co-ordinates, 2 local directions and the frequency of the wavelength(s) involved. ANSYS CFX has implemented a number of radiation models for various practical purposes. They are divided into two categories, namely directional and spectral models. Directional Models 1. 2. 3. 4.

Rosseland Model or Diﬀusion Approximation Model Spherical Harmonics or P1 Model Discrete Transfer Model Monte Carlo Model

652

R. Yousaf, D. Fiala, A. Wagner

Spectral Models 1. Gray Model 2. Multi-band Model 3. Weighted Sum of Gray Gases Model For directional approximation in optically thin media such as air under normal indoor climatic conditions, only Discrete Transfer Radiation Model (DTRM) or the Monte Carlo technique can be used [1, 4]. As a first approximation a gray spectral model was chosen in this study which treats the domain and surfaces as wavelength independent. 3.1 Comparison of Radiation Models As described above only DTRM and Monte Carlo were chosen and subjected to further analysis. The analysis evaluated the accuracy and the robustness of the respective models. Room geometries were modelled and view factors (defined as the amount of energy leaving one surface and arriving at the other) were predicted as a common criterion for model validation (see next section for more detail). It is important to note here that the Discrete Transfer Radiation Model (DTRM) method is characterized by the number of rays leaving a surface at definite discretized angles while the Monte Carlo method is characterized by the number of histories that are recorded for the trajectory of a photon leaving a surface (based on statistical sampling). These two characteristic parameters also dictate the accuracy of the individual radiation models and were varied in the study. All simulations were carried out on single compute node at the Computer Cluster XC4000 [5] each with 2 AMD Opteron processors having frequency of, 2,6 GHz and main memory of 16 GB, located at Scientific Computing Centre at the University of Karlsruhe. As shown in

Fig. 6. Accuracy comparison between DTRM and Monte Carlo Method

TC-CFD

653

Fig. 7. Robustness comparison between DTRM and Monte Carlo Method

Fig. 6, the accuracy of DTRM increases with an increasing number of rays that leave a bounded surface. The Monte Carlo method shows a rather oscillating behaviour regarding an increasing number of recorded histories but, on average and even at low numbers of histories, this method has a significantly improved accuracy compared to “Discrete Transfer Radiation Model. In addition to the above numerical accuracy test, a robustness check was also made in order to quantify the eﬀectiveness of the models in terms of calculation time. Figure 7 shows that the time requirements for the two models subject to wide ranging variations. The reason why DTRM requires in parts significantly more computational time suggests a detailed understanding of the numerical implementation of the model described elsewhere. [4, 6] 3.2 Validation of the Monte Carlo Radiation Model for Simple Geometries Before using the Monte Carlo radiation model for complex human geometrical shapes, for which no analytical solutions exist, it was necessary to validate the model for some assemblies of basic geometrical shapes encountered frequently in indoor conditions, for which analytical solutions do exist. Validation for Parallel Plates Parallel plates (Figs. 8 and 9) can be analogized to two walls facing each other in a room. The analytical solution for such a situation is given by [4]: 5 : 2 X (1 + X 2 )(1 + Y 2 ) F1−2 = { 1 + Y 2 tan−1 ( √ + X )} (2) πXY 1 + X2 + Y 2 1+Y2

654

R. Yousaf, D. Fiala, A. Wagner

Fig. 8. Two parallel opposing walls of a room with dimensions of a [m] * b [m] and held at a distance of c [m]

Fig. 9. Comparison of view factor calculated by analytical solution and CFD simulation

Where F1−2 is the view factor between the plates 1 and 2. X = a/c and Y = b/c. Validation for Perpendicular Plates Perpendicular plates (Figs. 10 and 11) can be analogized with the condition occurring at the junction between floor/ceiling and its adjacent wall in a room. The analytical solution for such a situation is given by [4]:

5 : 1 1 1 F1−2 = btan−1 + ctan−1 − c2 + b2 tan−1 +A (3) b c c2 + b 2

TC-CFD

655

Fig. 10. Two perpendicular walls of a room with dimensions of b [m] * c [m]

Fig. 11. Comparison of view factor calculated by analytical solution and CFD simulation

where

b2

2 1 + b 2 1 + c2 b 1 + b 2 + c2 1 A= ln 4 1 + b 2 + c2 (1 + b2 ) (b2 + c2 )

c2 c 2 1 + b 2 + c2 × (1 + c2 ) (b2 + c2 )

(4)

Where F1−2 is the view factor between plate 1 and 2. Validation for Cylinder Facing a Plate This case (Figs. 12 and 13) can be analogized with a human body modelled as cylinder enclosed in a regular shape room. Such approximation for a human

656

R. Yousaf, D. Fiala, A. Wagner

Fig. 12. A cylinder of radius r [m] and height a [m] is facing a wall of height a [m] and width b [m]

Fig. 13. Comparison of view factor calculated by analytical solution and CFD simulation

body have been employed so far in a number of studies [7]. The analytical solution as given by K. Kubaha [8] is given as B2 2 f (g)dg (5) φpc = B 0 Where φpc represents the view factor between the plate and cylinder. And f(g) is given by the relation [4]: A (A) f (g) = × (E) − A2 + g 2 π (A2 + g 2 ) where −

E = cos 1

Y X

−

1 2C

: X 2 + 4C 2 cos− 1

Y

X

: A2 + g 2

(6)

+ (F )

(7)

TC-CFD

where F = Y sin− 1

1

: A2 + g 2

−

πX 2

657

(8)

Where A = c/r, B = b/r, C = a/r X = A2 +C 2 +g 2 −1 and Y = C 2 −A2 −g 2 +1. Validation for Sphere Facing a Wall Some studies e.g. [9]treated human body as sphere. Analytical solution for such a case (Figs. 14 and 15) is given as [4]:

Fig. 14. A sphere of radius r [m] and at d [m] from a wall with section l1 [m] * l2 [m]

Fig. 15. Comparison of view factor calculated by analytical solution and CFD simulation

658

R. Yousaf, D. Fiala, A. Wagner

F1−2

1 tan−1 = 4π

1 D12 + D22 + D12

D22

0.5

(9)

Where F1−2 is the view factor between sphere and opposing wall D1 = d/l1 D2 = d/l2 .

4 Modelling Human Radiative Heat Transfer About 40% to 70% of the total heat loss from the human body surface to the environment takes place due to radiative heat exchange (both long and short wave radiation) [10, 11], this makes it inevitably important in human thermal comfort predictions. This simulation study focuses on numerical calculation of human project areas encountered when dealing with directional short wave radiation (e.g. solar radiation), view factors for dealing with of long wave radiation heat exchange and eﬀective radiation areas that take part in radiative heat exchanges. These factors incorporate the eﬀects of geometrical features and orientation of the participating objects. These factors together with material properties like emissivity, reflectivity, absorptivity etc influence the radiation heat transfer process [7, 8, 9, 10, 12]. In this study the use of numerical simulation techniques to model human radiation heat exchange phenomena for purposes of detailed human thermal comfort analysis is explored. 4.1 Need of Computational Resources Modelling radiation requires a solution of the Radiation Transport Equation (Sect. 4) by employing diﬀerent approximation models. CFD codes when used for numerical heat and mass transfer need an appropriate surface and domain mesh as a calculation field. Depending upon the type of the problem under consideration, the size of mesh can vary from a few thousand to millions of cells demanding high computational resources. The Scientific Computing Centre of the University of Karlsruhe is equipped with up to date ANSYS CFX software and provides high computational power by providing up to 20 parallel processes on Opteron II dual core processors on a high performance HP XC4000 [5] Cluster. For this study, up to 4 parallel processes have been used so far. A preliminary mesh independency study by the authors indicated that about 1 Gigabyte of RAM is required to starting a calculation using a mesh of approximately 1 million cells. Once the memory estimation and allocation is done by the program (ANSYS CFX), further calculation proceeds and time of these later calculations depend upon the simulation methodology and the question of interest.

TC-CFD

659

4.2 Choosing Benchmark In order to quantify the accuracy of the numerical methods employed to simulate the human radiative heat transfer phenomena, a bench mark had to be established. The most prominent work in the field of human thermal comfort is that conducted by P.O. Fanger [10], which is incorporated in many national and international standards on occupant comfort in buildings e.g. ISO 7730 and ASHRAE Standard 55. Fanger’s work is based on experimental investigations of human projected area (the area projected on a plane perpendicular to parallel rays) of standing and sedentary human subjects using photographic techniques [10, 15]. Based on the projected areas, Fanger devised eﬀective radiation areas (area of the human body that takes part in radiation exchange with the surroundings) as well as view factors between the human body and its surroundings. Once the human radiation data based on the above noted factors is acquired, one of the most influencing thermal comfort criteria i.e. the “Mean Radiant Temperature (MRT)”, which is defined as “that uniform temperature of an imaginary black enclosure which results in same heat loss by radiation from the person, as the actual enclosure” [7, 10] can be calculated. MRT is one of the most influencing criteria for estimation of human thermal comfort. 4.3 Simulation Methodology The experimental methodology followed by Fanger based the calculation of radiation data on “projected areas of human bodies”. This method leads to inaccuracies in case of an error in calculation of projected areas of human bodies for the dependent variables i.e. view factors and eﬀective radiation areas. Following a numerical approach however eliminates any such dependency of order thereby leading to controllable and less error prone results. For this study, the following order was chosen. 1. Calculation of view factors using the net radiation exchange method for the human body as a whole and comparing it with the experimental results obtained by Fanger. 2. Calculation of the eﬀective radiation area factor, defined as “the ratio between the body areas that takes part in radiation to the total body surface area”, and comparing it with the results obtained by Fanger. 3. Calculation of the projected area factor defined as “the ratio between the projected area and the eﬀective radiation area”, and comparing it with the results obtained by Fanger. In order to minimize the computational load various simplifications were employed in the simulation methodology as given below. •

The medium (indoor air) was considered as non participating medium for radiation heat transfer. This assumption is truly valid in indoor climates with very low optical thickness [1, 4] of the medium.

660

R. Yousaf, D. Fiala, A. Wagner

• The human body was not subdivided at this stage into sectors but was considered as a whole. • Flow and turbulence equations were not solved in parallel to radiation equations. • All surfaces were assumed to be gray diﬀuse. • Surfaces of non-interest were assumed to be black bodies (emission = absorption = 1) and setting their surface temperatures to 0 Kelvin in order to minimize the interference eﬀects. 4.4 Comparison Studies View Factor The human body geometry was enclosed by a rectangular space in such a way that its centre was at 3 m distance from the front wall as shown in Fig. 16. The length and height of the walls were then varied to get the dimensionless ratios a/c and b/c as shown in Fig. 17. The simulations were done for a number of a/c and b/c ratios and were compared with the results provided by the experimental work of Fanger [10, 16]. This comparison is given in Fig. 18. It is important to mention here that altering the a/c and b/c ratios led to changes in domain size. In order to keep consistent with the quality of mesh, mesh cells had to be increased with an increase in domain size. This resulted in number of mesh cells varying from a minimum of 500,000 (for the smallest a/c and b/c ratios) to a maximum of approx. 6 million (for the largest a/c and b/c ratios). As diﬀerent in-phase interferences were not included in radiation calculations, the time requirements for calculations using Monte Carlo method

Fig. 16. Human body in an enclosed surrounding.

TC-CFD

661

Fig. 17. Geometrical setup for view factor calculations.

Fig. 18. Comparison of View Factors

varied from 2 minutes on HP XC4000 [5] for the smallest mesh to about 6 minutes for largest mesh. Eﬀective Radiation Area Factor To investigate the dependency of posture on eﬀective radiation area factor, three diﬀerent standing postures were modelled as shown in Fig. 19.

662

R. Yousaf, D. Fiala, A. Wagner

Fig. 19. Various postures of the human body

Fig. 20. Variation in eﬀective area factor of with variation in posture

Fanger calculated an eﬀective radiation area factor of 0.725 for a standing person ignoring the eﬀects of clothing, postures and the convexities of the human body shape. In contrast to Fanger, the results of this numerical simulation study however indicate that even slight change in posture can significantly influence the eﬀective radiation area factor, namely from about 13% for posture P1 to about 7% for posture 3 (compared to Fanger’s Aeﬀ = 0.725). Projected Area Factor Based on the position of the human body in space, the direction of solar rays is defined by an azimuth angle α and an altitude angle β as shown in the Fig. 21. The area of human body facing direct radiation was projected on a plane perpendicular to the direction of solar radiation. Due to the diﬀerences in the eﬀective radiation area factors of the human body between the models used in this study and subjects in the Fanger experiments (stiﬀ postures), a direct comparison of simulated and experimentally measured projected area factors

TC-CFD

663

Fig. 21. Explaination of α and β as given by Fanger

Fig. 22. Projected area on the ﬂoor for altitude angle β = 90◦

is not appropriate to be made. In order to enable a comparison, the predicted projected area factors were normalised by dividing the predicted values by factor 0.725 which was the eﬀective area factor of Fanger’s subjects.The comparison is presented for β = 90◦ and α = 0◦ in Fig. 24.

5 Summary, Conclusions and Future Work In this study the performance of numerical techniques implemented in the software ANSYS CFX to simulate radiative heat transfer for simple and complex (human body) geometries was analysed. The validation studies carried out for simple geometrical assemblies showed a very good agreement with the

664

R. Yousaf, D. Fiala, A. Wagner

Fig. 23. Projected area on the wall for altitude angle β = 0◦ and azimuth angle α = 90◦

Fig. 24. Comparison between projected area factors as obtained by Fanger and as computed by detailed CFD simulations for altitude angles of β = 0◦ and azimuth angle α = 90◦ .

TC-CFD

665

corresponding analytical solutions. The maximum relative error encountered was typically less than 2% in most cases. It is however important to note that depending upon the domain size, the number of histories of photons should be adjusted. Moreover. the surface mesh should be fine enough especially at the junction of surfaces (preferably hexahedral) to capture the surface orientation correctly and minimize errors induced due to integration etc. In the case of human body geometries, the predicted view Factors varied slightly from those obtained experimentally by Fanger. A careful analysis of the results indicated that the discrepancies were associated with diﬀerences in the body posture and with some 3D characteristics such as surfaces convexities and concavities which were ignored in Fanger study. Literature sources [8, 13, 14, 15] indicate values for eﬀective radiation area factor for standing subjects between 0.75 up to 0.85. The values for eﬀective radiation area factors calculated by Fanger are much lower due to consideration of a rather stiﬀ posture. The present work, in agreement with the literature, indicates higher eﬀective radiation area factors. For the case of short wave radiation (directional radiation flux), the normalized projected area factors obtained using numerical simulations showed close agreement with data obtained experimentally by Fanger. As mentioned earlier the calculated projected area factor obtained using Monte Carlo radiation model is divided by the eﬀective area given by Fanger (0.725) in order to make the comparison. Further investigations are needed to prove the presumptions made so far regarding • • •

Eﬀect of the distance between the human body geometry and facing wall on view factors Eﬀect of body posture on view factors. Calculation of MRT.

References 1. ANSYS CFX 2. Poser 6.0 .The Premier 3D Design Tool. User Guide, Curious Labs Inc., Santa Cruz, CA, USA, 2006. 3. ANSYS ICEM CFD 4. R. Siegel, J.R. Hower, Thermal Radiation Heat Transfer (1992), 3rd Edition, Hemisphere Publishing. 5. http://www.rz.uni-karlsruhe.de/rz/docs/HP-XC/ug/ug4k.pdf 6. Astrid Mobiok, Roman Weber, Radiaton in Enclosures, Scientiﬁc Computation (2000), Springer. 7. Toshiyuki Miyanaga, Wataru Urabe, Yukio Nakano, Simpliﬁed human body model for evaluating thermal radiant environment in a radiant cooled space. 8. K. Kubaha, Asymmetric Radiant Fields and Human Comfort, PhD Thesis (2005), Institute for Energy and Sustainable Development, DeMontfort University, England. 9. M.M. Ardehali, Nirvan G. Panah, T. F. Smith, Proof of concept modeling of energy transfer mechanisms for radiant conditioning panels (2004), Energy Conversion & Management 45 2005–2017.

666

R. Yousaf, D. Fiala, A. Wagner

10. P.O.Fanger, Thermal Comfort, Analysis and Applications in Environmental Engineering, McGraw-Hill Book Company, 1972. 11. M. David Egan, (March 1975), Concepts in Thermal Comfort, Prentice Hall. 12. Shin-ichi Tanabe, Chie Narita, Yoshiichi Ozeki, Masaaki Konishi, Eﬀective radiation area of human body calculated by a numerical simulation, Energy and Buildings, 32 (2000) 205–215. 13. Raber, B.F., and Hutchinson, F.W., Optimum surface distribution in panel heating and cooling systems .ASHVE Trans. 48: 231–265, 1944. 14. Raber, B.F., and Hutchinson, F.W., Panel heating and cooling analysis .John Wiley & Sons, Inc., New York, 1947. 15. Dunkle, R.V.,Conﬁguration factors for radiant heat transfer calculations involving people. Journ. Of Heat Transfer, 83:71–76, 1963. 16. G .Cannistraro, G. Franzitta, C. Giaconia, G. Rizzo (1992), Algorithms for the calculation of the view factors between human body and rectangular surfaces in parallelepiped environments, Enery and Buildings, 19 (1992) 51–60.

Parallel Finite Element Methods with Weighted Linear B-Splines* Klaus H¨ ollig, J¨ org H¨ orner, and Martina Pfeil IMNG, University of Stuttgart, Pfaﬀenwaldring 57, 70569 Stuttgart, Germany [email protected]

Summary. Weighted extended B-splines (web-splines) combine the computational eﬃciency of B-splines and the geometric ﬂexibility of standard ﬁnite elements on unstructured meshes. These new ﬁnite elements on uniform grids (cf. [5] and www.webspline.de) are ideally suited for vectorization, parallelization and multilevel techniques. In this project we explore the potential of the web-method for large scale applications with performance tests on the NEC SX-8 cluster of the HLRS. We implement a new minimal degree variant which uses predeﬁned instruction sequences for matrix assembly and is almost as eﬃcient as a diﬀerence scheme on rectangular domains.

1 Introduction B-splines play an important role in approximation, numerical analysis, automated manufacturing, and computer graphics. Their use as finite elements suggests itself. However, in view of missing geometric flexibility and stability problems, this did not seem feasible. With the web-method, introduced in 2001 and patented in 2003, both diﬃculties have been overcome. Boundary conditions are incorporated via weight functions and stability is achieved by forming appropriate B-spline combinations. The resulting new type of a finite element basis, consisting of weighted extended B-splines (web-splines), combines the advantages of uniform splines and finite elements on unstructured grids: • • • • ∗

uniform grid with one basis function per grid point arbitrary smoothness and polynomial degree high accuracy with relatively few parameters exact fulfilment of essential boundary conditions The simulations were performed on the national super computer NEC SX-8 at the High Performance Computing Center Stuttgart (HLRS) under the grant number PFEMLWB/12799.

668

K. H¨ ollig, J. H¨ orner, M. Pfeil

• hierarchical bases for adaptive refinements • natural parallelism The potential of the new method, which bridges the gap between geometric modeling and numerical simulation, becomes apparent from a variety of tests for model problems in elasticity, heat conduction, and fluid flow (cf. www.webspline.de/examples/). In this project we introduce and implement a minimal degree variant of the web-method for three-dimensional boundary value problems. It is intended primarily for simulations where computational speed is of key importance and only moderate accuracy is required (e.g. because of missing precision of the physical model, measurement errors, etc.). Poisson’s equation as a basic model problem already exhibits the essential features of the new method, in particular, the treatment of domains with complicated boundaries via a special integration technique. The generalization to more general elliptic equations or systems is straightforward. In this report we first describe in Sect. 2 the weighted linear finite element basis. Section 3 is then devoted to a preprocessing technique which substantially accelerates the assembly of the Ritz-Galerkin system. A brief outline of the program structure is given in Sect. 4. Finally, we illustrate in Sect. 5 the performance of our algorithms.

2 Finite Element Basis We choose scaled translates bk = b(·/h − k),

(k1 , k2 , k3 ) ∈ K = {0, . . . , 1/h}3 , 1/h ∈ IN ,

of a linear box-spline b [1] to construct a minimal degree web-basis (cf. Fig. 1). While the underlying partition, which consists of 6 tetrahedra S per grid cell Q = ℓh + [0, 1]3 h, is more complicated than for trilinear tensor product B-splines (the standard linear web-basis function), this slight disadvantage

Fig. 1. Support and tetrahedral partition of a linear box-spline bk

Parallel Finite Element Methods with Weighted Linear B-Splines

669

is outweighed by the smaller support and the smaller number of monomial coeﬃcients. A spline on the standard domain Q∗ = (0, 1)3 is a linear combination of scaled box-spline translates: pk b k . ph = k∈K

Since bk is equal to 1 at the center kh of its support and vanishes at all other grid points k ′ h, the coeﬃcients pk coincide with the grid values p(kh) of the spline. In particular, on each tetrahedron S of the partition, ph (x) can be computed via linear interpolation from its 4 values at the vertices of S. We represent the simulation domain D in implicit form: D : w(x) > 0 , where w is referred to as a weight function. Assuming for convenience that D is a subset of the standard cube Q∗ , an approximate representation is given by Dh : wh (x) > 0 with wk bk , wk = w(kh) . w ≈ wh = k∈K

This yields a piecewise linear approximation of the boundary of order O(h2 ). Functions u, which vanish on the boundary of D (homogeneous Dirichlet boundary conditions), are approximated by weighted splines

h h h h u =w p = pk b k , pk (w bk ) = wk bk k∈K

k∈K

k∈K

where we set pk = 0 if the support of bk lies outside of Dh . In contrast to a standard finite element basis, consisting of hat functions (for example the box-splines bk ), we use the weighted basis functions wh bk , thus conforming to boundary conditions despite the regular grid. Regardless of the shape of the domain, the weighted spline approximation is determined by the two arrays wk , pk ,

k ∈ K,

of dimension (1 + 1/h)3 . Moreover, on each tetrahedron S, uh is a product of two linear functions which interpolate the 4 × 2 values of wh and ph at the vertices of S. Figure 2 illustrates an analogous construction in two dimensions. The examples of several weighted bivariate linear B-splines show that the qualitative form of the new basis is quite similar to standard elements. The modifications appear to be relatively minor, yet the simplications are substantial. No grid generation is required, and the data structure is extremely simple.

670

K. H¨ ollig, J. H¨ orner, M. Pfeil

Fig. 2. Weighted linear B-spline basis for a two-dimensional domain

3 Preprocessing of Ritz-Galerkin Integrals The matrix entries of the Ritz-Galerkin system for Poisson’s equation, which serves as our basic model problem, are grad(wh bk ) grad(wh bk′ ), k, k ′ ∈ K . Dh

These integrals are computed by summing the contributions from each grid cell Q: ... = ... . (1) Dh

Q

Q∩Dh

The latter integrals are nonzero only if both, bk and bk′ , have some support in Q, i.e., if kh and k ′ h are vertices of Q. Hence, we can write . . . = I(ℓ, α, α′ , wh ), ℓ ∈ L = {1, . . . , 1/h}3 , α, α′ ∈ {0, 1}3, Q∩Dh

where Q = ℓh + [0, 1]3 h, and α = k − ℓ, α′ = k ′ − ℓ. Usually, the Ritz-Galerkin integrals are computed on run-time. However, processing the boundary cells is time-consuming, since one has to take the intersection pattern of the tetrahedral partition with the boundary into account. There is a simple remedy. As is illustrated in Fig. 3, for each cell Q, the combinatorial type of the intersection of the tetrahedra S ⊂ Q with the boundary of Dh is determined by the signature s of the values Wℓ = {wℓ+γ : γ ∈ {0, 1}3 } of the weight function at the vertices of Q. Hence, there are only 28 = 256 possibilities. The diﬀerent patterns partition the lattice points L into disjoint sets Ls . For ℓ ∈ Ls , the integral I is a rational function of the weight function

Parallel Finite Element Methods with Weighted Linear B-Splines

671

− − +

−

−

− +

Fig. 3. Intersection pattern for a boundary cell determined by the signature sign(wγ ), γ = (0, 0, 0), (1, 0, 0), . . . , (1, 1, 1), of the weight function

values: I(ℓ, α, α′ , wh ) = Rs,α,α′ (Wℓ ),

ℓ ∈ Ls .

Clearly, the functions R do not depend on the shape of the domain and can, therefore, be compiled as subroutines. This allows the complete vectorization of the computations for combinatorially equivalent cells, cutting the assembly time to a minimum. Since the functions Rs,α,α′ are determined in advance, the time required for their automatic generation is irrelevant. Hence, we can optimize the instruction sequences for their evaluation. To find the best (or an almost best) procedure automatically is an interesting algebraic problem, which we are currently pursuing together with M. Clausen [2] from the University of Bonn. So far, we are using an extended version of the K-form algorithm of J. Koch [8] which is based on successive quadratic substitutions. Figure 4 shows one of the 12384 = 256 × 64 expressions. We see that the evaluation can be considerably simplified. Further simplifications are possible, but not provided by the K-Form. Of course, given the number of expressions, such simplified instruction sets must be generated automatically.

` 2 4 2 3 4 2 2w0,0,0 w1,1,0 w0,0,0 w1,1,1 + 6w0,0,0 w1,0,0 w1,1,1 + ...

(107 similar terms)

(w0,0,0 − w1,0,0 )3 (w0,0,0 − w1,1,0 )2 (w0,0,0 − w1,0,1 )2 (w0,0,0 − w1,1,1 )2

9 4 substitutions of type w ˜ = a/(b − c) = 17 substitutions of type w ˜ =a·b → ; 4 substitutions of type w ˜ = a/b

´

simpliﬁed instruction set

20w ˜5 − 6w ˜6 − w ˜7 − 10w ˜8 − 5w ˜9 − 10w ˜10 − w ˜11 − 5w ˜12 + 10w ˜13 + 2w ˜14 + 2w ˜15 −2w ˜16 − 8w ˜17 + 4w ˜18 − 2w ˜19 + w ˜20 + 2w ˜21 + 4w ˜22 + 2w ˜23 + w ˜24 + 2w ˜25 Fig. 4. Example of a function Rs,α,α′ for Poisson’s equation and a simpliﬁed instruction set

672

K. H¨ ollig, J. H¨ orner, M. Pfeil

4 Program Description As for conventional finite element approximations, the numerical solution of an elliptic boundary value problem with weighted linear box-splines consists of two steps: the assembly of the Ritz-Galerkin system and its iterative solution. We describe each component of the algorithm in turn, considering only the more diﬃcult matrix assembly. The Ritz-Galerkin matrix has a constant (generalized) band-width. Accordingly, the entry (k, k ′ ) is denoted by G(k, Δk),

k ∈ K, Δk = k ′ − k ∈ {−1, 0, 1}3 .

To generate the array G, we first compute the cell integrals I. In general terms, this is accomplished by the following program segment. for all s for all ℓ ∈ Ls for all α, α′ ∈ {0, 1}3 I(ℓ, α, α′ , wh ) = Rs,α,α′ (Wℓ ) end end end Summation of the cell integrals according to (1) then yields the matrix entries, as described in the following program segment. G=0 for all α, α′ ∈ {0, 1}3 for all ℓ ∈ L G(ℓ + α, α′ − α) = G(ℓ + α, α′ − α) + I(ℓ, α, α′ , wh ) end end In addition, we set zero diagonal entries G(k, 0) equal to one, in order to keep a simple data structure regardless of the shape of the domain. The corresponding entries of the right-hand side are set to 0. This is in agreement with the convention that the box-spline coeﬃcients pk are zero for Dh ∩supp bk = ∅. The notation and index structure of the pseudo-code reflects the mathematical description, not the FORTRAN implementation. However, it is apparent that the main loops are easily vectorized. To solve the Ritz-Galerkin system, we use a dynamic version of the multigrid algorithm described in [6]. The two basic components, Richardson smoothing and grid transfer with box-spline subdivision, trivially vectorize over the regular grid. In fact, the data structure of web-splines is ideally suited for any type of iterative solver. The simplicity of the above program fragments is deceptive – the complexity of the problem must be hidden somewhere. It is contained in the subroutines for the functions Rs,α,α′ . The corresponding automatically generated

Parallel Finite Element Methods with Weighted Linear B-Splines

673

code (244,582 lines) constitutes with approximately 98% the major portion of the program.

5 Implementation and Performance As a first test, we implemented a solver for the 2-dimensional Poisson equation. The code is written FORTAN90 with the following key routines: • • •

cell data: evaluation of the functions Rs,α,α′ ; matrix: assembly of the Ritz-Galerkin matrix; mg smooth: basic multigrid iteration.

The first routine, comprising the major portion of the code, is automatically generated with the aid of a Matlab-program. It not only supplies instruction sequences for integration, but also optimizes the complexity with K-form substitutions. Richardson’s iteration is used as a smoother, which in our experiments proved to be superior to SSOR and checkerboard-SSOR. An advantage of our new meshless method is the simplicity of the boundary treatment. Therefore, and in order to compare with automatic grid generation, we use random domains with fairly irregular boundaries. For the example shown in Fig. 5, a system with more than 6 million unknowns was solved on one processor in less than 2.5 seconds. As is typical for our B-spline based method, the assembly time (which corresponds to grid generation and numerical integration for a standard finite element method) is cut down substantially due to the regular grid and the use of the predefined instruction sets (routine: cell data).

Total time: 2.418 s

Other Solve

System

Fig. 5. Solution and computing times for a 2-dimensional domain

674

K. H¨ ollig, J. H¨ orner, M. Pfeil

Table 1 gives more detailed information about the program performance. We see that for all three routines the average vector length is larger than 200, i.e. very close to the maximal length 256. As a consequence, the eﬃciency is almost optimal. For the matrix assembly (routine: matrix) 7.9 Gflops are reached. The performance of the Richardson iteration with 6.1 Gflops is also quite good, in particular, since most of the 1204 iterations operate on coarse grids. Using a count weighted with the grid size, the solver requires 9.684 iterations in fine grid units. This roughly corresponds to an error reduction by a factor 0.17 per multigrid cycle. The routines, not listed in the statistic, also vectorize well (the vector ratio never drops below 98%); yet they contribute only a small fraction (< 14%) to the total time. Table 1. Statistics (ftrace) of the main routines for the two-dimensional test case (V.RATIO = ratio of vector operations, V.LEN = average vector length) PROG UNIT cell data matrix mg smooth

FREQ. 9 9 1204

TIME 0.196 0.096 1.794

MFLOPS 23.0 7905.4 6173.1

V.RATIO 98.24 99.76 99.40

V.LEN 230.3 245.2 211.8

Total time: 13.489 s Solve Other

System

PROG UNIT matrix mg smooth cell data 240 cell data 085 .. . cell data 186

FREQUENCY 5 499 5 5

TIME 0.273 3.011 0.267 0.264

MFLOPS 7210.1 4640.9 313.1 310.2

V.RATIO 99.70 97.99 45.91 48.97

V.LEN 124.1 98.1 51.6 54.8

5

0.024

162.5

86.18

138.1

Fig. 6. Domain and statistics for a 3-dimensional test case

Parallel Finite Element Methods with Weighted Linear B-Splines

675

The full potential of the new algorithm becomes apparent when considering three-dimensional examples. In fact, the random domain, shown in Fig. 6, would be nontrivial to mesh, as is necessary for a standard finite element scheme. The performance is not quite as good as for the two-dimensional case. While we achieve > 7.2 and > 4.6 Gflops for the routines matrix and mg smooth, the routine cell data (here split according to the diﬀerent cell types into routines cell data k, k=001:254) does not vectorize well. The reason is that not all of the 254 boundary intersection patterns occur frequently enough. Of course, for smaller grid-width, the vectorization would be close to optimal again. It is conceivable that significant improvements of the performance are still possible. First, our code can almost certainly be tuned further to the specific architecture. Second, the K-form can be augmented by additional algebraic simplification tools. In fact, it is an intriguing mathematical problem to automatically determine the best evaluation sequence for the functions Rs,α,α′ .

6 Concluding Remarks The numerical tests have confirmed that our new method is well suited for handling complicated boundaries. Moreover, due to the regular grid and an extremely simple data structure, the vectorization has led to a considerable speed up of our serial implementation (factor 15 in comparison to a 2.4 GHz AMD-CPU). The eﬃciency is partially also due to the idea of using predefined instruction sets, which avoids spending time on calculating quadrature points for boundary cells. The generalization to variable coeﬃcients problems and other elliptic problems, such as the Navier-Lame system of linear elasticity and eigenvalue problems for water waves, is relatively straightforward. It would also be interesting to apply the B-spline approach to moving boundary problems and form optimization. In both cases, the weight function technique could prove to be an alternative to existing methods, which require remeshing or have some topological limitations. Acknowledgements We thank U. K¨ uster and the HLRS parallel computing group who, with their advice and excellent support, contributed to a considerable percentage of the Gflops mentioned above.

References 1. C. de Boor, K. H¨ ollig, and S. Riemenschneider: Box Splines, Springer-Verlag, New York, 1993.

676

K. H¨ ollig, J. H¨ orner, M. Pfeil

2. M. Clausen, Private communication, 2007. 3. K. H¨ ollig, Finite Element Methods with B-Splines, SIAM, 2003. 4. K. H¨ ollig, U. Reif, and J. Wipper: Weighted extended B-spline approximation of Dirichlet problems, SIAM J. Numer. Anal. 39 (2001), 442–462. 5. K. H¨ ollig, U. Reif, and J. Wipper: Verfahren zur Erh¨ ohung der Leistungsf¨ ahigkeit einer Computereinrichtung bei Finite-Elemente-Simulationen und eine solche Computereinrichtung, Deutsches Patent DE 100 23 377 C2 (2003). 6. K. H¨ ollig, U. Reif, and J. Wipper: Multigrid methods with web-splines, Numer. Math. 91 (2002), 237–256. 7. J. H¨ orner: MIND: Multiple integration over NURBS domains, 2007. 8. J. Koch: Subdivision-Algorithmen zur L¨ osung polynomialer Gleichungssysteme, Master’s Thesis, University of Stuttgart (1991) 9. Finite Element Approximation with WEB-Splines (Slide collection): http://www.web-spline.de/publications/slide collection.pdf

High Performance Computing in Science and Engineering ' 06: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2006

High performance computing in science and engineering '08 transactions of the High Performance Computing Center, Stuttgart (HLRS) 2008

High Performance Computing in Science and Engineering '10: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2010

High Performance Computing in Science and Engineering '11: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2011

High Performance Computing on Vector Systems: Proceedings of the High Performance Computing Center Stuttgart, March 2005

High Performance Computing on Vector Systems 2006: Proceedings of the High Performance Computing Center Stuttgart, March 2006

Tools for High Performance Computing: Proceedings of the 2nd International Workshop on Parallel Tools for High Performance Computing, July 2008, HLRS, Stuttgart

High Performance Computing in Science and Engineering ' 07: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2007

High Performance Computing in Science and Engineering ' 06: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2006

High performance computing in science and engineering '08 transactions of the High Performance Computing Center, Stuttgart (HLRS) 2008

High Performance Computing in Science and Engineering '10: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2010

High Performance Computing in Science and Engineering '11: Transactions of the High Performance Computing Center, Stuttgart (HLRS) 2011

High Performance Computing on Vector Systems: Proceedings of the High Performance Computing Center Stuttgart, March 2005