Tensor Spaces and Numerical Tensor Calculus, 2nd edition

Springer Series in Computational Mathematics 56

Wolfgang Hackbusch

Tensor Spaces and Numerical Tensor Calculus Second Edition

Springer Series in Computational Mathematics Volume 56

Series Editors Randolph E. Bank, Department of Mathematics, University of California, San Diego, La Jolla, CA, USA Ronald L. Graham, Department of Computer Science & Engineering, University of California, San Diego, La Jolla, CA, USA Wolfgang Hackbusch, Max-Planck-Institut für Mathematik in den Naturwissenschaften, Leipzig, Germany Josef Stoer, Institut für Mathematik, University of Würzburg, Würzburg, Germany Richard S. Varga, Kent State University, Kent, OH, USA Harry Yserentant, Institut für Mathematik, Technische Universität Berlin, Berlin, Germany

This is basically a numerical analysis series in which high-level monographs are published. We develop this series aiming at having more publications in it which are closer to applications. There are several volumes in the series which are linked to some mathematical software. This is a list of all titles published in this series.

More information about this series at http://www.springer.com/series/797

Wolfgang Hackbusch

Tensor Spaces and Numerical Tensor Calculus Second Edition

123

Wolfgang Hackbusch Max Planck Institute for Mathematics in the Sciences Leipzig, Germany

ISSN 2198-3712 (electronic) ISSN 0179-3632 Springer Series in Computational Mathematics ISBN 978-3-030-35553-1 ISBN 978-3-030-35554-8 (eBook) https://doi.org/10.1007/978-3-030-35554-8 © Springer Nature Switzerland AG 2012, 2019 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Dedicated to my grandchildren Alina and Daniel

Preface

Large-scale problems have always been a challenge for numerical computations. An example is the treatment of fully populated n × n matrices when n2 is close to or beyond the computer’s memory capacity. Here the technique of hierarchical matrices can reduce the storage and the cost of numerical operations from O(n2 ) to almost O(n). Tensors of order (or spatial dimension) d can be understood as d-dimensional generalisations of matrices, i.e., arrays with d discrete or continuous arguments. For large d ≥ 3, the data size nd is far beyond any computer capacity. This book concerns the development of compression techniques for such high-dimensional data via suitable data sparse representations. Just as the hierarchical matrix technique was based on a successful application of the low-rank strategy, in recent years, related approaches have been used to solve high-dimensional tensor-based problems numerically. The results are quite encouraging, at least for data arising from suitably behaved problems, and even some problems of the size nd = 10001000 have become computable. The methods which can be applied to these multilinear problems are black boxlike. In this respect they are similar to methods used in linear algebra. On the other hand, most of the methods are approximate (computing suitably accurate approximations to quantities of interest) and in this respect they are similar to some approaches in analysis. The crucial key step is to construct an efficient new tensor representation, thus overcoming the drawbacks of the traditional tensor formats. In 2009 rapid progress could be achieved by introducing the hierarchical format, as well as the TT format, for tensor representation. Under suitable conditions, these formats allow a stable representation and a reduction of the data size from nd to O(dn). Another recent advancement is the so-called tensorisation technique, which may replace the size n with O(log n). Altogether, there is hope that problems of the size nd can be reduced to size O(d log(n)) = O(log(nd )); i.e., the problems are reduced to logarithmic size.

vii

viii

Preface

It turned out that some of the raw material for the methods described in this book was already known in the literature belonging to other (applied) fields outside of mathematics, such as chemistry. However, the particular language used to describe this material, combined with the fact that the algorithms (although potentially of general interest) were given names relating them only to a particular application, prevented the dissemination of the methods to a wider audience. One of the aims of this monograph is to introduce a more mathematically based treatment of this topic. Through this more abstract approach, the methods can be better understood, independently of the physical or technical details of the application. Accordingly, the book tries to introduce a nomenclature adapted to the mathematical background. The material in this monograph was used as the basis for a course of lectures at the University of Leipzig in the summer semester of 2010. The author’s research at the Max-Planck Institute of Mathematics in the Sciences has been supported by a larger group of researchers. In particular we would like to mention: B. Khoromskij, M. Espig, L. Grasedyck, and H.J. Flad. The help of H.J. Flad was indispensable for bridging the terminological gap between quantum chemistry and mathematics. The research programme has also benefited from the collaboration between the group in Leipzig and a group in Moscow headed by E. Tyrtyshnikov (Russian Academy of Sciences). Further inspiring cooperations have involved R. Schneider (TU Berlin, formerly University of Kiel) and A. Falc´o (CEU Cardenal Herrea University, Valencia). Later important contacts included D. Kressner (EPFL, Lausanne), H. Matthies (TU Braunschweig), L. De Lathauwer (University of Leuven), and A. Uschmajew and M. Michałek (both MPI, Leipzig). The author thanks many more colleagues for stimulating discussions. The first edition of this book appeared in 2012. The present revised edition not only contains corrections of the unavoidable misprints, but also includes new parts ranging from single additional statements to new subchapters. The additional chapters §4.5.4 and §10.1.5 allow L∞ estimates for truncations with respect to the Hilbert norm L2 . New statements about symmetric and antisymmetric tensors are added in §3.5.4, §6.9, and §7.7. The analysis of nonclosed formats has led to new results in §9.5 and §12.5. The cyclic matrix product representation and its siteindependent version are discussed in §12.5. The discussion of the ALS method is continued in §§9.6.2.3–9.6.2.7. The analysis of minimal subspaces of topological tensors is refined in §6.6. Other chapters are newly arranged: §4.2, §4.3, §7.4, §8.2, and §8.4. The number of references has increased by 50%. Finally, the author wishes to express his gratitude to the publisher, Springer, for their friendly cooperation. Leipzig, September 2019

Wolfgang Hackbusch

Contents

Part I Algebraic Tensors 1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 What are Tensors? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Tensor Product of Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Tensor Product of Matrices, Kronecker Product . . . . . . . . . . . 1.1.3 Tensor Product of Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Where do Tensors Appear? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Tensors as Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Tensor Decomposition for Inverse Problems . . . . . . . . . . . . . . 1.2.3 Tensor Spaces in Functional Analysis . . . . . . . . . . . . . . . . . . . 1.2.4 Large-Sized Tensors in Analysis Applications . . . . . . . . . . . . 1.2.5 Tensors in Quantum Chemistry . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Tensor Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Preview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Part I: Algebraic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Part II: Functional Analysis of Tensors . . . . . . . . . . . . . . . . . . 1.4.3 Part III: Numerical Treatment . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.4 Topics Outside the Scope of the Monograph . . . . . . . . . . . . . . 1.5 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Comments about the Early History of Tensors . . . . . . . . . . . . . . . . . . . 1.7 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 3 3 5 7 8 8 9 10 10 13 13 14 14 15 16 17 18 18 19

2

Matrix Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Matrix Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Matrix Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Semidefinite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Matrix Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Cholesky Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 QR Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23 23 25 27 29 30 30 31

ix

Contents

x

3

2.5.3 Singular-Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Low-Rank Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Linear Algebra Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Dominant Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33 39 41 44

Algebraic Foundations of Tensor Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Basic Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Free Vector Space over a Set . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Quotient Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.4 (Multi-)Linear Maps, Algebraic Dual, Basis Transformation 3.2 Tensor Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Constructive Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Characteristic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Isomorphism to Matrices for d = 2 . . . . . . . . . . . . . . . . . . . . . 3.2.4 Tensors of Order d ≥ 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 Different Types of Isomorphisms . . . . . . . . . . . . . . . . . . . . . . . 3.2.6 Rr and Tensor Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Linear and Multilinear Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Definition on the Set of Elementary Tensors . . . . . . . . . . . . . . 3.3.2 Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Tensor Spaces with Algebra Structure . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Symmetric and Antisymmetric Tensor Spaces . . . . . . . . . . . . . . . . . . . 3.5.1 Basic Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Quantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.4 Application of Functionals . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49 49 49 50 52 53 54 54 56 58 60 63 65 76 76 77 85 88 88 91 92 93

Part II Functional Analysis of Tensor Spaces 4

Banach Tensor Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1 Banach Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1.1 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.1.2 Basic Facts about Banach Spaces . . . . . . . . . . . . . . . . . . . . . . . 98 4.1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.1.4 Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.1.5 Dual Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.1.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.1.7 Weak Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.1.8 Continuous Multilinear Mappings . . . . . . . . . . . . . . . . . . . . . . 108 4.2 Topological Tensor Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.2.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.2.2 Continuity of the Tensor Product, Crossnorms . . . . . . . . . . . . 110 4.2.3 Projective Norm k·k∧(V,W ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 4.2.4 Duals and Injective Norm k·k∨(V,W ) . . . . . . . . . . . . . . . . . . . . 120

Contents

4.3

4.4

4.5

4.6

4.7

5

xi

4.2.5 Embedding of V ∗ into L(V ⊗ W, W ) . . . . . . . . . . . . . . . . . . 127 4.2.6 Reasonable Crossnorms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 4.2.7 Reflexivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 4.2.8 Uniform Crossnorms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4.2.9 Nuclear and Compact Operators . . . . . . . . . . . . . . . . . . . . . . . . 136 Tensor Spaces of Order d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.1 Continuity, Crossnorms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4.3.2 Recursive Definition of the Topological Tensor Space . . . . . . 140 4.3.3 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.3.4 Embedding into L(V, Vj ) and L(V, Vα ) . . . . . . . . . . . . . . . 147 4.3.5 Intersections of Banach Tensor Spaces . . . . . . . . . . . . . . . . . . 151 4.3.6 Tensor Space of Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.4.1 Scalar Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 4.4.2 Basic Facts about Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . 155 4.4.3 Operators on Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 4.4.4 Orthogonal Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Tensor Products of Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 4.5.1 Induced Scalar Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 4.5.2 Crossnorms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 4.5.3 Tensor Products of L(Vj , Vj ) . . . . . . . . . . . . . . . . . . . . . . . . . . 164 4.5.4 Gagliardo–Nirenberg Inequality . . . . . . . . . . . . . . . . . . . . . . . . 165 4.5.5 Partial Scalar Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 Tensor Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 4.6.1 Vector Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 4.6.2 Matrix-Vector Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.6.3 Matrix-Matrix Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 4.6.4 Hadamard Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 4.6.5 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 4.6.6 Function of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 Symmetric and Antisymmetric Tensor Spaces . . . . . . . . . . . . . . . . . . . 179 4.7.1 Hilbert Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 4.7.2 Banach Spaces and Dual Spaces . . . . . . . . . . . . . . . . . . . . . . . . 180

General Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 5.1 Vectorisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 5.1.1 Tensors as Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 5.1.2 Kronecker Tensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 5.2 Matricisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 5.2.1 General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 5.2.2 Finite-Dimensional Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 5.2.3 Hilbert Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 5.2.4 Matricisation of a Family of Tensors . . . . . . . . . . . . . . . . . . . . 198 5.3 Tensorisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Contents

xii

6

Minimal Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 6.1 Statement of the Problem, Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 6.2 Tensors of Order Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 6.2.1 Existence of Minimal Subspaces . . . . . . . . . . . . . . . . . . . . . . . 202 6.2.2 Use of the Singular-Value Decomposition . . . . . . . . . . . . . . . . 205 6.2.3 Minimal Subspaces for a Family of Tensors . . . . . . . . . . . . . . 206 6.3 Minimal Subspaces of Tensors of Higher Order . . . . . . . . . . . . . . . . . 207 6.4 Hierarchies of Minimal Subspaces and rankα . . . . . . . . . . . . . . . . . . . 210 6.5 Sequences of Minimal Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 6.6 Minimal Subspaces of Topological Tensors . . . . . . . . . . . . . . . . . . . . . 218 6.6.1 Setting of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 6.6.2 First Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 6.6.3 Second Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 6.7 Minimal Subspaces for Intersection Spaces . . . . . . . . . . . . . . . . . . . . . 224 6.7.1 Algebraic Tensor Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 6.7.2 Topological Tensor Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 6.8 Linear Constraints and Regularity Properties . . . . . . . . . . . . . . . . . . . . 226 6.9 Minimal Subspaces for (Anti-)Symmetric Tensors . . . . . . . . . . . . . . . 229

Part III Numerical Treatment 7

r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 7.1 Representations in General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.1.1 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.1.2 Computational and Memory Cost . . . . . . . . . . . . . . . . . . . . . . . 235 7.1.3 Tensor Representation versus Tensor Decomposition . . . . . . 236 7.2 Full and Sparse Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.3 r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 7.4 Tangent Space and Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 7.4.1 Tangent Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 7.4.2 Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.5 Representation of Vj . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 7.6 Conversions between Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 7.6.1 From Full Representation into r-Term Format . . . . . . . . . . . . 247 7.6.2 From r-Term Format into Full Representation . . . . . . . . . . . . 248 7.6.3 From r -Term into N -Term Format for r > N . . . . . . . . . . . . . 248 7.6.4 Sparse-Grid Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 7.6.5 From Sparse Format into r-Term Format . . . . . . . . . . . . . . . . . 251 7.7 Representation of (Anti-)Symmetric Tensors . . . . . . . . . . . . . . . . . . . . 253 7.7.1 Sums of Symmetric Rank-1 Tensors . . . . . . . . . . . . . . . . . . . . 254 7.7.2 Indirect Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 7.8 Modifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

Contents

xiii

8

Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 8.1 The Set Tr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 8.2 Tensor Subspace Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 8.2.1 General Frame or Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 8.2.2 Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 8.2.3 Tensors in KI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 8.2.4 Orthonormal Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 8.2.5 Summary of the Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 8.2.6 Hybrid Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 8.3 Higher-Order Singular-Value Decomposition (HOSVD) . . . . . . . . . . 273 8.3.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 8.3.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 8.3.3 Computation and Computational Cost . . . . . . . . . . . . . . . . . . . 277 8.4 Tangent Space and Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 8.4.1 Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 8.4.2 Tangent Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 8.4.3 Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 8.5 Conversions between Different Formats . . . . . . . . . . . . . . . . . . . . . . . . 287 8.5.1 Conversion from Full Representation into Tensor Subspace Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 8.5.2 Conversion from Rr to Tr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 8.5.3 Conversion from Tr to Rr . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 8.5.4 A Comparison of Both Representations . . . . . . . . . . . . . . . . . . 292 8.5.5 r-Term Format for Large r > N . . . . . . . . . . . . . . . . . . . . . . . 293 8.6 Joining two Tensor Subspace Representation Systems . . . . . . . . . . . . 293 8.6.1 Trivial Joining of Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 8.6.2 Common Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

9

r-Term Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 9.1 Approximation of a Tensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 9.2 Discussion for r = 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 9.3 Discussion in the Matrix Case d = 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 301 9.4 Discussion in the Tensor Case d ≥ 3 . . . . . . . . . . . . . . . . . . . . . . . . . . 303 9.4.1 Nonclosedness of Rr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 9.4.2 Border Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 9.4.3 Stable and Unstable Sequences . . . . . . . . . . . . . . . . . . . . . . . . . 306 9.4.4 A Greedy Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 9.5 General Statements on Nonclosed Formats . . . . . . . . . . . . . . . . . . . . . 309 9.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 9.5.2 Nonclosed Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 9.5.3 Discussion of F = Rr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 9.5.4 General Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 9.5.5 On the Strength of Divergence . . . . . . . . . . . . . . . . . . . . . . . . . 313 9.5.6 Uniform Strength of Divergence . . . . . . . . . . . . . . . . . . . . . . . . 314 9.5.7 Extension to Vector Spaces of Larger Dimension . . . . . . . . . . 317

Contents

xiv

Numerical Approaches for the r-Term Approximation . . . . . . . . . . . . 318 9.6.1 Use of the Hybrid Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 9.6.2 Alternating Least-Squares Method . . . . . . . . . . . . . . . . . . . . . . 320 9.6.3 Stabilised Approximation Problem . . . . . . . . . . . . . . . . . . . . . . 329 9.6.4 Newton’s Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 9.7 Generalisations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 9.8 Analytical Approaches for the r-Term Approximation . . . . . . . . . . . . 333 9.8.1 Quadrature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 9.8.2 Approximation by Exponential Sums . . . . . . . . . . . . . . . . . . . 335 9.8.3 Sparse Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

9.6

10

Tensor Subspace Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.1 Truncation to Tr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.1.1 HOSVD Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 10.1.2 Successive HOSVD Projection . . . . . . . . . . . . . . . . . . . . . . . . . 350 10.1.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352 10.1.4 Other Truncations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 10.1.5 L∞ Estimate of the Truncation Error . . . . . . . . . . . . . . . . . . . . 355 10.2 Best Approximation in the Tensor Subspace Format . . . . . . . . . . . . . . 358 10.2.1 General Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.2.2 Approximation with Fixed Format . . . . . . . . . . . . . . . . . . . . . . 359 10.2.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 10.3 Alternating Least-Squares Method (ALS) . . . . . . . . . . . . . . . . . . . . . . 362 10.3.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 10.3.2 ALS for Different Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 10.3.3 Approximation with Fixed Accuracy . . . . . . . . . . . . . . . . . . . . 367 10.4 Analytical Approaches for the Tensor Subspace Approximation . . . . 369 10.4.1 Linear Interpolation Techniques . . . . . . . . . . . . . . . . . . . . . . . . 369 10.4.2 Polynomial Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 10.4.3 Polynomial Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 10.4.4 Sinc Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 10.5 Simultaneous Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 10.6 R´esum´e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385

11 Hierarchical Tensor Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 11.1.1 Hierarchical Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 11.1.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 11.1.3 Historical Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 11.2 Basic Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.2.1 Dimension Partition Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.2.2 Algebraic Characterisation, Hierarchical Subspace Family . . 394 11.2.3 Minimal Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 11.2.4 Conversions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.3 Construction of Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400

Contents

xv

11.3.1 Hierarchical Basis Representation . . . . . . . . . . . . . . . . . . . . . . 400 11.3.2 Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 11.3.3 HOSVD Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 11.3.4 Tangent Space and Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . 422 11.3.5 Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 11.3.6 Conversion from Rr to Hr Revisited . . . . . . . . . . . . . . . . . . . 429 11.4 Approximations in Hr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 11.4.1 Best Approximation in Hr . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 11.4.2 HOSVD Truncation to Hr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 11.5 Joining two Hierarchical Tensor Representation Systems . . . . . . . . . 446 11.5.1 Setting of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 11.5.2 Trivial Joining of Frames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 11.5.3 Common Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 12

Matrix Product Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 12.1 Basic TT Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453 12.2 Function Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 12.3 TT Format as Hierarchical Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 12.3.1 Related Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 12.3.2 From Subspaces to TT Coefficients . . . . . . . . . . . . . . . . . . . . . 457 12.3.3 From Hierarchical Format to TT Format . . . . . . . . . . . . . . . . . 458 12.3.4 Construction with Minimal ρj . . . . . . . . . . . . . . . . . . . . . . . . . 460 12.3.5 Extended TT Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 460 12.3.6 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 12.3.7 HOSVD Bases and Truncation . . . . . . . . . . . . . . . . . . . . . . . . . 462 12.4 Conversions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 12.4.1 Conversion from Rr to Tρ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463 12.4.2 Conversion from Tρ to Hr with a General Tree . . . . . . . . . . 463 12.4.3 Conversion from Hr to Tρ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 12.5 Cyclic Matrix Products and Tensor Network States . . . . . . . . . . . . . . 467 12.5.1 Cyclic Matrix Product Representation . . . . . . . . . . . . . . . . . . . 467 12.5.2 Site-Independent Representation . . . . . . . . . . . . . . . . . . . . . . . 470 12.5.3 Tensor Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471 12.6 Representation of Symmetric and Antisymmetric Tensors . . . . . . . . . 472

13 Tensor Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473 13.1 Addition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 13.1.1 Full Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 13.1.2 r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 13.1.3 Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . 475 13.1.4 Hierarchical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 477 13.2 Entry-wise Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477 13.2.1 r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478 13.2.2 Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . 478 13.2.3 Hierarchical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 479

xvi

Contents

13.2.4 Matrix Product Representation . . . . . . . . . . . . . . . . . . . . . . . . . 480 13.3 Scalar Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 13.3.1 Full Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 13.3.2 r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481 13.3.3 Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . 482 13.3.4 Hybrid Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 13.3.5 Hierarchical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 13.3.6 Orthonormalisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489 13.4 Change of Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 13.4.1 Full Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490 13.4.2 Hybrid r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . 490 13.4.3 Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . 491 13.4.4 Hierarchical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 491 13.5 General Binary Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 13.5.1 r-Term Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 13.5.2 Tensor Subspace Representation . . . . . . . . . . . . . . . . . . . . . . . . 493 13.5.3 Hierarchical Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 13.6 Hadamard Product of Tensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495 13.7 Convolution of Tensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 13.8 Matrix-Matrix Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 13.9 Matrix-Vector Multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497 13.9.1 Identical Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 13.9.2 Separable Form (13.25a) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498 13.9.3 Elementary Kronecker Tensor (13.25b) . . . . . . . . . . . . . . . . . . 499 13.9.4 Matrix in p-Term Format (13.25c) . . . . . . . . . . . . . . . . . . . . . . 500 13.10Functions of Tensors, Fixed-Point Iterations . . . . . . . . . . . . . . . . . . . . 501 13.11Example: Operations in Quantum Chemistry Applications . . . . . . . . 503 14

Tensorisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 14.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 14.1.1 Notations, Choice for TD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507 14.1.2 Format Hρtens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 14.1.3 Operations with Tensorised Vectors . . . . . . . . . . . . . . . . . . . . . 510 14.1.4 Application to Representations by Other Formats . . . . . . . . . 512 14.1.5 Matricisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513 14.1.6 Generalisation to Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514 14.2 Approximation of Grid Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515 14.2.1 Grid Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515 14.2.2 Exponential Sums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516 14.2.3 Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516 14.2.4 Multiscale Feature and Conclusion . . . . . . . . . . . . . . . . . . . . . 520 14.2.5 Local Grid Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520 14.3 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 14.3.1 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 14.3.2 Separable Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523

Contents

xvii

14.3.3 Tensor Algebra A(`0 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524 14.3.4 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 14.4 Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534 14.4.1 FFT for Cn Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534 14.4.2 FFT for Tensorised Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535 14.5 Tensorisation of Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 14.5.1 Isomorphism ΦF n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 14.5.2 Scalar Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538 14.5.3 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 14.5.4 Continuous Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539 15

Multivariate Cross Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 15.1 Approximation of General Tensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 541 15.1.1 Approximation of Multivariate Functions . . . . . . . . . . . . . . . . 542 15.1.2 Multiparametric Boundary-Value Problem and PDE with Stochastic Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543 15.1.3 Function of a Tensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545 15.2 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 546 15.3 Properties in the Matrix Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548 15.4 Case d ≥ 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551 15.4.1 Matricisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551 15.4.2 Nestedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553 15.4.3 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555

16 Applications to Elliptic Partial Differential Equations . . . . . . . . . . . . . . 559 16.1 General Discretisation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559 16.2 Solution of Elliptic Boundary-Value Problems . . . . . . . . . . . . . . . . . . 560 16.2.1 Separable Differential Operator . . . . . . . . . . . . . . . . . . . . . . . . 561 16.2.2 Discretisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561 16.2.3 Solution of the Linear System . . . . . . . . . . . . . . . . . . . . . . . . . . 563 16.2.4 Accuracy Controlled Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 565 16.3 Solution of Elliptic Eigenvalue Problems . . . . . . . . . . . . . . . . . . . . . . . 566 16.3.1 Regularity of Eigensolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . 566 16.3.2 Iterative Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 568 16.3.3 Alternative Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 16.4 On Other Types of PDEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 569 17

Miscellaneous Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 17.1 Minimisation Problems on V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 17.1.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 17.1.2 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 572 17.2 Solution of Optimisation Problems Involving Tensor Formats . . . . . . 573 17.2.1 Formulation of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 17.2.2 Reformulation, Derivatives, and Iterative Treatment . . . . . . . 575 17.3 Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576

xviii

Contents

17.3.1 Tangent Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 576 17.3.2 Dirac–Frenkel Discretisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 576 17.3.3 Tensor Subspace Format Tr . . . . . . . . . . . . . . . . . . . . . . . . . . . . 577 17.3.4 Hierarchical Format Hr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 579 17.4 ANOVA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581 17.4.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581 17.4.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582 17.4.3 Combination with Tensor Representations . . . . . . . . . . . . . . . 584 17.4.4 Symmetric Tensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599

About the Author

Wolfgang Hackbusch works in the field of numerical mathematics for partial differential equations and integral equations. He has published a number of monographs, for example on the multi-grid method, the numerical analysis of elliptic partial differential equations, the iterative solution of large systems of equation, and the technique of hierarchical matrices.

xix

List of Symbols and Abbreviations

Symbols [a b . . .] [A B . . .] d·e b·c h·, ·i h·, ·iα h·, ·iH h·, ·iHS h·, ·ij h·, ·iF # * •|τ ×σ •⊥ •H •T •−T , •−H • ×

×dj=1 ×j ?

aggregation of vectors a, b ∈ KI , . . . into a matrix of size I × J aggregation of matrices A ∈ KI×J1 , B ∈ KI×J2 , . . . into a matrix of size I × (J1 ∪ J2 ∪ . . .) smallest integer ≥ · largest integer ≤ · scalar product; in KI usually the Euclidean scalar product; cf. §2.1, §4.4.1 partial scalar product; cf. §4.5.5 scalar product of a (pre-)Hilbert space H Hilbert-Schmidt scalar product; cf. Definition 4.140 Nd scalar product of the (pre-)Hilbert space Vj from V = j=1 Vj Frobenius scalar product of matrices; cf. (2.10) cardinality of a set weak convergence; cf. §4.1.7 restriction of a matrix • to the matrix block τ × σ; cf. §1.7 orthogonal complement, cf. §4.4.1 Hermitean transpose of a matrix or vector transpose of a matrix or vector inverse matrix of •T or •H , respectively either complex-conjugate value of a scalar or closure of a set Cartesian product of sets: A × B := {(a, b) : a ∈ A, b ∈ B} d-fold Cartesian product of sets j-mode product, cf. Footnote 8 on page 5; not used here convolution; cf. §4.6.5

xxi

xxii

List of Symbols and Abbreviations

∧ exterior product; cf. §3.5.1 Hadamard product; cf. (4.82) ⊕ direct sum; cf. footnote on page 13 ⊗d d-fold tensor product; cf. Notation 3.24 Nd (j) v ⊗ w, tensor product of two or more vectors; cf. §3.2.1 j=1 v Nd V ⊗ W, j=1 Vj tensor space generated by two or more vector spaces; cf. §3.2.1 Nd V ⊗a W, a j=1 Vj algebraic tensor space; cf. (3.9) and §3.2.4 Nd V ⊗k·k W, k·k j=1 Vj topological tensor space; cf. (3.10); §4.2 N cf. (3.17b) j6=k ⊂ the subset relation A ⊂ B includes the case A = B ∪˙ disjoint union ∼ equivalence relation; cf. §3.1.3, §4.1.1 ∼ •=• isomorphic spaces; cf. §3.2.5 •≤• semi-ordering of matrices; cf. (2.12) k·k norm; cf. §4.1.1 ∗ k·k dual norm; cf. Lemma 4.21 k·k2 Euclidean norm of vector or tensor (cf. §2.3 and Example 4.149) or spectral norm of a matrix (cf. (2.11)) k·kF Frobenius norm of matrices; cf. (2.9) k·kHS Hilbert-Schmidt norm; cf. Definition 4.140 k·kSVD,p Schatten norm; cf. (4.17) k·kX norm of a space X k·kX←Y associated matrix norm (cf. §2.3) or operator norm (cf. (4.6a)) k·k1 . k·k2 semi-ordering of norms; cf. §4.1.1 k·k∧(V,W ) , k·k∧ projective norm; cf. §4.2.3.1 k·k∨(V,W ) , k·k∨

injective norm; cf. §4.2.4.2

List of Symbols and Abbreviations

xxiii

Greek Letters α αc α1 , α2 δij ρ ρ(·) ρxyz (·) ρframe ρHOSVD ρHT ρHOSVD HT ρorth HT ρtens HT

often a subset of the set D of directions (cf. (5.3a,b)) or vertex of the tree TD ; cf. Definition 11.2 complement D\α; cf. (5.3c) often sons of a vertex α ∈ TD ; cf. §11.2.1 Kronecker delta; cf. (2.1) tuple of TT ranks; cf. Definition 12.1 spectral radius of a matrix; cf. §4.6.6 tensor representation by format ‘xyz’; cf. §7.1 general tensor subspace format; cf. (8.8c) HOSVD tensor subspace format; cf. (8.25) hierarchical format; cf. (11.24) hierarchical HOSVD format; cf. Definition 11.37 orthonormal hierarchical format; cf. (11.32) TT format for tensorised vectors; cf. (14.5a)

hybr ρhybr , ρhybr hybrid formats; cf. §8.2.6 orth , ρr-term ρj TT rank; cf. (12.1a) and Definition 12.1 ρorth orthonormal tensor subspace format; cf. (8.10b) ρr-term r-term format; cf. (7.7a) ρsparse sparse format; cf. (7.5) ρTS general tensor subspace format; cf. (8.5c) ρTT TT format; cf. (12.7) σ(·) spectrum of a matrix; cf. §4.6.6 σi singular value of the singular-value decomposition; cf. (2.16a), (4.65) Σ diagonal matrix of the singular-value decomposition; cf. (2.16a) ϕ, ψ often linear mapping or functional; cf. §3.1.4 Φ, Ψ often linear mapping or operator; cf. §4.1.4) Φ0 dual of Φ; cf. Definition 4.23 ∗ Φ adjoint of Φ; cf. Definition 4.136

xxiv~

List of Symbols and Abbreviations

Latin Letters a coefficient tensor, cf. Remark 3.31 A, B, . . . , A1 , A2 , . . . often used for linear mapping (from one vector space into another one). This includes matrices. A, B, C, . . . tensor products of operators or matrices A(j) mapping from L(Vj , Wj ), j-th component in a Kronecker product A(V ) tensor algebra generated by V ; cf. (3.38) A(V ) antisymmetric tensor space generated by V ; cf. Definition 3.74 Arcosh area [inverse] hyperbolic cosine: cosh(Arcosh(x)) = x (j)

(α)

bi , bi

B, Bj , Bα c0 (I)

basis vectors; cf. §8.2.1.1, (11.18a)  (j) (j)  basis (or frame), Bj = b1 , . . . , br , cf. §8.2.1.1;  (α) (α)  in the case of tensor spaces: Bα = b1 , . . . , brα , cf. (11.18a) subset of `∞ (I); cf. (4.4)

(α,`)

cij coefficients of the matrix C (α,`) ; cf. (11.20) C field of complex numbers 0 C(D), C (D) bounded, continuous functions on D; cf. Example 4.10 C(d, (ρj ), (nj ))) cyclic matrix-product representation; cf. §12.5.1 Cind (d, ρ, n) site-independent cyclic matrix-product representation; cf. §12.5.2 Cα tuple (C (α,`) )1≤`≤rα of C (α,`) from below; cf. (11.22b) C (α,`)

(α)

coefficient matrix at vertex α characterising the basis vector b` ; cf. (11.20) Cj , Cα contractions; cf. Definition 4.160 CN (f, h), C(f, h) sinc interpolation; cf. Definition 10.34 d order of a tensor; cf. §1.1.1 D set {1, . . . , d} of directions; cf. (5.3b) D(A) domain of the operator A; cf. page 227 Dδ analyticity stripe; cf. (10.33) depth(·) depth of a tree; cf. (11.6) det(·) determinant of a matrix diag{. . .} diagonal matrix with entries . . . dim(·) dimension of a vector space e(i) i-th unit vector of KI (i ∈ I); cf. (2.2) EN (f, h), E(f, h) sinc interpolation error; cf. Definition 10.34 Er (·) exponential sum; cf. (9.36a) Eρ regularity ellipse; cf. §10.4.2.2 F(W, V ) space of finite rank operators; cf. §4.2.9 G(·) Gram matrix of a set of vectors; cf. (2.13), (11.30)

List of Symbols and Abbreviations

xxv

H, H1 , H2 , . . . (pre-)Hilbert spaces H(Dδ ) Banach space from Definition 10.36 1,p H (D) Sobolev space; cf. Example 4.47 HS(V, W ) Hilbert–Schmidt space; cf. Definition 4.140 id identity mapping i, j, k, . . . index variables i, j, k multi-indices from a product index set I etc. I identity matrix or index set I, I[a,b] interpolation operator; cf. §10.4.3 I, J, K, I1 , I2 , . . . , J1 , J2 , . . . often used for index sets I, J index sets defined by products I1 × I2 × . . . of index sets j often index variable for the directions from {1, . . . , d} K underlying field of a vector space; usually R or C K(W, V ) space of compact operators; cf. §4.2.9 `(I) vector space KI ; cf. Example 3.1 `0 (I) subset of `(I); cf. (3.2) `p (I) Banach space from Example 4.7; 1 ≤ p ≤ ∞ level level of a vertex of a tree, cf. (11.5) lim inf limit inferior, smallest accumulation point L often depth of a tree, cf. (11.6) L lower triangular matrix in Cholesky decomposition; cf. §2.5.1 L(T ) set of leaves of the tree T ; cf. Remark 11.3b L(V, W ) vector space of linear mappings from V into W ; cf. §3.1.4 L(X, Y ) space of continuous linear mappings from X into Y ; cf. §4.1.4 Lp (D) Banach space; cf. Example 4.9; 1 ≤ p ≤ ∞ Mα , Mj matricisation isomorphisms; cf. Definition 5.3 n, nj often dimension of a vector space V, Vj N set {1, 2, . . .} of natural numbers N0 set N ∪ {0} = {0, 1, 2, . . .} N (W, V ) space of nuclear operators; cf. §4.2.9 Nxyz arithmetical cost of ‘xyz’ xyz storage cost of ‘xyz’; cf. (7.8a) Nmem NLSVD cost of a left-sided singular-value decomposition; cf. Corollary 2.24 NQR cost of a QR decomposition; cf. Lemma 2.22 NSVD cost of a singular-value decomposition; cf. Corollary 42 o(·), O(·) Landau symbols; cf. (4.12) P permutation matrix (cf. (2.15)), set of permutations, or projection PA alternator, projection onto A(V ); cf. (3.40)

xxvi

List of Symbols and Abbreviations

PS symmetriser, projection onto S(V ); cf. (3.40) P, Pj , etc. often used for projections in tensor spaces (j),HOSVD

PjHOSVD , Prj , PHOSVD HOSVD projections; cf. Remark 10.1 r P, Pp , Pp spaces of polynomials; cf. §10.4.2.1 Q unitary matrix of QR decomposition; cf. (2.14a) r matrix rank or tensor rank (cf. §2.2), representation rank (cf. Definition 7.3), or bound of ranks r rank (rα )α∈TD connected with hierarchical format Hr ; cf. §11.2.2 rα components of r from above r rank (r1 , . . . , rd ) connected with tensor subspace representation in Tr rj components of r from above rmin (v) tensor subspace rank; cf. Remark 8.4 range(·) range of a matrix or operator; cf. §2.1 rank(·) rank of a matrix or tensor; cf. §2.2 and (3.20) rank(·) border rank; cf. (9.11) rankα (·), rankj (·) α-rank and j-rank; cf. Definition 5.7 rmax maximal rank; cf. (2.5) and §3.2.6.5 R upper triangular matrix of QR decomposition; cf. (2.14a) R field of real numbers J R set of J-tuples; cf. page 4 Rr set of matrices or tensors of rank ≤ r; cf. (2.6) and (3.18) S(α) set of sons of a tree vertex α; cf. Definition 11.2 S(V ) symmetric tensor space generated by V ; cf. Definition 3.74 S(k, h)(·) see (10.31) sinc(·) sinc function: sin(πx)/(πx) span{· · · } subspace spanned by · · · supp(·) support of a mapping; cf. §3.1.2 Tα subtree of TD ; cf. Definition 11.6 TD dimension partition tree; cf. Definition 11.2 (`)

TD set of tree vertices at level `; cf. (11.7) TT TD linear tree used for the TT format; cf. §12 Tr set of tensors of representation rank r; cf. Definition 8.1 Tρ set of tensors of TT representation rank ρ; cf. (12.4) trace(·) trace of a matrix or operator; cf. (2.8) and (4.66) tridiag{a, b, c} tridiagonal matrix (a : lower diagonal entries, b : diagonal, c : upper diagonal entries) U vector space, often a subspace U, V unitary matrices of the singular-value decomposition; cf. (2.16b)

List of Symbols and Abbreviations

xxvii

ui , vi left and right singular vectors of SVD; cf. (2.18) u, v, w vectors u, v, w tensors U tensor space, often a subspace of a tensor space Uα subspace of the tensor space Vα ; cf. (11.8) {Uα }α∈TD hierarchical subspace family; cf. Definition 11.8 U 0 , V 0 , W 0 , . . . algebraic duals of U, V, W, . . .; cf. §3.1.4 UjI (v), UjII (v), UjIII (v), UjIV (v) see Lemma 6.12 Ujmin (v), Umin minimal subspaces of a tensor v; Def. 6.3, (6.8a), and §6.4 α (v) vi either the i-th component of v or the i-th vector of a set of vectors v (j) vector of Vj corresponding to the j-th direction of the tensor; cf. §3.2.4 v[k] tensor belonging to V[k] ; cf. (3.17d) Vfree (S) free vector space of a set S; cf. §3.1.2 N Vα tensor space V ; cf. (5.3d) Nj∈α j V[j] tensor space k6=j Vj ; cf. (3.17a) and §5.2 Vcycl space of cyclic tensors; cf. Definition 12.12 V, W, . . . , X, Y, . . . vector spaces V 0 , W 0 , . . . , X 0 , Y 0 , . . . algebraically dual vector spaces; cf. 3.1.4 V, W, X, Y tensor spaces X, Y often used for Banach spaces; cf. §4.1 ∗ ∗ X , Y , . . . dual spaces containing the continuous functionals; cf. §4.1.5 V ∗∗ bidual space; cf. §4.1.5 Z set of integers

xxviii

List of Symbols and Abbreviations

Abbreviations and Algorithms ALS alternating least-squares method; cf. §9.6.2 ANOVA analysis of variance; cf. §17.4 CANDELINC hybrid format; cf. Footnote 11 on page 271 CPD canonical polyadic decomposition; cf. Abstract on page 233 DCQR cf. (2.37) DFT density functional theory; cf. §13.11 DFT discrete Fourier transform; cf. §14.4.1 DMRG density matrix renormalisation group; cf. §17.2.2 FFT fast Fourier transform; cf. §14.4.1 HOOI higher-order orthogonal iteration; cf. §10.3.1 HOSVD higher-order singular-value decomposition; cf. §8.3 HOSVD(·), HOSVD∗ (·), HOSVD∗∗ (·) procedures constructing the hierarchical HOSVD format; cf. (11.41a–c) HOSVD-lw, HOSVD∗ -lw levelwise procedures; cf. (11.41a–c), (11.42a,b) HOSVD-TrSeq sequential truncation procedure; cf. (11.58) HOSVDα (v), HOSVDj (v) computation of HOSVD data; cf. §8.3.3 JoinBases joining two bases; cf. (2.32) JoinONB joining two orthonormal bases; cf. (2.33) LOBPCG locally optimal block preconditioned conjugate gradient; cf. (16.13) LSVD left-sided reduced SVD; cf. (2.29) MALS modified alternating least-squares method; cf. §17.2.2 MBI maximum block improvement variant of ALS; cf. page 321 MLSVD same as HOSVD; cf. Footnote 12 on page 273 MPS matrix product state, matrix product system; cf. §12 PD polyadic decomposition; cf. Abstract on page 233 PEPS projected entangled pairs states; cf. footnote 6 on page 467 PGD proper generalised decomposition; cf. (17.1.1) PQR pivotised QR decomposition; cf. (2.27) QR QR decomposition; cf. §2.5.2 REDUCE, REDUCE∗ truncation procedure; cf. §11.4.2 RQR reduced QR decomposition; cf. (2.26) RSVD reduced SVD; cf. (2.28) SVD singular-value decomposition; cf. §2.5.3

Part I

Algebraic Tensors

In Chapter 1, we start with an elementary introduction into the world of tensors (the precise definitions are in Chapter 3) and explain where large-sized tensors appear. This is followed by a description of the Numerical Tensor Calculus. Section 1.4 contains a preview of the material of the three parts of the book. We conclude with some historical remarks and an explanation of the notation. The numerical tools which will be developed for tensors, make use of linear algebra methods (e.g., QR and singular value decomposition). Therefore, these matrix techniques are recalled in Chapter 2. The definition of the algebraic tensor space structure is given in Chapter 3. This includes linear mappings and their tensor product.

Chapter 1

Introduction In view of all that . . . , the many obstacles we appear to have surmounted, what casts the pall over our victory celebration? It is the curse of dimensionality, a malediction that has plagued the scientist from earliest days. (Bellman [25, p. 94]). 1.1 What are Tensors? For a first rough introduction1 into tensors, we give a preliminary definition of tensors and the tensor product. The formal definition in the sense of multilinear algebra will be given in Chapter 3. In fact, below we consider three types of tensors which are of particular interest in later applications.

1.1.1 Tensor Product of Vectors While vectors have entries vi with one index and matrices have entries Mij with two indices, tensors will carry d indices. The natural number2 d defines the order of the tensor. The indices j ∈ {1, . . . , d} correspond to the ‘j-th direction’, ‘j-th position’, ‘j-th dimension’, ‘j-th axis’, ‘j-th site’, or3 ‘j-th mode’. The names ‘direction’ and ‘dimension’ originate from functions f (x1 , . . . , xd ) (cf. §1.1.3) for which the variable xj corresponds to the j-th spatial direction. For each j ∈ {1, . . . , d} we fix a (finite) index set Ij , e.g., Ij = {1, . . . , nj }. The Cartesian product of these index sets yields I := I1 × . . . × Id . The elements of I are multi-indices or d-tuples i = (i1 , . . . , id ) with ij ∈ Ij . A tensor v is defined by its entries vi = v[i] = v[i1 , . . . , id ] ∈ R. 1 2 3

Introductory articles are [136] and [153]. The letter d is chosen because of its interpretation as spatial dimension. The usual meaning of the term ‘mode’ is ‘eigenfunction’.

© Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_1

3

4

1 Introduction

We may write v := (v[i])i∈I . Mathematically, we can express the set of these tensors by RI . Note that for any index set J, RJ is the vector space RJ = {v = (vi )i∈J : vi ∈ R} of dimension #J (the sign # denotes the cardinality of a set). Notation 1.1. Both notations, vi with subscript i and v[i] with square brackets are used in parallel. The notation with square brackets is preferred for multiple indices and in the case of secondary subscripts: v[i1 , . . . , id ] instead of vi1 ,...,id . There is an obvious entry-wise definition of the multiplication λv of a tensor by a real number and of the (commutative) addition v + w of two tensors. Therefore the set of tensors has the algebraic structure of a vector space (here over the field R). In scientific areas more remote from mathematics and algebra, a tensor v[i1 , . . . , id ] is considered as a data structure and called a ‘d-way array’. The relation between the vector spaces RIj and RI is given by the tensor product. For vectors v (j) ∈ RIj (1 ≤ j ≤ d) we define the tensor product4,5 v := v (1) ⊗ v (2) ⊗ . . . ⊗ v (d) =

d O j=1

via its entries (1)

(d)

(2)

vi = v[i1 , . . . , id ] = vi1 · vi2 · . . . · vid The tensor space is written as tensor product of the vector spaces RIj defined by the span6 d O

v (j) ∈ RI

Nd

j=1

for all i ∈ I.

(1.1)

RIj = RI1 ⊗ RI2 ⊗ . . . ⊗ RId

 RIj = span v (1) ⊗ v (2) ⊗ . . . ⊗ v (d) : v (j) ∈ RIj , 1 ≤ j ≤ d .

(1.2)

j=1 (1) (2) 7 (d) The generating products Nd v Ij ⊗ v ⊗ . . . ⊗ v are called elementary tensors. of the tensor spaceN is called a (general) tensor. It is Any element v ∈ j=1 R d important to notice that, in general, a tensor v ∈ j=1 RIj is not representable as elementary tensor but only as a linear combination of such products. Nd The definition (1.2) implies j=1 RIj ⊂ RI . Taking all linearNcombinations of d Ij elementary tensors defined by the unit vectors, we easily prove = RI . j=1 R 4

In some publications the term ‘outer product’ is used instead of ‘tensor product’. This contradicts another definition of the outer product or exterior product satisfying the antisymmetric property u ∧ v = − (v ∧ u) (see page 90). 5 The index j indicating the ‘direction’ is written as upper index in brackets, in order to let space for further indices placed below. 6 The span is the set of all finite linear combinations. 7 Also the terms ‘decomposable tensors’ and ‘pure tensors’ are used. Further names are ‘dyads’ for d = 2, ‘triads’ for d = 3, etc. (cf. [211, p. 3]). In physics, elementary tensors represent ‘factorisable’ or ’pure states’. A non-pure quantum state is called ‘entangled’.

1.1 What are Tensors?

5

Qd In particular, because #I = j=1 #Ij , the dimension of the tensor space is ! Yd Od dim dim(RIj ). RIj = j=1

j=1

Remark 1.2. Let #Ij = n, i.e., dim(RIj ) = n for 1 ≤ j ≤ d. Then the dimension of the tensor space is nd . Unless both n and d are rather small numbers, nd is a huge number. In such cases, nd may far exceed the computer memory. This fact indicates a practical problem which must be overcome. The set of matrices with indices in I1 × I2 is denoted by RI1 ×I2 . Remark 1.3. (a) In the case of d = 2, the matrix space RI1 ×I2 and the tensor space RI1 ⊗ RI2 are isomorphic: matrix entries Mij correspond to tensor entries vi with i = (i, j) ∈ I1 × I2 . The isomorphism M : RI1 ⊗ RI2 → RI1 ×I2 can be defined by the linear extension of the map M (v ⊗ w) = v wT .

(1.3)

(b) For d = 1 the trivial identity RI = RI1 holds; i.e., vectors are tensors of order 1. (c) ForNthe degenerate case d = 0, the empty product is defined by the underlying 0 Ij := R . field: j=1 R

1.1.2 Tensor Product of Matrices, Kronecker Product Let d pairs of vector spaces Vj and Wj (1 ≤ j ≤ d) and the corresponding tensor spaces d d O O Wj Vj and W = V= j=1

j=1

be given together with linear mappings A(j) : Vj → Wj . The tensor product of the maps A(j) , the so-called Kronecker product, is the linear mapping d O (1.4a) A := A(j) : V → W j=1

defined by d O j=1

v

(j)

∈V

7→

A

d O j=1

! v

(j)

=

d  O

 A(j) v (j) ∈ W

(1.4b)

j=1

for8 all vj ∈ Vj . Since V is spanned by elementary tensors (cf. (1.2)), equation (1.4b) defines A uniquely on V (more details in §3.3). N (j) In De Lathauwer et al. [70, Def. 8], the matrix-vector multiplication Av by A := d j=1 A (1) (2) (d) is denoted by v ×1 A ×2 A · · · ×d A , where ×j is called the j-mode product. 8

6

1 Introduction

In the case of Vj = RIj and Wj = RJj , the mappings A(j) are matrices from Nd R . The Kronecker product j=1 A(j) belongs to the matrix space RI×J with Ij ×Jj

I = I1 × . . . × Id

and

J = J1 × . . . × Jd .

For d = 2 let I1 = {1, . . . , n1 } , I2 = {1, . . . , n2 } , J1 = {1, . . . , m1 }, and J2 = {1, . . . , m2 } be ordered index sets and use the lexicographical ordering9 of the pairs in I = I1 × I2 and J = J1 × J2 . Then the tensor product A ⊗ B ∈ RI×J of the matrices A ∈ RI1 ×J1 and B ∈ RI2 ×J2 has the block form 

 a11 B · · · a1m1 B   ... A ⊗ B =  ... . an1 1 B · · · an1 m1 B

(1.5)



 v (1) . P (µ) e ⊗ v (µ) Similarly, a vector v ∈ RJ is regarded as a block vector  ..  = (m1 ) µ∈J1 v with unit vectors e(µ) ∈ RJ1 (cf. (2.2)) and the vector blocks v (µ) ∈ RJ2 . Exercise 1.4. Let A, B, and v be as above. Check the following. P (a) The matrix-vector multiplication (A ⊗ B) v = µ∈J1 Ae(µ) ⊗ Bv (µ) requires #J1 = m1 multiplications Bv (µ) ; in total, 2#I2 #J1 #J2 operations.   w(1) .. P (ν) e ⊗w(ν) ∈ RI (e(ν) ∈ RI1 , w(ν) ∈ RI2 ) (b) If the vector w =  .  = (n1 ) ν∈J 1 w is similarly organised, the calculation of the scalar product by E D E XD X h(A ⊗ B) v, wi = aνµ Bv (µ) , w(ν) Ae(µ) ⊗ Bv (µ) , e(ν) ⊗ w(ν) = ν∈I1 µ∈J1

ν∈I1 µ∈J1

requires 2#I2 #J1 #J2 + 2#I1 (#I2 + 1) #J1 operations.   w ˆ (1) . P (ν) . ˆ ⊗ e(ν) ∈ RI , where w (c) Use the opposite blocking: w =  .  = (m1 ) ν∈I2 ˆ w w ˆ (ν) ∈ RI1 , e(ν) ∈ RI2 . Show that X X

AT w h(A ⊗ B) v, wi = ˆ (ν) µ Bv (µ) ν . ν∈I2 µ∈J1

What is the corresponding cost? The blockwise Kronecker products of two block matrices yield the Khatri–Rao product. Another related operation is the Tracy–Singh product. 9

This is the ordering (1, 1) , (1, 2) , . . . , (1, n2 ) , (2, 1) , . . . , (2, n2 ) , . . . , (n1 , n2 ) . If another ordering or no ordering is defined, definition (1.5) is incorrect or does not make sense.

1.1 What are Tensors?

7

1.1.3 Tensor Product of Functions Now we redefine Ij ⊂ R as an interval and consider infinite-dimensional vector spaces of functions as Vj = C(Ij ) or Vj = L2 (Ij ). C(Ij ) contains the continuous functions on Ij , while L2 (Ij ) is the space of the measurable and square-integrable functions on Ij . The tensor product of univariate functions fj (xj ) is the d-variate function10 f :=

d O

fj

with f (x1 , . . . , xd ) =

j=1

d Y

fj (xj )

(xj ∈ Ij , 1 ≤ j ≤ d) .

(1.6)

j=1

The product belongs to V=

d O

Vj ,

where V ⊂ C(I) or V ⊂ L2 (I), respectively.

j=1

for Vj = C(Ij ) or Vj = L2 (Ij ) (details in §4 and §4.4). In the infinite-dimensional case, the definition (1.2) must be modified if one wants to obtain a complete (Banach or Hilbert) space. The span of the elementary tensors must be closed with respect to a suitable norm (here norm of C(I) or L2 (I)): d O

Vj = span {v 1 ⊗ v 2 ⊗ . . . ⊗ v d : v j ∈ Vj , 1 ≤ j ≤ d}.

j=1

The tensor structure of functions is often termed separation of the variables. This means that a multivariate function f can either be written as an elementary tensor Nd product j=1 fj as in (1.6) or as a sum (or series) of such products. A particular example of a multivariate function is the polynomial X P (x1 , . . . , xd ) = ai xi ,

(1.7)

i

where each monomial xi :=

Qd

j=1 (xj )

ij

is an elementary product.

The definitions in §§1.1.1-3 may lead to the impression that there are different tensor products. This is only partially true. The cases of §§1.1.1–2 follow the same Nd concept. In Chapter 3, the algebraic tensor product V = a j=1 Vj of general vector spaces Vj (1 ≤ j ≤ d) will be defined. Choosing Vj = RIj , we obtain tensors as in §1.1.1, while for matrix spaces Vj = RIj ×Jj the tensor product coincides with the Kronecker product. The infinite-dimensional case of §1.1.3 is different since topological tensor spaces require a closure with respect to some norm (see Chapter 4). 10

According to Notation 1.1 we might write f [x1 , x2 , . . . , xd ] instead of f (x1 , x2 , . . . , xd ). In the sequel we use the usual notation of the argument list with round brackets.

8

1 Introduction

1.2 Where do Tensors Appear? At the first sight, tensors of order d ≥ 3 do not seem to be used so often. Vectors (the particular case d = 1) appear almost everywhere. Since matrices (case d = 2) correspond to linear mappings, they are also omnipresent. The theory of vectors and matrices has led to the field of linear algebra. However, there are no standard constructions in linear algebra which lead to tensors of order d ≥ 3. Instead, tensors are studied in the field of multilinear algebra.

1.2.1 Tensors as Coefficients The first purpose of indexed quantities is a simplification of notation. For instance, the description of the polynomial (1.7) in, say, d = 3 variables is easily readable if coefficients aijk with three indices are introduced. In §1.6 we shall mention such an approach already used by Cayley in 1845. Certain quantities in the partial differential equations of elasticity or in Maxwell’s equations are called tensors. However, these tensors are usually of order two and therefore, the term ‘matrix’ would be more appropriate. Moreover, in physics the term ‘tensor’ is often used with the meaning ‘tensor-valued function’. A true tensor c of order 4 is involved in Hooke’s law for three dimensions, where P the stress ‘tensor’ σ and the elasticity ‘tensor’ ε are connected via σij = − k` cijk` εk` (cf. Cauchy [52, pages 293–319]). In differential geometry, tensors are widely used for coordinate transformations. Typically, one distinguishes covariant and contravariant tensors and those of mixed type. The indices of the coefficients are placed either in lower position (covariant case) or in upper position (contravariant). For instance, aij k is a mixed tensor with two covariant and one contravariant component. For coordinate systems in Rn , all indices vary in {1, . . . , n}. The notational advantage of the lower and upper indices is the implicit Einstein summation rule: expressions containing a certain index in both positions are to be summed over this index. We give an example (cf. Kreyszig [201]). Let a smooth two-dimensional manifold be described by the function x(u1 , u2 ). First and second derivatives with respect to these coordinates are denoted by xuk and xui ,uj . Together with the normal vector n, the Gaussian formula for the second derivatives is xui ,uj = Γij k xuk + aij n

(apply summation over k),

where Γij k are the Christoffel symbols11 of the second kind (cf. Christoffel [59], 1869). 11

The notation Γij k is not used in Christoffel’s original paper [59].

1.2 Where do Tensors Appear?

9

The algebraic explanation of co- and contravariant tensors is as follows. The dual space to V := Rn is denoted by V 0 . Although V 0 is isomorphic to V, it is considered as a different vector space. In particular the basis transformations Nd are different (cf. Lemmata 3.8 and 3.9). Mixed tensors are elements of V = j=1 Vj , where Vj is either V (contravariant component) or V 0 (covariant component). The summation rule defines the dual form v 0 (v) of v 0 ∈ V 0 and v ∈ V .

1.2.2 Tensor Decomposition for Inverse Problems In many fields (psychometrics, linguistics, chemometrics,12 telecommunication, biomedical applications, information extraction,13 computer vision,14 etc.) matrixvalued data appear. M ∈ Rn×m may correspond to m measurements of different properties j, while i is associated to n different input data. For instance, in problems from chemometrics the input may be an excitation spectrum, while the output is the emission spectrum. Assuming linear behaviour, we obtain for one substance a matrix abT of rank one. In this case the inverse problem is trivial: the data abT allow us to recover the vectors a and b up to a constant factor. Having a mixture of r substances, we obtain a matrix M=

r X

(aν ∈ Rn , bν ∈ Rm ),

cν aν bT ν

ν=1

where cν ∈ R is the concentration of substance ν. The componentwise version of the latter equation is r X Mij = cν aνi bνj . ν=1

and B = [b1 b2 . . . br ] ∈ Rm×r , we may With A = [c1 a1 c2 a2 . . . cr ar ] ∈ R write M = AB T . n×r

Now the inverse problem is the task to recover the factors A and B. This, however, is impossible since A0 = A T and B 0 = T −T B satisfy M = A0 B 0T for any regular matrix T ∈ Rr×r . Tensors of order three come into play when we repeat the experiments with varying concentrations cνk (concentration of substance ν in the k-th experiment). The resulting data are r X cνk aνi bνj . Mijk = ν=1

12 13 14

See, for instance, Smile–Bro–Geladi [264] and De Lathauwer–De Moor–Vandevalle [71]. See, for instance, Lu–Plataniotis–Venetsanopoulos [218]. See, for instance, Wang–Ahuja [299].

10

1 Introduction

By the definition (1.1), we can rewrite the latter equation as15 M=

r X

aν ⊗ bν ⊗ cν .

(1.8)

ν=1

Under certain conditions, it is possible to recover the vectors aν ∈ Rn , bν ∈ Rm , cν ∈ Rr from the data M ∈ Rn×m×r (up to scaling factors; cf. Remark 7.4b). In these application fields, the above ‘inverse problem’ is called a ‘factor analysis’ or ‘component analysis’ (cf. [158], [71]). These techniques have been developed in the second part of the last century: Cattell [51] (1944), Tucker [284] (1966), Harshman [158] (1970), Appellof– Davidson [8] (1981), Henrion [160] (1994), De Lathauwer–De Moor–Vandevalle [71] (2000), Comon [61] (2002), Smilde–Bro–Geladi [264] (2004), Kroonenberg [202] (2008), and many more (see also the review by Kolda–Bader [197]). In this monograph, we shall not study these inverse problems. In §7.1.3, the difference between tensor representations and tensor decompositions will be discussed. Our emphasis lies on the tensor representation. We remark that the tensors considered above cannot really be large-sized as long as all entries Mijk can be stored.

1.2.3 Tensor Spaces in Functional Analysis The analysis of topological tensor spaces started by Schatten [254] (1950) and Grothendieck [129]. Chapter 4 introduces parts of their concepts. However, most of the applications in functional analysis concern tensor products X = V ⊗ W of two Banach spaces. The reason is that these tensor spaces of order two can be related to certain linear operator spaces. The interpretation of X as tensor product may allow us to transport certain properties from the factors V and W , which are easier to be analysed, to the product X which may be of a more complicated nature.

1.2.4 Large-Sized Tensors in Analysis Applications In analysis, the approximation of functions is well-studied. The quality of approximation is usually related to smoothness properties. If a function is the solution of a partial differential equation, a lot is known about its regularity (cf. [141, §9]). Below, we give an example how the concept of tensors may appear in the context of partial differential equations and their discretisations. 15

Representations as (1.8) are used by Hitchcock [164] in 1927 (see §1.6).

1.2 Where do Tensors Appear?

11

1.2.4.1 Partial Differential Equations Let Ω = I1 × I2 × I3 ⊂ R3 be the product of three intervals and consider an elliptic differential equation Lu = f on Ω, e.g., with Dirichlet boundary conditions u = 0 on the boundary Γ = ∂Ω. A second order differential operator L is called separable if ∂ ∂ ∂ L = L1 + L2 + L3 with Lj = aj (xj ) + bj (xj ) + cj (xj ). (1.9a) ∂xj ∂xj ∂xj Note that any differential operator with constant coefficients and without mixed derivatives is of this kind. According to §1.1.3, we may consider the three-variate function as a tensor of order three. Moreover, the operator L can be regarded as a Kronecker product:16 L = L1 ⊗ id ⊗ id + id ⊗ L2 ⊗ id + id ⊗ id ⊗ L3 .

(1.9b)

This tensor structure becomes more obvious when we consider a finite difference discretisation of Lu = f  . Assume, e.g., that I1 = I2 = I3 = [0, 1] and introduce the uniform grid Gn = ( ni , nj , nk ) : 0 ≤ i, j, k ≤ n of grid size h = 1/n. The discrete values of u and f at the nodes of the grid are denoted by17 uijk := u( ni , nj , nk ),

fijk := f ( ni , nj , nk ),

(1 ≤ i, j, k ≤ n − 1) .

Hence u and f are tensors of the size (n − 1)×(n − 1)×(n − 1). The discretisation of the one-dimensional differential operator Lj in (1.9a) yields a tridiagonal matrix L(j) ∈ R(n−1)×(n−1) . As in (1.9b), the matrix of the discrete system Lu = f is the Kronecker product L = L(1) ⊗ I ⊗ I + I ⊗ L(2) ⊗ I + I ⊗ I ⊗ L(3) .

(1.10)

I ∈ R(n−1)×(n−1) is the identity matrix. Note that L has size (n − 1)3 ×(n − 1)3 . The standard treatment of the system Lu = f views u and f as vectors from RN with N := (n − 1)3 and tries to solve N equations with N unknowns. If n ≈ 100, a system with N ≈ 106 equations can still be handled. However, for n ≈ 10000 or even n ≈ 106 , a system of the size N ≈ 1012 or N ≈ 1018 exceeds the capacity of standard computers. If we regard u and f as tensors of Rn−1 ⊗ Rn−1 ⊗ Rn−1 , it might be possible to find tensor representations with much less storage. Consider, for instance, a uniform load f = 1. Then f = 1 ⊗ 1 ⊗ 1 is an elementary tensor, where 1 ∈ Rn−1 is the vector with entries 1i = 1. The matrix L is already written as Kronecker product (1.10). In §9.8.2.6 we shall show that at least for positive-definite L a very accurate inverse matrix B ≈ L−1 can be constructed and that the matrix-vector multiplication u ˜ = Bf can be performed. The required storage for the representation of B and 16 17

The symbol id denotes the identity map. In the matrix case, we usually write I instead of id. Because of the boundary condition, uijk = 0 holds if one the indices equals 0 or n.

12

1 Introduction

u ˜ is bounded by O(n log2 ( 1ε )), where ε is related to the error L−1 − B 2 ≤ ε. The same bound holds for the computational cost. The following observations are important: (1) Under suitable conditions, the exponential cost nd can be reduced to O(dn) or even O(d log n) (here: d = 3). This allows computations in cases for which the standard approach fails and not even the storage of the data u, f can be achieved. (2) Usually, tensor computations will not be exact but yield approximations. In applications from analysis, there are many cases for which fast convergence holds. In the example from above the accuracy ε improves exponentially with a certain rank parameter so that we obtain the logarithmic factor of log2 (1/ε). Although such a behaviour is typical for many problems from analysis, it does not hold in general, in particular not for random data. (3) The essential key is a tensor representation with two requirements. First, low storage cost is an obvious option. Since the represented tensors are involved in operations (here: the matrix-vector multiplication Bf ), the second option is that such tensor operations should have a comparably low cost. Finally, we give an example for which the tensor structure can be successfully applied without any approximation error. Instead of the linear system Lu = f from above, we consider the eigenvalue problem Lu = λu. First, we discuss the undiscretised problem Lu = λu.

(1.11)

Here it is well known that the separation ansatz u(x, y, z) = u1 (x)u2 (y)u3 (z) yields three one-dimensional boundary eigenvalue problems L1 u1 (x) = λ(1) u1 ,

L2 u2 (y) = λ(2) u2 ,

L3 u3 (z) = λ(3) u3

with zero conditions at x, y, z ∈ {0, 1}. The product u(x, y, z) := u1 (x)u2 (y)u3 (z) satisfies Lu = λu with λ = λ(1) + λ(2) + λ(3) . The latter product can be understood as tensor product: u = u1 ⊗ u2 ⊗ u3 (cf. §1.1.3). Similarly, we derive from the Kronecker product structure (1.10) that the solutions of the discrete eigenvalue problems L(1) u1 = λ(1) u1 ,

L(2) u2 = λ(2) u2 ,

L(3) u3 = λ(3) u3

in Rn−1 yield the solution u = u1 ⊗ u2 ⊗ u3 of Lu = λu with λ = λ(1) + λ(2) + λ(3) . In this example we exploit that the eigensolution is exactly18 equal to an elementary tensor. In the discrete case, this implies that an object of the size 3 (n − 1) can be represented by three vectors of the size n − 1. 18

This holds only for separable differential operators (cf. (1.9a)), but also in more general cases tensor approaches apply as shown in [148] (see §16.3).

1.3 Tensor Calculus

13

1.2.4.2 Multivariate Function Representation The computational realisation of a special function f (x) in one variable may be based on a rational approximation, a recursion etc. or a combination of these tools. Computing a multivariate function f (x1 , . . . , xp ) is even more difficult. Such functions may be defined by complicated integrals, involving parameters x1 , . . . , xp in the integrand or in the integration domain. Consider the evaluation of f on I = I1 × . . . × Ip ⊂ Rp with Ij = [aj , bj ]. We may precompute f at grid points (x1,i1 , . . . , xp,ip ), xj,ij = aj + ij (bj − aj )/n for 0 ≤ ij ≤ n, followed by suitable interpolation at the desired x = (x1 , . . . , xp ) ∈ I. However, we fail as the required storage of the grid values is of the size np . Again, the hope is to find a suitable tensor approximation with storage O(pn) and an evaluation procedure of a similar cost. To give an example, a very easy task is the approximation of the function !−1/2 p X 1 2 = xi for kxk ≥ a > 0. kxk i=1 p √
We obtain a uniform accuracy of the size O exp(−π r/2)/ a with a storage of the size 2r and an evaluation cost O(rp). Details will follow in §9.8.2.5.2.

1.2.5 Tensors in Quantum Chemistry The Schr¨odinger equation determines ‘wave functions’ f (x1 , . . . , xd ) for which each variable xj ∈ R3 corresponds to one electron. Hence the spatial dimension 3d increases with the size of the molecule. A first ansatz19 is f (x1 , . . . , xd ) ≈ Φ(x1 , . . . , xd ) := ϕ1 (x1 )ϕ2 (x2 ) · . . . · ϕd (xd ), which leads to the Hartree–Fock Nd equation. According to (1.6), we can write Φ := j=1 ϕj as a tensor. More accurate approximations require tensors which are linear combinations of such products. The standard ansatz for the three-dimensional functions ϕj (x) are sums of 2 Gaussian functions20 Φν (x) := eαν kx−Rν k as introduced by Boys [39] in 1950. N3 αν (xk −Rν,k )2 Again, Φν is the elementary tensor . k=1 ek with ek (xk ) := e

1.3 Tensor Calculus The representation of tensors (in particular, with not too large storage requirements) is one goal of the efficient numerical treatment of tensors. Another goal is the efficient performance of tensor operations. In the case of matrices, we apply matrix-vector and matrix-matrix multiplications and matrix inversions. The same operations occur for tensors when the matrix is given by a Kronecker matrix and the vector by a tensor. Besides of these operations there are entry-wise multiplications, convolutions etc. 19 20

In fact, the product must be antisymmetrised yielding the Slater determinant from Lemma 3.86. Possibly multiplied by polynomials.

14

1 Introduction

In linear algebra, basis transformations are well known which lead to vector and matrix transforms. Such operations occur for tensors as well. There are matrix decompositions as the singular-value decomposition. Generalisations to tensors will play an important role. These and further operations are summarised under the term of ‘tensor calculus’.21 In the same way, as a library of matrix procedures is the basis for all algorithms in linear algebra, the tensor calculus enables computations in the realm of tensors. Note that already in the case of large-sized matrices, special efficient matrix representations are needed (cf. Hackbusch [138]), although the computational time grows only polynomially (typically cubically) with the matrix size. All the more important are efficient algorithms for tensors to avoid exponential run time. A review of the recent literature in this field is given by Grasedyck–Kressner– Tobler [124].

1.4 Preview 1.4.1 Part I: Algebraic Properties Matrices can be considered as tensors of second order. In Chapter 2 we summarise various properties of matrices, as well as techniques applicable to matrices. QR and singular-value decompositions will play an important role for later tensor operations. In Chapter 3, tensors and tensor spaces are introduced. The definition of the tensor space in §3.2 requires a discussion of free vectors spaces (in §3.1.2) and of quotient spaces (in §3.1.3). Furthermore, linear and multilinear mappings and algebraic dual spaces are discussed in §3.1.4. In §3.2 we introduce the tensor product and the (algebraic) tensor space and define the (tensor) rank of a tensor, which generalises the rank of a matrix. Later we shall introduce other vector-valued ranks of tensors. In §3.3 we have a closer look to linear and multilinear maps. In particular, tensor products of linear maps are discussed. Tensor spaces with additional algebra structure are different from tensor algebras. Both are briefly described in §3.4. In particular applications, symmetric or antisymmetric tensors are needed. These are defined in §3.5. Symmetric tensors are connected to quantics (cf. §3.5.2), while antisymmetric tensors are related to determinants (cf. §3.5.3). 21

The Latin word ‘calculus’ is the diminutive of ‘calx’ (lime, limestone) and has the original meaning ‘pebble’. In particular, it denotes the pieces used in the Roman abacus. Therefore the Latin word ‘calculus’ has also the meaning of ‘calculation’ or, in modern terms, ‘computation’.

1.4 Preview

15

1.4.2 Part II: Functional Analysis of Tensors Normed tensor spaces are needed as soon as we want to approximate certain tensors. Even in the finite-dimensional case, one observes properties of tensors which are completely unknown from the matrix case. In particular in the infinite-dimensional case, we have Banach (or Hilbert) spaces Vj endowed with a norm k·kj , as well Nd as the algebraic tensor space Valg = a j=1 Vj , which together with a norm k·k becomes a normed space. Completion yields the topological Banach space Vtop = Nd k·k j=1 Vj . The tensor space norm k·k is by no means determined by the single norms k·kj . In §4.2 we study the properties of tensor space norms. It turns out that continuity conditions on the tensor product limit the choice for k·k (cf. §4.2.2.1). There are two norms induced by {k·kj : 1 ≤ j ≤ d}, the projective norm (cf. §4.2.3.1) and the injective norm (cf. §4.2.4.2), which are the strongest and weakest possible norms. Further terms of interest are crossnorms (cf. §4.2.2.1), reasonable crossnorms (cf. §4.2.6.1), and uniform crossnorms (cf. §4.2.8). For the case d = 2 considered in §4.2, nuclear and compact operators are discussed (cf. §4.2.9). The extension to d ≥ 3 discussed in §4.3 is almost straightforward except that we also N need suitable norms, e.g., for the tensor spaces a j∈{1,...,d}\{k} Vj of order d − 1. While Lp or C 0 norms of tensor spaces belong to the class of crossnorms, the usual spaces C m or H m (m ≥ 1) cannot be described by crossnorms but by intersections of Banach (or Hilbert) tensor spaces (cf. §4.3.5). The corresponding construction by crossnorms leads to so-called mixed norms. Hilbert spaces are discussed in §4.4. In this case the scalar products h·, ·ij of Vj define the induced scalar product of the Hilbert tensor space (cf. §4.4.1). In the Hilbert case, the infinite singular-value decomposition can be used to define the Hilbert–Schmidt and the Schatten norms (cf. §4.4.3). Besides the usual scalar product the partial scalar product is of interest (cf. §4.5.5). In §4.6 the tensor operations are enumerated which later are to be performed numerically. Particular subspaces of the tensor space ⊗d V are the symmetric and antisymmetric tensor spaces discussed in §4.7. Chapter 5 concerns algebraic tensor spaces, as well as topological ones. We consider different isomorphisms which allow us to regard tensors either as vectors (vectorisation in §5.1) or as matrices (matricisation in §5.2). In particular, the matricisation will become an important tool. The opposite direction is the tensorisation considered in §5.3 and later, in more detail, in Chapter 14. Here vectors from Rn are artificially reformulated as tensors. Another important tool for the analysis and for concrete constructions are the minimal subspaces studied in Chapter 6. Given some tensor v, we ask for Nd the smallest subspaces Uj such that v ∈ j=1 Uj . Of particular interest is their behaviour for sequences vn * v of tensors.

16

1 Introduction

1.4.3 Part III: Numerical Treatment The numerical treatment of tensors is based on a suitable tensor representation. Chapters 7 to 10 are devoted to two well-known representations, the r-term format (also called the canonical or CP format) and the tensor subspace format (also called the Tucker format). We distinguish the exact representation from the approximation task. Exact representations are discussed in Chapter 7 (r-term format) and Chapter 8 (tensor subspace format). If the tensor rank is moderate, the r-term format is a very good choice, whereas the tensor subspace format is disadvantageous for larger tensor order d because of its exponentially increasing storage requirement. Tensor approximations are discussed separately in Chapters 9 (r-term format) and 10 (tensor subspace format). In the first case, many properties known from the matrix case (see §9.3) do not generalise to tensor orders d ≥ 3. A particular drawback is mentioned in §9.4: the set of r-term tensors is not closed. This property ˜ in the may cause a numerical instability. An approximation of a tensor v by some v r-term format may be computed numerically using a regularisation (cf. §9.6). In some cases, analytical methods allow us to determine very accurate r-term approximations to functions and operators (cf. §9.8). In the case of tensor subspace approximations (§10) there are two different options. The simpler approach is based on the higher-order singular-value decomposition (HOSVD; cf. §10.1). This allows a projection to smaller rank similar to the standard singular-value decomposition in the matrix case. The result is not necessarily the best one but quasi-optimal. The second option is the best approximation considered in §10.2. In contrast to the r-term format, the existence of a best approximation is guaranteed. A standard numerical method for its computation is the alternating least-squares method (ALS, cf. §10.3). For particular cases, analytical methods are available to approximate multivariate functions (cf. §10.4). While the r-term format suffers from a possible numerical instability, the storage size of the tensor subspace format increases exponentially with the tensor order d. A format avoiding both problems is the hierarchical format described in Chapter 11. Here the storage is strictly bounded by the product of the tensor order d, the maximal rank involved, and the maximal dimension of the vector spaces Vj . Again, HOSVD techniques can be used for a quasi-optimal truncation. Since the format is closed, numerical instability does not occur. The hierarchical format is based on a dimension partition tree. A particular choice for the tree leads to the matrix product representation or TT format described in Chapter 12. The essential part of the numerical tensor calculus is the performance of tensor operations. In Chapter 13 we describe all operations, their realisation in the different formats, and the corresponding computational cost. The tensorisation briefly mentioned in §5.3 is revisited in Chapter 14. When applied to grid functions, tensorisation corresponds to a multiscale approach. The tensor truncation methods allow an efficient compression of the data size. As shown in §14.2, the approximation can be proved to be at least as good as analytical methods like hp methods, . approximations, or wavelet compression techniques.

1.4 Preview

17

In §14.3 the computation of the convolution is described. The fast Fourier transform is explained in §14.4. The method of tensorisation can also be applied to functions, instead of grid functions as detailed in §14.5. Chapter 15 is devoted to the multivariate cross approximation. The underlying problem is the approximation of general tensors, which has several important applications. In Chapter 16, the application of the tensor calculus to elliptic boundary-value problems (§16.2) and eigenvalue problems (§16.3) is discussed. The final Chapter 17 collects a number of additional topics. §17.1 considers general minimisation problems. Another minimisation approach described in §17.2 applies directly to the parameters of the tensor representation. Dynamic problems are studied in §17.3, while the ANOVA method is mentioned in §17.4.

1.4.4 Topics Outside the Scope of the Monograph As already mentioned in §1.2.2, we do not aim for inverse problems, where the parameters of the representation (decomposition) have a certain external interpretation (see the references in the footnotes 12–14). High-dimensional tensors arise also in data mining (cf. Kolda–Sun [198]). However, in contrast to mathematical applications (e.g., in partial differential equations), weaker properties hold concerning data smoothness, the desired order of accuracy, and even the availability of data. We do not consider data completion (approximation of incomplete data; cf. Liu et al. [214], Kahle et al. [176], Grasedyck–Kr¨amer [123]), which is a typical Nproblem d for data from non-mathematical sources. Entries v[i] of a tensor v ∈ j=1 Rnj d may be available only for i ∈ ˚ I ⊂ I := ×j=1 {1, . . . , nj }. Other I of a subset ˚ examples are cases in which data are lost or deleted. Approximation of the remain˜ of a certain format yields the desired completion (cf. Footnote ing data by a tensor v 10 on page 321). Instead, in Chapter 15 we are discussing quite another kind of data completion, where an approximation v ˜ is constructed by a small part of the data. However, unlike usual data completion, we assume that all data are available on demand, although possibly with high arithmetic cost. Another subject, which is not discussed here, is the detection and determination of principal manifolds of smaller dimension (‘manifold learning’: see, e.g., Feuers¨anger–Griebel [101] or Griebel–Hullmann [127]). The order d of the tensor considered here, is always finite. In fact, the numerical cost of storage or arithmetic operations is at least increasing linearly in d. Infinite dimensions (d = ∞) may appear theoretically (as in §15.1.2.2), but only truncations to finite d are discussed. There are approaches to generalise purely algebraic problems from linear algebra to multilinear algebra; e.g., eigenvalue problems (cf. Cartwright–Sturmfels [50], Qi–Luo [245]) and certain decompositions. Unfortunately, usually such problems are NP-hard (cf. Hillar–Lim [162, §9]) and do not help for large-sized tensors.

18

1 Introduction

1.5 Software Free software for tensor applications is offered by the following groups: • M ATLAB Tensor Toolbox by Bader–Kolda [14]: http://www.tensortoolbox.org/ • Hierarchical Tucker Toolbox by Tobler–Kressner [200]: https://anchp.epfl.ch/index-html/software/htucker/ • Tensorlab 3.0 by Vervliet–Debals–Sorber–Van Barel–De Lathauwer: https://www.tensorlab.net/ • TT TOOLBOX http://github.com/oseledets/TT-Toolbox/ by I. Oseledets • TensorCalculus http://www.swmath.org/software/13726 by M. Espig et al. • B.M. Wise and N.B. Gallagher: http://www.eigenvector.com • Andersson–Bro [6]: http://www.models.life.ku.dk/nwaytoolbox

1.6 Comments about the Early History of Tensors The word ‘tensor’ seems to be used for the first time in an article by William Rowan Hamilton [155] from 1846. The meaning, however, was quite different. Hamilton is well known for his quaternions. Like complex numbers, a modulus of a quaternion can be defined. For this nonnegative real number he introduced the name ‘tensor’. The word ‘tensor’ is again used in a book by Woldemar Voigt [298] in 1898 for quantities which come closer to our understanding.22 In May 1845, Arthur Cayley [53] submitted a paper in which he described hyperdeterminants.23 There he considers tensors of general order. For instance, he gives an illustration of a tensor from R2 ⊗ R2 ⊗ R2 (p. 11 in [298]): e m = 2, et prenons Soit n = 3, posons pour plus simpicit´ 111 = a, 211 = b, 121 = c, 221 = d, 22

112 = e, 212 = b, 122 = g, 222 = h,

From [298, p. 20]: Tensors are “. . . Zust¨ande, die durch eine Zahlgr¨osse und eine zweiseitige Richtung charakterisiert sind. . . . Wir wollen uns deshalb nur darauf st¨utzen, dass Zust¨ande der geschilderten Art bei Spannungen und Dehnungen nicht starrer K¨orper auftreten, und sie deshalb tensorielle, die f¨ur sie charakteristischen physikalischen Gr¨ossen aber Tensoren nennen.” 23 p q r See also [73, §5.3]. The hyperdeterminant vanishes P for a tensor v ∈ R ⊗ R ⊗ R if and only if the associated multilinear form ϕ(x, y, z) := i,j,k v[i, j, k]xi yj zk allows nonzero vectors x, y, z such that ∇x ϕ, ∇y ϕ, or ∇z ϕ vanish at (x, y, z).

1.7 Notations

19

de mani` a consid´ ere que la fonction ` erer est U = ax1 y1 z1 + bx2 y1 z1 + cx1 y2 z1 + dx2 y2 z1 + ex1 y1 z2 + f x2 y1 z2 + gx1 y2 z2 + hx2 y2 z2 . Next, he considers linear transformations Λ(1) , Λ(2) , Λ(3) in all three directions, e.g., Λ(1) is described as follows. ´quations pour la transformation sont Les e x1 = λ11 x˙ 1 + λ12 x˙ 2 , x2 = λ21 x˙ 1 + λ22 x˙ 2 , .. . The action of the transformations Λ(1) , Λ(2) , Λ(3) already represents the Kronecker product Λ(1) ⊗ Λ(2) ⊗ Λ(3) . The paper of Hitchcock [164] from 1927 has a similar algebraic background. The author states that ‘any covariant tensor Ai1 ..ip can be expressed as the sum of a finite number of which is the product of p covariant vectors’. In [164] the ranks are defined which we introduce in Definition 5.7. Although in this paper he uses the name ‘tensor’, in the following paper [163] of the same year he prefers the term ‘matrix’ or ‘p-way matrix’. The tensor product of vectors a, b is denoted by ab without any special tensor symbol. In §1.1.2 we named the tensor product of matrices ‘Kronecker product’. In fact, this term is well-introduced, but historically it seems to be unfounded. The ‘Kronecker product’ (and its determinant) has first been studied by Johann Georg Zehfuss [308] in 1858, while it is questionable whether there exists any notice of Kronecker about this product (see Henderson–Pukelsheim–Searle [174] for historical remarks). Zehfuss’ result about determinants can be found in Exercise 4.164.

1.7 Notations A list of symbols, letters etc. can be found on page xix. Here we collect the notational conventions which we use in connection with vectors, matrices, and tensors. Index Sets. I, J, K are typical letters used for index sets. In general, we do not require that an index set be ordered. This allows, e.g., to define a new index set K := I × J as the product of index sets I, J without prescribing an ordering of the pairs (i, j) of i ∈ I and j ∈ J.

20

1 Introduction

Fields. A vector space is associated with some field, which will be denoted by K. The standard choices24 are R and C. When we use the symbol K instead of the special choice R, we use the complex-conjugate value λ of a scalar whenever this is required in the case of K = C. Vector Spaces Kn and KI . Let n ∈ N. Kn is the standard notation for the n vector space of the n-tuples v = (vi )i=1 with vi ∈ K. The more general notation I K abbreviates the vector space {v = (vi )i∈I : vi ∈ K}. Equivalently, we may define KI as the space of mappings from I into K. Note that this definition makes sense for non-ordered index sets. If, e.g., K = I × J is the index set, a vector v ∈ KK has entries vk = v(i,j) for k = (i, j) ∈ K. The notation v(i,j) must be distinguished from vi,j which indicates a matrix entry. The simple notation Kn is identical to KI for I = {1, . . . , n}. Vectors will be symbolised by small letters. Vector entries are usually denoted by vi . The alternative notation v[i] is used if the index carries a secondary index (example: v[i1 ]) or if the symbol for the vector is already indexed (example: vν [i] for vν ∈ KI ). Typical symbols for vector spaces are V , W , U , etc. Often, U is used for subspaces. Matrices and Matrix Spaces KI×J . Any linear mapping Φ : KI → KJ (I, J index sets) can be represented by a matrix25 M ∈ KI×J with entries Mij ∈ K and we may write M = (Mij )i∈I,j∈J or M = (Mij )(i,j)∈I×J . The alternative notation Kn×m is used for the special index sets I = {1, . . . , n} and J = {1, . . . , m}. Even the mixed notation KI×m appears if J = {1, . . . , m}, while I is a general index set. Matrices will be symbolised by capital letters. Matrix entries are denoted by Mi,j = Mij or by M [i, j]. Given a matrix M ∈ KI×J , its i-th row or its j-th column will be denoted by Mi,• = M [i, •] ∈ KJ

or M•,j = M [•, j] ∈ KI , respectively.

If τ ⊂ I and σ ⊂ J are index subsets, the restriction of a matrix is written as M |τ ×σ = (Mij )i∈τ,j∈σ ∈ Kτ ×σ . More about matrix notations will follow in §2.1. 24

Finite fields are of less interest, since approximations do not make sense. Nevertheless, there are applications of tensor tools for Boolean data (cf.Lichtenberg–Eichler[210]). Most of the results not involving norms and absolute values hold for general fields. At some places we require that K must have characteristic 0, i.e., finite fields are excluded. 25 M ∈ KI×J is considered as a matrix, whereas v ∈ KK for K = I × J is viewed as a vector.

1.7 Notations

21

Tensors. Tensors are denoted by small boldface letters: v, w, . . . , a, b, . . . Their entries are usually indexed in square brackets: v[i1 , . . . , id ]. In simple cases, subscripts may be used: vijk . The boldface notation v[i1 , . . . , id ] is also used in the case of a variable d which possibly takes the values d = 1 [vector case] or 2 [matrix case]. The standard notation for a tensor space of order d is V=

d O

Vj .

j=1

Here Vj (1 ≤ j ≤ d) are vector spaces generating the tensor space V. As in this example, tensor spaces are denoted by capital letters in bold type. U is the typical letter for a subspace of a tensor space. Nd Elementary tensors from V = j=1 Vj have the form v=

d O

v (j) = v (1) ⊗ . . . ⊗ v (d) .

j=1

The superscript in round brackets indicates the vector corresponding to the j-th direction. The preferred letter for the direction index is j (or k if a second index (j) is needed). The entries of v (j) may be written as vi or v (j) [i]. A lower subscript Pr Nd (j) (j) may also denote the ν-th vector vν ∈ Vj as required in v = ν=1 j=1 vν . In (j) (j) this case the entries of vν are written as vν [i]. To be precise, we N have to distinguish algebraic and topological tensor Nbetween d d spaces denoted by a j=1 Vj and k·k j=1 Vj , respectively. Details can be found in Notation 3.10.

Chapter 2

Matrix Tools

Abstract In connection with tensors, matrices are of interest for two reasons. Firstly, they are tensors of order two and therefore a nontrivial example of a tensor. Differently from tensors of higher order, matrices allow us to apply practically realisable decompositions. Secondly, operations with general tensors will often be reduced to a sequence of matrix operations (realised by well-developed software). Sections 2.1–2.3 introduce the notation and recall well-known facts about matrices. Section 2.5 discusses the important QR decomposition and the singular-value decomposition (SVD) and their computational cost. The (optimal) approximation by matrices of lower rank explained in Section 2.6 will be used later in truncation procedures for tensors. In Part III we shall apply some linear algebra procedures introduced in Section 2.7 based on QR and SVD.

2.1 Matrix Notations In this subsection, the index sets I, J are assumed to be finite. As soon as complex conjugate values appear,1 the scalar field is restricted to K ∈ {R, C}. We recall the notation KI×J explained in §1.7. The entries of a matrix M ∈ KI×J are denoted by Mij (i ∈ I, j ∈ J). Conversely, numbers αij ∈ K (i ∈ I, j ∈ J) may be used to define M := (αij )i∈I, j∈J ∈ KI×J . Let j ∈ J. The j-th column of M ∈ KI×J is the vector M [•, j] = (Mij )i∈I ∈ KI , while vectors c(j) ∈ KI generate a matrix M := [ c(j) : j ∈ J ] ∈ KI×J . If J is ordered, we may write M := [ c(j1 ) , c(j2 ) , . . .]. δij (i, j ∈ I) is the Kronecker symbol defined by  δij = 1

1 0

if i = j ∈ I, otherwise.

(2.1)

In the case of K = R, α = α ¯ holds for all α ∈ K.

© Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_2

23

2 Matrix Tools

24

The unit vector e(i) ∈ KI (i ∈ I) is defined by e(i) := (δij )j∈I .

(2.2)

The symbol I = (δij )i,j∈I is used for the identity matrix. Since matrices and index sets do not appear at the same place, the simultaneous use of I for a matrix and for an index set should not lead to any confusion (example: I ∈ KI×I ). If M ∈ KI×J , the transposed matrix M T ∈ KJ×I is defined by Mij = (M T )ji (i ∈ I, j ∈ J). A matrix from KI×I is symmetric if M = M T . The Hermitian transposed matrix M H ∈ KJ×I is M T , i.e., Mij = (M T )ji , where • is the complex conjugate value. If K = R, M H = M T holds. This allows us to use H for the general case of K ∈ {R, C}. A Hermitian matrix satisfies M = M H . The range of a matrix M ∈ KI×J is2 range(M ) := {M x : x ∈ KJ }. The Euclidean scalar product in KI is given by X hx, yi = y H x = xi yi , i∈I

where, in the real case K = R, the conjugate sign can be ignored. In the case of K = C, the scalar product is a sesquilinear form, i.e., it is antilinear in the second argument.3 Two vectors x, y ∈ KI are orthogonal (symbolic notation: x⊥y) if hx, yi = 0. A family of vectors {xν }ν∈F ⊂ KI is orthogonal if the vectors are pairwise orthogonal, i.e., hxν , xµ i = 0 for all ν, µ ∈ F with ν 6= µ. Similarly, two vectors x, y ∈ KI or a family {xν }ν∈F ⊂ KI are orthonormal, if, in addition, all vectors are normalised: hx, xi = hy, yi = 1 or hxν , xν i = 1 (ν ∈ F ). A matrix M ∈ KI×J is called orthogonal, if the columns of M are orthonormal. An equivalent characterisation is M H M = I ∈ KJ×J .

(2.3)

Note that the (Hermitian) transpose of an orthogonal matrix is, in general, not orthogonal. M ∈ KI×J can be orthogonal only if #J ≤ #I. An orthogonal square4 matrix M ∈ KI×I is called unitary (if K = R, often the term ‘orthogonal’ is preferred). In contrast to the remark above, unitary matrices satisfy 2

Also the notation colspan(M ) exists, since range(M ) is spanned by the columns of M. A mapping ϕ is called antilinear, if ϕ(x + αy) = ϕ(x) + αϕ(y) for α ∈ C. 4 We may assume M ∈ KI×J with #I = #J and different I, J. Then M H M = I ∈ KI×I and M M H = I ∈ KJ×J are the precise conditions. 3

2.2 Matrix Rank

25

M H M = M M H = I ∈ KI×I , i.e., M H = M −1 holds. Assume that the index sets satisfy either I ⊂ J or J ⊂ I. Then a (rectangular) matrix M ∈ KI×J is diagonal, if Mij = 0 for all i 6= j, (i, j) ∈ I ×J. Given numbers δi (i ∈ I ∩ J), the associated diagonal matrix M with Mii = δi is written as diag{δi : i ∈ I ∩ J}. If the index set I ∩ J is ordered, an enumeration of the diagonal entries can be used: diag{δi1 , δi2 , . . .}. Again, assume I ⊂ J or J ⊂ I and a common ordering of I ∪ J. A (rectangular) matrix M ∈ KI×J is lower triangular, if Mij = 0 for all (i, j) ∈ I ×J with i > j. Similarly, Mij = 0 for all i < j defines the upper triangular matrix.

2.2 Matrix Rank Remark 2.1. Let M ∈ KI×J . The following statements are equivalent and each may be used as definition of the ‘matrix rank’ r = rank(M ): (a) r = dim range(M ), (b) r = dim range(M T ), (c) r is the maximal number of linearly independent rows in M, (d) r is the maximal number of linearly independent columns in M, (e) r ∈ N0 is minimal with the property M=

r X

ai biT ,

where ai ∈ KI and bi ∈ KJ ,

(2.4)

i=1

(f) r is maximal with the property that there exists a regular r×r submatrix5 of M . (g) r is the number of positive singular values (see (2.16a)). In (b) and (e) we may replace •T with •H . Part (e) states in particular that products ai biT of nonvanishing vectors represent all rank-1 matrices. The rank of M ∈ KI×J is bounded by the maximal rank rmax := min {#I, #J} ,

(2.5)

and this bound is attained for the so-called full-rank matrices. The definition of linear independency depends on the field K. This leads to the following question. A real matrix M ∈ RI×J may also be considered as an element of CI×J . Hence, in principle, such an M may possess a ‘real’ rank and a 5

That means that there are I 0 ⊂ I and J 0 ⊂ J with #I 0 = #J 0 = r and M |I 0 ×J 0 regular.

2 Matrix Tools

26

‘complex’ rank. However, the equivalent characterisations (f) and (g) of Remark 2.1 are independent of the choice K ∈ {R, C} and prove the next remark. Remark 2.2. For M ∈ RI×J ⊂ CI×J the value of rank(M ) is independent of the field K ∈ {R, C}. Corollary 2.3. Let r = rank(M ) and define A := [a1 , . . ., ar ] and B := [b1 , . . ., br ] by ai and bi in (2.4). Then (2.4) is equivalent to M = AB T . An interesting matrix family is the set of matrices of rank not exceeding r :  Rr := M ∈ KI×J : rank(M ) ≤ r . (2.6) Any M ∈ Rr may be written in the form (2.4). Lemma 2.4. The sets Rr ⊂ KI×J for r ∈ N0 are closed. Any convergent sequence R(k) ∈ Rr satisfies   lim inf rank(R(k) ) ≥ rank lim R(k) . (2.7) k→∞

k→∞

 Proof. For s ∈ N0 set Ns := k ∈ N : rank(R(k) ) = s ⊂ N and r∞ = min{s ∈ N0 : #Ns = ∞} = lim inf rank(R(k) ) ≤ r. k→∞

We restrict R(k) to the subsequence with k ∈ Nr∞ , i.e., rank(R(k) ) = r∞ . For full rank, i.e., r∞ = min{#I, #J}, nothing has to be proved. Otherwise we use the criterion from Remark 2.1f: all (r∞ +1)×(r∞ +1) submatrices (I 0 ⊂ I, J 0 ⊂ J, #I 0 = #J 0 = r∞ + 1)
are singular, in particular, det R(k) |I 0 ×J 0 = 0 holds. 0 = lim det(R(k) |I 0 ×J 0 ) = det(lim(R(k) )|I 0 ×J 0 ) follows from the continuity of determinants and proves that rank(lim R(k) ) ≤ r∞ ≤ r; hence, lim R(k) ∈ Rr , proving that Rr is closed. t u R(k) |I 0 ×J 0

Remark 2.5. A matrix M ∈ KI×J with random entries has maximal rank rmax with probability one. Proof. Matrices of smaller rank form a subset of measure zero.

t u

Definition 2.6 (universal k-column independence). For an m-tuple of vectors (a1 , . . . , am ) ∈ V m let k ∈ {0, . . . , m} be the largest integer such that all subtuples of size k consist of linearly independent vectors. Then krank(a1 , . . . , am ) := k is called the Kruskal rank (cf. Kruskal [204, page 13]).

2.3 Matrix Norms

27

2.3 Matrix Norms Before the Euclidean, spectral and Frobenius norms will be discussed, the trace of a square matrix is introduced. For a generalisation of the trace to operators, see (4.66). Definition 2.7. The mapping trace : KI×I → K is defined by X Mii . trace(M ) :=

(2.8)

i∈I

Exercise 2.8. (a) trace(AB) = trace(BA) for any A ∈ KI×J and B ∈ KJ×I . (b) trace(M ) = trace(UM U H ) for M ∈ KI×I and any orthogonal matrix U∈ KJ×I (in particular, for a unitary matrix U ∈ KI×I ). (c) Let λi (i ∈ I) bePall eigenvalues of M ∈ KI×I according to their multiplicity. Then trace(M ) = i∈I λi . The general definition of norms and scalar products can be found in §4.1.1 and §4.4.1. The Frobenius norm s X 2 kM kF = |Mi,j | for M ∈ KI×J (2.9) i∈I,j∈J

is also called the Schur norm or Hilbert–Schmidt norm. This norm is generated by the Frobenius scalar product X hA, BiF := Ai,j Bi,j = trace(AB H ) = trace(B H A), (2.10) i∈I,j∈J 2

2

since hM, M iF = kMkF . In particular, kMkF = trace(MM H ) = trace(M HM ) holds. Remark 2.9. Let I×J and I 0×J 0 define two matrix formats with the same number of entries: #I · #J = #I 0 · #J 0 . Any bijective mapping π : I × J → I 0×J 0 generates 0 0 a mapping P : M ∈ KI×J 7→ P (M ) = M 0 ∈ KI ×J via M 0 [i0 , j 0 ] = M [i, j] 0 0 for (i , j ) = π(i, j). Then the Frobenius norm and scalar product are invariant with respect to P , i.e., kP (M )kF = kM kF

and

hP (A), P (B)iF = hA, BiF .

Let k·kX and k·kY be vector norms on X = KI and Y = KJ , respectively. Then the associated matrix norm is6   kM ykX kM k := kM kX←Y := sup : 0 6= y ∈ KJ for M ∈ KI×J . kykY If k·kX and k·kY coincide with the Euclidean vector norm 6

In the degenerate case of Y = {0} set kM k := 0.

2 Matrix Tools

28

kuk2 :=

sX

2

for u ∈ KK ,

|ui |

i∈K

the associated matrix norm kM kX←Y is the spectral norm denoted by kM k2 . Exercise 2.10. Let M ∈ KI×J . (a) Another equivalent definition of k·k2 is yH M x

 kM k2 = sup

p

J

y H y · xH x

: 0 6= x ∈ K , 0 6= y ∈ K

I

 .

(2.11)

(b) kM k2 = kU M k2 = kM V H k2 = kU M V H k2 holds for orthogonal matrices 0 0 U ∈ KI ×I and V ∈ KJ ×J . From Lemma 2.23b we shall learn that the squared spectral norm kM k22 is the largest eigenvalue of both matrices M H M and M M H . Both matrix norms k·k2 and k·kF are submultiplicative, i.e., kABk ≤ kAk kBk. The example of A = B = I ∈ Rn×n shows the equality in 1 = kI · Ik2 ≤ √ kIk2 kIk2 = 1, while n = kI · IkF ≤ kIkF kIkF = n is a rather pessimistic estimate. In fact, spectral and Frobenius norms can be mixed to get better estimates. Lemma 2.11. The product of A ∈ KI×J and B ∈ KJ×K is estimated by kABkF ≤ kAk2 kBkF

as well as

kABkF ≤ kAkF kBk2 .

P 2 2 Proof. C[•, j] denotes the j-th column of C ∈ KI×K . kCkF = j∈J kC[•, j]k2 involves the Euclidean norm of the columns. For C := AB the columns satisfy C[•, j] = A · B[•, j] and the estimate kC[•, j]k2 ≤ kAk2 kB[•, j]k2 . Together 2 2 2 with the above identity, kAkF ≤ kAk2 kBkF follows. The second inequality can be obtained from the first one because kXkF = kX T kF and kXk2 = kX T k2 . t u A particular consequence is kAk2 ≤ kAkF (use B = I in the second inequality). Exercise 2.12. Let U ∈ KI

0

×I

and V ∈ KJ

0

×J

be orthogonal matrices and prove:

(a) kM kF = kU M kF = kM V H kF = kU M V H kF for M ∈ KI×J .

 (b) hA, BiF = U AV H , U BV H F for A, B ∈ KI×J . Exercise 2.13. For index sets I, J, and K let A ∈ KI×K and B ∈ KJ×K with rank(B) = #J ≤ #K. Show that the matrix C ∈ KI×J minimising kA − CBkF is given by C := AB H (BB H )−1 .

2.4 Semidefinite Matrices

29

2.4 Semidefinite Matrices A matrix M ∈ KI×I is called positive semidefinite if M = MH

and

hM x, xi ≥ 0

for all x ∈ KI .

In addition, a positive-definite matrix has to satisfy hM x, xi > 0 for 0 6= x ∈ KI . Remark 2.14. Let M ∈ KI×I be positive [semi]definite. (a) The equation X 2 = M has a unique positive-[semi]definite solution in KI×I , which is denoted by M 1/2 . (b) M has positive [nonnegative] diagonal entries Mii (i ∈ I). In the set of Hermitian matrices from KI×I a semi-ordering can be defined via A≤B

:⇐⇒

B − A positive semidefinite.

(2.12)

When we write A ≤ B, we always tacitly assume that A and B are Hermitian. Remark 2.15. Let A, B ∈ KI×I be Hermitian. (a) A ≤ B is equivalent to hAx, xi ≤ hBx, xi

for all x ∈ KI .

(b) For any matrix T ∈ KI×J the inequality A ≤ B implies T HAT ≤ T HB T . (c) A ≤ B implies trace(A) ≤ trace(B). (d) A ≤ B implies trace(T HAT ) ≤ trace(T HBT ) for all T. ˆ ≤ E ∈ KI×I and arbitrary Ci ∈ KJ×K (i ∈ I), we have Lemma 2.16. For 0 ≤ E X X ˆij Ci C H ≤ X := ˆ := Eij Ci CjH ∈ KJ×J . E 0≤X j i,j∈I

i,j∈I

ˆ = U diag{λk : k ∈ I} U H holds with λk ≥ 0. Set E−E Proof. Diagonalisation P H ˆ = P ˆ Bk := i∈I Uik Ci . Then X − X k∈I λk Bk Bk proves X ≤ X because H t u λk ≥ 0 and Bk Bk ≥ 0. A tuple x := (xi : i ∈ I) of vectors7 xi ∈ KJ leads to the scalar products hxj , xi i for all i, j ∈ I. Then the Gram matrix of x is defined by
(2.13) G := G(x) = hxj , xi i i,j∈I . Exercise 2.17. (a) Gram matrices are always positive semidefinite. (b) The Gram matrix G(x) is positive definite if and only if x is a tuple of linearly independent vectors. (c) Any positive-definite matrix G ∈ KI×I can be interpreted as a Gram matrix of I a basis xP by defining a scalar product via hv, wi := bH Ga := (xi : i ∈ I) of KP for v = i∈I ai xi and w = i∈I bi xi . 7

Here, KJ can also be replaced by an infinite dimensional Hilbert space.

2 Matrix Tools

30

Lemma 2.18. The spectral norm of G(x) can be characterised by ) (

2 X 2

X

: ξi ∈ K with kG(x)k2 = max ξi xi |ξi | = 1 .

2

i∈I

i∈I

Proof. Let ξ := (ξi )i∈I ∈ KI . kG(x)k2 = max {|hGξ, ξi| : kξk = 1} holds since

P  P P G(x) is symmetric. hGξ, ξi = i,j hxj , xi i ξj ξi = j∈I ξj xj , i∈I ξi xi =

P

2

ξi xi proves the assertion. t u i∈I

2

2.5 Matrix Decompositions Three well-known decompositions will be recalled. The numbers of arithmetic operations8 given below are reduced to the leading term, i.e., terms of lower order are omitted.

2.5.1 Cholesky Decomposition Remark 2.19. Given a positive-definite matrix M ∈ Kn×n , there is a unique lower triangular matrix L ∈ Kn×n with positive diagonal entries such that M = LLH . Computing L costs 31 n3 operations. Matrix-vector multiplications La or LH a or the solution of linear systems Lx = a or LH x = b require n2 operations. For semidefinite matrices (or positive-definite matrices with rather small eigenvalues) there are pivotised versions such that M is equal to P HLLHP with a permutation matrix P and the side condition Lii ≥ 0, instead of Lii > 0 (cf. Hackbusch [138, §9.4.6]). As explained below, the Cholesky decomposition can be used to orthonormalise the m-tuple x := (xi )1≤i≤m of vectors xi ∈ Kn . We associate the tuple x with the Gram matrix G(x) = X H X of X := [x1 x2 · · · xm ] ∈ Kn×m (cf. (2.13)). Lemma 2.20. (a) If x is linearly independent, the Cholesky decomposition G(x) = LLH exists. Then Y := XL−H is an orthogonal matrix, i.e., its columns form an orthonormal m-tuple y := (yi )1≤i≤m . (b) If the vectors x are linearly dependent, the pivotised version of the Cholesky decomposition yields P G(x)P H = LLH with a permutation matrix P and a rect√ Here, we count all arithmetical operations (+, −, ∗, /, , etc.) equally. Sometimes, the combination of one multiplication and one addition is counted as one unit (‘flop’, cf. [36, p. 43]). 8

2.5 Matrix Decompositions

31

 0 0 0 0 angular matrix L = LL00 ∈ Km×m (m0 < m), where L0 ∈ Km ×m is lower 0 triangular with a positive diagonal. Split XP H into [X 0 X 00 ] with X 0 ∈ Kn×m . 0 The orthonormal m -tuple y is defined by columns of the orthogonal matrix 0 Y = X 0 L0−H ∈ Kn×m . Proof. Part (a) can be considered as the particular case of (b) with m0 = m and
H H P = I. Comparing LLH and P G(x)P H = XP H XP H = [X 0 X 00 ] [X 0 X 00 ] 0 0H 0H 0 = X X . Hence the shows that L L assertion follows from the identity Y H Y =
0−H
0−H 0 0H 0−1 0H 0 0−1 LL L = I. L t u X X L =L Remark 2.21. The cost of the procedure described in Lemma 2.20a consists of nm (m + 1) operations for X 7→ G(x), 31 m3 operations for G(x) 7→ L, and m2 n operations for computing Y = XL−H .

2.5.2 QR Decomposition The letter ‘R’ in ‘QR decomposition’ stands for a right (or upper) triangular matrix. Since an upper triangular matrix R is defined by Rij = 0 for all i > j, this requires suitably ordered index sets. The QR decomposition (or ‘QR factorisation’) is a helpful tool for orthogonalisation (cf. [247, §3.4.3], [115, §5.2]) and can be viewed as algebraic formulation of the Gram–Schmidt9 orthogonalisation. For details about different variants and their numerical stability we recommend the book of Bj¨orck [36]. Lemma 2.22 (QR factorisation). Let M ∈ Kn×m . (a) Then there are a unitary matrix Q ∈ Kn×n and an upper triangular matrix R ∈ Kn×m with M = QR

(Q unitary, R upper triangular matrix).

(2.14a)

Q can be constructed as product of Householder transforms (cf. [276, §4.7]). The computational work is 2mn min(n, m) − 23 min(n, m)3 for calculating R (while Q is defined implicitly as a product of Householder matrices), and 43 n3 for forming Q explicitly as a matrix (cf. [115, §5.2.1]).  0 (b) If n > m, the matrix R has the block structure R0 containing the submatrix R0 as an upper triangular matrix of the size m × m. The corresponding block decomposition Q = [ Q0 Q00 ] yields the reduced QR factorisation M = Q0 R0

(Q0 ∈ Kn×m , R0 ∈ Km×m ).

(2.14b)

The computational work is10 9

A modified Gram–Schmidt algorithm was already derived by Laplace in 1816 (see reference in Bj¨orck [36, p. 61] together with additional remarks concerning history). 10 Half of the cost of NQR (n, m) is needed for 12 (m2 + m) scalar products. The rest is used for scaling and adding column vectors.

2 Matrix Tools

32

NQR (n, m) := 2nm2

(cf. [115, Alg. 5.2.5], [247, §3.4]).

(c) If r := rank(M ) < min{n, m}, the sizes of Q0 and R0 can be further reduced: M = Q0 R0

(Q0 ∈ Kn×r , R0 ∈ Kr×m ).

(2.14c)

In particular, if M does not possess full rank as in Part (c) of the lemma above, we want R0 in (2.14c) to be of the form R0 = [R10 R20 ] ,

R10 ∈ Kr×r upper triangular,

rank(R10 ) = r,

(2.14d)

i.e., the diagonal entries of R10 do not vanish. This form of R0 can be achieved if and only if the part (Mij )1≤i,j≤r of M has also rank r. Otherwise we need a suitable permutation M 7→ MP of the columns of M . Then the factorisation takes the form M P = Q0 [R10 R20 ]

(P permutation matrix, Q0 , R10 , R20 in (2.14c,d)).

(2.15)

An obvious pivot strategy for a matrix M ∈ Kn×m with r = rank(M ) is the Gram–Schmidt orthogonalisation in the following form (cf. [115, §5.4.1]). (1) Let mi ∈ Kn (1 ≤ i ≤ m) be the i-th columns of M . (2) For i := 1 to r do (2a) Choose k ∈ {i, . . . , m} such that kmk k = max{kmν k : i ≤ ν ≤ m}. If k 6= i, interchange the columns mi and mk . (2b) Now mi has maximal norm. Normalise: mi := mi / kmi k. Store mi as i-th column of the matrix Q. (2c) Set mk := mk − hmk , mi i mi for i + 1 ≤ k ≤ m. Here k·k is the Euclidean norm and h·, ·i the corresponding scalar product. The column exchanges in Step (2a) lead to the permutation matrix11 P in (2.15). The operations in Step (2b) and Step (2c) define [R10 R20 ]. The presupposition r = rank(M ) guarantees that all mi appearing in Step (2b) do not vanish, while mi = 0 (r + 1 ≤ i ≤ m) holds after the r-th iteration for the remaining columns. In usual applications, the rank is unknown. In that case we may introduce a tolerance τ > 0 and redefine Step (2b) as follows: (2b’) If kmi k ≤ τ set r := i − 1 and terminate. Otherwise proceed as in Step (2b). The principle of the QR decomposition can be generalised to tuples V m , where the column vectors from Kn are replaced with functions from the space V (cf. Trefethen [283]). A permutation matrix P ∈ Kr×r (corresponding to a permutation π : {1, . . . , r} → {1, . . . , r}) is defined by (P v)i = vπ(i) . Any permutation matrix P is unitary. 11

2.5 Matrix Decompositions

33

2.5.3 Singular-Value Decomposition 2.5.3.1 Definition and Computational Cost The singular value decomposition (abbreviation: SVD) is the generalisation of the diagonalisation of square matrices (cf. [247, §1.9]). Lemma 2.23 (SVD). (a) Let M ∈ Kn×m be any matrix. Then there are unitary matrices U ∈ Kn×n , V ∈ Km×m , and a diagonal rectangular matrix Σ ∈ Rn×m , 

σ1 0 . . .   0 σ2 . . . Σ= . . .  .. . . . . 0 ... 0

0 0 ... 0 0 .. .

0 .. .



  0 ..  .





illustration  for the case  , of n ≤ m

(2.16a)

σn 0 . . . 0

with so-called singular values12 σ1 ≥ σ2 ≥ . . . ≥ σi = Σii ≥ . . . ≥ 0

(1 ≤ i ≤ min{n, m})

such that 13 M = U Σ V T.

(2.16b)

The columns of U are the left singular vectors, the columns of V are the right singular vectors. (b) The spectral norm of M has the value kM k2 = σ1 . (c) The Frobenius norm of M is equal to v u umin{n,m} X kM kF = t σi2 .

(2.16c)

i=1

Proof. Assume without loss of generality that n ≤ m and set A := M M H ∈ Kn×n . Diagonalise the positive-semidefinite matrix, A = UDU H with U ∈ Kn×n unitary, n×n where the (nonnegative) eigenvalues in D = diag{d are ordered 1 , . . ., dn } ∈ R √ by size: d1 ≥ d2 ≥ . . . ≥ 0. Defining σi := di in (2.16a), we rewrite D = ΣΣ T = ΣΣ H . With W := M H U = [w1 , . . ., wn ] ∈ Km×n we have 12

For indices ` > min{#I, #J} we formally define σ` := 0. The usual formulation is M = U Σ V H with the Hermitean transposed V H . Here we use V T also for K = C because of Remark 1.3a. 13

2 Matrix Tools

34

D = U HAU = U H M M H U = W H W. Hence the columns wi of W are pairwise orthogonal and wiH wi = di = σi2 . Next, we are looking for a unitary matrix V = [v1 , . . ., vm ] ∈ Km×m with W = V ΣT,

i.e., wi = σi vi

(1 ≤ i ≤ m)

(note that the complex conjugate values vi are used13 ). Let r := max{i : σi > 0}. For 1 ≤ i ≤ r, the condition above leads to vi := σ1i wi ; i.e., vi is normalised: viH vi = 1. Since the vectors wi of W are already pairwise orthogonal, the vectors {vi : 1 ≤ i ≤ r} are orthonormal. For r + 1 ≤ i ≤ n, σi = 0 implies wi = 0. Hence wi = σi vi holds for any choice for vi . To obtain a unitary matrix V , we may choose any orthonormal extension {vi : r + 1 ≤ i ≤ m} of {vi : 1 ≤ i ≤ r}. The relation W = V Σ T (with Σ T = Σ H ) implies W H = ΣV T . By the definition of W we have M = U W H = U Σ V T so that (2.16b) is proved. Exercises 2.10b and 2.12a imply that kM k2 = kΣk2 and kM kF = kΣkF , proving the parts (b) and (c). t u If n < m, the last m − n columns of V are multiplied by the zero part of Σ. Similarly, for n > m, certain columns of U are not involved in the representation of M . Reduction to the first min{n, m} columns yields the following result. Corollary 2.24. (a) Let ui ∈ KI and vi ∈ KJ be the (orthonormal) columns of U and V , respectively. Then the statement M = U Σ V T in (2.16b) is equivalent to min{n,m} X M= σi ui viT . (2.17) i=1

The computational cost is about  NSVD (n, m) := min 14nmN + 8N 3 , 6nmN + 20N 3 , where N := min{n, m} (cf. [115, §5.4.5]). (b) The decomposition (2.17) is not unique. Let σi = σi+1 = . . . = σi+k−1 Pi+k−1 (k ≥ 2) be a k-fold singular value. The part j=i σj uj vjT in (2.17) is equal to T

σi [ui , . . . , ui+k−1 ] [vi , . . . , vi+k−1 ] . For any unitary k × k matrix Q, the transformed vectors [ˆ ui , . . . , u ˆi+k−1 ] := [ui , . . . , ui+k−1 ] Q and [ˆ vi , . . . , vˆi+k−1 ] := [vi , . . . , vi+k−1 ] Q

2.5 Matrix Decompositions

35

Pi+k−1 ˆj vˆjT . Even in the case k = 1 of a simple singular yield the same sum j=i σj u ˆi := zui , vˆi := z1 vi with value, each pair ui , vi of columns may be changed into u z ∈ K and |z| = 1. (c) Often the subspace span{ui : 1 ≤ i ≤ r} for some r ≤ min{n, m} is of interest. This space is uniquely determined if and only if σr < σr+1 . The same statement holds for span{vi : 1 ≤ i ≤ r}. Remark 2.25. The orthonormality of {ui } and {vi } corresponds to the usual Euclidean scalar product hx, yi = y H x. If KIi (i = 1, 2) are endowed with other scalar products h·, ·ii , there are positive definite matrices Wi such that hx, yii = (Wi y)H (Wi x). Given a matrix M ∈ KI1 ×I2 , perform the usual SVD P ˆ := W1 M W2 = P σν u of M ˆν vˆνT . Then M = ν σν uν vνT holds with h·, ·i1 ν orthonormal vectors uν := W1−1 u ˆν and h·, ·i2 -orthonormal vectors vν := W2−1 vˆν . Next, we consider a convergent sequence M (ν) → M of matrices together with the singular-value decompositions M (ν) = U (ν) Σ (ν) V (ν)T and M = UΣ V T . Exercise 2.26. Prove the following. (a) Let M (ν) = U (ν) Σ (ν) V (ν)T ∈ Kn×m be the singular-value decompositions of M (ν) → M . Then there is a subsequence {νi : i ∈ N} ⊂ N such that U (νi ) → U,

Σ (νi ) → Σ,

V (νi ) → V,

(ν)

M = UΣ V T .

(ν)

(b) Subsequences of the spaces Sr := span{ui : 1 ≤ i ≤ r} converge to (ν) Sr := span{ui : 1 ≤ i ≤ r}, where ui and ui are the columns of U (ν) and U from Part (a).

2.5.3.2 Reduced and One-Sided Singular-Value Decompositions If M is not of full rank, there are singular values σi = 0 so that further terms can be omitted from the sum in (2.17). Let14 r := max{i : σi > 0} = rank(M ) as in the proof above. Then (2.17) can be rewritten as M=

r X

σi ui viT

i=1

 with

r

r

{ui }i=1 , {vi }i=1 orthonormal, σ1 ≥ . . . ≥ σr > 0,

(2.18)

where only nonzero terms appear. The corresponding matrix formulation is

M = U 0 Σ 0 V 0T

14

 0 n×r orthogonal,   U = [u1 , . . . , ur ] ∈ K 0 m×r orthogonal, with V = [v1 , . . . , vr ] ∈ K (2.19)   0 r×r Σ = diag{σ1 , . . . , σr } ∈ R , σ1 ≥ . . . ≥ σr > 0.

If σi = 0 for all i, set r := 0 (empty sum). This happens for M = 0.

2 Matrix Tools

36

Definition 2.27 (reduced SVD). The identities (2.18) or (2.19) are called the reduced singular-value decomposition (since the matrices U , Σ, V in (2.16b) are reduced to the essential nonzero part). There are cases—in particular when m  n—in which one is only interested in the left singular vectors ui and the singular values σi in (2.18) or equivalently only in U 0 and Σ 0 in (2.19). Then we say that we need the left-sided singular-value decomposition. The proof of Lemma 2.23 has already shown how to solve for U 0 and Σ 0 only: (1) Set A := M M H ∈ Kn×n . (2) Diagonalise A = U D U H with the nonnegative diagonal matrix D = diag{d1 , . . . , dn } ∈ Rn×n ,

d1 ≥ d2 ≥ . . . ≥ 0.

(3) Set r := max{i : di > 0}, σi :=

p di , and Σ 0 := diag{σ1 , . . . , σr }.

(4) Restrict U to the first r columns: U 0 = [u1 , . . . , ur ]. Remark 2.28. (a) Steps (1–4) from above define the matrices U 0 and Σ 0 in (2.19). −1 The third matrix V 0 is theoretically available via V 0 = (Σ 0 ) M H U 0 . The product n(n+1) M M H in Step 1 requires computing scalar products hmi , mj i (i, j ∈ I), 2 involving the rows mi := M [i, •] ∈ KJ of M . The computational cost for these scalar products will crucially depend on the underlying data structure (cf. Remark 7.16). Steps (2–4) are independent of the size of J. Their cost is asymptotically 8 3 3 n (cf. Golub–Van Loan [115, §8.3.1]). ˆ := U 0H M . M ˆ has orthogonal rows (b) The knowledge of U 0 suffices to define M m ˆ i (1 ≤ i ≤ n) which are ordered by size: km ˆ 1 k = σ1 > km ˆ 2 k = σ2 > . . . > 0. Proof. Let M = U 0 Σ 0 V 0T be the reduced singular-value decomposition. Since ˆ := U 0H M = Σ 0 V 0T . It follows that U 0H U 0 = I ∈ K
r×r , Part (b) defines M H 0 0T 0 0 2 0 ˆM ˆ = ΣV M (V Σ ) = Σ , i.e., hm ˆ i, m ˆ j i = 0 for i 6= j and km ˆ i k = σi . t u The analogously defined right-sided singular-value decomposition of M is identical to the left-sided singular-value decomposition of the transposed matrix M T since M = U 0 Σ 0 V 0T ⇐⇒ M T = V 0 Σ 0 U 0T .

2.5.3.3 Inequalities of Singular Values Finally, we discuss estimates about eigenvalues and singular values of perturbed matrices. The following lemma states the characterisation of eigenvalues by Fischer [102] and Courant [65]. For a general matrix A ∈ Kn×n we denote its eigenvalues corresponding to their multiplicity by λk (A). If λk (A) ∈ R, we order the eigenvalues such that λk (A) ≥ λk+1 (A). Formally, we set λk (A) := 0 for k > n.

2.5 Matrix Decompositions

37

Remark 2.29. For matrices A ∈ Kn×m and B ∈ Km×n the identity λk (AB) = λk (BA) is valid. If A and B are positive semidefinite, the eigenvalues λk (AB) are nonnegative. Proof. (i) If e 6= 0 is an eigenvector of AB with nonzero eigenvalue λ, the vector Be does not vanish. Then (BA)(Be) = B(AB)e = B(λe) = λ(Be) proves that Be is an eigenvector of BA for the same λ. Hence AB and BA share the same nonzero eigenvalues, while λk (AB) = λk (BA) = 0 for k > rank(AB). (ii) If B ≥ 0, the square root B 1/2 is defined (cf. Remark 2.14). Part (i) shows that λk (AB) = λk (AB 1/2 B 1/2 ) = λk (B 1/2 AB 1/2 ). The latter matrix is positive semidefinite, proving λk (B 1/2 AB 1/2 ) ≥ 0. u t Lemma 2.30. Let the matrix A ∈ Kn×n be positive semidefinite. Then the eigenvalues λ1 (A) ≥ . . . ≥ λn (A) ≥ 0 can be characterised by λk (A) =

min V ⊂ Kn subspace with dim(V) ≤ k − 1

xH Ax. max n x ∈ K with xH x = 1 and x⊥V

(2.20)

Proof. A can be diagonalised: A = UΛU H with Λii = λi . Since xH Ax = y H Λy for y = U H x, the assertion can also be stated in the form λk =

min

W with dim(W)≤k−1

 max y H Λy : y ∈ Kn with y H y = 1, y⊥W

(W = U H V). Fix W with dim(W) ≤ k−1 . All y ∈ Kn with yi = 0 for i > k form a k-dimensional subspace Y. Since dim(W) ≤ k − 1, there is at least one vector in k k P P λi yi2 ≥ λk yi2 ≥ λk . 0 6= y ∈ Y with y H y = 1, y⊥W. Obviously, y H Λy = i=1

i=1

The choice W = {w ∈ Kn : wi = 0 : k ≤ i ≤ n} yields equality: y H Λy = λk . t u We use the notation λk (A) for the k-th eigenvalue of a positive-semidefinite matrix A, where the eigenvalues are ordered by size (see Lemma 2.30). Similarly, σk (A) denotes the k-th singular value of a general matrix A. Note that k·k2 is the spectral norm in (2.11). Lemma 2.31. (a) Let A, B ∈ Kn×n be two positive-semidefinite matrices. Then λk (A) ≤ λk (A + B) ≤ λk (A) + kBk2

for 1 ≤ k ≤ n.

In particular, 0 ≤ A ≤ B implies λk (A) ≤ λk (B) for 1 ≤ k ≤ n.

(2.21)

2 Matrix Tools

38 0

(b) Let the matrices A ∈ Kn×m and B ∈ Kn×m satisfy AAH ≤ BB H . Then the singular values15 σk (A) and σk (B) of both matrices are related by for 1 ≤ k ≤ n.

σk (A) ≤ σk (B)

0

The same statement holds for A ∈ Km×n and B ∈ Km ×n with AH A ≤ B H B. 0

0

(c) Let M ∈ Kn×m be any matrix, while A ∈ Kn ×n and B ∈ Km×m have to satisfy AH A ≤ I and B H B ≤ I. Then15 σk (AM B) ≤ σk (M )

for k ∈ N.

Proof. (i) λk (A) ≤ λk (A + B) is a consequence of Remark 2.15 and Lemma 2.30. (ii) Let VA and VA+B be the subspaces which are the minimisers in (2.20) for A and A + B, respectively. Abbreviate the maximum in (2.20) over x ∈ Kn with xH x = 1 and x⊥V by max. Then V

  λk (A + B) = max xH (A + B)x ≤ max xH (A + B)x = max xH Ax + xH Bx VA+B

VA

VA

H

H

≤ max x Ax + max x Bx = λk (A) + kBk2 . VA

xH x=1

(iii) For Part (b) use λk (A) ≤ λk (A + B) with A and B replaced by AAH and BB H − AAH in the case of AAH ≤ BB H . Otherwise use the fact that the eigenvalues of X H X and XX H coincide (cf. Remark 2.29). (iv) Let M 0 := AM B and use σk (M 0 )2 = λk (M 0H M 0 ). Remark 2.15b implies that M 0 (M 0 )H = AM BB H M H AH ≤ AM M H AH , so that λk (M 0H M 0 ) ≤ λk (AM M H AH ). Remark 2.29 states that λk (AM M H AH ) = λk (M H AH AM ), u and from AH A ≤ I we infer that λk (M H AH AM ) ≤ λk (M H M ) = σk (M )2 . t Let n = n1 + n2 , A ∈ Kn1 ×m and B ∈ Kn2 ×m . Then the agglomerated matrix A A n×m . . In the next lemma we compare singular values of A and B B belongs to K Lemma 2.32. For general A ∈ Kn1 ×m and B ∈ Kn2 ×m , the singular values satisfy q A 2 ) ≤ σk2 (A) + kBk2 . σk (A) ≤ σk ( B The same estimate holds for σk ([A B]) containing A ∈ Kn×m1 and B ∈ Kn×m2 . A HA Proof. Use σk2 (A) = λk (AH A) and σk2 ( B ) = λk ([A B] B ) = λk (AH A + B H B)   2 A and apply (2.21): σk2 ( B ) ≤ λk (AH A) + kB H Bk2 = σk2 (A) + kBk2 . t u  Exercise 2.33. Prove σk (A) ≤ σk 15

See Footnote 12 on page 33.

A C B

 ≤

q 2 2 σk2 (A) + kBk2 + kCk2 .

2.6 Low-Rank Approximation

39

2.6 Low-Rank Approximation Given a matrix M , we ask for a matrix R ∈ Rs of lower rank (i.e., s < rank(M )) such that kM −Rk is minimised. The answer is given by16 Erhard Schmidt (1907) [257, §18]. In his paper, he studies the infinite singular-value decomposition for operators (cf. Theorem 4.137). The following finite case is a particular application. Lemma 2.34. (a) Let M, R ∈ Kn×m with r := rank(R). The singular values of M and M − R are denoted by σi (M ) and σi (M − R), respectively. Then17 (2.22) for all 1 ≤ i ≤ min{n, m}. σi (M − R) ≥ σr+i (M )  (b) Let s ∈ 0, 1, . . . , min{n, m} . Use the singular-value decomposition M = UΣ V T to define  σi for i = j ≤ s, T (2.23a) with (Σs )ij = R := U Σs V 0 otherwise, i.e., Σs results from Σ by replacing all singular values σi = Σii for i > s with zero. Then the approximation error is v u umin{n,m} X kM − RkF = t kM − Rk2 = σs+1 and σi2 . (2.23b) i=s+1

Inequalities (2.22) becomes σi (M − R) = σs+i (M ). Proof. (i) If r + i > min{n, m}, (2.22) holds because σr+i (M ) = 0. Therefore suppose r + i ≤ min{n, m}. 2 (ii) First, σi (M − R) is investigated for i = 1. λr+1 (M M H ) := σr+1 (M ) is H the (r + 1)-th eigenvalue of A := M M (see proof of Lemma 2.23). The minimisation in (2.20) yields  2 σr+1 (M ) ≤ max xH Ax : x ∈ Kn with xH x = 1, x⊥V for any fixed subspace V of dimension ≤ r. Choose V := ker(RH )⊥ . As x⊥V is equivalent to x ∈ ker(RH ), we conclude that xH Ax = xH M M H x = M H x


H


H
H H
M H x = (M − R) x (M − R) x

H

= xH (M − R) (M − R) x. H

Application of (2.20) to the first eigenvalue λ1 = λ1 ((M − R) (M − R) ) of the H matrix (M − R) (M − R) shows that 16

Occasionally, this result is (incorrectly) attributed to Eckart–Young [84] (1936). Compare also the historical comments about SVD by Stewart [275]. 17 See Footnote 12 on page 33.

2 Matrix Tools

40

 max xH Ax : x ∈ Kn with xH x = 1, x⊥V H

= max{xH (M − R) (M − R) x : xH x = 1, x⊥V} H

≤ max{xH (M − R) (M − R) x : x ∈ KI with xH x = 1} H
= λ1 (M − R) (M − R) (in the case of the first eigenvalue, the requirement x⊥V
with dim(V) = 0 is an H empty condition). Since again λ1 (M − R) (M − R) = σ12 (M − R), we have 2 2 proved σr+1 (M ) ≤ σ1 (M − R), which is statement (a) for i = 1. (iii) For i > 1 choose V := ker(RH )⊥ + W, where W with dim(W) ≤ i − 1 is arbitrary. Analogous to part (ii), we obtain the bound o n H max xH (M − R) (M − R) x : x ∈ Kn with xH x = 1, x⊥W . H

Minimisation over all W yields λi (M − R) (M − R)




= σi2 (M − R).

(iv) The choice in (2.23a) eliminates the singular values σ1 , . . . , σs so that σi (M − R) = σs+i (M ) for all i ≥ 1. t u Ps Pr T Using the notation M = i=1 σi ui viT in (2.18), we write R as i=1 σi ui vi . A connection with projections is given next. Ps Ps (s) (s) := i=1 vi viH are the orthogonal Remark 2.35. P1 := i=1 ui uH i and P2 projections onto span{ui : 1 ≤ i ≤ s} and span{vi : 1 ≤ i ≤ s}, respectively. (s) (s) (s) (s) Then R in (2.23a) can be written as R = P1 M (P2 )T = P1 M = M (P2 )T . Conclusion 2.36 (best rank-k approximation). For M ∈ Kn×m construct R as in (2.23a). Then R is the solution of the following two minimisation problems: min rank(R)≤r

kM − Rk2

and

min rank(R)≤r

kM − RkF .

(2.24)

The values of the minima are given in (2.23b). The minimising element R is unique if and only if σr > σr+1 . P 2 Proof. (i) Since kM − R0 k2 = σ1 (M − R0 ) and kM − R0 kF = i>0 σi2 (M − R0 ) follows from Lemma 2.23b,c, we obtain from Lemma 2.34a that 2

kM − R0 k2 ≥ σr+1 (M ), kM − R0 kF ≥

X

σi2 (M ) for R0 with rank(R0 ) ≤ r.

i>r

Since equality holds for R0 = R, this is the solution of the minimisation problems. (ii) If σk = σk+1 , we may interchange the r-th and (r + 1)-th columns in U u and V obtaining another singular-value decomposition. Thus, another R results. t Next, we consider a convergent sequence M (ν) and use Exercise 2.26.

41

2.7 Linear Algebra Procedures

Lemma 2.37. Consider M (ν) ∈ Kn×m with M (ν) → M . Then there are best approximations R(ν) according to (2.24) so that a subsequence of R(ν) converges to R , which is the best approximation to M . Remark 2.38. The optimisation problems (2.24) can also be interpreted as the best approximation of the range of M : max {kP M kF : P orthogonal projection with rank(P ) = r} .

(2.25)

Proof. The best approximation R ∈ Rr to M has the representation R = P M (r) for P = P1 (cf. Remark 2.35). By orthogonality, 2

2

2

kP M kF + k(I − P )M kF = kM kF holds. Hence minimisation of 2

2

k(I − P )M kF = kM − RkF 2

is equivalent to maximising kP M kF .

u t

2.7 Linear Algebra Procedures For later use, we formulate procedures based on the previous techniques. The reduced QR decomposition is characterised by the dimensions n and m, the input matrix M ∈ Kn×m , the rank r, and resulting factors Q and R. The corresponding procedure is denoted by {reduced QR decomposition} procedure RQR(n, m, r, M, Q, R); input: M ∈ Kn×m ; output: r = rank(M ), Q ∈ Kn×r orthogonal, R ∈ Kr×m upper triangular.

(2.26)

and requires NQR (n, m) operations (cf. Lemma 2.22). The modified QR decomposition in (2.15) produces an additional permutation matrix P and the decomposition of R into [R1 R2 ]: {pivotised QR decomposition} procedure PQR(n, m, r, M, P, Q, R1 , R2 ); input: M ∈ Kn×m ; (2.27) output: Q ∈ Kn×r orthogonal, P ∈ Km×m permutation matrix, r×(m−r) r×r upper triangular with r = rank(M ), R2 ∈ K R1 ∈ K . A modified version of PQR will be presented in (2.37).

2 Matrix Tools

42

The (two-sided) reduced singular-value decomposition from Definition 2.27 leads to procedure RSVD(n, m, r, M, U, Σ, V ); {reduced SVD} input: M ∈ Kn×m ; output: U ∈ Kn×r , V ∈ Km×r orthogonal with r = rank(M ), Σ = diag{σ1 , . . . , σr } ∈ Rr×r with σ1 ≥ . . . ≥ σr > 0.

(2.28)

Here the integers n, m may also be replaced with index sets I and J. For the cost NSVD (n, m), see Corollary 2.24a. The left-sided reduced singular-value decomposition (cf. Remark 2.28) is denoted by procedure LSVD(n, m, r, M, U, Σ); input: M ∈ Kn×m ; output: U, r, Σ as in (2.28).

{left-sided reduced SVD} (2.29)

Its cost is NLSVD (n, m) :=

1 8 n (n + 1) Nm + n3 , 2 3

where Nm is the cost of the scalar product of rows of M . In general, Nm = 2m − 1 holds, but it may be smaller for structured matrices (cf. Remark 7.16). In the procedures above, M is a general matrix from Kn×m . Matrices M ∈ Rr (cf. (2.6)) may be given in the form

M=

r X r X ν=1 µ=1

H cνµ aν bH µ = ACB

aν ∈ Kn , A = [a1 a2 · · · ] ∈ Kn×r , bν ∈ Km , B = [b1 b2 · · · ] ∈ Km×r

! . (2.30)

Then the following approach has a cost proportional to n + m if r  n, m (cf. [138, Alg. 2.17]), but also for r  n, m it is cheaper than the direct computation18 of the product M = A CB H followed by a singular-value decomposition. Remark 2.39. For M = A CB H in (2.30) compute the reduced QR decompositions19  QA ∈ Kn×rA , rA := rank(A), A = QA RA and B = QB RB with QB ∈ Kn×rB , rB := rank(B), 18 19

For instance, the direct computation is cheaper if n = m = r = rA = rB . Possibly, permutations according to (2.27) are necessary.

43

2.7 Linear Algebra Procedures

H followed by the computation of the singular-value decomposition RA CRB = H H ˆ ˆ UΣ V . Then the singular-value decomposition of M is given by UΣ V with ˆ and V = QB Vˆ . The cost of this calculation is U = QA U
2 2 n + rB m rr2 + 2 rA NQR (n, r) + NQR (m, r) + NLSVD (rA , rB ) + 2rA rB r + 2¯

with r¯ := min{rA , rB } ≤ min{n, m, r}. In the symmetric case of A = B and n = m with r¯ := rank(A), the cost reduces to
NQR (n, r) + 2¯ rr2 + r¯2 r + 2n + 38 r¯ . 0

00

Let B 0 = [b01 , . . . , b0r0 ] ∈ Kn×r and B 00 = [b001 , . . . , b00r00 ] ∈ Kn×r contain two systems of vectors. Often, B 0 and B 00 correspond to bases of subspaces U 0 ⊂ V and U 00 ⊂ V . A basic task is the construction of a basis20 B = [b1 , . . . , br ] of 0 00 U := U 0 + U 00 . Furthermore, the matrices T 0 ∈ Kr×r and T 00 ∈ Kr×r with B 0 = BT 0 and B 00 = BT 00 , i.e., b0j =

r X

Tij0 bi , b00j =

i=1

r X

Tij00 bi ,

(2.31)

i=1

are of interest. The corresponding procedure is procedure JoinBases(B 0 , B 00 , r, B, T 0 , T 00 ); {joined bases} 0 n×r 0 00 n×r 00 input: B ∈ K , B ∈K , output: r = rank[B 0 B 00 ]; B basis of range([B 0 B 00 ]), 0

(2.32)

00

T 0 ∈ Kr×r and T 00 ∈ Kr×r with (2.31). 0

00

A possible realisation starts from B = [b01 ,. . ., b0r0 , b001 ,. . ., b00r00 ] ∈ Kn×(r +r ) and performs the reduced QR factorisation B = QR by RQR(n,r0 + r00,r,B,P,Q,R1,R2). Then the columns of P T Q form the basis B, while [T 0 , T 00 ] = R := [R1, R2]. If B 0 is a basis, it may be advantageous to let the basis vectors bi = b0i from B0  I 0 0 unchanged, whereas for i > r , bi is the i-th column of Q. Then T = 0 holds, while T 00 is as before. If all bases B 0 , B 00 , B are orthonormal, the second variant from above completes the system B 0 to an orthonormal basis B: procedure JoinONB(b0 , b00 , r, b, T 0 , T 00 ); {joined orthonormal basis} 0 00 input: B 0 ∈ Kn×r , B 00 ∈ Kn×r orthonormal bases, (2.33) output: B orthonormal basis of range([B 0 B 00 ]); r, T 0 , T 00 as in (2.32). The cost of both procedures is NQR (n, r0 + r00 ).

20

We call B = [b1 , . . . , br ] a basis, meaning that the set {b1 , . . . , br } is the basis.

2 Matrix Tools

44

2.8 Dominant Columns Again, we consider the minimisation problem (2.24): minrank(M 0 )≤k kM − M 0 k for M ∈ Kn×m and k·k = k·k2 or k·k = k·kF . Without loss of generality, we assume that the minimising matrix Mk satisfies rank(Mk ) = k; otherwise replace k with min0 0 kM − M 0 k. min0 kM − M 0 k = k 0 := rank(Mk ) and note that rank(M )≤k

rank(M )≤k

The minimising matrix Mk ∈ Rk is of the form Mk = AB T ,

where A ∈ Kn×k and B ∈ Km×k

and range(Mk ) = range(A). The singular-value decomposition M = UΣV T yields the matrices A = U 0 Σ 0 and B = V 0 , where the matrices U 0 , Σ 0 , V 0 consist of the first k columns of U, Σ, V . Since A = U 0 Σ 0 = M V 0T , the columns of A are linear combinations of all columns of M . The latter fact is a disadvantage is some cases. For a concrete numerical approach we have to represent A. If the columns mj of M are represented as full vectors from Kn , a linear combination is of the same kind and leads to no difficulty. This can be different if other representations are involved. To give an example of an extreme case, replace Kn×m = (Kn )m with X m , where X is a subspace of, say, L2 ([0, 1]). Let the columns be functions as xν or exp(αx). Such functions can be simply coded together with procedures for pointwise evaluation and mutual scalar products. However, linear combinations cannot be simplified. For instance, scalar products of linear combinations must be written as double sums of elementary scalar products. As a result, the singular-value decomposition reduces the rank of M to k, but the related computational cost may be larger than before. This leads to a new question. Can we find AB T ∈ Rk approximating M such that A = [cj1 · · · cjk ] consists of k (different) columns of M ? In this case, AB T involves only k columns of M , instead of all. For this purpose, we define Rk (M ) := {AB T ∈ Rk : A = [M [·, j1 ], · · · , M [·, jk ]] with 1 ≤ jκ ≤ m}, (2.34) using the notations from above. The minimisation (2.24) is now replaced with find Mk ∈ Rk (M ) with

kM − Mk k =

min

M 0 ∈Rk (M )

kM − M 0 k .

(2.35)


Since there are m k different combinations of columns, we do not try to solve this combinatorial problem exactly. Instead, we are looking for an approximate solution. By procedure PQR in (2.27), we obtain the QR decomposition M P = QR with R = [R1 R2 ]. First we discuss the case r := rank(M ) = m, which is equivalent to M possessing full rank. Then R2 does not exist (m − r = 0 columns) and R := R1 is a square upper triangular matrix with nonvanishing diagonal entries. Thanks to the pivoting strategy, the columns of R are of decreasing Euclidean norm. Let k ∈ {1, . . . , m − 1} be the desired rank from problem (2.35). We split the matrices into the following blocks:

2.8 Dominant Columns

R=

R0 S 0 R00

Q = Q0 Q00

45

 with

R0 ∈ Kk×k , R00 ∈ K(m−k)×(m−k) upper triangular, S ∈ Kk×(m−k) ,

with Q0 ∈ Kn×k , Q00 ∈ Kn×(m−k) .

Then Q0 R0 corresponds to the first k columns of M P . As P is a permutation matrix, these columns form the matrix A as required in (2.34). The approximating matrix is defined by MkPQR := Q0 [R0 S] with [R0 S] ∈ Kk×m ; i.e., the matrix B T in (2.34) is B = [R0 S] . Proposition 2.40. The matrix MkPQR := Q0 [R0 S] constructed above belongs to Rk (M ) and satisfies the following estimates: kM − MkPQR k2 ≤ kR00 k2 ,

kM − MkPQR kF ≤ kR00 kF , q 2 σk (MkPQR ) ≤ σk (M ) ≤ σk2 (MkPQR ) + kR00 k2 .

Proof. (i) By construction, M − MkPQR = Q (ii) (2.36b) follows from Lemma 2.32.

0

0 0 R00



(2.36a) (2.36b)

holds and leads to (2.36a). t u

Now we investigate the case r < m. Then the full QR decomposition would lead  0 to QR with R = R0 with zeros in the rows r +1 to m. These zero rows are omitted by the reduced QR decomposition. The remaining part R0 (again denoted by R) is of the shape R = [R1 R2 ], where R1 has upper triangular form. As rank(M ) = r, the approximation rank k in (2.35) should vary in 1 ≤ k < r. The columns of R1 are again decreasing, but the first k columns may not be chosen optimally. This is illustrated by the following example.   Let M = 20 11 11 11 with r = 2 < m = 4. Since the first column has largest norm,  1 0 procedure PQR produces P = I (no permutations) and Q = , R1 =     0 1 2 1 1 1 , R2 = . Let k = 1. Choosing the first column of Q and first 0 1 1 1   [1] row of R, we obtain the matrix M1 := 20 10 10 10 . The approximation error is

  √ [1] ε1 := kM − M1 k = 00 01 01 01 = 3.  Note that we cannot choose the second column 01 of Q and the second row of R 0 instead since 1 is not a column of M ; i.e., the resulting approximation does not belong to R1 (M ). A remedy is to change the pivot strategy in Step 2a from page 32. We choose the second column of M as first column of Q. For this purpose, let P be the permutation matrix corresponding to 1 ↔ 2. The QR decomposition applied   to M P = 11 20 11 11 yields       √ 1 1 √ 1 1 1 1 1 Q= √ , R1 = 2 , R2 = 2 . 0 1 0 0 2 1 −1 [2]

The first column of Q and the first row of R result in M1 P with the smaller approximation error

2 Matrix Tools

46

  √

1000 √ [2]

= 2 < 3.

ε2 := kM − M1 k = −1 0 0 0   The reason for ε2 < ε1 is obvious: although 11 is of smaller norm than 20 , it has a higher weight because it appears in three columns of M . To take this weight into consideration, we need another pivot strategy. Let M = [c1 · · · cm ] ∈ Kn×m . Each column cj 6= 0 is associated with the −2 orthogonal projection Pj := kcj k cj cH j onto span{cj }. We call ci a dominant column if kPi M k = max kPj M k . 1≤j≤m

Equivalently, kM − Pj M k is minimal for j = i. Let P be the permutation matrix corresponding to the exchange 1 ↔ i. Then M P = QR leads to Q with ci / kci k −1 as first column. The first row of R is r1H := kci k cH i M P . Hence the choice for the dominant column ensures that the approximation (2.35) with k = 1 is given by −1 kci k ci r1H P H = Pi M . The calculation of a dominant column is discussed in the next lemma. Lemma 2.41. For M = [c1 · · · cm ] set with ζjk := hck , cj i / kcj k .

Z = (ζjk )1≤j,k≤m

Then the index imax ∈ {1, . . . , m} with kζimax ,• k = max1≤j≤m kζj,• k characterises the dominant column. Proof. Because Pj M = kcj k its norm is kPj M k =

−2

pP

k

cj cH j ck 1≤k≤m


 =

ζjk cj kcj k

 , 1≤k≤m

|ζjk |2 = kζj,• k .

t u

The concept of the dominant column leads to the following variant of PQR: procedure DCQR(n, m, r, M, P, Q, R1 , R2 ); {pivot by dominant column} input: M ∈ Kn×m ; output: Q ∈ Kn×r orthogonal, P ∈ Km×m permutation matrix, R1 ∈ Kr×r upper triangular with r = rank(M ), R2 ∈ Kr×(m−r) . for j := 1 to r do begin determine i ∈ {j, . . . , m} such that ci := M [•, i] (2.37) is the dominant column of M [•, j : m]; permute j ↔ i (change of P ) Q[j, •] := ci ; M [•, i] := M [•, j]; −2 M [•, j + 1 : m] := (I − Pi ) M [•, j + 1 : m] (Pi := kci k ci cH i ) end; {determination of R1 , R2 as usual; here omitted}

2.8 Dominant Columns

47

Corollary 2.42. In practical applications it suffices to determine ˆιmax , instead of imax , where kζˆιmax ,• k is sufficiently close to kζimax ,• k. For this purpose, order the columns by decreasing norm: M P =: [c1 · · · cm ]

with kcj k ≥ kcj+1 k . Pm 2 Choose m0 ∈ {1, . . . , m − 1} with ε2 := k=m0 +1 kck k sufficiently small to the first m0 columns: ˆιmax is the maximiser of and reduce the pPmaximisation m0 2 . Then the two maxima are related by max1≤j≤m0 |ζ | jk k=1 q 2 kζˆιmax ,• k ≤ kζimax ,• k ≤ kζˆιmax ,• k + ε2 . To estimate the cost, we start with the determination of the matrix Z from Lemma 2.41. The 12 m(m + 1) scalar products are to be computed only once. After one elimination step (ck 7→ Pi ck ) they can be updated21 without new scalar products (cost: 2(m−1)m). The application of Pi to M does also not require scalar products since they are precomputed. The cost of the original procedure DCQR is   NDCQR = 4 mr − 12 r2 n + m2 n + 2rm(m − r) plus lower order terms. If m is immediately reduced to m0 , the cost is reduced correspondingly. Remark 2.43. Assume that n = n ˆ p with n ˆP , p ∈ N. An additional reduction of n cost is possible if the scalar product ha, bi = i=1 ai bi is approximated by ha, bip := p

n ˆ X

ai bi .

i=1

h·, ·ip is not a scalar product in Kn , but if a, b ∈ Kn are smooth grid functions, ha, bip approximates ha, bi. With this modification in DCQR, n in NDCQR can be reduced to n ˆ . Note that the computed Q is not strictly orthogonal. Following the ideas from [187], we may start with n ˆ = O(1) to select rˆ columns, where r < rˆ = O(r). Then the final call of DCQR (with exact scalar product) is applied to the reduced matrix M ∈ Kn׈r . Note that this procedure yields the same result as DCQR applied to the original matrix M if the finally chosen r columns are among the rˆ columns selected by the first heuristic step. The total work is O(mr + m20 r + r2 n + r3 ) with m0 from Corollary 2.42.

21

To be precise, only hck , cj i and |ζjk |2 need to be updated.

Chapter 3

Algebraic Foundations of Tensor Spaces

Abstract Since tensor spaces are in particular vector spaces, we start in Section 3.1 with vector spaces. Here, we introduce the free vector space (§3.1.2) and the quotient vector space (§3.1.3) which are needed later. Furthermore, the spaces of linear mappings and dual mappings are discussed in §3.1.4. The core of this chapter is Section 3.2 containing the definition of the tensor space. Section 3.3 is devoted to linear and multilinear mappings as well as to tensor spaces of linear mappings. Algebra structures are discussed in Section 3.4. Finally, symmetric and antisymmetric tensors are defined in Section 3.5.

3.1 Vector Spaces 3.1.1 Basic Facts We recall that V is a vector space (also named linear space) over the field K, if V 6= ∅ is a commutative group (where the group operation is written as addition) and if a multiplication ·:K×V →V is defined with the following properties: (αβ) · v (α + β) · v α · (v + w) 1·v 0·v

= α · (β · v) = α·v+β·v = α·v+α·w =v =0

for α, β ∈ K, v ∈ V, for α, β ∈ K, v ∈ V, for α ∈ K, v, w ∈ V, for v ∈ V, for v ∈ V,

(3.1)

where on the left-hand side 1 and 0 are the respective multiplicative and additive unit elements of the field K, while on the right-hand side 0 is the zero element of the group V . © Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_3

49

3 Algebraic Foundations of Tensor Spaces

50

The sign ‘·’ for the multiplication · : K × V → V is usually omitted, i.e., αv is written instead of α · v. When α (v + w) may be misunderstood as a function α evaluated at v + w, we prefer the original notation α · (v + w). Any vector space V has a basis {vi : i ∈ B} ⊂ V with the property that it is In the infinite case of linearly independent and spans V = span{vi : i ∈ B}. P #B = ∞, linear independence means that all finite sums i ai vi vanish P if and only if ai = 0. Analogously, span{vi : i ∈ B} consists of all finite sums i ai vi . Here ‘finite sum’ means a sum with finitely many terms or equivalently a (possibly infinite) sum with only finitely many nonzero terms. The cardinality #B is independent of the choice of a basis and called the dimension, denoted by dim(V ). Note that there are many infinite cardinalities. Equality dim(V ) = dim(W ) holds if and only if there exists a bijection BV ↔ BW between the corresponding index sets of the bases. Vector spaces having identical dimensions are isomorphic. For finite dimension n ∈ N0 , the model vector space is Kn . The isomorphism P between a general vector space V with basis {v1 , . . . , vn } and Kn is given by v = αν vν 7→ (α1 , . . . , αn ) ∈ Kn . Example 3.1. Let I be an infinite index set. Then `(I) = KI denotes the set of all sequences (ai )i∈I . The set `(I) may also be viewed as the set of all mappings I → K. A subset of `(I) is `0 (I) := {a ∈ `(I) : ai = 0 for almost all i ∈ I} .

(3.2)

The unit vectors {e(i) : i ∈ I} in (2.2) form a basis of `0 (I) so that the dimension is equal to dim(`0 (I)) = #I. However, the vector space `(I) has a basis of larger infinite cardinality: dim(`(I)) > dim(`0 (I)). Above the case of a finite index set I is less interesting since then `0 (I) = `(I) does not yield a new set. An example of a uncountable index set is I = R (i.e., `(I) is isomorphic to the set of K-valued functions on R).

3.1.2 Free Vector Space over a Set Given an arbitrary set S, we are aiming for a vector space such that S is a basis. Let S be any nonempty set and K a field. Consider a mapping ϕ : S → K. Its support is defined by supp(ϕ) := {s ∈ S : ϕ(s) 6= 0} ⊂ S, where 0 is the zero element in K. Requiring # supp(ϕ) < ∞ means that ϕ = 0 holds for almost all s ∈ S. This property defines the set V := {ϕ : S → K : # supp(ϕ) < ∞} .

3.1 Vector Spaces

51

We introduce an addition in V . For ϕ, ψ ∈ V , the sum σ := ϕ + ψ is the mapping σ : S → K defined by their images σ(s) := ϕ(s) + ψ(s) for all s ∈ S. Note that the support of σ is contained in supp(ϕ) ∪ supp(ψ), which again has finite cardinality, so that σ ∈ V . Since ϕ(s) + ψ(s) is the addition in K, the operation is commutative: ϕ + ψ = ψ + ϕ. Obviously, the zero function 0V ∈ V with 0V (s) = 0 ∈ K for all s ∈ S satisfies ϕ + 0V = 0V + ϕ = ϕ. Furthermore, ϕ− : S → K defined by ϕ− (s) = −ϕ(s) is the inverse of ϕ, i.e., ϕ− + ϕ = ϕ + ϕ− = 0V . Altogether, (V, +) is a commutative group. Scalar multiplication · : K×V → V maps α ∈ K and ϕ ∈ V into the mapping ψ := αϕ defined by ψ(s) = αϕ(s) for all s ∈ S. Therefore, all axioms in (3.1) are satisfied so that V represents a vector space over the field K. The characteristic functions χs are of particular interest for an element s ∈ S:  1 if t = s ∈ S, χs (t) = 0 if t ∈ S\{s}. Every ϕ ∈ V may be written as a linear combination of such χs : X X ϕ= ϕ(s) χs . ϕ(s) χs = s∈S

s∈supp(ϕ)

Here two different notations are used: the first sum is finite, while the second one is infinite, but contains only finitely many nonzero terms. Note that any finite subset of {χs : s ∈ S} is linearly independent. Assuming X αs χs = 0V for some S0 ⊂ S with #S0 < ∞ and αs ∈ K, s∈S0

the evaluation at t ∈ S0 yields ! αt =

X

αs χs

(t) = 0V (t) = 0

s∈S0

proving linear independence. Conversely, any finite linear combination X with αs ∈ K, S0 ⊂ S, #S0 < ∞ αs χs

(3.3)

s∈S0

belongs to V . sets {χs : s ∈ S} Let Φχ : χs 7→ s be the one-to-one correspondence between the P and S. We can extend Φχ to ΦV defined on V such that v = s∈S0 αs χs in P (3.3) is mapped onto the formal linear combination s∈S0 αs s of elements of S. The image Vfree (S) := ΦV (V ) is called the free vector space over the set S.

3 Algebraic Foundations of Tensor Spaces

52

3.1.3 Quotient Vector Space Let V0 be a subspace of a vector space V. Then V0 defines an equivalence relation on V : v∼w

if and only if v − w ∈ V0 .

Any v ∈ V can be associated with an equivalence class cv := {w ∈ V : w ∼ v} .

(3.4)

Here v is called a representative of the class cv , which is also written as v + V0 . Because of the definition of an equivalence relation, the classes are either equal or disjoint. Their union is equal to V . The set {cv : v ∈ V } of all equivalence classes is denoted by the quotient V / V0 . We may define c0 + c00 := {v 0 + v 00 : v 0 ∈ c0 , v 00 ∈ c00 } for two classes c0 , c00 ∈ V /V0 and check that the resulting set is again an equivalence class, i.e., an element in V /V0 . Similarly, we define λ · c ∈ V /V0 for c ∈ V /V0 . Using the notation cv for the classes generated by v ∈ V , we find the relations cv + cw = cv+w and λ · cv = cλv . In particular, c0 is the zero element. Altogether, V /V0 is again a vector space over the same field, called the quotient vector space. Exercise 3.2. Prove the identity dim(V ) = dim (V /V0 ) dim(V0 ) and the particular cases V /V = {0} and V / {0} = V. A mapping ϕ : V /V0 → X (X any set) may possibly be induced by a map Φ : V → X via ϕ(cv ) := Φ(v) (cv in (3.4)). (3.5) Whether (3.5) is a well-defined formulation, hinges upon the following consistency condition. Lemma 3.3. (a) Let Φ : V → X be a general mapping. Then (3.5) for all v ∈ V defines a mapping ϕ : V /V0 → X if and only if Φ is constant on each equivalence class, i.e., v ∼ w implies Φ(v) = Φ(w). (b) If Φ : V → X (X a vector space) is a linear mapping, the necessary and sufficient condition reads Φ(v) = 0 for all v ∈ V0 .

3.1 Vector Spaces

53

3.1.4 (Multi-)Linear Maps, Algebraic Dual, Basis Transformation Let X, Y be two vector spaces. ϕ : X → Y is a linear mapping if ϕ(λx0 + x00 ) = λϕ(x0 ) + ϕ(x00 )

for all λ ∈ K and all x0 , x00 ∈ X.

The set of linear mappings ϕ is denoted by L(X, Y ) := {ϕ : X → Y is linear} .

(3.6) d

Let Xj (1 ≤ j ≤ d) and Y be vector spaces. A mapping ϕ : ×j=1 Xj → Y is called multilinear (or d-linear) if ϕ is linear in all d arguments: ϕ(x1 , . . . , xj−1 , xj0 + λx00j , xj+1 , . . . , xd ) = ϕ(x1 , . . . , xj−1 , x0j , xj+1 , . . . , xd ) + λϕ(x1 , . . . , xj−1 , xj00 , xj+1 , . . . , xd ) for all xi ∈ Xi , xj0 , xj00 ∈ Xj , 1 ≤ j ≤ d, λ ∈ K. In the case of d = 2, the term bilinear mapping is used. Definition 3.4. Φ ∈ L(X, X) is called a projection, if Φ2 = Φ. It is called a projection onto Y if Y = range(Φ). Note that no topology is defined and, therefore, no continuity is required. Remark 3.5. Let {xi : i ∈ B} be a basis of X. ϕ ∈ L(X, Y ) is uniquely determined by the values ϕ(xi ), i ∈ B. In the particular case of Y = K, linear mappings ϕ : X → K are called linear forms. They are elements of the vector space X 0 := L(X, K), which is called the algebraic dual of X. A multilinear (bilinear) map into Y = K is called a multilinear form (bilinear form). Definition 3.6. Let S := {xi : i ∈ B} ⊂ X be a system of linearly independent vectors. A dual system {ϕi : i ∈ B} ⊂ X 0 is defined by ϕi (xj ) = δij

for i, j ∈ B

(cf. (2.1)).

If {xi : i ∈ B} is a basis and #B < ∞, {ϕi : i ∈ B} is called a dual basis.1 Remark 3.7. The dual Psystem allows us to determine the coefficients αi in the basis αi xi ∈ X by αi = ϕi (x). representation x = i∈B

Note that a dual system is not a basis of X 0 if dim(X) = ∞. The functional Φ ∈ X 0 defined by Φ(xi ) = 1 for all i ∈ B is not a finite linear combination of the dual system. 1

3 Algebraic Foundations of Tensor Spaces

54

So far we used abstract vector spaces. An alternative representation follows from Remark 3.7 as soon as a basis {xi : i ∈ B} is fixed. Then the coefficient tuples (αi )i∈B belong to KB (if #B = n we may also write Kn ). In this case the reference to the underlying basis is essential. For another basis {ˆ xi : i ∈ B} we obtain a new coefficient vector (ˆ αi )i∈B ∈ KB . The following result follows from X X α ˆix ˆi . αi xi = i∈B

i∈B

P ˆj and Lemma 3.8 (basis transformation). Assume two bases with xi = j∈B tji x P x ˆj = i∈B sij xi . Set T = (tij )i,j∈B and S = (sij )i,j∈B . Let α := (αi )i∈B ∈ αi )i∈B ∈ KB be the coefficient vectors corresponding to the KB and α ˆ := (ˆ xi : i ∈ B}. Then respective bases {xi : i ∈ B} and {ˆ ˆ, α = Sα

T = S −1 .

α ˆ = T α,

Also the dual basis {ϕi : i ∈ B} refers to {xi : i ∈ B} . } and {ˆ xi }, Lemma 3.9. Let the dual bases {ϕi } and {ϕˆi } correspond to {xiP ϕˆj ϕ = s respectively. The basis transformation in Lemma 3.8 leads to ij i j∈B P and ϕˆj = i∈B tji ϕi . A general vector (functional) ϕ ∈ X 0 can be written as X X βˆi ϕˆi ϕ= βi ϕ i = i∈B

i∈B

with coefficient vectors β = (βi )i∈B ∈ KB and βˆ = (βˆi )i∈B ∈ KB . Using S and T from Lemma 3.8, the relations involve the transposed matrices: ˆ β = T T β,

βˆ = S T β.

3.2 Tensor Product 3.2.1 Constructive Definition There are various ways to define the tensor product of two vector spaces. We follow the quotient space formulation (cf. [301]). Other constructions yield isomorphic objects (see comment after Proposition 3.23). Given two vector spaces V and W over some field K, we start with the free vector space Vfree (S) over the pair set S := V × W as introduced in §3.1.2. Note that Vfree (V ×W ) does not make use of the vector space properties of V or W. We recall that elements of Vfree (V ×W ) are linear combinations of pairs from V ×W : m X i=1

λi (vi , wi )

for any (λi , vi , wi ) ∈ K×V ×W, m ∈ N0 .

3.2 Tensor Product

55

A particular subspace of Vfree (V ×W ) is   m  m n n    P P αi βj (vi , wj ) − P αi vi , P βj wj  . N := span i=1 j=1 i=1 j=1    for m, n ∈ N, α , β ∈ K, v ∈ V, w ∈ W  i j i j

(3.7)

The algebraic tensor space is defined by the quotient vector space V ⊗a W := Vfree (V × W ) / N

(3.8)

(cf. §3.1.3). The equivalence class c(v,w) ∈ V ⊗a W generated by a pair (v, w) ∈ V × W is denoted by v ⊗ w. Note that the tensor symbol ⊗ is used for two different purposes:2 (i) In the tensor space notation V ⊗ W the symbol ⊗ connects vector spaces and may carry the suffix ‘a’ (meaning ‘algebraic’) or a norm symbol in the later case of Banach tensor spaces (cf. (3.10) and §4). (ii) In v ⊗ w the quantities v, w, v ⊗ w are vectors, i.e., elements of the respective vector spaces V, W, V ⊗a W . As Vfree (V ×W ) is the set of linear combinations of (vi , wi ), the quotient space Vfree (V × W )/N consists of all linear combinations of vi ⊗ wi : V ⊗a W = span {v ⊗ w : v ∈ V, w ∈ W } .

(3.9)

If a norm topology is given, the completion with respect to the given norm k·k yields the topological tensor space V ⊗k·k W := V ⊗a W .

(3.10)

In §4 we discuss the properties of the tensor product for Banach spaces. This includes the Hilbert spaces, which are considered in §4.4. Notation 3.10. (a) For finite-dimensional vector spaces V and W , the algebraic tensor space V ⊗a W is already complete with respect to any norm and, therefore, coincides with the topological tensor space V ⊗k·k W . In this case we omit the suffices and simply write V ⊗ W. (b) Furthermore, the notation V ⊗ W is used when both choices V ⊗a W and V ⊗k·k W are possible or if the distinction between ⊗a and ⊗k·k is irrelevant. (c) The suffices of ⊗a and ⊗k·k will be moved to the left side when indices appear Nd Nd at the right side as in a j=1 Vj and k·k j=1 Vj . 2

Similarly, the sum v + w of vectors and the sum V + W := span{v + w : v ∈ V, w ∈ W } of vector spaces use the same symbol.

3 Algebraic Foundations of Tensor Spaces

56

Definition 3.11 (tensor space, tensor). (a) V ⊗a W (or V ⊗k·k W ) is again a vector space, which is now called a tensor space. (b) The explicit term algebraic tensor space emphasises that V ⊗a W , and not V ⊗k·k W , is meant. (c) Elements of V ⊗a W or V ⊗k·k W are called tensors, in particular, x ∈ V ⊗a W is an algebraic tensor, while x ∈ V ⊗k·k W is a topological tensor. (d) Any product v ⊗ w (v ∈ V, w ∈ W ) is called an elementary tensor.

3.2.2 Characteristic Properties Lemma 3.12. The characteristic algebraic properties of the tensor space V ⊗a W is the bilinearity: (λv) ⊗ w = v ⊗ (λw) = λ · (v ⊗ w) (v 0 + v 00 ) ⊗ w = v 0 ⊗ w + v 00 ⊗ w v ⊗ (w0 + w00 ) = v ⊗ w0 + v ⊗ w00 0⊗w =v⊗0=0

for for for for

λ ∈ K, v ∈ V, w ∈ W, v 0 , v 00 ∈ V, w ∈ W, v ∈ V, w0 , w00 ∈ W, v ∈ V, w ∈ W.

(3.11)

Proof. The first equality in (3.11) follows from λ (v, w) − (λv, w) ∈ N ; i.e., λ · (v ⊗ w) − (λv) ⊗ w = 0 holds in the quotient space. The other identities are u t derived similarly. Here the standard notational convention holds: the multiplication ⊗ has priority over the addition +, i.e., a ⊗ b + c ⊗ d means (a ⊗ b) + (c ⊗ d). Multiplication by a scalar needs no bracket since the interpretation of λv ⊗ w by (λv) ⊗ w or λ · (v ⊗ w) does not change the result (see first identity in (3.11)). In the last line3 of (3.11) the three zeros belong to the different spaces V, W , and V ⊗a W . The following statements also hold for infinite-dimensional spaces. Note that in this case the dimensions must be understood as set theoretical cardinal numbers. Lemma 3.13. (a) Let {vi : i ∈ BV } be a basis of V and {wj : j ∈ BW } a basis of W . Then a basis of V ⊗a W is given by B := {vi ⊗ wj : i ∈ BV , j ∈ BW } .

(3.12)

(b) dim (V ⊗a W ) = dim(V ) · dim(W ). P Proof. Assume i,j aij vi ⊗ wj = 0. For the linear independence of all products vi ⊗ wj ∈ B we have to show aij = 0. The properties (3.11) show X X aij wj . for wi0 := (3.13) vi ⊗ wi0 = 0 i 3

j

The last line can be derived from the first one by setting λ = 0.

3.2 Tensor Product

57

Let ϕi ∈ V 0 (cf. §3.1.4) be the linear form on V with ϕi (vj ) = δij (cf. (2.1)). Define Φi : V ⊗a W → W by Φi (v ⊗ w) = ϕi (v)w. Application of Φi to (3.13) yields wi0 = 0. Since {wj : j ∈ BW } is a basis, aij = 0 follows for all P j. As i is chosen arbitrarily, we have shown aij = 0 for all coefficients in i,j aij vi ⊗ wj = 0. Hence B is a system of linearly independent vectors. P By definition, a general tensor x ∈ V ⊗a W has the form x = ν v (ν) ⊗ w(ν) . P (ν) be expressed by the basis vectors: v (ν) = i αi vi , and similarly, Each v (ν) P can(ν) (ν) w = j βj wj . Note that all sums have finitely many terms. The resulting sum x=

XX ν

(ν)



αi vi

⊗

i

X



(ν)

βj wj

j

XX

= (3.11)

i,j

(ν) (ν)

αi βj

 vi ⊗ w j

ν

is again finite and shows that span{B} = V ⊗a W , i.e., B is a basis. Since #B = #BV · #BW , we obtain the dimension identity of Part (b).

u t

The last two statements characterise the tensor space structure. Proposition 3.14. Let V, W , and T be vector spaces over the field K. A product ⊗ : V × W → T is a tensor product and T a tensor space, i.e., it is isomorphic to V ⊗a W, if the following properties hold: (i)

span property: T = span{v ⊗ w : v ∈ V, w ∈ W };

(ii)

bilinearity (3.11);

(iii)

linearly independent vectors {vi : i ∈ BV } ⊂ V and {wj : j ∈ BW } ⊂ W lead to linearly independent vectors {vi ⊗ wj : i ∈ BV , j ∈ BW } in T.

Proof. Properties (i)–(iii) imply that B in (3.12) is again a basis.

u t

Lemma 3.15. For any tensor x ∈ V ⊗a W there exist an r ∈ N0 and a representation r X vi ⊗ wi (3.14) x= i=1

with linearly independent vectors {vi : 1 ≤ i ≤ r} ⊂ V and {wi : 1 ≤ i ≤ r} ⊂ W . Pn Proof. Take any representation x = i=1 vi ⊗ wi . If, e.g., the system of vectors {vi : 1 ≤ i ≤ n} is not linearly independent, one vi can be expressed by the others. Without loss of generality assume vn =

n−1 X

αi vi .

i=1

Then vn ⊗ w n =

n−1 X i=1

! αi vi

⊗ wn =

n−1 X i=1

vi ⊗ (αi wn )

3 Algebraic Foundations of Tensor Spaces

58

shows that x possesses a representation with only n−1 terms: ! n−1 n−1 X X vi ⊗ wi + vn ⊗ wn = vi ⊗ wi0 with wi0 := wi + αi wn . x= i=1

i=1

Since each reduction step decreases the number of terms by one, this process terminates at a certain number r of terms, i.e., we obtain a representation with r u t linearly independent vi and wi . The number r appearing in Lemma 3.15 will be called the rank of the tensor x (cf. §3.2.6.2). This is in accordance with the usual matrix rank as seen in §3.2.3.

3.2.3 Isomorphism to Matrices for d = 2 Let V and W be vector spaces of finite dimensions dim V = #I and dim W = #J (I, J index sets). As discussed in the final part of §3.1.4, a particular choice of bases in V, W leads to the (isomorphic) spaces KI , KJ of coefficient vectors. The transfer to the coefficients is necessary to obtain the matrix space KI×J . In part (a) of Proposition 3.16 the isomorphism V ⊗ W ∼ = KI ⊗ KJ is described. I×J J I is already described in Remark 1.3a. The isomorphism M : K ⊗ K → K Proposition 3.16. Let {vi : i ∈ I} be a basis of V and {wj : j ∈ J} a basis of W , where dim(V ) < ∞ and dim(W ) < ∞. Let Φ : KI → V and Ψ : KJ → W denote the isomorphisms X X Ψ : (βj )j∈J 7→ αi vi , βj wj . Φ : (αi )i∈I 7→ i∈I

j∈J

(a) Then the corresponding canonical isomorphism of the tensor spaces is given by XX Ξ := Φ⊗Ψ : KI ⊗KJ → V ⊗W with (αi )i∈I ⊗(βj )j∈J 7→ αi βj vi ⊗wj . i∈I j∈J

(b) Together with M : KI ⊗ KJ → KI×J from Remark 1.3a we obtain an isomorphism between matrices from KI×J and tensors from V ⊗ W : XX Ξ 0 := ΞM−1 : KI×J → V ⊗ W with (aij )i∈I,j∈J 7→ aij vi ⊗ wj . i∈I j∈J

Concerning the usual isomorphism between matrices from KI×J and linear mappings W → V , we refer to Proposition 3.68. The difference between matrices and tensors leads to different representations of basis transformations.

3.2 Tensor Product

59

Remark 3.17 (basis transformation). If we change the bases {vi : i ∈ I} and {wj : j ∈ J} from Proposition 3.16 by transformations S and T : X X Sni vˆn and wj = ˆm , vi = Tmj w n∈I

P

m∈J

P

ˆnm vˆn ⊗ w ˆm shows that the matrices A = (aij ) then i,j aij vi ⊗ wj = n,m a aij ) are related by and Aˆ = (ˆ Aˆ = S A T T . On the side of the tensors, this transformation takes the form ˆ a = (S ⊗ T ) a

a := (ˆ with a := (ai,j )(i,j)∈I×J and ˆ ai,j )(i,j)∈I×J ,

where S ⊗ T is the Kronecker product acting on KI ⊗ KJ (cf. §1.1.2 and §3.3.2.1), i.e., M ((S ⊗ T ) a) = S M(a) T T . If V = KI and W = KJ , Ξ is the identity and the isomorphism between V ⊗ W and the matrix space KI×J is given by M in Remark 1.3a. Remark 3.18. Suppose dim(V ) = 1. Then the vector space V may be identified with the field K. V ⊗a W is isomorphic to K ⊗a W and to W . In the latter case, we identify λ ⊗ w (λ ∈ K, w ∈ W ) with λw. Lemma 3.19 (reduced singular-value decomposition). Let K ∈ {R, C} , and suppose dim(V ) < ∞ and dim(W ) < ∞. Then for any x ∈ V ⊗ W there exist a number r ≤ min{#I, #J} and two families (xi )i=1,...,r and (yi )i=1,...,r of linearly independent vectors such that r X σi xi ⊗ yi x= i=1

with singular values σ1 ≥ . . . ≥ σr > 0. Proof. The isomorphism Ξ 0 : KI×J → V ⊗ W from Proposition 3.16b de0−1 fines the decomposition Prmatrix AT:= Ξ x, for which the reduced singular-value that Ξ 0 (ai bT A = i=1 σi ai bi can be determined (cf. (2.18)). Note P i ) = ai ⊗ bi r 0 Ξ x A = Ξ(a σ = (cf. (1.3)). Backtransformation yields i i ⊗ bi ) = i=1 Pr i=1 σi Φ(ai ) ⊗ Ψ (bi ). The statement follows by setting xi := Φ(ai ) and u yi := Ψ (bi ). Note that linearly independent ai yield linearly independent Φ(ai ). t We remark that the vectors xi (as well as yi ) are not orthonormal since such properties are not (yet) defined for V (and W ). Lemma 3.19 yields a second proof of Lemma 3.15 but restricted to K ∈ {R, C}. Remark 3.20. The tensor spaces V ⊗a W and W ⊗a V are isomorphic vector spaces via the (bijective) transposition T : V ⊗a W → W ⊗a V

via x = v ⊗ w 7→ xT = w ⊗ v.

If x ∈ V ⊗ W corresponds to a matrix M = Ξ 0−1 (x) (cf. Proposition 3.16b), then xT ∈ W ⊗ V corresponds to the transposed matrix M T .

3 Algebraic Foundations of Tensor Spaces

60

3.2.4 Tensors of Order d ≥ 3 3.2.4.1 Algebraic Properties In principle we can extend the construction in §3.2.1 to the case of more than two factors. However, this is not necessary as the next lemma shows. Lemma 3.21. (a) The tensor product is associative: the left- and right-hands sides in U ⊗a (V ⊗a W ) = (U ⊗a V ) ⊗a W, are isomorphic vector spaces as detailed in the proof. We identify both notations and use the neutral notation U ⊗a V ⊗a W instead. (b) If U, V, W are finite dimensional with dim(U ) = n1 , dim(V ) = n2 , and dim(W ) = n3 , the isomorphic model tensor space is Kn1 ⊗ Kn2 ⊗ Kn3 . Proof. Let ui (i ∈ BU ) , vj (j ∈ BV ) , wk (k ∈ BW ) be bases of U, V,
W . As seen in Lemma 3.13, V ⊗a W has the basis vj ⊗ wk (j, k) ∈ BV × BW , while U ⊗a (V ⊗a W ) has the basis ui ⊗ (vj ⊗ wk ) with (i, (j, k)) ∈ BU ×(BV ×BW ). Similarly, (U ⊗a V ) ⊗a W has the basis (ui ⊗ vj ) ⊗ wk

with

((i, j) , k) ∈ (BU × BV ) × BW .

By the obvious bijection between BU × (BV × BW ) and (BU × BV ) × BW , the spaces U ⊗a (V ⊗a W ) and (U ⊗a V ) ⊗a W are isomorphic. This proves part (a). t u For part (b), see Remark 3.31. Repeating the product construction (d − 1)-times, we get the generalisation of Nd the previous results to the algebraic tensor product a j=1 Vj (cf. Notation 3.10). Proposition 3.22. Let Vj (1 ≤ j ≤ d, d ≥ 2) be vector spaces over K. d N (a) The algebraic tensor space4 V := a Vj is independent of the order in j=1

which the pairwise construction (3.8) is performed (more precisely, the resulting spaces are isomorphic and can be identified). (b) T is the algebraic tensor space V if the following properties hold: nN o d (j) (j) v : V T = span span property: (i) v ∈ j ; j=1 (ii)

multilinearity; i.e., for all λ ∈ K and v (j) , w(j) ∈ Vj with j ∈ {1, . . . , d}:
v (1) ⊗ v (2) ⊗ . . . ⊗ λv (j) + w(j) ⊗ . . . ⊗ v (d) = λv (1) ⊗ v (2) ⊗...⊗ v (j) ⊗...⊗ v (d) + v (1) ⊗ v (2) ⊗...⊗ w(j) ⊗...⊗ v (d) ;

(iii)

linearly independent vectors {vi : i ∈ Bj } ⊂ Vj (1 ≤ j ≤ d) lead to Nd (j) linearly independent vectors { j=1 vij : ij ∈ Bj } in T.

(j)

N The product d j=1 Vj is to be formed in the order of the indices j, i.e., V1 ⊗ V2 ⊗ . . . If we N write j∈K Vj for an ordered index set K, the ordering of K determines the order of the factors. 4

3.2 Tensor Product

61

(c) The dimension is given by dim(V) =

d Y

dim(Vj ).

j=1 (j)

If {vi : i ∈ Bj } are bases of Vj (1 ≤ j ≤ d), then {vi : i ∈ B} is a basis of V, where d O (j) vi j for i = (i1 , . . . , id ) ∈ B := B1 × . . . × Bd . vi = j=1

Nd An alternative definition of a j=1 Vj follows the construction in §3.2.1 with pairs (v, w) replaced with d-tuples and an appropriately defined subspace N . The following expression of the multilinear mapping ϕ by the linear mapping Φ is also called the ‘linearisation’ of ϕ. Proposition 3.23 (universality of the tensor product). Let Vj (1 ≤ j ≤ d) and U be vector spaces over K. Then, for any multilinear mapping ϕ : V1 × . . . × Vd → U , i.e., ϕ(v (1) , . . . , λv (j) + w(j) , . . . , v (d) ) = λ ϕ(v (1) , . . . , v (j) , . . . , v (d) ) + ϕ(v (1) , . . . , w(j) , . . . , v (d) ) for all v

(j)

,w

(j)

(3.15)

∈ Vj , λ ∈ K, 1 ≤ j ≤ d,

there exists a unique linear mapping Φ :

a

Nd

j=1

Vj → U such that

ϕ(v (1) , v (2) , . . . , v (d) ) = Φ v (1) ⊗ v (2) ⊗ . . . ⊗ v (d)




for all v (j) ∈ Vj , 1 ≤ j ≤ d. (j)

Proof. Let {vi : i ∈ Bj } be a basis of Vj . {vi : i ∈ B} from Proposition 3.22c Nd (1) (d) is a basis of V := a j=1 Vj . Define Φ(vi ) := ϕ(vi1 , . . . , vid ). This determines Φ : V → U uniquely (cf. Remark 3.5). Analogously, the multilinear mapping ϕ (1) (d) is uniquely determined by ϕ(vi1 , . . . , vid ). Multilinearity of ϕ and V yields

ϕ v (1) , v (2) , . . . , v (d) = Φ v (1) ⊗ . . . ⊗ v (d) for all v (j) ∈ Vj . t u The statement of Proposition 3.23 may also be Nd used as an equivalent definition of a j=1 Vj (cf. Greub [125, Chap. I, §2]). The universal factorisation property stated in Proposition 3.23 is visualised by the commutative diagram to the right.

V1 × . . . × Vd

a

⊗↓ Nd

j=1

→ U ϕ

Φ% Vj

Notation 3.24. If all Vj = V are identical vector spaces, the notation Nd simplified by ⊗d V . For a vector v ∈ V , we set ⊗d v := j=1 v.

Nd

j=1

Vj is

3 Algebraic Foundations of Tensor Spaces

62

3.2.4.2 Nondegeneracy To avoid trivial situations, we define nondegenerate tensor spaces. Nd Definition 3.25. A tensor space V = a j=1 Vj is called nondegenerate, if d > 0 and dim(Vj ) ≥ 2 for all 1 ≤ j ≤ d. Otherwise V is called degenerate. This definition is justified by the following remarks. If dim(Vj ) = 0 for one j, V = {0} is the trivial vector space. If d = 0, the empty product is defined by V = K, which is also a trivial case. In the case of dim(Vj ) = 1 for some j, the Nd formulation by a j=1 Vj can be reduced (see next remark). Remark 3.26. (a) If dim(Vk ) = 1 for some k, the isomorphism V = a N a j∈{1,...,d}\{k} Vj allows us to omit the factor Vk .

Nd

j=1 Vj

∼ =

(b) After eliminating all factors Vj with dim(Vj ) = 1 and renaming the remaining Ndred vector spaces, we obtain V ∼ Vj . If still dred > 0, the representation = Vred = a j=1 is nondegenerate. Otherwise the tensor space is degenerate because of dred = 0. As an illustration, we may consider a matrix space (i.e., a tensor space of order d = 2). If dim(V2 ) = 1, the matrices consist of only one column. Hence they may be considered as vectors (tensor space with d = 1). If dim(V1 ) = dim(V2 ) = 1 , the 1 × 1-matrices may be identified with scalars from the field K.

3.2.4.3 Tuples Finally, we mention an isomorphism between the space of tuples of tensors and an extended tensor space. Nd Lemma 3.27. Let V = j=1 Vj be a tensor space over the field K and m ∈ N. The vector space of m-tuples (v1 , . . . , vm ) with vi ∈ V is denoted by Vm . Then the following vector space isomorphism is valid: Vm ∼ =

d+1 O

Vj = V ⊗ Vd+1

with Vd+1 := Km .

j=1

Pm (i) (i) Proof. (v1 , . . . , vm ) ∈ Vm corresponds to ∈ Km i=1 vi ⊗ e , where e is the i-th unit vector. The opposite direction of the isomorphism is described by m ∼ (x1 v, x2 v, . . . , xm v) with x(d+1) = (xi ) v ⊗ x(d+1) = t u i=1,...,m ∈ K . Exercise 3.28. Let Pmt be the tensor corresponding to the tuple (v1 , . . . , vm ) . Prove that rank(t) ≤ i=1 rank(vi ). A generalisations to infinite dimensions can be defined by V ⊗a `0 (I) with #I = ∞ (cf. (3.2)).

3.2 Tensor Product

63

3.2.5 Different Types of Isomorphisms In algebra it is common to identify isomorphic structures. All characteristic properties of an algebraic structure should be invariant under isomorphisms. The question is what algebraic structures are meant. All previous isomorphisms were vector space isomorphisms. As it is well known, two vector spaces are isomorphic if and only if the dimensions coincide (note that in the infinite-dimensional case, the cardinalities of the bases are decisive). Any tensor space is a vector space, but not any vector space isomorphism preserves the tensor structure. An essential part of the tensor structure is the d-tuple of vector spaces (V1 , . . . , Vd ) together with the dimensions of Vj (cf. ProposiNd tion 3.23). In fact, each space Vj can be regained from V := a j=1 Vj as Nd the range of the mapping A = k=1 Ak (cf. (1.4a)) with Aj = id, while 0 6= Ak ∈ Vk0 for k 6= j. Therefore a tensor space isomorphism must satisfy Nd ∼ W implies that W = a Nd Wj holds with isomorphic that V = a j=1 Vj = j=1 ∼ Vj for all 1 ≤ j ≤ d. In particular, the order d of the tensor vector spaces Wj = spaces must coincide. This requirement is equivalent to the following definition. Definition 3.29 (tensor space isomorphism). A tensor space isomorphism A : V := a

d O j=1

Vj → W = a

d O

Wj

j=1

Nd is any bijection of the form A = j=1 Aj (cf. (3.30a)), where Aj : Vj → Wj (1 ≤ j ≤ d) are vector space isomorphisms. For instance, K2 ⊗ K8 , K4 ⊗ K4 , and K2 ⊗ K2 ⊗ K4 are isomorphic vector spaces (since all have dimension 16), but their tensor structures are not identical. For the definition of U ⊗a V ⊗a W we use in Lemma 3.21 that U ⊗a (V ⊗a W ) ∼ = (U ⊗a V ) ⊗a W . Note that these three spaces are isomorphic only in the sense of vector spaces, whereas their tensor structures are different. In particular, the latter spaces are tensor spaces of order two, while U ⊗a V ⊗a W is of order three. We see the difference when we consider the elementary tensors as in the next example. Example 3.30. Let both {v1 , v2 } ⊂ V and {w1 , w2 } ⊂ W be linearly independent. Then u ⊗ (v1 ⊗ w1 + v2 ⊗ w2 ) is an elementary tensor in U ⊗a (V ⊗a W ) (since u ∈ U and v1 ⊗ w1 + v2 ⊗ w2 ∈ V ⊗a W ), but it is not an elementary tensor of U ⊗a V ⊗a W . Remark 3.31. In the finite-dimensional case of nj := dim(Vj ) < ∞, the isomorphic Nd (j) model tensor space is W := j=1 Knj . Choose some bases {bν : 1 ≤ ν ≤ nj } Nd of Vj . Any v ∈ V := j=1 Vj has a unique representation of the form X (d) (1) (3.16) v= ai bi1 ⊗ . . . ⊗ bid . i1 ···id

The coefficients ai define the coefficient tensor a ∈ W. The mapping A : V → W by A(v) = a is a tensor space isomorphism.

3 Algebraic Foundations of Tensor Spaces

64

The next generalisation of Remark 3.17 describes a general basis transformation. (j) (j) Lemma 3.32 (basis transformation). (a) Let {bi }i∈Ij and {ˆbi }i∈Ij be bases P (j) (j) (j) of Vj with bk = i∈Ij Tik ˆbi . Then the coefficient tensors a and ˆ a in (3.16) are related by d O (j) with T = T (j) , T (j) = (Tik ). a = Ta ˆ j=1

T:

Nd

Ij j=1 K

→

Nd

j=1 K

Ij

is a tensor space isomorphism.

(b) If the spaces Vj = V coincide, i.e., V = ⊗d V, also the transformations T (j) = T coincide and T becomes T = ⊗d T. (c) If either Vj = V or Vj = V 0 , the T (j) are either T or (T −1 )T , respectively. t u

Proof. For part (c) use Lemma 3.9.

∼ is an equivalence relation in the set of the respective The isomorphism sign = ∼ structure. Let =vec denote the vector space isomorphism, while ∼ =ten is the tensor space isomorphism. Then ∼ =ten W =ten is the finer equivalence relation since V ∼ implies V ∼ =vec W but not vice versa. There are additional equivalence relations ∼ which are between ∼ =ten and ∼ =W ⇒ V∼ =, =vec , i.e., V ∼ =ten W ⇒ V ∼ =vec W. We give three examples. 1) We may not insist upon a strict ordering of the vector spaces Vj . Let π : {1, . . . , d} → {1, . . . , d} be a permutation. Then V := V1 ⊗ V2 ⊗ . . . ⊗ Vd and Vπ := Vπ(1) ⊗ Vπ(2) ⊗ . . . ⊗ Vπ(d) are considered as isomorphic. In a second step, each Vj may be replaced with an isomorphic vector space Wj . 2) In Remark 3.26 we omitted vector spaces Vj ∼ = K of dimension one. This ∼ V1 or K⊗V2 = ∼ V2 , where Vj may be additionally leads to an isomorphism V1 ⊗K = replaced with an isomorphic vector space Wj . Note that the order of the tensor spaces is changed, but the nontrivial vector spaces are still pairwise isomorphic. 3) The third example will be of broader importance. Fix some k ∈ {1, . . . , d}. Using the isomorphism from Item 1, we may state that V1 ⊗ . . . ⊗ Vk ⊗ . . . ⊗ Vd ∼ = Vk ⊗ V1 ⊗ . . . ⊗ Vk−1 ⊗ Vk+1 ⊗ . . . ⊗ Vd using the permutation 1 ↔ k. Next, we make use of the argument of Lemma 3.21: associativity allows the vector space isomorphism h i ∼ Vk ⊗ V1 ⊗. . .⊗Vk−1 ⊗Vk+1 ⊗. . .⊗Vd . Vk ⊗V1 ⊗. . .⊗Vk−1 ⊗Vk+1 ⊗. . .⊗Vd = The tensor space in parentheses will be abbreviated by O Vj , V[k] := a j6=k

where

(3.17a)

3.2 Tensor Product

65

O

O

.

(3.17b)

Vj ∼ = Vk ⊗ V[k] .

(3.17c)

means

j6=k

j∈{1,...,d}\{k}

Altogether, we have the vector space isomorphism V=

d O a j=1

We notice that V is a tensor space of order d, whereas Vk ⊗ V[k] has order 2. Nevertheless, a part of the tensor structure (the space Vk ) is preserved. The importance of the isomorphism (3.17c) is already obvious from Lemma 3.21 since this allows a reduction to tensor spaces of order two. We shall employ (3.17c) to introduce the matricisation in §5.2. In order to simplify the notation, we shall often replace the ∼ = sign with equality: V = Vk ⊗ V[k] . This allows us to write v ∈ V, as well as v ∈ Vk ⊗ V[k] , whereas the more exact notation is v ∈ V and v ˜ = Φ(v) ∈ Vk ⊗V[k] with the vector space isomorphism Φ : V → Vk ⊗ V[k] . In fact, we shall see in Remark 3.36 that v and v ˜ have different properties. For elementary tensors of V we write v=

d O

v (j) = v (k) ⊗ v[k] ,

where v[k] :=

j=1

O

v (j) ∈ V[k] .

(3.17d)

j6=k

For a general (algebraic) tensor, the corresponding notation is v=

d XO i

(j)

vi

=

j=1

X

(k)

vi

[k]

⊗ vi

[k]

with vi :=

O

i

(j)

vi

∈ V[k] .

j6=k

3.2.6 Rr and Tensor Rank 3.2.6.1 The Set Rr Let Vj (1 ≤ j ≤ d) be vector spaces generating V := combinations of r elementary tensors are contained in ( r ) X vν(1) ⊗ . . . ⊗ vν(d) : vν(j) ∈ Vj Rr := Rr (V) :=

a

Nd

j=1

Vj . All linear

(r ∈ N0 ).

(3.18)

ν=1

Deliberately, we use the same symbol Rr as in (2.6) as justified by Remark 3.38a. S Remark 3.33. V = r∈N0 Rr holds for the algebraic tensor space V. (3.9), v ∈ V is a finite linear combination of elementary Proof. By the definition Ps tensors, i.e., v = ν=1 αν eν for some s ∈ N0 and suitable elementary tensors eν . (1) (d) The factor αν can S be absorbed by the elementary tensor: αν eν =: vν ⊗. . .⊗vν . S Hence v ∈ Rs ⊂ r∈N0 Rr proves V ⊂ r∈N0 Rr ⊂ V. t u

3 Algebraic Foundations of Tensor Spaces

66

Remark 3.34. The sets Rr , which in general are not subspaces, are nested: {0} = R0 ⊂ R1 ⊂ . . . ⊂ Rr−1 ⊂ Rr ⊂ . . . ⊂ V

for all r ∈ N,

(3.19a)

and satisfy the additive property Rr + Rs = Rr+s .

(3.19b)

Proof. Note that R0 = {0} by the empty sum convention. Since we may choose Pr (1) (j) (d) u vr = 0 in ν=1 vν ⊗ . . . ⊗ vν , all sums of r−1 terms are included in Rr . t

3.2.6.2 Tensor Rank (1)

(d)

A nonvanishing elementary tensor vν ⊗ . . . ⊗ vν in (3.18) becomes a rank-1 matrix in the case of d = 2 (cf. §2.2). Remark 2.1 states that the rank r is the Pr (1) (2) smallest integer such that a representation M = is valid (cf. ν=1 vν ⊗ vν (1.3)). This leads to the following generalisation. Nd Definition 3.35 (tensor rank). The tensor rank of v ∈ a j=1 Vj is defined by rank(v) := min {r : v ∈ Rr } ∈ N0 .

(3.20)

The definition makes sense since subsets of N0 have always a minimum. As in (2.6), we can characterise the set Rr by Rr = {v ∈ V : rank(v) ≤ r} .

(3.18∗ )

We shall use the shorter ‘rank’ instead of ‘tensor rank’. Note that there is an ambiguity if v is a matrix as well as a Kronecker tensor (see Remark 3.38b). If necessary, we use the explicit terms ‘matrix rank’ and ‘tensor rank’. Another ambiguity is caused by the isomorphisms discussed in §3.2.5. Remark 3.36. Consider the isomorphic vector spaces U ⊗a V ⊗a W and U ⊗a X with X := V ⊗a W from Example 3.30. Let Φ : U ⊗a V ⊗a W → U ⊗a X be the vector space isomorphism. Then rank(v) and rank(Φ(v)) are in general different. If we identify v and Φ(v), the tensor structure should be explicitly mentioned, e.g., by writing rankU ⊗V ⊗W (v) or rankU ⊗X (v). The same statement holds for Nd V := a j=1 Vj and Vk ⊗a V[k] in (3.17a). The latter remark shows that the rank depends on the tensor structure (U ⊗a X versus U ⊗a V ⊗a W ). However, Lemma 3.39 will prove invariance with respect to tensor space isomorphisms. Practically, it may be hard to determine the rank. It is not only that the rank is a discontinuous function so that any numerical rounding error may change the rank (as for the matrix rank). Even with exact arithmetic computing the rank is, in general, not feasible for large-size tensors because of the next statement.

3.2 Tensor Product

67

Proposition 3.37 (H˚astad [159]). In general, the determination of the tensor rank is an NP-hard problem. If Vj are matrix spaces KIj ×Jj , the tensor rank is also called the Kronecker rank. Remark 3.38. (a) For d = 2, the rank of v ∈ V1 ⊗a V2 is given by r in (3.14) and can be constructed as in the proof of Lemma 3.15. If, in addition, the spaces Vj are finite dimensional, Proposition 3.16 yields an isomorphism between V1 ⊗a V2 and matrices of the size dim(V1 ) × dim(V2 ). Independently of the bases chosen in Proposition 3.16, the matrix rank of the associated matrix coincides with the tensor rank. Nd (b) For Vj = KIj ×Jj the (Kronecker) tensors A ∈ V := j=1 Vj are matrices. In this case the matrix rank of A is completely unrelated to the tensor rank of A. Qd For instance, the identity matrix I ∈ V has (full) matrix rank j=1 #Ij , whereas the Nd tensor rank of the elementary tensor I = j=1 Ij (Ij = id ∈ KIj×Ij ) is equal to 1. (c) For tensors of order d ∈ {0, 1}, the rank is trivial:   0 for v = 0 . rank(v) = 1 otherwise Nd So far, we have only considered algebraic tensor spaces Valg := a j=1 Vj . A Nd Banach tensor space Vtop := k·k j=1 Vj (cf. (3.10)) is the closure (completion) S of r∈N0 Rr (cf. Remark 3.33).We can extend the definition of the tensor rank by5 rank(v) := ∞

if

v∈

d O k·k j=1

Vj \ a

d O

Vj .

j=1

Nd Nd Lemma 3.39 (rank invariance). Let V = j=1 Vj and W = j=1 Wj be either algebraic or topological tensor spaces. (a) Assume that V and W are isomorphic tensor spaces, Nd i.e., the vector spaces Vj ∼ = Wj are isomorphic (cf. Definition 3.29). Let Φ = j=1 φ(j) : V → W be an isomorphism. Then the tensor rank of v ∈ V is invariant under Φ: rank(v) = rank(Φ(v)) (b) Let A =

Nd

j=1

for all v ∈ V.

A(j) : V → W with A(j) ∈ L(Vj , Wj ). Then rank(Av) ≤ rank(v)

for all v ∈ V.

Pr Nd (j) Proof. For Part (b) consider v = ν=1 j=1bν . Since the number of terms in Pr Nd (j) Av = ν=1 j=1(A(j) bν ) is unchanged, rank(Av) ≤ rank(v) follows. For Part (a) we use this inequality twice for Φ and Φ−1 : rank(v) ≥ rank(Φv) ≥ t u rank(Φ−1 Φv) = rank(v). 5

This does not mean, in general, that v can be written as an infinite sum (but see §4.2.3.3 and Theorem 4.133).

3 Algebraic Foundations of Tensor Spaces

68

Corollary 3.40. Remark 3.31 states a tensor space isomorphism Φ between a Nd finite-dimensional tensor space V := j=1 Vj with nj := dim(Vj ) and its Nd nj . Let a be the coefficient tensor of v. K coefficient tensor space W := j=1 Then rank(v) = rank(a). As a consequence, Φ is a bijection between Rr (V) and Rr (W). By the definition of the rank of algebraic tensors, there exists a representation v=

d r O X

vν(j)

with r := rank(v) < ∞.

(3.21)

ν=1 j=1

Other names for (3.21) are ‘rank decomposition’ or ‘rank-retaining decomposition’. Sometimes (3.21) is viewed as the generalisation of the singular-value decomposition (SVD) to d ≥ 3. From the view point of numerical analysis the opposite is true. SVD yields orthonormal systems which is a good basis for stable computations. Concerning the vectors in (3.21) compare Remark 3.50. The singular values in SVD clearly indicate what terms can be omitted for a truncation. Analogously, the eleNd Nd (j) (j) mentary tensor j=1 vν may be rewritten as eν := σν j=1 vˆν with normalised P (j) (j) (j) vectors vˆν := vν /kvν k. However, there may be a partial sum v0 = µ eνµ of small size kv0 k involving terms with large σνµ . Thus v0 is a perfect candidate for truncation in spite of the large σνµ (cf. §9.4). The following lemma offers a necessary condition for (3.21). The proof is known from Lemma 3.15. (j)

Lemma 3.41. Assume r = rank(v). Using vν in (3.21), define the elementary tensors O O Vk vν[j] := vν(k) ∈ V[j] = a k∈{1,...,d}\{j}

k∈{1,...,d}\{j} [j]

(cf. (3.17d)). Then (3.21) implies that the tensors {vν : 1 ≤ ν ≤ r} are linearly (j) independent for all 1 ≤ j ≤ d, while vν 6= 0 for all ν and j. [1]

Proof. Let j = 1. Assume that the elementary tensors {vν : 1 ≤ ν ≤ r} are [1] linearly dependent. Without loss of generality, suppose that vr may be expressed Pr−1 [1] [1] by the other tensors: vr = ν=1 βν vν . Then v=

r X

vν(1) ⊗ vν[1] =

ν=1

=

r−1  X

r−1  X

 vν(1) + βν vr(1) ⊗ vν[1]

ν=1

 vν(1) + βν vr(1) ⊗ vν(2) ⊗ . . . ⊗ vν(d)

ν=1 (j)

implies rank(v) < r. Similarly, if vν = 0, the j-th term can be omitted implying t u the contradiction rank(v) < r. Analogously, j > 1 is treated.

3.2 Tensor Product

69

Remark 3.42. Note that Lemma 3.41 states linear independence only for the [1] (1) tensors vν , 1 ≤ ν ≤ r. The vectors vν are nonzero, but they may be linearly dependent. An example is the tensor in (3.24), which has rank 3, while all subspaces (j) Uj = span{vν : 1 ≤ ν ≤ 3} have only dimension 2. For symmetric tensors (cf. §3.5) a symmetric rank will be defined in Definition 3.83.

3.2.6.3 Expansion with Respect to a Basis Below we consider tensors belonging to a subspace U ⊂ V that is also a tensor Nd space U = a j=1 Uj generated by subspaces Uj ⊂ Vj (possibly Uj = Vj ). The statement u ∈ U ⊂ V can be understood in two ways. It can be regarded as P N (j) (j) a tensor u = ν j vν in V, i.e., vν ∈ Vj , which belongs to U. On the Nd other hand, the definition of u ∈ U = a j=1 Uj requires a representation by P N (j) (j) u = ν j uν with uν ∈ Uj (cf. (3.9) and span property (i) in Proposition 3.22b). Both interpretations are equivalent as will be proved in §6. Lemma 3.43. Let Uj ⊂ Vj be subspaces generating the tensor (sub)space U=

d O a

Uj ⊂ V.

j=1

P N (j) (j) Assume that u = ν j vν with vν ∈ Vj satisfies u ∈ U. Then there are P N (j) (j) uν ∈ Uj such that u = ν j uν . For later use we introduce the following notation (3.22a). Let Ud ⊂ Vd be a finite-dimensional subspace and consider a tensor6 v ∈ V1 ⊗ V2 ⊗ . . . ⊗ Vd−1 ⊗ Ud = V[d] ⊗ Ud (d)

and fix a basis {bi } of Ud . Any elementary tensor in V[d] ⊗ Ud can be written as P [d] P [d] (d) (d)
= i vi ⊗ bi with vi := αi v[d] . Using this v[d] ⊗ v (d) = v[d] ⊗ i αi bi P [d] (d) representation for all terms of an algebraic tensor v = i vi ⊗ vi ∈ V[d] ⊗ Ud (d)

(vi

∈ Ud by Lemma 3.43) we obtain a representation of the form v=

X i

(d)

vhii ⊗ bi

with vhii ∈ V[d] =

d−1 O

Vj .

(3.22a)

j=1 (d)

The tensors vhii ∈ V[d] are uniquely determined by the choice of the basis {bi } and the index i. An explicit description of vhii will be given in Remark 3.65. 6

We may assume as well that v ∈ V[j] ⊗ Uj for j 6= d. For the sake of a simple notation we choose j = d.

3 Algebraic Foundations of Tensor Spaces

70

In the case of d = 1, V[d] is the field K and the expansion coefficients vhii are scalars. Nd−2
If d ≥ 2 and v ∈ Vj ⊗ Ud−1 ⊗ Ud , we may fix bases of Ud−1 and Ud . Nd−2
j=1 Then vhii ∈ j=1 Vj ⊗ Ud−1 in (3.22a) can again be expanded with respect to Prd−1 (d−1) (d−1) the basis {bi vhiihmi ⊗ bm } of Ud−1 and we obtain vhii = m=1 and thus v=

X

(d)

(d−1) vhiihmi ⊗ bm ⊗ bi .

i,m

We introduce the shorter notation vhi,mi := vhiihmi ∈

d−2 O a

Vj .

(3.22b)

j=1

3.2.6.4 Dependence on the Field Nd So far, the field K has been fixed. Note that the ‘real’ tensor space VR := j=1 Rnj Nd can be embedded into the ‘complex’ tensor space j=1 Cnj . Concerning the tensor rank, the following problem arises. Let v ∈ VR be a ‘real’ tensor. The tensor rank (j) is the minimal number r = rR of terms in (3.21) with vν ∈ Rnj . We may also ask for the minimal number r = rC of terms in (3.21) under the condition that (j) vν ∈ Cnj . Since RIj ⊂ CIj , the inequality rC ≤ rR is obvious, which already proves statement (a) below. Nd Proposition 3.44. Let VR = j=1 Vj be a tensor space over the field R. Define Nd VC = j=1 Vj,C as the corresponding complex version over C. Let rR (v) be the (real) tensor rank within VR , while rC (v) is the (complex) tensor rank within VC . (a) For any v ∈ VR , the inequality rC (v) ≤ rR (v) holds. (b) (vector and matrix case) If 0 ≤ d ≤ 2, rC (v) = rR (v) holds for all v ∈ VR . Nd (c) (proper tensor case) If d ≥ 3 and VR = j=1 Vj is a nondegenerate tensor space (cf. Definition 3.25), there are v ∈ VR with strict inequality rC (v) < rR (v). Proof. (i) Part (a) is already proved above. (ii) If d = 1, Remark 3.38c shows that rC = rR . If d = 2, v ∈ VR may be interpreted as a real-valued matrix M ∈ RI1 ×I2 . By Remark 2.2, the matrix rank is independent of the field: rC (v) = rR (v). (iii) Example 3.48 below presents a counterexample for which rC (v) < rR (v) in the case of d = 3. It may be easily embedded into tensor spaces with larger d by u setting v0 := v ⊗ a4 ⊗ a5 ⊗ . . . ⊗ ad with arbitrary 0 6= aj ∈ Vj (4 ≤ j ≤ d). t Bergman [29] was the first describing the rank dependence on K. Another dependence on the field will be mentioned in §3.2.6.5.

3.2 Tensor Product

71

3.2.6.5 Maximal, Typical, and Generic Ranks Nd The sequence R0 ⊂ R1 ⊂ . . . ⊂ V := a j=1 Vj in (3.19a) is properly increasing for infinite-dimensional tensor spaces. On the other hand, for finite-dimensional tensor spaces there must be a smallest rmax so that Rr = Rrmax for all r ≥ rmax . As a consequence, V = Rrmax , while Rrmax −1 $ V. This number rmax = rmax (V) is called the maximal rank V (see (2.5) for the matrix case). Lemma 3.45. Let nj := dim(Vj ) < ∞ for 1 ≤ j ≤ d. Then ! d Y Y rmax ≤ nj / max ni = min j=1

1≤i≤d

1≤i≤d

nj

(3.23)

j∈{1,...,d}\{i}

describes an upper bound of the maximal rank. For equal dimensions nj = n, this is rmax ≤ nd−1 . The statement Q can be extended to the case that one space Vi is infinite dimensional: rmax ≤ j6=i nj . Proof. After a permutation of the factors we may assume that nd = max1≤i≤d ni . Consider the full representation (3.16) of any v ∈ V: v=

(d)

(1)

X

a[i1 , . . . , id ] bi1 ⊗ . . . ⊗ bid

i1 ,...,id−1 ,id

! =

X

(1) bi1

⊗ ... ⊗

(d−1) bid−1

⊗

i1 ,...,id−1

X

(d) a[i1 , . . . , id ] bid

.

id

The sum in the last line is taken over r¯ := Rr¯ = V proves rmax ≤ r¯.

Qd−1 j=1

nj elementary tensors. Hence t u

The true value rmax may be clearly smaller than the bound from above. For instance, Kruskal [204] proves   max{n1 , n2 } rmax = min{n1 , n2 } + min n1 , n2 , for V = Rn1 ⊗ Rn2 ⊗ R2 , 2 rmax = 5 for V = R3 ⊗ R3 ⊗ R3 . For equal dimensions n := n1 = . . . = nd , Howell [169] proves7 the asymptotic upper bound X d rmax ≤ d (n − 2ν)d−1 = nd−1 + O(nd−2 ). 2(d − 1) n−d 0≤ν≤

7

2

Howell considers tensor spaces over a ring instead of a field K. The cited result holds for rings which are principal ideal domains. Note that fields are in particular principal ideal domains.

3 Algebraic Foundations of Tensor Spaces

72

A general lower bound is proved by Howell [169] and Pamfilos [243]:  Y  . d d X 1−d+ nj rmax ≥ nj . j=1

j=1

For nj = n, this inequality implies rmax > nd−1 /d. Concerning the maximal rank, there is a remarkable difference to the matrix case. Random matrices and their rank are described in Remark 2.5. Random tensors may attain more than one rank with positive probability. Such ranks are called typical. Kruskal [204] proves that {2, 3} are the typical ranks of V = R2 ⊗R2 ⊗R2 , while 3 is the maximal rank. Note that such results also depend on the field. For algebraically closed fields as C there is only one typical rank (cf. Strassen [278], Comon–Golub– Lim–Mourrain [63]). In this case, the typical rank becomes the generic rank since almost all tensors have this rank (cf. Comon et al. [62]).

3.2.6.6 Examples As an illustration, we consider the tensor v ∈ V ⊗ V defined by v = a ⊗ a + b ⊗ a + a ⊗ b + b ⊗ a, where a, b ∈ V are linearly independent. The given representation proves v ∈ R4 and rank(v) ≤ 4. The fact that all four terms are linearly independent does not imply rank(v) = 4 . In fact, another representation is v = (a + b) ⊗ (a + b) proving rank(v) = 1 since v 6= 0 excludes rank(v) = 0. For later use we exercise the determination of the rank for a special tensor. Lemma 3.46. (a) Let Vj (1 ≤ j ≤ 3) be vector spaces of dimension ≥ 2 and consider the tensor space V := V1 ⊗ V2 ⊗ V3 . For linearly independent vectors v (j) , w(j) ∈ Vj define v := v (1) ⊗ v (2) ⊗ w(3) + v (1) ⊗ w(2) ⊗ v (3) + w(1) ⊗ v (2) ⊗ v (3) .

(3.24)

Then rank(v) = 3 holds; i.e., the given representation is already the shortest one. (b) The generalisation to general d yields the tensor    ! j−1 d d X O O  sd := v (k)  . v (k) ⊗ w(j) ⊗  j=1

k=1

(3.25)

k=j+1

If v (j) , w(j) ∈ Vj are linearly independent for each 1 ≤ j ≤ d, the tensor satisfies rank(sd ) = d.

3.2 Tensor Product

73

Proof. The inductive proof of the more general Part (b) is due to Buczy´nski– Landsberg [45]. For d = 1, 2, the statement is trivial. Let the induction hypothesis rank(sd−1 ) = d−1 hold. By the definition of sd , the inequality r := rank(sd ) ≤ d holds; i.e., there are ujν ∈ Vj with sd =

r O d X

uν(j) .

ν=1 j=1

Let ϕ, ψ ∈ Vd0 be dual to v (d) , w(d) ; i.e., ϕ(v (d) ) = ψ(w(d) ) = 1 and ψ(v (d) ) = ϕ(w(d) ) = 0. Setting  (1) v ⊗ . . . ⊗ v (j−1) ⊗ w(j) ⊗ v (j+1) ⊗ . . . ⊗ v (d−1) for j ≤ d − 1, ˆj := v v (1) ⊗ . . . ⊗ v (d−1) for j = d, we observe that ϕ(sd ) = sd−1 and ψ(sd ) = v ˆd . . Using the notation of Remark 3.64b — the functionals are applied to the last factor from Vd — we obtain on the other hand that ϕ(sd ) = sd−1 =

r X

αν

ν=1

d−1 O

uν(j) ,

ψ(sd ) = v ˆd =

r X ν=1

j=1

βν

d−1 O

u(j) ν .

j=1

(d)

(d)

holds with αν := ϕ(uν ) and βν := ψ(uν ). The inductive hypothesis implies rank(sd−1 ) = d − 1 so that r ≥ d − 1. r Since sd−1 and v ˆd are linearly independent, the vectors α = (αν )ν=1 and r β = (βν )ν=1 must also be linearly independent. As β 6= 0, we may assume without loss of generality that βr does not vanish. Set λ := αr /βr and vd = ˆ sd−1 := sd−1 − λˆ

r X ν=1

(αν − λβν )

d−1 O j=1

u(j) ν =

r−1 X

(αν − λβν )

ν=1

d−1 O

uν(j) .

j=1

The latter (r − 1)-term representation shows that rank(ˆ sd−1 ) ≤ r − 1. However, we verify that    !   j−1 d−1 d X O O λ (j)  ˆ sd−1 = v (k) ⊗ w(j) − v ⊗ v (k)  d − 1 j=1 k=1

k=j+1

λ is of the form (3.25) with v (j) and w(j) − d−1 v (j) being linearly independent. Hence the inductive hypothesis states that rank(ˆ sd−1 ) = d − 1, which implies r ≥ d. Together with the inequality r ≤ d from the beginning, the statement is proved. t u Nd Nd Exercise 3.47. Consider v = j=1 vj + j=1 wj with nonvanishing vectors vj and wj . Show that rank(v) ≤ 1 holds if and only if vj and wj are linearly dependent for at least d − 1 indices j ∈ {1, . . . , d}. Otherwise, rank(v) = 2.

For the distinction of rankR and rankC we give the following example.

3 Algebraic Foundations of Tensor Spaces

74

Example 3.48. Let a, b, c, a0 , b0 , c0 ∈ Rn with n ≥ 2 such that (a, a0 ) , (b, b0 ), (c, c0 ) are pairs of linearly independent vectors. The real part of the complex tensor (a + ia0 ) ⊗ (b + ib0 ) ⊗ (c + ic0 ) ∈ Cn ⊗ Cn ⊗ Cn has a representation v=

1 2

(a + ia0 )⊗(b + ib0 )⊗(c + ic0 ) +

1 2

(a − ia0 )⊗(b − ib0 )⊗(c − ic0 ) ∈ R2

in Cn ⊗ Cn ⊗ Cn . Exercise 3.47 proves that rankC (v) = 2 in Cn ⊗ Cn ⊗ Cn . Multilinearity yields the representation v = a ⊗ b ⊗ c − a0 ⊗ b0 ⊗ c − a0 ⊗ b ⊗ c0 − a ⊗ b0 ⊗ c0 within Rn ⊗ Rn ⊗ Rn . We verify that also v = (a − a0 ) ⊗ (b + b0 ) ⊗ c + a0 ⊗ b ⊗ (c − c0 ) − a ⊗ b0 ⊗ (c + c0 ) .

(3.26)

An additional reduction is not possible so that rankR (v) = 3 > 2 = rankC (v) is valid. Proof. Assume that v = A ⊗ B ⊗ C + A0 ⊗ B 0 ⊗ C 0 . Applying suitable functionals to the first two components, we see that C, C 0 ∈ span{c, c0 }. If C and C 0 are linearly dependent, this leads to a quick contradiction. So assume that they 0 are linearly independent and choose functionals γ ∈ (Rn ) with γ(C) = 1 and 0 γ(C ) = 0. Note that at least two of the numbers γ(c), γ(c − c0 ), γ(c + c0 ) are nonzero. Hence application of id ⊗ id ⊗ γ to v = A ⊗ B ⊗ C + A0 ⊗ B 0 ⊗ C 0 yields A ⊗ B with matrix rank equal to 1, while the result for v in (3.26) is a linear combination of (a − a0 ) ⊗ (b + b0 ) and a0 ⊗ b, a ⊗ b0 , where at least two terms are present. One verifies that the matrix rank is 2. This contradiction excludes rankR (v) = 2. It is even easier to exclude the smaller ranks 1 and 0. t u The next example of different n-term representations over the real or complex field is taken from Mohlenkamp–Monz´on [230] and Beylkin–Mohlenkamp [31].
Pd d Example 3.49. Consider the function f (x1 , . . . , xd ) := sin j=1 xj ∈ ⊗ V for V = C(R). If C(R) is regarded as a vector space over K = C, rank(f ) = 2 holds and is realised by ! d d d Pd X 1 Pd 1 1 O ixj 1 O −ixj sin xj = ei j=1 xj − e−i j=1 xj = e − e . 2i 2i 2i j=1 2i j=1 j=1 If C(R) is considered as vector space over K = R, the following representation needs d terms: ! ! ! d d ν−1 d X X O O sin(xj +αj −αν ) sin(xj +αj −αν ) sin xj = ⊗ sin(xν ) ⊗ sin(αj −αν ) sin(αj −αν ) j=1

ν=1

j=1

j=ν+1

with arbitrary αj ∈ R satisfying sin(αj − αν ) 6= 0 for all j 6= ν. Lim [212, §15.3] gives an example of rR (v) < rQ (v) involving the field Q of rational numbers.

3.2 Tensor Product

75

2, the r-term representation (3.21) corresponds to a matrix In the Prcase of d = I×J M = i=1 ai bT with vectors ai ∈ KI and bi ∈ KJ . Here the minimal i ∈ K (tensor and matrix) rank r is attained if the vectors {ai } and the vectors {bi } are linearly independent. PrMoreover,T the singular-value decomposition yields the particular form M = i=1 σi ai bi with σi > 0 and orthonormal ai and bi . Generalisations of these properties to d ≥ 3 are not valid. Remark 3.50. (a) A true generalisation of the singular-value decomposition to Pr Nd Nd (j) nj with ∈ V := d dimensions would be v = j=1 vν j=1 K ν=1 σν (j) r = rank(v), orthonormal vectors {vν : 1 ≤ ν ≤ r} for all 1 ≤ j ≤ d, and σν > 0. Unfortunately, such tensors form only a small subset of V; i.e., in general, v ∈ V does not possess such a representation. (j)

(b) Even the requirement that the vectors {vν : 1 ≤ ν ≤ r} be linearly independent cannot be satisfied in general. Proof. The tensors a ⊗ a ⊗ a + a ⊗ b ⊗ b cannot be reduced to rank ≤ 1, although the first factors are equal. This proves Part (b), while (b) implies (a). u t

3.2.6.7 Application: Strassen’s Algorithm Standard matrix-matrix multiplication of two n×n matrices costs 2n3 operations. A reduction to 4.7nlog2 7 = 4.7n2.8074 proposed by Strassen [277] is based on the fact that two 2×2 block matrices can be multiplied as follows:      a1 a2 c1 c2 b1 b2 = , ai , bi , ci submatrices (3.27) a3 a4 c3 c4 b3 b4 with c1 = m1 + m4 − m5 + m7 ,

c2 = m2 + m4 ,

c4 = m1 + m3 − m2 + m6 ,

c3 = m3 + m5 ,

m1 = (a1 + a4 )(b1 + b4 ),

m2 = (a3 + a4 )b1 ,

m3 = a1 (b2 − b4 ),

m4 = a4 (b3 − b1 ),

m5 = (a1 + a2 )b4 ,

m6 = (a3 − a1 )(b1 + b2 ),

m7 = (a2 − a4 )(b3 + b4 ),

where only 7 multiplications of block matrices are needed. The entries of the following tensor m ∈ ⊗3 K4 are defined by cν =

4 X

mµλν aµ bλ

(1 ≤ ν ≤ 4).

(3.28a)

µ,λ=1

For instance for ν = 1, the identity c1 = a1 b1 +a2 b3 shows that m111 = m231 = 1, and mµλ1 = 0 otherwise. Assume a representation of m using r terms:

3 Algebraic Foundations of Tensor Spaces

76

m=

3 r O X

(j)

mi .

(3.28b)

i=1 j=1

Insertion into (3.28a) yields cν =

r 4 X X

(1)

(2)

(3)

mi [µ] mi [λ] mi [ν] aµ bλ

i=1 µ,λ=1

=

r X i=1

(3) mi [ν]

4 X

! (1) mi [µ] aµ

µ=1

4 X

! (2) mi [λ] bλ

,

λ=1

i.e., only r multiplications of block matrices are needed. Algorithm (3.27) corresponds to a representation (3.28b) with r = 7, implying rank(m) ≤ 7. Theorem 3.51. rank(m) = rank(m) = 7 (see Definition 9.14 for rank). Proof. The first equation rank(m) = 7 is proved by Winograd [304]. For the stronger statement rank(m) = 7 see Landsberg [206, Theorem 11.0.2.11]. t u The general matrix-matrix multiplication is discussed in Example 3.72.

3.3 Linear and Multilinear Mappings Now we consider linear mappings defined on V ⊗a W or, more generally, on Nd a j=1 Vj . The image space might be the field K — then the linear mappings are called linear forms or functionals — or another tensor space. In §3.3.1 we shall justify that it suffices to define a (multi-)linear mapping by its values for elementary tensors. Often a linear mapping ϕk : Vk → Wk for a Nd fixed k is extended to a linear mapping defined on V := a j=1 Vj . This leads to an embedding explained in §3.3.2. Functionals are a subclass of linear maps. Nevertheless, there are special properties which are addressed in §3.3.2.4.

3.3.1 Definition on the Set of Elementary Tensors If a linear mapping φ ∈ L(V, W ) has to be defined on a vector space V spanned by a basis {vj }, it is sufficient to describe the images φ(vj ) (cf. Remark 3.5). In the case of a linear mapping φ ∈ L(V, X) we know that V is spanned Nd (j) by elementary tensors e := . Hence it is sufficient to know the image j=1 v φ(e) ∈ X of the elementary tensors. Note that φ(e) cannot take arbitrary values since elementary tensors are not linearly independent.

3.3 Linear and Multilinear Mappings

77

Lemma 3.52. A linear mapping φ : V := by its values for elementary tensors.

a

Nd

j=1

Vj → X is completely defined


Nd (j) Remark 3.53. Often the value of φ is given in terms of the vectors j=1 v v (1) , . . . , v (d) . That means there is a mapping Φ : V1 × . . . × Vd → X with ! d O (j) = Φ(v (1) , . . . , v (d) ) for all v (j) ∈ Vj . φ v (3.29) j=1

In this case we have to require that Φ : V1 × . . . × Vd → U be multilinear. Then the ‘universality of the tensor product’ formulated in Proposition 3.23 implies that (3.29) defines a unique linear mapping φ. Example 3.54. Let ϕ : V → U be a linear mapping. Then the definition φ (v ⊗ w) := ϕ(v) ⊗ w

for all v ∈ V, w ∈ W

defines a unique linear mapping φ from V ⊗a W into X = U ⊗a W. In this example the mapping Φ : V × W → X is given by Φ(v (1) , v (2) ) = ϕ(v (1) ) ⊗ v (2) . Linearity of ϕ implies multilinearity of Φ.

3.3.2 Embeddings 3.3.2.1 Embedding of Spaces of Linear Maps Next we consider two d-tuples (V1 , . . . , Vd ) and (W1 , . . . , Wd ) of vector spaces Nd Nd and the corresponding tensor spaces V := a j=1 Vj and W := a j=1 Wj . Since L(Vj , Wj ) ( 1 ≤ j ≤ d) are again vector spaces, we can build the tensor space L :=

d O a

L(Vj , Wj ) .

j=1

Nd Elementary tensors of L are of the form Φ = j=1 ϕ(j) with ϕ(j) ∈ L(Vj , Wj ). In §1.1.2 we called Φ the Kronecker product8 of the mappings ϕ(j) . They have a natural interpretation as mappings of L(V, W) via (1.4b): O  O d d Φ v (j) = ϕ(j) (v (j) ) ∈ W j=1

j=1

for any

d O

v (j) ∈ V.

(3.30a)

j=1

At least, this term is used for matrix spaces L(Vj , Wj ) with Vj = Knj and Wj = Kmj . As mentioned in §1.6, the attribution to Kronecker is questionable. 8

3 Algebraic Foundations of Tensor Spaces

78

Note that (3.30a) defines Φ for all elementary tensors of V. The right-hand side in (3.30a) is multilinear in v (j) . Therefore Remark 3.53 proves that Φ can be uniquely extended to Φ ∈ L(V, W). Linear combinations of such elementary tensors Φ are again elements of L(V, W). This leads to the embedding described below. Nd Proposition 3.55. Let Vj , Wj , V, W be as above. We identify a j=1 L(Vj , Wj ) with a subspace of L(V, W) via (3.30a): L=

d O a

L(Vj , Wj ) ⊂ L(V, W).

(3.30b)

j=1

If at least two of the vector spaces Vj are infinite dimensional and dim(W) > 0, L is a proper subspace. On the other hand, if all vector spaces Vj are finite dimensional, the spaces coincide: d O a

L(Vj , Wj ) = L(V, W).

(3.30c)

j=1

Proof. (a) Definition (3.30a) yields a linear mapping Υ : L → L(V, W). Υ describes an embedding if and only if Υ is injective. For this purpose, we use induction on d and start with d = 2. We have to disprove Υ (Λ) = 0 for 0 6= Λ ∈ L = L(V1 , W1 ) ⊗a L(V2 , W2 ). If Υ (Λ) = 0, the interpretation (3.30a) of Λ produces the zero mapping Υ (Λ) in L(V, W). By Lemma 3.15, there Pr (2) (1) (2) exists a representation Λ = with linearly independent ϕν ν=1 ϕν ⊗ ϕν (1) and 9 ϕ1 6= 0. Application to any v := v (1) ⊗ v (2) yields 0 = Λ(v) =

r X

(2) (1) ) ⊗ ϕ(2) ϕ(1) ) ∈ W = W 1 ⊗a W 2 . ν (v ν (v

ν=1 (1)

Fix v (1) such that ϕ1 (v (1) ) 6= 0. Then there is some functional χ ∈ W10 with (1) χ(ϕ1 (v (1) )) 6= 0. Application of χ ⊗ id to Λ(v) yields 0 = (χ ⊗ id) (Λ(v)) =

r X

(2) ) ∈ W2 αν ϕ(2) ν (v

ν=1 (1)

(cf. Remark 3.64) with αν := χ(ϕν (v (1) )). The choice of v (1) and χ ensures Pr (2) (2) that α1 6= 0. Linear independence of {ϕν } implies ν=1 αν ϕν 6= 0. Hence P (2) r (2) there exists some v (2) ∈ V2 with ) 6= 0 in contradiction to ν=1 αν ϕν (v
(1) (2) 0 = (χ ⊗ id) Λ(v)(v ⊗ v ) . This proves injectivity of Υ for d = 2. 9

(1)

In fact, ϕν

(1)

are linearly independent, but only ϕ1

6= 0 is needed.

3.3 Linear and Multilinear Mappings

79

Let the assertion be valid for d − 1. Represent Λ in the form r X [1] ϕ(1) with r = rankV1 ⊗V[1] (Λ) Λ= ν ⊗ ϕν ν=1 [1]

(cf. Remark 3.36). As stated in Lemma 3.41, {ϕν } is linearly independent, while Nd (1) [1] ϕν 6= 0. By induction, ϕν ∈ L(V[1] , W[1] ) holds with V[1] := a j=2 Vj and Nd W[1] := a j=2 Wj . Now all arguments from the inductive start d = 2 can be repeated. (b) Assume that all vector spaces Vj and Wj are finite dimensional. Then both Qd L and L(V, W) have the same dimension j=1 (dim Vj dim Wj ). Therefore the inclusion (3.30b) implies (3.30c). (c) Now only the spaces Vj are required to be finite dimensional. Fix some Φ ∈ L(V, W). Since V is finite dimensional, also the image Φ(V) ⊂ W is Nd ˆ ˆ j := U min (Φ(V)) ⊂ Wj and W ˆ = finite dimensional. Hence W j j=1 Wj are ˆ part (b) proves finite dimensional (Ujmin (·) defined in §6). Since Φ ∈ L(V, W), Nd ˆ j ) ⊂ a Nd L(Vj , Wj ) = L . that Φ ∈ a j=1 L(Vj , W j=1 (d) If at least two vector spaces Vj are infinite dimensional and dim(W) > 0, the counterexample L(`0 ⊗a `0 , K) of Example 3.63 can be embedded into the u t space L(V, W). Proposition 3.55 does not state what happens if exactly one space Vj is infinite dimensional. In this case the answer depends on the dimensions of Wj . If at most one of these spaces is infinite dimensional, L = L(V, W) holds; otherwise L $ L(V, W). Two canonical cases are explained below. Remark 3.56. (a) Let V = V1 ⊗ V2 and W = W1 ⊗ W2 with finite-dimensional (j) (2) spaces V2 and W2 . Let {vν : ν ∈ Bj } be bases of Vj , while {wk : 1 ≤ k ≤ n} with n := dim(W2 ) < ∞ is a basis of W2 . A linear map Φ ∈ L(V, W) is char(2) (1) acterised by the images wν,µ := Φ(vν ⊗ vµ ). There is a unique representation of Pn (j) (2) the form wν,µ = k=1 wν,µ,k ⊗wk (cf. (3.22a)). Let {ˆ vν : ν ∈ Bj } (j = 1, 2) be the dual bases. Then the mapping ( P (1) n X X ϕµ,k (v) := ν∈B1 vˆν (v) wν,µ,k for v ∈ V1 , 0 0 ϕµ ,k ⊗ψµ ,k ∈ L with (2) (2) ψµ0 ,k (v) := vˆν (v) wk for v ∈ V2 k=1 µ0 (1)

coincides with Φ. Note that vˆν (v) is nonzero for only finitely many ν ∈ B1 so that ϕµ,k ∈ L(V1 , W1 ). Hence, L = L(V, W). (b) Let V = V1 ⊗ K and W = W1 ⊗ W2 with infinite-dimensional spaces V1 , W1 , W2 . Let {bν } be a basis of V1 and define Φ ∈ L(V, W) by Φ(bν ⊗ 1) = wν with rank(wν ) ≥ ν. Any mapping φ ∈ L has some finite rank rφ so that rank(φ(bν ⊗ 1)) ≤ rφ (cf. Exercise 3.60). This shows φ(bν ⊗ 1) 6= Φ(bν ⊗ 1) for ν > rφ and proves Φ ∈ / L, i.e., L $ L(V, W).

3 Algebraic Foundations of Tensor Spaces

80

3.3.2.2 Notational Details In Example 3.54 we use the mapping v ⊗ w 7→ ϕ(v) ⊗ w, which can be generalised to d   O Φ: v (j) 7→ v (1) ⊗ . . . ⊗ v (k−1) ⊗ ϕ v (k) ⊗ v (k+1) ⊗ . . . ⊗ v (d) . j=1

The latter mapping can be formulated as  (j) d O ϕ = ϕ, Wj := W (j) ϕ Φ= with ϕ(j) = id, Wj := Vj j=1

for j = k, for j 6= k,

(3.31a)

or Φ = id ⊗ . . . ⊗ id ⊗ ϕ ⊗ id ⊗ . . . ⊗ id. Since such a notation is rather cumbersome, we identify10 ϕ and Φ as stated below. Notation 3.57. (a) Let k ∈ {1, . . . , d} be fixed. A mapping ϕ ∈ L(Vk , Wk ) is synonymously interpreted as Φ in (3.31a). This defines the embedding L(Vk , Wk ) ⊂ L(V, W) where V =

a

Nd

j=1

Vj and W =

a

Nd

j=1

(3.31b)

Wj with Wj = Vj for j 6= k.

(b) Let α ⊂ {1, . . . , d} be a nonempty subset. Then the embedding O L(Vk , Wk ) ⊂ L(V, W) a k∈α

is defined analogously by inserting identity N maps for all j ∈ {1, . . . , d}\α. If we want to distinguish the map ϕα ∈ a k∈α L(Vk , Wk ) from Φ ∈ L(V, W), we denote Φ by ϕα ⊗ idαc N which indicates that the identity idαc is applied to a k∈αc Vk corresponding to the complement αc = {1, . . . , d}\α. If α = {j}, the notation idαc = id[j] is used (cf. §5.2.1). We repeat the composition rule, which in §4.6.3 is formulated for Kronecker matrices. Nd Remark 3.58. Composing the elementary tensors Ψ = j=1 ψ (j) ∈ L(U,V) and Nd Φ = j=1 ϕ(j) ∈ L(V,W) yields d   O ϕ(j) ◦ ψ (j) ∈ L(U, W). Φ◦Ψ = j=1 10

The identifications (3.31a–c) are standard in other fields. If we use the multi-index notation ∂ n f = ∂xn1 ∂yn2 ∂zn3 f for the partial derivative, the identities are expressed by ∂yn2 = ∂zn3 = id ∂ n1 if, e.g., n2 = n3 = 0. However, more often ( ∂x ) f (x, y, z) is written omitting the identities n1 n1 in ∂ /∂x ⊗ id ⊗ id.

3.3 Linear and Multilinear Mappings

81

An obvious conclusion are the next statements (use that id commutes with any map). Conclusion 3.59. (a) Let Φ and Ψ as above and assume that ϕ(j) and ψ (j) commute: ϕ(j) ◦ ψ (j) = ψ (j) ◦ ϕ(j) (1 ≤ j ≤ d). Then also Φ ◦ Ψ = Ψ ◦ Φ holds. ϕ(β) ∈ L(Vβ , Wβ ) belong to disjoint subsets (b) Let ϕ(α) ∈ L(Vα , Wα ) and N α, β of {1, . . . , d} and Vα = a j∈α Vj , etc. (cf. §5.2.1). Then the mappings Φ = ϕ(α) ⊗ idαc and Ψ = ϕ(β) ⊗ idβ c commute. Moreover, the composition is Φ ◦ Ψ = ϕ(α) ⊗ ϕ(β) ⊗ id{1,...,d}\(α∪β) . (c) Let ϕ(j) ∈ L(Vj , Wj ) for all 1 ≤ j ≤ d. Then ϕ(1) ◦ ϕ(2) ◦ . . . ◦ ϕ(d) interpreted Nd (j) by (3.31b) is equal to j=1 ϕ . 3.3.2.3 Kronecker Matrices and Rank Amplification Now we assume that Vj = Knj and Wj = Kmj . In this case the linear maps ϕ(j) ∈ L(Vj , Wj ) can be realised by matrices A(j) ∈ Knj ×mj . The tensor product Nd Φ = j=1 ϕ(j) ∈ L(V,W) corresponds to the Kronecker matrix A=

d O

A(j) .

j=1

Definition (3.30a) becomes Av =

d  O j=1

A(j) v (j)



∈

d O

Kmj

j=1

for v =

d O j=1

v (j) ∈

d O

Kn j .

j=1

Pr Nd (j) The (tensor) rank r is again the smallest integer such that A = ν=1 j=1 Aν . Note that the tensor rank of A is not related to the matrix rank of A (cf. Remark 3.38b). In such a case we explicitly distinguish the matrix rank and tensor rank. Otherwise, rank(·) always means the tensor rank. Exercise 3.60. If rank(A) = r and rank(v) = s, then rank(Av) ≤ rs. The bound rs from above yields an upper bound, which may be too pessimistic. Obviously, rs may be replaced with min{rs, rmax (W)}, where rmax (W) is the Nd maximal rank of W = j=1 Kmj (cf. §3.2.6.5). But even when rs < rmax (W), there are cases for which rank(Av) may be strictly smaller than in the exercise. For this purpose, we define the rank amplification by ramp (A) := max v6=0

rank(Av) , rank(v)

3 Algebraic Foundations of Tensor Spaces

82

which implies that rank(Av) ≤ ramp (A) · rank(v). Exercise 3.60 states that ramp (A) ≤ rank(A). Example 3.61. For V = Kn ⊗ Km and W = Km ⊗ Kn with n, m ≥ 2 define T ∈ L(V, W) by the transposition from Remark 3.20; i.e., T (v ⊗ w) = w ⊗ v. Then ramp (T ) = 1, whereas rank(T ) > 1. Proof. The proof of rank(T v) = rank(v) is trivial. For proving rank(T ) > 1 it is sufficient the case n = m = 2. V = K2 ⊗ K2 is isomorphic to v1  u1  to consider T 4 ∼ K via u2 ⊗ v2 = [u1 v1 u2 v1 u1 v2 u2 v2 ] . T corresponds to the matrix   1000

0 0 1 0  0 1 0 0 , which cannot be equal to a rank-1 tensor product A⊗B (A, B ∈ K2×2 ) 0001

u since a11 b11 = 1 and a11 b22 = 0 imply b22 = 0, in contradiction to a22 b22 = 1. t

3.3.2.4 Embedding of Linear Functionals The previous linear mappings become functionals if the image space Wj is the trivial vector space K. However, the difference is seen from the following example. Consider the mapping ϕ : V → U from Example 3.54 and the induced mapping Φ : v ⊗ w 7→ ϕ(v) ⊗ w ∈ U ⊗a W For U = K, the mapping ϕ is a functional and the image of Φ belongs to K⊗a W As K ⊗a W is isomorphic11 to W , it is standard to simplify U ⊗a W to W (cf. Remark 3.26a). This means that ϕ(v) ∈ K is considered as scalar factor: ϕ(v) ⊗ w = ϕ(v) · w ∈ W . When we want to reproduce the notation in §3.3.2.1, we must modify W by omitting all factors Wj = K. d N K The counterpart of Proposition 3.55 is the statement below. Here W = j=1 degenerates to K, i.e., L(V, W) = V0 . Nd Proposition 3.62. Let Vj (1 ≤ j ≤ d) be vector spaces generating V := a j=1 Vj . Nd N d Elementary tensors of a j=1 Vj0 are Φ = j=1 ϕ(j) , ϕ(j) ∈ Vj0 . Their application to tensors from V is defined via ! d d Y O (j) v ϕ(j) (v (j) ) ∈ K . = Φ j=1

j=1

This defines the embedding 11

Concerning isomorphisms compare the last paragraph in §3.2.5.

3.3 Linear and Multilinear Mappings d O a

83

Vj0 ⊂ V0 ,

and

Vj0 = V0 ,

if dim(Vj ) < ∞ for 1 ≤ j ≤ d.

(3.32)

j=1 d O a j=1

The next example shows that, in general, (3.32) holds with proper inclusion. Example 3.63. Choose V := `0 ⊗a `0 withP`0 = `0 (N) defined in (3.2) and consider the functional Φ ∈ V0 with Φ(v ⊗ w) := i∈N vi wi . Note that this infinite sum is well defined since vi wi = 0 for almost all i ∈ Z. Since Φ does not belong to `00 ⊗a `00 , 0 the latter space is a proper subspace of (`0 ⊗a `0 ) . The result does not change when the index set N is replaced with any other infinite index set I. Note that any infinite-dimensional space V is isomorphic to `0 (I) with #I = dim(V ). Therefore this example holds for V := V ⊗a W with V and W of infinite dimension. Pk Proof. For an indirect proof, assume that Φ = ν=1 ϕν ⊗ ψν ∈ `00 ⊗a `00 for some ϕν , ψν ∈ `00 and k ∈ N. Choose any integer m > k. Let e(i) ∈ `0 be the i-th unit Pk vector. The assumed identity Φ = ν=1 ϕν ⊗ ψν tested for all e(i) ⊗ e(j) ∈ `0 ⊗a `0 with 1 ≤ i, j ≤ m yields m2 equations Xk ϕν (e(i) ) · ψν (e(j) ) δjk = Φ(e(i) ⊗ e(j) ) = (1 ≤ i, j ≤ m) . (3.33) ν=1

Define matrices A, B ∈ Km×k by Aiν := ϕν (e(i) ) and Bjν := ψν (e(j) ). Then equation (3.33) becomes I = AB T . Since rank(A) ≤ min{m, k} = k, the rank of the product AB T is also bounded by k, contradicting rank(I) = m > k. t u A very N important case is the counterpart of (3.31b) in Notation 3.57a. We use the notations j6=k and V[k] introduced in (3.17a,b). Nd Remark 3.64. (a) Let Vj (1 ≤ j ≤ d) be vector spaces generating V := a j=1 Vj . For a fixed index k ∈ {1, . . . , d} let ϕ(k) ∈ Vk0 be a linear functional. Then ϕ(k) induces the definition of Φ ∈ L(V, V[k] ) by Φ

d O j=1

! v

(j)


O (j) v . := ϕ(k) v (k) · j6=k

(b) According to (3.31b), we identify Φ = id ⊗ . . . ⊗ ϕ(k) ⊗ . . . ⊗ id = id[k] ⊗ ϕ(k)
N
Nd (j) and ϕ(k) and write ϕ(k) = ϕ(k) v (k) · j6=k v (j) . This leads to the j=1 v embedding (3.34) Vk0 ⊂ L(V, V[k] ). The tensor vhii ∈ V[d] defined in (3.22a) can be obtained as follows. Part (b) follows by linearity.

3 Algebraic Foundations of Tensor Spaces

84 (d)

Remark 3.65. (a) Let {bi : i ∈ B} be a basis of Ud ⊂ Vd with some dual system (d) {ϕi } ⊂ Vd0 . Then v ∈ V[d] ⊗ Ud has the representation (3.22a) with (d)

vhii := ϕi (v) ∈ V[d] . (b) v = 0 implies vhii = 0 for all i. Part (b) from above proves the first part of the next lemma. The second part is left as exercise. Pr Nd Nd (j) Lemma 3.66. Let V = a j=1 Vj and v := i=1 j=1 vi . If, for some index (k)

k ∈ {1, . . . , d}, the vectors {vi v = 0 implies that

: 1 ≤ i ≤ r} ⊂ Vk are linearly independent, [k]

[k]

for all vi :=

vi = 0

O

(j)

(1 ≤ i ≤ r).

vi

j6=k (k)

[k]

Conversely, for linearly dependent vectors {vi : 1 ≤ i ≤ r}, there are vi ∈ V[k] , Pr [k] (k) not all vanishing, with v = i=1 vi ⊗ vi = 0. 0 . Here the elementary tensors are ϕ[k] = Another extreme case occurs for V[k] N (j) ϕ . In this case, the image space is K ⊗ . . . ⊗ K ⊗ Vk ⊗ K ⊗ . . . ⊗ K ∼ = Vk . j6=k

Nd Remark 3.67. Let Vj (1 ≤ j ≤ d) be vector spaces generating V := a j=1 Vj . For a fixed index k ∈ {1, . . . , d} define V[k] by (3.17a). Then elementary tensors N (j) 0 ϕ ∈ V[k] ϕ[k] = (ϕ(j) ∈ Vj0 ) are considered as mappings from L(V, Vk ) via j6=k

d O

[k]

ϕ

! v

(j)

! :=

j=1

Y

(j)

ϕ

(v

(j)

· v (k) .

)

(3.35a)

j6=k

This justifies the embedding 0 V[k] ⊂ L(V, Vk ).

(3.35b)

The generalisation of the above case is as follows. Let α ⊂ {1, . . . , d} be any nonempty subset and define the complement αc := {1, . . . , d}\α. Then the embedding O 0 (3.35c) with Vα := a Vα ⊂ L(V, Vαc ) Vj j∈α

is defined via ! O j∈α

ϕ

(j)

d O j=1

! v

(j)

! :=

Y j∈α

(j)

ϕ

(v

(j)

)

·

O j∈αc

v (j) .

85

3.4 Tensor Spaces with Algebra Structure

3.3.2.5 Further Embeddings Proposition 3.68. (a) The algebraic tensor space V ⊗a W 0 can be embedded into L(W, V ) via w ∈ W 7→ (v ⊗ w0 ) (w) := w0 (w) · v ∈ V. (b) Similarly, V ⊗a W can be embedded into L(W 0 , V ) via w0 ∈ W 0 7→ (v ⊗ w) (w0 ) := w0 (w) · v ∈ W.

(3.36a)

(c) The embeddings from above show that V ⊗a W 0 ⊂ L(W, V )

and

V ⊗a W ⊂ L(W 0 , V ).

(3.36b)

Corollary 3.69. If dim(W ) < ∞, V ⊗a W ∼ = L(W, V ) are isomorphic. = V ⊗a W 0 ∼ Proof. dim(W ) < ∞ implies W 0 ∼ = V ⊗a W 0 . = W and, therefore, also V ⊗a W ∼ Thanks to (3.36b), V ⊗a W 0 is isomorphic to a subspace of L(W, V ). To prove V ⊗a W ∼ = L(W,V ), we must demonstrate that any ϕ ∈ L(W,V ) can be realised by some x ∈ V ⊗a W 0. Let P {wi } be a basis of W and {ωi } a dual basis with ωi (wj ) = δij . Set x := i ϕ(wi ) ⊗ ωi . One easily verifies that x(wi ) = ϕ(wi ) in the sense of the embedding (3.36a). u t

3.4 Tensor Spaces with Algebra Structure Throughout this section, all tensor spaces are algebraic tensor spaces. Therefore we omit the index ‘a’ in ⊗a . So far, we have considered tensor products of vector spaces Aj (1 ≤ j ≤ d). Now we suppose that Aj possesses an additional operation12 ◦ : Aj × Aj → Aj , which we call the multiplication (to be quite precise, we should introduce individual symbols ◦j for each Aj ). We require that (a + b) ◦ c = a ◦ c + b ◦ c a ◦ (b + c) = a ◦ b + a ◦ c (λa) ◦ b = a ◦ (λb) = λ · (a ◦ b) 1◦a=a◦1=a

for all a, b, c ∈ Aj , for all a, b, c ∈ Aj , for all λ ∈ K and all a, b ∈ Aj , for some 1 ∈ Vj and all a ∈ Aj .

(3.37)

These rules define a (noncommutative) algebra with unit element 1. Ignoring the Nd algebra structure, we define the tensor space A := a j=1 Aj as before and establish 12

µ : Aj ×Aj → Aj defined by (a, b) 7→ a ◦ b is called the structure map of the algebra Aj .

3 Algebraic Foundations of Tensor Spaces

86

an operation ◦ : A × A → A by ! ! d d d O O O aj ◦ bj = (aj ◦ bj ) , j=1

j=1

j=1

1 :=

d O

1

j=1

for elementary tensors. The first two axioms in (3.37) are used to define the multiNd plication of tensors in a j=1 Aj . Example 3.70. (a) Consider the matrix spaces Aj := KIj ×Ij (Ij : finite index sets). Here ◦ is the matrix-matrix multiplication in KIj ×Ij . The unit element 1 is the Nd identity matrix I. Then A := j=1 Aj = KI×I (I = I1 × . . . × Id ) is the space containing the Kronecker matrices. (b) The vector spaces Aj and A from Part (a) yield another algebra if ◦ is defined by the Hadamard product (entry-wise product, cf. (4.82)): (a ◦ b)i = ai ·bi for i ∈ Ij×Ij and a, b ∈ Aj . The unit element 1 is the matrix with all entries being one. (c) Let Aj := C(Ij ) be the set of continuous functions on the interval Ij ⊂ R. Aj becomes an algebra with the pointwise multiplication ◦. The unit element is Nd the function with constant value 1 ∈ K. Then A := j=1 Aj ⊂ C(I) contains multivariate functions on the product domain I = I1 × . . . × Id . (d) Let Aj := `0 (Z) (cf. (3.2)). The multiplication ◦ in Aj may be defined by the convolution ? in (4.83a). The unit element 1 is the sequence with 1i = δi0 for allNi ∈ Z (δi0 in (2.1)). This defines the d-dimensional convolution ? in d A := j=1 Aj = `0 (Zd ). Another relation of between algebras and tensor spaces is already exercised in §3.2.6.7 for the matrix-matrix multiplication. Let d = 1, i.e., the vector space V carries a second operation ◦ : V × V → V . Fix any basis {bi : i ∈ I} of V. The operation ◦ is completely defined by the data sijk in X bi ◦ bj = sijk bk . k∈I

These coefficients define the structure tensor s ∈ KI ⊗ KI ⊗ KI of the algebra (cf. Ye–Lim [305, §4]). The importance of rank(s) is emphasised in §3.2.6.7. An equivalent tensor, also denoted by s, is X sijk bi0 ⊗ bj0 ⊗ bk ∈ V 0 ⊗ V 0 ⊗ V, s := i,j,k∈I

where b0i ∈ V 0 is the functional with b0i (b` ) = δi` for basis vectors b` . We recall the bidual embedding V ⊂ V 00 by v(ϕ) := ϕ(v) (v ∈ V, ϕ ∈ V 0 ). Therefore the left-hand side of the following expression belongs to V 00 ⊗ V 00 ⊗ L(V, V ). Remark 3.71. v ◦ w = (v ⊗ w ⊗ id) s holds for all v, w ∈ V. P P Inserting v = w ⊗ id) s = w = Proof. j wj bj , we obtain i vi bi and P P (v ⊗P P sijk bi0 (v) ⊗ b0j (w) ⊗ bk = vi wj k sijk bk = sijk vi wj bk = i,j,k i,j,k i,j P P P t u j wj bj ) = v ◦ w. i vi bi ) ◦ ( i,j vi wj bi ◦ bj = (

87

3.4 Tensor Spaces with Algebra Structure

Example 3.72. Let V1 = Kn1 ×n2 , V2 = Kn2 ×n3 , V3 = Kn1 ×n3 be matrix spaces. (j) (j) The respective bases consist of E(k,`) ∈ Vj with E(k,`) [p, q] = δkp δ`q . The characteristic tensor of the matrix-matrix multiplication is defined by m :=

n3 n1 X n2 X X

(1)0

(3)

(2)0

E(i1 ,i2 ) ⊗ E(i2 ,i3 ) ⊗ E(i1 ,i3 ) ∈ V1 ⊗ V2 ⊗ V3 .

i1 =1 i2 =1 i3 =1

The tensor m is already defined in (3.28a) for n1 = n2 = n3 = 2 (up to the isomorphism K4 ' K2×2 ). Proof. Note that (1)0

X

(A ⊗ B ⊗ id) m =

(2)0

(3)

E(i1 ,i2 ) (A)E(i2 ,i3 ) (B)E(i1 ,i3 )

i1 ,i2 ,i3

" =

#

X X i1 ,i3

(3)

A[i1 , i2 ]B[i2 , i3 ] E(i1 ,i3 ) =

X

(3)

(AB) [i1 , i3 ]E(i1 ,i3 ) = AB

i1 ,i3

i2

for A ∈ V1 and B ∈ V2 .

t u

The complexity of matrix multiplication is closely related to the rank of m (cf. §3.2.6.7). For a detailed discussion we refer to Landsberg [206, Chap. 11]. The term tensor algebra is used for another algebraic construction. Let V be a vector space and consider the tensor spaces ⊗d V :=

d O

V

j=1

(cf. Notation 3.24) and define the direct sum13 of ⊗d V for all d ∈ N0 : X ⊗d V. A(V ) :=

(3.38)

d∈N0

Elements of the tensor algebra are finite sums multiplicative structure is given by ◦ = ⊗: vn ∈ ⊗d V, vm ∈ ⊗m V

7→

P

d∈N0

vd with vd ∈ ⊗d V . The

vn ⊗ vm ∈ ⊗n+m V

(cf. Greub [125, Chap. III]). The algebra A(V ) has the unit element 1 ∈ K = ⊗0 V ⊂ A(V ). The algebra A(`0 ) will be used in §14.3.3. If we replace N0 in (3.38) with N , ∞ X ⊗d V A(V ) := d=1

is an algebra without unit element. 13

V = X + Y is the direct sum of the vector spaces X, Y, if each v ∈ V has a unique representation v = x + y with x ∈ X, y ∈ Y.

3 Algebraic Foundations of Tensor Spaces

88

3.5 Symmetric and Antisymmetric Tensor Spaces 3.5.1 Basic Definitions (Anti)symmetric tensors are associated with coinciding vector spaces Vj denoted by V : V := V1 = V2 = . . . = Vd . The d-fold tensor product is now denoted by V = ⊗d V , where d ≥ 2 is required. Here we refer to the algebraic tensor space, i.e., V = ⊗da V . The completion to a Banach or Hilbert tensor space will be considered in §4.7.2. A bijection π : D → D of the set D := {1, . . . , d} is called a permutation. Let P := {π : D → D bijective} be the set of all permutations. Note that its cardinality #P = d! increases fast with increasing d . (P, ◦) is a group for which ◦ is defined by composition: (τ ◦ π) (j) = τ (π(j)). The inverse of π is denoted by π −1 . As known from the description of determinants, sign : P → {−1, +1} can be defined such that transpositions (pairwise permutations) have the sign −1, while the function sign is multiplicative: sign(τ ◦ π) = sign(τ ) · sign(π). A permutation π ∈ P gives rise to a mapping V → V, again denoted by π: d d O O −1 (3.39) v (j) 7→ π: v (π (j)) . j=1

j=1

Exercise 3.73. For v ∈ ⊗d Kn show that (π(v))i = vπ(i) for all i ∈ {1, . . . , n}d , where
π (i1 , . . . , id ) := iπ(1) , . . . , iπ(d) is the action of π ∈ P onto a tuple from {1, . . . , n}d . Definition 3.74. (a) v ∈ V = ⊗d V is symmetric if π(v) = v for all14 π ∈ P . (b) The symmetric tensor space is defined by S := S(V ) := Sd (V ) := ⊗dsym V := {v ∈ V : v symmetric} . (c) A tensor v ∈ V = ⊗d V is antisymmetric (synonymously with ‘skew-symmetric’ or ‘alternating’) if π(v) = sign(π)v for all π ∈ P . (d) The antisymmetric tensor space is defined by d V := {v ∈ V : v antisymmetric} . A := A(V ) := Ad (V ) := ⊗anti 14

Replacing all π ∈ P by a subgroup of permutations (e.g., only permutations of certain positions) one can define tensor spaces with partial symmetry.

89

3.5 Symmetric and Antisymmetric Tensor Spaces

Remark 3.75. An equivalent definition of a symmetric [antisymmetric] tensor v is v = π(v) [v = −π(v)] for all transpositions π : (1, . . . , i, . . . , j, . . . , d) 7→ (1, . . . , j, . . . , i, . . . , d) , since all permutations from P are products of pairwise permutations. For d = 2 and V = Kn , tensors from S and A correspond to symmetric matrices (Mij = Mji ) and antisymmetric matrices (Mij = −Mji ), respectively. Formally we define Sd = Ad = ⊗d V for d = 0 and d = 1. Proposition 3.76. (a) S and A are subspaces of V = ⊗d V . (b) The projections PS and PA from V onto S and A, respectively, are given by PS (v) :=

1 X π(v), d!

PA (v) :=

π∈P

1 X sign(π)π(v). d!

(3.40)

π∈P

PS is called the symmetriser, and PA the alternator. Any matrix can be split into a symmetric and an antisymmetric part, i.e., S ⊕ A = V holds for d = 2. This is not true for d ≥ 3. Remark 3.77. (a) Exercise 3.73 shows that (PS (v))i = (PA (v))i =

1 d!

P

π∈P

vπ(i) and

1 X sign(π)vπ(i) for v ∈ ⊗d Kn and all i ∈ {1, . . . , n}d . d! π∈P

d

(b) Let V = Kn , In := {1, . . . , n} , and Isym := {i ∈ In : 1 ≤ i1 ≤ i2 ≤ . . . ≤ id ≤ n} . n Then v ∈ ⊗dsym Kn is completely determined by the entries vi for i ∈ Isym n . All other entries vi coincide with vπ(i) , where π ∈ P is chosen such that π(i) ∈ Isym n . (c) With V and In from Part (b) let Ianti := {i ∈ In : 1 ≤ i1 < i2 < . . . < id ≤ n} . n Then v ∈ ⊗danti Kn is completely determined by the entries vi with i ∈ Ianti n . For i ∈ In \Ianti n , two cases must be distinguished. If i contains two equal elements, i.e., ij = ik for some j 6= k, then vi = 0. Otherwise there is some π ∈ P with π(i) ∈ Ianti and vi = sign(π)vπ(i) . n Conclusion 3.78. (a) Ad (Kn ) = {0} for d > n since Ianti = ∅. n N d (b) For n = d , Ad (Kd ) = span{PA ( j=1 e(j) )} is one-dimensional (e(j) : unit vectors, cf. (2.2)) since #Ianti = 1. n

3 Algebraic Foundations of Tensor Spaces

90

Proposition 3.79. For dim(V ) = n < ∞ and d ≥ 2 the dimensions of S and A satisfy

dim(Ad (V )) = nd < nd /d! < dim(Sd (V )) = n+d−1 . d Bounds are  d  d d−1 d−1 dim(Ad (V )) ≤ n − /d! and dim(Sd (V )) ≤ n + /d!. 2 2 Proof. S is isomorphic to S(Kn ). Remark 3.77c shows that dim(S) = #Isym n .
n sym By induction we show that #In = d . The proof for A is analogous. t u As Proposition 3.79 shows, A(V ) has a dimension smaller than S(V ), but the other side of the coin is that V must be higher dimensional to form Ad (V ) of a certain dimension. As long as n = dim(V ) < d, Ad (V ) = {0} is zero-dimensional. Let NA = ker(PA ), NS = ker(PS ) be the kernels and note that AN = range(PA ) d and S = range(PS ) are the images. Then the tensor space V = V admits the direct decomposition V = NA ⊕ A = NS ⊕ S, i.e., any v ∈ T has a unique decomposition into v = vN + vX with either vN ∈ NA , vX ∈ A or vN ∈ NS , vX ∈ S. Consequently, S and A are isomorphic to the quotient spaces (cf. §3.1.3): S∼ = V/NS ,

A∼ = V/NA .

(3.41)

V/NA = ∧d V is called the d-th exterior power of V , and v (1) ∧v (2) ∧ . . . ∧v (d) is the isomorphic image of PA (v (1) ⊗ v (2) ⊗ . . . ⊗ v (d) ). The operation ∧ is called the exterior product. Analogously, V/NS = ∨d V is called the d-th symmetric power of V and v (1) ∨v (2) ∨. . .∨v (d) is the isomorphic image of PS (v (1) ⊗v (2) ⊗. . .⊗v (d) ). In the context of (anti)symmetric tensor spaces a mapping A ∈ L(V, V) is called symmetric if A commutes with all π ∈ P , i.e., Aπ = πA. This property implies that APS = PS A and APA = PA A, and proves the following result. Remark 3.80. If A ∈ L(V, V) is symmetric, S and A are invariant under A, i.e., the restrictions of A to S and A belong to L(S, S) and L(A, A), respectively. The Hadamard product will be explained in (4.82). In the case of functions, it is the usual pointwise product. Exercise 3.81. Let s, s0 ∈ S and a, a0 ∈ A. Show that s s0 , a a0 ∈ S, whereas s a, a s ∈ A.

91

3.5 Symmetric and Antisymmetric Tensor Spaces

3.5.2 Quantics The term ‘quantics’ introduced by Arthur Cayley [54] in 1854 is used for homogeneous polynomials in multiple variables, i.e., polynomials in the variables xi (i ∈ B) which are a finite sum of terms Y aν xν = aν xνi i∈B i P with multi-indices ν of length |ν| := i∈B νi = d ∈ N0 and aν ∈ K. Such quantics have the property p(λx) = λd p(x). Proposition 3.82. Let V be a vector space15 with the algebraic basis {bi : i ∈ B}. Then the algebraic symmetric tensor space Sd (V ) is isomorphic to the set of quantics in the variables {xi : i ∈ B}. This statement also holds for dim(V ) = #B = ∞. Note that the infinite product Y xν = xνi i i∈B

makes sense since at most d exponents νi are different from zero. Nd V . Fix a For the proof of Proposition 3.82 consider a general tensor v ∈ a basis {bi : i ∈ B} of V . To each bi we associate a variable xi . We may write v=

X i∈B d

ai

d O

bi j

(almost all ai vanish).

j=1

P Nd Therefore any symmetric tensor can be written as PS v = i∈B d PS (ai j=1 bij ). The isomorphism into the quantics of degree d is given by ! d d O Y bi j Φ : PS ai (3.42) 7→ ai xij . j=1

j=1

Note that d Y j=1

xij =

Y

xνi i

with νi := #{j ∈ {1, . . . , d} : ij = i}

for all i ∈ B.

i∈B

The symmetry of PS v corresponds to the fact that all factors xi commute, i.e., xi xj = xj xi . Above we used the range of PS to define Sd (V ). In the case of d = 2, e.g., v := PS (a ⊗ b) = 21 (a ⊗ b + b ⊗ a) represents a symmetric tensor. Instead, we may use v = 14 ⊗2 (a + b) − 14 ⊗2 (a − b) . In the latter representation each term itself is symmetric. In general, Sd (V ) can be generated by d-fold products ⊗d v: 15

An additional requirement is that either K have characteristic 0 or that the polynomials be understood in such a way that xν for different ν are linearly independent, i.e., they are not viewed as functions defined on K (note that x3 − 1 = 0 as a function on K = Z3 ).

3 Algebraic Foundations of Tensor Spaces

92

S(V ) = span{⊗d v : v ∈ V }.

(3.43)

In the language of quantics this means that the polynomial is written as sum of
d Pn of linear forms. The decomposition of a homogeneous d-th powers i=1 ai xi polynomial into this special form is addressed by Brachat–Comon–Mourrain– Tsigaridas [40]. Such a decomposition is the basis of the following definition. Definition 3.83. For symmetric tensors16 s ∈ Sd (V ) the symmetric rank (also called Waring rank) is defined by ( ) r X d ⊗ vi with vi ∈ V . ranksym (s) := min r ∈ N0 : s = i=1

An immediate conclusion is ranksym (s) ≥ rank(s). In the case of real matrices we have ranksym (s) = rank(s). The conjecture that this equality also holds for general tensors is wrong. A counterexample is given by Shitov [261]. A related quantity is the Chow rank defined in §7.7.2.

3.5.3 Determinants Since tensor products are related to multilinear forms (cf. Proposition 3.23) and the determinant is a special antisymmetric multilinear form, it is not surprising to find relations between antisymmetric tensors and determinants. We recall that the determinant det(A) of a matrix A ∈ Kd×d is equal to d X Y (3.44) det(A) = aj,π(j) . sign(π) j=1

π∈P

antisymmetric tensors we have to use their antisymmetrisation Lemma 3.84. The antisymmetrised elementary tensor v := PA v (j) ∈ V := Kn has the entries  (1) (2)  (d) vi 1 vi 1 · · · vi 1  (1) (2) (d)   vi 2 vi 2 · · · vi 2  1 v[i1 , . . . , id ] = det  .. . . .   .. . d! . ..   . . (1)

(2)

Nd

(j) j=1 u . For
Nd (j) . PA j=1 u

The building block of usual tensors are the elementary tensors

Nd

j=1


v (j) with

(d)

vi d vi d · · · vi d 16

Here we assume P that the underlying field K has characteristic 0 (as R and C). Otherwise the representation s = ri=1 ⊗d vi may be impossible. Consider, e.g., the field K = Z2 = {0, 1}. Then V := K2 contains only 3 nontrivial vectors so that dim{⊗3 v : P v ∈ V } ≤ 3. Since dim(S3 (V )) = 4, not all symmetric tensors allow a representation s = ri=1 ⊗d vi .

93

3.5 Symmetric and Antisymmetric Tensor Spaces

Proof. Set e :=

Nd

j=1

v (j) . By the definition (3.44), the right-hand side is equal to

d Y 1 X 1 X (j) sign(π) viπ(j) = sign(π)eπ(i) = (PA (e))i = vi = v[i1 , ..., id ], d! d! j=1 π∈P

π∈P

t u

proving the assertion.

For V = Kd the antisymmetric space Ad (Kd ) is one-dimensional (cf. Conclusion 3.78b). Therefore any transformation by A = ⊗d A is identical to a multiple of the identity. Nd Remark 3.85. Let V = Kd . For any A ∈ Kd×d and v = PA ( j=1 v (j) ) ∈ Ad (Kd ) one has O  O  d d A(v) := PA Av (j) = det(A) PA v (j) . j=1

j=1

Proof. By Conclusion 3.78, tensors u from A(Kd ) are determined by u[1, . . . , d]. 1 Lemma 3.84 shows that v[1,..., d] = d! det(M ), where M = [v (1) ,..., v (d) ] ∈ Kd×d , 1 1 and that A(v)1,...,d = d! det(AM ) = det(A) · d! det(M ) = det(A) · v[1,..., d]. t u For function spaces V (e.g., V = L2 (R), V = C[0, 1], etc.) there is an analogue of Lemma 3.84. Lemma 3.86. For functions f1 , . . . , fd ∈ V the antisymmetrisation of the elementary tensor d d O Y F = fj , i.e., F (x1 , . . . , xd ) = fj (xj ), j=1

j=1

yields G := PA (F ) with 

 f1 (x1 ) f2 (x1 ) · · · fd (x1 )  f1 (x2 ) f2 (x2 ) · · · fd (x2 )  1   G(x1 , . . . , xd ) = det  . .. ..  . .. d!  .. . . .  f1 (xd ) f2 (xd ) · · · fd (xd ) G is also called the Slater determinant of f1 , . . . , fd . Also the scalar product of two antisymmetrised elementary tensors is expressed by a determinant as stated in (4.90).

3.5.4 Application of Functionals For a subset ∅ $ α $ D = {1, . . . , d} let φ ∈ (a (without requiring any symmetry property).

N

j∈α

V )0 be a functional

3 Algebraic Foundations of Tensor Spaces

94

Lemma 3.87. Let φ and α as above and set k := #α. (a) Then φ interpreted as in (3.35c) is a map of Sd (V ) into Sd−k (V ). The value φ(s) of s ∈ Sd (V ) only depends on k = #α, not on the particular choice of α ⊂ D. (b) φ is also a mapping from Ad (V ) into Ad−k (V ). Here, the sign of φ(a) for a ∈ Ad (V ) depends on the choice of α. Proof. Let α and β be subsets of D with #α = #β and denote the corresponding maps by φα and φβ . Define the permutation π by π(β) = α. Then φβ ◦ π = φα holds, i.e., φα (v) = φβ (π(v)) . In the symmetric case we have π(v) = v, i.e., φα (v) = φβ (v). In the antisymmetric case the sign may change. In the sequel we assume that α = {d − k + 1, . . . , d}. Let π be a transposition of i 6= j with i, j ∈ {1, . . . , d − k}. It follows that πφ = φπ, i.e., π(φ(v)) = φ(π(v)). Hence v ∈ Sd (V ) implies π(v) = v and π(φ(v)) = φ(v) ∈ ⊗d−k V. Remark 3.75 proves that φ(v) ∈ Sd−k (V ). The u t antisymmetric case is analogous. Since the expansion term vhii can be defined by a functional (cf. Remark 3.65), Lemma 3.87 proves the next statement. For that purpose we recall (3.22a): X (3.45) vhii ⊗ bi , v= {bi : i ∈ B} basis of V. i

Conclusion 3.88. If v ∈ Sd (V ) or v ∈ Ad (V ), also the tensors vhii ∈ ⊗d−1 V defined in (3.45) belong to Sd−1 (V ) or v ∈ Ad−1 (V ), respectively. Concerning the notation vhi,mi we refer to (3.22b). Theorem 3.89. Assume that v is given by (3.45) with vhii ∈ ⊗d−1 V. (a) Then v ∈ Sd (V ) holds if and only if vhii ∈ Sd−1 (V ) and vhi,mi = vhm,ii

for all i, m ∈ B.

(b) v ∈ Ad (V ) holds if and only if vhii ∈ Ad−1 (V ) and vhi,mi = −vhm,ii for all different i, m ∈ B. Proof. (i) Assume vhiiP ∈ Sd−1 (V ) and define the tensor v by (3.45). The second = v expansion yields i,m vhi,mi ⊗ bm ⊗ bi . Symmetry of v implies that v = P vhi,mi ⊗ bi ⊗ bm . Interchanging the symbols i, m in this expression yields i,mP v = i,m vhm,ii ⊗bm ⊗bi . Therefore, vhi,mi = vhm,ii follows (cf. Remark 3.66a). (ii) For the opposite direction we have to show that πv = v for all transposition of j 6= k (cf. Remark 3.75). If 1 ≤ j, k ≤ d − 1, πv = v is a consequence of vhii ∈ Sd−1 (V ). It remains to consider the case j ≤ d − 1 and k = d; without loss of generality we choose j = d − 1. The considerations in part (i) show that vhi,mi = vhm,ii implies πv = v. (iii) The antisymmetric case is completely analogous. t u

Part II

Functional Analysis of Tensor Spaces

Algebraic tensor spaces yield a suitable fundament for the finite-dimensional case. But already in the finite-dimensional case we want to formulate approximation problems, which require the introduction of a topology. Topological tensor spaces are a subject of functional analysis. Standard examples of infinite-dimensional tensor spaces are function spaces, since multivariate functions can be regarded as tensor products of univariate ones. To obtain a Banach tensor space, we need the completion with respect to a norm, which is not fixed by the normed spaces generating the tensor space. The scale of norms is a particular topic of the discussion of Banach tensor spaces in Chapter 4. A particular, but important case are Hilbert tensor spaces. Chapter 5 has a stronger connection to algebraic tensor spaces than to topological ones. But, in particular, the technique of matricisation is a prerequisite required in Chapter 6. In Chapter 6 we discuss the so-called minimal subspaces which are important for the analysis of the later tensor representations in Part III.

Chapter 4

Banach Tensor Spaces

Abstract The discussion of topological tensor spaces has been started by Schatten [254] and Grothendieck [129, 130]. In Section 4.2 we discuss the question how the norms of V and W are related to the norm of V ⊗W. In particular, we introduce the pro- and injective norms. From the viewpoint of functional analysis, tensor spaces of order two are of special interest, since they are related to certain operator spaces (cf. §4.2.9). However, for our applications we are more interested in tensor spaces of order ≥ 3. These spaces are considered in Section 4.3. As preparation for the aforementioned sections and later ones, we need more or less well-known results from Banach space theory, which we provide in Section 4.1. Sections 4.4 and 4.5 discuss the case of Hilbert spaces and Hilbert tensor spaces. This is important, since many applications and many of the numerical methods require the Hilbert structure. The reason is that, unfortunately, the solution of approximation problems with respect to general Banach norms is much more involved than those with respect to a scalar product. In Section 4.6 we recall the tensor operations. The final Section 4.7 is devoted to symmetric and antisymmetric tensor spaces.

4.1 Banach Spaces 4.1.1 Norms In general, we consider vector spaces X over one of the fields K ∈ {R, C}. The topological structure will be generated by norms. We recall the axioms of a norm on X: k·k : X → [0, ∞) , kxk = 0 kλxk = |λ| kxk kx + yk ≤ kxk + kyk

if and only if x = 0, for all λ ∈ K and x ∈ X, for all x, y ∈ X (triangle inequality).

© Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_4

(4.1)

97

98

4 Banach Tensor Spaces

The map k·k : X → [0, ∞) is continuous because of the inverse triangle inequality kxk − kyk ≤ kx − yk (4.2) for all x, y ∈ X. Combining a vector space X with a norm defined on X, we obtain a normed vector space denoted by the pair (X, k·k). If there is no doubt about the choice of a norm, the notation (X, k·k) is shortened by X. A vector space X may be equipped with two different norms; i.e., (X, k·k1 ) and (X, k·k2 ) may be two different normed spaces although the set X is the same in both cases. For the set of all norms on X we can define a semi-ordering. A norm k·k1 is called weaker (or not stronger) than k·k2 , in symbolic notation k·k1 . k·k2 , (equivalently: k·k2 stronger than k·k1 , or k·k2 & k·k1 ) if there is a constant C such that kxk1 ≤ C kxk2 for all x ∈ X. Two norms k·k1 and k·k2 on X are equivalent, in symbolic notation k·k1 ∼ k·k2 , if k·k1 . k·k2 . k·k1 , or equivalently, if there are C1 , C2 ∈ (0, ∞) with 1 kxk1 ≤ kxk2 ≤ C2 kxk1 C1

for all x ∈ X.

4.1.2 Basic Facts about Banach Spaces A sequence xi ∈ X in a normed vector space (X, k·k) is called a Cauchy sequence (with respect to k·k) if supi,j≥n kxi − xj k → 0

as n → ∞.

A normed vector space (X, k·k) is called a Banach space if it is also complete (with respect to k·k). Completeness means that any Cauchy sequence xi ∈ X has a limit x := limi→∞ xi ∈ X (i.e., kx − xi k → 0). A subset X0 ⊂ X of a Banach space (X, k·k) is dense, if for any x ∈ X there is a sequence xi ∈ X0 with kx − xi k → 0. An equivalent criterion is that for any ε > 0 and any x ∈ X there is some xε ∈ X0 with kx − xε k ≤ ε. A dense subset may be, in particular, a dense subspace. An important property of dense subsets is noted in the next remark.

4.1 Banach Spaces

99

Remark 4.1. Let Φ : X0 → Y be a continuous linear mapping, where X0 is dense in the Banach space (X, k·k) while (Y, k·kY ) is some Banach space. Then there is a unique continuous extension Φ : X → Y with Φ(x) = Φ(x) for all x ∈ X. If a normed vector space (X, k·k) is not complete, it has a unique completion
X, ||| · ||| —up to isomorphisms—such that X is a dense subspace of the Banach space X and ||| · ||| is the continuous extension of k·k (similarly to Remark 4.1, Ψ := k·k : X → R has a unique extension Ψ =||| · ||| to a norm on X). From now on we shall use the same symbol k·k for the norm on the closure X (instead of ||| · |||). Further, the continuous extension Φ of any continuous Φ : X → Y is again denoted by Φ. Remark 4.2. Let (X, k·k1 ) and (X, k·k2 ) be normed vector spaces with identical sets X but different norms. Then we must distinguish between the completions (X1 , k·k1 ) and (X2 , k·k2 ) with respect to the corresponding norms. If X is of infinite dimension, the identity X1 = X2 holds if and only if k·k1 ∼ k·k2 . If k·k1 . k·k2 , then1 X1 ⊃ X2 ⊃ X. Definition 4.3. A Banach space is separable, if there is a countable dense subset. Lemma 4.4. Any subspace of a separable Banach space is separable. Proof. This result is stated by Hausdorff in 1927 (for metric spaces instead of Banach spaces; cf. Taskovi´c [280]). Let U be a subspace of B = {xi : i ∈ N}. If {u ∈ U : ku − xi k ≤ 1/n}

(n ∈ N)

is nonempty, select an element ui,n . The obtained set S := {ui,n } is again count3 3 able. Given u ∈ U and 0 < ε ≤ 1, there is an xi with ku − xi k < 3ε . For n ∈ [ 2ε , ε ], 2ε an element ui,n ∈ S exists with kui,n − xi k ≤ 3 . Hence ku − ui,n k ≤ ε proves that S ⊂ U is dense in U. t u Definition 4.5. A closed subspace U of a Banach space X is called complemented, if there is a subspace W such that X = U ⊕ W is a direct sum (cf. [175, p. 4]). Closedness of U in X = U ⊕ W implies that W is closed too. In 1933, Banach–Mazur [20] showed that there exist non-complemented closed subspaces. Lindenstrauss–Tzafriri [213] prove for infinite-dimensional Banach spaces X: Each closed subspace is complemented if and only if X is isomorphic to a Hilbert space. Lemma 4.6. Any finite-dimensional subspace U ⊂ X is complemented. Proof. Let Φ ∈ L(X, U ) be the projection from Theorem 4.16. Then X = U ⊕ W holds with W := (I − Φ)X. t u 1

More precisely, the completion X1 can be constructed such that X1 ⊃ X2 .

100

4 Banach Tensor Spaces

4.1.3 Examples Let I be a (possibly infinite) index set. Examples for I are I = {1, . . . , n}, N, Z, or products of these sets. The vector spaces `(I) = KI and `0 (I) ⊂ `(I) are already explained in Example 3.1. Example 4.7. `p (I) consists of all a ∈ `(I) with bounded norm
P p 1/p kak`p (I) := kakp := for p ∈ [1, ∞) i∈I |ai | kak`∞ (I) := kak∞ := sup{|ai | : i ∈ I}

for p = ∞.

(4.3)

(`p (I), k·kp ) is a Banach space for all 1 ≤ p ≤ ∞. If I is a uncountable index set (i.e., #I > ℵ0 := #N ) and p < ∞, the finiteness of kak`p (I) implies that the support supp(a) := {i ∈ I : ai 6= 0} is at most countable. Remark 4.8. (a) For p < ∞, `0 (I) is a dense subspace of `p (I). (b) For an infinite set I, the completion of `0 (I) under the norm k·k∞ yields the proper subset (c0 (I), k·k∞ ) $ `∞ (I) of zero sequences:2 c0 (I) = {a ∈ `∞ (I) : lim aν = 0}. ν→∞

(4.4)

Proof. (i) For finite sets I, `0 (I) = `(I) holds and nothing has to be proved. Next, I is assumed to be infinite and countable. P∞ p Since (ii) Case ν=1 |aν | < ∞, for any ε > 0, there is a νε P p ∈ [1, ∞). p such that ν>νε |aν | ≤ εp , which proves ka − a0 kp ≤ ε for a0 = (ai0 )i∈I with a0ν := aν for the finitely many 1 ≤ ν ≤ νε and with a0ν := 0 otherwise. Since a0 ∈ `0 (I), `0 (I) is dense in `p (I). (iii) Case p = ∞. Assume `0 (I) 3 a(n) → a ∈ `∞ (I) with respect to k·k∞ . (n) For any n ∈ N there is a νn such that aν = 0 for ν > νn . For any ε > 0, there (n) is an nε such that ka − ak∞ ≤ ε for n ≥ nε . Hence |aν | = |aν(nε ) − aν | ≤ ka(nε ) − ak∞ ≤ ε for ν > νnε proves limν aν = 0, i.e., a ∈ c0 (I). This proves `0 (I) ⊂ c0 (I). (iv) Assume a ∈ c0 (I). For any n ∈ N, there is a νn such that |aν | ≤ 1/n (n) (n) for ν > νn . Define a(n) ∈ `0 (I) by aν = aν for 1 ≤ ν ≤ νn , and aν = 0 (n) (n) otherwise. Then ka −ak∞ ≤ 1/n holds; i.e., a → a and the reverse inclusion c0 (I) ⊂ `0 (I) follow. (v) Let I be uncountable. For p < ∞, we use the statement from Example 4.7: # supp(a) ≤ ℵ0 . For p = ∞, use ℵ20 = ℵ0 to prove # supp(a) ≤ ℵ0 for all a ∈ `0 (I). Neglecting the zero components, repeat the proof with I0 := supp(a) instead of I. t u 2

If #I > ℵ0 , all countable subsequences must be zero sequences.

4.1 Banach Spaces

101

Example 4.9. Let D ⊂ Rm be a domain. (a) Assume 1 ≤ p < ∞. Then3 Z o n p Lp (D) := f : D → K measurable and |f (x)| dx < ∞ d

defines a Banach space with the norm kf kLp (D) = kf kp =

R d


1/p p . |f (x)| dx

(b) For p = ∞, L∞ (D) := {f : D → K measurable and kf k∞ < ∞} equipped with the norm kf k∞ := ess sup |f (x)| is a Banach space. x∈D

Example 4.10. Let D ⊂ Rm be a domain. The following sets of continuous functions and n-times continuously differentiable functions are Banach spaces: (a) C(D) = C 0 (D) := {f : D → K with kf kC(D) < ∞}, where kf kC(D) = sup {|f (x)| : x ∈ D} . (b) Let n ∈ N0 and define C n (D) := {f : I → K with kf kC n (D) < ∞} with the norm kf kC n (D) = max|ν|≤n k∂ ν f kC(D) , where the maximum is taken over Pm all multi-indices ν = (ν1 , . . . , νm ) ∈ N0m with |ν| := i=1 νi . The mixed partial derivatives are abbreviated by Ym  ∂ νi ∂ ν := . (4.5) i=1 ∂xi ∂ ν f denotes the weak derivative (considered as a distribution). Example 4.11. Let D ⊂ Rm be a domain. The Sobolev space n o H 1,2 (D) := f : D → K with kf kH 1,2 (D) < ∞ is a Banach space with the norm v v u

m uX X u

∂ 2 u 2 2 ν t

kf kH 1,2 (D) := t k∂ f kL2 (D) = kf kL2 (D) + .

∂xi f 2 L (D) i=1 |ν|≤1

4.1.4 Operators We recall the definition (3.6) of the set L(X, Y ) of linear mappings. The following remark states basic facts about continuous linear mappings, which are also called operators. The set of operators will be denoted by L(X, Y ). Remark 4.12. Let (X, k·kX ) and (Y, k·kY ) be normed spaces, and Φ ∈ L(X, Y ). (a) The following conditions (i) and (ii) are equivalent: (i) Φ is continuous; (ii) Φ is bounded, i.e., sup {kΦ(x)kY : x ∈ X with kxkX ≤ 1} < ∞. 3

To be precise, we have to form the quotient space {. . .} /N with N := {f : f = 0 on D\S for all S with measure µ(S) = 0}.

102

4 Banach Tensor Spaces

(b) The supremum in (ii) defines the operator norm4 kΦkY ←X := sup {kΦ(x)kY : x ∈ X with kxkX ≤ 1} .

(4.6a)

A simple consequence is kΦ(x)kY ≤ kΦkY ←X kxkX

for all x ∈ X.

(4.6b)

Another notation for the boundedness of Φ reads as follows: there is a constant C < ∞ such that kΦ(x)kY ≤ C kxkX

for all x ∈ X.

(4.6c)

Then the minimal possible C in (4.6c) coincides with kΦkY ←X (cf. (4.6b)).

(c) Let X and Y be the completions of X, Y so that X, k·kX and Y , k·kY are Banach spaces. Then the continuation Φ : X → Y discussed in Remark 4.1 has an identical operator norm: kΦkY ←X = kΦkY ←X . Because of this equality, we shall not distinguish between k·kY ←X and k·kY ←X . (d) The set of continuous linear mappings (operators) from X into Y is denoted by L(X, Y ). Together with (4.6a), (L(X, Y ), k·kY ←X ) forms a normed space. If (Y, k·kY ) is a Banach space, (L(X, Y ), k·kY ←X ) is also a Banach space. Proof. (α) If the boundedness (ii) holds, the definition of kΦkY ←X makes sense and (4.6b) follows by linearity of Φ. (β) (boundedness ⇒ continuity) For any ε > 0 set δ := ε/ kΦkY ←X . Whenever kx0 − x00 kX ≤ δ, we conclude that kΦ(x0 ) − Φ(x00 )kY = kΦ(x0 −x00 )kY ≤ kΦkY ←X kx0 −x00 kX ≤ kΦkY ←X δ = ε, i.e., Φ is continuous. (γ) The direction ‘continuity ⇒ boundedness’ is proved indirectly. Assume that (ii) is not valid. Then there are xi with kxi kX ≤ 1 and αi := kΦ(xi )kY → ∞. αi 1 The scaled vectors x0i := 1+α xi satisfy x0i → 0, whereas kΦ(x0i )kY = 1+α → i i 1 6= 0 = k0kY = kΦ(0)kY . Hence Φ is not continuous. Steps (β) and (γ) prove part (a). (δ) For the last part of (d) let Φi ∈ L(X, Y ) be a Cauchy sequence, i.e., sup kΦi − Φj kY ←X → 0

as n → ∞.

i,j≥n

For any x ∈ X, also the images yi := Φi (x) form a Cauchy sequence as seen from kyi − yj kY ≤ kΦi − Φj kY ←X kxkX → 0. By the Banach space property, y := lim yi ∈ Y exists uniquely giving rise to a mapping Φ(x) := y. One verifies that Φ : X → Y is linear and bounded with kΦi − ΦkY ←X → 0, i.e., Φ ∈ L(X, Y ). t u 4

In (4.6a) one may replace kxkX ≤ 1 by kxkX = 1, as long as the vector space is not the trivial space X = {0} containing no x with kxkX = 1. Since this trivial case is of minor interest, we will often use kxkX = 1 instead.

4.1 Banach Spaces

103

From now on, operator is used as a shorter name for ‘continuous linear map’. In later proofs we shall use the following property of the supremum in (4.6a). Remark 4.13. For all operators Φ ∈ L(X, Y ) and all ε > 0, there is an xε ∈ X with kxε kX ≤ 1 such that kΦkY ←X ≤ (1 + ε) kΦ(xε )kY

and

kΦ(xε )kY ≥ (1 − ε) kΦkY ←X .

A subset K of a normed space is called compact, if any sequence xν ∈ K possesses a convergent subsequence with limit in K. Definition 4.14. An operator Φ ∈ L(X, Y ) is called compact if the unit ball B := {x ∈ X : kxkX ≤ 1} is mapped onto Φ(B) := {Φ(x) : x ∈ B} ⊂ Y and the closure Φ(B) is a compact subset of Y . K(X, Y ) denotes the set of compact operators. The approximation property of a Banach space, which will be explained in Definition 4.89, is also related to compactness. Finally, we add results about projections (cf. Definition 3.4). The next statement follows from closed graph theorem (cf. Yosida [306, §II.6]). Lemma 4.15. Let X = U ⊕ W be the direct sum of closed subspaces U, W . The decomposition x = u + w (u ∈ U , w ∈ W ) of any x ∈ X defines projections P1 , P2 ∈ L(X, X) onto these subspaces by P1 x = u and P2 x = w. Vice versa, a projection P1 ∈ L(X, X) onto U = P1 X together with P2 := I − P1 and W = P2 X defines a direct sum of closed subspaces. According to Definition 4.5, the subspaces U and W are complemented. Theorem 4.16. Let Y ⊂ X be a subspace of a Banach space X with dim(Y ) ≤ n. Then there exists a projection Φ ∈ L(X, X) onto Y such that √ kΦkX←X ≤ n. The proof can be found in DeVore–Lorentz [77, Chap. 9, §7] or Meise [224, Prop. 12.14]. The bound is sharp for general Banach spaces, but can be improved to kΦkX←X ≤ n| 2 − p | 1

1

for X = Lp .

(4.7)

If there is a uniform bound of kΦkX←X for all finite-dimensional subspaces Y, then X is isomorphic to a Hilbert space (cf. Albiac–Kalton [3, Theorem 13.4.3]). Definition 4.17 (embedding). Let X and Y be Banach spaces with (different) norms, while the sets satisfy X ⊂ Y. Then X is continuously embedded into Y if the identity map id : X → Y is continuous, i.e., kxkY ≤ C kxkX for all x ∈ X. The embedding is dense and continuous if, in addition, X is dense in Y. More generally, the embedding ι : X → Y is an injective map so that X and ι(X) ⊂ Y are isomorphic. Identifying X and ι(X), we get the above situation. In the last case of Remark 4.2 X2 is continuously and densely embedded into X1 . Concerning the density use that already X is dense in (X1 , k·k1 ).

104

4 Banach Tensor Spaces

4.1.5 Dual Spaces A trivial example of a Banach space (Y, k·kY ) is the field K ∈ {R, C} with the absolute value |·| as norm. The operators X → K are called continuous functionals or continuous forms. The Banach space L(X, K) is called the (continuous) dual of X and denoted by5 X ∗ := L(X, K). The norm k·kX ∗ of X ∗ follows from the general definition (4.6a): kϕkX ∗ = sup {|ϕ(x)| : x ∈ X with kxkX ≤ 1}

(4.8)

= sup {|ϕ(x)| / kxkX : 0 6= x ∈ X} . ∗

Instead of k·kX ∗ , we also use the notation k·kX (meaning the dual norm corresponding to k·kX ). The next statement is one of the many versions of the Hahn–Banach theorem (cf. Yosida [306, §IV.6]). Theorem 4.18. Let (X, k·kX ) be a normed linear space and U ⊂ X a subspace. If a linear form is bounded on U , it can be extended to a continuous functional on X with the same bound. In particular, for x0 ∈ X there is some ϕ ∈ X ∗ such that ϕ(x0 ) = kx0 kX

and

kϕkX ∗ = 1.

(4.9)

This implies that we recover the norm k·kX from the dual norm k·kX ∗ via the following maximum (no supremum is needed!):   |ϕ(x)| ∗ kxkX = max {|ϕ(x)| : kϕkX ∗ = 1} = max : 0 6= ϕ ∈ X . (4.10) kϕkX ∗ Corollary 4.19. Let {xν ∈ X : 1 ≤ ν ≤ n} be linearly independent. Then there are functionals ϕν ∈ X ∗ such that ϕν (xµ ) = δνµ (cf. (2.1)). The functionals n n (ϕν )ν=1 are called to be dual to (xν )ν=1 . Proof. Let U := span {xν : 1 ≤ ν ≤ n}, and define the dual system {ϕν } as in Definition 3.6. Since dim(U ) < ∞, the functionals ϕν are bounded on U . By Theorem 4.18 there exists an extension of all ϕν to X ∗ with the same bound. t u The following Lemma of Auerbach is proved, e.g., in Meise [224, Lemma 10.5] or Light–Cheney [211, page 131].. Lemma 4.20. For any n-dimensional subspace of a Banach space X, there exists a basis {xν : 1 ≤ ν ≤ n} and a corresponding dual system {ϕν : 1 ≤ ν ≤ n} such that ∗ kxν k = kϕν k = 1 for 1 ≤ ν ≤ n. 5

Note that X ∗ is a subset of X 0 , the space of the algebraic duals.

4.1 Banach Spaces

105

Lemma 4.21. (a) Let (X, k·k1 ) and (X, k·k2 ) be two normed vector spaces with k·k1 ≤ C k·k2 . By Remark 4.2, completion yields Banach spaces (X1 , k·k1 ) and
∗ ∗
(X2 , k·k2 ) with X2 ⊂ X1 . The corresponding duals X1∗ , k·k1 and X2∗ , k·k2 ∗ ∗ ∗ ∗ ∗ satisfy X1 ⊂ X2 . The dual norms fulfil kϕk2 ≤ C kϕk1 for all ϕ ∈ X1 with the same constant C. If k·k1 and k·k2 are equivalent, X1∗ = X2∗ holds. ∗
∗
(b) If X ∗ , k·k1 and X ∗ , k·k2 are identical sets with equivalent dual norms ∗ ∗ k·k1 and k·k2 generated by normed vector spaces (X, k·k1 ) and (X, k·k2 ) , then k·k1 and k·k2 are also equivalent. Proof. (i) Let ϕ ∈ X1∗ . Since X2 ⊂ X1 , ϕ(x2 ) is well defined for any x2 ∈ X2 ∗ ∗ ∗ with kx2 k2 = 1, we obtain |ϕ(x2 )| ≤ kϕk1 kx2 k1 ≤ C kϕk1 kx2 k2 = C kϕk1 . ∗ ∗ Taking the supremum over all x2 ∈ X2 with kx2 k2 = 1, we get kϕk2 ≤ C kϕk1 . Again, by Remark 4.2, X1∗ ⊂ X2∗ follows. (ii) For equivalent norms both inclusions X1∗ ⊂ X2∗ ⊂ X1∗ prove X1∗ = X2∗ .   ∗ ∗ (iii) The identity kxk1 = max |ϕ(x)| : kϕk1 = 1 = maxϕ6=0 |ϕ(x)| / kϕk1 ∗ ∗ follows from (4.10). By norm equivalence, kϕk1 ≤ C kϕk2 follows so that   ∗ ∗ max |ϕ(x)| / kϕk1 ≥ C1 max |ϕ(x)| / kϕk2 = C1 kxk2 ϕ6=0

ϕ6=0

t u

and vice versa, proving Part (b).

The Banach space (X ∗ , k·kX ∗ ) has again a dual (X ∗∗ , k·kX ∗∗ ) called the bidual of X. The embedding X ⊂ X ∗∗ has to be understood as identification of x ∈ X with the bidual mapping χx ∈ X ∗∗ defined by χx (ϕ) := ϕ(x) for all ϕ ∈ X ∗ . If X = X ∗∗ , the Banach space X is called reflexive. Lemma 4.22. Let ϕ ∈ X ∗ . Then kϕkX ∗ = sup

06=x∈X

|ϕ(x)| |Φ(ϕ)| = max ∗∗ . 06=Φ∈X kxkX kΦkX ∗∗

If X is reflexive, kϕkX ∗ = max06=x∈X |ϕ(x)| / kxkX holds (max instead of sup!). Proof. The left equality holds by the definition of k·kX ∗ . The right equality is the identity (4.10) with x, X, ϕ replaced with ϕ, X ∗ , Φ. In the reflexive case, Φ(ϕ) = ϕ(xΦ ) holds for some xΦ ∈ X and proves the second part. t u Definition 4.23. If Φ ∈ L(X, Y ), the dual operator Φ∗ ∈ L(Y ∗ , X ∗ ) is defined via Φ∗ : η 7→ ξ := Φ∗ η with ξ(x) := η(Φx) for all x ∈ X. Lemma 4.24. kΦ∗ kX ∗ ←Y ∗ = kΦkY ←X . The embedding properties of X ⊂ Y are almost inherited by their dual spaces (cf. Definition 4.17). For a proof compare Hackbusch [141, Lemma 6.63]. Lemma 4.25. Let X ⊂ Y be a dense and continuous embedding of the Banach spaces. Then also Y 0 is continuously embedded in X 0 . If X is reflexive, the embedding Y 0 ⊂ X 0 is also dense.

106

4 Banach Tensor Spaces

4.1.6 Examples In §4.1.3 examples of Banach spaces are given. Some of the corresponding dual spaces are easy to describe. Example 4.26. (a) The dual of `p (I) for 1 ≤ p < ∞ is (isomorphic to) `q (I), where ∗ the conjugate q is defined by p1 + 1q = 1 and the embedding `q (I) ,→ (`p (I)) is defined by X ϕ (a) := ai ϕi ∈ K for a = (ai )i∈I ∈ `p (I) and ϕ = (ϕi )i∈I ∈ `q (I). i∈I

The subspace c0 (I) ⊂ `∞ (I) (cf. (4.4)) has the dual `1 (I). If #I < ∞, equality ∗ ∗ (`∞ (I)) = `1 (I) holds; otherwise, (`∞ (I)) % `1 (I). ∗ (b) Similarly, (Lp (D)) R ∼ = Lq (D) is valid for 1 ≤ p < ∞ with the embedding q g ∈ L (D) 7→ g(f ) := d f (x)g(x)dx for all f ∈ Lp (D). ∗

(c) Let I = [a, b] ⊂ R be an interval. Any functionalR ϕ ∈ (C(I)) corresponds to a function g of bounded variation such that ϕ(f ) = I f (x)dg(x) exists as Stieljes integral for f ∈ C(I). The latter integral with g chosen as the step function  all x≤s gs (x) := 01 for for x>s leads to the Dirac functional δs , which satisfies δs (f ) = f (s)

for all f ∈ C(I) and s ∈ I.

(4.11)

4.1.7 Weak Convergence Let (X, k·k) be a Banach space. We say that (xn )n∈N converges weakly to x ∈ X, if lim ϕ(xn ) = ϕ(x) for all ϕ ∈ X ∗ . In this case we write xn * x. Standard (strong) convergence xn → x implies xn * x. Lemma 4.27. If xn * x, then kxk ≤ lim inf kxn k . n→∞

∗

Proof. Choose ϕ ∈ X ∗ with kϕk = 1 and |ϕ(x)| = kxk (cf. (4.9)) and note that kxk ← |ϕ(xn )| ≤ kxn k. t u Exercise 4.28. Let the sequence {xn } be bounded. Show that xn * x already holds if lim ϕ(xn ) = ϕ(x) for all ϕ ∈ Ξ, where Ξ is dense in X ∗ . Conversely, xn * x implies that {xn } is bounded. Hint: For the latter statement use the uniform boundedness theorem (cf. [137, §3.4.7]). (i)

Lemma 4.29. Let N ∈ N. Assume that the sequences (xn )n∈N for 1 ≤ i ≤ N (i) converge weakly to linearly independent limits x(i) ∈ X (i.e., xn * x(i) ). Then (i) there is an n0 such that for all n ≥ n0 , the N -tuples (xn : 1 ≤ i ≤ N ) are linearly independent.

4.1 Banach Spaces

107

Proof. There exist functionals ϕ(j) ∈ X ∗ (1 ≤ j ≤ N ) with ϕ(j) (x(i) ) = δij (cf. Corollary 4.19). Set   N ∆n := det (ϕ(j) (x(i) )) n i,j=1 . (i)

(i)

xn * x(i) implies ϕ(j) (xn ) → ϕ(j) (x(i) ). Continuity of the determinant proves N ∆n → ∆∞ := det((δij )i,j=1 ) = 1. Hence there is an n0 such that ∆n > 0 for (i) u t all n ≥ n0 , but ∆n > 0 proves linear independence of {xn : 1 ≤ i ≤ N }. For the local sequential weak compactness used next see Yosida [306, Chap. V.2]. Lemma 4.30. If X is a reflexive Banach space, any bounded sequence xn ∈ X has a subsequence xnν converging weakly to some x ∈ X. Corollary 4.31. Pr Let X be a reflexive Banach space, xn ∈ X a bounded sequence with xn = i=1 ξn,i , ξn,i ∈ X, and kξn,i k ≤ C kxn k. Then there are Prξi ∈ X and a subsequence such that ξnν ,i * ξi and, in particular, xnν * x := i=1 ξi . Proof. By Lemma 4.30, weak convergence xn * x holds for n ∈ N(0) ⊂ N, where N(0) is an infinite subset of N. Because kξn,1 k ≤ C kxn k, ξn,1 is also a bounded sequence for n ∈ N(0) . A second infinite subset N(1) ⊂ N(0) exists with the property ξn,1 * ξ1 (n ∈ N(1) ) for some ξ1 ∈ X. Next, ξn,2 * ξ2 ∈ X can ξn,i be shown for n ∈ N(2) ⊂ N(1) , etc. Finally, for N(r) 3 n → ∞ all Psequences r u t converge weakly to ξi and summation over i yields xn * x := i=1 ξi . Definition 4.32. A subset M ⊂ X is called weakly closed, if xn ∈ M and xn * x imply x ∈ M . Note the implication ‘M weakly closed ⇒ M closed’, i.e., ‘weakly closed’ is a stronger statement than ‘closed’. Theorem 4.33. Let (X, k·k) be a reflexive Banach space with a weakly closed subset ∅ 6= M ⊂ X. Then the following minimisation problem has a solution: for any x ∈ X find v ∈ M with kx − vk = inf{kx − wk : w ∈ M }. Proof. Choose any sequence wn ∈ M with kx − wn k & inf{kx − wk : w ∈ M }. Since (wn )n∈N is a bounded sequence in X, Lemma 4.30 ensures the existence of a weakly convergent subsequence wni * v ∈ X. v belongs to M because wni ∈ M and M is weakly closed. Since x − wni * x − v is also valid, u Lemma 4.27 shows that kx − vk ≤ lim inf kx − wni k ≤ inf{kx − wk : w ∈ M }. t Since the assumption of reflexivity excludes important spaces, we add some remarks on this subject. The existence of a minimiser (‘nearest point’) v in a certain set A ⊂ X to some v ∈ V \A is a well-studied subject. A set A is called ‘proximinal’ if for all x ∈ X\A the best approximation problem kx − vk = inf w∈A kx − wk has at least one solution v ∈ A. Without the assumption of

108

4 Banach Tensor Spaces

reflexivity, there are statements ensuring under certain conditions that the set of points x ∈ X\A possessing nearest points in A are dense (e.g., Edelstein [85]). However, in order to conclude from the weak closedness6 of the minimal subspaces that they are proximinal, requires reflexivity as the following statement elucidates. Theorem 4.34 ([107, p. 61]). For a Banach space X the following is equivalent: (a) X is reflexive, (b) All closed subspaces are proximinal. (c) All weakly closed nonempty subsets are proximinal.

4.1.8 Continuous Multilinear Mappings Multilinearity is defined in (3.15). The equivalence of continuity and boundedness shown in Remark 4.12a also holds for bilinear, or generally, for multilinear mappings. The proof follows using the same arguments. Lemma 4.35. (a) A bilinear mapping B : (V, k·kV ) × (W, k·kW ) → (X, k·k) is continuous if and only if there is some C ∈ R such that kB(v, w)k ≤ C kvkV kwkW

for all v ∈ V and w ∈ W.


d (b) A multilinear mapping A : ×i=1 Vj , k·kj → (X, k·k) is continuous if and only if there is some C ∈ R such that Yd for all v (j) ∈ Vj . kA(v (1) , . . . , v (d) )k ≤ C kv (j) kj j=1

4.2 Topological Tensor Spaces 4.2.1 Notations The algebraic tensor product V ⊗a W has been defined in (3.9) by the span of all elementary tensors v ⊗w for v ∈ V and w ∈ W . In pure algebraic constructions such a span is always a finite linear combination. Infinite sums, as well as limits of sequences, cannot be defined without topology. In the finite-dimensional case, the algebraic tensor product V ⊗a W is already complete. Corollary 4.71 will even show a similar case with one factor being infinite dimensional. As already announced in (3.10), the completion X := X0 of X0 := V ⊗a W with respect to some norm k·k yields a Banach space (X, k·k), which is denoted On the other hand, if X coincides with the dual Y ∗ of another Banach space Y , every weak∗ closed set in X is proximinal (cf. Holmes [166, p. 123]). However, the later proofs of weak closedness of Ujmin (v) in §6 do not allow to conclude also weak∗ closedness. 6

4.2 Topological Tensor Spaces

109

by V ⊗k·k W := V ⊗ W := V ⊗a W

k·k

k·k

and now called a Banach tensor space. Note that the result of the completion depends on the norm k·k as already discussed in §4.1.2. A tensor x ∈ V ⊗k·k W is defined as a limit x = limn→∞ xn of some xn ∈ V ⊗a W from the algebraic tensor space, e.g., xn is the sum of say n elementary tensors. In general, such a limit of a sequence cannot be written as an infinite sum (but see §4.2.3.3). Furthermore, the convergence xn → x may be arbitrarily slow. However, in practice, one is interested in fast convergence, in order to approximate x by xn with reasonable n (if n is the number of elementary tensors involved, the storage will be related to n). In that case the statement xn → x should be replaced with a quantified error estimate: kxn − xk ≤ O(ϕ(n)) with ϕ(n) → 0 as n → ∞, (4.12) or kxn − xk ≤ o(ψ(n)) with sup ψ(n) < ∞, i.e., kxn − xk ≤ Cϕ(n) for some constant C or kxn − xk /ψ(n) → 0 as n → ∞. The notation ⊗k·k becomes a bit cumbersome if the norm sign k·k carries additional suffices, as e.g., k·kC n (I) . To shorten the notation, we often copy the (shortened) suffix7 of the norm to the tensor sign, e.g., the association p ↔ k·k`p (Z) or ∧ ↔ k·k∧(V,W ) is used in V ⊗ W = V ⊗p W, p

V ⊗ W = V ⊗∧ W, ∧

etc.

The neutral notation V ⊗W is used only if (i) there is not doubt about the norm of the Banach tensor space or (ii) a statement holds both for the algebraic tensor product and the topological one. In the finite-dimensional case, in which no completion is necessary, we shall use V ⊗ W for the algebraic product space (without any norm), as well as for any normed tensor space V ⊗ W . If V and W are infinite-dimensional spaces, it turns out that the algebraic tensor space V ⊗a W is only a rather small part of V ⊗k·k W . As remarked by 8 Uschmajew [289], V ⊗a W is a set of first category, S i.e., there are closed sets Xk (k ∈ N) with empty interior such that V ⊗a W = k∈N Xk . The precise statement Nd in Proposition 4.36 refers to the tensor space Valg := j=1 Vj of order d and will be proved at the end of §8.1. Proposition 4.36. Under the conditions (4.54a) or (4.54b) and if two of the spaces Vj are infinite dimensional, Valg is a set of first category. 7

The letter a must be avoided, since this denotes the algebraic product ⊗a . The complete space V ⊗k·k W cannot be of first category due to the theorem of Baire–Hausdorff (cf. [137, Theorem 3.40]). 8

110

4 Banach Tensor Spaces

4.2.2 Continuity of the Tensor Product, Crossnorms 4.2.2.1 Basic Facts We start from Banach spaces (V, k·kV ) and (W, k·kW ). The question arises as to whether the norms k·kV and k·kW define a norm k·k on V ⊗W in a canonical way. A suitable choice seems to be the definition kv ⊗ wk = kvkV kwkW

for all v ∈ V and w ∈ W

(4.13)

for the norm of elementary tensors. However, in contrast to linear maps (cf. Lemma 3.52), the definition of a norm on the set of elementary tensors does not determine k·k on the whole of V ⊗a W. Hence, unlike algebraic tensor spaces, the topological tensor space (V ⊗ W, k·k) is not uniquely determined by the components (V, k·kV ) and (W, k·kW ). Definition 4.37. Any norm k·k on V ⊗a W satisfying (4.13) is called a crossnorm. A necessary condition for k·k is the continuity of the tensor product; i.e., the mapping (v, w) ∈ V × W 7→ v ⊗ w ∈ V ⊗ W must be continuous. Since ⊗ is bilinear (cf. Lemma 3.12), we may apply Lemma 4.35a. Remark 4.38. The tensor product ⊗ : (V, k·kV ) × (W, k·kW ) → (V ⊗ W, k·k) is continuous if and only if there exists some C < ∞ such that kv ⊗ wk ≤ C kvkV kwkW

for all v ∈ V and w ∈ W.

(4.14)

Here (V ⊗W, k·k) may be the algebraic (possibly incomplete) tensor space V ⊗a W equipped with a norm k·k or a Banach tensor space (V ⊗k·k W, k·k). Inequality (4.14) is, in particular, satisfied by a crossnorm. Without continuity of ⊗, unexpected and unwanted effects may happen. Assume for instance that V is a Banach space and consider a sequence vi ∈ V with v = lim vi . Then the elementary tensors vi ⊗ w and v ⊗ w belong to V ⊗a W and therefore also to the completion V ⊗k·k W , but vi ⊗ w → v ⊗ w may not hold. For a proof, we use the following lemma (cf. Defant–Floret [74, I.1.2]). Lemma 4.39. Let V be a Banach space (W may be incomplete). Assume separate continuity: kv ⊗ wk ≤ Cw kvkV and kv ⊗ wk ≤ Cv kwkW hold for all v ∈ V and w ∈ W with Cv [Cw ] depending on v [w]. Then (4.14) follows for some C < ∞. Assuming that (4.14) is not true, we conclude from Lemma 4.39 that at least one of the statements about separate continuity is not valid. Without loss of generality, there is some w ∈ W such that sup06=v∈V kv ⊗ wk / kvkV = ∞. Fixing such a w, we obtain a norm ||| v |||w := kv ⊗ wk which is strictly stronger than k·kV . Then there are sequences vi → v ∈ V which do not converge with respect to the |||·|||w norm (cf. Remark 4.2).

4.2 Topological Tensor Spaces

111

In the definition of V ⊗k·k W we have not fixed whether V and W are complete or not. The next lemma shows that in any case the same Banach tensor space V ⊗k·k W results. Any incomplete normed space (V, k·kV ) may be regarded as a dense subset of the Banach space (V , k·kV ). Replacing the notations V , V by V, V0 , we can apply the following lemma. There we replace V and W in V ⊗ W with dense subspaces V0 , W0 . Three norms are involved: k·kV for V0 and V , k·kW for W0 and W , and k·k for the tensor spaces V0⊗aW0 , V ⊗aW , V ⊗k·kW. Lemma 4.40. Let V0 be dense in (V, k·kV ) and W0 be dense in (W, k·kW ). Assume ⊗ : (V, k·kV ) × (W, k·kW ) → (V ⊗k·k W, k·k) to be continuous; i.e., (4.14) holds. Then V0 ⊗a W0 is dense in V ⊗k·k W , i.e., V0 ⊗a W0 = V ⊗k·k W. Proof. For any ε > 0 and any x ∈ V ⊗k·k W we must show that there is an xε ∈ V0 ⊗a W0 with kx − xε k ≤ ε. By the definition of V ⊗k·k W , there is an Pn x0 ∈ V ⊗a W with kx − x0 k ≤ 2ε and a finite sum representation x0 = i=1 vi0 ⊗ wi0 with vi0 ∈ V and wi0 ∈ W . We set Cmax := max {kvi0 kV , kwi0 kW : 1 ≤ i ≤ n} , and choose δ so small that nCδ (2Cmax + δ) ≤ ε/2, where C is the equally named constant in (4.14). Select vi P ∈ V0 and wi ∈ W0 n with kvi0 − vi kV ≤ δ and kwi0 − wi kW ≤ δ and set xε := i=1 vi ⊗ wi . Then

n

X

0 0 0 kx − xε k = (vi ⊗ wi − vi ⊗ wi )

i=1

n

X

= {(vi0 − vi ) ⊗ wi0 + vi0 ⊗ (wi0 − wi ) + (vi − vi0 ) ⊗ (wi0 − wi )}

i=1

≤ ≤

n X  i=1 n X

k(vi0 − vi ) ⊗ wi0 k + kvi0 ⊗ (wi0 − wi )k + k(vi − vi0 ) ⊗ (wi0 − wi )k



≤ (4.14)

{C kvi0 − vi kV kwi0 kW + C kvi0 kV kwi0 − wi kW + C kvi − vi0 k kwi0 − wi k}

i=1

≤ nCδ {2Cmax + δ} ≤ ε/2 proves kx − xε k ≤ ε.

t u

112

4 Banach Tensor Spaces

Remark 4.41. If (V, k·kV ) and (W, k·kW ) are separable (cf. Definition 4.3) and continuity (4.14) holds, (V ⊗k·k W, k·k) is also separable. Proof. Let {vi : i ∈ N} and {wj : j ∈ N} be dense subsets of V and W , respectively. The set E := {vi ⊗ wj : i, j ∈ N} ⊂ V ⊗a W is again countable. By continuity of ⊗, the closure E contains all elementary tensors v ⊗ w ∈ V ⊗a W . Let B := span{E} be the set of finite linear combinations of vi ⊗ wj ∈ E. The previous result shows that V ⊗k·k W ⊃ B ⊃ V ⊗a W and proves B = V ⊗k·k W. t u

4.2.2.2 Examples Example 4.42 (`p ). Let V := (`p (I), k·k`p (I) ) and W := (`p (J), k·k`p (J) ) for 1 ≤ p ≤ ∞ and for some finite or countable I and J. Then the tensor product of the vectors a = (aν )ν∈I ∈ `p (I) and b = (bµ )µ∈J ∈ `p (J) may be considered as the (infinite) matrix a ⊗ b =: c = (cνµ )ν∈I,µ∈J with entries cνµ := aν bµ . Any elementary tensor a ⊗ b belongs to9 `p (I×J), and k·k`p (I×J) is a crossnorm: ka ⊗ bk`p (I×J) = kak`p (I) kbk`p (J)

for a ∈ `p (I), b ∈ `p (J).

(4.15)

Proof. For p < ∞, (4.15) follows from XX XX p p p p ka ⊗ bk`p (I×J) = |aν bµ | = |aν | |bµ | ν∈I µ∈J

=

X

|aν |

ν∈I µ∈J p

 X

ν∈I

p

|bµ |



 p = kak`p (I) kbk`p (J) .

µ∈J

For p = ∞, use |cνµ | = |aν bµ | = |aν | |bµ | ≤ kak`∞ (I) kbk`∞ (J) to show that ka ⊗ bk`∞ (I×J) ≤ kak`∞ (I) kbk`∞ (J) . Since there are indices ν ∗ , µ∗ with |aν ∗ | ≥ (1 − ε) kak`∞ (I)

and

|bµ∗ | ≥ (1 − ε) kbk`∞ (J)

the reverse inequality follows.

t u

Since linear combinations of finitely many elementary tensors again belong to `p (I × J), we obtain `p (I) ⊗a `p (J) ⊂ `p (I × J). Hence the completion with respect to k·k`p (I×J) yields a Banach tensor space `p (I) ⊗p `p (J) ⊂ `p (I × J). The next statement shows that, except for p = ∞, even equality holds. Remark 4.43. `p (I) ⊗ `p (J) = `p (I × J) holds for 1 ≤ p < ∞. p

9

Note that I ×J is again countable.

4.2 Topological Tensor Spaces

113

Proof. Let 1 ≤ p < ∞. It is sufficient to show that `p (I) ⊗a `p (J) is dense in `p (I × J). Let iν and jµ (ν, µ ∈ N) be any enumerations of the index sets I and J, respectively. Note that c ∈ `p (I ×J) has entries indexed by (i, j) ∈ I ×J. Then for each c ∈ `p (I × J), X p lim |ciν ,jµ |p = kck`p (I×J) n→∞

1≤ν,µ≤n

holds. Define c(n) ∈ `p (I × J) by  ciν ,jµ (n) ciν ,jµ = 0

for 1 ≤ ν, µ ≤ n, otherwise.

The previous limit shows c(n) → c as n → ∞. Let eνI ∈P `p (I)v and eµJ ∈ `p (J) (n) (n) be the unit vectors (cf. (2.2)). Then the identity c = 1≤ν,µ≤n ciν ,jµ eνI ⊗ eJµ shows that c(n) ∈ `p (I)⊗a `p (J). Since c = lim c(n) ∈ `p (I ×J) is an arbitrary element, `p (I) ⊗a `p (J) is dense in `p (I × J). t u For p = ∞ we consider the proper subspace c0 (I) $ `∞ (I) endowed with the same norm k·k`∞ (I) (cf. (4.4)). The previous proof idea can be used to show c0 (I) ⊗ c0 (J) = c0 (I × J). ∞

Next, we consider the particular case of p = 2, i.e., V := (`2 (I), k·k`2 (I) ) and W := (`2 (J), k·k`2 (J) ) for some finite or countable I and J. Again, c := a ⊗ b with a ∈ `2 (I) and b ∈ `2 (J), as well as any linear combination from V ⊗a W , may be considered as a (possibly infinite) matrix c = (cνµ )ν∈I,µ∈J . Theorem 4.137 will provide an infinite singular-value decomposition of c : c=

∞ X i=1

σ i vi ⊗ w i

with σ1 ≥ σ2 ≥ . . . ≥ 0 and orthonormal systems {vi } and {wi }.

(4.16)

Pn Algebraic tensors v = i=1 xi ⊗ yi lead to σi = 0 for all i > n. Therefore the sequence σ := (σi )i∈N belongs to `0 (N). Sequences (σi )i∈N with infinitely many nonzero entries can only appear for topological tensors. ∞

Definition 4.44 (Schatten norms). If the singular values σ = (σν )ν=1 of c in (4.16) have a finite `p norm kσkp , we set kckSVD,p := kσkp :=

∞ X

!1/p p

|σν |

for 1 ≤ p < ∞.

(4.17)

ν=1

As already seen in (4.15), k·k`2 (I×J) is a crossnorm. The next example shows that there is more than one crossnorm.

114

4 Banach Tensor Spaces

Example 4.45. Consider Example 4.42 for p = 2, i.e., V := (`2 (I), k·k`2 (I) ) and W := (`2 (J), k·k`2 (J) ). For any 1 ≤ p ≤ ∞, the Schatten10 norm k·kSVD,p is a crossnorm on V ⊗a W : ka ⊗ bkSVD,p = kak`2 (I) kbk`2 (J)

for a ∈ `2 (I), b ∈ `2 (J), 1 ≤ p ≤ ∞. (4.18)

As a consequence, ⊗ : V × W → V ⊗SVD,p W is continuous for all 1 ≤ p ≤ ∞. In particular, k·kSVD,2 = k·k`2 (I×J) holds. Proof. The rank-1 matrix c := a⊗b has the singular values σ1 = kak`2 (I) kbk`2 (J) and σi = 0 for i ≥ 2. Since the sequence σ := (σi ) has at most one nonzero entry, kσkp = σ1 holds for all 1 ≤ p ≤ ∞ and implies (4.18). For the last t u statement compare (2.16c). The norm (4.18) defines Banach spaces called Schatten classes. For details we refer to Schatten [255] and Meise [224, §II.16]. Example 4.46. (a) Let V = C(I) and W = C(J) be the spaces of continuous functions on some domains I and J with supremum norms k·kV = k·kC(I) and k·kW = k·kC(J) (cf. Example 4.10). Then the norm on V ⊗a W = C(I) ⊗a C(J)

k·k∞ = k·kC(I×J)

satisfies the crossnorm property (4.13). (b) If I and J are compact sets, the completion of V ⊗a W with respect to the maximum norm k·k∞ = k·kC(I×J) yields C(I) ⊗ C(J) = C(I × J). ∞

(4.19)

(c) For V = Lp (I) and W = Lp (J) (1 ≤ p < ∞), the crossnorm property (4.13) and the following identity hold: Lp (I) ⊗ Lp (J) = Lp (I × J) p

for 1 ≤ p < ∞.

Proof. (a) For Part (a) one proves (4.13) as in Example 4.42. (b) Obviously, polynomials in C(I × J) belong to C(I) ⊗a C(J). Since I, J, and hence I × J are compact, the Stone–Weierstrass Theorem (cf. Yosida [306, §0.2]) states that polynomials are dense in C(I × J). This implies that C(I) ⊗a C(J) is dense in C(I × J) , and (4.19) follows. t u (c) In the case of Lp , we may replace polynomials with step functions. 10

Brief remarks about Robert Schatten including his publication list can be found in [211, p. 138].

4.2 Topological Tensor Spaces

115

An obvious generalisation of Lp is the Sobolev space   ∂ 1,p p p H (D) := f ∈ L (D) : f ∈ L (D) for 1 ≤ j ≤ d ∂xj

for D ⊂ Rd

with the norm kf k1,p := kf kH 1,p (D) :=

p kf kp

d X

∂f p

+

∂xj

!1/p for 1 ≤ p < ∞.

p

j=1

Let I1 , I2 be intervals. Then the algebraic tensor space H 1,p (Ij ) ⊗a H 1,p (I2 ) is a dense subset of H 1,p (I1×I2 ). Hence the completion with respect to k·kH 1,p (I1 ×I2 ) yields H 1,p (I1 × I2 ) = H 1,p (I1 ) ⊗1,p H 1,p (I2 ). (4.20) Example 4.47 (H 1,p ). The tensor space (H 1,p (I1 × I2 ), k·kH 1,p (I1×I2 ) ) in (4.20) for11 1 ≤ p ≤ ∞ satisfies the continuity inequality (4.14) with C = 1, but the norm is not a crossnorm. Proof. For f = g ⊗ h, i.e., f (x, y) = g(x) h(y), we have p

p

p

p

p

p

p

kg ⊗ hkH 1,p (I1 ×I2 ) = kgkp khkp + kg 0 kp khkp + kgkp kh0 kp p

p

≤ kgk1,p khk1,p , where equality holds if and only if kg 0 kp kh0 kp = 0.

t u

For later use we introduce the anisotropic Sobolev space H (1,0),p (I1 × I2 ) := {f ∈ Lp (I1 × I2 ) : ∂f /∂x1 ∈ Lp (I1 × I2 )} with the norm  kf k(1,0),p := kf kH (1,0),p (I1 ×I2 ) := p

∂f p 1/p

p kf kp +

∂x1 p p

p

p

for 1 ≤ p < ∞. p

p

p

In this case kg ⊗ hkH (1,0),p (I1 ×I2 ) = kgkp khkp + kg 0 kp khkp = kgk1,p khkp proves the following result. Example 4.48. Let 1 ≤ p < ∞. The tensor space   H (1,0),p (I1 × I2 ), k·kH (1,0),p (I1 ×I2 ) = H 1,p (I1 ) ⊗(1,0),p Lp (I2 ) satisfies (4.13); i.e., k·kH (1,0),p (I1 ×I2 ) is a crossnorm. 11

To include p = ∞, define kf k1,∞ := max{kf k∞ , k∂f /∂x1 k∞ , k∂f /∂x2 k∞ }.

116

4 Banach Tensor Spaces

4.2.3 Projective Norm k·k∧(V,W ) 4.2.3.1 Definition and Properties Let k·k1 and k·k2 be two norms on V ⊗a W and denote the corresponding completions by V ⊗1 W and V ⊗2 W . If k·k1 . k·k2 , we already stated that V ⊗1 W ⊃ V ⊗2 W (cf. Remark 4.2). If the mapping (v, w) 7→ v ⊗ w is continuous with respect to the three norms k·kV , k·kW , k·k2 of V , W , V ⊗2 W , then it is also continuous for any weaker norm k·k1 of V ⊗1 W . For a proof, combine kv ⊗ wk2 ≤ C kvkV kwkW in (4.14) with k·k1 ≤ C 0 k·k2 to obtain boundedness kv ⊗ wk1 ≤ C 00 kvkV kwkW with the constant C 00 := CC 0 . On the other hand, if k·k1 is stronger than k·k2 , continuity may fail. Therefore we may ask for the strongest possible norm still ensuring continuity. Note that the strongest possible norm yields the smallest possible Banach tensor space containing V ⊗a W (with continuous ⊗). Since . is only a semi-ordering of the norms, it is not trivial that there exists indeed a strongest norm. The answer is given in the following exercise. Exercise 4.49. Let N be the set of all norms α on V ⊗aW satisfying α(v ⊗ w) ≤ Cα kvkV kwkW (v ∈ V , w ∈ W ). Replacing α by the equivalent norm α/Cα , we obtain the set N 0 ⊂ N of norms with α(v ⊗ w) ≤ kvkV kwkW . Define kxk := sup{α(x) : α ∈ N 0 } for x ∈ V ⊗a W and show that k·k ∈ N 0 is a norm α . k·k for all α ∈ N . The norm defined next will be the candidate for the strongest possible norm. Definition 4.50 (projective norm). The normed spaces (V, k·kV ) and (W, k·kW ) induce the projective norm12 k·k∧(V,W ) = k·k∧ on V ⊗a W defined by kxk∧(V,W ) := kxk∧ X  n n X vi ⊗ wi := inf kvi kV kwi kW : x =

(4.21) for x ∈ V ⊗a W.

i=1

i=1

Completion of V ⊗a W with respect to k·k∧(V,W ) defines the Banach tensor space 

 V ⊗ W, k·k∧(V,W ) . ∧

Note that the infimum in (4.21) is taken over all representations of x. Lemma 4.51. k·k∧(V,W ) is not only a norm but also a crossnorm. 12

Grothendieck [130] introduced the notations k·k∧ for the projective norm and k·k∨ for the injective norm from §4.2.4.2. The older notations by Schatten [254] are γ(·) for k·k∧(V,W ) and λ(·) for k·k∨(V,W ) .

4.2 Topological Tensor Spaces

117

Proof. (i) The first and third norm axioms in (4.1) are trivial. For the proof of the triangle inequality kx0 + x00 k∧ ≤ kx0 k∧ + kx00 k∧ choose any ε > 0. By Pn0 definition of the infimum in (4.21), there are representations x0 = i=1 vi0 ⊗ wi0 Pn00 and x00 = i=1 vi00 ⊗ wi00 with 0

n X

00

ε 2

n X

ε kvi00 kV kwi00 kW ≤ kx00 k∧ + . 2 i=1 i=1 Pn0 Pn00 A possible representation of x = x0 + x00 is x = i=1 vi0 ⊗ wi0 + i=1 vi00 ⊗ wi00 . Hence a bound of the infimum involved in kxk∧ is

kvi0 kV kwi0 kW ≤ kx0 k∧ +

0

kxk∧ ≤

n X

and

00

kvi0 kV

kwi0 kW

i=1

+

n X

kvi00 kV kwi00 kW ≤ kx0 k∧ + kx00 k∧ + ε.

i=1

Since ε > 0 is arbitrary, kxk∧ ≤ kx0 k∧ + kx00 k∧ follows. It remains to prove kxk∧ > 0 for x 6= 0. In §4.2.4.2 we shall introduce another norm kxk∨ for which kxk∨ ≤ kxk∧ will be shown in Lemma 4.61. Hence the norm property of k·k∨ proves that x 6= 0 implies 0 < kxk∨ ≤ kxk∧ . (ii) Definition (4.21) implies the inequality kv ⊗ wk∧ ≤ kvkV kwkW . Pn (iii) For the reverse inequality consider any representation v ⊗ w = i=1 vi ⊗wi . By Theorem 4.18 there is a continuous functional Λ ∈ V ∗ with Λ(v) = kvkV and kΛkV ∗ ≤ 1. The latter estimate implies |Λ (vi )| ≤ kvi kV . Applying Λ to the first Pn component in v ⊗ w = i=1 vi ⊗ wi , we conclude that Xn kvkV w = Λ(vi )wi i=1

(cf. Remark 3.64). The triangle inequality of k·kW yields Xn Xn kvkV kwkW ≤ |Λ(vi )| kwi kW ≤ kvi kV kwi kW . i=1

i=1

The infimum over all representations yields kvkV kwkW ≤ kv ⊗ wk∧ .

t u

Proposition 4.52. Given (V, k·kV ) and (W, k·kW ), the norm (4.21) is the strongest one ensuring continuity of (v, w) 7→ v ⊗ w. More precisely, if some norm k·k satisfies (4.14) with a constant C, then k·k ≤ C k·k∧ holds with the same constant. Proof. Let k·k be a norm on V ⊗a W such that (v, w) 7→ vP ⊗ w is continuous. Then n (4.14), i.e., kv ⊗ wk ≤ C kvkV kwkW holds. Let x = i=1 vi ⊗ wi ∈ V ⊗a W. Pn The triangle inequality yields kxk ≤ kv ⊗ w k. Together with (4.14), i i i=1 Pn kxk ≤ C kv k kw k follows. Taking the infimum over all representations i W Pn i=1 i V x = i=1 vi ⊗ wi , kxk ≤ C kxk∧ follows; i.e., k·k∧ is stronger than k·k. Since k·k is arbitrary under the continuity side condition, k·k∧ is the strongest norm. t u The property of k·k∧ as the strongest possible norm ensuring continuity justifies calling k·k∧(V,W ) a canonical norm of V ⊗a W induced by k·kV and k·kW. The completion V ⊗∧ W with respect to k·k∧ is the smallest one. Any other norm k·k with (4.14) leads to a larger Banach space V ⊗k·k W.

118

4 Banach Tensor Spaces

4.2.3.2 Examples Example 4.53 (`1 ). Consider V = (`1 (I), k·k`1 (I) ) and W = (`1 (J), k·k`1 (J) ) for some finite or countable sets I and J (cf. Example 4.42). The projective norm and the resulting Banach tensor space are k·k∧(V,W ) = k·k`1 (I×J)

and `1 (I) ⊗ `1 (J) = `1 (I × J). ∧

Proof. (i) c ∈ V ⊗a W ⊂ `1 (I × J) has entries cνµ for ν ∈ I and µ ∈ J. A possible representation of c is c=

XX

(ν)

(µ)

cνµ eV ⊗ eW

ν∈I µ∈J (ν)

(µ)

(ν)

(µ)

with unit vectors eV and eW (cf. (2.2)). By keV k`1 (I) = keW k`1 (J) = 1 and the definition of k·k∧(V,W ) , we obtain the inequality kck∧(V,W ) ≤

XX

|cνµ | = kck`1 (I×J) .

ν∈I µ∈J

(ii) k·k`1 (I×J) is a crossnorm (cf. (4.15)); i.e., (4.14) holds with C = 1. Hence Proposition 4.52 shows that kck`1 (I×J) ≤ kck∧(V,W ) . Together with the previous part, k·k∧(V,W ) = k·k`1 (I×J) follows. (iii) The latter identity together with Remark 4.43 (for p = 1) implies that the tensor product is equal to `1 (I) ⊗ `1 (J) = `1 (I × J). t u ∧

Example 4.54 (`2 ). Let V := (`2 (I), k·k`2 (I) ) and W := (`2 (J), k·k`2 (J) ) for some finite or countable sets I and J. Then k·k∧(V,W ) = k·kSVD,1

(cf. (4.17) for k·kSVD,p ).

Note that k·kSVD,1 k·kSVD,2 = k·k`2 (I×J) for #I, #J > 1. P Proof. (i) Let c ∈ V ⊗a W . Its singular-value decomposition c = i σi vi ⊗ wi has only finitely many nonzero singular values σi . By the definition of k·k∧(V,W ) we have X σi kvi kV kwi kW . kck∧(V,W ) ≤ i

kvi kV = kwi kW = 1 (vi , wi orthonormal) yields kck∧(V,W ) ≤ kσk1 = kckSVD,1 . (ii) k·kSVD,1 satisfies (4.14) with C = 1 (cf. (4.18)) so that Proposition 4.52 implies the opposite inequality kckSVD,1 ≤ kck∧(V,W ) . Together, the assertion t u kckSVD,1 = kck∧(V,W ) is proved.

4.2 Topological Tensor Spaces

119

4.2.3.3 Absolutely Convergent Series By definition, any topological tensor x ∈ V ⊗k·k W is the limit of some sequence Pnν (ν) (ν) (ν) (ν) vi ⊗ wi ∈ V ⊗a W. If, in particular, vi = vi and wi = wi x(ν) = i=1 P∞ are independent of ν, the partial sums xν define the series x = i=1 vi ⊗ wi . P∞ Below we state that there is an absolutely convergent series x = i=1 vi ⊗ wi , whose representation is almost optimal compared with the infimum kxk∧ . Proposition 4.55. For any ε > 0 and any x ∈ V ⊗∧ W, there is an absolutely convergent infinite sum x=

∞ X

(vi ∈ V, wi ∈ W )

vi ⊗ w i

(4.22a)

i=1

with

∞ X

kvi kV kwi kW ≤ (1 + ε) kxk∧ .

(4.22b)

i=1

Proof. We abbreviate k·k∧ by k·k and set εν := 3ε kxk /2ν . If x = 0, nothing has to be done. Otherwise choose some s1 ∈ V ⊗a W with kx − s1 k ≤ ε1 .

(4.23a)

Hence ks1 k P ≤ kxk + ε1 follows. By the definition of the norm, there is a represenn1 tation s1 = i=1 vi ⊗ wi with n1 X

kvi kV kwi kW ≤ ks1 k + ε1 ≤ kxk + 2ε1 .

(4.23b)

i=1

Set d1 := x − s1 and approximate d1 by s2 ∈ V ⊗a W such that kd1 − s2 k ≤ ε2 , s2 =

n2 X

vi ⊗ wi ,

i=n1 +1

n2 X

kvi kV kwi kW ≤ ks2 k + ε2 ≤ ε1 + 2ε2

i=n1 +1

(4.23c) (here we use ks2 k ≤ kd1 k + ε2 and kd1 k ≤ ε1 ; cf. (4.23a)). Analogously, we set d2 := d1 − s2 and choose s3 and its representation such that kd2 − s3 k ≤ ε3 , s3 =

n3 X

vi ⊗ wi ,

i=n2 +1

n3 X

kvi kV kwi kW ≤ ks3 k + ε3 ≤ ε2 + 2ε3 .

i=n2 +1

(4.23d) By induction, using (4.23a,c,d) we obtain statement (4.22a) in the form of ∞ ∞ P P x = sν = vi ⊗ wi . The estimates of the partial sums in (4.23b–d) show ν=1 i=1 P P∞ ∞ that i=1 kvi kV kwi kW ≤ kxk + 3 ν=1 εν = kxk + ε kxk, proving (4.22b). t u

120

4 Banach Tensor Spaces

4.2.4 Duals and Injective Norm k·k∨(V,W ) 4.2.4.1 Tensor Products of Dual Spaces The normed spaces (V, k·kV ) and (W, k·kW ) give rise to the dual spaces V ∗ and W ∗ endowed with the dual norms k·kV ∗ and k·kW ∗ described in (4.8). Consider the tensor space V ∗ ⊗a W ∗ . Elementary tensors ϕ ⊗ ψ from V ∗ ⊗a W ∗ may be viewed as linear forms on V ⊗a W via the definition (ϕ ⊗ ψ) (v ⊗ w) := ϕ(v) · ψ(w) ∈ K . As discussed in §3.3.2.4, any x∗ ∈ V ∗ ⊗a W ∗ is a linear form on V ⊗a W . Hence, 0

V ∗ ⊗a W ∗ ⊂ (V ⊗a W ) .

(4.24)

0

Note that (V ⊗a W ) is the algebraic dual since continuity is not yet ensured. ∗

A norm k·k on V ⊗a W leads to a dual space (V ⊗a W ) with a dual norm ∗ denoted by k·k . We would like to have ∗

V ∗ ⊗a W ∗ ⊂ (V ⊗a W )

(4.25) ∗

instead of (4.24). Therefore the requirement on the dual norm k·k
(and indirectly ∗ on k·k) is that ⊗ : (V ∗ , k·kV ∗ )×(W ∗ , k·kW ∗ ) → V ∗ ⊗ a W ∗ , k·k be continuous. The latter property, as seen in §4.2.2.1, is expressed by ∗

kϕ ⊗ ψk ≤ C kϕkV ∗ kψkW ∗

for all ϕ ∈ V ∗ and ψ ∈ W ∗ .

(4.26)

Lemma 4.56. Inequality (4.26) implies that kvkV kwkW ≤ C kv ⊗ wk

for all v ∈ V, w ∈ W,

(4.27)

which coincides with (4.14) up to the direction of the inequality sign. Furthermore, (4.26) implies the inclusion (4.25). Proof. Given v ⊗ w ∈ V ⊗a W , choose ϕ and ψ according to Theorem 4.18: kϕkV ∗ = kψkW ∗ = 1 and ϕ(v) = kvkV , ψ(w) = kwkW . Then ∗

kvkV kwkW = ϕ(v)ψ(w) = |(ϕ ⊗ ψ) (v ⊗ w)| ≤ kϕ ⊗ ψk kv ⊗ wk ≤ ≤ C kϕkV ∗ kψkW ∗ kv ⊗ wk = C kv ⊗ wk proves (4.27) with the same constant C as in (4.26).

(4.26)

t u

4.2 Topological Tensor Spaces

121 ∗

A similar result with a ‘wrong’ inequality sign for k·k is stated next. Lemma 4.57. Let k·k satisfy the continuity condition (4.14) with constant C. Then ∗ C kϕ ⊗ ψk ≥ kϕkV ∗ kψkW ∗ holds for all ϕ ∈ V ∗ and ψ ∈ W ∗ . Proof. The desired inequality follows from |ϕ(v)| |ψ(w)| |(ϕ ⊗ ψ) (v ⊗ w)| kϕ ⊗ ψk kv ⊗ wk = ≤ kvkV kwkW kvkV kwkW kvkV kwkW

≤ C kϕ ⊗ ψk (4.14)

for all 0 6= v ∈ V and 0 6= w ∈ W .

t u ∗

According
∗ to Remark 4.12c, we need not distinguish between (V ⊗a W ) and ∗ V ⊗k·k W , where k·k is the norm of V ⊗a W. The norm dual to k·k is k·k . ∗ ∗ ∗ The algebraic tensor product V ⊗a W can be completed with respect to k·k . Then (4.25) becomes
∗ V ∗ ⊗k·k∗ W ∗ ⊂ V ⊗k·k W . (4.28) Assuming infinite dimensions, the question arises whether the inclusion can be replaced by ‘=’ or ‘$’ (see Proposition 3.55 and Example 3.63 for the algebraic tensor case). In the topological case, both cases are possible as shown in Exercise 4.68. As in §4.2.2.1, we may ask for the strongest dual norm satisfying inequality (4.26). As seen in Lemma 4.21, weaker norms k·k correspond to stronger dual norms ∗ k·k . Therefore the following two questions are equivalent: ∗

• Which norm k·k on V ⊗a W yields the strongest dual norm k·k satisfying (4.26)? • What is the weakest norm k·k on V ⊗a W such that the corresponding dual norm ∗ k·k satisfies (4.26)? A candidate will be defined below. Since this will be again a norm determined only by k·kV and k·kW , it may also be called an induced norm.

4.2.4.2 Injective Norm Definition 4.58 (injective norm). Normed spaces (V, k·kV ) and (W, k·kW ) induce the injective norm k·k∨(V,W ) on V ⊗a W defined by kxk∨(V,W ) := kxk∨ :=

sup

|(ϕ ⊗ ψ) (x)| .

ϕ∈V ∗ ,kϕkV ∗ =1 ψ∈W ∗ ,kψkW ∗ =1

The completion of V ⊗a W with respect to k·k∨ defines (V ⊗∨ W, k·k∨ ).

(4.29)

122

4 Banach Tensor Spaces

Lemma 4.59. (a) k·k∨(V,W ) defined in (4.29) is a crossnorm on V ⊗a W ; i.e., (4.13) holds implying (4.14) and (4.27). ∗ (b) The dual norm kϕ ⊗ ψk∨(V,W ) is a crossnorm on V ∗ ⊗a W ∗ , i.e., ∗

kϕ ⊗ ψk∨(V,W ) = kϕkV ∗ kψkW ∗

for all ϕ ∈ V ∗ , ψ ∈ W ∗

(4.30)

holds, implying (4.26). Proof. (i) The norm axiom kλxk∨ = |λ| kxk∨ and the triangle inequality are standard. To show positivity Pr kxk∨ > 0 for 0 6= x ∈ V ⊗a W, apply Lemma 3.15: x has a representation x = i=1 vi ⊗ wi with linearly independent vi and wi . Note that r ≥ 1 because x 6= 0. Then there are normalised functionals ϕ ∈ V ∗ and ψ ∈ W ∗ with ϕ(v1 ) 6= 0 and ψ(w1 ) 6= 0, while ϕ(vi ) = ψ(wi ) = 0 for i ≥ 2. This leads to X  X r r |(ϕ⊗ψ)(x)| = (ϕ⊗ψ) vi ⊗wi = ϕ(vi )ψ(wi ) = |ϕ(v1 )ψ(w1 )| > 0. i=1

i=1

Hence kxk∨ ≥ |(ϕ ⊗ ψ) (x)| is also positive. (ii) Application of (4.29) to an elementary tensor v ⊗ w yields kv ⊗ wk∨(V,W ) = =

|(ϕ ⊗ ψ) (v ⊗ w)| =

sup 

|ϕ(v)|

sup

|ϕ(v)| |ψ(w)|

sup kϕkV ∗ =1 kψkW ∗ =1

kϕkV ∗ =1 kψkW ∗ =1



kϕkV ∗ =1

sup kψkW ∗ =1

|ψ(w)|



= kvkV kwkW .

(4.10)

(iii) For 0 6= ϕ⊗ψ ∈ V ∗ ⊗a W ∗ introduce the normalised continuous functionals ϕˆ := ϕ/ kϕkV ∗ and ψˆ := ψ/ kψkW ∗ . Then for all x ∈ V ⊗a W, the inequality
|(ϕ ⊗ ψ) (x)| = kϕkV ∗ kψkW ∗ ϕˆ ⊗ ψˆ (x) ≤ kϕkV ∗ kψkW ∗

sup 0

∗

0

ϕ ∈V ,kϕ kV ∗ =1 ψ 0 ∈W ∗ ,kψ 0 kW ∗ =1

|(ϕ0 ⊗ ψ 0 ) (x)| = kϕkV ∗ kψkW ∗ kxk∨

follows. The supremum over all x ∈ V ⊗a W with kxk∨ = 1 yields the dual norm ∗ so that kϕ ⊗ ψk∨(V,W ) ≤ kϕkV ∗ kψkW ∗ . This is already (4.26) with C = 1. (iv) Let ε > 0 and ϕ ⊗ ψ ∈ V ∗ ⊗a W ∗ be arbitrary. According to Remark 4.13, there are vε ∈ V and wε ∈ W with kvε kV = kwε kW = 1 and |ϕ(vε )| ≥ (1 − ε) kϕkV ∗

and

|ψ(wε )| ≥ (1 − ε) kψkW ∗ .

Note that, by (4.13), xε := vε ⊗ wε satisfies kxε k∨ = kvε kV kwε kW = 1. Hence ∗

kϕ ⊗ ψk∨(V,W ) = sup |(ϕ ⊗ ψ) (x)| ≥ |(ϕ ⊗ ψ) (xε )| = kxk∨ =1

= |(ϕ ⊗ ψ) (vε ⊗ wε )| =

|ϕ(vε )| |ψ(wε )|

2

≥ (1 − ε) kϕkV ∗ kψkW ∗ .

4.2 Topological Tensor Spaces

123 ∗

As ε > 0 is arbitrary, the reverse inequality kϕ ⊗ ψk∨(V,W ) ≥ kϕkV ∗ kψkW ∗ follows. Together with step (iii), we have proved (4.30). u t Proposition 4.60. k·k = k·k∨(V,W ) is the weakest norm on V ⊗a W subject to the ∗ additional condition that the dual norm k·k satisfy (4.26). Proof. Let x ∈ V ⊗a W . For any ε > 0, there is some ϕε ⊗ ψε ∈ V ∗ ⊗a W ∗ ∗ ∗ with kϕε kV = kψε kW = 1 and ∗

kxk∨(V,W ) ≤ (1 − ε)| (ϕε ⊗ ψε ) (x)| ≤ (1 − ε) kϕε ⊗ ψε k kxk ≤

(4.26)

≤ C (1 −

∗ ε) kϕε kV

∗ kψε kW

kxk = C (1 − ε) kxk

| {z }| {z } =1

=1

proving k·k∨(V,W ) ≤ C k·k; i.e., k·k∨(V,W ) is weaker than k·k.

t u

Lemma 4.61. k·k∨(V,W ) ≤ k·k∧(V,W ) holds on V ⊗a W. P Proof. Choose any ε > 0. Let x = i vi ⊗ wi ∈ V ⊗a W be some representation P with i kvi kV kwi kW ≤ kxk∧(V,W ) + ε. Choose normalised functionals ϕ and ψ with kxk∨(V,W ) ≤ |(ϕ ⊗ ψ) (x)| + ε. Then X X  kxk∨(V,W ) ≤ (ϕ ⊗ ψ) vi ⊗ wi + ε = ϕ(vi )ψ(wi ) + ε i i X X ≤ |ϕ(vi )| |ψ(wi )| + ε ≤ kvi kV kwi kW + ε ≤ kxk∧(V,W ) + 2ε. i

i

As ε > 0 is arbitrary, kxk∨(V,W ) ≤ kxk∧(V,W ) holds for all x ∈ V ⊗a W .

t u

Exercise 4.62. For any a1 , . . . , an ≥ 0 and b1 , . . . , bn > 0 show that min

1≤i≤n

ai a1 + . . . + an ai ≤ ≤ max . 1≤i≤n bi bi b1 + . . . + bn

So far, we have considered the norm kxk∨(V,W ) on V ⊗a W . Analogously, we can define kxk∨(V ∗ ,W ∗ ) on V ∗ ⊗a W ∗ . For the latter norm we shall establish a connection with k·k∧(V,W ) in §4.2.3.1, which states that in a certain sense the injective norm is dual to the projective norm.13 ∗

Proposition 4.63. k·k∨(V ∗ ,W ∗ ) = k·k∧(V,W ) on V ∗ ⊗a W ∗ . ∗

Proof. (i) The norm kΦk∧(V,W ) of Φ ∈ V ∗ ⊗a W ∗ is bounded by ∗

kΦk∧(V,W ) = 13

P |Φ( i vi ⊗ wi )| |Φ(x)| P = sup P 06=x∈V ⊗a W kxk∧(V,W ) 06=x= i vi ⊗wi ∈V ⊗a W i kvi kV kwi kW sup

The reverse statement is not true, but nearly (see Defant–Floret [74, §I.6]).

124

4 Banach Tensor Spaces

≤

sup P 06=x= i vi ⊗wi ∈V ⊗a W

P |Φ (vi ⊗ wi )| Pi kv i kV kwi kW i

≤

sup

Exercise 4.62 06=v∈V 06=w∈W

|Φ (v ⊗ w)| . kvkV kwkW

On the other hand, the elementary tensor x = v ⊗ w appearing in the last expression is only a subset of those x used in P |Φ ( i vi ⊗ wi )| ∗ P sup = kΦk∧(V,W ) P kv k kw k i i 06=x= i vi ⊗wi i V W ∗

so that kΦk∧(V,W ) must be an upper bound. Together, we arrive at ∗

kΦk∧(V,W ) =

sup 06=v∈V 06=w∈W

|Φ (v ⊗ w)| . kvkV kwkW

(4.31a)

P P (ii) Let Φ = ϕi ⊗ψi and set ϕ := ψi (w)ϕi ∈ V ∗ for some fixed w ∈ W . i i Then P | i ϕi (v)ψi (w)| |ϕ(v)| |v ∗∗ (ϕ)| sup = sup = sup ∗∗ ∗∗ kvk 06=v∈V 06=v∈V kvk Lemma 4.22 06=v ∗∗ ∈V ∗∗ kv k P ∗∗ | i v (ϕi )ψi (w))| = sup . ∗∗ ∗∗ ∗∗ kv ∗∗ k 06=v ∈V Similarly, the supremum over w can be replaced with a supremum over w∗∗ : P X | i ϕi (v)ψi (w)| |(v ∗∗ ⊗w∗∗ ) (Φ)| sup = sup for Φ = ϕ i ⊗ ψi . ∗∗ ∗∗ ∗∗ ∗∗ kvk kwk 06=v∈V 06=v ∗∗ ∈V ∗∗ kv k kw k i 06=w∗∗ ∈W ∗∗

06=w∈W

(4.31b) (iii) The left-hand side of (4.31b) coincides with the right-hand side of (4.31a) since P |Φ (v ⊗ w)| = | i ϕi (v)ψi (w)|. The right-hand side of (4.31b) coincides with the ∗ definition of kΦk∨(V ∗ ,W ∗ ) . Together, kΦk∧(V,W ) = kΦk∨(V ∗ ,W ∗ ) is shown. t u Corollary 4.64. The norm k·k∧(V,W ) satisfies not only (4.26) but also ∗

kϕ ⊗ ψk∧(V,W ) = kϕkV ∗ kψkW ∗

for all ϕ ∈ V ∗ , ψ ∈ W ∗

(cf. (4.30)).

∗

Proof. This is the crossnorm property for k·k∨(V ∗ ,W ∗ ) = k·k∧(V,W ) (cf. Proposition 4.63) stated in Lemma 4.59. t u Exercise 4.65. Let U ⊂ V be a closed subspace. The norm k·k = k·k∨(U,W ) is defined involving functionals ψ ∈ U ∗ . On the other hand we may restrict the norm k·k = k·k∨(V,W ) to U ⊗ W (involving ϕ ∈ V ∗ ). Show that both approaches lead to the same closed subspace U ⊗k·k W .

4.2 Topological Tensor Spaces

125

4.2.4.3 Examples Again, we consider the spaces `p (I) and `p (J). To simplify the reasoning, we first restrict the analysis to finite index sets I = J = {1, . . . , n}. The duals of `p (I) and `p (J) for 1 ≤ p < ∞ are `q (I) and `q (J) with p1 + 1q = 1 (cf. Example 4.26). Let ϕ ∈ `q (I) and ψ ∈ `q (J). The definition of k·k∨ makes use of (ϕ ⊗ ψ) (x), where for x = v ⊗ w the definition (ϕ ⊗ ψ) (v ⊗ w) = ϕ(v) · ψ (w) holds. The interpretation of ϕ(v) for a vector v ∈ `p (I) and a (dual) vector ϕ ∈ `q (I) is ϕ(v) := ϕT v. Similarly, ψ (w) = ψ T w. Elements from `p (I) ⊗ `p (J) = `p (I × J) are standard n × n matrices, which we shall denote by M . We recall that v ⊗ w (v ∈ `p (I), w ∈ `p (J)) corresponds to the matrix vwT . Hence, with M = vwT , the definition of (ϕ ⊗ ψ) (v ⊗ w) becomes ϕT M ψ ∈ K. This leads to the interpretation of kxk∨(V,W ) in (4.29) by kM k∨(`p (I),`p (J)) =

sup

|ϕTM ψ| . ϕ,ψ6=0 kϕkq kψkq

T ϕ M ψ = sup

kϕkq =kψkq =1

Remark 4.66. (a) Let 1 ≤ p < ∞ and assume #I > 1 and #J > 1. Then inequality k·k∨(`p (I),`p (J)) ≤ k·k`p (I×J) holds, but the corresponding equality is not valid. (b) For p = 2, k·k∨(`2 (I),`2 (J)) = k·kSVD,∞ 6= k·kSVD,2 = k·k`2 (I×J) holds with k·kSVD,p defined in (4.17). Proof. To prove k·k∨(`p (I),`p (J)) 6= k·k`p (I×J) , choose M =

h

1 1 1 −1

i

for the case14

I = J = {1, 2}. For instance for p = 1, kM k`1 (I×J) = 4, while an elementary |ϕTM ψ| analysis of kϕk kψk shows that kM k∨(`1 (I),`1 (J)) = 2. t u ∞

∞

In the case of the projective norm k·k∧(`p (I),`p (J)) in §4.2.3.1, we saw in §4.2.3.2 that the norms k·k∧(`p (I),`p (J)) and k·k`p (I×J) coincide for p = 1. Now coincidence happens for p = ∞. Remark 4.67. k·k∨(`∞ (I),`∞ (J)) = k·k`∞ (I×J) . (i)

(j)

(i)

(j)

Proof. Choose unit vectors eI ∈ `1 (I), eJ ∈ `1 (J). Then (eI )TM eJ = Mij (i) (j) (i) (i) (j)
and keI k1 = keJ k1 = 1 show that |(eI )TM eJj |/ keI k1 keJ k1 = |Mij | and kM k∨(`∞ (I),`∞ (J)) =

sup

sup

06=ϕ∈(`∞ (I))∗ 06=ψ∈(`∞ (J))∗

|ϕTM ψ| ∗ ∗ . kϕk∞ kψk∞

∗

`1 (I) is a subset of the dual space (`∞ (I)) . The particular choice of the unit (i) (j) vectors eI ∈ `1 (I) (i ∈ I) and eJ ∈ `1 (J) (j ∈ J) yields 14

This 2 × 2 example can be embedded in any model with #I, #J ≥ 2.

126

4 Banach Tensor Spaces (i)

kM k∨(`∞ (I),`∞ (J)) ≥ sup

i∈I,j∈J

(j)

|(eI )TM eJ | (i)

(j)

keI k1 keJ k1

= sup |Mij | = kM k`∞ (I×J) . i∈I,j∈J

On the other hand, for all i ∈ I we have X  X |(M ψ)i | = Mij ψj ≤ sup |Mij | |ψj | ≤ kM k`∞ (I×J) kψk1 j

j

j

implying kM ψk∞ ≤ kM k`∞ (I×J) kψk1 . Finally, |ϕTM ψ| ≤ kϕk1 kM ψk∞ proves the reverse inequality kM k∨(`∞ (I),`∞ (J)) ≤ kM k`∞ (I×J) . t u Set V := `∞ (I) ⊗∨ `∞ (J). The conclusion we can draw from Remark 4.67 is not V = `∞ (I × J), but that V is a closed subspace of `∞ (I × J). Exercise 4.68. (a) `1 (I)⊗a `1 (J) is dense in `1 (I×J), i.e., `1 (I)⊗`1 (I×J) `1 (J) = `1 (I ×J). The dual spaces of `1 (I), `1 (J), `1 (I × J) are isomorphic to `∞ (I), `∞ (J), `∞ (I × J). Hence (4.28) becomes `∞ (I) ⊗∨ `∞ (J) = `∞ (I) ⊗`∞ `∞ (J) ⊂ `∞ (I × J). Prove that the left-hand side is a proper subspace. (b) Let `2 (I) ⊗a `2 (J) be equipped with the norm k·k := k·kSVD,2 = k·k`2 (I×J) and prove that (4.28) holds with an equal sign. Among the function spaces, C(I) is of interest since again the supremum norm k·k∞ is involved. Remark 4.69. Let V = (C(I), k·kC(I) ) and W = (C(J), k·kC(J) ) with certain domains I and J (cf. Example 4.10). Then k·k∨(C(I),C(J)) = k·kC(I×J) . Proof. (i) Let f (·, ·) ∈ C(I × J). The duals C(I)∗ and C(J)∗ contain the delta functionals δx , δy (x ∈ I, y ∈ J) with kδx kC(I)∗ = 1 (cf. (4.11)). Hence, | (ϕ ⊗ ψ) f | kϕk ϕ,ψ6=0 C(I)∗ kψkC(J)∗

kf k∨(C(I),C(J)) = sup ≥

sup | (δx ⊗ δy ) f | = x∈I,y∈J

sup |f (x, y)| = kf kC(I×J) .

x∈I,y∈J

(ii) For the reverse inequality, consider the function fy := f (·, y) for fixed y ∈ J. Then fy ∈ C(I) has the norm kfy kC(I) ≤ kf kC(I×J) for all y ∈ J. Application of ϕ yields g(y) := ϕ(fy ) and |g(y)| = |ϕ(fy )| ≤ kϕkC(I)∗ kf kC(I×J) for all y ∈ J; hence, kgkC(J) ≤ kϕkC(I)∗ kf kC(I×J) . Application of ψ to g gives | (ϕ ⊗ ψ) f | = |ψ(g)| ≤ kψkC(J)∗ kgkC(J) ≤ kf kC(I×J) kϕkC(I)∗ kψkC(J)∗ implying kf k∨(C(I),C(J)) ≤ kf kC(I×J) . t u

4.2 Topological Tensor Spaces

127

4.2.5 Embedding of V ∗ into L(V ⊗ W, W ) The following result will be essential for the properties of the minimal subspaces discussed in §6. According to Remark 3.64, we regard V ∗ ⊂ V 0 as a subspace of L(V ⊗a W, W ) via ! X X X ∗ ϕ ∈ V 7→ ϕ ϕ(vi ) wi ∈ W. vi ⊗ w i = ϕ (vi ⊗ wi ) = i

i

i

The crucial question is as to whether the map x 7→ ϕ(x) is continuous, i.e., whether V ∗ ⊂ L (V ⊗a W, W ) . The supposition k·k & k·k∨(V,W ) of the next proposition is satisfied for all reasonable crossnorms (cf. §4.2.6.1 and Proposition 4.60). Proposition 4.70. If V ⊗a W is equipped with a norm k·k & k·k∨(V,W ) , the embedding V ∗ ⊂ L (V ⊗a W, W ) is valid. In particular, kϕ(x)kW ≤ kϕkV ∗ kxk∨(V,W ) ≤ C kϕkV ∗ kxk

for all ϕ ∈ V ∗ (4.32) and x ∈ V ⊗a W.

An analogous result holds for W ∗ ⊂ L (V ⊗ a W, V ) . Proof. For w := ϕ(x) ∈ W choose ψ ∈ W ∗ with ψ(w) = kwkW and kψkW ∗ = 1 (cf. Theorem 4.18). Then X  kϕ(x)kW = |ψ(ϕ(x))| P= ϕ(vi )wi ψ i x= i vi ⊗wi X ∗ = ϕ(vi )ψ(wi ) = |(ϕ ⊗ ψ) (x)| ≤ kϕ ⊗ ψk∨(V,W ) kxk∨(V,W ) = i

(4.30)

= kϕkV ∗ kψkW ∗ kxk∨(V,W ) = kϕkV ∗ kxk∨(V,W ) and kxk∨(V,W ) ≤ C kxk prove (4.32).

t u

Corollary 4.71. Let V and W be two Banach spaces, where either V or W are finite dimensional. Equip V ⊗a W with a norm k·k & k·k∨(V,W ) . Then V ⊗a W is already complete. Proof. Let dim(W ) = n and choose a basis {w1 , . . . , wn } of W .P According to n Remark 3.65a, all tensors x ∈ V ⊗a W may be written as x = i=1 vi ⊗ wi with vi = ϕi (x) ∈ V using a dual basis {ϕ1 , . . . , ϕn } . By dim(W ) < ∞ the functionals belong to W ∗ , and Proposition 4.70 implies that ϕi ∈ L (V ⊗a W, V ) . Let xk ∈ V ⊗a W be a Cauchy sequence. By continuity of ϕi , also vik = ϕi (xk ) k is a Cauchy sequence.PSince V is a Banach Pn space, vi → vi ∈ W converges and n k t u proves lim xk = lim i=1 vi ⊗ wi = i=1 vi ⊗ wi ∈ V ⊗a W.

128

4 Banach Tensor Spaces

4.2.6 Reasonable Crossnorms 4.2.6.1 Definition and Properties Now we combine the inequalities (4.14) and (4.26) (with C = 1); i.e., we require, simultaneously, continuity of ⊗ : (V, k·kV ) × (W, k·kW ) → (V ⊗a W, k·k) and ∗ ∗ ⊗ : (V ∗ , k·kV ∗ ) × (W ∗ , k·kW ∗ ) → (V ∗ ⊗a W ∗ , k·k ). Note that k·k is the dual norm of k·k. Definition 4.72. A norm k·k on V ⊗a W is a reasonable crossnorm15 if k·k satisfies kv ⊗ wk ≤ kvkV kwkW ∗

kϕ ⊗ ψk ≤ kϕkV ∗ kψkW ∗

for all v ∈ V and w ∈ W,

(4.33a)

for all ϕ ∈ V ∗ and ψ ∈ W ∗ .

(4.33b)

Lemma 4.73. If k·k is a reasonable crossnorm, then (4.33c,d) holds: kv ⊗ wk = kvkV kwkW

for all v ∈ V and w ∈ W,

∗

kϕ ⊗ ψk = kϕkV ∗ kψkW ∗

∗

(4.33c)

∗

for all ϕ ∈ V and ψ ∈ W .

(4.33d)

Proof. (i) Note that (4.33b) is (4.26) with C = 1. By Lemma 4.56, inequality (4.27) holds with C = 1, i.e., kv ⊗ wk ≥ kvkV kwkW . Together with (4.33a) we obtain (4.33c). ∗ (ii) Similarly, (4.33a) is (4.14) with C = 1. Lemma 4.57 proves kϕ ⊗ ψk ≥ t u kϕkV ∗ kψkW ∗ so that, together with (4.33b), identity (4.33d) follows. By the previous lemma, an equivalent definition of a reasonable crossnorm k·k ∗ is: k·k and k·k are crossnorms. Lemma 4.51, Corollary 4.64, and Lemma 4.59 prove that the norms k·k∧(V,W ) and k·k∨(V,W ) are particular reasonable crossnorms. Furthermore, Lemma 4.61, together with Propositions 4.52 and 4.60, shows the next statement. Proposition 4.74. k·k∨(V,W ) is the weakest and k·k∧(V,W ) is the strongest reasonable crossnorm; i.e., any reasonable crossnorm k·k satisfies k·k∨(V,W ) . k·k . k·k∧(V,W ) .

(4.34) ∗

Proposition 4.75. If k·k is a reasonable crossnorm on V ⊗ W , then k·k is also a reasonable crossnorm on V ∗ ⊗ W ∗ . ∗

Proof. (i) We must show that k·k satisfies (4.33a,b) with k·k, V , W replaced with ∗ k·k , V ∗ , W ∗ . The reformulated inequality (4.33a) is (4.33b). Hence this condition is satisfied by assumption. It remains to show the reformulated version of (4.33b): 15

Also the name ‘dualisable crossnorm’ has been used (cf. [263]). Schatten [254] used the term ‘crossnorm whose associate is a crossnorm’ (‘associate norm’ means dual norm).

4.2 Topological Tensor Spaces ∗∗

kv ∗∗ ⊗ w∗∗ k

129

≤ kv ∗∗ kV ∗∗ kw∗∗ kW ∗∗ for all v ∗∗ ∈ V ∗∗ , w∗∗ ∈ W ∗∗ . (4.35a)

(ii) By Proposition 4.74, k·k∨(V,W ) ≤ k·k ≤ k·k∧(V,W ) is valid. Applying ∗∗ ∗∗ ∗∗ Lemma 4.21 twice, we see that k·k∨(V,W ) ≤ k·k ≤ k·k∧(V,W ) holds for the bidual norm; in particular, kv ∗∗ ⊗ w∗∗ k

∗∗

∗∗

≤ kv ∗∗ ⊗ w∗∗ k∧(V,W ) .

(4.35b)

∗

Proposition 4.63 states that k·k∧(V,W ) = k·k∨(V ∗ ,W ∗ ) implying ∗∗

∗

k·k∧(V,W ) = k·k∨(V ∗ ,W ∗ ) .

(4.35c)

Since k·k∨(V ∗ ,W ∗ ) is a reasonable crossnorm on V ∗ ⊗ W ∗ (cf. Proposition 4.74), it satisfies the corresponding inequality (4.33b): ∗

kv ∗∗ ⊗ w∗∗ k∨(V ∗ ,W ∗ ) ≤ kv ∗∗ kV ∗∗ kw∗∗ kW ∗∗ for all v ∗∗ ∈ V ∗∗ , w∗∗ ∈ W ∗∗ . (4.35d) t u Now the equations (4.35b–d) prove (4.35a). In general, different reasonable crossnorms are not equivalent. Exceptions occur for finite-dimensional spaces or subspaces. However, the next statement of Fern´andez-Unzueta [100] also shows equivalence on the sets Rr of tensors of rank bounded by r (cf. §3.2.6). Nd Theorem 4.76. Let k·k and |||·||| be two reasonable crossnorms on V = j=1 Vj . Then for all r ∈ N the following estimates hold: kvk ≤ rd−1 |||v|||,

kv − wk ≤ (2r)

d−1

|||v − w|||

for all v, w ∈ Rr .

As a consequence, the closure of Rr is independent of the particular reasonable crossnorm.

4.2.6.2 Examples and Counterexamples Example 4.77 (`2 ). Let V = `2 (I) and W = `2 (J) for finite or countable index sets I, J with norms k·k`2 (I) and k·k`2 (J) . Then all norms k·kSVD,p for 1 ≤ p ≤ ∞ are reasonable crossnorms on `2 (I) ⊗a `2 (J). In particular, k·k∨(`2 (I),`2 (J)) = k·kSVD,∞ ≤ k·kSVD,p ≤ k·kSVD,q ≤ k·kSVD,1 = k·k∧(`∞ (I),`∞ (J))

for all p ≥ q ≥ 1.

Example 4.78 (`p ). k·k`p (I×J) is a reasonable crossnorm for 1 ≤ p < ∞. Proof. (4.15) proves that k·k`p (I×J) satisfies (4.33a). The same statement (4.15) t u for p replaced with q (defined by p1 + 1q = 1) shows (4.33b).

130

4 Banach Tensor Spaces

The next example can be shown analogously. Example 4.79 (Lp ). Let V = Lp (I) and W = Lp (J) for intervals I and J. Then k·kLp (I×J) is a reasonable norm on V ⊗a W = Lp (I × J) for 1 ≤ p < ∞. For the next example we recall that kf kC 1 (I) = max {|f (x)| , |f 0 (x)|} x∈I

is the norm of continuously differentiable functions in one variable x ∈ I ⊂ R. The name k·k1,mix of the following norm is derived from the mixed derivative involved. Example 4.80. Let I, J be compact intervals in R and set V = (C 1 (I), k·kC 1 (I) ), W = (C 1 (J), k·kC 1 (J) ). For the tensor space V ⊗a W we introduce the mixed norm kϕkC 1

mix (I×J)

:= kϕk1,mix  :=

max (x,y)∈I×J

(4.36) 2  ∂ϕ(x, y) ∂ϕ(x, y) ∂ ϕ(x, y) , , . |ϕ(x, y)| , ∂x ∂y ∂x∂y

Then k·k1,mix is a reasonable crossnorm. Proof. kf ⊗ gkC 1 (I×J) ≤ kf kC 1 (I) kgkC 1 (J) is easy to verify. The proof of mix (4.33b) uses similar ideas to those from the proof of Remark 4.69. t u However, the standard norm for C 1 (I × J) is  kϕkC 1 (I×J) =

max (x,y)∈I×J

As k·kC 1 (I×J) ≤ k·kC 1

mix (I×J)

 ∂ ∂ |ϕ(x, y)| , ϕ(x, y) , ϕ(x, y) . ∂x ∂y

, the inequality

kf ⊗ gkC 1 (I×J) ≤ kf kC 1 (I) kgkC 1 (J) proves (4.33a). However, the second inequality (4.33b) characterising a reasonable crossnorm cannot be satisfied. For a counterexample, choose the continuous functionals δx0 0 ∈ V ∗ (x0 ∈ I) and δy0 0 ∈ V ∗ (y0 ∈ J) defined by δx0 0 (f ) = −f 0 (x0 )
and δy0 0 (g) = −g 0 (y0 ). Then δx0 0 ⊗δy0 0 (f ⊗g) = f 0 (x0 )g 0 (y0 ) cannot be bounded ∂2 by kf ⊗gkC 1 (I×J) , since the term | ∂x∂y ϕ(x, y)| from (4.36) is missing. For the treatment of this situation we refer to §4.3.5. An analogous situation happens for H 1,p (I × J) = H 1,p (I) ⊗1,p H 1,p (J) from Example 4.47. As mentioned in Example 4.47, k·kH 1,p (I×J) is no crossnorm. 1,p Furthermore, it does not satisfy (4.26). On the other hand, Hmix (I × J) allows the

p
1/p p p p crossnorm kf k + k∂f /∂xk + k∂f /∂yk + ∂ 2 f /∂x∂y . p

p

p

p

4.2 Topological Tensor Spaces

131

The anisotropic Sobolev space H (1,0),p (I1 × I2 ) = H 1,p (I1 ) ⊗(1,0),p Lp (I2 ) is introduced in Example 4.48. Remark 4.81. The norm k·k(1,0),p = k·kH (1,0),p (I1 ×I2 ) on H (1,0),p (I1 × I2 ) is a reasonable crossnorm for 1 ≤ p < ∞. Proof. As already stated in Example 4.48, the norm satisfies (4.33c). It remains to prove (4.33b). The functionals ϕ ∈ V ∗ := (H 1,p (I1 ))∗ and ψ ∈ W ∗ := Lp (I2 )∗ ∗ may be normalised: kϕkV ∗ = kψkW ∗ = 1. Then kϕ ⊗ ψk ≤ 1 has to be proved. ∗ By the definition of k·k , it is sufficient to show for all f ∈ H (1,0),p (I1 × I2 ).

|(ϕ ⊗ ψ) (f )| ≤ kf k(1,0),p

Next, we may restrict f ∈ H (1,0),p (I1 ×I2 ) to the dense subset f ∈ C ∞ (I1 × I2 ). As stated in Example 4.26b, the dual space W ∗ can be identified with Lq (I2 ), i.e., ψ ∈ Lq (I2 ) and kψkq = 1. Application of ψ ∈ W ∗ to f yield the following function of x ∈ I1 : Z

f (x, y)ψ(y)dy ∈ C ∞ (I1 ).

F (x) := I2

The functional ϕ ∈ V ∗ acts with respect to the x-variable: Z (ϕ ⊗ ψ) (f ) = ϕ(F ) =

ϕ [f (·, y)] ψ(y)dy. I2

For a fixed y ∈ I2 , the estimate |ϕ [f (·, y)]| ≤ kϕkV ∗ kf (·, y)k1,p = kf (·, y)k1,p implies that Z |(ϕ ⊗ ψ) (f )| ≤ I2

≤

Z I2

p kf (·, y)k1,p

kf (·, y)k1,p ψ(y)dy dy

1/p  Z I2

q

|ψ(y)| dy

1/q

sZ

= kψkq p | {z }

I2

p

kf (·, y)k1,p dy

=1

sZ Z =

p

I2

I1

p  ∂ p |f (x, y)| + f (x, y) dx dy = kf k(1,0),p . ∂x

t u

Although the solution of elliptic partial differential equations is usually a 1,p function of the standard Sobolev spaces and not of mixed spaces as Hmix , there are important exceptions. As proved by Yserentant [307], the solutions of the electronic Schr¨odinger equation have mixed regularity because they must be antisymmetric (Pauli principle).

132

4 Banach Tensor Spaces

4.2.7 Reflexivity Let k·k be a crossnorm norm on V ⊗aW . The dual space of V ⊗k·kW or16 V ⊗aW
∗ is V ⊗k·k W . From (4.25) we derive that V ∗ ⊗k·k∗ W ∗ ⊂ V ⊗k·k W


∗

.

(4.37)

Lemma 4.82. Assume that V ⊗k·k W is equipped with a reasonable crossnorm k·k and is reflexive. Then the identity V ∗ ⊗k·k∗ W ∗ = V ⊗k·k W


∗

(4.38)

holds. Furthermore, the spaces V and W are reflexive. Proof. (i) For an indirect proof, assume that (4.38) is not valid. Then there is some / V ∗ ⊗k·k∗ W ∗ . By Hahn–Banach there is some bidual φ ∈ (V ⊗k·k W )∗ with φ ∈ Φ ∈ (V ⊗k·k W )∗∗ such that Φ(φ) 6= 0, while Φ(ω) = 0 for all ω ∈ V ∗ ⊗k·k∗ W ∗ . Because of the reflexivity, Φ(φ) has a representation as φ(xΦ ) for some 0 6= xΦ ∈ V ⊗k·k W . As 0 6= xΦ implies kxΦ k∨(V,W ) > 0, there is some ω = ϕ ⊗ ψ ∈ V ∗ ⊗a W ∗

such that |ω(xΦ )| > 0.

This is in contradiction to 0 = Φ(ω) = ω(xΦ ) for all ω ∈ V ∗ ⊗a W ∗ . Hence identity (4.38) must hold. (ii) The statement analogous to (4.37) for the dual spaces is
∗
∗∗ = V ⊗k·k W, = V ⊗k·k W V ∗∗ ⊗k·k∗∗ W ∗∗ ⊂ V ∗ ⊗k·k∗ W ∗ (4.38)

implying V ∗∗ ⊂ V and W ∗∗ ⊂ W . Together with the general property V ⊂ V ∗∗ and W ⊂ W ∗∗ , we obtain reflexivity of V and W . t u The last lemma shows that reflexivity of the Banach spaces V and W is necessary for V ⊗k·k W to be reflexive. One might expect that reflexivity of V and W is also sufficient; i.e., the tensor product of reflexive spaces is again reflexive. This is not the case as the next example shows (for a proof see Schatten [254, p. 139]; note that the Banach spaces `p (N) are reflexive for 1 < p < ∞). Example 4.83. For 1 < p < ∞ and

1 p

+

1 q

= 1, `p (N) ⊗ `q (N) is nonreflexive. ∨

A Banach space X and any dense subspace X0 ⊂ X yield the same dual space X ∗ = X0∗ (cf. Lemma 4.40). 16

4.2 Topological Tensor Spaces

133

4.2.8 Uniform Crossnorms Let (V ⊗a W, k·k) be a tensor space with crossnorm k·k and consider operators A ∈ L(V, V ) and B ∈ L(W, W ) with operator norms kAkV ←V and kBkW ←W . As discussed in §3.3.2.1, A ⊗ B is defined on elementary tensors v ⊗ w via (A ⊗ B) (v ⊗ w) := (Av) ⊗ (Bw) ∈ V ⊗a W. While A ⊗ B is well-defined on finite linear combinations from V ⊗a W , the question is as to whether A ⊗ B : V ⊗a W → V ⊗a W is (uniformly) bounded, i.e., A ⊗ B ∈ L(V ⊗a W, V ⊗a W ). In the positive case, A ⊗ B also belongs to L(V ⊗k·k W, V ⊗k·k W ). For elementary tensors, the estimate k(A ⊗ B) (v ⊗ w)k = k(Av) ⊗ (Bw)k = kAvk kBwk

(4.39)

≤ kAkV ←V kBkW ←W kvkV kwkW = kAkV ←V kBkW ←W kv ⊗ wk follows by the crossnorm property. However, this inequality does not automatically extend to general tensors from V ⊗a W . Instead, the desired estimate is the subject of the next definition (cf. Schatten [254]). Definition 4.84. A crossnorm on V ⊗a W is called uniform if A⊗B for all A ∈ L(V, V ) and B ∈ L(W, W ) belongs to L(V ⊗a W, V ⊗a W ) with the operator norm kA ⊗ BkV ⊗a W ←V ⊗a W ≤ kAkV ←V kBkW ←W . (4.40) Taking the supremum over all v ⊗ w with kv ⊗ wk = 1, we conclude from (4.39) that kA ⊗ BkV ⊗a W ←V ⊗a W ≥ kAkV ←V kBkW ←W . Therefore we may replace inequality (4.40) with kA ⊗ BkV ⊗a W ←V ⊗a W = kAkV ←V kBkW ←W . Proposition 4.85. k·k∧(V,W ) and k·k∨(V,W ) are uniform crossnorms. P

vi ⊗wi ∈ V ⊗a W and A ∈ L(V, V ), B ∈ L(W, W ). Then

X

X kAvi kV kBwi kW ≤ (Avi )⊗(Bwi ) k(A ⊗ B) (x)k∧(V,W ) = i i ∧(V,W ) X ≤ kAkV ←V kBkW ←W kvi kV kwi kW

Proof. (i) Let x =

i

i

P

holds for all representations x = i vi ⊗ wi . The infimum over all representations yields k(A ⊗ B) (x)k∧(V,W ) ≤ kAkV ←V kBkW ←W kxk∧(V,W ) ,

134

4 Banach Tensor Spaces


i.e., (4.40) holds for V ⊗a W, k·k∧(V,W ) . P (ii) Let x = i vi ⊗ wi ∈ V ⊗a W and note that k(A ⊗ B) xk∨(V,W ) =

sup 06=ϕ∈V

=

∗ ,06=ψ∈W ∗

sup 06=ϕ∈V ∗ ,06=ψ∈W ∗

=

sup 06=ϕ∈V ∗ ,06=ψ∈W ∗

|(ϕ ⊗ ψ) ((A ⊗ B) x)| kϕkV ∗ kψkW ∗ P | i (ϕ ⊗ ψ) ((Avi ) ⊗ (Bwi ))| kϕkV ∗ kψkW ∗ P | i (ϕ(Avi ) · ψ(Bwi )| . kϕkV ∗ kψkW ∗

By Definition 4.23, A∗ ∈ L(V ∗ , V ∗ ) and B ∗ ∈ L(W ∗ , W ∗ ) satisfy X X ϕ(Avi ) · ψ(Bwi ) = (A∗ ϕ) (vi ) · (B ∗ ψ) (wi ) i

i

= |((A∗ ϕ) ⊗ (B ∗ ψ)) (x)| . We continue: k(A ⊗ B) xk∨(V,W ) = =

|((A∗ ϕ) ⊗ (B ∗ ψ)) (x)| kϕkV ∗ kψkW ∗

sup

06=ϕ∈V ∗ ,06=ψ∈W ∗ kA∗ ϕkV ∗

sup

06=ϕ,06=ψ

kϕkV ∗

kB ∗ ψkW ∗ |((A∗ ϕ) ⊗ (B ∗ ψ)) (x)| . kψkW ∗ kA∗ ϕkV ∗ kB ∗ ψkW ∗

By Lemma 4.24, the inequalities kA∗ ϕkV ∗ ≤ kA∗ kV ∗ ←V ∗ = kAkV ←V kϕkV ∗ and

kB ∗ ψkW ∗ ≤ kB ∗ kW ∗ ←W ∗ = kBkW ←W kψkW ∗

hold, while |((A∗ ϕ) ⊗ (B ∗ ψ)) (x)| ≤ kxk∨(V,W ) . kA∗ ϕkV ∗ kB ∗ ψkW ∗ Together, k(A ⊗ B) xk∨(V,W ) ≤ kAkV ←V kBkW ←W kxk∨(V,W ) proves that k·k∨(V,W ) is also uniform.

t u

By definition, a uniform crossnorm is a crossnorm. Exercise 4.86. Condition (4.40) already implies that, up to a suitable scaling, the norm is a crossnorm. Hint: use similar maps Φ and Ψ as in the next proof.

4.2 Topological Tensor Spaces

135

As shown in Simon [263], it is also a reasonable crossnorm. Lemma 4.87. A uniform crossnorm is a reasonable crossnorm. Proof. Let ϕ ∈ V ∗ and ψ ∈ W ∗ and choose some 0 6= v ∈ V and 0 6= w ∈ W . Define the operator Φ ∈ L(V, V ) by Φ = vϕ (i.e., Φ(x) = ϕ(x) · v) and, similarly, Ψ ∈ L(W, W ) by Ψ := wψ. The identities kΦkV ←V = kvkV kϕkV ∗ and kΨ kW ←W = kwkW kψkW ∗ are valid, as well as (Φ ⊗ Ψ ) (x) = ((ϕ ⊗ ψ) (x)) · (v ⊗ w)

for all x ∈ X : = V ⊗a W.

The crossnorm property yields k(Φ ⊗ Ψ ) (x)k = |(ϕ ⊗ ψ) (x)| kv ⊗ wk = |(ϕ ⊗ ψ) (x)| kvkV kwkW , while the uniform crossnorm property allows the estimate |(ϕ ⊗ ψ) (x)| kvkV kwkW = k(Φ ⊗ Ψ ) (x)k ≤ kΦkV ←V kΨ kW ←W kxk = kvkV kϕkV ∗ kwkW kψkW ∗ kxk . Dividing by kvkV kwkW 6= 0, we obtain |(ϕ ⊗ ψ) (x)| ≤ kϕkV ∗ kψkW ∗ kxk for u t all x ∈ X. Hence k·k is a reasonable crossnorm. Proposition 4.88. Suppose that the Banach spaces V and W are reflexive. If k·k ∗ is a uniform crossnorm on V ⊗a W , then k·k is also a uniform and reasonable ∗ ∗ crossnorm on V ⊗a W . Proof. By Lemma 4.87, k·k is a reasonable crossnorm, while by Proposition 4.75 ∗ k·k is also a reasonable crossnorm. To prove uniformity, let A∗ ∈ L(V ∗ , V ∗ ) and ∗ ∗ B ∗ ∈ L(W ∗ , W ∗ ). Because of the reflexivity, the adjoint Poperators of A ∗ and B ∗ ∗ ∈ For x V ψ B = W ∈ L(V, are A∗∗ = ∈ L(W, W ). ) A and ϕ ⊗ ⊗ V i a i i P and x = j vj ⊗ wj ∈ V ⊗a W we have
XX ∗ ∗ (A∗ ⊗ B ∗ ) (x∗ ) (x) = (A ϕi ) (vj ) · (B ψi ) (wj ) i

j

∗ XX
= ϕi (Avj ) · ψi (Bwj ) = x (A ⊗ B) (x) i

j

∗ ∗

∗

≤

≤ kx k k(A ⊗ B) (x)k

k·k uniform ∗

From k(A∗ ⊗ B ∗ ) (x∗ )k = sup |((A x6=0

∗

kx∗ k kAkV ←V kBkW ←W kxk .

⊗B ∗ )(x∗ ))(x)| kxk

∗

≤ kAkV ←V kBkW ←W kx∗ k ,

kAkV ←V = kA∗ kV ∗ ←V ∗ , and kBkW ←W = kB ∗ kW ∗ ←W ∗ we derive that the ∗ dual norm k·k is uniform. t u

136

4 Banach Tensor Spaces

4.2.9 Nuclear and Compact Operators Suppose that V and W are Banach spaces and consider the tensor space V ⊗a W ∗ . The inclusion V ⊗a W ∗ ⊂ L(W, V ) is defined via (v ⊗ ψ) (w) := ψ(w)v ∈ V

for all v ∈ V, ψ ∈ W ∗ , w ∈ W.

Similarly as in Proposition 3.68a, V ⊗a W ∗ is interpreted as a subspace of L(W, V ) and denoted by F(W, V ). Elements Φ ∈ F(W, V ) are called finite-rank operators. We recall Definition 4.14: K(W, V ) is the set of compact operators. Definition 4.89. A Banach space X has the approximation property if for any compact set K ⊂ X and ε > 0 there is ΦK,ε ∈ F(X, X) with sup kΦK,ε x − xkV ≤ ε. x∈K

Proposition 4.90. (a) The completion with respect to the operator norm k·kV ←W in (4.6a) yields F(W, V ) ⊂ K(W, V ). (b) Sufficient for F(W, V ) = K(W, V ) is the approximation property of W ∗ . Proof. Φ ∈ F(W,V ) is compact since its range is finite dimensional. Part (a) follows, because limits of compact operators are compact. For Part (b), see [211, p. 17]. t u Next, we relate the operator norm k·kV ←W with the crossnorms of V ⊗k·k W ∗ . Lemma 4.91. kΦkV ←W ≤ kΦk∨(V,W ∗ ) holds for all Φ ∈ V ⊗∨ W ∗ . Reflexivity of W implies the equality k·kV ←W = k·k∨(V,W ∗ ) . Proof. kΦk∨(V,W ∗ ) is the supremum of |(ϕ⊗w∗∗ )(Φ)| over all normalised ϕ ∈ V ∗ and w∗∗ ∈ W ∗∗ . Replacing W ∗∗ by its subspace W , we get a lower bound: kΦk∨(V,W ∗ ) ≥ =

sup

|(ϕ ⊗ w) (Φ)| =

kϕkV ∗ =kwkW =1

sup kwkW =1

sup

|ϕ(Φ(w))|

kϕkV ∗ =kwkW =1

kΦ(w)kV = kΦkV ←W .

If W = W ∗∗ , equality follows.

t u

Corollary 4.92. As all reasonable crossnorms k·k are stronger than k·k∨(V,W ∗ ) , we have V ⊗k·k W ∗ ⊂ F(W, V ) ⊂ K(W, V ). This holds in particular for k·k∧(V,W ∗ ) . The definition of nuclear operators can be found in Grothendieck [129]. Definition 4.93. N (W, V ) := V ⊗∧ W ∗ is the space of nuclear operators. If V and W are assumed to be Hilbert spaces, the infinite singular-value decomposition enables additional conclusions which will be given in §4.4.3.

4.3 Tensor Spaces of Order d

137

4.3 Tensor Spaces of Order d Most of the definitions and properties stated for d = 2 carry over to d > 2 in an obvious way. However, a new problem arises. In the case of d = 2, all three spaces V, N W, V ⊗ W are equipped with norms. For d > 2 spaces as for instance ∗ V[j] = k6=j Vk are of interest. Since we want to embed the dual space V[j] into L(V, Vj ) (cf. §4.3.4) we need a suitable norm. Such norms will be defined in §4.3.2.

4.3.1 Continuity, Crossnorms 4.3.1.1 General Definitions and Properties In this chapter, k·kj are the norms associated with the vector spaces Vj , while k·k Nd is the norm of the algebraic tensor space a j=1 Vj and of the Banach tensor space Nd k·k j=1 Vj . Lemma 4.35b implies the following result. Remark 4.94. Let (Vj , k·kj ) be normed vector spaces for 1 ≤ j ≤ d. The d-fold tensor product d O : V1 × . . . × Vd → V1 ⊗a . . . ⊗a Vd j=1

is continuous if and only if there is some constant C such that

d

d Y

O (j)

≤C v for all v (j) ∈ Vj (1 ≤ j ≤ d) . kv (j) kj

j=1

We call k·k a crossnorm if

d

d

Y

O (j)

= v kv (j) kj

j=1

(4.41)

j=1

for all v (j) ∈ Vj

(1 ≤ j ≤ d)

(4.42)

j=1

holds for elementary tensors. Similarly, we may consider the d-fold tensor product d O j=1

: V1∗ × . . . × Vd∗ →

d O a

Vj∗

j=1

Nd of the dual spaces. We recall that the normed space ( a j=1 Vj , k·k) has a dual Nd ∗ ( a j=1 Vj )∗ equipped with the dual norm k·k . We interpret ϕ1 ⊗ . . . ⊗ ϕd ∈ N Nd Nd d ∗ ∗ a j=1 Vj as functional on a j=1 Vj , i.e., as an element of ( a j=1 Vj ) , via   (ϕ1 ⊗ . . . ⊗ ϕd ) v (1) ⊗ . . . ⊗ v (d) := ϕ1 (v (1) ) · ϕ2 (v (2) ) · . . . · ϕd (v (d) ). Nd Nd ∗ ∗ ∗ Then continuity of j=1 : V1 × . . . × Vd → a j=1 Vj is equivalent to

138

4 Banach Tensor Spaces

d d Y

O

∗

kϕj k∗j C ≤ ϕ j

for all ϕj ∈ Vj∗ (1 ≤ j ≤ d) .

(4.43)

j=1

j=1

Nd A crossnorm k·k on j=1 Vj is called a reasonable crossnorm if

O d

d

∗ Y

=

for all ϕj ∈ Vj∗ (1 ≤ j ≤ d) . ϕ kϕj kj∗ j

j=1

(4.44)

j=1

Nd A crossnorm k·k on V := k·k j=1 Vj is called a uniform crossnorm, if elemenNd tary tensors A := j=1 A(j) have the operator norm kAkV←V =

d Y

kA(j) kVj ←Vj



 A(j) ∈ L(Vj ,Vj ), 1 ≤ j ≤ d

(4.45)

j=1

(we may write ≤ instead, but equality follows, compare Definition 4.84 and the subsequent comment on page 133). The proofs of Lemma 4.87 and Proposition 4.88 can easily be extended to d factors yielding the following result. Nd Lemma 4.95. (a) A uniform crossnorm on j=1 Vj is a reasonable crossnorm. Nd (b) Let k·k be a uniform crossnorm on j=1 Vj with reflexive Banach spaces Vj . Nd ∗ ∗ Then k·k is a uniform and reasonable crossnorm on j=1 Vj . 4.3.1.2 Projective Norm k·k∧ Nd As for d = 2 (cf. §4.2.3.1), the projective norm kxk∧(V1 ,...Vd ) on a j=1 Vj is induced by the norms k·kj = k·kVj for 1 ≤ j ≤ d, which can be defined as follows. Nd Remark 4.96. (a) For x ∈ a j=1 Vj define k·k∧(V1 ,...Vd ) by X  d n Y n O d X (j) (j) kvi kj : x = kxk∧(V1 ,...Vd ) := kxk∧ := inf vi . i=1 j=1

(b) k·k∧ satisfies the crossnorm property (4.42). (c) k·k∧ is the strongest norm for which the map is continuous.

i=1 j=1 d N j=1

: V1 × . . . × Vd →

a

d N

Vj

j=1

The parts (b) and (c) are proved analogously to the case of d = 2 in Lemma 4.51 and Proposition 4.52. Nd Proposition 4.97. k·k∧ is a uniform and reasonable crossnorm on a j=1 Vj . Proof. The proof of the uniform crossnorm property in Proposition 4.85 can easily be extended from d = 2 to d ≥ 3. The result of Part (i) together with Lemma 4.87 implies that k·k∧ is a reasonable crossnorm. t u

4.3 Tensor Spaces of Order d

139

4.3.1.3 Injective Norm k·k∨ The analogue of the definition of k·k∨ in §4.2.4.2 for d factors yields the following formulation. Let ϕ1 ⊗ ϕ2 ⊗ . . . ⊗ ϕd be an elementary tensor of the tensor space Nd ∗ a j=1 Vj generated by the dual spaces. The proof of the next remark uses the same arguments as used in Lemma 4.59 and Proposition 4.60. Remark 4.98. (a) For v ∈

a

Nd

j=1

kvk∨(V1 ,...,Vd ) := kvk∨ :=

Vj define k·k∨(V1 ,...Vd ) by17

sup 06=ϕj ∈Vj∗ 1≤j≤d

|(ϕ1 ⊗ ϕ2 ⊗ . . . ⊗ ϕd ) (v)| . Qd ∗ j=1 kϕj kj

(4.46)

(b) For elementary tensors the crossnorm property (4.42) holds:

O

d (j)

v

j=1

∨(V1 ,...,Vd )

=

d Y

kv (j) kj

for all v (j) ∈ Vj (1 ≤ j ≤ d) .

j=1

(c) k·k∨(V1 ,...Vd ) is the weakest norm with

N

d

d N

j=1

j=1

: × Vj∗ → a

Vj∗ being continuous.

The definition of the injective norm can be simplified for symmetric tensors (cf. Remark 4.175). Even in the finite-dimensional case with d ≥ 3, the computation of kvk∨ is NP-hard (cf. Hillar–Lim [162]).

4.3.1.4 Examples Example 4.53 can be generalised to tensors of order d. Example 4.99. Let Vj := `1 (Ij ) with finite or countable index set Ij for 1 ≤ j ≤ d. Nd The projective norm k·k∧(V1 ,...,Vd ) of a j=1 Vj coincides with k·k`1 (I1×I2×...×Id ) . Remark 4.69 leads to the following generalisation. Example 4.100. Let Vj = (C(Ij ), k·kC(Ij ) ) with certain domains Ij (e.g., intervals). Then k·k∨(C(I1 ),...,C(Id )) = k·kC(I1 ×I2 ×...×Id ) . Proof. Remark 4.69 shows that k·k∨(C(I1 ),C(I2 )) = k·kC(I1 ×I2 ) . Iteration yields k·k∨(C(I1 ),C(I2 ),C(I3 )) = k·k∨(C(I1 ×I2 ),C(I3 )) = k·kC(I1 ×I2 ×I3 ) . The proof is completed by induction. t u 17

In Zhang–Golub [309, Def. 2.1], the expression (ϕ1 ⊗ ϕ2 ⊗ . . . ⊗ ϕd ) (v)/ is introduced as the generalised Rayleigh quotient.

Qd

j=1

kϕj k∗ j

140

4 Banach Tensor Spaces

Since kckSVD,2 = k·k`2 (I1 ×I2 ) is not equivalent to k·kSVD,1 , the explicit interpretation of k·k∧(V1 ,...,Vd ) for (Vj , k·k`2 (Ij ) ) and d ≥ 3 is not obvious. A later counterexample will use the following unusual uniform crossnorm. Exercise 4.101. For α ⊂ {1, . . . , d} with 2 ≤ #α ≤ d − 2 we regard V = #α 2 d 2 ` (N) and Vαc = a ⊗d−#α `2 (N) a ⊗ ` (N) also as Vα ⊗Vαc with Vα = a ⊗ (cf. Example 4.7). On Vα and Vαc we use the injective and projective norms. Then the norm on V is defined as n nX o o X X kvk := inf min kxν k∧ kyν k∨ : v = kxν k∨ kyν k∧ , xν ⊗ yν , ν

ν

ν

where xν ∈ Vα , yν ∈ Vαc . Prove that k · k is a uniform crossnorm. 4.3.1.5 Dual Spaces Lemma 4.82 together with its proof can easily be generalised to order-d tensors. Nd Lemma 4.102. Assume that V = k·k j=1 Vj is equipped with a reasonable Nd crossnorm k·k and is reflexive. Then the identity k·k∗ j=1 Vj∗ = V∗ holds. Furthermore, the spaces Vj are reflexive. Exercise 4.103. Under the assumptions of Lemma 4.102 show that weak converNd Nd (j) (j) (j) . Hint: gence vn * v (j) ∈ Vj (1 ≤ j ≤ d) implies j=1 vn * j=1 v Exercise 4.28.

4.3.2 Recursive Definition of the Topological Tensor Space 4.3.2.1 Setting of the Problem As mentioned in §3.2.4, the algebraic tensor space Valg := constructed recursively by pairwise products: X2alg := V1 ⊗a V2 ,

a

Nd

j=1

Vj can be

alg alg alg Xalg 3 := X2 ⊗a V3 , . . . , Valg := Xd := Xd−1 ⊗a Vd .

For a similar construction of the topological tensor space V := need, in addition, suitable norms k·kXk on Xk so that Xk := Xk−1 ⊗k·kX Vk k

k·k

Nd

for k = 2, . . . , d with X1 := V1

j=1

Vj we (4.47)

yielding V = Xd with k·k = k·kXd for k = d. In the case of a (reasonable) crossnorm, it is natural to require that k·kXk be also a (reasonable) crossnorm. The crossnorm property is not only a property of k·k but also describes its relation to the norms of the generating normed spaces. For d ≥ 3, different interpretations are possible as explained below.

4.3 Tensor Spaces of Order d

141

Remark 4.104. There are two interpretations of the crossnorm property of k·kXk : Nk (i) A crossnorm on Xk = k·kX j=1 Vj requires k

O Yk

k

(4.48a) v (j) kv (j) kj (v (j) ∈ Vj ),

= j=1

j=1

Xk

(ii) whereas a crossnorm on Xk = Xk−1 ⊗k·kX Vk requires the stronger condition k

kx ⊗ v

(k)

kXk = kxkXk−1 kv

(k)

kk

for x ∈ Xk−1 and v (k) ∈ Vk .

(4.48b)

The difference between (i) and (ii) corresponds to the different meaning of an ‘elementary tensor’ as illustrated in Example 3.30. Remark 4.105. (a) Let α ⊂ D := {1, . . . , d} be arbitrary. We may renumber the spaces Vj such that α = {1, . . . , #α}. Then the above mentioned construction of N k·k the norm k · kα of Vα := X#α yields Banach tensor spaces Vα = a j∈α Vj α for all subsets α. (b) A crossnorm property even stronger than (4.48b) is  (α) v ∈ Vα , v(β) ∈ Vβ , (α) (β) (α) (β) kv ⊗ v kγ = kv kα kv kβ for all ˙ ⊂ D (disjoint union). γ = α∪β (4.48c) The implications (4.48c) ⇒ (4.48b) ⇒ (4.48a) are valid. (c) There are uniform crossnorms not satisfying (4.48c). It may even happen that v(α) ⊗ v(β) (v(α) ∈ Vα , v(β) ∈ Vβ ) does not belong to Vγ . c Proof. The proof of (c) uses the norm in Exercise 4.101 N with(j)β = α(j). The definition of the norm k · kα by kxkα = kx ⊗ yk with y = j∈α y , ky k2 = 1 leads to kxkα = kxk∨ since kyk∨ = kyk∧ = 1 and kxk∨ ≤ kxk∧ . Analogously, we obtain k · kαc = k · k∨ . The completed Banach spaces are the injective spaces Vα and Vαc . Choose tensors x ∈ Vα and y ∈ Vαc not belonging to the projective spaces. Then kx ⊗ yk = ∞ holds for the uniform crossnorm of Exercise 4.101. t u

4.3.2.2 Construction for General Uniform Crossnorms If V is a Banach tensor space with a uniform crossnorm k·k, it turns out that the required norm k · kα of Vα (α ⊂ D := {1, . . . , d}) is uniquely determined. Lemma 4.106. If k·kXk is a crossnorm in the sense of (4.48b), k·kXk is uniquely defined by the norm k·k of V via

Od for arbitrary v (j) ∈ Vj

kxkXk = x ⊗ v (j) (4.49) j=k+1 with kv (j) kj = 1. N Similarly, kv(α) kα = kv(α) ⊗ j∈αc v (j) k holds for all v(α) ∈ Vα , v (j) ∈ Vj , kv (j) kj = 1, α ⊂ D. The uniformity of the crossnorm k·k can be replaced by the stronger crossnorm property (4.48c).

142

4 Banach Tensor Spaces

Proof. The induction starts with k = d−1. The crossnorm property (4.48b) implies kx ⊗ v (d) k = kxkXd−1 kv (d) kd = kxkXd−1 for any normalised vector v (d) ∈ Vd . The cases k = d − 2, . . . , 2 follow by recursion. Case (4.48c) is left as exercise. t u Note that Lemma 4.106 requires that a crossnorm k·kXk exist in the sense of (4.48b). Requiring that k·k be a uniform crossnorm on V, we now show the opposite direction: The intermediate norms k·kXk in (4.49) are well defined and have the desired properties. Note that X1 = V1 and Xd = V; in particular, the norms k·kXd and k·k coincide. Nd Theorem 4.107. Assume that k·k is a uniform crossnorm on V = k·k j=1 Vj . Then the definition (4.49) does not depend on the choice of v (j) ∈ VjN , and the k resulting norm k·kXk is a uniform and reasonable crossnorm on k·kX j=1 Vj . k Furthermore, k·kXk is a reasonable crossnorm on Xk−1 ⊗k·kX Vk . k

The proof is in §4.3.3. Remark 4.105a yields the following corollary. Corollary 4.108. LetNα ⊂ {1, . . . , d}. As in Theorem 4.107 uniform crossnorms k·kα of Vα = k·kα j∈α Vj can be defined. It is also a reasonable crossnorm on
N ˙ Vα = Vα1 ⊗ k∈α2 Vk for all disjoint subsets α1 , α2 6= ∅ with α = α1 ∪ α2 , i.e.,

!

O Y

(α1 ) (k) kv (k) kk . ⊗ v

x

= kx(α1 ) kα1

k∈α2

Lemma 4.109. Let Xd−1 = k·kX

Nd−1 d−1

(d)

defined in Theorem 4.107. All ϕ




I ⊗ . . . ⊗ I ⊗ ϕ(d) (v) X

k∈α2

α

d−1

∈

j=1 Vj Vd∗ and

be equipped with the norm k·kXd−1 ψ ∈ X∗d−1 satisfy

≤ kϕ(d) kd∗ kvk

∗ kvk k(ψ ⊗ I) (v)kd ≤ kψkX d−1

for all v ∈ V = Xd . (4.50)


∗ Proof. Any ψ ∈ Xd−1 satisfies ψ ⊗ ϕ(d) = ψ ◦ I ⊗ . . . ⊗ I ⊗ ϕ(d) (I = id).
∗ with kψk∗X =1 For v[d] := I ⊗ . . . ⊗ I ⊗ ϕ(d) (v) there is a ψ ∈ Xd−1 d−1
and ψ v[d] = kv[d] kXd−1 (cf. (4.10)). Hence,

(I ⊗ . . . ⊗ I ⊗ ϕ(d) )(v) X

d−1


= ψ (I ⊗ . . . ⊗ I ⊗ ϕ(d) )(v)

= |(ψ ⊗ ϕ(d) )(v)| ≤ kψ ⊗ ϕ(d) k∗ kvk

= (4.33d)

kψk∗Xd−1 kϕ(d) kd∗ kvk

= kϕ(d) kd∗ kvk u proves the first inequality in (4.50). The second one can be proved analogously. t

4.3 Tensor Spaces of Order d

143

4.3.2.3 Injective and Projective Norms The norm k · kα of Vα defines the operator norm k · kα←α of L(Vα , Vα ). Definition 4.110 (strong uniformity). A crossnorm k·k of V is called strongly uniform if the induced norms k·kα (α ⊂ D = {1, . . . , d}) satisfy the following conS` dition. For all disjoint decompositions α = ν=1 αν and all φ(αν ) ∈ L(Vαν , Vαν ) the norm k · kα←α satisfies the equality:

`

`

O

Y

(αν )

(αν ) = φ (4.51) .

φ



αν ←αν ν=1

α←α

ν=1

The injective and projective norms can easily be extended to the intermediate spaces Vα . They even satisfy the strongly uniform crossnorm property of Definition 4.110. The proof of the theorem is postponed to §4.3.3. Nd Nd Theorem 4.111. Let V := ∧ j=1 Vj [or, respectively, V := ∨ j=1 Vj ] be the standard projective [injective] tensor space. The norm k · kα of Vα defined in Corollary 4.108 coincides with the projective [injective] norm k · k∧(Vj :j∈α) [k · k∨(Vj :j∈α) ]. Both norms are strongly uniform crossnorms. In particular, the ∗ stronger crossnorm property (4.48c) holds for Vα , Vβ as well as for Vα , Vβ∗ . ˙ For a disjoint union γ = α∪β, Vγ = Vα ⊗∧ Vβ [Vγ = Vα ⊗∨ Vβ ] holds. An immediate consequence is Xk−1 = Vα with α = {1, . . . , k − 1} for the spaces defined in Theorem 4.107. Another strongly uniform crossnorm is the induced Hilbert norm which will be introduced in §4.5.1.

4.3.3 Proofs Proof of Theorem 4.107. (i) It is sufficient to consider the case k = d − 1 so that definition (4.49) becomes kxkXd−1 := kx ⊗ v (d) k

with kv (d) kd = 1.

There is some ϕ(d) ∈ Vd∗ with kϕ(d) k∗d = 1 and ϕ(d) (v (d) ) = 1. Let w(d) ∈ Vd with kw(d) kd = 1 be another choice. Set A(d) := w(d) ϕ(d) ∈ L(Vd , Vd ), i.e., A(d) v = ϕ(d) (v)w(d) . Because kA(d) kVd ←Vd = kϕ(d) kd∗ kw(d) kd = 1, the uniform Nd crossnorm property (4.45) with A := j=1 A(j) , where A(j) := I for 1 ≤ j ≤ d−1, implies that kx ⊗ w(d) k = kA(x ⊗ v (d) )k ≤ kx ⊗ v (d) k. Interchanging the roles of w(d) and v (d) , we obtain kx ⊗ v (d) k = kx ⊗ w(d) k. Obviously, k·kXd−1 = k · ⊗v (d) k is a norm on Xd−1 .

144

4 Banach Tensor Spaces

Nd−1 Nd (ii) For x := j=1 v (j) we form x ⊗ v (d) = j=1 v (j) with some kv (d) kd = 1 . The crossnorm property of k·k implies the crossnorm property of k·kXd−1 on Nd−1 (d) kd = 1): k·kX j=1 Vj (use kv d−1

d−1

O (j)

v

j=1

Xd−1

Y

d d d−1 Y

O (d) (j)

kv (j) kj = = x⊗v = v = kv (j) kVj ←Vj . j=1

j=1

j=1

Similarly, the uniform crossnorm property can be shown:

d−1 !






O (j) A x = A(1) ⊗ . . . ⊗ A(d−1) x ⊗ v (d)

j=1 Xd−1





(1) = A ⊗ . . . ⊗ A(d−1) ⊗ I x ⊗ v (d) ! d−1 d−1 Y Y ≤ kA(j) kVj ←Vj kIkVd ←Vd kx ⊗ v (d) k = kA(j) kVj ←Vj kxkXd−1 . | {z } j=1 j=1 =1

As a consequence, by Lemma 4.87, k·kXd−1 is also a reasonable crossnorm. (iii) Now we consider V as the tensor space Xd−1 ⊗a Vd (interpretation (ii) of Remark 4.104). Let v := x ⊗ w(d) with x ∈ Xd−1 and 0 6= w(d) ∈ Vd . Set v (d) := w(d) /kw(d) kd . Then kvk = kx ⊗ w(d) k = kw(d) kd kx ⊗ v (d) kV = kxkXd−1 kw(d) kd follows by the definition (4.49) of kxkXd−1 . This proves that k·k is a crossnorm on Xd−1 ⊗a Vd . Since k·k is not necessarily uniform on Xd−1 ⊗a Vd , we need another argument to prove that k·k is a reasonable crossnorm on Xd−1 ⊗ Vd . Let ψ ∈ X∗d−1 and ϕ(d) ∈ Vd∗ . We need to prove that kψ ⊗ ϕ(d) k∗ = kψk∗Xd−1 kϕ(d) k∗d . Using the crossnorm property for an elementary tensor v = x ⊗ v (d) , we get |(ψ⊗ϕ(d) )(v)| |ψ(x)| |ϕ(d)(v (d) )| = ≤ kψk∗Xd−1 kϕ(d) k∗d kvk kxkXd−1 kv (d) kd

if v = x⊗v (d) 6= 0. (4.52)

Taking the supremum over all v = x ⊗ v (d) 6= 0 (x ∈ Xd−1 ), we obtain |(ψ ⊗ ϕ(d) )(v)| |(ψ ⊗ ϕ(d) )(v)| ≥ sup = kψk∗Xd−1kϕ(d) k∗d . kvk kvk (d) v∈Xd v=x⊗v

kψ ⊗ ϕ(d) k∗ = sup

Nd Define the operator A := j=1 A(j) ∈ L(V, V) by A(j) = I (1 ≤ j ≤ d − 1) and A(d) = vˆ(d) ϕ(d) with 0 6= vˆ(d) ∈ Vd . Then Av is an elementary vector of the form x ⊗ vˆ(d) (x ∈ Xd−1 ), and kA(d) kVd ←Vd = kˆ v (d) kd kϕ(d) k∗d holds. (d) (d) ∗ This fact and the crossnorm property kAvk ≤ kˆ v kd kϕ kd kvk lead us to

4.3 Tensor Spaces of Order d

145

kψk∗Xd−1 kϕ(d) k∗d ≥

(4.52)

|(ψ ⊗ ϕ(d) )(Av)| |(ψ ⊗ ϕ(d) )(Av)| ≥ (d) . kAvk kˆ v kd kϕ(d) k∗d kvk

Since (ψ ⊗ ϕ(d) )(Av) = (ψ ⊗ (ϕ(d) A(d) ))(v) = ϕ(d) (ˆ v (d) ) · (ψ ⊗ ϕ(d) )(v), the estimate can be continued by |ϕ(d) (ˆ v (d) )| |(ψ ⊗ ϕ(d) )(v)| for all 0 6= vˆ(d) ∈ Vd . kvk kˆ v (d) kd kϕ(d) k∗d Since supvˆ(d) 6=0 ϕ(d) (ˆ v (d) ) /kˆ v (d) kd = kϕ(d) k∗d it follows that
ψ ⊗ ϕ(d) (v) ≤ kψk∗Xd−1 kϕ(d) k∗d for all v ∈ V, kvk kψk∗Xd−1 kϕ(d) k∗d ≥

so that kψ ⊗ ϕ(d) k∗ ≤ kψk∗Xd−1 kϕ(d) k∗d . Together with the opposite inequality from above, we have proved kψ ⊗ ϕ(d) k∗ = kψk∗Xd−1 kϕ(d) k∗d . t u N Proof of Theorem 4.111 (case of projective norm). (i) Define Vα = ∧ j∈α Vj with the projective norm k·kα := k·k∧(Vj :j∈α) . Endow W := Vα ⊗∧ Vβ for disjoint α, β ⊂ D with the projective norm k·kW := k·k∧(Vα ,Vβ ) , while Vγ for γ := α ∪ β and its norm are defined as above. We want to prove W = Vγ and k·kW = k·kγ . P Nd (j) Choose any v ∈ Vγ and ε > 0. There is a representation v = ν j=1 vν P Qd N N (j) (α) (j) (β) (j) with kvk ≥ ν j=1 kvν kj −ε. Set xν := j∈α vν and xν := j∈β vν . P (α) P (β) (α) (β) Since v = ⊗ xν , the inequality kvkW ≤ ν xν ν kxν kα kxν kβ = P Qd (j) ν j=1 kvν kj ≤ kvk + ε holds for all ε > 0. Hence, kvkW ≤ kvkγ . P (α) (β) (ii) For the reverse inequality choose a representation v = ⊗ xν ν xν P (α) (β) with kvkW ≥ ν kxν kα kxν kβ − ε. Without loss of generality, the factors P 2 (α) (β) are equally scaled: ων := kxν kα = kxν kβ , i.e., kvkW ≥ ν ων − ε. P N P N (α) (j) (β) (j) Choose representations xν = µ j∈α xνµ , and xν = χ j∈β xνχ with P Q P Q (α) (j) (β) (j) kxν kα ≥ µ j∈α kxνµ kj − 2−ν ε and kxν kβ ≥ χ j∈β kxνχ kj − 2−ν ε.
P N N (j) (j) Then the representation v = ν,µ,χ j∈α xνµ ⊗ j∈β xνχ yields kvkγ ≤

X Y

kx(j) νµ kj

ν,µ,χ j∈α

≤

X

kx(j) νχ kj =

XhX Y ν

j∈β −ν

ε


2

X

= ν q ≤ kvkW + O(ε + ε kvkW ) ν

ων + 2

Y

ων2

+2

kx(j) νµ kj

ih X Y

µ j∈α 1−ν

εων + 2

kx(j) νχ kj

i

χ j∈β −2ν 2

ε




for all ε > 0, proving kvkγ ≤ kvkW . In the last line we use the Schwarz inequality qP pP P 1−ν 2(1−ν) 2 2 ω ≤ ν ν ν2 ν ων .

146

4 Banach Tensor Spaces

(iii) The special choice α = {1, . . . , k − 1} and β = {k} shows that (4.47) N` holds with X` = ∧ j=1 Vj (` = k − 1, k). (iv) Considering Vγ for γ = α ∪ β as the projective

tensor space Vα ⊗

∧ Vβ of order 2, Proposition 4.85 proves φ(α) ⊗ φ(β) γ←γ = φ(α) α←α φ(β) β←β . t u Recursive application of this result yields (4.51). N Proof of Theorem 4.111 (case of injective norm). (i) Define Vα = ∨ j∈α Vj with the injective norm k·kα := k·k∧(Vj :j∈α) . Endow W := Vα ⊗∨ Vβ for disjoint α, β ⊂ D with the injective norm k·kW := k·k∨(Vα ,Vβ ) , while Vγ for γ := α∪β and its norm are defined as above. We want to prove W = Vγ and k·kW = k·kγ . N ∗ (similar for β) we Since elementary tensors of j∈α Vj∗ from a subset of Vα have kvkW = supΦ∈Vα∗ ,Ψ ∈Vβ∗

|(Φ⊗Ψ )v| ∗ kΦk∗ α kΨ kβ

≥ kvkγ .

P (α) (β) (ii) For the reverse inequality consider an arbitrary tensor v = i xi ⊗ xi ∈ (β) ∗ ∗ Vα ⊗a Vβ . For any Φ ∈ Vα and Ψ ∈ Vβ∗ set λi := Ψ (xi )/ kΨ kβ ∈ K. Then P (α) (β) (Φ ⊗ Ψ ) i xi ⊗ xi

|(Φ ⊗ Ψ ) v| ∗ ∗ = kΦkα kΨ kβ

P (α) (β) i Φ(xi )Ψ (xi )

= ∗ ∗ ∗ ∗ kΦkα kΨ kβ kΦkα kΨ kβ P P  (α) (α)

i Φ(λi xi ) Φ i λi xi

X (α) = = ≤ λi xi . ∗ ∗ i α kΦkα kΦkα

According to the definition of k·kα , for any ε > 0 there are ϕ(j) ∈ Vj∗ with P N P (α) (α) kϕ(j) k∗j = 1 such that k i λi xi kα ≤ |( j∈α ϕ(j) ) i λi xi | + ε. Inserting N (β) (α) ∗ λi = Ψ (xi )/ kΨ kβ and setting µi = ( j∈α ϕ(j) )xi , we arrive at

X |(Φ ⊗ Ψ ) v|

X (α) (β) ∗ λi xi ≤ µi Ψ (xi )/ kΨ kβ + ε ∗ ∗ ≤ i i α kΦkα kΨ kβ X

X



(β) (β) ∗ ≤ Ψ µi xi / kΨ kβ + ε ≤ µi xi + ε. i

i

β

∗ Again there are ϕ(j) ∈ Vj∗ with ϕ(j) j = 1 for j ∈ β so that

X

O X

(β) (β) µi xi + ε ≤ ϕ(j) µi xi + 2ε.

i

j∈β

β

N



i

|(Φ⊗Ψ )v| (j) Using µi = xi , we conclude that kΦk ∗ ∗ is bounded by j∈α ϕ α kΨ kβ O O X    (α) (β) ϕ(j) xi ⊗ xi + 2ε = ϕ(j) v + 2ε ≤ kvkγ + 2ε. j∈γ

i

(α)

j∈γ

Since this inequality holds for all ε > 0, we obtain kvkW = supΦ,Ψ

|(Φ⊗Ψ )v| ∗ kΦk∗ α kΨ kβ

≤

kvkγ . Together with part (i), kvkW = kvkγ follows. The further conclusions are as in the proof for the projective case. t u

4.3 Tensor Spaces of Order d

147

4.3.4 Embedding into L(V, Vj ) and L(V, Vα ) N N Let ϕ ∈ k6=j Vk∗ denote a functional on V[j] := a k6=j Vk . Usually we identify ϕ with the induced map VN→ Vj (cf. Remark 3.64b). To be quite precise, we now use the symbol ϕ for k6=j ϕk and Φ = ϕ ⊗ idj for the map of V into Vj . If Φ is continuous, i.e., Φ ∈ L(V, Vj ), then the composition of some Nd ϕj ∈ Vj∗ and Φ is again continuous. Since the composition ϕj ◦ Φ = k=1 ϕk is a functional on V, continuity of ϕj ◦ Φ means that ϕj ◦ Φ ∈ V∗ and ∗

kϕj ◦ ΦkV∗ ≤ kϕj kj kΦkVj ←V .

(4.53)

Part (b) of the next lemma shows that also the reverse statement is true. Lemma 4.112. (a) ϕj ∈ Vj∗ and Φ = ϕ ⊗ idj ∈ L(V, Vj ) imply ϕj ◦ Φ ∈ V∗ . (b) Let Φ = ϕ ⊗ idj : V → Vj be a linear map. If ϕj ◦ Φ ∈ V∗ for all ϕj ∈ Vj∗ , then Φ ∈ L(V, Vj ). Proof. Part (a) is proved above. For Part (b) let v ∈ V be arbitrary. Define vj := Φv. The norm of vj is kvj kj = ϕj (vj ) = (ϕj ◦ Φ) (v) ≤ kϕj ◦ ΦkV∗ kvk for a ∗ suitable ϕj ∈ Vj∗ with kϕj kj = 1 (cf. (4.10)). This proves Φ ∈ L(V, Vj ) with t u the norm kΦkVj ←V ≤ kϕj ◦ ΦkV∗ .

4.3.4.1 Basic Requirement I Now we give a reformulation of the previous statements. We rearrange the order in
Nd ∗ ∗ ∗ ∗ ∗ require that the and ⊗ V V ⊗ . . . ⊗ ⊗ V to get ⊗ . . . V V ⊗ V j+1 1 j j−1 d k=1 k tensor product mapping
∗ ∗ × Vj+1 × . . . × Vd∗ → V∗ be continuous. (4.54a) ⊗ : Vj∗ × V1∗ × . . . × Vj−1 Nd Note that (4.54a) coincides with (4.43). Let V∨ := ∨ k=1 Vk be the Banach tensor space with the injective norm k·k∨ = k·k∨(V1 ,...,Vd ) . Lemma 4.113. (a) The following statements are equivalent: (i) property (4.54a), (ii) k·k∨ . k·k , and (iii) the inclusion V ⊂ V∨ of the Banach tensor spaces. N (b) If k·k∨ = k·k (and hence V = V∨ ), then ϕ = k6=j ϕk gives rise to Φ = ϕ ⊗ idj ∈ L(V, Vj ) and Y

∗

Φ = kϕk kk . Vj ←V k∈{1,...,d}\{j}

N (c) Under condition (4.54a), any ϕ ∈ a k6=j Vk∗ leads to Φ ∈ L(V, Vj ) with kΦkVj ←V ≤ C kΦkVj ←V∨ < ∞ and C from k·k∨ ≤ C k·k .

148

4 Banach Tensor Spaces

Proof. (a) Since k·k∨ is the weakest norm satisfying (4.54a) (cf. Remark 4.98c), we obtain ‘(i)⇒(ii)’. Remark 4.2 shows that ‘(ii)⇒(iii)’. V ⊂ V∨ implies the con∗ tinuous embedding V∨ ⊂ V∗ (cf. Lemma 4.25). (4.54a) holds for the injective ∗ ∗ ⊂ V∗ leads to (4.54a). norm, i.e., with V∨ on the right-hand side, while V∨ Hence, also ‘(iii)⇒(i)’ is proved. Nd the proofQ of Lemma 4.112b and use that kϕj ◦ Φk∗∨ = k k=1 ϕk k∗∨ = Qd(b) Repeat ∗ ∗ = k=1 kϕk kk k6=j kϕk kk . kϕj k∗ j =1 N (c) ϕ is a (finite) linear combination of elementary tensors k6=j ϕk from part (a). Part (b) shows that kΦkVj ←V∨ is bounded. Hence k·k∨ ≤ C k·k proves kΦkVj ←V ≤ kΦkVj ←V∨ kidkV∨ ←V ≤ C kΦkVj ←V∨ . u t ∗

A quantitative statement about kΦkVj ←V can be given if we define a norm k·k[j] N Nd ∗ on V[j] = ( k6=j Vk )∗ . Assuming a uniform crossnorm on V = k·k j=1 Vj , we can introduce the auxiliary norms k·kV[j] = k·k[j] as defined in Remark 4.105a for N ∗
∗ = ( k6=j Vk )∗ , k·k[j] . α = D\{j}. The continuous dual space is V[j] N ∗ Exercise 4.114. The algebraic space a k6=j Vk∗ is a subspace of V[j] from above. Hint: use Lemma 4.95a. N ∗ ∗ ∗ As a consequence, the Banach space k·k[j] k6=j Vk is a subspace of V[j] . The spaces even coincide if V[j] is reflexive (cf. Lemma 4.102). N ∗ ∗ Theorem 4.115. Choose the tensor N norms∗ as explained above. ϕ ∈ k·k[j] k6=j Vk or, more general, ϕ ∈ (k·k[j] k6=j Vk ) can be regarded as a continuous map Φ = ϕ ⊗ idj : V → Vj . Moreover, the norms of ϕ and Φ coincide: ∗

kϕk[j] = kΦkVj ←V . Proof. (a) The second inequality in (4.50) (with V[j] instead of Xd−1 ) implies ∗ kΦkVj ←V = sup06=v∈V kΦ(v)kj / kvk ≤ kϕk[j] .

(b) The definition kxk[j] = x ⊗ v (j) of the norm (4.49) involves some v (j) ∈ Vj with kv (j) kj = 1. Choose ψ (j) ∈ Vj∗ with the property kψ (j) kj∗ = ψ (j) (v (j) ) = 1. Then we have
ϕ ⊗ ψ (j) (x ⊗ v (j) ) |ϕ(x)| · ψ (j) (v (j) ) |ϕ(x)| |ϕ(x)|



= =

x ⊗ v (j) =

x ⊗ v (j)

x ⊗ v (j) kxk[j]
ϕ ⊗ ψ (j) (v) ≤ sup = kϕ ⊗ ψ (j) k∗ . kvk 06=v∈V ϕ ⊗ ψ (j) can be written as the composition ψ (j) ◦ (ϕ ⊗ idj ) = ψ (j) ◦ Φ so that kϕ ⊗ ψ (j) k∗ = kψ (j) ◦ ΦkK←V ≤ kψ (j) kK←Vj kΦkVj ←V = kψ (j) k∗j kΦkVj ←V = kΦkVj ←V . Together with the previous inequality, |ϕ(x)| kxk[j] ≤ kΦkVj ←V is proved. ∗ The supremum over all x leads us to the reverse inequality kϕk[j] ≤ kΦkVj ←V . t u

4.3 Tensor Spaces of Order d

149

4.3.4.2 Modified Requirement I So far we assumed that V is equipped with a norm k·k satisfying k·k∨ . k·k . However, there are norms of practical importance which are weaker than k·k∨ . Nd (k) Then the tensor of functionals ϕ(k) ∈ Vk∗ may be an unbounded map k6=j ϕ Nd of a k6=j Vk into Vj . Since, in this case, the injective norm k·k∨ is too strong, we define a weaker norm as follows. In (4.54a) the norms of Vj and V remain fixed, while we replace all (Vk , k·kk ) with k 6= j by larger Banach spaces (V0,k , k·k0,k ) with a weaker norm k·k0,k . k·kk , i.e., Vk ⊂ V0,k

be continuously embedded for k 6= j

(cf. Definition 4.17). §4.3.5 will show examples of practical importance involving weaker spaces V0,k . Then Theorem 4.126 yields (4.54b). Requirement (4.54a) becomes
∗ ∗ ∗ ∗ ⊗ : Vj∗ × V0,1 × . . . × V0,j−1 → V∗ × . . . × V0,d × V0,j+1

be continuous. (4.54b) Set k·k0,∨ = k·k∨(Vj ,V0,1 ,...,V0,j−1 ,V0,j+1 ,...,V0,d ) . In spite of k·k∨ 6. k·k , the estimate k·k0,∨ .N k·k might be true. The latter inequality is equivalent to (4.54b). Set V0,[j] := ∨ k6=j V0,k . Theorem 4.111 yields V0 = Vj ⊗∨ V0,[j] . The statement analogous to Lemma 4.113 is as follows. Lemma 4.116. (a) The following three requirements are equivalent: (4.54b), V0 ⊂ V,, and k·k0,∨ . k·k . N ∗ corresponds to the continuous (b) Under condition (4.54b), any ϕ ∈ a k6=j V0,k map Φ = ϕ ⊗ id : V → Vj , i.e., Φ ∈ L(V, Vj ).

4.3.4.3 Weak Convergence Theorem 4.117. (a) For vn , v ∈ V assume vn * v. Let O O Vk∗ ϕk ∈ a ϕ= k6=j

k6=j

and define un := ϕ(vn ) ∈ Vj

and

u := ϕ(v) ∈ Vj .

Under condition (4.54a), un converges weakly to u: un * u. N ∗ (b) The same result holds for ϕ ∈ a k6=j V0,k under condition (4.54b).

150

4 Banach Tensor Spaces

Proof. (a) un * u is equivalent to ϕj (un ) → ϕj (u) for all ϕj ∈ Vj∗ . Since ϕj ◦ Φ in ϕj (un ) = (ϕj ◦ Φ) vn belongs to V∗ according to (4.54a), the assumption vn * v shows that ϕj (un ) = (ϕj ◦ Φ) vn → (ϕj ◦ Φ) v = ϕj (u). u t

(b) Part (b) is completely analogous.

4.3.4.4 Basic Requirement II Now we replace Vj by Vα for some nonempty subset α ⊂ {1, . . . , d} with nonempty complement αc . Vα must be equipped with some norm k·kα . Requirements (4.54a) and (4.54b) become

×V × ×V

∗ × ⊗ : Vα

∗ k

→ V∗

be continuous,

(4.54c)

k∈αc

∗ ⊗ : Vα

∗ 0,k

→ V∗ be continuous for V0,k ⊃ Vk .

(4.54d)

k∈αc

Lemma 4.118. Assume that the norm k·k of V is not weaker than the injective ∗ as described in the norm k·k∨(V1 ,...,Vd ) . Then statement (4.54c) holds with Vα proof. N Proof. (i) Define Vα = ∨ k∈α Vk by means of the injective norm k·k∨(Vk :k∈α) , Nd ∗ instead of V∗ is stated in while V∨ = ∨ j=1 Vj . Then (4.54c) with V∨ Theorem 4.111. If the norm of V is a uniform crossnorm, one may alternatively ∗ ) as in §4.3.2.2. define the norm of Vα (and therefore of Vα ∗ (ii) k·k & k·k∨ implies V ⊂ V∨ and V∗ ⊃ V∨ . Hence, the result of part (i) proves (4.54c). t u

Again, sufficiently weak norms of V0,k ⊃ Vk may lead to (4.54d). condition (4.54c) [or (4.54d)] a functional ϕ ∈ a Theorem N4.119. Under ∗ ∗ [ϕ ∈ a k∈αc V0,k ). ] gives rise to Φ = ϕ ⊗ idα ∈ L(V, Vα

N

∗ k∈αc Vk

Corollary 4.120. Strongly uniform crossnorms satisfy the strengthened require∗ ∗ ∗ ∗ ∗ ∗ × Vα be continuous. × V0,α or Vα ments that ⊗ : Vα c → V c → V

4.3.4.5 Weak Convergence The proof of the following theorem is the same as for Theorem 4.117. N Theorem 4.121. (a) For vn , v ∈ V assume vn * v. Let ϕ = k6=j ϕk ∈ N ∗ ) ∈ u V := V V and define := ∈ ϕ(v u ϕ(v) and c α . Under α a n n k k∈α condition (4.54c), un * u holds. N ∗ under condition (4.54d). (b) The same result holds for ϕ ∈ a k∈αc V0,k

4.3 Tensor Spaces of Order d

151

4.3.5 Intersections of Banach Tensor Spaces If two Banach spaces (X, k·kX ) and (Y, k·kY ) have a nonempty intersection Z :=X ∩Y , the intersection norm k·kZ is given by kzkZ := max{kzkX , kzkY } or equivalent ones. Below, we shall make use of this construction. At the end of §4.2.6.2 we studied the example C 1 (I × J). This space can be obtained as the closure of C 1 (I) ⊗a C 1 (J) with respect to the norm k·kC 1 (I×J) , however, this norm is not a reasonable crossnorm, it satisfies (4.33a) but not (4.33b). Instead, the mixed norm k·k1,mix in (4.36) is a reasonable crossnorm, but the 1 (I × J) is a proper subspace of C 1 (I × J), since functions resulting space Cmix 1 f ∈ Cmix (I × J) possess continuous mixed derivatives fxy . There is another way to obtain C 1(I × J). First we consider the anisotropic spaces ∂ C (1,0) (I ×J) := {f : f, ∂x f ∈ C(I ×J)},

kf k(1,0) := max{kf k∞ , kfx k∞ },

∂ C (0,1) (I ×J) := {f : f, ∂y f ∈ C(I ×J)},

kf k(0,1) := max{kf k∞ , kfy k∞ },

with kf k∞ := sup(x,y)∈I×J |f (x, y)|. Then we obtain C 1 (I ×J) and its norm by C 1 (I × J) = C (1,0) (I × J) ∩ C (0,1) (I × J), k · kC 1 (I×J) = max{kf k(1,0) , kf k(0,1) }. The proof of Remark 4.69 can be extended to show that k·k(1,0) [k·k(0,1) ] is a reasonable crossnorm of C (1,0) (I × J) [C (0,1) (I × J)]. We give another important example. Here N ∈ N is a fixed degree. Example 4.122. For Ij ⊂ R (1 ≤ j ≤ d) and 1 ≤ p < ∞, the Sobolev space H N,p (Ij ) consists of all functions f from Lp (Ij ) with bounded norm 18 p 1/p X N Z n d kf kN,p;Ij := , (4.55a) dxn f dx I j n=0 whereas H N,p (I) for I = I1 × . . . × Id ⊂ Rd is endowed with the norm  X Z 1/p p kf kN,p := |∂ n f | dx (4.55b) 0≤|n|≤N

where n ∈

Nd0

I

is a multi-index of length |n| :=

Pd

j=1

nj , and ∂ n as in (4.5).

Again, the norm k·kN,p satisfies (4.33a) but not (4.33b), in particular, it is not a reasonable crossnorm. Instead, for each n ∈ Nd0 with |n| ≤ N we define the space H n,p (I) := {f ∈ Lp (I) : ∂ n f ∈ Lp (I)} 18

It suffices to have the terms for n = 0 and n = N in (4.55a). The derivatives are to be understood as weak derivatives (cf. [141, §6.2.1]).

152

4 Banach Tensor Spaces

with the reasonable crossnorm p

p

kf k0,p + k∂ n f k0,p

kf kn,p :=


1/p

.

T Then the Sobolev space H N,p (I) is equal to the intersection 0≤|n|≤N H n,p (I), and its norm (4.55b) is equivalent to max0≤|n|≤N k·kn,p . Note that H n,p for n = (1, 0) is considered in Example 4.48. If n ∈ N0d is a multiple of a unit vector, i.e., ni = 0 except for one i, the proof of Remark 4.81 can be used to show that k·kn,p is a reasonable crossnorm for 1 ≤ p < ∞. The Sobolev spaces H m,p (Ij ) for m = 0, 1, . . . , Nj are an example of a scale of Banach spaces. We fix integers Nj and denote the j-th scale by (Nj )

Vj := Vj

(Nj −1)

⊂ Vj

(0)

⊂ . . . ⊂ Vj

with dense embeddings.

(4.56)

The corresponding norms satisfy k·kj,n & k·kj,m for Nj ≥ n ≥ m ≥ 0 on Vj . (n)

(0)

(1 ≤ n ≤ Nj ) are dense in (Vj , k·kj,0 ).

Lemma 4.123. By (4.56), all Vj

Let numbers Nj ∈ N0 be given and define N as a subset of N0d satisfying n ∈ N ⇒ 0 ≤ n j ≤ Nj ,

(4.57a)

Nj := (0, . . . , 0, Nj , 0, . . . , 0) ∈ N . | {z } | {z } j−1

(4.57b)

d−j d

The standard choice for N is N ⊂ {0, . . . , N } with Nj = N for all 1 ≤ j ≤ d. Often  N := Nj ∈ Nd0 : 1 ≤ j ≤ d (4.57c) is sufficient. For each n ∈ N we define the tensor space V(n) :=

d O a

(nj )

Vj

.

(4.58a)

j=1

Then we choose a reasonable crossnorm k·kn on V(n) or an equivalent one. The intersection Banach tensor space is defined by \ V := V(n) with intersection norm kvk := max kvkn (4.58b) n∈N

n∈N

or an equivalent norm. (0)

(Nj )

Remark 4.124. Assume Vj % Vj and let V be defined by (4.58b).

(this excludes the finite-dimensional case)

(a) V = Vmix := V(N1 ,N2 ,...,Nd ) holds if and only if (N1 , . . . , Nd ) ∈ N (cf. (4.36)). (b) Otherwise Vmix $ V $ V(0) holds, and Vmix is dense in V. (c) In Case (a) a reasonable crossnorm k·kmix may exist, whereas in Case (b) condition (4.33b) required for a reasonable crossnorm cannot be satisfied. Nd (Nj ) ∗ Proof. For Part (c) note that ϕ ∈ ) are continuous functionals on j=1 (Vj Vmix but not necessarily on V $ Vmix endowed with a strictly weaker norm. t u

4.3 Tensor Spaces of Order d

153

Proposition 4.125. Under the conditions (4.57a,b), the Banach tensor space V in (4.58b) satisfies the inclusion ! d d d O O O (0) ∩V = a Vj , Vj Vj ⊂ Vmix := k·k(N ,...,N ) a 1

j=1

d

j=1

j=1

i.e., an algebraic tensor in V does not differ from an algebraic tensor in Vmix . Pr Nd Nd (0) (j) with Each v ∈ a j=1 Vj ∩ V has a representation v = i=1 j=1 vi (j)

vi

(Nj )

∈ Vj = Vj

.

Proof. By definition (4.58a), ! " d O \ (0) ∩V = Vj a j=1

n∈N

d O a

! (0) Vj

#

∩

V(n)

j=1

Nd (0)
holds. Since v ∈ a j=1 Vj ∩ V(n) is an algebraic tensor, it belongs to the
Nd (0) space a j=1 Vj ∩ V(n) = V(n) . Lemma 6.11 will show that " # d \ O \ (n ) j v∈ V(n) = a Vj . j=1

n∈N

Nd

T

n∈N

(nj )
n∈N Vj

By condition (4.57b), v ∈ a j=1 = a from the fact that one of the numbers nj is equal to Nj . (0)

Nd

j=1 Vj

can be obtained t u

(1)

Application to Vj = C 0 (Ij ) and Vj = C 1 (Ij ) yields that all functions from
Nd 1 the algebraic tensor space a j=1 C 0 (Ij ) ∩C 1 (I) are already in Vmix = Cmix (I) 1 (cf. (4.36)), which is a proper subspace of C (I).
∗ P The dual space V∗ is the sum (span) of the duals of V(n) : V∗ = V(n) . n∈N

Theorem 4.126. Let k·kn be reasonable crossnorms on V(n) for all n ∈ N . Then (4.54b) holds in the form   (0)∗ (0)∗ (0)∗ (0)∗ ⊗ : Vj∗ × V1 × . . . × Vj−1 × Vj+1 × . . . × Vd → V∗ is continuous (Nj )

(we recall that Vj = Vj (nj )∗

⊗ : Vj

). Moreover, for any n ∈ N we have that

  (n )∗ (n )∗ (n )∗ × V1 1 × . . . × Vj−1j−1 × Vj+1j+1 × . . . → V∗

is continuous.

Proof. The first statement is the special case of the second one with n = Nj ∈ N (cf. (4.57b)). Since k·kn is a reasonable crossnorm, the tensor product ⊗ is contin(n )∗ d uous from ×j=1 Vj j into V(n)∗ . From the continuous embedding V(n)∗ ⊂ V∗ we obtain the desired result. t u

154

4 Banach Tensor Spaces

4.3.6 Tensor Space of Operators Nd Nd Let V = a j=1 Vj and W = a j=1 Wj be two Banach tensor spaces with the respective norms k·kV and k·kW , while k·kVj and k·kWj are the norms of Vj and Wj . The space L(Vj , Wj ) is endowed with the operator norm k·kWj ←Vj . Their algebraic tensor space is d O L := a L(Vj , Wj ) . j=1 d d N N A(j) ∈ L on v = The obvious action of an elementary tensor A = v (j) ∈ V j=1 j=1 yields the following tensor from W: ! ! d d d O O O A(j) Av = v (j) = A(j) v (j) ∈ W. j=1

j=1

j=1

If k·kV and k·kW are crossnorms, we estimate kAvkW by d Y j=1

kA(j) v (j) kWj

d h i Y kA(j) kWj ←Vj kv (j) kVj = ≤ j=1

d Y

! kA(j) kWj ←Vj

kvkV .

j=1

Hence kAvkW ≤ kAkkvkV holds for all elementary tensors. However, we cannot expect that all crossnorms k·kV and k·kW satisfy the estimate kAvkW ≤ kAkkvkV for general tensors v ∈ V. In the special case of V = W, we called crossnorms uniform if they satisfy this estimate (cf. §4.2.8). We show that the pro- and injective norms have the desired property. Proposition 4.127. (a) If k·kV = k·k∧(V1 ,...,Vd ) and k·kW = k·k∧(W1 ,...,Wd ) , then Nd A = j=1 A(j) ∈ L has the operator norm kAkW←V =

d Y

kA(j) kWj ←Vj .

(4.59)

j=1

(b) If k·kV = k·k∨(V1 ,...,Vd ) and k·kW = k·k∨(W1 ,...,Wd ) , (4.59) again holds with respect to the corresponding operator norm. Proof. The same arguments as in the proof of Proposition 4.85 can be applied.

t u

Nd Since kAkW←V is finite for elementary tensors A = j=1 A(j) , boundedness Nd holds for all A ∈ L := a j=1 L(Vj , Wj ) . The completion of (L, k·kW←V ) yields the tensor space d O L(Vj , Wj ) ⊂ L(V, W). k·kW←V j=1

Nd Exercise 4.128. Let k·k be a uniform crossnorm on V := j=1 Vj . Prove that Nd the operator norm k·kV←V is a crossnorm for L := j=1 L(Vj , Vj ).

4.4 Hilbert Spaces

155

4.4 Hilbert Spaces 4.4.1 Scalar Product Again, we restrict the field to either K = R or K = C. A normed vector space (V, k·k) is a pre-Hilbert space if the norm is defined by p kvk = hv, vi < ∞ for all v ∈ V, (4.60) where h·, ·i : V ×V → K is a scalar product on V . In the case of K = R, a scalar product is a bilinear form, which, in addition, must be symmetric and positive: hv,wi = hw, vi

for v, w ∈ V,

(4.61a)

hv,vi > 0

for v 6= 0.

(4.61b)

In the complex case K = C, the form must be sesquilinear, i.e., bilinearity and (4.61a) is replaced with19 hv,wi = hw, vi hu + λv,wi = hu, wi + λ hv,wi ¯ hw, vi hw, u + λvi = hw, ui + λ

for v, w ∈ V, for all u, v, w ∈ V, λ ∈ C, for all u, v, w ∈ V, λ ∈ C.

The triangle inequality of the norm (4.60) follows from the Schwarz inequality |hv,wi| ≤ kvk kwk

for v, w ∈ V.

We describe a pre-Hilbert space by (V, h·, ·i) and note that this defines uniquely a normed space (V, k·k) via (4.60). If (V, h·, ·i) is complete, i.e., if (V, k·k) is a Banach space, we call (V, h·, ·i) a Hilbert space. Example 4.129. The Euclidean scalar product on KI is defined by X hv, wi = vi w i . i∈I

4.4.2 Basic Facts about Hilbert Spaces Vectors u, v ∈ V are orthogonal if hv, wi = 0. A subset S ⊂ V is an orthogonal system if all pairs of different v, w ∈ S are orthogonal. If an orthogonal system is a basis, it is called an orthogonal basis. If, in addition, kvk = kwk = 1 holds, we have orthonormal vectors, an orthonormal system, and an orthonormal basis, respectively. In the infinite-dimensional Hilbert case, the term ‘orthonormal basis’ 19

In physics, the opposite ordering is common: the scalar product is antilinear in the first and linear in the second argument.

156

4 Banach Tensor Spaces

has to be understood as ‘complete basis’, which is different from the algebraic basis: if any v ∈ V can uniquely be written b = {bν : ν ∈ B} is a complete basis of V P as unconditionally20 convergent series v = ν∈B αν bν (αν ∈ K). If V is separable, B P is (at most) countable; otherwise B is not countable, but for each v ∈ V the series ν∈B αν bν contains only countably many nonzero coefficients. The orthogonal complement of a subset S ⊂ V is S ⊥ = {v ∈ V : hv,wi = 0 for all w ∈ S} . Remark 4.130. (a) Any orthogonal complement is closed. (b) If S ⊂ V is a closed subspace, V = S ⊕ S ⊥ is a direct sum, i.e., every v ∈ V has a unique decomposition v = s + t with s ∈ S and t ∈ S ⊥ . An unpleasant feature of general Banach spaces is the possible non-reflexivity X ∗∗ % X. This does not happen for Hilbert spaces as stated next. Remark 4.131. (a) All Hilbert spaces satisfy V = V ∗∗ . (b) The dual space V ∗ is isomorphic to V : for any ϕ ∈ V ∗ there is exactly one vϕ ∈ V with for all v ∈ V ϕ(v) = hv,vϕ i (4.62) (Fr´echet–Riesz theorem, cf. Riesz [250, §II.30]). Conversely, every element vϕ ∈ V generates a functional ϕ ∈ V ∗ via (4.62). This defines the Fr´echet–Riesz isomorphism J : V → V ∗ with hv,wiV = hJv,JwiV ∗ . Notation 4.132. (a) For v ∈ V we shall denote Jv ∈ V ∗ by v ∗ , i.e., v ∗ (·) = h·, vi. For finite-dimensional vector spaces, v ∗ is equal to v H (cf. §2.1). (b) It is possible (but not necessary) to identify V with V ∗ by setting v = v ∗ . (c) Let v ∈ V and w ∈ W . Then wv ∗ ∈ L(V, W ) denotes the operator (wv ∗ ) (x) := v ∗ (x) · w ∈ W

for all x ∈ V.

(4.63)

Theorem 4.133. For every Hilbert space V there exists an orthonormal basis {φi : i ∈ S}. It satisfies X X 2 2 hv, φi i φi , kvk = for all v ∈ V. (4.64) |hv, φi i| v= i∈S

i∈S

The second identity in (4.64) is the Parseval equality. Exercise 4.134. Let v, w ∈ V . Show that X hv, φi i hφi , wi hv, wi = i∈S

for any orthonormal basis {φi : i ∈ S} of V. 20

An unconditionally convergent series gives the same finite value for any ordering of the terms.

4.4 Hilbert Spaces

157

4.4.3 Operators on Hilbert Spaces Throughout this subsection, V and W are Hilbert spaces. Exercise 4.135. The operator norm kΦkW ←V of Φ ∈ L(V, W ) defined in (4.6a) coincides with the definition |hΦv,wiW | p sup kΦkW ←V = . hv,viV hw,wiW 06=v∈V,06=w∈W Definition 4.136. (a) The operator Φ ∈ L(V, W ) gives rise to the adjoint operator21 Φ∗ ∈ L(W, V ) defined by hΦv, wiW = hv, Φ∗ wiV . (b) If V = W and Φ = Φ∗ ∈ L(V, V ), the operator is called selfadjoint. Next, we consider the subspace K(V, W ) ⊂ L(V, W ) of compact operators (cf. Definition 4.14 and §4.2.9). We recall that W ⊗ V ∗ can be interpreted as a subspace of K(V, W ) (cf. Corollary 4.92). The (finite) singular-value decomposition from Lemma 2.23 can be generalised to the infinite-dimensional case. Theorem 4.137 (infinite singular-value decomposition). (a) For Φ ∈ K(V, W ) there are singular values σ1 ≥ σ2 ≥ . . . with σν & 0 and orthonormal systems {wν ∈ W : ν ∈ N} and {vν ∈ V : ν ∈ N} such that ∞ X Φ= σν wν vν∗ (cf. (4.63)), (4.65) ν=1

where the sum converges with respect to the operator norm k·kW ←V : kΦ − Φ(k) kW ←V = σk+1 & 0

for Φ(k) :=

k X

σν wν vν∗ .

ν=1

(b) Conversely, any Φ defined by (4.65) with σk & 0 belongs to K(V, W ). Proof. Set Ψ := Φ∗ Φ ∈ L(V, V ). As product of compact operators, Ψ is compact. The Riesz–Schauder theory (cf. [141, Theorem 6.89]) states that Ψ has eigenvalues λν with λν → 0. Since Ψ is selfadjoint, there are corresponding eigenfunctions wν which can be chosen orthonormally defining an orthonormal system {vν : ν ∈ N}. As Ψ is positive semidefinite, i.e., hΨ v, viV ≥ √ 0 for all v ∈ V , we conclude that λν ≥ 0. Hence the singular values σν := + λν are well-defined. Finally, set wν := Φvν / kΦvν k = σ1ν Φvν (the latter equality follows 2 from kΦvν k = hΦvν , Φvν i = hvν , Φ∗ Φvν i = hvν , Ψ vν i = λν hvν , vν i = λν ). The vectors wν are already normalised. Since 21

There is a slight difference between the adjoint operator defined here and the dual operator from Definition 4.23, since the latter belongs to L(W ∗ , V ∗ ). As we may identify V = V ∗ and W = W ∗ , this difference is not essential.

158

4 Banach Tensor Spaces

hwν , wµ i kΦvν k kΦvµ k = hΦvν , Φvµ i = hvν , Φ∗ Φvµ i = λµ hvν , vµ i = 0 for ν 6= µ, {wν : ν ∈ N} is an orthonormal system in W . Besides Φvν = σν wν (by the definition of wν ) also Φ(k) vν = σν wν holds for ν ≤ k since vµ∗ (vν ) = hvν , vµ i = δνµ and ! Xk Xk ∗ σµ wµ δνµ = σν wν . σµ wµ vµ (vν ) = µ=1

µ=1



We conclude that Φ − Φ(k) (vν ) = 0 for ν ≤ k, while Φ − Φ(k) (vν ) = Φ(vν ) for

∗ 2 2 ≥ σk+2 ν > k. Hence Φ − Φ(k) Φ − Φ(k) has the eigenvalues σk+1 ≥ ... This implies that kΦ − Φ(k) kW ←V = σk+1 . Convergence follows by σν & 0. For the opposite direction use the facts that Φ(k) is compact because of the finitet u dimensional range and that limits of compact operators are again compact. Corollary 4.138. If κ(·, ·) is the Schwartz kernel of Φ, i.e., Z Φ(v) := κ(·, y)v(y)dy for v ∈ V, Ω P∞ we may write (4.65) as κ(x, y) = ν=1 σν wν (x)vν (y). Representation (4.65) allows us to define a scale of norms k·kSVD,p (cf. (4.17)), ∞ which use the `p norm of the sequence σ = (σν )ν=1 of singular values.22 Remark 4.139. (a) kΦkSVD,∞ = kΦk∨(W,V ) = kΦkV ←W is the operator norm. (b) kΦkSVD,2 = kΦkHS is the Hilbert–Schmidt norm. (c) kΦkSVD,1 = kΦk∧(W,V ) determines the nuclear operators. In the context of Hilbert spaces it is of interest that the Hilbert–Schmidt operators form again a Hilbert space with scalar product defined via the trace, which, for the finite-dimensional case, has already been defined in (2.8). In the infinitedimensional case, the definition of the trace is generalised by X trace(Φ) := hφi , Φφi i for any orthonormal basis {φi : i ∈ S} . (4.66) i∈S

To show that this definition makes sense, we must prove that the right-hand side does not depend on the particular basis. Let {ψj : j ∈ T } be another orthonormal basis. Then Exercise 4.134 shows that X  X XX X hφi , ψj i hψj , Φφi i = ψj , Φ hφi , ψj i φi hφi , Φφi i = i∈S j∈T

i∈S

=

X

i∈S

j∈T

hψj , Φψj i .

j∈T 22

In physics, in particular quantum information, the entropy −

P

ν

σν log(σν ) is of interest.

4.4 Hilbert Spaces

159

Definition 4.140 (Hilbert–Schmidt space). The Hilbert–Schmidt scalar product of Φ, Ψ ∈ L(V, W ) is defined by hΦ, Ψ iHS := trace(Ψ ∗ Φ) and defines the norm q p kΦkHS := hΦ, ΦiHS = trace(Φ∗ Φ). Operators Φ ∈ L(V, W ), with finite norm kΦkHS form the Hilbert–Schmidt space23 HS(V, W ). As stated in Remark 4.139b, the norms kΦkSVD,2 = kΦkHS coincide. Since P∞ 2 finiteness of ν=1 σν implies σν & 0, Theorem 4.137b proves the next result. Remark 4.141. HS(V, W ) ⊂ K(V, W ). A tensor from V ⊗ W 0 may be interpreted as map (v ⊗ w0 ) : w 7→ hw, w0 iW · v from L(W, V ). In §4.2.9 this approach has led to the nuclear operator equipped with the norm kΦkSVD,1 . For Hilbert spaces, the norm kΦkSVD,2 = kΦkHS is more natural. Lemma 4.142. Let V ⊗k·k W = V ⊗k·k W 0 be the Hilbert tensor space generated by the Hilbert spaces V, W . Interpreting Φ = v⊗w ∈ V ⊗k·kW as a map in L(W, V ), the tensor norm kv⊗wk coincides with the Hilbert–Schmidt norm kΦkHS . P∞ ∗ Proof. By Theorem 4.137, there is a representation Φ = ν=1 σν vp ν wν ∈ L(W, V ) P∞ 2 with orthonormal vν , wν . The Hilbert–Schmidt norm is equal to ν=1 σν (cf. Remark 4.139b). Interpreting Φ as a tensor from V ⊗k·k W, we use the notation ∞ ∞ P P 2 Φ= σν vν ⊗ wν . By orthonormality, kv⊗wk = σν2 leads to the same norm. t u ν=1

ν=1

Combining this result with Remark 4.141, we derive the next statement. Remark 4.143. Φ ∈ V ⊗k·k W interpreted as mapping from L(W, V ) is compact.

4.4.4 Orthogonal Projections A (general) projection has already been defined in Definition 3.4. Definition 4.144. Φ ∈ L(V, V ) is called an orthogonal projection, if it is a projection and selfadjoint. Remark 4.145. (a) Set R := range(Φ) := {Φv : v ∈ V } for a projection Φ ∈ L(V, V ). Then Φ is called a projection onto R. v = Φ(v) holds if and only if v ∈ R. (b) Let Φ ∈ L(V, V ) be an orthogonal projection onto R. Then R is closed and Φ is characterised by 23

This space is also called Schmidt class and denoted by (sc) in Schatten-von Neumann [256, Def. 2.1] or by (σc) in Schatten [255].

160

4 Banach Tensor Spaces

 Φv =

v for v ∈ R 0 for v ∈ R⊥

 ,

where V = R ⊕ R⊥ (cf. Remark 4.130b). (4.67)

(c) Let a closed subspace R ⊂ V and w ∈ V be given. Then the best approximation problem find a minimiser vbest ∈ R

of

kw − vbest k = min kw − vk v∈R

has the unique solution vbest = Φw, where Φ is the projection onto R from (4.67). (d) An orthogonal projection 0 6= Φ ∈ L(V, V ) has the norm kΦkV ←V = 1. (e) If Φ is the orthogonal projection onto R ⊂ V , then I − Φ is the orthogonal projection onto R⊥ . (f) Let {b1 , . . . , br } be an orthonormal basis of a subspace R ⊂ V . Then the orthogonal projection onto R is explicitly given by Φ=

r X

bν b∗ν ,

i.e.,

Φv =

ν=1

r X

hv, bν i bν .

ν=1

In the particular case of V = Kn with the Euclidean scalar product, we form the orthogonal matrix U := [ b1 , . . . , br ]. Then Φ = U U H ∈ Kn×n is the orthogonal projection onto R = range{U }. Lemma 4.146. (a) Let P1 , P2 ∈ L(V, V ), where P1 is an orthogonal projections. Then k (I − P1 P2 ) vkV2 ≤ k (I − P1 ) vkV2 + k (I − P2 ) vkV2

for any v ∈ V.

(b) Let Pj ∈ L(V, V ) be orthogonal projections for 1 ≤ j ≤ d − 1. Then k(I − P1 P2 · · · Pd−1 Pd )vkV2 ≤

d X

k (I − Pj ) vkV2

for any v ∈ V.

j=1

Proof. In (I − P1 P2 ) v = (I − P1 ) v + P1 (I − P2 ) v the two terms on the righthand side are orthogonal. Therefore k (I − P1 P2 ) vk2V = k (I − P1 ) vkV2 + kP1 (I − P2 ) vk2V 2

≤ k (I − P1 ) vkV2 + kP1 kV ←V k (I − P2 ) vk2V ≤ k (I − P1 ) vk2V + k (I − P2 ) vk2V using kP1 kV ←V ≤ 1 from Remark 4.145d proves Part (a). Part (b) follows by Qd induction: replace P2 in Part (a) with j=2 Pj . t u

4.5 Tensor Products of Hilbert Spaces

161

4.5 Tensor Products of Hilbert Spaces 4.5.1 Induced Scalar Product Let h·, ·ij be a scalar product defined on Vj (1 ≤ j ≤ d); i.e., Vj is a pre-Hilbert Nd space. Then V := a j=1 Vj is again a pre-Hilbert space with a scalar product Nd Nd h·, ·i which is defined for elementary tensors v = j=1 v and w = j=1 w(j) by *

d O

v

(j)

j=1

,

d O

+ w

(j)

:=

d Y

hv (j) , w(j) ij

for all v (j), w(j) ∈ Vj .

(4.68)

j=1

j=1

In the case of norms, we saw that a norm defined on elementary tensors does not determine the norm on the whole tensor space. This is different for a scalar product. One verifies that hv, wi is a sesquilinear form. Hence its definition on elementary tensors extends to V×V. The symmetry hv, wi = hw, vi also follows immediately from the symmetry of h·, ·ij . It remains to prove the positivity (4.61b). Lemma 4.147. Equation (4.68) defines a unique scalar product on which we call the induced scalar product.

a

Nd

j=1

Vj ,

Proof. (i) Consider d = 2, i.e., a scalar Pnproduct on V ⊗a W . Let h·, ·iV , h·, ·iW be scalar products of V, W , and x = i=1 vi ⊗wi 6= 0. Without loss of generality we may assume that {vi } and {wi } are linearly independent (cf. Lemma 3.15). Consequently, the Gram matrices Gv = hvi , vj iV


n i,j=1

and

Gw = hwi , wj iW


n i,j=1

are positive definite (cf. Exercise 2.17b). The scalar product hx, xi is equal to n X

hvi , vj iV hwi , wj iW =

n X

Gv,ij Gw,ij = trace(Gv GT w ).

i,j=1

i,j=1 1/2

1/2

T (cf. Remark 2.14a) yields Exercise 2.8a with A := Gv and B := Gv Gw 1/2 1/2 T 1/2 1/2 T T Gv trace(Gv Gw ) = trace(Gv Gw Gv ). The positive-definite matrix Gv Gw has positive diagonal elements (cf. Remark 2.14b), proving hx, xi > 0. ! d d−1 N N (ii) For d ≥ 3 the assertion follows by induction: a Vj = a Vj ⊗a Vd j=1 j=1 Nd−1 u with the scalar product of a j=1 Vj as in (4.68) but with d replaced by d − 1. t

Definition (4.68) implies that elementary tensors v and w are orthogonal if and only if vj ⊥ wj for at least one index j. A simple observation is stated next.

162

4 Banach Tensor Spaces (j)

Remark 4.148. Orthogonal [orthonormal] systems {φi : i ∈ Bj } ⊂ Vj for 1 ≤ j ≤ d induce the orthogonal [orthonormal] system in V consisting of φi :=

d O

(j)

φij

for all i = (i1 , . . . , id ) ∈ B := B1 × . . . × Bd .

j=1 (j)

If {φi } are orthonormal bases of Vj , {φi } is an orthonormal basis of V. Example 4.149. Consider Vj = KIj endowed with the Euclidean scalar product Nd from Example 4.129. Then the induced scalar product of v, w ∈ V = j=1 Vj is given by X X X hv, wi = vi wi = ··· v[i1 · · · id ] w[i1 · · · id ]. i∈I

i1 ∈I1

id ∈Id

The corresponding (Euclidean) norm is denoted by k·k or more specifically by k·k2 . There is qaPslight mismatch in the case of d = 2 . If we treat v as a tensor, 2 kvk2 = denotes the Euclidean norm. However, if M = v is i,j |vij | considered as a matrix, kM k2 denotes the spectral norm, whereas the previous Euclidean norm is called Frobenius norm and denoted by kM kF (cf. §2.3). The standard Sobolev space H N is a Hilbert space corresponding to p = 2 in Example 4.122. As seen in §4.3.5, H N is an intersection space with a particular Nd intersection norm. Therefore we cannot define H N = j=1 Vj by the induced (n) scalar product (4.68). The Hilbert space Vj is endowed with the scalar product h·, ·ij,n involving the L2 scalar product of the functions and their n-th derivatives. The space V(n) , and the set N are defined as in §4.3.5. Then the scalar product on \ V= V(n) n∈N

is defined by * d + d d O O XY (N ) (j) (j) v , w := hv (j) , w(j) ij,nj for all v (j), w(j) ∈ Vj j . (4.69) j=1

j=1

n∈N j=1

In this definition v and w are elementary tensors of the space Vmix , which by Remark 4.124b is dense in V. The bilinear (sesquilinear) form defined in (4.69) is positive since a convex combination of positive forms is again positive. The corresponding norm sX kvk = kvk2n n∈N

is equivalent to max kvkn in (4.58b). n∈N

4.5 Tensor Products of Hilbert Spaces

163

4.5.2 Crossnorms Proposition 4.150. The norm derived from the scalar product (4.68) is a reasonable crossnorm. Furthermore, it is a strongly uniform crossnorm (cf. Definition 4.110). Qd Proof. (i) Taking v = w in (4.68) shows that kvk = i=1 kvi k for all v ∈ V. (ii) Since the dual spaces Vi∗ and V∗ may be identified with Vi and V, respectively, part (i) proves the crossnorm property (4.44) for V∗ . Nd (iii) First we consider the finite-dimensional case. Let A = j=1 A(j) with A(j) ∈ L(Vj , Vj ) and v ∈ V. Diagonalisation yields A(j)∗A(j) = Uj∗ Dj Uj (Uj (j) unitary, Dj diagonal). The columns {φi : 1 ≤ i ≤ dim(Vj )} of Uj form an orthonormal bases of Vj . Define the orthonormal basis {φi } according to Remark 4.148 P P 2 2 and represent v as v = i ci φi . Note that kvk = i |ci | (cf. (4.64)). Then

* d + d d

X O

2 X O O

(j) (j) (j) A(j) (φkj ) A(j) (φij ), ck kAvk = ci A(j) (φij ) = ci

2

j=1

i

j=1

j=1

i,k

d D d D E E X Y X Y (j) (j) (j) (j) = ci ck A(j) φij , A(j) φkj = ci ck φij , A(j)∗ A(j) φkj . j

j=1

i,k

j=1

i,k

(j)

(j)

(j)

Since φkj are eigenvectors of A(j)∗A(j) , the products hφij , A(j)∗ A(j) φkj i vanish for ij 6= kj . Hence, 2

kAvk =

X

d D d

E X Y Y

(j) (j) 2 (j) (j) 2 φij , A(j)∗ A(j) φij = |ci |

A φij j=1

i

≤

2

|ci |

Y d

kA(j) kVj ←Vj

2 X

j=1

=

Y d

|ci |

2

i

kA(j) kVj ←Vj

2 X

j=1

i

d Y

j

j=1

i (j)

kφij k2j j=1 | {z } =1

2

|ci | =

Y d

kA(j) kVj ←Vj

2

2

kvk

j=1

proves that the crossnorm is uniform. Nd (iv) Now consider the infinite-dimensional case. The tensor v ∈ V := a j=1 Vj Pn Nd Nd (j) has some representation v = i=1 j=1 vi and, therefore, v ∈ V0 := j=1 V0,j (j) holds with the finite-dimensional subspaces V0,j := span{vi : 1 ≤ i ≤ n}. Let Φj = Φ∗j ∈ L(Vj ,Vj ) be the orthogonal projection onto V0,j . An easy exercise shows that 2

kAvk = hv, A∗ Avi  O    O d d = v, A(j)∗ A(j) v = v, j=1

j=1

  Φ∗j A(j)∗ A(j) Φj v .

164

4 Banach Tensor Spaces

Set Cj := Φ∗j A(j)∗ A(j) Φj = (A(j) Φj )∗ (A(j) Φj ) = B (j)∗ B (j) for the well1/2 defined square root B (j) := Cj . Since the operator acts in the finite-dimensional subspace V0,j only, part (iii) applies. The desired estimate follows from kB (j) k2Vj ←Vj = kB (j)∗ B (j) kVj ←Vj = k(A(j) Φj )∗ (A(j) Φj )kVj ←Vj = kA(j) Φj kV2 j ←Vj ≤ kA(j) kV2 j ←Vj kΦj k2Vj ←Vj = kA(j) k2Vj ←Vj (cf. Remark 4.145). (v) Let N γ = α ∪ β be a disjoint union of α, β ∈ D = {1, . . . , d}. Define Vα = j∈α Vj and Vβ by means of the induced scalar products h·, ·iα and h·, ·iβ . The scalar product h·, ·iγ of Vγ can be constructed in two ways: (i) induced by h·, ·ij for all j ∈ γ and (ii) induced by h·, ·iα and h·, ·iβ . It is easy to see that both constructions lead to identical results. The uniform crossnorm property of t u Vγ = Vα ⊗ Vβ (construction (ii)) proves the strong uniformity. The projective crossnorm k·k∧ for `2 (I)×`2 (J) is discussed in Example 4.54. The result shows that the generalisation for d ≥ 3 does not lead to a standard norm. Nd The injective crossnorm k·k∨ of a j=1 Vj is defined in (4.46). For instance, Vj = `2 (Ij ) endowed with the Euclidean scalar product leads to P P (d) (1) i1 ∈I1 · · · id ∈Id v[i1 · · · id ]·wi1 ·. . .·wid kvk∨(`2 ,...,`2 ) = sup . kw(1) k1 · . . . · kw(d) kd 06=w(j) ∈Vj 1≤j≤d

If d = 1, kvk∨ coincides with kvk2 . For d = 2, kvk∨ is the spectral norm24 kvk2 of v interpreted as matrix (cf. (2.11)).

4.5.3 Tensor Products of L(Vj , Vj ) The just proved uniformity shows that the Banach spaces L(Vj , Vj ), k · kVj ←Vj form the tensor space d O L(Vj , Vj ) ⊂ L(V, V) a




j=1

and that the operator norm k·kV←V is a crossnorm (cf. Exercise 4.128). It is even a crossnorm in the stronger sense of (4.48c). Note that (L(V, V), k·kV←V ) is a Banach space but not a Hilbert space. To obtain a Hilbert space, we have to consider the space HS(Vj , Vj ) of the Hilbert– Schmidt operators with the scalar product h·, ·ij,HS (cf. Definition 4.140). The scalar 24

By this reason—also for d ≥ 3 —the injective crossnorm is sometimes called the spectral norm.

4.5 Tensor Products of Hilbert Spaces

165

Nd products h·, ·ij,HS induce the scalar product h·, ·iHS on H := a j=1 HS(Vj , Vj ) . Equation (4.81) shows that h·, ·iHS is defined by the trace on H. Nd Exercise 4.151. For A(j) v (j) = λj v (j) (v (j) 6= 0) and A := j=1 A(j) prove: Nd (a) The elementary tensor v := j=1 v (j) is an eigenvector of A with eigenvalue Qd λ := j=1 λj , i.e., Av = λv. (b) Assume that A(j) ∈ L(Vj , Vj ) has dim(Vj ) < ∞ eigenpairs (λj , v (j) ). Then all eigenpairs (λ, v) constructed in Part (a) yield the complete set of eigenpairs of A. Exercise 4.151b requires that all A(j) be diagonalisable. The next lemma treats the general case. Lemma 4.152. Let A(j) ∈ CIj ×Ij be a matrix with #Ij < ∞ for 1 ≤ j ≤ d and Nd form the Kronecker product A := j=1 A(j) ∈ CI×I . Let (λj,k )k∈Ij be the tuple of eigenvalues of A(j) corresponding to their multiplicity. Then (λk )k∈I

with λk :=

d Y

λj,kj

(4.70)

j=1

represents all eigenvalues of A together with their multiplicity. Note that λk might be a multiple eigenvalue by two reasons: (a) λj,kj = λj,kj +1 = . . . = λj,kj +µ−1 is a µ-fold eigenvalue of A(j) with µ > 1, (b) different factors λj,kj 6= λj,kj0 may produce the same product λk = λk0 . Proof. For each matrix A(j) there is a unitary similarity transformation R(j) = U (j) A(j) U (j)H (U (j) unitary) into an upper triangular matrix R(j) (Schur normal form; cf. [140, Theorem A.34]). Hence A(j) and R(j) have identical eigenvalues Nd Nd including their multiplicity. Set U := j=1 U (j) and R := j=1 R(j) . U is again unitary (cf. (4.80a,b)), while R ∈ CI×I is of upper triangular form with λk in (4.70) as diagonal entries. Since the eigenvalues of triangular matrices are given by the diagonal elements of R(j) (including the multiplicity), the assertion follows. t u

4.5.4 Gagliardo–Nirenberg Inequality It will be very convenient to use the induced Hilbert norm (usually Euclidean or L2 norm). On the other hand, there is a need to control other norms, in particular, the supremum norm k·k∞ . The Gagliardo–Nirenberg inequality (cf. Nirenberg [232, p. 125] and Maz’ja [222]25 ) relates the supremum norm to the L2 norm, provided that the function is smooth enough. For the latter property, the following semi-norm is introduced, sZ Xd ∂ m u 2 dx, (4.71) |u|m := j=1 ∂xm Ω j 25

Use the parameters q = ∞, j = 0, p = r = 2, ` = m, n = d in Maz’ja [222, Eq. (2.3.50)].

166

4 Banach Tensor Spaces

which is defined for u ∈ H m (Ω) = H m,2 (Ω) (cf. Example 4.122). Note that |·|m involves no mixed derivatives. We discuss the inequality for three model cases of Ω: d Rd , [0, ∞)d , and [0, 1]q . In the latter case, the semi-norm |·|m must be replaced with a norm, e.g., with

2

2

|·|m + k·k .

 Theorem 4.153. Let Ω ∈ Rd , [0, ∞)d , [0, 1]d and suppose that m > d/2 . Then any function u ∈ H m (Ω) satisfies the inequality  d  d 1− 2m 2m  cΩ for Ω ∈ Rd , [0, ∞)d , m |ϕ|m kϕkL2 h i d kuk∞ ≤ d  cΩ |ϕ|2 + kϕk2 4m kϕk1−2 2m for Ω = [0, 1]d . m m L The constant cΩ m will be discussed in more detail below. For this purpose, we give an explicit proof based on the properties of certain Green functions corresponding to a variational problem described next. Let Ω = Ω1 × . . . × Ωd with Ωj ⊂ R. The scalar product of L2 (Ω) is denoted by (·, ·), while k·k is the corresponding norm. For m ∈ N, we define the Pd  ∂ m u ∂ m v  bilinear form hu, vim := j=1 ∂xm , ∂xm corresponding to the semi-norm |·|m j j introduced above. For positive numbers α, β, we define the bilinear form 2 2 a(u, v) := aΩ m,α,β (u, v) := α · hu, vim + β · (u, v) .

The corresponding norm is denoted by Ω

|||u||| = |||u|||m,α,β :=

p

a(u, u) .

Ω

|||u|||m,α,β , for different α, β > 0, are equivalent norms of the Sobolev space H m (Ω). The Sobolev embedding theorem ensures H m (Ω) ⊂ C(Ω) for m > d2 (cf. [141, Theorem 6.48]); i.e., k·k∞ ≤ γ · ||| · ||| holds. We set Ω

Ω γ = γm,α,β := sup{kuk∞ /|||u|||m,α,β : 0 6= u ∈ H m (Ω)}.

(4.72)

As a consequence of the embedding H m (Ω) ⊂ C(Ω) for m > d/2, the Dirac functional δξ (ξ ∈ Ω) with δξ (u) = u(ξ) belongs to H m (Ω)0 . The Green function Gξ = G(·, ξ) = GΩ m,α,β (·, ξ) is the solution of the variational formulation aΩ m,α,β (Gξ , v) = v(ξ)

for all v ∈ H m (Ω) and a fixed ξ ∈ Ω.

(4.73)

The solution Gξ ∈ H m (Ω) satisfies the following properties. 2

Lemma 4.154. (a) G(ξ, ξ) = |||Gξ ||| > 0, p (b) |u(ξ)| ≤ G(ξ, ξ)|||u||| for all u ∈ V , and the maximum of the ratio |u(ξ)| |||u||| is taken for u = Gξ . (c) γ in (4.72) satisfies γ 2 = sup{|G(x, y)| : x, y ∈ Ω} = sup{G(ξ, ξ) : ξ ∈ Ω}.

4.5 Tensor Products of Hilbert Spaces

167

Proof. (i) v = Gξ in (4.73) yields G(ξ, ξ) = a(Gξ , Gξ ) > 0. p (ii) |u(ξ)| =(4.73) |a(Gξ , u)| ≤ |||Gξ ||||||u||| =(a) G(ξ, ξ)|||u|||. Equality holds for u = Gξ . (iii) u = Gy and ξ = x in (c) yield p p |G(x, y)| ≤ G(x, x)|||Gy ||| =(a) G(x, x)G(y, y); i.e., |G(x, y)| ≤ max{G(x, x), G(y, y)} and the supremum is taken along the diagonal {(ξ, ξ) : ξ ∈ Ω}. t u First, we consider the case Ω = Rd . The translation operator Tδ (δ ∈ Rd ) is defined by (Tδ u) (x) = u(x + δ). Tδ is unitary in L2 (Ω): Tδ∗ = Tδ−1 = T−δ , d and the bilinear form a = aR m,α,β satisfies a(Tδ u, v) = a(u, T−δ v).

(4.74)

Conclusion 4.155. Under assumption (4.74), G depends only on the difference of its arguments: G(x, y) = G(x − y). In particular, G(x, x)p= G(y, y) for all x, y ∈ Rd , and γ from Lemma 4.154d can be defined by γ = G(0, 0). Proof. We must show that G(x, y) = G(x + δ, y + δ). Use G(x + δ, y + δ) = (Tδ Gy+δ ) (x) = a(Gx , Tδ Gy+δ ) = a(Tδ Gy+δ , Gx ) = a(Gy+δ , T−δ Gx ) = (T−δ Gx )(y + δ) = Gx (y) = G(y, x) = G(x, y). t u Lemma 4.156. In the case of Ω = [0, ∞)d , the maximum is also taken at ξ = 0: γ 2 = supξ∈Ω G(ξ, ξ) = G(0, 0). Proof. Assume that for some 0 6= ξ ≥ 0 (pointwise inequality), G(ξ, ξ) > G(0, 0) holds. Define g(x) := G(x+ξ, ξ) and note that g(0) = G(ξ, ξ). The squared norm 2 |||g||| is an integral over [0, ∞)d and equal to the integral over [ξ1 , ∞)×. . .×[ξd , ∞) 2 with g replaced by Gξ . Obviously, the latter integral is not larger than |||Gξ ||| , i.e., |||g||| ≤ |||Gξ |||. p By Lemma 4.154b, G(ξ,ξ) = |g(0)| ≤ G(0,0) = G(0, 0) holds, while |||g||| |||g||| |||G(0,0)||| p the previous inequality yields the contradiction G(ξ,ξ) ≥ G(ξ,ξ) = G(ξ, ξ) > |||g||| |||Gξ ||| p G(0, 0). t u  For Ω ∈ [0, ∞)d , Rd we can define26 the dilatation Mλ (λ > 0) centred at 0 ∈ Rd by (Mλ u)(x) := u(λx) (x ∈ Ω). The substitution rule yields (Mλ u, Mλ v) = λ−d/2 (u, v) and the chain rule ∂m ∂m m Mλ ∂x proves ∂x m (Mλ u) = λ m u; hence, j

26

j

In general, Ω = Ω1 × . . . × Ωd must be assumed to be a cone (with origin at 0), i.e., x ∈ Ω implies λx ∈ Ω for all λ ≥ 0.

168

4 Banach Tensor Spaces Ω −d Ω (Mλ u, Mλ v) = aΩ am,αλm ,β (u, v). am,α,β m,αλm−d/2 ,βλ−d/2 (u, v) = λ

Ω Choosing λm = β/α, we can relate am,α,β to aΩ m,1,1 :

Ω am,α,β

 − md  − md β β Ω 2 M β m1 u, M β m1 v = am,β,β (u, v) = β aΩ m,1,1 (u, v) (α) (α) α α


or equivalently 2 −d/m Ω aΩ am,1,1 (M( β )−1/m u, M( β )−1/m v). m,α,β (u, v) = β (β/α) α

α

Ω Lemma 4.157. The Green functions GΩ m,α,β belonging to am,α,β satisfy a similar relation:
−2 1/m GΩ (β/α)d/m GΩ x, (β/α)1/m y . (4.75) m,α,β (x, y) = β m,1,1 (β/α) 1/m Proof. Set u(x) = β −2 (β/α)d/m GΩ x, (β/α)1/m y) and test with m,1,1 ((β/α) some function v: Ω aΩ m,α,β (u, v) = am,1,1 (M

1

1

β −m (α )

1

β m β m GΩ m,1,1 (( α ) •, ( α ) y), M

1

β −m (α )

v)

Ω 1/m = aΩ y), M(β/α)−1/m v) m,1,1 (Gm,1,1 (•, (β/α)

= δ(β/α)1/m y (M(β/α)−1/m v) = (M(β/α)−1/m v)((β/α)1/m y) = v(y) = δy (v), i.e., u satisfies the variational problem defining GΩ m,α,β (·, y).

t u

Since the supremum norm is invariant under dilatation, we obtain the following result from (4.75). Ω Lemma 4.158. The quantity γ = γm,α,β defined in Lemma 4.154d is the following function of α, β :

q   d q −1 β 2m kGΩ ( · , 0)k = β kGΩ ∞ m,1,1 ( · , 0)k∞ m,α,β α d   2m β Ω = β −1 α γm,1,1 .

Ω γm,α,β =

Let Qa := [0, a]d be the cube with side length a. Finally, we mention the case of Ω = Q1 = [0, 1]d . Application of the dilatation operator Mλ to u ∈ H m (Ω) yields a function Mλ u ∈ H m (Qλ ) , where, in contrast to the situation above, the domain Qλ depends on λ. Now (4.75) takes the form Q

(β/α) −2 GΩ (β/α)d/m Gm,1,1 m,α,β (x, y) = β

1/m


(β/α)1/m x, (β/α)1/m y .

(4.76)

4.5 Tensor Products of Hilbert Spaces

169

We conjecture that again G(0, 0) ≥ G(ξ, ξ) is valid, however the previous proof does not apply to the cube. For different domains the constants γ compare as follows. If Ω 0 ⊂ Ω 00 and q 00

Ω γm,1,1 =

0

00

00

0 Ω Ω GΩ m,1,1 (ξ, ξ) for some ξ ∈ Ω , then γm,1,1 ≥ γm,1,1 .

Now we return to the proof of Theorem 4.153 and specify the involved constants. Proposition 4.159. Theorem 4.153 holds with constants cΩ m satisfying  ( Ω am γm,1,1 if Ω ∈ Rd , [0, ∞)d , Ω cm = √ Q(1+|u|2 /kuk2 )1/(2m) (4.77) 2 γm,1,1 m if Ω = Q1 = [0, 1]d , where o n d d am := min sin 2m −1 (δ) cos− 2m (δ) : 0 < δ < π/2

√ d d d d = sin 2m −1 12 arccos( m − 1) cos− 2m 12 arccos( m − 1) ≤ 2 has the asymptotic behaviour am = 1 + −d/2 If Ω = Rd , lim cΩ holds. m =π

d 4m

log

2m d

d + O( m ) → 1 for m → ∞.

Proof. (i) Fix a function u ∈ H m (Ω). Let M := |u|m and L := kuk, and set α := cos(δ)/M and β := sin(δ)/L for any δ ∈ (0, π/2). By construction, Ω

|||u|||m,α,β = 1 holds. We infer from (4.72) and Lemma 4.158 that d

Ω

Ω Ω Ω kuk∞ ≤ γm,α,β |||u|||m,α,β = γm,α,β = β −1 (β/α) 2m γm,1,1
d d d 1− d Ω Ω 2m = γm,1,1 · (β −1 )1− 2m · α−1 2m = a(δ) · γm,1,1 · kuk 2m · |u|m

 d d for Ω ∈ Rd , [0, ∞)d . The factor a(δ) is equal to sin 2m −1 (δ) cos− 2m (δ). The √ choice δ = π/4 yields a(π/4) = 2, while the minimum is taken at the value d δ = 12 arccos( m − 1) yielding the asymptotic behaviour described above. q √ 2 2 If Ω = [0, 1]d , set M := |u|m + kuk , L := kuk, and27 α := 1/( 2 M ), √ β := 1/( 2 L). Now, using (4.76), we can proceed as above. Because of the 2 2 2 different definition of M , |u|m must be replaced with |u|m + kuk . (ii) The Fourier transform (cf. [135]) shows that the constant in (4.77) is equal to s −1 Z  Xd −d/2 Ω cm = am · (2π) 1+ ξj2m dξ. Rd

j=1

Pd For m → ∞, the function (1+ j=1 ξj2m )−1 tends to 0 when kξk∞ > 1, and to 1 −d/2 when kξk∞ < 1. Together with am → 1, the statement cΩ follows. t u m →π 27

The estimate is not optimised √ concerning the choice of α and β. The√present choice corresponds to cos(δ) = sin(δ) = 1/ 2 in the previous proof. Hence, am = 2 may be improved.

170

4 Banach Tensor Spaces

4.5.5 Partial Scalar Products Let X := V1 ⊗a W and Y := V2 ⊗a W be two tensor spaces sharing a pre-Hilbert space (W, h·, ·iW ). We define a sesquilinear form (again denoted by h·, ·iW ) via h·, ·iW : X × Y → V1 ⊗a V2 , hv1 ⊗ w1 , v2 ⊗ w2 iW := hw1 , w2 iW · v1 ⊗ v2 for v1 ∈ V1 , v2 ∈ V2 , and w1 , w2 ∈ W. We call this operation a partial scalar product, since it acts on the W part only. We now assume V1 = V 2 so that X = Y. We rename X and Y by V with N the usual structure V = aN j∈D Vj , D = {1, . . . , d}. In this case W from above corresponds to Vα = a j∈α Vj for a nonempty subset α ⊂ D. The notation h·, ·iW is replaced with h·, ·iα :

*

d O

vj ,

j=1

d O j=1

h·, ·iα : V × V → VD\α ⊗a VD\α , + " # ! O Y wj := hvj , wj i · vj ⊗ j∈α

α

! O

j∈D\α

wj .

j∈D\α

The partial scalar product h·, ·iα : V × V → Vαc ⊗a Vαc can be constructed as composition of the following two mappings: 1) sesquilinear concatenation (v, w) 7→ v ⊗ w ∈ V ⊗a V followed by 2) contractions28 explained below. N Definition 4.160. For a nonempty, finite index set D let V = Vd = a j∈D Vj be a pre-Hilbert space with induced scalar product. For any j ∈ D, the contraction Cj : V ⊗a V → VD\{j} ⊗a VD\{j} is defined by ! !! ! ! O O O O Cj vk ⊗ wk := hvj , wj i vk ⊗ wk . k∈D

k∈D

k∈D\{j}

k∈D\{j}

For a Q subset α ⊂ D, the contraction Cα : V ⊗a V → VD\α ⊗a VD\α is the product Cα = j∈α Cj with the action  Cα 

j∈D

28

!

! O

vj ⊗

O j∈D

wj  =

"

# Y j∈α

hvj , wj i ·

! O j∈D\α

vj

! ⊗

O

wj .

j∈D\α

N In tensor algebras, contractions are applied to tensors from j Vj , where Vj is either the space 0 0 V its dual V . If, e.g., N V1 = V and V2 = V, the corresponding contraction is defined by N or (j) 7→ v (1) (v (2) ) · j≥3 v (j) (cf. Greub [125, p. 72]). jv

171

4.6 Tensor Operations

§5.2 will show additional matrix interpretations of these partial scalar products. The definition allows us to compute partial scalar product recursively. Formally, we may define C∅ (v) := v and hv, wi∅ := v ⊗ w. Corollary 4.161. If ∅ $ α $ β ⊂ D, then hv, wiβ = Cβ\α (hv, wiα ).

4.6 Tensor Operations In this section we specify operations which later are to be realised numerically in the various formats. With regard to practical applications, we mainly focus on a finite-dimensional setting. In §13 we shall quantify the arithmetical cost of the various operations.

4.6.1 Vector Operations The trivial · v (λ ∈ K , Nd vector space operations are the scalar multiplication λN d v ∈ a j=1 Vj ) and the addition v + w. By definition, v, w ∈ a j=1 Vj have representations as finite linear combinations. Obviously, the sum might have a representation with even more terms. This will become a source of trouble. Nd Nd The scalar product of two elementary tensors v = j=1 vj and w = j=1 wj reduces by the definition to the scalar product of the simple vectors vj , wj ∈ Vj : hv, wi =

d Y

hvj , wj i .

(4.78)

j=1

The naive computation of the scalar product by X hv, wi = vi wi i∈I d

would be much too costly. Therefore the reduction to scalar products in Vj is very helpful. Nd General vectors v, w ∈ a j=1 Vj are sums of elementary tensors. Assume that the (minimal) number of terms is nv and nw , respectively. Then nv nw scalar products (4.78) must be performed and added. Again, it becomes obvious that large numbers nv and nw cause problems. The evaluation of hv, wi is not restricted to the discrete setting Vj = RIj with finite index sets Ij . Assume the infinite-dimensional case of continuous functions from Vj = C([0, 1]) . As long as vj , wj belong to Ra (possibly infinite) family of 1 functions for which the scalar products hvj , wj i = 0 vj (x)wj (x)dx are exactly known, the evaluation of hv, wi can be realised.

172

4 Banach Tensor Spaces

4.6.2 Matrix-Vector Multiplication Nd Again, we consider Vj = KIj and V = KI = j=1 Vj with I = I1 × . . . × Id . Nd Ij ×Ij Matrices from KI×I are described by Kronecker products in . For j=1 K elementary tensors A=

d O

A(j) ∈

j=1

d O

KIj ×Ij

and

v=

Av =

vj ∈

j=1

j=1

the evaluation of

d O

d  O

A(j) v (j)

d O

KI j

j=1



j=1

requires d simple matrix-vector multiplications, whereas the naive evaluation of Av may be beyond the computer capacities. Nd The same holds for a rectangular matrix A ∈ KI×J = j=1 KIj ×Jj .

4.6.3 Matrix-Matrix Operations Nd Ij ×Ij For the addition of Kronecker tensors A, B ∈ the same statement j=1 K holds as for the addition of vectors. The multiplication rule for elementary Kronecker tensors is ! ! d d d O O O (j) (j) = A A(j) B (j) for all A(j), B (j) ∈ KIj ×Ij . (4.79) B j=1

j=1

j=1

Similarly for A(j) ∈ KIj ×Jj , B (j) ∈ KJj ×Kj . If A (B) is a linear combination of nA (nB ) elementary Kronecker tensors, nA nB evaluations of (4.79) are needed. Further rules for elementary Kronecker tensors are: !−1 d d  −1 O O (j) = A(j) A for all invertible A(j) ∈ KIj ×Ij , (4.80a) j=1

j=1 d O j=1

!T A

(j)

=

d  O

A(j)

T

for all A(j) ∈ KIj ×Jj .

(4.80b)

j=1

Exercise 4.162. Assume that all matrices A(j) ∈ KIj ×Jj have one of the properties {regular, symmetric, Hermitian, positive-definite, diagonal, lower triangular, upper triangular, orthogonal, unitary, positive, permutation matrix}. Show that the Nd (j) Kronecker matrix possesses the same property. What statements hold j=1 A for negative, negative-definite, or antisymmetric matrices A(j) ?

173

4.6 Tensor Operations

Nd

Exercise 4.163. Let A :=

j=1 A

A(j) = Q(j) R(j)

(j)

(QR),

. Assume that one of the decompositions A(j) = L(j) L(j)H

(Cholesky),

or A(j) = U (j) Σ (j) V (j)T

(SVD)

is given for all 1 ≤ j ≤ d. Prove that A possesses the respective decomposition QR (QR), LLH (Cholesky), UΣVT (SVD) with the Kronecker matrices Q :=

d O

Q(j) ,

R :=

j=1

d O

R(j) ,

etc.

j=1

Exercise 4.164. Prove the following statements about the matrix rank (cf. Remark 2.1):   d d O Y rank  A(j)  = rank(A(j) ), j=1

j=1

and the trace of a matrix (cf. (2.8)):   d d Y O trace(A(j) ). trace  A(j)  = j=1

(4.81)

j=1

The determinant involving A(j) ∈ KIj ×Ij is equal to  det 

d O j=1

 A(j)  =

d  Y

n[j] det(A(j) )

Y

with n[j] :=

j=1

#Ik .

k∈{1,...,d}\{j}

The latter identity for d = 2 is treated in the historical paper by Zehfuss [308] (cf. §1.6): matrices A ∈ Kp×p and B ∈ Kq×q lead to the determinant q

p

det(A ⊗ B) = (det A) (det B) . The connection with the Frobenius scalar (2.10) product is given by hA ⊗ B, X ⊗ Y iF = hA, XiF hB, Y iF . Further statements about elementary Kronecker products can be found in Langville–Stewart [208] and Van Loan–Pitsianis [292].

174

4 Banach Tensor Spaces

4.6.4 Hadamard Multiplication As seen in §1.1.3, univariate functions may be the subject of a tensor product, producing multivariate functions. For two functions f (x) and g(x) of the variable x = (x1 , . . . , xd ) ∈ [0, 1]d pointwise multiplication f · g is a standard operation. Replace [0, 1]d by a finite grid Gn := {xi : i ∈ I} ⊂ [0, 1]d ,

where

I = {i = (i1 , . . . , id ) : 0 ≤ ij ≤ n}, xi = (xi1 , . . . , xid ) ∈ [0, 1]d , xν = ν/n. Nd Then the entries ai := f (xi ) and bi := g(xi ) define tensors in KI = j=1 KIj , where Ij = {0, . . . , n}. Pointwise multiplication f · g corresponds to entry-wise multiplication of a and b, which is called the Hadamard product:29 a b ∈ KI

with entries

(a b)i = ai bi

for all i ∈ I.

(4.82)

Performing the multiplication for all entries would be too costly. For elementary tensors it is much cheaper to use     d d d   O O O  b(j)  = a(j) b(j) . a(j)   j=1

j=1

j=1

The following rules are valid: a b = b a, (a + a ) b = a0 b + a00 b, 0

00

a (b0 + b00 ) = a b0 + a b00 .

4.6.5 Convolution There are various versions of a convolution a ? b. First, we consider sequences in `0 (Z) (cf. Example 3.1). The convolution in Z is defined by X (a, b, c ∈ `0 (Z)) . c := a ? b with cν = aµ bν−µ (4.83a) µ∈Z

Sequences a ∈ `0 (N0 ) can be embedded into a0 ∈ `0 (Z) by setting a0i = ai for i ∈ N0 and a0i = 0 for i < 0. Omitting the zero terms in (4.83a) yields the convolution in N0 : 29

Although the name ‘Hadamard product’ for this product is widely used, it does not go back to Hadamard. However, Issai Schur mentions this product in his paper [258] from 1911. In this sense, the term ‘Schur product’ would be more correct.

175

4.6 Tensor Operations

c := a ? b

with cν =

ν X

aµ bν−µ

(a, b, c ∈ `0 (N0 )) .

(4.83b)

µ=0

The convolution of two vectors a = (a0 , . . . , an−1 ) and b = (b0 , . . . , bm−1 ), with possibly n 6= m, yields min{n−1,ν}

c := a ? b

with cν =

X

aµ bν−µ

for 0 ≤ ν ≤ n + m − 2. (4.83c)

µ=max{0,ν−m+1}

Note that the resulting vector has increased length: c = (c0 , . . . , cn+m−2 ) . For finite tuples a = (a0 , a1 , . . . , an−1 ) ∈ `({0, 1, . . . , n − 1}) = Kn , the periodic convolution (with period n) is explained by c := a ? b

with cν =

n−1 X

aµ b[ν−µ]

(a, b, c ∈ Kn ) ,

(4.83d)

µ=0

where [ν − µ] is the rest class modulo n, i.e., [m] ∈ {0, 1, . . . , n − 1} with [m] − m being a multiple of n. Remark 4.165. For a, b ∈ Kn let c ∈ K2n−1 be the result of (4.83c) and define per := cn−1 . Then cper ∈ Kn by cνper := cν + cν+n for 0 ≤ ν ≤ n − 2 and cn−1 per c is the periodic convolution result in (4.83d). The index sets Z, N, In := {0, 1, . . . , n − 1} may be replaced with the d-fold products Zd , Nd , Ind . For instance, (4.83a) becomes X
c := a ? b with cν = a, b, c ∈ `0 (Zd ) . aµ bν−µ µ∈Zd

For any I ∈ {Z, N, In }, the space `0 (I d ) is isomorphic to a ⊗d `0 (I). For elementary tensors a, b ∈ a ⊗d `0 (I) , we may apply the following rule: O d j=1

a(j)

 O  O d  d  ? b(j) = a(j) ? b(j) , j=1

a(j) , b(j) ∈ `0 (I).

(4.84)

j=1

Note that a ? b is again an elementary tensor. Since almost all entries of a ∈ `0 are zero, the sums in (4.83a) and (4.84) contain only finitely many nonzero terms. If we replace `0 with some Banach space `p , the latter sums may contain infinitely many terms and we have to check its convergence. Lemma 4.166. For a ∈ `p (Z) and b ∈ `1 (Z), the sum in (4.83a) is finite and produces a ? b ∈ `p (Z) for all 1 ≤ p ≤ ∞; furthermore, ka ? bk`p (Z) ≤ kak`p (Z) kbk`1 (Z) .

176

4 Banach Tensor Spaces

Proof. Choose any d ∈ `q (Z) with kdk`q (Z) = 1 and p1 + 1q = 1. Then the scalar P P P product hd, a ? bi = ν,µ∈Z aµ bν−µ dν can be written as α∈Z bα ν∈Z aν−α dν . Since also the shifted sequence (aν−α )ν∈Z has the norm kak`p (Z) , we obtain X aν−α dν ≤ kak`p (Z) kdk`q (Z) = kak`p (Z) . ν∈Z

P |hd, a ? bi| can be estimated by α∈Z |bα | kak`p (Z) = kak`p (Z) kbk`1 (Z) . Since `q is isomorphic to (`p )0 for 1 ≤ p < ∞, the assertion is proved except for m = ∞. The latter case is an easy conclusion from |cν | ≤ kak`∞ (Z) kbk`1 (Z) (cf. (4.83a)). t u So far, discrete convolutions have been described. Analogous integral versions for univariate functions are Z ∞ Z x (f ? g) (x) = f (t)g(x − t)dt, (f ? g) (x) = f (t)g(x − t)dt, (4.85a) −∞ 1

0

Z

f (t)g([x − t])dt,

(f ? g) (x) =

where [x] ∈ [0, 1), [x] − x ∈ Z.

0

The multivariate analogue of (4.85a) is Z (f ? g) (x) = f (t1 , . . . , td ) g(x1 − t1 , . . . , xd − td ) dt1 . . . dtd . Rd

Qd Qd Again, elementary tensors f (x) = j=1 f (j) (xj ) and g(x) = j=1 g (j) (xj ) satisfy the counterpart of (4.84): ! ! d d d   O O O (j) (j) f ? g = f (j) ? g (j) , (4.85b) j=1

j=1

j=1

i.e., the d-dimensional convolution can be reduced to d one-dimensional ones.

4.6.6 Function of a Matrix A square matrix of the size n × n has n eigenvalues λi ∈ C (1 ≤ i ≤ n, counted according to their multiplicity). They form the spectrum σ(M ) := {λ ∈ C : λ eigenvalue of M } . The spectral radius is defined by ρ(M ) := max {|λ| : λ ∈ σ(M )} .

(4.86)

177

4.6 Tensor Operations

Let f : Ω ⊂ C → C be a holomorphic function30 in an open domain Ω. The application of f to a matrix M is possible if σ(M ) ⊂ Ω . Proposition 4.167. (a) Assume M ∈ CI×I and let D be an (open) domain with σ(M ) ⊂ D ⊂ D ⊂ Ω. Then a holomorphic function on Ω gives rise to a matrix f (M ) ∈ CI×I defined by Z 1 −1 f (M ) := (ζI − M ) f (ζ) dζ . (4.87) 2πi ∂D P∞ ν (b) Assume that f (z) = converges for |z| < R with R > ρ(M ). ν=0 aν z Then an equivalent definition of f (M ) is f (M ) =

∞ X

aν M ν .

ν=0

Important functions are, e.g., f (z) = exp(z),

√ f (z) = exp( z),

f (z) = 1/z

if σ(M ) ⊂ {z ∈ C : <e(z) > 0}, if 0 ∈ / σ(M ).

Lemma 4.168. If f (M ) is defined, then f (I ⊗. . .⊗I ⊗ M ⊗ I ⊗. . .⊗I) = I ⊗. . .⊗I ⊗ f (M ) ⊗ I ⊗. . .⊗I. Nd Proof. Set M := I ⊗. . .⊗I ⊗M ⊗I ⊗. . .⊗I and I := j=1 I. As σ(M) = σ(M ) (cf. Lemma 4.152), f (M ) can be defined by (4.87) if and only if f (M) is well R −1 1 defined. (4.87) yields f (M) := 2πi (ζI − M) f (ζ)dζ. Use ∂D ζI − M = I ⊗. . .⊗I ⊗ (ζI) ⊗ I ⊗. . .⊗I − I ⊗. . .⊗I ⊗ M ⊗ I ⊗. . .⊗I = I ⊗. . .⊗I ⊗ (ζI − M ) ⊗ I ⊗. . .⊗I and (4.80a) and proceed by Z   1 −1 f (M) = I ⊗. . .⊗I ⊗ (ζI − M ) ⊗ I ⊗. . .⊗I f (ζ) dζ 2πi ∂D   Z 1 −1 = I ⊗. . .⊗I ⊗ (ζI − M ) f (ζ)dζ ⊗ I ⊗. . .⊗I 2πi ∂D = I ⊗. . .⊗I ⊗ f (M ) ⊗ I ⊗. . .⊗I . For later use, we add rules about the exponential function. 30

Functions with other smoothness properties can be considered too. Compare §14.1 in [138].

t u

178

4 Banach Tensor Spaces

Lemma 4.169. (a) If A, B ∈ CI×I are commutative matrices (i.e., AB = BA), then exp(A) exp(B) = exp(A + B).

(4.88a)

(b) Let A(j) ∈ KIj ×Ij and A = A(1) ⊗ I ⊗ . . . ⊗I + I ⊗ A(2) ⊗ . . . ⊗ I + . . .

(4.88b)

+ I ⊗ . . . ⊗I ⊗ A(d−1) ⊗ I + I ⊗ . . . ⊗I ⊗ A(d) ∈ KI×I . Then exp(tA) =

d O

exp(tA(j) )

(t ∈ K).

(4.88c)

j=1

Proof. The d terms in (4.88b) are pairwise commutative and, therefore, (4.88a) proves exp(A) =

d Y

exp(A(j) ) for A(j) := I ⊗ . . . ⊗ I ⊗ A(j) ⊗ I ⊗ . . . ⊗ I.

j=1

Lemma 4.168 shows that exp(A(j) ) = I ⊗ . . . ⊗ I ⊗ exp(A(j) ) ⊗ I ⊗ . . . ⊗ I. Nd (j) (j) Thanks to (4.79), their product yields by tA(j) , j=1 exp(A ). Replacing A u t we prove (4.88c). Exercise 4.170. Prove sin((A ⊗ I) + (I ⊗ B)) = sin(A) ⊗ cos(B) + cos(A) ⊗ sin(B) and the analogue for cos ((A ⊗ I) + (I ⊗ B)). Finally, we mention quite a different kind of a function application to a tensor. Let v ∈ KI with I = I1 × . . . × Id . Then the entry-wise application of a function f : K → K yields f (v) ∈ KI

with f (v)i := f (vi ) for all i ∈ I.

For a matrix v ∈ Kn×m this is a rather unusual operation. It becomes more natural when we consider multivariate functions C(I) defined on I = I1 × . . . × Id (product Nd of intervals). Let ϕ ∈ C(I) = k·k∞ j=1 C(Ij ) . Then the definition f (ϕ) ∈ C(I) with

(f (ϕ)) (x) = f (ϕ(x)) for all x ∈ I

shows that f (ϕ) = f ◦ ϕ is nothing than the usual composition of mappings. If f is a polynomial, we can use the fact that the power function f (x) = xn applied to v coincides with the n-fold Hadamard product (4.82). However, in general, not even for elementary tensors v the result f (v) has an easy representation.

4.7 Symmetric and Antisymmetric Tensor Spaces

179

4.7 Symmetric and Antisymmetric Tensor Spaces 4.7.1 Hilbert Structure Given a Hilbert space (V, h·, ·iV ), define h·, ·i on V by the induced scalar product (4.68). We recall the set P of permutations and the projections PS , PA (cf. §3.5.1): PS and PA are orthogonal projections from V onto the symmetric tensor space S and the antisymmetric tensor space A, respectively (cf. Proposition 3.76). As a consequence, e.g., the identities hPA (u), PA (v)i = hPA (u), vi = hu, PA (v)i ,

(4.89)

hPA(u), APA(v)i = hPA(u), PA(Av)i = hPA(u), Avi = hu, APA (v)i hold for all u, v ∈ V and symmetric A ∈ L(V, V). By the definition of the induced scalar product (4.68), the scalar product hu, vi of elementary tensors u and v reduces to products of scalar products in V . In A,
Nd Nd (j) (j) . elementary tensors u = have to be replaced with PA j=1 u j=1 u Their scalar product reduces to determinants of scalar products in V as stated by the following L¨owdin rule (cf. L¨owdin [217]): *  d  O + d   O 1 (j) (j) , PA u = v PA det hu(i) , v (j) iV . d! i,j=1,...,d j=1 j=1

(4.90)

For a proof use (4.89) so that the left-hand side in (4.90) is equal to * d O j=1

u

(j)

, PA

O d j=1

v

(j)

+

1 = d!

* d O

(j)

u

j=1

,

X π∈P

sign(π)π

O d

v

(j)

+

j=1

* d + d d D E O Y 1 O (j) X 1X (π(j)) u , sign(π) v = = sign(π) u(j) , v (π(j)) . d! j=1 d! V j=1 j=1 π∈P

π∈P

A comparison with (3.44) yields the result in (4.90). Corollary 4.171. For biorthonormal systems {u(j) } and {v (j) }, i.e., hu(i) , v (j) iV = δij , the right-hand side in (4.90) becomes 1/d! . The systems are in particular biorthonormal if u(j) = v (j) forms an orthonormal system. Let {bi : i ∈ I} be an orthonormal system in V with #I ≥ d. For (i1 , . . . , id ) ∈ I d define the elementary tensor d O e(i1 ,...,id ) := bij . j=1

If (i1 , . . . , id ) contains two identical indices, PA (e(i1 ,...,id ) ) = 0 follows.

180

4 Banach Tensor Spaces

Remark 4.172. For two tuples (i1 , . . . , id ) and (j1 , . . . , jd ) consisting of d different indices, the following identity holds: E D PA (e(i1 ,...,id ) ), PA (e(j1 ,...,jd ) )  sign(π)/d! if π(i1 , . . . , id ) = (j1 , . . . , jd ) for some π ∈ P, = otherwise. 0 Proof. hPA (ei ), PA (ej )i = hei , PA (ej )i follows from (4.89). If i = j, there is i π ∈ P with π(i) = j such that PA (ej ) contains a term sign(π) d! e . All other terms i are orthogonal to e . t u An analogue of the L¨owdin rule (4.90) holds for symmetrised elementary tensors. Here the determinant must be exchanged with the permanent:    + * d d   O O 1 PS  fj  , PS  gj  = Perm (hfi , gj i)1≤i,j≤d , d! j=1 j=1 where Perm(M ) =

d X Y

Mi,π(i)

for M ∈ Kd×d .

π∈Pd i=1

The practical disadvantage is the fact that, in general, the computation of the permanent is NP-hard (cf. Valiant [291]).

4.7.2 Banach Spaces and Dual Spaces Let V be a Banach space (possibly, a Hilbert space) with norm k·kV . The norm of the algebraic tensor space Valg = a ⊗d V is denoted by k·k. We require that k·k be invariant with respect to permutations, i.e., kvk = kπ (v)k

for all π ∈ P and v ∈ ⊗da V.

(4.91)

Conclusion 4.173. Assume (4.91). The mapping π : a ⊗d V → a ⊗d V corresponding to π ∈ P, as well as the mappings PS and PA , are bounded by 1. Proof. The bound for π is obvious. For PS use that

1 X 1 X 1

kPS k = π ≤ kπk = d! = 1

d!

d! d! π∈P

π∈P

holds for the operator norm. Similarly for PA . Vk·k := ⊗dk·k V is defined by completion with respect to k·k.

t u

4.7 Symmetric and Antisymmetric Tensor Spaces

181

Lemma 4.174. Assume (4.91). Denote the algebraic symmetric and antisymmetric tensor spaces by Salg (V ) and Aalg (V ). Both are subspaces of Valg . The completion of Salg (V ) and Aalg (V ) with respect to k·k yields subspaces Sk·k (V ) and Ak·k (V ) of Vk·k . An equivalent description of Sk·k (V ) and Ak·k (V ) is  Sk·k (V ) = v ∈ Vk·k : v = π (v) for all π ∈ P ,  Ak·k (V ) = v ∈ Vk·k : v = sign(π) π(v) for all π ∈ P . Proof. (i) By Conclusion 4.173, π is continuous. For any sequence vn ∈ Salg (V ) with vn → v ∈ Vk·k the property vn = π (vn ) is inherited by v ∈ Sk·k (V ). (ii) Conversely, let v ∈ Vk·k with v = π (v) for all π ∈ P . This is equivalent to v = PS (v). Let vn → v for some vn ∈ Valg and construct un := PS (vn ) ∈ Salg (V ). Continuity of PS (cf. Conclusion 4.173) implies that u := lim un = PS (lim vn ) = PS (v) ∈ Vk·k ; hence v = u lies in the completion Sk·k (V ) of Salg (V ). Analogously for the u t space Ak·k (V ). Any dual form ϕ ∈ a ⊗d V 0 is also a dual form on the subspaces Salg (V ) and Nd Aalg (V ). For π ∈ P let π 0 be the dual mapping; i.e., π 0 ( j=1 ϕj ) ∈ a ⊗d V 0 acts
Nd Nd Nd as π 0 j=1 ϕj (v) = ϕ (π(v)). We conclude that π 0 j=1 ϕj = j=1 ϕπ(j) 0 and that all ϕ ∈ a ⊗d V 0 with PS ϕ = 0 represent the zero mapping on Salg (V ). d 0 0 . A comparison with Thus, a ⊗ V reduces to the quotient space a ⊗d V 0 / ker PS d 0 0 (3.41) shows that a ⊗ V / ker PS can be viewed as the symmetric tensor space ∼ a ⊗d V 0 / ker P 0 . Aalg (V 0 ) derived from V 0 . Similarly, Aalg (V 0 ) = A The same statements hold for the continuous functionals:   ∼ ⊗d ∗ V ∗ / ker P ∗ and Sk·k∗ (V ∗ ) = S k·k   ∗ d ∗ Ak·k∗ (V ) ∼ = ⊗k·k∗ V / ker PA∗ , ∗ where by the previous considerations ker PS and ker PA∗ are closed subspaces. The next remark states that in the Hilbert case the definition of the injective norm on S(V ) may be formulated with symmetric tensors ⊗d ϕ of functionals instead Nd of j=1 ϕj . More general results including those about the projective norm can be found in Floret [108]. The original proof of Banach [19] refers to the Hilbert space V = L2 ([0, 1]).
⊗d ϕ (v) Remark 4.175. Let V be a Hilbert space. Then kvk∨ = sup ∗ ∗ ϕ∈V ,kϕkV =1 holds for all symmetric tensors v ∈ S(V ).

Chapter 5

General Techniques

Abstract In this chapter, isomorphisms between the tensor space of order d and vector spaces or other tensor spaces are considered. The vectorisation from Section 5.1 ignores the tensor structure and treats the tensor space as a usual vector space. In finite-dimensional implementations this means that multivariate arrays are organised as linear arrays. After vectorisation, linear operations between tensor spaces become matrices expressed by Kronecker products (cf. §5.1.2). While vectorisation ignores the tensor structure completely, matricisation keeps one of the tensor products and leads to a tensor space of order two (cf. Section 5.2). In the finite-dimensional case, this space is isomorphic to a matrix space. The interpretation as matrix allows to formulate typical matrix properties like the rank leading to the j-rank for a direction j and the α-rank for a subset α of the directions 1, . . . , d. In the finite-dimensional or Hilbert case, the singular-value decomposition can be applied to the matricised tensor. In Section 5.3, the tensorisation is introduced, which maps a vector space (usually without any tensor structure) into an isomorphic tensor space. The artificially constructed tensor structure allows interesting applications. While Section 5.3 gives an introduction into this subject, details about tensorisation will follow in Chapter 14.

5.1 Vectorisation 5.1.1 Tensors as Vectors In program languages, matrices or multi-dimensional arrays are mapped internally into a linear array (vector) containing the entries in a lexicographical ordering. Note that the ordering is not uniquely determined (there are program languages using different lexicographical orderings). Without additional data, it is impossible to restore a matrix or even its format from the vector (e.g., a 4×4 and a 2×8 matrix yield the same linear array). This fact expresses that structural data are omitted. © Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_5

183

184

5 General Techniques

Vectorisation is implicitly expressed by the notation (1.5) for Kronecker matrices. Applying this notation to n × 1 matrices which are regarded as (column) vectors, (1.5) becomes       a1 b a1 b1  a2 b   a2   b2  n n·m for a =   ∈ K and b =   ∈ Km . a⊗b= ∈K .. . . . .. .. Hence the resulting tensor is immediately expressed as vector in Kn·m . Only with this vectorisation, can Kronecker products of matrices A ∈ Kn×n and B ∈ Km×m be interpreted as matrices from Kn·m×n·m . For a mathematical formulation of the vectorisation Q assume Vj = KIj with d nj := #Ij < ∞. Choose any index set J with #J = j=1 nj together with a Nd d bijection φ : ×j=1 Ij → J. This defines the isomorphism Φ : V = j=1 Vj → KJ between the tensor space V and the vector space KJ . Tensor entries v[i1 , . . . , id ] are mapped into vector entries v[φ(i1 , . . . , id )] and vice versa. Note that Φ is a vector space isomorphism in the sense of §3.2.5. In the case of a linear system with n equations and unknowns, we are used to dealing with vectors x, b and a matrix M :
(5.1a) Mx = b M ∈ KJ×J , x, b ∈ KJ . In particular, LU and Cholesky decompositions require an ordered index set J, e.g., J = {1, . . . , N } with N := #J. Matrix equations are an example of systems described differently, e.g., the Lyapunov matrix equation is AX + XA> = B,

(5.1b)

where matrices A, B ∈ KI×I are given and the solution X ∈ KI×I is sought. Let n := #I. The number of unknown entries Xij is n2 . Furthermore, Eq. (5.1b) is linear in all unknowns Xij ; i.e., (5.1b) turns out to represent a linear system of n2 equations for n2 unknowns. Lemma 5.1 from below allows us to translate the Lyapunov equation (5.1b) into Ax = b, where x, b ∈ V := KI ⊗ KI are tensors and A ∈ L(V, V) is the following Kronecker product: A = A ⊗ I + I ⊗ A ∈ L(V, V)

(5.1c)

(cf. [27]). Using the vectorisation isomorphism Φ : V → KJ from above, we obtain the linear system (5.1a) with M = Φ A Φ−1 , x = Φ x, and b = Φ b. Lemma 5.1. The matrices U, V ∈ KI×I define Kronecker products U = U ⊗ I, V = I ⊗ V, W = U ⊗ V ∈ L(KI ⊗ KI , KI ⊗ KI ). The products Ux, Vx, Wx correspond to U X, XV > , and U XV > , where X ∈ KI×I is the matrix interpretation of the tensor x ∈ KI ⊗ KI .

185

5.1 Vectorisation

P Proof. W := U XV > has the matrix coefficients Wi,j = k,`∈I Ui,k Xk,` Vj,` . The Kronecker matrix W = U ⊗V has the entries W(i,j),(k,`) = Ui,k Vj,` . Hence, (Wx)(i,j) =

X

W(i,j),(k,`) x(k,`) = Wi,j .

(k,`)∈I×I

u t

The special cases U = I or V = I yield the first two statements.

From (5.1c) we easily conclude that a positive-definite matrix A in (5.1b) leads to a positive-definite matrix M and, therefore, enables a Cholesky decomposition.

5.1.2 Kronecker Tensors As already mentioned in the previous section, the interpretation of a Kronecker tensor product as a matrix is based on vectorisation. However, there is a second possibility for vectorisation. Matrices, which may be seen as tensors of order two, can be mapped isomorphically into vectors. This will be done by the mappings φj from below. Let Ij and Jj be the index sets characterising the matrix space Mj := KIj ×Jj . An isomorphic vector space is Vj := KKj with Kj = Ij × Jj . The following isomorphism φj describes the vectorisation: φj : Mj → Vj ,

(j)

A(j) = A`,m


`∈Ij ,m∈Jj

(j)
i∈Kj

7→ a(j) := φj (A(j) ) = ai

.

Nd d We identify M := j=1 Mj with the matrix space KI×J , where I := ×j=1 Ij N d d and J := ×j=1 Jj , while V := j=1 Vj is identified with the vector space d

d

KK

with K :=

× j=1

Kj =

× (I ×J ) . j

j

j=1

P Note that the matrix-vector multiplication y = Mx is written as yi = j∈J Mij xj Nd Nd (j) (j) ∈ M and a = ∈V for i ∈ I. Elementary tensors A = j=1 A j=1 a have the entries d d Y Y (j) (j) A[(`1 , . . . , `d ) , (m1 , . . . , md )] = A`j ,mj and a[i1 . . . , id ] = aij . j=1

j=1


Nd Define a(j) by φj (A(j) ). Then a = j=1 φj (A) holds and gives rise to the following definition: d O Φ= φj : A ∈ M 7→ a = Φ(A) ∈ V with j=1

a[(`1 , m1 ) ,..., (`d , md )] = A[(`1 ,..., `d ) , (m1 ,..., md )] for (`j , mj ) ∈ Kj .

186

5 General Techniques

Φ can be regarded as a vectorisation of the Kronecker matrix space M. For d ≥ 3, we have the clear distinction that M is a matrix space, whereas V is a tensor space of order d ≥ 3. For d = 2, however, V can also be viewed as a matrix space (cf. Van Loan–Pitsianis [292]). Remark 5.2. Suppose that d = 2. (a) The matrix A with entries A[(`1 , `2 ) , (m1 , m2 )] is mapped by Φ into a with entries a[(`1 , m1 ) , (`2 , m2 )]. Since d = 2, the tensor a can again be viewed as a matrix from KK1 ×K2 . Note that, in general, a ∈ KK1 ×K2 and A ∈ KI×J have different shape. However, #(K1 × K2 ) = #(I × J) holds, and A and a have the same Frobenius norm; i.e., Φ is also isometric (cf. Remark 2.9). (b) The singular-value decomposition of A in the sense of Lemma can be P3.19 r applied as follows. Apply Lemma 3.19 to a = Φ(A) resulting in a = i=1 σi xi⊗yi (xi ∈ V1 , yi ∈ V2 ). Then application of Φ−1 yields A=

r X

σi Xi ⊗ Yi

−1 with Xi = φ−1 1 (xi ), Yi = φ2 (yi ).

(5.2)

i=1

The orthonormality of the singular vectors {xi }, {yi } translates into orthonormality of the matrices {Xi }, {Yi } with respect to the Frobenius norm (2.9). The Kronecker singular-value decomposition (5.2) doesPnot coincide with the r standard singular-value decomposition of the matrix A = i=1 σiA ui ⊗ viT , since Xi ∈ KK1 and ui ∈ KI as well as Yi P ∈ KK2 and vi ∈ KJ are of different size r T A (in general #K1 6= #I). The terms in i=1 σi ui ⊗ vi have the matrix rank 1, whereas the terms in (5.2) have tensor rank 1. Ps (c) The decomposition (5.2) can be used for truncation. As = i=1 σi Xi ⊗ Yi (0 ≤ s ≤ r) is the best approximation of A with respect to the Frobenius norm. As an illustration of Part (a), consider the identity matrix A = I for the index sets I1 = J1 = {1, 2} and I2 = J2 = {a, b, c}. The matrices A and a = Φ(A) are given below together with the indices for the rows and columns (only nonzero entries are indicated): 1a 1b 1c 2a 2b 2c 1a 1 1 1b 1 A = 1c 2a 1 2b 1 2c 1

7→ Φ

aa ab ac ba bb bc ca cb cc 11 1 1 1 a = 12 21 22 1 1 1

5.2 Matricisation

187

5.2 Matricisation Synonyms for matricisation are matrix unfolding or flattening. The term ‘matricisation’ corresponds to the fact that tensors are turned into matrices (at least in the finite-dimensional case). We recall the two types of isomorphisms discussed in §3.2.5. The strongest form is the tensor space isomorphism which preserves the tensor structure (cf. Definition 3.29). The weakest form is the vector space isomorphism which identifies all tensor spaces V and W having the same dimension not regarding the tensor strucNd ture. An intermediate form groups the d spaces Vj from V = j=1 Vj such that N5 the order is reduced. For instance, j=1 Vj is isomorphic to the rearrangement (V1 ⊗ V5 )⊗(V2 ⊗ V3 )⊗V4 , which is a tensor space of order dnew = 3. Matricisation is characterised by dnew = 2. An example is (V1 ⊗ V5 ) ⊗ (V2 ⊗ V3 ⊗ V4 ) . Since tensor spaces of order two are close to matrix spaces, matricisation tries to exploit all features of matrices. In this setting, vectorisation corresponds to dnew = 1. Below we use the sign ⊗ without subscripts ⊗a or ⊗k·k since both cases are allowed and need not be distinguished.

5.2.1 General Case To get dnew = 2, we must divide the whole index set {1, . . . , d} into two (disjoint) subsets. For a systematic approach we introduce the set D = {1, . . . , d}

(5.3a)

∅ $ α $ D.

(5.3b)

and consider proper subsets

The complement of α is denoted by αc := D\α . We define the (partial) tensor spaces O Vj Vα =

for α ⊂ D,

(5.3c)

(5.3d)

j∈α

which include the special cases V∅ = K for α = ∅ and VD = V for α = D. For singletons we have the synonymous notation V{j} = Vj

(j ∈ D).

Instead of V{j}c for the complement {j}c , we already introduced the symbol

188

5 General Techniques

O

V[j] =

Vk

k ∈D\{j}

(cf. (3.17a)). Depending on the context, the spaces Vα and V[j] may be algebraic or topological tensor spaces. For the norm of the partial tensor space Vα in the case of ∅ $ α $ D, we refer to §4.3.2. Below, we introduce the isomorphism Mα from V onto the binary tensor space Vα ⊗ Vαc : O Vj ∼ (5.4) V= = Vα ⊗ Vαc . j∈D

Often, α is a singleton {j}, i.e., Vα = V{j} = Vj . In this case (5.4) becomes V∼ = Vj ⊗ V[j] and the isomorphism is denoted by Mj . Definition 5.3 (Mα , Mj ). The matricisation Mα with α from1 (5.3b) is the isomorphism N Mα : k∈D Vk → Vα ⊗ Vαc N (k) (αc ) N (k) N c (k) v . 7→ v(α) ⊗ v(α ) with v(α) = v , v = k∈D v k∈αc

k∈α

In particular, for j ∈ D, Mj is the isomorphism Mj :

N N

k∈D

Vk → Vj ⊗ V[j]

k∈D

v (k) 7→ v (j) ⊗ v[j]

with v[j] =

N

k∈D\{j}

v (k) .

Next, we check how Mα (v) behaves when we apply an elementary Kronecker Nd (j) product : V → W to v. This includes the case of a tensor space j=1 A isomorphism Φ : V → W (cf. Definition 3.29). Nd Nd Nd Remark 5.4. For V = j=1 Vj and W = j=1 Wj let A := j=1 A(j) : V → W N be an elementary Kronecker product. For α in (5.3b) set A(α) := j∈α A(j) and N c A(α ) := j∈αc A(j) . Then   c Mα (Av) = A(α) ⊗ A(α ) Mα (v)

for all v ∈ V. c

If A(j) : Vj → Wj are isomorphisms, A(α) ⊗ A(α between Vα ⊗ Vαc and Wα ⊗ Wαc .

)

describes the isomorphism

By condition (5.3b) we have avoided the empty set α = ∅ and α = D (⇒ αc = ∅). Since the empty tensor product is interpreted as the field K, one may view MD : V → V ⊗ K as the vectorisation (column vector) and M∅ : V → K ⊗ V as mapping into a row vector. 1

5.2 Matricisation

189

5.2.2 Finite-Dimensional Case 5.2.2.1 Example For finite dimensions, the binary tensor space Vα ⊗ Vαc resulting from the matricisation may be interpreted as matrix space (cf. §3.2.3). If, e.g., Vj = KIj , then Mα maps into2 KIα ×Iαc, where Iα = ×k∈α Ik and Iαc = ×k∈αc Ik . Hence a tensor v with entries v[(iκ )κ∈D ] becomes a matrix M = Mα (v) with entries M [(iκ )κ∈α , (iλ )λ∈αc ]. To demonstrate the matricisations, we illustrate all Mα for a small example. Example 5.5. Below, all matricisations are given for the tensor v ∈ KI1 ⊗ KI2 ⊗ KI3 ⊗ KI4

with I1 = I2 = I3 = I4 = {1, 2}.

The matrix M1 (v) belongs to KI1 ×J with J = I2 × I3 × I4 . For the sake of the following notation we introduce the lexicographical ordering of the triples from I2 × I3 × I4 : (1, 1, 1) , (1, 1, 2) , (1, 2, 1) , . . . , (2, 2, 2). Under these assumptions, KI1 ×J becomes K2×8 : 3   v1111 v1112 v1121 v1122 v1211 v1212 v1221 v1222 . M1 (v) = v2111 v2112 v2121 v2122 v2211 v2212 v2221 v2222 M2 (v) belongs to KI2 ×J with J = I1 ×I3 ×I4 . Together with the lexicographical ordering in J we get   v1111 v1112 v1121 v1122 v2111 v2112 v2121 v2122 M2 (v) = . v1211 v1212 v1221 v1222 v2211 v2212 v2221 v2222 Similarly,  M3 (v) =  M4 (v) =

v1111 v1112 v1211 v1212 v2111 v2112 v2211 v2212 v1121 v1122 v1221 v1222 v2121 v2122 v2221 v2222



v1111 v1121 v1211 v1221 v2111 v2121 v2211 v2221 v1112 v1122 v1212 v1222 v2112 v2122 v2212 v2222



, .

Next, we consider α = {1, 2}. M{1,2} (v) belongs to KI×J with I = I1 × I2 and J = I3 × I4 . Lexicographical ordering of I and J yields a matrix from K4×4 :   v1111 v1112 v1121 v1122  v1211 v1212 v1221 v1222   . M{1,2} (v) =  v2111 v2112 v2121 v2122  v2211 v2212 v2221 v2222 This means that Mα is replaced by Ξ 0−1 ◦ Mα , where Ξ 0 is the isomorphism from the matrix space KIα ×Iαc onto the tensor space Vα ⊗ Vαc (see Proposition 3.16b). For simplicity, we write Mα instead of Ξ 0−1 ◦ Mα . 3 Bold face indices correspond to the row numbers. 2

190

5 General Techniques

Similarly, 

v1111  v1121 M{1,3} (v) =   v2111 v2121  v1111  v1112  M{1,4} (v) =  v2111 v2112

v1112 v1122 v2112 v2122

v1211 v1221 v2211 v2221

v1121 v1122 v2121 v2122

v1211 v1212 v2211 v2212

 v1212 v1222  , v2212  v2222  v1221 v1222  . v2221  v2222

The other Mα (v) are transposed versions of the already described matrices: M{2,3} = MT {1,4} ,

M{2,4} = MT {1,3} ,

M{3,4} = MT {1,2} ,

T T T M{1,2,3} = MT 4 , M{1,2,4} = M3 , M{1,3,4} = M2 , M{2,3,4} = M2 .

5.2.2.2 Invariant Properties and α-Rank The interpretation of tensors v as matrices M enables us (i) to transfer the matrix terminology from M to v, (ii) to apply all matrix techniques to M. In Remark 3.17 we considered the isomorphism M(v) = M and stated that multiplication of a tensor
by a Kronecker product A⊗B has the isomorphic expression M (A ⊗ B) v = A M(v) B T = AM B T . More generally, the following statement holds, which is the matrix interpretation of Remark 5.4. N N N Lemma 5.6. Let v ∈ V = j∈D KIj and A = j∈D A(j) ∈ j∈D L(KIj , KJj ). N The product Av ∈ W = j∈D KJj satisfies O O c c A(j), A(α ) = Mα (Av) = A(α) Mα (v)A(α )T with A(α) = A(j) . (5.5) j∈αc

j∈α

In particular, if all A Mα (v) coincide.

(j)

are regular matrices, the matrix ranks of Mα (Av) and

Proof. Define the index sets Iα := ×j∈α Ij and Iαc := ×j∈αc Ij , and similarly Jα and Jαc . The indices i ∈ I := ×j∈D Ij are written as (i0 , i00 ) with i0 ∈ Iα and i00 ∈ Iαc . Similarly for j = (j0 , j00 ) ∈ J. Note that •j0 ,j00 denotes a matrix entry, while •j = •(j0 ,j00 ) is a tensor entry. The identity X X X A(j0,j00),i vi = Mα (Av)j0,j00 = (Av)(j0,j00) = A(j0,j00),(i0,i00) v(i0,i00) i0 ∈Iα i00 ∈Iαc

i∈I

=

X X i0 ∈Iα i00 ∈Iαc

proves (5.5).

(α) (αc ) Aj0 ,i0 v(i0 ,i00) Aj00 ,i00

=

X X

(α)

(αc )

Aj0 ,i0 Mα (v)i0 ,i00 Aj00 ,i00

i0 ∈Iα i00 ∈Iαc

t u

5.2 Matricisation

191

According to item (i), we may define the matrix rank of Mα (v) as a property of v. By Lemma 5.6, the rank of Mα (v) is invariant under tensor space isomorphisms. Definition 5.7 (rankα ). For any4 α ⊂ D in (5.3b) and all j ∈ D we define rankα (v) := rank (Mα (v)) ,

(5.6a)

rankj (v) := rank{j} (v) = rank (Mj (v)) .

(5.6b)

In 1927, Hitchcock [164, p. 170] has introduced rankj (v) as ‘the rank on the j th index’. Even rankα (v) is already defined by him as the ‘α-plex rank’. We shall call it the j-rank or the α-rank, respectively. Further properties of the α-rank will follow in Lemma 6.20 and Corollary 6.21. The tensor rank defined in (3.20) is not directly related to the family of ranks {rankα (v) : ∅ $ α $ D} in (5.6a) or the ranks {rankj (v) : j ∈ D} in (5.6b). Later, in Remark 6.24, we shall prove rankα (v) ≤ rank(v) for all α ⊂ D. N Remark 5.8. A tensor v ∈ j∈D Vj with random entries satisfies rankα (v) = Q Q min j∈α dim(Vj ), j∈αc dim(Vj ) with probability one for all α ⊂ D. u t

Proof. Apply Remark 2.5 to Mα (v).

Nd Remark 5.9. Let k·k be the Euclidean norm in V = j=1 KIj (cf. Example 4.149), while k·kF is the Frobenius norm for matrices (cf. (2.9). Then the norms coincide: kvk = kMα (v)kF

for all ∅ $ α $ D and all v ∈ V. t u

Proof. Use Remark 2.9.

5.2.2.3 Singular-Value Decomposition Later, singular values of Mα (v) will be important. The next proposition compares the singular values of Mα (v) and Mα (Av). N c Proposition 5.10. Let v ∈ V = κ∈D KIκ . The Kronecker matrix A = A(α) ⊗A(α ) c is assumed to be composed of A(α) ∈ L(Vα , Vα ) and A(α ) ∈ L(Vαc , Vαc ) with c c the properties A(α)HA(α) ≤ I and A(α )HA(α ) ≤ I. Then the singular values fulfil for all k ∈ N.

σk (Mα (Av)) ≤ σk (Mα (v)) c

Proof. Combine Mα (Av) = A(α) Mα (v)A(α 4

)T

in (5.5) and Lemma 2.31c. t u

Usually, we avoid α = ∅ and α = D. Formally, the definition of M∅ , Md from Footnote 1 yields rank∅ (v) = rankd (v) = 1 for v 6= 0 and rank∅ (0) = rankd (0) = 0, otherwise.

192

5 General Techniques

Corollary 5.11. The assumptions of Proposition 5.10 are in particular satisfied if c A(α) and A(α ) are orthogonal projections. Remark 5.12. The reduced singular-value decomposition Mα (v) = UΣ V T =

rα X

(α)

σi ui viT

i=1 (α)

σi


> 0, ui , vi columns of U and V, rα = rankα (v) translates5 into v=

rα X

(α)

σi

(ui ∈ Vα , vi ∈ Vαc ).

ui ⊗ vi

i=1

Here ui and vi are the isomorphic vector interpretations of the tensors ui , vi . Remark 5.13. (a) In the case of matrices (i.e., D = {1, 2}), the ranks are equal:6 (2) (1) rank1 (v) = rank2 (v). Further, the singular values of Mα (v) coincide: σi = σi . (b) If d ≥ 3, the values rankj (v) (j ∈ D) may not coincide. Furthermore, the (j) singular values σi of Mj (v) may be different; however, the following quantity is invariant: rankj (v) 

X

(j)

σi

2

2

= kvk2

for all j ∈ D

(k·k2 in (4.149)).

i=1

Proof. (i) Let M(v) = M be the isomorphism (1.3) between v ∈ KI1 ⊗ KI2 and the matrix M ∈ KI1 ×I2 . Then M1 (v) = M , while M2 (v) = M T . Since M and M T have identical rank and identical singular values, Part (a) follows. (ii) Consider the tensor v = a1 ⊗ a2 ⊗ a3 + a1 ⊗ b2 ⊗ b3 ∈ with ai =

1 0




(i = 1, 2, 3) and bi =

0 1




O3 j=1

K2

(i = 2, 3). We have


M1 (v) = a1 ⊗ c ∈ K2 ⊗ K2 ⊗ K2 ∼ = K2 ⊗ K4
∼ (1 0 0 1) ∈ K4 (i.e., M1 (v) ∼ with c := a2 ⊗ a3 + b2 ⊗ b3 = = 10 00 00 10 has rank 1), whereas    ∼ d := (1 0 0 0) , d d := a1 ⊗ a3 = M2 (v) = a2 ⊗ d + b2 ⊗ e ∼ with = ∼ e := a1 ⊗ b3 = e := (0 1 0 0) e Let the tensors ui and vi be vectorised into ui and vi (cf. §5.1). Then M(ui ⊗vi ) = ui viT is used (cf. (1.3)). 6 The true generalisation of this property for general d is described by (6.13a). 5

5.2 Matricisation

193

has two linearly independent rows and, therefore, rank 2. (iii) Since the rank is also the number of positive singular values, they must be different for the given example. The sum of the squared singular values is the 2 squared Frobenius norm of the corresponding matrix: kMj (v)kF . Since the matrix entries of Mj (v) are only a permutation of the entries of v (cf. Example 5.5), 2 t u the sum of their squares is equal to kvk2 . (j)

We may pose the opposite question: Given any real values σi ≥ 0 subject (j) to (4.149), are there tensors possessing these σi as singular values? The answer is negative (cf. Hackbusch–Uschmajew [154], Hackbusch–Kressner–Uschmajew [151]).

5.2.2.4 Matricisation of Kronecker Matrices So far, the tensor v involves a single index tuple i = (ij )j∈D . In the case of a tensor space formed from matrix spaces Xj = CIj ×Jj = L(Uj , Vj ), the Kronecker Nd matrix M ∈ X = j=1 Xj has two multi-indices: M[i, j], i ∈ I, j ∈ J. The matricisation Mα : X = L(U, V) → Xα ⊗ Xαc = L(Uα , Vα ) ⊗ L(Uαc , Vαc ) yields Mα (M) with entries for [(iα , jα ) , (iαc , jαc )]

with iα ∈ Iα , jα ∈ Jα , iαc ∈ Iαc , jαc ∈ Jαc .

The rank of Mα (M) is rankα (M) = dim(span{M[(•, •), (iαc , jαc )] ∈ Xα : iαc ∈ Iαc , jαc ∈ Jαc }). We emphasise that the α-rank of M is completely different from the matrix rank of the Kronecker matrix M. The multiplication rule from Remark 5.4 for Mα (Av) has a matrix equivalent: Mα (AB) = Mα (A) · Mα (B) for A ∈ Xα ⊗ Xαc , B ∈ Yα ⊗ Yαc , where Yj = L(Vj , Wj ) and Yα = L(Vα , Wα ). On the left-hand side, AB is the matrix product between X = L(U, V) and Y = L(V, W) (Kronecker matrices of tensor order d), while on the right-hand side the operation · is the multiplication between Kronecker matrices of tensor order 2. In tensor notation Mα (Av) = Mα (A) · Mα (v) for A ∈ Xα ⊗ Xαc , v ∈ Vα ⊗ Vαc holds, while (5.5) is the matrix interpretation.

(5.7)

194

5 General Techniques

5.2.2.5 Infinite-Dimensional Spaces For infinite-dimensional vector spaces Vj , these quantities generalise as follows. In the finite-dimensional case, rank(Mα (v)) is equal to the dimension of the range of Mα (v) (cf. Remark 2.1), which is given by range(Mα (v)) = {Mα (v)z : z ∈ Vαc }. P Since Mα (v) ∈ Vα ⊗Vαc has the form ν xν ⊗yν (cf. PDefinition 5.3), the matrixvector multiplication Mα (v)z can be considered as ν z(yν ) · xν ∈ Vα , where 0 z ∈ Vα c is regarded as an element of the dual vector space (for dim(Vαc ) < ∞, 0 Vαc may be identified with Vαc ). The mapping X X z(yν ) · xν xν ⊗ yν 7→ ν

ν

is denoted by id ⊗ z. Then the matrix-vector multiplication Mα (v)z may be rewritten as (id ⊗ z) Mα (v). This leads to the notation 0 rankα (v) := dim {(id ⊗ z) Mα (v) : z ∈ Vα c} . 0 ∗ The transition to the dual space Vα c (or Vαc ) is necessary since, in the infinitedimensional case, Mα (v) cannot be interpreted as a mapping from Vαc into Vα 0 0 but as a mapping from Vα c into Vα . The set {(id ⊗ z) Mα (v) : z ∈ Vαc } on min (v), so that the right-hand side will be defined in §6 as the minimal subspace Uα min (v)) rankα (v) := dim(Uα

(5.8)

is the generalisation to infinite-dimensional (algebraic) vector spaces, as well as to Banach spaces. ∗ An identification of Vα c with Vαc becomes possible if all Vj are Hilbert spaces. This case is discussed next.

5.2.3 Hilbert Structure N Next, we consider a pre-Hilbert space V = a j∈D Vj and the left-sided singularvalue decomposition problem of Mα (v) for some v ∈ V. The singular-value decomposition is r X (α) σi ui ⊗ vi , (5.9) Mα (v) = i=1

where

5.2 Matricisation

195

ui ∈ Vα :=

O a

vi ∈ Vαc :=

Vj ,

O

Vj

a j∈αc

j∈α

(α)

(α)

are two orthonormal families {ui }, {vi }, and σ1 ≥ . . . ≥ σr > 0 . The (α) left-sided singular-value decomposition problem asks for {ui } and {σi }. If (α) Ij Vj = K allows us to interpret Mα (v) as a matrix, the data {ui }, {σi } are determined by LSVD(Iα , Iαc , r, Mα (v), U, Σ) (cf. (2.29)). We recall that its computation may use the diagonalisation Mα (v)Mα (v)H =

r X (α) (σi )2 ui uH i . i=1

In the infinite-dimensional setting, the latter expression can be expressed by the partial scalar product7 in §4.5.5: hMα (v), Mα (v)iαc = hv, viαc ∈ Vα ⊗ Vα . Pr (α) Assuming a singular-value decomposition Mα (v) = i=1 σi ui ⊗ vi (possibly with r = ∞, cf. (4.16)), the partial scalar product yields the diagonalisation hMα (v), Mα (v)iαc =

X r

(α)

σi

ui ⊗ vi ,

i=1

=

r r X X

(α) (α)

σi σj hvi , vj iαc ui ⊗ uj = | {z } i=1 j=1 =δij

r X

j=1 r X

(α)

σj

 uj ⊗ vj αc

(α)

(σi )2 ui ⊗ ui .

i=1

We summarise. N Lemma 5.14. Let V = a j∈D Vj ∼ = Vα ⊗a Vαc with Vα , Vαc as in (5.9). (α) (α) The left singular vectors ui ∈ Vα and the singular values σi are obtainable from the diagonalisation hMα (v), Mα (v)iαc =

r X

(α)
2

σi

(α)

ui

(α)

⊗ ui .

(5.10)

i=1 (α)

Analogously, the right singular vectors vi are obtainable from the diagonalisation hMα (v), Mα (v)iα =

r X

(α)

∈ Vαc and the singular values σi

(α)
2

σi

(α)

vi

(α)

⊗ vi .

i=1 c

If the image Mα (v) = v(α) ⊗ v(α ) under the isomorphism Mα : V → V(α) ⊗ V(α an elementary tensors, the partial scalar product is defined by D E c c hMα (v), Mα (v)iαc = v(α ) , v(α ) · v(α) ⊗ v(α) ∈ Vα ⊗ Vα . 7

αc

The expression hv, viαc has the same meaning.

c

)

is

196

5 General Techniques

Corollary 4.161 allows us to determine the partial scalar product h·, ·iβ from h·, ·iα if α $ β ⊂ D. As a consequence, hMα (v), Mα (v)iαc can be obtained from hMβ (v), Mβ (v)iβ c :   hMα (v), Mα (v)iαc = Cβ\α hMβ (v), Mβ (v)iβ c

(5.11)

with the contraction Cβ\α from Definition 4.160. In order to apply Cβ\α , the tensor hMβ (v), Mβ (v)iβ c ∈ Vβ ⊗ Vβ (α)

is interpreted as a tensor in Vα ⊗ Vβ\α ⊗ Vα ⊗ Vβ\α . Let {bi a basis of Vα . We obtain the following result.

: 1 ≤ i ≤ rα } be

Theorem 5.15. Assume ∅ $ α1 $ α ⊂ D and rα X

hMα (v), Mα (v)iαc =

(α)

(α)

eij bi

(α)

⊗ bj

∈ Vα ⊗ Vα .

(5.12a)

i,j=1 (α) Set α2 := α\α1 . Then α = α1 ∪˙ α2 holds. Consider bi ∈ Vα as elements of Vα1 ⊗ Vα2 with the representation (α) bi

=

rα 1 rα 2 X X

(α1 ) 2) c(i) ⊗ b(α . νµ bν µ

(5.12b)

ν=1 µ=1 (i)
We introduce the matrices Ci := cνµ ∈ Krα1 ×rα2 for 1 ≤ i ≤ rα . Then

hMαk (v), Mαk (v)iαc =

rα X

(α )

(αk )

eij k bi

k

(αk )

⊗ bj

∈ Vαk ⊗Vαk

(k = 1, 2)

i,j=1

  (α ) holds with coefficient matrices Eαk = eij k ∈ Krαk ×rαk defined by Eα1 =

rα X

(α)

H eij Ci GT α2 Cj ,

Eα2 =

i,j=1

rα X

(α)

eij CiT GT α1 Cj ,

(5.12c)

i,j=1

 

(α ) (α )  (α ) (α ) where Gαk = gνµk is the Gram matrix with entries gνµk = bµ k , bν k .

Proof. The insertion of (5.12b) in (5.12a) yields hMα (v), Mα (v)iαc =

rα X i,j=1

(α)

eij

X

(j)

(α1 )

(α1 ) 2) c(i) ⊗ b(α ⊗ bσ νµ cστ bν µ

ν,µ,σ,τ

Applying (5.11) with β\α replaced by α2 = α\α1 yields

(α2 )

⊗ bτ

.

5.2 Matricisation

197

hMα1 (v), Mα1 (v)iα1c =

rα X

(α)

X

eij

i,j=1

 (α1 ) (j) (α2 ) (α ) 2) c(i) , b(α bν ⊗ bσ 1 νµ cστ bµ τ

ν,µ,σ,τ

proving 1) e(α νσ =

rα X

(α)

eij

i.e., Eα1 =

i,j=1

(j)

(α2 ) c(i) νµ cστ gτ µ ,

µ,τ

i,j=1

P

X

(α)

H eij Ci GT α2 Cj . The case of Eα2 is analogous.

t u

(α)

Corollary 5.16. Assume the finite-dimensional case with orthonormal basis {bi }. (α) (α) Form the matrix Bα = [b1 b2 · · · ]. Then Mα (v)Mα (v)H = Bα Eα BH α (α)

(α)

holds in the matrix interpretation. In particular, (σi )2 = λi is valid for the (α) (α) singular values σi of Mα (v) and the eigenvalues λi of Eα . In the following example, we apply the matricisation to a topological Hilbert N tensor space V = k·k j∈D Vj with induced scalar product. Example 5.17. Let Vj = L2 (Ij ) with Ij ⊂ R for 1 ≤ j ≤ d. Then L2 (I) = Nd d 2 k·k j=1 Vj holds for I = ×j=1 Ij . Consider a function f ∈ L (I). To obtain the (j)

left singular vectors ui ∈ L2 (Ij ), we form the operator Kj := Mj (f )M∗j (f ) = hMj (f ), Mj (f )i[j] ∈ L (Vj , Vj ) . The application of Kj to g ∈ L2 (Ij ) is given by Z Kj (g)(ξ) = kj (ξ, ξ 0 )g(ξ 0 )dξ 0 with I[j] = Ij

kj (ξ, ξ 0 ) :=

Z

×I , dx k

k6=j

[j]

=

Y

dxk in

k6=j

f (. . . , xj−1 , ξ, xj+1 , . . .) f (. . . , xj−1 , ξ 0 , xj+1 , . . .) dx[j] .

I[j] (j)

The singular vectors ui value problem

(j)

and singular values σi (j)

(j)
2 (j) ui

Kj (ui ) = σi

can be obtained from the eigen(i ∈ N) .

Nd If f ∈ a j=1 Vj is an algebraic tensor, Kj has finite rank and delivers only finitely (j) (j) many singular vectors ui with positive singular values σi .

198

5 General Techniques

5.2.4 Matricisation of a Family of Tensors N Let F = (vi )i∈I be a family of tensors vi ∈ Vd = j∈D Vj . According to Lemma 3.27, the tuple (vi )i∈I may be considered as an element of the tensor space Vd ⊗ KI . If D = {1, . . . , d}, define the extended index set Dex := D ∪ {d + 1} N and the extended tensor space Vex = j∈Dex Vj , where Vd+1 := KI . Using the identification described in Lemma 3.27, we may view F as an element of Vex . This allows us to define Mα (F) for all α ⊂ Dex . For instance, α = D yields X
vi ∈ Vd , ei ∈ KI : i-th unit vector . Md (F) = vi ⊗ ei i∈I

From this representation we conclude the following result about the left-sided singular-value decomposition (cf. (5.10)). P Remark 5.18. hMα (F), Mα (F)iDex\α = hMα (vi ), Mα (vi )iD\α for α ⊂ D. i∈I

5.3 Tensorisation Tensorisation is the opposite of vectorisation: a vector is isomorphically transformed into a tensor, even when the tensor structure is not given beforehand. One example of tensorisation has been presented in §1.2.4.1. There, u and f are grid functions and are usually considered as vectors. Because of the special shape of the grid Gn , the entries of u and f are of the form uijk and fijk (1 ≤ i, j, k ≤ n) and u and f can be regarded as tensors from Kn ⊗ Kn ⊗ Kn . However, the tensorisation may also be rather artificial. Consider, e.g., any vector from KI with I := {0, . . . , n − 1} and assume that n is not a prime so that a factorisation n = n1 n2 (n1 , n2 ≥ 2) exists. Choose index sets J1 := {0,. . . , n1 −1} and J2 := {0, . . . , n2 − 1} and set J := J1 × J2 . Since #J =#I, there is a bijection α : J → I, leading to an isomorphism x ∈ KI ←→ v ∈ KJ = KJ1 ⊗ KJ2

with

(5.13)

v[j1 , j2 ] = x[α(j1 , j2 )] and α(j1 , j2 ) = j2 n2 + j1 . In the latter case, v ∈ KJ is an n1 × n2 matrix or a tensor of order two. Obviously, tensors of higher order can be obtained by exploiting a factorisation n = n1 n2 · . . . · nd (assuming nj ≥ 2 to avoid trivial cases): KI ∼ =

d O j=1

KJj

for #I =

d Y j=1

#Jj .

199

5.3 Tensorisation

An extreme case is KI with the dimension #I = 2d : d ∼ K2 =

Od j=1

K2 .

The advantage of the representation as tensor is the fact that vectors (of length n) rewritten as tensors may require less storage. In the following example, the corresponding tensors are elementary tensors (cf. Khoromskij [185]). Consider x ∈ K{0...,n−1}

with xν = ζ ν

for 0 ≤ ν ≤ n − 1,

(5.14)

where ζ ∈ K is arbitrary. Such vectors appear in . approximations, as well as in Fourier representations. The isomorphism (5.13) based on n = n1 n2 yields the matrix    M = 

ζ0 ζ1 . ..

ζ n1 ζ n1 +1 . ..

ζ n1 −1 ζ 2n1 −1

  0  · · · ζ (n2 −1)n1 ζ   ζ1   · · · ζ (n2 −1)n1 +1      =  ..  ζ 0 ζ n1 · · · ζ (n2 −1)n1 , .. ..   .  . . n2 n1 −1 ζ n1 −1 ··· ζ

which corresponds to the tensor product    x ∈ K{0...,n−1} ←→  

ζ0 ζ1 .. .



ζ n1 −1

ζ0 ζ n1 ...



    ⊗  



ζ (n2 −1)n1

   ∈ K{0...,n1 −1} ⊗ K{0...,n2 −1} . 

The decomposition can be repeated for the first vector, provided that n1 = n0 n00 (n0 , n00 ≥ 2), and yields (after renaming n0 , n00 , n2 by n1 , n2 , n3 )    x ∈ K{0...,n−1} ←→  

ζ0 ζ1 .. . ζ n1 −1



ζ0 ζ n1 .. .



    ⊗  

ζ (n2 −1)n1





    ⊗  

ζ0



ζ n1 n2 ...

  . 

ζ (n3 −1)n1 n2

By induction, this proves the next statement. Remark 5.19. Let n = n1 n2 · · · nd with nj ≥ 2. The vector x in (5.14) corresponds to the elementary tensor v (1) ⊗ . . . ⊗ v (d) with v (j) ∈ K{0...,nj −1} defined by   ζ0 pj j−1   Y   ζ (j) v = with pj := nk .  ..   . k=1 ζ (nj −1)pj

200

5 General Techniques j−1

d

In the case of n = 2 , i.e., nj = 2, pj = 2 is reduced from n to 2d = 2 log2 n.

, and v

(j)

 =

 2j−1 , the data size 1

ζ

Interestingly, this representation does not only save storage but also provides a more stable representation. As an example consider the integral involving the oscillatory function f (t) = exp(−αt) for α = 2πik + β, k ∈ N, β > 0, and g(t) = 1. For large k, the value of the integral Z

1

f (t)g(t)dt = 0

1 − exp(−β) β + 2πik

R1 is small compared with 0 |f (t)g(t)| dt = (1 − exp(−β)) /β. For usual numerical integration we have to expect a cancellation error of the size machine precision multiplied by the amplification factor Z κ := 0

1

Z |f (t)g(t)| dt /

0

1

p 2πk f (t)g(t)dt = 1 + (2πk/β)2 ∼ , β

which becomes large for large k and small β. If we approximate the integral by8 S :=

n−1 1 X ν  ν  f g n ν=0 n n

for n = 2d ,

the floating point errors are amplified by κ from above. Using the tensorised grid Nd Nd functions f = j=1 f (j) and g = j=1 g (j) with f (j) =



1 exp{−(2πik + β)2j−1−d }



and g (j) =

  1 1

according to Remark 5.19, we rewrite9 the sum (scalar product) as Qd (j) (j)  1 ,g : j=1 f n

1 n

hf , gi =

d  1 Y S= 1 + exp(−(2πik + β)2j−1−d ) . n j=1

In this case the amplification factor for the floating point errors is O(d + 1) and does not deteriorate as k → ∞ and β → 0. More details about tensorisation will follow in Chapter 14. 8

The sum S is slightly different from the trapezoidal rule because of the quadrature weights for ν = 0 and n. For β = 1/10, k = 1000, and d = 20 (i.e., n = 220 = 1 048 576), the value of S is 4.56210 -8 − 1.515 i (exact integral value: 2.41010 -10 − 1.51510 -5 i).  10 -5j−1 Q P d −1 ν 9 2 Note that d 1 + x = 2ν=0 x . j=1

Chapter 6

Minimal Subspaces

Abstract The notion of minimal subspaces is closely connected with the representations of tensors, provided these representations can be characterised by (dimensions of) subspaces. A separate description of the theory of minimal subspaces can be found in Falc´o–Hackbusch [96] (see also [97]. The tensor representations discussed in the later Chapters 8, 11, 12 will lead to subsets Tr , Hr , Tρ of a tensor space. The results of this chapter will prove weak closedness of these sets. Another result concerns the question of a best approximation: is the infimum also a minimum? In the positive case, it is guaranteed that the best approximation can be found in the same set. Nd For tensors v ∈ a j=1 Vj we shall define ‘minimal subspaces’ Ujmin (v) ⊂ Vj in Sects. 6.1–6.4. In Section 6.5 we consider weakly convergent sequences vn * v and analyse the connection between Ujmin (vn ) and Ujmin (v). The main result will be presented in Theorem 6.29. While Sections 6.1–6.5 discuss minimal subspaces of Nd algebraic tensors v ∈ a j=1 Vj , Section 6.6 investigates Ujmin (v) for topological Nd tensors v ∈ k·k j=1 Vj . The final Section 6.7 is concerned with intersection spaces.

6.1 Statement of the Problem, Notations Nd Consider an algebraic tensor space V = a j=1 Vj and a fixed tensor v ∈ V. Among the subspaces Uj ⊂ Vj with the property d O v ∈ U := a Uj (6.1) j=1

we are looking for the smallest ones. We must show that minimal subspaces Uj exist and that these minimal subspaces can be obtained simultaneously in (6.1) for all 1 ≤ j ≤ d. Since it will turn out that the minimal subspaces are uniquely determined by v, we denote them by Ujmin (v) ⊂ Vj . © Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_6

201

202

6 Minimal Subspaces

The determination of Ujmin (v) will be given in (6.5) and (6.8). We shall characterise the features of Ujmin (v), e.g., the dimension rj := dim(Ujmin (v)). Furthermore, the properties of Ujmin (v) for varying v are of interest. In particular, we consider Ujmin (vn ) and its dimension for a sequence vn * v. First, in §6.2, we explore the case d = 2. In §6.6 we replace the algebraic tensor space with a Banach tensor space. An obvious advantage of (6.1) is the fact that the subspaces Uj are of finite dimension even when dim(Vj ) = ∞ as stated next. Nd Lemma 6.1. For v ∈ a j=1 Vj there are always finite-dimensional subspaces Uj ⊂ Vj satisfying (6.1). More precisely, dim(Uj ) ≤ rank(v) can be achieved. Proof. By the definition of the algebraic tensor space, v ∈ there is a finite linear combination v=

d n O X

a

Nd

j=1 Vj

vν(j)

means that

(6.2a)

ν=1 j=1 (j)

with some integer n ∈ N0 and certain vectors vν ∈ Vj . Define Uj := span{vν(j) : 1 ≤ ν ≤ n}

for 1 ≤ j ≤ d.

(6.2b)

Nd Then v ∈ U := a j=1 Uj proves (6.1) with subspaces of dimension dim(Uj ) ≤ n. u By the definition of the tensor rank, the smallest n in (6.2a) is n := rank(v). t

6.2 Tensors of Order Two 6.2.1 Existence of Minimal Subspaces First, we consider the matrix case d = 2 and admit any field K. To ensure the existence of minimal subspaces, we need the lattice property (X1 ⊗a X2 ) ∩ (Y1 ⊗a Y2 ) = (X1 ∩ Y1 ) ⊗a (X2 ∩ Y2 ) , which is formulated in the next lemma more generally. Lemma 6.2. Let A be an index set of possibly infinite cardinality. Then   \  \  \ U1,α ⊗a ( U1,α ⊗a U2,α ) = U2,α . α∈A

α∈A

α∈A

(6.3)

6.2 Tensors of Order Two

203


T
T T Proof. The inclusion α∈A U1,α ⊗a α∈A U2,α ⊂ α∈A (U1,α ⊗a U2,α ) is U ⊗ ∈ β ∈ A implies that v U for all obvious.TIt remains
to show that a 2,β 1,β
T U . Choose some ∈ A U and let γ ∈ A be β ⊗ v ∈ 1,α 2,α a α∈A α∈A arbitrary. By assumption, v has representations v=

nβ X

(1) uν,β

⊗

(2) uν,β

ν=1

=

nγ X

(2) u(1) ν,γ ⊗ uν,γ

(j)

(j) ∈ Uj,γ . with uν,β ∈ Uj,β , uν,γ

ν=1 (2)

(1)

Thanks to Lemma 3.15, we may assume that the systems {uν,β } and {uν,β } are (2) linearly independent. Choose a dual system ϕµ ∈ V20 of {uν,β }. By Defini(2) tion 3.6 it satisfies ϕµ (uν,β ) = δνµ . Application of id ⊗ ϕµ to the first repre(1) sentation yields (id ⊗ ϕµ )(v) = uµ,β , while the second representation leads to Pnγ Pnγ (1) (1) (1) (2) (2) ν=1 ϕµ (uν,γ ) uν,γ . The resulting equation uµ,β = ν=1 ϕµ (uν,γ ) uν,γ shows (1)

(1)

(1)

that uµ,β is a linear combination of vectors uν,γ ∈ U1,γ , i.e., uµ,β ∈ U1,γ . Since T (1) γ ∈ A is arbitrary, uµ,β ∈ α∈A U1,α follows. T (2) (1) Analogously, using the system of {u
dual T
ν,β }, we prove uν,γ ∈ α∈A U2,α . T Hence v ∈ t u α∈A U2,α . α∈A U1,α ⊗a Definition 6.3. For an algebraic tensor v ∈ V1 ⊗a V2 , subspaces U1min (v) ⊂ V1 and U2min (v) ⊂ V2 are called minimal subspaces if they satisfy v ∈ U1min (v) ⊗a U2min (v), v ∈ U1 ⊗a U2

⇒

U1min (v)

(6.4a) ⊂ U1 and

U2min (v)

⊂ U2 .

(6.4b)

Proposition 6.4. All v ∈ V1 ⊗a V2 possess unique minimal subspaces Ujmin (v) for j = 1, 2. Proof. To prove existence and uniqueness of minimal subspaces, define the set F := F(v) := {(U1 , U2 ) : v ∈ U1 ⊗a U2 for subspaces Uj ⊂ Vj } . T F is nonempty since (V1 , V2 ) ∈ F. Then Ujmin (v) := (U1 ,U2 )∈F Uj holds for (v) holds and proves j = 1, 2. In fact, by Lemma 6.2, v ∈ U1min (v) ⊗a U2minT t u (6.4a), while (6.4b) is a consequence of the construction by (U1 ,U2 )∈F Uj . Pr (2) (1) Lemma 6.5. Assume (3.14); i.e., v = ν=1 uν ⊗ uν holds with linearly inde(j) pendent vectors {uν : 1 ≤ ν ≤ r} for j = 1, 2 . Then these vectors span the minimal subspace:  Ujmin (v) = span uν(j) : 1 ≤ ν ≤ r for j = 1, 2. Proof. Apply the proof of Lemma 6.2 to the set A := {β, γ} and the subspaces (j) Uj,β := span{uν : 1 ≤ ν ≤ r}, Uj,γ := Ujmin (v). It shows that Uj,β ⊂ Ujmin (v) . Since a strict inclusion is excluded, Uj,β = Ujmin (v) proves the assertion. t u

204

6 Minimal Subspaces

As a consequence of Lemma 6.5, dim(Ujmin (v)) = r holds for j = 1 and j = 2, proving the following result. Corollary 6.6. U1min (v) and U2min (v) have identical finite dimensions. A constructive algorithm for the determination of Ujmin (v) has already been givenPin the proof of Lemma 3.15: as long as the vectors vν or wν in n v = ν=1 vν ⊗wν are linearly dependent, we can reduce the number of terms by one. This process must terminate after at most n steps. By Lemma 6.5, the resulting vectors {vν } and {wν } span U1min (v) and U2min (v). In the proof of Lemma 6.5 we already made indirect use of the following characterisation of Ujmin (v). The tensor id ⊗ ϕ2 ∈ L(V1 , V1 ) ⊗ V20 can be considered as a mapping belonging to L(V1 ⊗a V2 , V1 ) (cf. §3.3.2.4): v ⊗ w ∈ V1 ⊗a V2 7→ (id ⊗ ϕ2 ) (v ⊗ w) := ϕ2 (w) · v ∈ V1 . The action of ϕ1 ⊗ id ∈ V10 ⊗ L(V2 , V2 ) ⊂ L(V1 ⊗a V2 , V2 ) is analogous. Proposition 6.7. For v ∈ V1 ⊗a V2 the minimal subspaces are characterised by U1min (v) = {(id ⊗ ϕ2 ) (v) : ϕ2 ∈ V20 } ,

(6.5a)

V10 } .

(6.5b)

U2min (v)

= {(ϕ1 ⊗ id) (v) : ϕ1 ∈

(1)

Proof. Repeat the proof of Lemma 6.2: there are maps id ⊗ ϕ2 yielding uν . (1) By Lemma 6.5, the vectors uν span U1min (v). Similarly for (6.5b). Note that the u t right-hand sides in (6.5a,b) are linear subspaces. For V1 = Kn1 and V2 = Kn2 , tensors from V1 ⊗V2 are isomorphic to matrices from Kn1 ×n2 . Then definition (6.5a) may be interpreted as U1min (v) = range{M } = {M x : x ∈ V2 }, where M = M1 (v) is the matrix corresponding to v. Similarly, (6.5b) becomes U2min (v) = range{M T } since M T = M2 (v). Corollary 6.8. (a) Once U1min (v) and U2min (v) are given, we may select any basis Pr (2) (2) (1) {uν : 1 ≤ ν ≤ r} of U2min (v) and find a representation v = ν=1 uν ⊗ uν (2) (1) (cf. (3.14)) with the given vectors {uν } and some basis {uν } of U2min (v). (1) Conversely, we may select a basis {uν : 1 ≤ ν ≤ r} of U1min (v), and obtain Pr (2) (1) (1) (2) v = ν=1 uν ⊗uν with the given uν and some basis {uν } of U2min (v). (2)

(b) If {uν : 1 ≤ ν ≤ s} is a basis of a larger subspace U2 % U2min (v), a representaPs (1) (2) (1) tion v = ν=1 uν ⊗ uν still exists, but the vectors uν are linearly dependent.

6.2 Tensors of Order Two

205 (2)

(c) If we fix a basis {uν : 1 ≤ ν ≤ r} of some subspace U2 ⊂ V2 , there are mappings {ψν : 1 ≤ ν ≤ r} ⊂ L(V1 ⊗a U2 , V1 ) such that ψν (v) ∈ U1min (v) and v=

r X

ψν (v) ⊗ uν(2)

for all v ∈ V1 ⊗ U2 .

ν=1 (2)

Here the functionals ψν ∈ V20 form a dual system of {uν } and are interpreted as mappings of L(V1 ⊗a U2 , V1 ). Proof. (i) Part (c) is a repetition of Remark 3.65, where ψν (v) = vhνi . By definition of U1min (v), vhνi ∈ U1min (v) holds. (ii) In the case of Part (a) dim U2min (v) = r holds and implies dim U1min (v) = r (cf. Corollary 6.6). Set U1 := span1≤ν≤r {vhνi }. Since v ∈ U1 ⊗a U2min (v), (6.4b) implies U1 ⊃ U1min (v). Thus dim(U1 ) = r follows and {vhνi } must be a basis. (iii) If dim(U2 ) = s > dim(U2min (v)) = dim(U1min (v)), the s vectors vhνi in u t must be linearly dependent.

U1min (v)

6.2.2 Use of the Singular-Value Decomposition If v ∈ U1 ⊗a U2 holds for subspaces Uj ⊂ Vj of not too large dimension, the singular-value decomposition offers a practical construction of U1min (v) and U2min (v). Although the singular-value decomposition produces orthonormal bases, no Hilbert structure of V1 and V2 is required. The approach is restricted to the fields R and C. Remark 6.9. Let K ∈ {R, C}. Suppose that a representation of v ∈ U1 ⊗a U2 is Pn (1) (2) (j) given by v = ν=1 vν ⊗ vν with vν ∈ Uj and dim(Uj ) < ∞ . (j)

(a) Choose bases {ui : 1 ≤ i ≤ nj } of Uj (j = 1, 2) and determine the coefficients (j) of vν : vν(j) =

nj X

(j)

(j)

(j = 1, 2).

cνi ui

i=1

Hence v =

Pn1 Pn2 i=1

j=1

(1)

(2)

Mij ui ⊗uj

has the coefficients Mij :=

(1) (2) ν=1 cνi cνj .

Pn

(b) Determine the reduced singular-value decomposition of the matrix M ∈ Kn1×n2 by calling the procedure RSVD(n1 , n2 , r, M, U, Σ, V ), i.e., M = UΣ V T =

r X ν=1

σν aν bνT .

206

6 Minimal Subspaces

(c) Define {ˆ aν : 1 ≤ ν ≤ r} ⊂ U1 and {ˆbν : 1 ≤ ν ≤ r} ⊂ U2 by1 n2 n1 X X (2) (1) for 1 ≤ ν ≤ r . bν [j] uj ˆν := a σν aν [i] ui and ˆbν := i=1

j=1

U1min (v)

They span the minimal subspaces := span {ˆ aν : 1 ≤ ν ≤ r} ⊂ U1 and Pr ˆν ⊗ ˆbν holds. U2min (v) := span{ˆbν : 1 ≤ ν ≤ r} ⊂ U2 , and v = ν=1 a min min (d) dim(U1 (v)) = dim(U2 (v)) = rank(M ) = r. Proof. Singular-value decomposition yields v=

n2 n1 X X

(1) Mij ui

⊗

(2) uj

=

i=1 j=1

=

nv r X X ν=1

n2 n1 X r X X i=1 j=1

(1)

σν aν [i] ui



⊗

nw X j=1

i=1

! σν aν [i]bν [j]

(1)

ui

(2)

⊗ uj

ν=1 (2)

bν [j] uj



=

r X

a ˆν ⊗ ˆbν .

ν=1

Since the vectors aν are linearly independent, the a ˆν are also linearly independent. Similarly, {ˆbν } forms a basis. t u

6.2.3 Minimal Subspaces for a Family of Tensors The minimal subspaces Ujmin (v) belong to a single tensor v ∈ V1 ⊗a V2 . Now we replace the tensor v with a subset F ⊂ V1 ⊗a V2 and ask for minimal subspaces U1min (F ) and U2min (F ) so that v ∈ U1min (F ) ⊗a U2min (F ) holds for all v ∈ F . The obvious result is summarised below. Proposition 6.10. Let F ⊂ V1 ⊗a V2 be a nonempty subset. Then the minimal subspaces U1min (F ) and U2min (F ) are 2 X X (6.6a) U1min (v) and U2min (F ) := U2min (v). U1min (F ) := v∈F

v∈F

Another characterisation is U1min (F ) = span {(id ⊗ ϕ2 ) (v) : ϕ2 ∈ V20 , v ∈ F } , U2min (F ) = span {(ϕ1 ⊗ id) (v) : ϕ1 ∈ V10 , v ∈ F } .

(6.6b)

Proof. v ∈ F and v ∈ U1min (F )⊗a U2min (F ) require Ujmin (v) ⊂ Ujmin (v)(F ) for S j = 1, 2 and all v ∈ F ; hence, v∈F Ujmin (v) ⊂ Ujmin (v)(F ). The smallest S P subspace containing v∈F Ujmin (v) is the sum v∈F Ujmin (v), implying (6.6a). Equations (6.5a,b) prove (6.6b). t u 1 2

aν [i] is the i-th entry of aν ∈ Kn1 , etc. The sum of subspaces in (6.6a) is defined by the span of their union.

207

6.3 Minimal Subspaces of Tensors of Higher Order

6.3 Minimal Subspaces of Tensors of Higher Order In this section we assume that d ≥ 3 and generalise some of the features of tensors Nd of second order. Given v ∈ a j=1 Vj we now look for Ujmin (v) of minimal Nd Nd dimension with v ∈ j=1 Ujmin (v), i.e., v ∈ j=1 Uj must imply Uj ⊃ Ujmin (v). Nd By Lemma 6.1, we may assume v ∈ U := j=1 Uj with finite-dimensional subspaces Uj ⊂ Vj . The lattice structure from Lemma 6.2 generalises to higher order. Lemma 6.11. Let Xj , Yj ⊂ Vj be subspaces for 1 ≤ j ≤ d. Then the identity ! ! d d d O O O = X ∩ Y (Xj ∩ Yj ) a a j j a j=1

j=1

j=1

holds and can be generalised to infinitely many intersections. Proof. For the start of the induction d = 2 use Lemma 6.2. AssumeNthat the Nat d d assertion holds for d −1 and use a j=1 Xj = X1 ⊗X[1] with X[1] := a j=2 Xj
Nd and a j=1 Yj = Y1 ⊗Y[1] . Lemma 6.2 states that v ∈ (X1 ∩Y1 )⊗ X[1] ∩Y[1] . By Nd inductive hypothesis, X[1] ∩Y[1] = a j=2 (Xj ∩Yj ) holds, proving the assertion. t u Again, the minimal subspaces Ujmin (v) can be defined by the intersection of all Nd subspaces Uj ⊂ Vj satisfying v ∈ a j=1 Uj . TheN algebraic characterisation of Ujmin (v) is similar to the case d = 2. Here we N write k∈{1,...,d}\{j} . Note that the right-hand sides of (6.7a–d) k6=j instead of N N N involve the possibly different spaces a k6=j Vk0 , (a k6=j Vk )0 , a k6=j Vk∗ , N ( k6=j Vk )∗ . Nevertheless, the image spaces are identical. Nd Lemma 6.12. Let v ∈ V = a j=1 Vj . (a) The two spaces n o O UjI (v) := ϕ(v) : ϕ ∈ a Vk0 , (6.7a) k6=j   0  O (6.7b) Vk UjII (v) := ϕ(v) : ϕ ∈ a k6=j

UjI (v)

UjII (v).

coincide: = (b) If Vj are normed spaces, we may replace algebraic functionals with continuous functionals: n o O Vk∗ . (6.7c) UjIII (v) := ϕ(v) : ϕ ∈ a k6=j

Then (c) If

UjI (v) a

UjII (v)

UjIII (v)

is valid. = = is a normed space, we may define V k6=j k O n UjIV (v) := ϕ(v) : ϕ ∈

N

k6=j

Then UjI (v) = UjII (v) = UjIV (v) holds.

Vk

∗ o

.

(6.7d)

208

6 Minimal Subspaces

Nd Proof. (i) Since the mappings ϕ are applied to v ∈ U := j=1 Uj (cf. Lemma 6.1), N N N we may replace ϕ ∈ a k6=j Vk0 with ϕ ∈ a k6=j Uk0 and ϕ ∈ (a k6=j Vk )0 N by ϕ ∈ (a k6=j Uk )0 without changing ϕ(v). Since dim(Uk ) < ∞, Proposition N N 3.62c states that a k6=j Uk0 = (a k6=j Uk )0 . This proves Part (a). N N (ii) As in part (i) we may restrict ϕ to a k6=j Uk0 = (a k6=j Uk )0 . Since dim(Uk ) < ∞, algebraic duals are continuous, i.e., ϕ∈

O a

Uk0 =

 O a

k6=j

k6=j

Uk

0

=

O a

Uk∗ .

k6=j

By Hahn–Banach (Theorem 4.18), such mappings can be extended to This proves Part (b), while Part (c) is analogous.

a

N

∗ k6=j Vk .

t u

Nd Theorem 6.13. (a) For any v ∈ V = a j=1 Vj there exist minimal subspaces Ujmin (v) (1 ≤ j ≤ d). An algebraic characterisation of Ujmin (v) is ( Ujmin (v)

= span

)
ϕ(1) ⊗ . . . ⊗ ϕ(j−1) ⊗ id ⊗ ϕ(j+1) ⊗ . . . ⊗ ϕ(d) (v) with ϕ(k) ∈ Vk0 for k 6= j (6.8a)

or equivalently n Ujmin (v) = ϕ(v) : ϕ ∈

O a

k6=j

Vk0

o

,

(6.8b)

where the action of the functional ϕ is understood as in (6.8a). Ujmin (v) coincides with the sets UjI (v), UjII (v), UjIII (v), UjIV (v) in (6.7a–d). Nd (b) For a subset F ⊂ V = a j=1 Vj of tensors, the minimal subspaces Vj,F Nd with F ⊂ a j=1 Vj,F are Ujmin (F ) =

X

Ujmin (v).

(6.8c)

v∈F

(c) For finite-dimensional Vj , the j-rank is defined in (5.6b) and satisfies rankj (v) = dim(Ujmin (v)),

(6.8d)

while, for the infinite-dimensional case, Eq. (6.8d) is the true generalisation of the definition of rankj . Proof. (i) The equivalence of N (6.8a) and (6.8b) is easily seen: Linear combinations of elementary tensors k6=j ϕk are expressed by span {. . .} in (6.8a) N 0 and by ϕ ∈ a k6=j Vk in (6.8b).

209

6.3 Minimal Subspaces of Tensors of Higher Order

d N (ii) The isomorphism Mj (‘matricisation’) from Definition 5.3 maps a Vk k=1 into Vj ⊗a V[j] . Proposition 6.7 states that n o   O 0  min 0 Uj (v) = ϕ(v) : ϕ ∈ V[j] = ϕ(v) : ϕ ∈ a Vk k6=j

is the minimal subspace. The set on the right-hand side is UjII (v) (cf. (6.7b)) and Lemma 6.12 states that UjII (v) = UjI (v), where UjI (v) coincides with the set on the right-hand side of (6.8b). So far, we have proved v ∈ Ujmin (v) ⊗a V[j] . Thanks to Lemma 6.11, the intersection may be performed componentwise yielding Nd v ∈ j=1 Ujmin (v). (iii) For families of tensors, the argument of Proposition 6.10 proves Part (b). u t (iv) For rankj , see the discussion in §6.4. The right-hand side in (6.8a) is the span of a subset. For d = 2, the symbol ‘span’ may be omitted, since the subset is already a subspace (cf. Proposition 6.7). Exercise 6.14. (a) For a subset F ⊂ V let UF := span{F } ⊃ F . Show that Ujmin (F ) = Ujmin (UF ). (b) Let F ⊂ V be a subspace of finite dimension. Show that dim(Ujmin (F )) < ∞. The determination of Ujmin (v) by (6.8a,b) is not very constructive since it requires the application of all dual mappings ϕ(k) ∈ Vk0 . Another approach has already been used in the proof above. Remark 6.15. For j ∈ {1, . . . , d} apply the matricisation Mj := Mj (v) ∈ Vj ⊗a V[j]

with V[j] :=

O a

Vk .

k∈{1,...,d}\{j}

The techniques of §6.2.1 and §6.2.2 may be used to determine the minimal submin min spaces Ujmin (v) and U[j] (v). In particular, if a (v): Mj (v) ∈ Ujmin (v) ⊗ U[j] singular-value decomposition is required, we can make use of Remark 2.28 since only the first subspace Ujmin (v) is of interest. Remark 6.16. While dim (U1min (v)) = dim (U2min (v)) holds for d = 2 (cf. Corollary 6.6), the dimensions of Ujmin (v) are in general different for d ≥ 3. Nd Ujmin (v). Consider Remark 6.17. Let v ∈ V = j=1 Vj with corresponding Nd larger Banach spaces V0,j ⊃ Vj and set V0 := j=1 V0,j (cf. §4.3.4.2). Using ∗ 0 the duals ϕ(j) ∈ V0,j or ϕ(j) ∈ V0,j in (6.7a,c), we define the minimal subspaces min min min (v) coincide, since the minimality (6.5a,b) U0,j (v). However, Uj (v) = U0,j implies uniqueness. The same statement holds for Uαmin (v) defined below.

210

6 Minimal Subspaces

6.4 Hierarchies of Minimal Subspaces and rankα So far, we have defined minimal subspaces Ujmin (v) for a single index j in D := {1, . . . , d}. We can extend this definition to Umin α (v) for all subsets ∅ $ α $ D (cf. (5.3b)). For illustration we consider the example 7 O v∈V= Vj = (V1 ⊗ V2 ) ⊗ (V3 ⊗ V4 ) ⊗ (V5 ⊗ V6 ⊗ V7 ) = Vα ⊗Vβ ⊗Vγ , | {z } | {z } | {z } j=1

=Vα

=Vβ

=Vγ

N7

in which we use the isomorphism between V = j=1 Vj and Vα ⊗ Vβ ⊗ Vγ . Ignoring the tensor structure of Vα , Vβ , Vγ , we regard V = Vα ⊗Vβ ⊗Vγ as tensor space of order 3. Consequently, for v ∈ V there are minimal subspaces Umin α (v) ⊂ min Vα = V1 ⊗ V2 , Umin (v) ⊂ V = V ⊗ V , and U (v) ⊂ V = V ⊗ V6 ⊗ V7 β 3 4 γ 5 γ β min min min such that v ∈ Uα (v) ⊗ Uβ (v) ⊗ Uγ (v) . These minimal subspaces may be constructively determined from Mα (v), Mβ (v), Mγ (v). As in the example, we use the notations (5.3a–d) for D, α, αc , and Vα . By V = Vα ⊗Vαc , any v ∈ V gives rise to minimal subspaces Umin α (v) ⊂ Vα and c Umin (v) ⊂ V . c α α Nd Proposition 6.18. Let v ∈ V = a j=1 Vj , and ∅ 6= α ⊂ D. Then the minimal min subspace Umin (v) for j ∈ α are related by α (v) and the minimal subspaces Uj O Umin Ujmin (v) . (6.9) α (v) ⊂ a j∈α

Nd

Proof. We know that v ∈ U := a j=1 Ujmin (v) . Writing U as Uα ⊗Uαc with N N Uα := a j∈α Ujmin (v) and Uαc := a j∈αc Ujmin (v) , we see that Umin α (v) N min must be contained in Uα = a j∈α Uj (v) . t u An obvious generalisation is the following. Nd Corollary 6.19. Let v ∈ V = a j=1 Vj . Assume that ∅ = 6 α1 , . . . , αm , β ⊂ D are Sm subsets such that β = µ=1 αµ is a disjoint union. Then Om Umin Umin β (v) ⊂ αµ (v). µ=1

In particular, if ∅ = 6 α, α1 , α2 ⊂ D satisfy α = α1 ∪˙ α2 (disjoint union), then min min Umin α (v) ⊂ Uα1 (v) ⊗ Uα2 (v).

If

Y

min dim(Vµ ) ≥ dim(Umin α1 (v)) · dim(Uα2 (v)),

µ∈αc

there are tensors v ∈ V such that (6.10) holds with equality sign.

(6.10)

6.4 Hierarchies of Minimal Subspaces and rankα

211

(1)

(2)

Proof. For the last statement let {bi } be a basis of Umin α1 (v) and {bj } a basis (1) (2) min min min ⊗ Uα } ⊗ Then (v). {b of Uα b U (v) is a basis of (v). For all pairs α1 j i 2 2 N (i, j) choose linearly independent tensors wij ∈ µ∈αc Vµ and set X (2) (1) v := wij ⊗ bi ⊗ bj ∈ V. i,j

(1)

min (v) = spani,j {bi One verifies that Uα

(2)

min ⊗ bj } = Umin α1 (v) ⊗ Uα2 (v).

min (v) is analogous to (6.8a,b): The algebraic characterisation of Uα o n O 0 (j) (j) c (v) : ϕαc = V ∈ Umin ϕ = span , ϕ ϕ (v) α j , α c

(6.11)

j∈α

where ϕαc

Nd

j=1

u t



N N (j) v (j) := ϕαc · j∈α v (j) . j∈αc v

In Definition 5.7 rankα is introduced by rankα (v) := rank (Mα (v)), where Mα (v) is interpreted as a matrix. In the finite-dimensional case, rank(Mα (v)) is equal to the dimension of range(Mα (v)). In general, Mα (v) is a mapping 0 from Vα c into Vα , whereas the interpretation of the range of the matrix Mα (v) considers Mα (v) as a mapping from Vαc into Vα , which is true for the finite0 dimensional case since then Vα c and Vαc may be identified. As announced in (5.8), the true generalisation is min rankα (v) = dim(Uα (v))

(cf. Theorem 6.13c), which includes the case of rankα(v) = ∞ for v ∈ k·k with dim(Umin α (v)) = ∞. For completeness, we define  1 if v 6= 0, rank∅ (v) = rankd (v) = 0 if v = 0

(6.12) Nd

j=1 Vj

(cf. Footnote 4 on page 191). The α-ranks satisfy the following basic rules. Lemma 6.20. (a) The ranks for α ⊂ D and for the complement αc coincide: rankα (v) = rankαc (v).

(6.13a)

(b) If α ⊂ D is the disjoint union α = β ∪˙ γ, then rankα (v) ≤ rankβ (v) · rankγ (v)

(6.13b)

(c) If Y

dim(Vµ ) ≥ rankβ (v) · rankγ (v),

(6.13c)

µ∈αc

then there are v such that equality holds in (6.13b): rankα (v) = rankβ (v) · rankγ (v)

(6.13d)

In particular, under condition (6.13c), random tensors satisfy (6.13d) with probability one.

212

6 Minimal Subspaces

Proof. (i) In the finite-dimensional case we can use Mαc (v) = Mα (v)T to derive (6.13a) from Definition 5.7. In general, use V = Vα ⊗ Vαc and Corollary 6.6. (ii) Definition (6.12) together with (6.10) yields (6.13b). (iii) The last statement in Corollary 6.19 yields Part (c). A random tensor yields t u a random matrix Mα (v) so that Remark 2.5 applies. Corollary 6.21. (a) Decompose D = {1, . . . , d} disjointly into D = α ∪˙ β ∪˙ γ. Then the following inequalities hold: rankα (v) ≤ rankβ (v) · rankγ (v), rankβ (v) ≤ rankα (v) · rankγ (v), rankγ (v) ≤ rankα (v) · rankβ (v). (b) Let α = {j , j + 1, . . . , j}, β = {1, . . . , j − 1}, γ = {1, . . . , j}. Then (6.13b) holds again. Proof. (a) Since αc = β ∪˙ γ, the combination of (6.13a,b) proves the first inequality of Part (a). Since α, β, γ are symmetric in their properties, the other two inequalities follow. (b) For Part (b) note that D = α ∪˙ β ∪˙ γ c . t u min Umin α (v) can be determined from Uβ (v) with β % α (cf. notation (6.6b)).

Proposition 6.22. Let ∅ $ α $ β ⊂ D. Then the following identity holds for all v ∈ V: min min Umin α (v) = Uα (Uβ (v)). c

α α Proof. N (i) Any tensor vα ∈ Umin α (v) has the representation v = ϕ (v) with αc 0 c c ˙ ϕ ∈ j∈αc Vj . Using the disjoint union α = (β\α) ∪ β , there is a splitting P β\α N N c c c β\α ϕα = ν ϕν ⊗ ϕβν with ϕν ∈ j∈β\α Vj0 and ϕβν ∈ j∈β c Vj0 . Define P β\α c c α vν := ϕβν (v) ∈ Umin = ϕα (v) = (vν ) belongs to β (v). Then v ν ϕν min min min min Umin (U (v)), proving that U (v) ⊂ U (U (v)). α α α β β P min (ii) Any w ∈ Umin (U (v)) is of the form w = α β ν wν with O wν = ϕβ\α (vν ), where ϕβ\α ∈ Vj0 and vν ∈ Umin ν ν β (v) j∈β\α c

N c (cf. (6.6a)). Since vν := ϕβν (v) for some functional ϕβν ∈ j∈β c Vj0 , we obtain P β\α P c c c c β\α w = ν ϕν (ϕβν (v)) = ν (ϕν ⊗ ϕβν )(v), i.e., w = ϕα (v) with ϕα = P β\α N c 0 min ⊗ ϕβν ∈ ν ϕν j∈αc Vj . This shows that w ∈ Uα (v) and proves the min min opposite inclusion Umin t u α (v) ⊃ Uα (Uβ (v)). N Corollary 6.23. Let ∅ $ α $ β ⊂ D and ϕ ∈ j∈β c Vj0 . ϕ maps v ∈ V into w := ϕ(v) ∈ Vβ . Then rankα (w) ≤ rankα (v) as well as rank(w) ≤ rank(v).

6.5 Sequences of Minimal Subspaces

213

min min min min Proof. w ∈ Umin β (v) implies Uα (w) ⊂ Uα (Uβ (v)) = Uα (v). The proof for the tensor rank is left to the reader. u t

We conclude with a comparison of the α-rank and the tensor rank introduced in Definition 3.35. Nd Remark 6.24. rankα (v) ≤ rank(v) holds for v ∈ a k=1 Vk and α ⊂ {1, . . ., d}. While rank(·) may depend on the underlying field K ∈ {R, C} (cf. §3.2.6.4), the value of rankα (·) is independent. Pr Nd (j) Proof. (i) Rewrite v = i=1 j=1 ui with r = rank(v) as O Xr (j) (α) (α) (αc ) ui ⊗ ui , where ui := ui . j∈α

i=1

The dimension of Uα :=

(j) span{uα,i

: 1 ≤ i ≤ r} satisfies

rankα (v) = dim(Umin α (v)) ≤ dim(Uα ) ≤ r = rank(v). (ii) Because rankα (v) is the matrix rank of Mα (v), Remark 2.2 proves independence of the field. t u (d)

(d)

Exercise 6.25. Let {bi : i ∈ B} be an (algebraic) basis of Vd and {ϕi : i ∈ B} min (v) = span{vhii : i ∈ B}, where the dual system in Vd0 . Show that U{1,...,d−1} vhii are the expansion terms in (3.22a) (here with Ud = Vd ).

6.5 Sequences of Minimal Subspaces Nd Next we consider sequences vν ∈ a j=1 Vj . In general, the limit with respect to k·k is not an algebraic tensor. Therefore we have to define Ujmin (v) for topological Nd tensors in V := k·k j=1 Vj . Let V be a Banach tensor space with norm k·k and assume that either condition (4.54a) or (4.54b) be valid.

(6.14)

We recall that k·k & k·k∨ is a sufficient condition and that all reasonable crossnorms satisfy this inequality (cf. Proposition 4.74). In (6.7c), UjIII (v) was defined as one of the equivalent definitions of Ujmin (v) for algebraic v (cf. Lemma 6.12). Now we follow this definition and set
  (1) ϕ ⊗ . . . ⊗ ϕ(j−1) ⊗ id ⊗ ϕ(j+1) ⊗ . . . ⊗ ϕ(d) (v), min Uj (v) := span where ϕ(k) ∈ Vk∗ for k ∈ {1, . . . , d}\{j} O  = ϕ(v) : ϕ ∈ a Vk∗ , (6.15a) k∈{1,...,d}\{j}

U(v) :=

d O k·k j=1

Ujmin (v) .

(6.15b)

214

6 Minimal Subspaces

For algebraic tensors v these definitions coincide with the previous ones. Note that Ujmin (v) may be infinite dimensional for topological tensors. Lemma 6.26. (a) Assume (4.54a). Then Ujmin (v) in (6.15a) is well defined for any Nd v ∈ V = k·k j=1 Vj . Ujmin (v) does not depend on k·k . (b) Ujmin (v) may also be defined by the closure of the right-hand side in (6.15a), however, this does not change the tensor subspace U(v) in (6.15b). Proof. By Lemma 4.113c the map ϕ(1) ⊗. . .⊗ϕ(j−1) ⊗id⊗ϕ(j+1) ⊗. . .⊗ϕ(d) is u t continuous. This proves part (a). For part (b) apply Lemma 4.40. ∗ Corollary 6.27. Alternatively we can choose functionals ϕ(k) ∈ V0,k for Banach spaces V0,k ⊃ Vk appearing in (4.54b): O  ∗ V0,k . Ujmin (v) := ϕ(v) : ϕ ∈ a k∈{1,...,d}\{j}

Then, by Lemma 4.116b, ϕ : V → Vj is continuous. In §6.6 we shall discuss the meaning of Ujmin (v) and U(v) for non-algebraic tensors. In the algebraic case, an alternative definition N of Ujmin (v) is UjIV (v) in (6.7d). This definition requires that a norm on V[j] = k6=j Vk be defined. If V is equipped with a uniform crossnorm, we may choose the norm of V[j] according N ∗ ∗ in V[j] . According to Lemma 4.102 to §4.3.2. Note that k6=j Vk is contained N ∗ these spaces may coincide, but in general k6=j Vk is a proper subspace. The next lemma states that, nevertheless, the closures of the image spaces UjIII (v) and UjIV (v) coincide. Nd Lemma 6.28. Assume that V = k·k j=1 Vj is a Banach tensor space with a N uniform crossnorm k·k. For V[j] = k6=j Vk use the uniform crossnorm defined in Corollary 4.108. The dual norm is used in o n ∗ . (6.15c) UjIV (v) := ϕ(v) : ϕ ∈ V[j] Then UjIV (v) = Ujmin (v) is valid. In particular, (6.15b) may be replaced by Nd U(v) = k·k j=1 UjIV (v) . Proof. (i) Since k·k is uniform, it is also a reasonable crossnorm. This ensures that Ujmin (v) is well defined. According to Theorem 4.115, ϕ is well defined on V and therefore also UjIV (v). N ∗ (ii) Since k·k∗[j] k6=j Vk∗ ⊂ V[j] , we have Ujmin (v) ⊂ UjIV (v). (iii) Assuming that UjIV (v) is strictly larger than Ujmin (v), there are ψ ∈ N (k·k k6=j Vk )∗ and u := ψ(v) ∈ UjIV (v) such that u ∈ / Ujmin (v). By Hahn–

6.5 Sequences of Minimal Subspaces

215

Banach, there is a functional ϕ(j) ∈ Vj∗ with ϕ(j) (u) 6= 0 and ϕ(j) |U min (v) = 0. j The tensor w := ϕ(j) (v) ∈ V[j] does not vanish, since   ψ(w) = ϕ(j) ⊗ ψ (v) = ϕ(j) (ψ(v)) = ϕ(j) (u) 6= 0. Hence the definition of kwk∨ > 0 implies that there is an elementary tensor N Nd ϕ[j] = k6=j ϕ(k) with |ϕ[j] (w)| > 0. Set ϕ := ϕ(j) ⊗ ϕ[j] = k=1 ϕ(k) . Now, ϕ(v) = ϕ[j] (ϕ(j) (v)) = ϕ[j] (w) 6= 0 is a contradiction to ϕ(v) = ϕ(j) (ϕ[j] (v)) = 0 because ϕ[j] (v) ∈ Ujmin (v) and ϕ(j) |Ujmin (v) = 0 . Hence UjIV (v) ⊂ Ujmin (v) is valid. Combining this inclusion with part (ii) yields UjIV (v) = Ujmin (v). The statement about U(v) corresponds to Lemma 6.26b. t u So far we used the continuity ϕ[j] (vn ) → ϕ[j] (v) for vn → v. Now we replace convergence by weak convergence and recall Theorem 4.117: Assume (6.14). For Nd all vn , v ∈ k·k j=1 Vj with vn * v, we have weak convergence ϕ[j] (vn ) * ϕ[j] (v) for all ϕ[j] ∈

a

N

k6=j

in Vj

Vk∗ [if (4.54a)] or for all ϕ[j] ∈

[if (4.54b), Remark 6.17]. Theorem 6.29. Assume (6.14). If vn ∈ a then

Nd

j=1 Vj

a

N k6=j

∗ V0,k with V0,k ⊃ Vk

satisfies vn * v ∈ k·k

dim(Ujmin (v)) ≤ lim inf dim(Ujmin (vn )) n→∞

Nd

j=1 Vj

,

for all 1 ≤ j ≤ d.

Proof. Choose a subsequence (again denoted by vn ) such that dim(Ujmin (vn )) is weakly increasing. In the case of dim(Ujmin (vn )) → ∞, nothing has to be proved. Therefore let lim dim(Ujmin (vn )) = N < ∞. For an indirect proof, assume that N dim(Ujmin (v)) > N . Since {ϕ(v) : ϕ ∈ a k6=j Vk∗ } spans Ujmin (v), there are N + 1 linearly independent vectors O [j] [j] b(i) = ϕi (v) with ϕi ∈ a Vk∗ for 1 ≤ i ≤ N + 1. k6=j (i)

[j]

By Theorem 4.117, weak convergence bn := ϕi (vn ) * b(i) holds. By Lemma (i) 4.29, for n large enough, the tuple (bn )1≤i≤N +1) is also linearly independent. (i) [j] Because bn = ϕi (vn ) ∈ Ujmin (vn ), this contradicts dim(Ujmin (vn )) ≤ N. t u These statements about Ujmin (·) can be generalised to Umin α (·) for subsets α.

216

6 Minimal Subspaces

Theorem 6.30. We suppose that (6.16)

either condition (4.54c) or (4.54d) be valid. Nd Nd If vn ∈ a j=1 Vj satisfies vn * v ∈ k·k j=1 Vj , then min dim(Umin α (v)) ≤ lim inf dim(Uα (vn )) n→∞

c

for all α $ D = {1, . . . , d}. c

c

4.121 shows that ϕα (vn ) * ϕα (v) in Vα for all ϕα ∈ Proof. N Theorem α αc αc α α ∗ a k∈αc Vk . The proof requires ϕ (ϕ (vn )) → ϕ (ϕ (v)) for all ϕ ∈ αc ∗ α Vα . (6.16) ensures that ϕ ◦ ϕ is continuous. u t If the space Ujmin (v) becomes infinite dimensional, we may ask about its (infinite) cardinality. The proof of the next remark shows that Ujmin (v) is contained in the completion of a space of dimension ≤ ℵ0 = #N. Remark 6.31. Even when the tensor space V is nonseparable, there is a separable Nd subspace S = k·k j=1 S (j) ⊂ V such d O

Ujmin (v) ⊂ S (j) ,

k·k

Ujmin (v) ⊂ S,

and

v ∈ S.

j=1

In particular, Ujmin (v) and

k·k

d N j=1

Ujmin (v) are separable.

Proof. Let vi → v be a converging sequence with algebraic tensors vi ∈ Valg . ∞ P The sum S (j) := Ujmin (vi ) of the finite-dimensional miminal subspaces has an i=1 ∞ P at most countable dimension: dim(Ujmin (vi )) ≤ ℵ0 , i.e., S (j) is separable and i=1

therefore also S (cf. Remark 4.41). Nd The property vi ∈ a j=1 Ujmin (vi ) ⊂ S together with the closedness of S proves that v = lim vi ∈ S. N ∗ [j] min Similarly, for any ϕ[j] ∈ (vi ) ⊂ S (j) has its k6=j Vk , ϕ (vi ) ∈ Uj limit lim ϕ[j] (vi ) = ϕ[j] (v) in S (j) . Hence, Ujmin (v) ⊂ S (j) holds and implies Nd Nd Nd min (v) ⊂ k·k j=1 S (j) = k·k j=1 S (j) = S (cf. Lemma 4.40). k·k j=1 Uj Nd u Separability of Ujmin (v) and k·k j=1 Ujmin (v) follows from Lemma 4.4. t We can revert the latter statement. Nd Proposition 6.32. Let V = k·k j=1 Vj , where at least two Banach spaces Vj are infinite dimensional. The norm k·k is assumed to satisfy (4.41). Then for any closed separable subspace Uj ⊂ Vj there is some v ∈ V with Uj = Ujmin (v).

6.5 Sequences of Minimal Subspaces

217

Proof. (i) Without loss of generality we consider the case of d = 2 and j = 1. Let {xi : i ∈ N} be a countable and dense subset of U1 . Choose any sequence {yi ∈ V2 : i ∈ N} of linearly independent vectors. Both sequences are normalised: kxi k1 = kyi k2 = 1. On span{yi : 1 ≤ i ≤ n} define the functional ϕn by ϕn (yi ) = 0 for i < n and ϕn (yn ) = 1 and extend ϕn by Hahn-Banach, so that |ϕn (yi )| ≤ 1 for all i > n. Define numbers anm for 1 ≤ n ≤ m < ∞ by ann := 1 and the recursion an,n+j = −

j X

4−i ϕn (yn+i ) an+i,n+j

for j = 1, 2, . . .

i=1

P∞ P∞ Set v := ν=1 4−ν xν0 ⊗ yν with xν0 := µ=ν aνµ xµ (convergence of both sums is still to be shown). Application of id ⊗ ϕn yields ∞ X

(id ⊗ ϕn ) (v) =

4−ν ϕn (yν ) x0ν =

ν=n

=

X

= 4−n

∞ X

j X

j=0

i=0

4−ν ϕn (yν )

ν=n

4−ν ϕn (yν ) aνµ xµ

n≤ν≤µ<∞

∞ X

=

i=ν−n,j=µ−n

∞ X

aνj xj

j=ν

4−n

X

4−i ϕn (yn+i ) an+i,n+j xn+j

0≤i≤j<∞

! 4−i ϕn (yn+i ) an+i,n+j

 = 4−nϕn (yn )ann xn +

∞ X

an,n+j +

j=1

xn+j j X

! 4−i ϕn (yn+i ) an+i,n+j  = 4−n xn

i=1

since ϕn (yn ) = ann = 1 and (. . .) = 0 by definition of an,n+j . This proves U1min (v) = span{xn : n ∈ N } = U1 . (ii) Set bj := supn |an,n+j |. Obviously b0 = 1. By induction we shall prove bj ≤ 2−j−1 . For j = 1 we have |an,n+1 | = 4−1 |ϕn (yn+1 )| an+1,n+1 ≤ 1/4 = 2−1−1 =: b1 , since an+1,n+1 = 1. Assume that bi ≤ 2−i−1 for i ≤ j − 1. The definition of an,n+j yields |an,n+j | ≤

j X i=1

=

j X

4−i |ϕn (yn+i )| |an+i,n+j | ≤ | {z } ≤1

j X i=1

4−i bj−i ≤

j X

4−i 2−(j−i)−1

i=1

2−i 2−j−1 < 2−j−1

i=1

proving bj ≤ 2−j−1 for j. P∞ P∞ (iii) The norm of x0ν = µ=ν aνµ xµ is bounded by µ=ν |aνµ | kxµ k1 ≤ 1 + P∞ −j−1 P∞ = 3/2. The sum defining v is bounded by kvk ≤ 32 C i=1 4−i = j=1 2 C/2 with C in (4.41). t u

218

6 Minimal Subspaces

6.6 Minimal Subspaces of Topological Tensors 6.6.1 Setting of the Problem In Lemma 6.26b, under condition (6.14), we defined Ujmin (v) for topological Nd tensors from k·k j=1 Vj . Accordingly, we may define the Banach subspace U(v) :=

d O k·k

Ujmin (v) .

(6.17)

j=1

For algebraic tensors we know that3 v ∈ U(v). However, the corresponding conjecture v ∈ U(v) for topological tensors turns out to be not quite obvious.4 Nd If v ∈ U(v), there must be a sequence of algebraic tensors un ∈ a j=1 Ujmin (v) Nd tending to v. Since v is the limit of algebraic tensors vn ∈ Valg := a j=1 Vj , one Nd can try to transfer vn → v into a sequence un → v with un ∈ a j=1 Ujmin (v) . We present two approaches. 1. If Ujmin (v) is complemented, v ∈ U(v) can be proved (cf. §6.6.2). This argument5 applies to the Hilbert case (see §6.6.2.3) and is trivial for a finitedimensional Ujmin (v) (cf. §6.6.2.2). 2. The second approach in §6.6.3 requires that the algebraic tensors vn converge fast enough.

6.6.2 First Approach The following criterion refers to complemented subspaces (cf. Definition 4.5). We recall Lemma 4.15: a subspace U is complemented if and only if there exists a continuous projection onto U. 3

Since dim(Ujmin (v)) < ∞,

a

Nd

j=1

Ujmin (v) =

k·k

Nd

j=1

Ujmin (v) = U(v) holds.

4

To repeat the proof of the existence of minimal
subspaces, we need the counterpart of (6.3)
which would be X1 ⊗k·k X2 ∩ Y1 ⊗k·k Y2 = (X1 ∩ Y1 ) ⊗k·k (X2 ∩ Y2 ) . Again,

(X1 ∩ Y1 )⊗k·k (X2 ∩ Y2 ) ⊂ X1 ⊗k·k X2 ∩ Y1 ⊗k·k Y2 is a trivial statement. For the op



posite direction one needs that X1 ⊗k·k X2 ∩ Y1 ⊗k·k Y2 = X1 ⊗a X2 ∩ Y1 ⊗a Y2 is a subset of (X1 ⊗a X2 ) ∩ (Y1 ⊗a Y2 ) = (X1 ∩ Y1 ) ⊗a (X2 ∩ Y2 ) = (X1 ∩ Y1 ) ⊗k·k (X2 ∩ Y2 ) . (6.3)

However, the closure and the intersection of sets satisfy the rule A ∩ B ⊂ A ∩ B, whereas the previous argument requires the reverse inclusion. 5 This approach is communicated to the author by A. Falc´o.

6.6 Minimal Subspaces of Topological Tensors

219

6.6.2.1 Projection onto U(v) Lemma 6.33. Let Uj be complemented for all 1 ≤ j ≤ d. Assume that V = Nd k·k j=1 Vj is a Banach tensor space with a uniform crossnorm k·k.˙ Then the following intersection property holds: d O \ O
Vk . Uj , where V[j] := a Uj ⊗k·k V[j] = k·k j=1

1≤j≤d

k6=j

Nd Proof. The result for the algebraic tensor spaces implies that k·k j=1 Uj =
Nd T T T Uj ⊗a V[j] ⊂ Uj ⊗a V[j] = Uj ⊗k·k V[j] a j=1 Uj = 1≤j≤d

1≤j≤d

1≤j≤d

because of the general rule A ∩ B ⊂ A ∩ B . It remains to prove the opposite inclusion d \ O Uj ⊗k·k V[j] ⊂ k·k Uj . j=1

1≤j≤d

For any v ∈ Uj ⊗k·k V[j] there is a sequence vn ∈ Uj ⊗a V[j] with vn →
v. The projection Pj ∈ L(V Nj , Vj ) from Lemma 4.15 onto Uj satisfies Pj ⊗ I[j] vn = vn , where I[j] = k6=j Ik (Ik : Vk → Vk identity). By the uniform crossnorm property, Pj ⊗ I[j] is continuous on V = Vj ⊗k·k V[j] so that


v = lim vn = lim Pj ⊗ I[j] vn = Pj ⊗ I[j] (lim vn ) = Pj ⊗ I[j] v. Nd Pj ⊗I[j] commutes with Pk ⊗I[k] (k 6= j) and yields the product P := j=1 Pj =
Qd Nd j=1 Pj ⊗ I[j] . This proves v = Pv. Let a j=1 Vj 3 vn → v. Note Nd that un = Pvn ∈ a j=1 Uj and lim un = P lim vn = Pv = v, i.e., Nd v ∈ k·k j=1 Uj , which implies the desired reverse inclusion. t u Nd Theorem 6.34. Assume that V = k·k j=1 Vj is a Banach tensor space with a uniform crossnorm k·k. For v ∈ V assume that Ujmin (v) is complemented for 1 ≤ j ≤ d. Then6 d O v ∈ U(v) := k·k Ujmin (v) . j=1

Proof. According to Lemma 6.33, we must show that v ∈ Ujmin (v) ⊗k·k V[j] for all 1 ≤ j ≤ d. Let Ij ∈ L(Vj
, Vj ) and I[j] ∈ L(V[j] , V[j] ) be the identity mappings. We split v = Ij ⊗ I[j] v into

v = Pj ⊗ I[j] v + (Ij − Pj ) ⊗ I[j] v (6.18) with Pj ∈ L(Vj , Vj )
as in the proof of Lemma 6.33. We want to show that d := (Ij − Pj ) ⊗ I[j] v vanishes. By Lemma 4.87, k·k is a reasonable cross6

Note that

k·k

Nd

j=1

Ujmin (v) =

k·k

Nd

j=1

Ujmin (v) by Lemma 4.40.

220

6 Minimal Subspaces

norm. Therefore the injective norm kdk∨ is defined by the supremum of all |ϕ(d)| Nd N with ϕ = k=1 ϕk and normalised ϕk . Let ϕ[j] = k6=j ϕk and write ϕj ⊗ ϕ[j] as ϕj ◦ (Ij ⊗ ϕ[j] ). Note that   
 (Ij ⊗ ϕ[j] )d = (Ij ⊗ ϕ[j] ) ◦ (Ij − Pj ) ⊗ I[j] v = (Ij − Pj ) ◦ (Ij ⊗ ϕ[j] ) v. By the definition of Ujmin (v), (Ij ⊗ ϕ[j] )v ∈ Ujmin (v) holds, proving   (Ij − Pj ) (Ij ⊗ ϕ[j] )v = 0. This implies
that kdk∨ = 0; hence, d = 0. From (6.18) we conclude that v = Pj ⊗ I[j] v ∈ Ujmin (v) ⊗k·k V[j] . t u The assumption of a uniform crossnorm can be weakened. Instead, we require that tensor products of projections be uniformly bounded: ( kPvk ≤ CP kvk

for all v ∈ V and Od all P = Pj , Pj ∈ L(Vj , Vj ) projection.

(6.19)

j=1

The reason is that the proof from above involves only projections. Also in this case the norm is not weaker than the injective norm. Remark 6.35. A crossnorm k·k satisfying condition (6.19) fulfils k·k∨ ≤ CP k·k (cf. (4.34)). Proof. Note that the proof of Lemma 4.87 uses projections.

t u

6.6.2.2 Finite-Dimensional Ujmin (v) A rather trivial case occurs for finite-dimensional subspaces Ujmin (v). Note that this case may occur even if limn→∞ dim(Ujmin (vn )) = ∞. Remark 6.36. Assume that V is equipped with a norm k·k not weaker than the injective norm. If all subspaces Ujmin (v) (1 ≤ j ≤ d) are finite dimensional, v ∈ U(v) holds. In particular, v is an algebraic tensor. Proof. (i) First we assume that k·k is a uniform crossnorm. By Lemma 4.6 all minimal subspaces Ujmin (v) are complemented so that Theorem 6.34 applies. The Nd finite dimension implies that U(v) is the algebraic tensor product a j=1 Ujmin (v) . Nd (ii) V∨ := ∨ j=1 Vj satisfies V ⊂ V∨ . The definition of Ujmin (v) does not depend on the norm (cf. Lemma 6.26a). Since the injective norm is a uniform crossnorm, part (a) applies and proves that v ∈ V∨ is an algebraic tensor. Therefore Nd v ∈ U(v) = a j=1 Ujmin (v) also belongs to V. t u

6.6 Minimal Subspaces of Topological Tensors

221

6.6.2.3 Case of Hilbert Spaces Since in Hilbert spaces every closed subspace U is complemented: Vj = U ⊕ U ⊥ , Theorem 6.34 yields the following result. Nd Theorem 6.37. For Hilbert spaces Vj let V = k·k j=1 Vj be the Hilbert tensor space with the induced scalar product. Then v ∈ U(v) holds for all v ∈ V with U(v) in (6.17).

6.6.2.4 General Case In the non-Hilbert case we show a negative result. Consider the Banach space V1 . If V1 is not isomorphic to a Hilbert space, there are xi ∈ V1 (i ∈ N) such that the original norm kxi k1 and a Hilbert space norm kxi k1,H are not equivalent, i.e., kxi k1 / kxi k1,H approaches 0 or ∞. Define the Banach subspace V0,1 := span{xi : i ∈ N}. Since V0,1 is not isomorphic to a Hilbert space, Lindenstrauss–Tzafriri [213] prove that V0,1 contains closed separable subspaces U ⊂ V0,1 which are not complemented. By Proposition 6.32 there is a tensor v with U = U1min (v). This proves the following statement. Proposition 6.38. If (at least) Nd two Banach spaces Vk and V` are infinite dimensional, there are v ∈ k·k j=1 Vj with k·k satisfying (4.41) such that Ukmin (v) is not complemented.

6.6.3 Second Approach For the next theorem we need an additional assumption on the norm k·k. A sufficient condition is that k·k is a uniform crossnorm (cf. §4.3.1). The uniform crossnorm property implies that k·k is a reasonable crossnorm (cf. Lemma 4.87). Hence condition (6.14) is ensured (cf. Proposition 4.74). The proof of the theorem requires that the speed of convergence be fast enough. Since there are several cases for which exponential convergence vn → v holds, the condition from below is not too restrictive. Here the index n in vn refers to the tensor rank. Hence for all j we have7 there are vn → v 7

with

rank{1,...,j} (vn ) ≤ n

for n ∈ N.

(6.20)

The ranks rank{1,...,j} (vn ) occur since we combine the vector spaces in the order (. . . ((V1 ⊗ V2 ) ⊗ V3 ) ⊗ . . .) ⊗ Vd . We may choose another order. Instead of n ∈ N it is sufficient to use n ∈ N0 for any infinite subset N0 ⊂ N.

222

6 Minimal Subspaces

Nd Theorem 6.39. Assume that V = k·k j=1 Vj is a Banach tensor space with a uniform crossnorm k·k. If v ∈ V satisfies (6.20) with the rate8 kvn − vk ≤ o(n−3/2 ), then v ∈ U(v) =

k·k

d N j=1

Ujmin (v) , i.e., v = lim un with un ∈

a

d N j=1

Ujmin (v).

Proof for d = 2. For the start of the induction we consider the case d = 2. Each vn ∈ Valg := V1 ⊗a V2 has a representation in U1min (vn ) ⊗ U2min (vn ) with Ujmin (vn ) ⊂ Vj (j = 1, 2), and r := dim U1min (vn ) = dim U2min (vn ) ≤ n. Without loss of generality we assume the worst case r = n and obtain the represenPn (2) (1) tation vn = i=1 vi,n ⊗ vi,n . According to Corollary 6.8c, we can fix any basis (2) min {vi } of U2 (vn ) and recover vn =

n X

(2)

ψi (vn ) ⊗ vi

i=1 (2)

(2)

from a dual basis {ψi } which defines the map ψ i = id ⊗ ψi ∈ L(Valg , V1 ) (2) (2) (cf. Notation 3.57). We choose vi and ψi according to Lemma 4.20 with (2) (2) kvi k2 = kψi k∗2 = 1 and define unI :=

n X

(2)

ψ i (v) ⊗ vi

∈ U1min (v) ⊗a V2 .

i=1

The triangle inequality yields

kunI

n

X (2) − vn k = (ψ i (v) − ψ i (vn )) ⊗ vi

i=1

n

n

X X

(2) (2) = ψ i (v − vn ) ⊗ vi ≤

ψ i (v − vn ) ⊗ vi

i=1 n X

= (4.33c)

(∗)

i=1

(2)

kψ i (v − vn )k1 kvi k2 | {z } i=1 =1

≤

n X

(2)

kψi k∗2 kv − vn k = n kv − vn k.

i=1 (2)

The last line follows from Theorem 4.115: kψ i kV1 ←V = kψi k∗2 = 1. Note that uIn ∈ U1,n ⊗ V2 with U1,n := span {ψi (v) : 1 ≤ i ≤ n} ⊂ U1min (v), where dim U1,n ≤ n. 8

The condition can be weakened to o (n−1−|1/2−1/p| ) if (4.7) applies.

6.6 Minimal Subspaces of Topological Tensors

223

We repeat the previous construction with interchanged directions: we choose a (1) (1) n basis {vi }ni=1 of U1min (vn ) ⊂ V1 and a corresponding dual system {χi }i=1 ⊂ V2∗ . P (1) n An analogous proof shows that unII := i=1 vi ⊗ χi (v) ∈ V1 ⊗a U2,n satisfies the estimate (∗∗) kunII − vn k ≤ nkv − vn k, where U2,n := span{χi (v) : 1 ≤ i ≤ n} ⊂ U2min (v). We choose the projection P2 onto the subspace U2,n according to Theorem 4.16, set P := id ⊗ P2 , and define un := P (uIn ) ∈ U1,n ⊗a U2,n ⊂ U1min (v) ⊗a U2min (v). The uniform crossnorm property (4.45) implies the estimate kP kV←V = kidkV1 ←V1 kP2 kV2 ←V2 ≤

√ n,

in which the latter bound is given by Theorem 4.16 because dim(U2,n ) ≤ n. Since II P2 (uII n ) = un , the estimates
√ II 3/2 kP2 (vn ) − uII n kvn − uII kv − vn k, n k = kP2 vn − un k ≤ nk ≤ n (∗∗)
√ kun − P2 (vn )k = kP2 uIn − vn k ≤ n kuIn − vn k ≤ n3/2 kv − vn k (∗)

are valid. Altogether, we get the estimate    II  kun − vk = k [un − P2 (vn )] + P2 (vn ) − uII n + un − vn + [vn − v] k   ≤ 2n3/2 + n + 1 kv − vn k. The assumption kv − vn k ≤ o(n−3/2 ) implies that kun − vk → 0, i.e., v ∈ U1min (v) ⊗k·k U2min (v). t u Before we continue the proof for d ≥ 3, it must be checked whether the inclusion O min min Umin Ujmin (v) (α = α1 ∪˙ α2 ) (6.22) α (v) ⊂ Uα1 (v) ⊗ Uα2 (v) ⊂ k·kX k·kXα

α

j∈α

corresponding to (6.9) also holds for topological tensors v. Lemma 6.40. Let the assumptions of Theorem 6.39 be valid. Corollary 4.108 can used to construct uniform crossnorms on Umin β (v) for all β ⊂ {1, . . . , d}. Then (6.22) is valid. N (j) Proof. By definition, Umin α (v) is spanned by x := Φ(v) with Φ := j∈α ϕ . ∗ −3/2 Set xn := Φ(vn ) and note that kx − xn kXα ≤ kΦk kv − vn k = o(n ), while rankα2 (xn ) ≤ rankα2 (vn ) ≤ n (cf. Corollary 6.23). We consider x and N xn as tensors of the tensor space Vα1 ⊗ Vα2 with Vαν = k·kX j∈αν Vj . α Applying Theorem 6.39 (d = 2) with Vα1 , Vα2 replaced with V1 , V2 , we obtain min min x ∈ Vα1 ⊗ Vα2 which proves Umin α (v) ⊂ Uα1 (v) ⊗ Uα2 (v).

224

6 Minimal Subspaces

min Applying this statement recursively to Umin α1 (v) and Uα2 (v), we obtain also the last inclusion in (6.22). u t

Nd Proof of Theorem 6.39 (continued). Consider V = j=1 Vj as V[d] ⊗ Vd . Then min min (v). Together with U[d] Theorem 6.39 (d = 2) yields v ∈ Umin (v) ⊂ [d] (v) ⊗ Ud Nd−1 min t u (v) in (6.22) we prove the assertion. k·kX j=1 Uj [d]

6.7 Minimal Subspaces for Intersection Spaces Banach spaces which are intersection spaces (see §4.3.5) do not satisfy the basic assumption (6.14). Therefore we have to check whether the previous results can be extended to this case. We recall the general setting. For each 1 ≤ j ≤ d we have a (n) scale of spaces Vj , 0 ≤ n ≤ Nj , which leads to tensor spaces V(n) =

d O k·kn

(nj )

Vj

for multi-indices n ∈ N ⊂ Nd0 ,

j=1

where N satisfies the conditions (4.57a,b). The final tensor subspace is \ Vtop := V(n) n∈N

endowed with the intersection norm (4.58b). The algebraic counterparts are denoted by d O \ (n) (n ) (n) Valg := a Vj j and Valg := Valg . j=1

n∈N

The results for algebraic tensor spaces are in §6.7.1, while topological tensor spaces are treated in §6.7.2.

6.7.1 Algebraic Tensor Space All spaces V(n) are dense subspaces of V(0) =

d O k·k0

Vj ,

(0)

where Vj = Vj

j=1

Nd (cf. Lemma 4.123). Here we consider algebraic tensors from a j=1 Vj . Each N (n) (n ) multi-index n ∈ N defines another space V[j] = a k∈{1,...,d}\{j} Vk k . Nevertheless, each n yields the same minimal subspace Ujmin (v).

6.7 Minimal Subspaces for Intersection Spaces

225

Remark 6.41. For all n ∈ Nd0 with nk ≤ Nk and v ∈ Valg , the minimal subspace is given by  (N ) (n) (6.23) Ujmin (v) = ϕ(v) : ϕ ∈ (V[j] )0 ⊂ Vj j . (N ,...,Nd )

Proof. By Proposition 4.125, v ∈ Valg1 (n)

From v ∈ Valg

(n)

⊂ V holds for all n with nk ≤ Nk . Nd alg min (N ,...,Nd ) we derive (6.23) and v ∈ j=1 Uj (v). By v ∈ Valg1 (Nj )

and minimality of Ujmin (v), the inclusion Ujmin (v) ⊂ Vj

follows.

t u

6.7.2 Topological Tensor Space Remark 6.41 does not hold for non-algebraic tensors in V := Vtop . A simple counter-example is f ∈ C 1 (I × J) with f (x, y) = F (x + y) and F ∈ / C 2. 0 0 1 ∗ Choose the functional ϕ = δη ∈ C (J) . Then ϕ(f )(x) = −F (x + η) ∈ C 0 (I), but ϕ(f ) is not in C 1 (I) in contrast to Remark 6.41. (n)

While in Remark 6.41 we could take functionals from V[j] for any n bounded by nk ≤ Nk , we now must restrict the functionals to n = 0. The statement of Lemma 6.26 becomes: The following definition is well defined for all v ∈ V(n) . ∗ o  O n (0) Ujmin (v) := ϕ(v) : ϕ ∈ a Vk k∈{1,...,d}\{j} n o O (0) ϕ(k) , ϕ(k) ∈ (Vk )∗ = span ϕ(v) : ϕ = k∈{1,...,d}\{j}

We now show that the same results can be derived as in the standard case. Condition (6.14) used before must be adapted to the situation of the intersection space. Consider the tuples Nj = (0, . . . , 0, Nj , 0, . . . , 0) ∈ N in (4.57c) and the corresponding topological tensor spaces (0)

V(Nj ) = V1

(Nj )

(0)

⊗ . . . ⊗ Vj−1 ⊗ Vj

(0)

(0)

⊗ Vj+1 ⊗ . . . ⊗ Vd

endowed with the norm k·kNj . We require k·k

(0)

∨(V1

(0)

(Nj )

,...,Vj−1 ,Vj

(0)

(0)

,Vj+1 ,...,Vd

)

. k·kNj

for all 1 ≤ j ≤ d.

(6.24)

This is a particular case of (4.54b). Therefore Theorem 4.117b yields the following result. N (0) Lemma 6.42. Assume (6.24) and ϕ[j] ∈ a (Vk )∗ . Let vm , v ∈ V with vm * v. k6=j

(N )

(a) Then weak convergence ϕ[j] (vm ) * ϕ[j] (v) holds in Vj = Vj j . (b) The estimate kϕ[j] (v − vm )kj,Nj ≤ C kv − vm kNj N N (0) holds for elementary tensors ϕ[j] = k6=j ϕ(k) ∈ a k6=j (Vk )∗ with normalised factors kϕ(k) k∗k,0 = 1, where C is the norm constant involved in (6.24).

226

6 Minimal Subspaces (N )

Proof. Note that a functional ϕ(j) ∈ (Vj j )∗ composed with an elementary tensor Nd N N Nd (n ) (0) ϕ[j] = k6=j ϕ(k) ∈ a k6=j (Vk )∗ yields ϕ = k=1 ϕ(k) ∈ a k=1 (Vk j )∗ , where nj are the components of n = Nj . By (6.24), ϕ belongs to (V(Nj ) )∗ . (Nj )

Conclusion 6.43. Under assumption (6.24), Ujmin (v) ⊂ Vj = Vj v ∈ V and all 1 ≤ j ≤ d.

u t

holds for all

Proof. Let vm ∈ Valg be a sequence with vm → v ∈ V. By the definition (4.58b) of the intersection norm, kvm − vkNj → 0 holds for all j. Then Lemma 6.42b shows that kϕ[j] (v − vm )kj,Nj → 0. Since ϕ[j] (vm ) ∈ Vj by Proposition 4.125, the limit ϕ[j] (v) also belongs to Vj . t u

Theorem 6.44. Assume (6.24) and vm ∈ Valg with vm * v ∈ V. Then dim(Ujmin (v)) ≤ lim inf dim(Ujmin (vm )) m→∞

for all 1 ≤ j ≤ d.

Proof. We can repeat the proof from Theorem 6.29.

t u

6.8 Linear Constraints and Regularity Properties For a fixed k let ϕk ∈ Vk∗ be a continuous linear functional. We say that a tensor Nd v ∈ j=1 Vj satisfies the linear constraint ϕk if (id ⊗ . . . ⊗ id ⊗ ϕk ⊗ id ⊗ . . . ⊗ id) v = 0.

(6.25)

A single constraint can be replaced with a family Φ ⊂ Vk∗ : v satisfies the linear constraints Φ ⊂ Vk∗ if (6.25) holds for all ϕk ∈ Φ. If, for instance, Vk = Knk ×nk is a matrix space, the subset of symmetric matrices is characterised by ϕνµ (M ) := Mνµ −Mµν = 0 for all 1 ≤ ν ≤ µ ≤ nk . In the same way, tridiagonal matrices, sparse matrices with a fixed sparsity pattern, etc., can be defined by linear constraints. The next statement is mentioned by Tyrtyshnikov [285]. Remark 6.45. Under the assumption (6.14), statements (a) and (b) are equivalent: Nd (a) v ∈ j=1 Vj satisfies a linear constraint ϕk , (b) the minimal subspace Ukmin (v) fulfils ϕk (Ukmin (v)) = 0; i.e., ϕk (uk ) = 0

holds for all uk ∈ Ukmin (v).

227

6.8 Linear Constraints and Regularity Properties

Proof. The direction (b)⇒(a) is trivial. Assume (a) and choose any uk ∈ Ukmin (v). ∗ There is some ϕ[k] ∈ V[k] with uk = ϕ[k] (v). Since ϕk ◦ ϕ[k] = ϕ[k] ◦ ϕk , we t u conclude that ϕk (uk ) = ϕk (ϕ[k] (v)) = ϕ[k] (ϕk (v)) = ϕ[k] (0) = 0. In the case of infinite-dimensional function cases, regularity properties (i.e., smoothness properties) of v are characterised by the boundedness Nd of 2certain 2 functionals. For instance, consider the case of V = L (Ω) = k·kL2 j=1 L (Ωj ) . The derivative Ak = ∂/∂xk is an unbounded linear operator from Vk = L2 (Ωk ) into Vk . Nevertheless there are functions v ∈ V that are differentiable in xk for a particular k, so that Av ∈ V is well defined, where A = id[k] ⊗ A(k) , i.e., v belongs to the domain of A: D(A) := {v ∈ V : Av ∈ W}

for an (unbounded) linear operator A. (6.26)

In the following we assume that the unbounded operator A(k) is a pointwise limit of (k) (k) a sequence of bounded operators Aν ∈ L(Vk , Vk ), i.e., k(A(k) − Aν )v (k) kk → 0 (k) (k) (k) for all v ∈ D(A ). For instance A = ∂/∂xk is the limit of the difference (k) quotient operator. A is regarded as the pointwise limit of id[k] ⊗ Aν . Assuming Nd that the norm of V is a uniform crossnorm, it follows that D(Ak ) ⊗a j6=k Vj = D(A). Proposition 6.46. Let A = id ⊗ . . . ⊗ id ⊗ A(k) ⊗ id ⊗ . . . ⊗ id : V → V be an unbounded operator as explained above. The norm of V is required to be a uniform crossnorm. Then v ∈ D(A) implies Ukmin (v) ⊂ D(A(k) ). (k) min Proof. All v (k)∈U N k (v) are linear∗combinations of vectors of the form v = Φ(v) for some Φ = j6=k ϕj , ϕj ∈ Vj . Theorem 4.115 ensures that Φ ∈ L(V, Vk ). (k)

(k)

(k)

Note that Aν ∈ L(Vk , Vk ) satisfies Φ(Aν v) = Aν Φ(v) for Aν = id[k] ⊗Aν . Since the left-hand side passes to the limit Φ(Av), also the right-hand side has a (k) limit limν Aν Φ(v) = A(k) (Φ(v)), i.e., Φ(v) ∈ D(Ak ). u t Nd In the case of a Hilbert tensor space V = k·k j=1 Vj with induced scalar product and ∅ $ α $ D := {1, . . . , d}, the minimal subspace Uαmin (v) can be determined from the singular-value decomposition of Mα (v) =

r X (α) σi ui ⊗ vi i=1

N (α) (cf. §5.2.3) with σi > 0, orthonormal systems {ui } ⊂ Vα = k·k j∈α Vj , {vi } ⊂ Vαc , and r ∈ N0 ∪ {∞}. The norms of Vα and Vαc are defined by the correspondingly induced scalar product.

228

6 Minimal Subspaces

Proposition 6.47. Suppose a Hilbert tensor space V, a tensor v ∈ V, and the singular-value decomposition of Mα (v) as defined above. (a) Then Uαmin (v) = span{ui : 1 ≤ i ≤ r} holds (note that r = ∞ is possible). (b) Let Uα ⊂ Uαmin (v) be any subspace of the form Uα = span{ui : i ∈ Jα }, where Jα ⊂ {1, . . . , r}, and let Π (α) be the orthogonal projection onto Uα . Suppose that β ⊂ D satisfies either β ⊂ α or β ⊂ αc . Let A(β) be a (possibly unbounded) linear operator on Vβ . Extend Π (α) and A(β) to the full tensor space by c

Πα := Π (α) ⊗ I (α ) ,

c

Aβ := A(β) ⊗ I (β ) ,

where I (γ) is the identity on Vγ for the complements γ = αc , β c . If v ∈ D(Aβ ), then also Πα v ∈ D(Aβ ) and kAβ Πα vk ≤ kAβ vk . Proof. Part (a) is left to the reader. For Part (b) assume first that β ⊂ αc . We may write Πα = Π (α) ⊗ I (α

c

\β)

⊗ I (β)

c

and Aβ = I (α) ⊗ I (α

\β)

⊗ A(β

and see that these operators commute. Hence kAβ Πα vk ≤ kΠα Aβ vk ≤ kAβ vk because kΠα k ≤ 1. If β ⊂ α, we choose the corresponding projection c

Παc := I (α) ⊗ Π (α ) , c

where Π (α ) projects onto Uαc = span{vi : i ∈ Jα } with vi from Mα (v) = Pr (α) c c i=1 σi ui ⊗vi . Although Πα 6= Πα , Πα v = Πα v holds. Since Aβ commutes u with Παc , we have again kAβ Πα vk = kAβ Παc vk ≤ kΠαc Aβ vk ≤ kAβ vk . t Concrete examples of Aβ are differential operators with respect to the variables xi , i ∈ β. Particular subspaces Uα are span{ui } for a single i. Then Proposition 6.47 with α = β shows that (α)

kA(α) ui k ≤ kAα vk /σi , which is a quantified version of the statement Uαmin (v) ⊂ D(A(α) ). Another description of regularity is the boundedness of a norm. In (4.56) scales of spaces with corresponding norms k·kj,n are introduced. The degree of regularity (smoothness) is given by the maximal n with kf kj,n < ∞ (for a function f ∈ Vj ). The analogue of Proposition 6.46 is the statement in Conclusion 6.43: v ∈ V(Nj ) (N ) implies Ujmin (v) ⊂ Vj j .

6.9 Minimal Subspaces for (Anti-)Symmetric Tensors

229

6.9 Minimal Subspaces for (Anti-)Symmetric Tensors Lemma 6.48. Let v ∈ Sd (V ) [v ∈ Ad (V )] be an [anti-]symmetric tensor. For min any subset ∅ $ α $ D, the minimal subspace satisfies9 Vα (v) ⊂ Sα (V ) min min [Vα (v) ⊂ Aα (V )]. Moreover, Vα (v) only depends on #α. Proof. This is a consequence of Lemma 3.87. Let {bi : i ∈ B} be an (algebraic) basis of V and {ϕi : i ∈ B} the dual basis. We denote the interpretation of ϕi as mapping from ⊗d V into ⊗d−1 V (d) (d) Nd by ϕi to express that ϕi is applied to the d-th direction: ϕi ( j=1 v (j) ) = Nd−1 ϕi (v (d) ) j=1 v (j) . Nd While { j=1 bij : ij ∈ B} is the basis of ⊗d V (cf. Proposition 3.22), Nd {PS ( j=1 bij ) : ij ∈ B} is the basis of Sd (V ) (considered as algebraic tensor Nd (d) space). Applying ϕν to PS ( j=1 bij ) for arbitrary i1 , . . . , id yields        d d d O X O X O 1 1 PS  ϕ(d) bij = ϕν (bij ) PS  bik = δν,ij PS  bi k  ν d d j=1 j=1 j=1 k6=j

k6=j

for in to the space of polynomials P ν ∈ νB. As stated Q§3.5.2,νi Sd (V ) is isomorphic P ν a x with x = x and |ν| := ν = d. This space defines the ν ν i∈B i i∈B i Nd Qd quantics Q(d, B). The isomorphism is given by Φ : PS ( j=1 bij ) → j=1 xij (d)

defined in (3.42). Via the isomorphism Φ the map ϕν ∈ L(Sd (V ), Sd−1 (V )) (d) −1 corresponds to ϕΦ ∈ L(Q(d, B), Q(d − 1, B)): ν = Φ ◦ ϕν ◦ Φ   d d Y X Y   ϕΦ x = δν,ij xik ∈ Q(d − 1, B). ij ν j=1

j=1

k6=j

Since the derivative dxdν xij coincides with δν,ij , the product rule yields ϕΦ ν = This proves the first part of the following statement.

d dxν .

Proposition 6.49. The minimal subspace of a homogeneous polynomial q ∈ Q(d, B)   is d min Uj (q) = span q:ν∈B . dxν For α ⊂ D with k := #α, the minimal subspace is   dk Uαmin (q) = span q : ν1 , . . . , νk ∈ B ⊂ Q(d − k, B). dxν1 dxν2 · · · dxνk Proof. For Uαmin (q) apply induction on k and use Proposition 6.22. As stated in Lemma 6.48, the minimal subspace only depends on k, not α. The first part corresponds to k = 1. t u 9

The case #α = 1 is uninteresting. Tensors of order 1 are both symmetric and antisymmetric (the only permutation π ∈ P is the identity).

Part III

Numerical Treatment

The numerical treatment of tensors is based on a suitable tensor representation. The first four chapters are devoted to two well-known representations. Chapter 7 describes the r-term format (also called canonical or CP format), while Chapter 8 is concerned with the tensor subspace format (also called Tucker format). In both chapters, tensors are exactly represented. Quite another topic is the approximation of tensors. Chapter 7 studies the approximation within the r-term format. Here, it becomes obvious that tensors of larger order than two have much less favourable properties than matrices, which are tensors of order two. Approximation within the tensor subspace format is addressed in Chapter 8. Here, the technique of higher order singular value decomposition (HOSVD; cf. §10.1) is very helpful, both theoretically and practically. While the r-term format suffers from a possible numerical instability, the storage size of the tensor subspace format increases exponentially with the tensor order d. A format avoiding both drawbacks is the hierarchical format described in Chapter 11. Here, the storage is strictly bounded by the product of the maximal involved rank, the maximal dimension of the vector spaces Vj , and d, the order of the tensor. Again, HOSVD techniques can be used for a quasi-optimal truncation. Since the format is closed, numerical instability does not occur. The hierarchical format is based on a dimension partition tree. A particular choice of the tree leads to the matrix product representation or TT format described in Chapter 12. The essential part of the numerical tensor calculus is the performance of tensor operations. In Chapter 13 we describe all operations, their realisation in the different formats, and the corresponding arithmetical cost. Chapter 14 contains the details of the tensorisation technique. When applied to (grid) functions, tensorisation corresponds to a multiscale approach. Chapter 15 is devoted to the multivariate cross approximation, which has several important applications. If a tensor can be evaluated entry-wise, this method allows us to construct a tensor approximation in the hierarchical format. In Chapter 16, the application of the tensor calculus to elliptic boundary value problems and elliptic eigenvalue problems is discussed. The final Chapter 17 collects a number of further topics. Section 17.1 considers general minimisation problems. Another minimisation approach described in Sect. 17.2 applies directly to the parameters of the tensor representation. Dynamic problems are studied in Sect. 17.3, while the ANOVA method is mentioned in Sect. 17.4.

Chapter 7

r-Term Representation

Abstract The r-term representation v=

r O d X

vν(j) ,

ν=1 j=1

i.e., a representation by sums of r elementary tensors, is already used in the algebraic definition (3.9) of tensors. In different fields, the r-term representation has different names: ‘canonical decomposition’ in psychometrics (cf. [48]), ‘parallel factors model’ (cf. [158]) in chemometrics.1 The word ‘representation’ is often replaced by ‘format’. The short form ‘CP’ is proposed by Comon [62] meaning ‘canonical polyadic decomposition’. Sometimes the polyadic decomposition (PD) is distinguished from the canonical polyadic decomposition (CPD) by the fact that in the latter case the used number of terms should be minimal (equal to the tensor rank; cf. Vervliet et al [296]). Here, the notation ‘r-term representation’ is used, where ‘r’ is considered as a variable from N0 , which may be replaced by other variable names or numbers. Before we discuss the r-term representation in Section 7.3, we consider representations in general (Section 7.1) and the full representation (Section 7.2). The sensitivity of the r-term representation is analysed in Section 7.4. Section 7.5 discusses possi(j) ble representations of the vectors vν ∈ Vj . We briefly mention the conversion from full format to r-term representation (cf. §7.6.1). We conclude with the discussion of (anti-)symmetric tensors (in Section 7.7) and other modifications of the r-term format (in Section 7.8). The discussion of arithmetical operations with tensors in r-term representation is postponed to Chapter 13. In this chapter we restrict our considerations to the exact representation in the r-term format. Approximations, which are of greater interest in practice, will be discussed in Chapter 9. 1

The combination of both names has led to the awful abbreviation ‘CANDECOMP/PARAFAC decomposition’. © Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_7

233

234

7 r-Term Representation

7.1 Representations in General 7.1.1 Concept For any practical implementation we must distinguish between the mathematical objects and the way we represent them for the purpose of a computer implementation. To give a simple example: any rational number r ∈ Q may be represented by a pair (p, q) of integers. Here we need the semantic explanation that r = p/q. From this example we see that the representation is not necessarily unique. The representation concept may be iterated, e.g., in the previous example we need a representation of the integers, e.g., by a bit sequence together with a coding of the sign. Let S be a mathematical set. In general, a representation of s ∈ S is based on a mapping ρS : PS → S (7.1) where the set PS usually consists of tuples p = (p1 , . . . , pn ) of parameters which are assumed to be realisable in computer language implementations. The mapping ρS only serves as a definition of p ∈ PS and cannot be used in implementations. First we discuss surjectivity of ρS . Since most of the mathematical sets are infinite, whereas a real computer has only finite size, surjectivity cannot hold in general. There are two ways to overcome this problem. Like in the concept of the Turing machine, we may base our considerations on a virtual computer with infinite storage. Then, e.g., all integers can be represented by bit sequences of arbitrary length. The second remedy is the replacement of S with a finite subset S0 ⊂ S such that ρS : PS → S0 becomes surjective. For instance, we may restrict the integers to an interval S0 = Z∩[−imax , imax ]. A consequence is that we have to expect problems when we try to perform the addition imax +1. Another type of replacement S0 ⊂ S is known for the case S = R. Here the set S0 of machine numbers satisfies a density property: any real number2 x ∈ S = R can be approximated by x0 ∈ S0 such that the relative error is smaller than the so-called machine precision eps. From now on we assume that the mapping ρS : PS → S is surjective (since infinite storage is assumed or / and S is replaced with a finite subset again denoted by S). There is no need to require injectivity of ρS . In general, the inverse ρ−1 S (s) of s∈S is set-valued. Any p ∈ ρ−1 S (s) may be used equally well to represent s ∈ S. 2

Here, we ignore the problems of overflow and underflow, which is a difficulty of the same kind as discussed above for integers.

7.1 Representations in General

235

7.1.2 Computational and Memory Cost An important property of s ∈ S is the storage size needed for its representation. For instance, a natural number n ∈ S = N needs 1 + blog2 nc bits. In general, we must deal with the storage needed for the parameter tuple p = (p1 , . . . , pn ). We denote the necessary storage by Nmem (p). Since s ∈ S may have many representations, we associate s with the memory size3  Nmem (s) := min Nmem (p) : p ∈ ρ−1 S (s) . −1 Practically when ρ−1 S (s) is a large set, it might be hard to find p ∈ ρS (s) with Nmem (p) = Nmem (s).

Usually we want to perform some operations between mathematical objects or we want to evaluate certain functions. Assume, e.g., a binary operation within the set S. The assignment s := s1 s2 requires finding a representation p of s, provided that representations pi of si (i = 1, 2) are given. The corresponding operation b on the side of the parameter representations becomes p := p1 b p2

ρS (p) = ρS (p1 )

(7.2) ρS (p2 )

ρS (p2 ) is in general not unique . The right-hand note that p ∈ ρS−1 ρS (p1 ) side in (7.2) explains only the meaning of b . It cannot be used for the implementation since ρS and ρ−1 S are not implementable. We assume that there is some algorithm mapping the arguments p1 , p2 ∈ PS into some p = p1 b p2 ∈ PS with finite Nmem (p) such that this computation requires a finite number of arithmetic operations. The latter number is denoted by N and may be a function of the arguments. The standard unit of N are the arithmetic operations +, −, ∗, / of real numbers (machine numbers). :⇐⇒

The same setting holds for an n-variate function ϕ : S1 × . . . × Sn → S0 . Assume representations ρi : Pi → Si for 0 ≤ i ≤ n. Then, on the level of representation, ϕ becomes ϕˆ : P1 × . . . × Pn → P0

with

ˆ 1 , . . . , pn )) = ϕ(ρ1 (p1 ), . . . , ρn (pn )). ρ0 (ϕ(p The required number of arithmetic operations is denoted by Nϕ . 3

The memory size Nmem (p) is assumed to be a natural number. Any subset of N has a minimum.

7 r-Term Representation

236

7.1.3 Tensor Representation versus Tensor Decomposition The term ‘decomposition’ is well known, e.g., from the QR decomposition or singular-value decomposition. We may define a decomposition as an (at least essentially) injective representation. As an example we take the singular-value decomposition. We may represent a matrix M by the three matrix-valued parameters p1 = U (unitary matrix), p2 = Σ (diagonal matrix), and V (unitary matrix). The semantic explanation is ρSVD (U, Σ, V ) = M := UΣ V T . However, the representation of M is not the purpose of SVD. Instead, the parameters U, Σ, V of ρSVD (U, Σ, V ) = M are of interest since they indicate important properties of M . Injectivity of ρSVD is necessary to speak about the singular vectors ui , vi and the singular values Σii . We know from Corollary 2.24b that injectivity does not hold for multiple singular values. Therefore the vague formulation ‘essentially injective’ has been used above. In a certain way, ‘representation’ and ‘decomposition’ play opposite roles as synthesis and analysis. • Representation. The parameters in ρS (p1 , . . . , pn ) = s are only of auxiliary nature. In the case of non-injectivity, in principle, any parameter tuple is as good as another. If the data sizes are different, one may be interested in the cost-optimal choice. Furthermore, there may be stability reasons to prefer certain parameters. The representation of s is illustrated by the direction p1 , . . . , pn 7→ s. • Decomposition. For a given s ∈ S one likes to obtain the parameters pi in ρS (p1 , . . . , pn ) = s. Therefore, the direction is s 7→ p1 , . . . , pn . ‘Tensor decomposition’ it is applied when features of a concrete object should be characterised by parameters of tensor-valued data about this object. The r-term representation can be considered as decomposition since often essential injectivity holds (cf. Remark 7.6 and Theorem 7.5). The HOSVD decomposition in §8.3 is another example. Since our main interest is the calculation with tensors, here we are only interested in representations of tensors. However, in many practical applications the decomposition is an essential tool for data analysis (cf. §1.2.2).

7.2 Full and Sparse Representation

237

7.2 Full and Sparse Representation Consider the tensor space V=

d O

Vj

j=1

of finite-dimensional spaces Vj . As mentioned before, in the finite-dimensional case, we need not distinguish between algebraic and topological spaces. Introducing bases (j) (j) {b1 , b2 , . . .} of Vj and index sets Ij := {1, . . . , dim(Vj )}, we obtain a repreNd Nd Ij I sentation of elements from with j=1 Vj by elements from K = j=1 K index sets I = I1 × . . . × Id . This representation ρ : P = KI → S = ρfull (a) =

X

(1)

(d)

ai bi1 ⊗ . . . ⊗ bid

Nd

i=1

Vi (cf. (7.1)) is defined by

with a ∈ KI and i = (i1 , . . . , id ) ∈ I . (7.3)

i∈I

In the case of Vj = KIj , the bases are formed by the unit vectors. Notation 7.1. The full representation uses the coefficients ai ∈ KI with the interpretation (7.3). The data size is ! d d O Y full dim(Vj ). Vj = Nmem = #I = dim (7.4) j=1

j=1

In the model case of dim(Vj ) = n for all 1 ≤ j ≤ d, the storage size is nd . Unless n and d are very small numbers, the value of nd is too huge for practical realisations. In particular when d becomes large, the exponential growth of nd is a severe hurdle. In the case of matrices, the format of sparse matrices is very popular. For completeness, we formulate the sparse-tensor format. A concrete example will follow in §7.6.5. I, the sparse representation Remark 7.2. Given a subset ˚ I ⊂ I and ai ∈ K for all i ∈˚ and the represents tensor consists of the data ˚ I and (ai )i∈˚ I   X (d) (1) ˚ ρsparse I, (ai )i∈˚ a b ⊗ . . . ⊗ bid . (7.5) I = ˚ i i1 i∈I

= 2#˚ I. The data size is The traditional understanding in linear (and multilinear) algebra is that their objects are completely given. A vector v ∈ KI needs knowledge of all vi (i ∈ I) and, correspondingly, all coefficients vi should be stored simultaneously. In the field of analysis, the alternative concept of functions is dominating. Note that KI for arbitrary, possibly infinite sets I is isomorphic to the set of functions I → K. Although a function f , say from I = [0, 1] to K is defined as the set sparse Nmem

7 r-Term Representation

238

{(x, f (x)) : x ∈ [0, 1]} of all pairs, implementations of f do not suffer from the fact that there are infinitely many function values. Instead of requiring all values to be present, we ask for the possibility to determine f (x) only for a given x ∈ I. Accordingly, the full functional representation of some a ∈ KI needs the implementation of a function function a(i1 , i2 , . . . , id )

(7.6)

which returns the entry ai ∈ K for any single index i = (i1 , . . . , id ). Often tensors which can be represented by (7.6), are called function related tensors. This naming is a bit vague. In principle, any tensor a ∈ KI with finite I can be implemented by (7.6). A typical function related tensor is for ij ∈ {0, 1, . . . , n} and h = 1/n,

f (i1 h, i2 h, . . . , id h)

describing the restriction of a function f : [0, 1]d → K to a uniform grid Gh ⊂ [0, 1]d of grid size h (such a restriction we call a ‘grid function’). The evaluation time of f (x) for an x ∈ Kd is assumed to be independent of the grid size h. (1) (2) (d) The uniform grid may equally well be replaced with f (xi1 , xi2 , . . . , xid ), where (j) {xi : i ∈ Ij } is a non-equidistant grid in the j-th direction. The functional representation is of particular interest if partial evaluations of a ∈ KI are required (cf. §15).

7.3 r-Term Representation The set Rr defined in (3.18) or (3.18∗ ) is fundamental for the r-term representation. Definition 7.3. For variable r ∈ N0 , the r-term representation is explained by the mapping r O d d   X O vν(j) ∈ V := a (7.7a) Vj ρr-term r, (vν(j) )1≤j≤d := 1≤ν≤r

with r ∈ N0 ,

(j) vν

ν=1 j=1

j=1

∈ Vj . For fixed r ∈ N0 , the r-term representation S r ρr-term : N0 × r∈N0 (V1 × . . . × Vd ) → V ,   r N d P (j) (j) ρr-term r, (vν )1≤j≤d = vν ∈ Rr 1≤ν≤r

(7.7b)

ν=1 j=1

refers to all tensors from Rr . We call r the representation rank. Note that the representation rank refers to the representation by the parameters not to the represented tensor. Because Rr ⊂ Rr+1 , a tensor expressed with representation rank r can also be expressed by any larger representation rank (e.g., after adding zero terms).

(j) (vν )1≤j≤d,1≤ν≤r ,

7.3 r-Term Representation

239

The representation rank and the tensor rank are connected as follows: (i) If v ∈ V is represented by a representation rank r, then rank(v) ≤ r. (ii) Let r := rank(v). Then there exists a representation of v with representation rank r. However, finding this representation may be NP-hard (cf. Proposition 3.37). (j)

Remark 7.4. (a) The r-term representation is by no means injective; Qd e.g., vν may (j) be replaced by λj,ν vν with scalars λj,ν ∈ K satisfying j=1 λj,ν = 1. To reduce this ambiguity, we may consider the modified representation v=

r X

aν

ν=1

d O

vν(j)

with kvν(j) kVj = 1 for all 1 ≤ j ≤ d

j=1 (j)

(j)

with normalised vν and factors aν ∈ K. Still the sign of vν is not fixed (and (j) (j) in the case of K = C, vν may be replaced by λj,ν vν with |λj,ν | = 1 and Qd j=1 λj,ν = 1). For large d, this representation is not the best choice for practical use since over- or underflow of the floating point numbers aν may occur. A better normalisation4 is v=

r O d X

vν(j)

with kvν(j) kVj = kvν(k) kVk for all 1 ≤ j, k ≤ d.

ν=1 j=1

Another trivial ambiguity of the r-term representation is the ordering of the terms. Pr Nd (j) (b) The representation v = ν=1 j=1 vν is called essentially unique, if the scalar factors and the ordering of the terms mentioned above are the only ambiguities. Conditions implying essential uniqueness5 for d = 3 are given by Kruskal [203] (note his correcting remarks in [204, page 14]). This subject is also discussed by Kolda–Bader [197, §3.2], De Lathauwer [69], Stegeman–Ten Berge [273], Stegeman [270], and Chiantini–Ottaviani [55]. The general case d ≥ 3 is considered by Sidiropoulos–Bro [262]: Pr Nd (j) Theorem 7.5. If the representation v = ν=1 j=1 vν satisfies6 d X

(j)

krank(v1 , . . . , vr(j) ) ≥ 2r + d − 1

j=1

(cf. Definition 2.6), then the representation is essentially unique with r = rank(v). Remark 7.6. Even more, under certain conditions it is possible to construct the (unique) decomposition explicitly. In particular, this mapping is analytic. Details can be found in Sørensen et al. [266, 265, 267] and Domanov–Lathauwer [81, 82, 83]. 4

Cf. Mohlenkamp [229, Remark 2.1]. Tensors with a unique r-term representation are also called r-identifiable. 6 Lim [212] hints to a personal communication with H. Derksen stating that the above inequality cannot be improved. 5

7 r-Term Representation

240

The second formula in Example 3.49 shows that there are tensors which allow more than countably many, essentially different representations. As long as we are only interested in a cheap representation of tensors and cheap realisations of tensor operations, (essential) uniqueness is not relevant. This is different in applications in (j) which the vectors vν are used for an interpretation of certain data v.
(j) Remark 7.7. (a) The storage size for the parameter p = r, (vν )1≤j≤d, 1≤ν≤n is d X r-term size(vν(j) ). (7.8a) Nmem (p) = r · j=1 (j)

Here the representation is iterated: we need some representation of vν ∈ Vj and (j) the related storage size is denoted by size(vν ). More details about size(·) will follow in §7.5. (b) For the standard choice Vj = KIj with nj := #Ij the full representation of a (j) (j) vector vν ∈ Vj requires size(vν ) = nj . This results in r-term Nmem (p) = r ·

d X

nj .

(7.8b)

j=1

(c) Assume nj = n for all 1 ≤ j ≤ d. Then the characteristic size is r-term (p) = r · d · n. Nmem

(7.8c)

By Corollary 3.40, a tensor space isomorphism Φ : V → W is a bijection Rr (V)  Rr (W), i.e., the same representation rank r can be chosen for v ∈ V and Φ(v). The following remark is a reformulation of Lemma 6.1. Nd Pr Nd (j) is an element of Remark 7.8. v = ν=1 j=1 vν j=1 Uj with subspaces (j) Uj := span{vν : 1 ≤ ν ≤ r} for 1 ≤ j ≤ d. In particular, Ujmin (v) ⊂ Uj holds. In the next lemma we state two results involving the minimal subspaces Ujmin (v) Pr Nd (j) introduced in §6. Given some r-term representation v = ν=1 j=1 vν , the first (j)

(j)

statement shows that the vectors vν may be projected to vˆν ∈ Ujmin (v) and Pr Nd (j) v = ν=1 j=1 vˆν is still a correct representation. This implies that the search for (j)

candidates of vν may immediately be restricted to Ujmin (v). Lemma 7.9. For v ∈ (j)

(a) The vectors vi

Nd

j=1

Vj let Pj ∈ L(Vj , Vj ) be a projection onto Ujmin (v). (j)

in the r-term representation may be replaced with Pj vi : v=

r O d X i=1 j=1

(j)

vi

=

d r O X

(j)

P j vi .

(7.9)

i=1 j=1

(b) Equation (7.9) is also valid if some of the Pj are replaced with the identity.

7.4 Tangent Space and Sensitivity

Proof. Set P :=

Nd

j=1

241

u t

Pj and use v = Pv.

Pr Nd (j) The second result concerns representations v = ν=1 j=1 vν with minimal r = rank(v) and states that in this case equality Ujmin (v) = Uj must hold. Hence (j) vν ∈ Ujmin (v) is a necessary condition for a representation with minimal r. Pr Nd (j) Proposition 7.10. If v = ν=1 j=1 vν is a representation with r = rank(v), the subspaces from Remark 7.8 are Uj = Ujmin (v) (cf. (6.8a)). Proof. Let j ∈ {1, . . . , d}. Pj is some projection onto Ujmin (v). By Lemma 3.41, Pr N [j] [j] (j) v = ν=1 vν ⊗ vν holds with linearly independent vν ∈ a k6=j Vk . Apply Pj (regarded as a map from L(V, V), cf. Notation 3.57) to v. By Lemma 7.9b, Pr Pr (j) [j] (j) [j] (j) v = Pj v = ν=1 (Pj vν )⊗vν is valid implying 0 = ν=1(vν −Pj vν )⊗vν . [j] (j) (j) Linear independence of vν proves vν − Pj vν = 0 (cf. Lemma 3.66); i.e., all (j) vν belong to Ujmin (v). u t A consequence are the following conclusions from §6.8. Pr Nd (j) Remark 7.11. Suppose that v = ν=1 j=1 vν with r = rank(v). (a) If v satisfies a linear constraint ϕk (as defined in §6.8), then ϕk (vν(k) ) = 0 holds for all 1 ≤ ν ≤ r (cf. Tyrtyshnikov [285, Theorem 2.1]). T (b) Let V = n∈N V(n) be the intersection Banach spaces in §4.3.5. Then (j)

(nj )

v ∈ V(n) [v ∈ V] implies that vν ∈ Vj

(j)

(Nj )

[vν ∈ Vj

] for all 1 ≤ ν ≤ r.

(k)

(c) Let A be an unbounded operator on Vk . If v ∈ D(A) for A = A(k) ⊗ id[k] , (k) then vν ∈ D(A(k) ) (cf. Proposition 6.46).

7.4 Tangent Space and Sensitivity 7.4.1 Tangent Space (j)

Let the tuple v := (vν )1≤j≤d,1≤ν≤r determine the tensor v = ρr- term (r, v). Fix (j) (j) v, choose any wν ∈ Vj and set w = (wν )1≤j≤d,1≤ν≤r . For t ∈ R define v(t) := ρr-term (r, v + t · w). The chain rule yields the following derivative v(0) ˙ at t = 0 (derivative in direction w): r X d X dv(t) = vν(1) ⊗ . . . ⊗ vν(k−1) ⊗ wν(k) ⊗ vν(k+1) . . . ⊗ vν(d) . (7.10) dt t=0 ν=1 k=1

7 r-Term Representation

242

Definition 7.12. The tangent space Tr-term (v) of ρr-term (r, v) is the span of (7.10) for all w. Some of the properties of Tr- term (v) are mentioned below. Remark 7.13. (a) Different parameters v0 6= v00 representing the same tensor v = ρr-term (r, v0 ) = ρr-term (r, v00 ), may lead to different tangent spaces Tr-term (v0 ) 6= Tr-term (v00 ). (b) If the representation v = ρr-term (r, v) is essentially unique because of Theorem 7.5, also v0 = ρr-term (r, v0 ) with v0 is essentially unique in a sufficiently small neighbourhood of v. (c) v = ρr-term (r, v) ∈ Tr-term (v). (d) All x ∈ Tr-term (v) satisfy rank(x) ≤ rd and rank(x) ≤ 2r (cf. (9.11)). Proof. (a) r = 1 and v0 = (0, 0), i.e., v = 0 ⊗ 0 = 0, yields Tr-term (v0 ) = {0}. However, v00 = (a, 0), i.e., v = a ⊗ 0 = 0, yields Tr-term (v00 ) = span{a} ⊗ V2 . (b) The Kruskal rank does not decrease for small enough perturbations. (c) Set w := d1 v. (d) (7.10) proves rank(x) ≤ rd, and

v(t)−v(0) t

→

dv dw

shows rank(x) ≤ 2r. t u

7.4.2 Sensitivity In §7.1 we started with general comments about representations ρS (p1 , . . . , pn ) by parameters pj . From the numerical point of view it is important to know how ρS behaves under perturbations of pj . The derivative ∂ρS /∂pj may be called the sensitivity with respect to pj (cf. [137, §2]). There are several reasons why one is interested in these numbers. Since we are almost never working with exact data, the true parameter pj may be perturbed by rounding or other effects. These errors are amplified by the factor ∂ρS /∂pj . Whether such perturbations lead to dangerous effects for ρS (p1 , . . . , pn ) is detected by the sensitivity analysis. We start from the r-term representation v=

r O d X

vν(j) = ρr-term (r, v)

(7.11a)

ν=1 j=1 (j)

and set v ˜ = ρr-term (r, v + d), where the components dν perturbations: r O d   X v ˜= vν(j) + d(j) . ν

of d are considered as

ν=1 j=1

It is convenient to consider the relative error (j)

δν(j) :=

kdν k (j)

kvν k

.

(7.11b)

7.4 Tangent Space and Sensitivity

243

Nd Proposition 7.14. Let k·k be any crossnorm on V = k·k j=1 Vj (cf. Definition 4.37 Pr and (4.42)). Write the tensor v ∈ V in (7.11a) as v = ν=1 vν with the elementary Nd (j) tensors vν := j=1 vν . Then the following error estimate holds: " d # r Y X (j) (7.12) k˜ δν kvν k v − vk ≤ with δν := (1 + δν ) − 1. ν=1 (j)

For small δν

j=1

the first order approximation is δν ≈

(j) j=1 δν .

Pd

(j)

(1 ≤ j ≤ d). Induction on d will prove 

d  d d d  O Y Y

O (j) (j) (j) (j)

  (1 + + − v 1 d δ v − kvν(j) k. ≤ ) ν ν ν ν

Proof. The term vν is only affected by δν



j=1

j=1

j=1

j=1

For d = 1 the statement is trivial. Assume that the statement holds for d − 1, i.e., Nd for the products · · · . Then j=2

d

d 

 O

O (j) (j) (j) vν vν + dν −

j=1 j=1

( d ) d d

   O O O

(1) (1) (j) (1) (j) (j) (j) vν ≤ = vν +dν ⊗ + dν ⊗ vν vν +dν −

(4.42)

j=2 j=2 j=2

d

d d

O  O Y

(j) (1) (j) (j) + ≤ kvν(1) +d(1) kvν(j) k ≤ inductive v − kd +d v k k

ν ν ν ν ν

hypothesis j=2

≤ (1 + δν(1) ) kvν(1) k

Y d d Y kvν(j) k + δν(1) kvν(1) k kvν(j) k (1+δν(j) )−1

=

h



 i 1 + δν(j) − 1

j=1

j=2

j=2

j=2 d Y

j=2

j=2

Y d

d Y

kvν(j) k

j=1

u t

proves the statement.

The error estimate (7.12) should be combined with the stability estimate

r

r O

X Od X

d

(j) (j)

v ≤ κ v , (7.13)

j=1 i j=1 i

i=1

i=1

which will be discussed in more detail in Definition 9.20. The best (smallest) ˜: stability constant is κ = 1. Together, we can estimate the relative error of v k˜ v − vk ≤ κδ kvk

with δ := max{δν : 1 ≤ ν ≤ r}.

7 r-Term Representation

244

Since the condition κ may be as large as possible (cf. §9.4), a small relative (j) perturbation in vi does not guarantee a similarly small relative error of v. r Pd  (j) 2 (j) 2 Finally, we consider the ` norm . For general errors dν , the j=1 δν quantity δν =

d Y

(1 + δν(j) ) − 1 ≈

j=1

d X

δν(j)

j=1

√ qPr Pd (j) 2 2 is the best result so that k˜ v − vk ≤ δ with δ ≈ rd ν=1 kvν k j=1 (δν ) . (j)

The estimate improves a bit if the error dν

is a projection error.

Nd Remark 7.15. Let V = k·k j=1 Vj be a Hilbert tensor space with induced scalar product. Consider orthogonal projections Pj : Vj → Vj and the resulting errors (j) (j) (j) dν := (Pj − I)vν (vν in (7.11a)). Then the following error estimate holds: v

d

u r d

O

X X √ u

(j) 2 2 (j) t k˜ v − vk ≤ r kvν k (δν ) with kvν k = vν .

ν=1

j=1

j=1

Proof. We repeat the inductive proof. The first and second lines from above are Nd Nd (j) (j) (j) (1) (1) (1) + dν ) − = (vν + dν ) ⊗ · · · + dν ⊗ · · · . Since j=1 (vν j=1 vν (1)

(1)

(1)

(1)

(j)

(1)

(1)

vν +dν = P1 vν and dν = (Pj −I)vν are orthogonal and kP1 vν k ≤ kvν k, Nd Nd (j) (j) (j) the squared norm of is bounded by j=1 (vν + dν ) − j=1 vν kvν(1) k2

d

2

d

2 d

O

O

O

(j) (j) (j) (j) 2 (j) (vν + dν ) − vν + (δν ) vν .



j=2

j=2

Induction leads us to k˜ vν − vν k2 ≤ kvν k2 the sum over ν proves the assertion.

j=1

(j) 2 j=1 (δν ) .

Pd

Schwarz’ inequality of t u

7.5 Representation of Vj (j)

In (7.7a) v = ρr-term (r, (vν )j,ν ) is described as a representation of the tensor v. However, representations may become recursive if the parameters of the represen(j) tation need again a representation. In this case the involved vectors vi ∈ Vj must be implementable. If Vj = KIj with nj := #Ij , we might try to store the vector (j) vi by full representation (i.e., as an array of length nj ). But depending on the size (j) of nj and the nature of the vectors vi , there may be other approaches:

7.5 Representation of Vj

245

1. Sparse representation. If the vectors contain mostly zero components only these should be stored together with a description of the support. In applications from physics, functions may occur that are mainly locally supported, i.e., outside of a small neighbourhood of the centre the corresponding grid function can be set to zero (cf. [28]). 2. Tensorisation. Another approach has been mentioned in §5.3 and will be continued in §14: usual vectors from Knj may be interpreted as higher-order tensors. Under certain assumptions the storage of such tensor representations may be much cheaper than nj (possibly, it becomes O(log nj )). These considerations are in particular of interest if approximations are exceptable (see §9). 3. Hierarchical matrices. The spaces Vj may be matrix spaces: Vj = KIj ×Jj . Full representation of large-scale matrices is usually avoided. Possibly, we can (j) exploit the sparsity of the matrices vi ∈ KIj ×Jj . An efficient data-sparse (j) representation of matrices vi are hierarchical matrices (cf. Hackbusch [138]). In all these cases the required storage size may strongly deviate from dim(Vj ). Therefore we shall use the notation size(v (j) )

for v (j) ∈ Vj

as already done in (7.8a). (j)

Using special representations for vi , one has to take into account that also the operations involving these data must be implemented accordingly. If more than one representation format is used, all combinations of formats may occur in binary operations v op w. Remark 7.16. Many computations require scalar products hu, vij for u, v ∈ Vj . We denote its computational cost by Nj . The standard Euclidean scalar product in Vj = Knj costs Nj = 2nj − 1 arithmetic operations. Nj may be smaller than 2nj − 1 for certain representations of u, v ∈ Vj , while it may be larger for a scalar product hu, vij = v H Aj u, involving some positive-definite matrix Aj . Full representations are obviously impossible if dim(Vj ) = ∞. This happens, e.g., for Vj = C([0, 1]). Really general functions cannot be represented in a finite way. A remedy is the approximation, e.g., by interpolation. Such an approach will be studied in §10.4. In this chapter we discuss exact representations. Often, the involved (j) functions can be described by well-known function classes, e.g., vi ∈ C([0,1]) are polynomials, trigonometric polynomials or sums of exponentials exp(αx). Then we are led to the following situation: (j)

vi

=

X ν∈Bj

(j)

βν,i b(j) ν ∈ Uj ⊂ Vj

with Uj = span{b(j) ν : ν ∈ Bj }.

(7.14)

7 r-Term Representation

246 (j)

(j)

Now vi is represented by its coefficients (βν,i )ν∈Bj , while the basis functions (j) bν are fully characterised by the index ν ∈ Bj (e.g., the monomial xn is completely described by the integer n ∈ N0 ). In §8.2.6 we shall obtain (7.14) in a different way and call it the ‘hybrid format’. Representation (7.14) via a basis of a subspace Uj may even be of interest when Vj = KIj contains standard vectors (of large size nj = #Ij ). If more than one tensor shares the same subspaces Uj , the costly storage of the basis vectors (j) is needed only once, while the storage for the coefficients βν,i is of minor size. Bj ×Bj (j) Having precomputed the Gram matrix Γ of the scalar products ∈ K (j) (j) (j) Γν,µ := hbν , bµ i, we can reduce a scalar product hv, ui of v, u ∈ Uj to the scalar product hΓ βv , βu i of the respective coefficients in KBj . The resulting cost is Nj = 2(#Bj )2 + #Bj . A similar situation is discussed in the next remark. (j)

Remark 7.17. Assume that, according to (7.14), {vi : 1 ≤ i ≤ r} ⊂ Vj = KIj (j) is represented by coefficients βν,i , where rj := #Bj and nj = dim(Vj ). If the task is to compute the Gram matrix with entries Mνµ := hvν(j) , vµ(j) i,

Mj

the direct approach would cost 21 r(r + 1)Nj ≈ r2 nj operations. The computation of Γ (j) ∈ KBj ×Bj mentioned above requires rj2 nj operations. The Cholesky decomposition Γ (j) = L(j) L(j)H can be determined by 13 rj3 operations. The coef(j) (j) (j) ficients βν,i define the vectors βi := (βν,i )ν∈Bj (1 ≤ i ≤ r). Using



 hvν(j) , vµ(j) iVj = Γ (j) βν(j) , βµ(j) KBj = L(j)H βν(j) , L(j)H βµ(j) KBj , we compute all Mνµ by rj2 r + r2 rj operations. The total cost of this approach is rj2 nj + 13 rj3 + rj2 r + r2 rj . It is cheaper than the direct computation if (r2 − rj2 )nj > 13 rj3 + rj2 r + r2 rj . Remark 7.18. In the case of Vj = KIj , the standard method for producing a basis (j) (j) bν and the coefficients βν,i in (7.14) is the reduced QR decomposition. Form the (j) (j) matrix A := [v1 · · · vr ] and decompose A = QR

with Q ∈ KIj ×rj , rj := rank(A), R ∈ Krj ×r (j)

(cf. Lemma 2.22). Then vi

b(j) ν := Q•,ν ,

=

Pr

ν=1

Q•,ν Rν,i holds; i.e.,

(j)

βν,i := Rν,i , and Bj := {1, . . . , rj }.

The computational cost is NQR (nj , r).

7.6 Conversions between Formats

247

7.6 Conversions between Formats Nd In this section we consider the tensor space V := j=1 KIj with index sets of the size nj := #Ij . The tensor index set is I := I1 × . . . ×Id . The number   Yd nj / max ni , N := (7.15) j=1

1≤i≤d

appears in Lemma 3.45 as bound of the maximal rank in V. To simplify the notation, we assume that n1 = max{ni : 1 ≤ i ≤ d} and introduce the index set I0 := I2 × . . . × Id

(7.16)

0

of the size N = #I . In particular if r > N , it can be interesting to convert a tensor from r-term format into another one. We discuss the conversion from full format (abbreviation: F) or r-term format (Rr ) into other ones. At the right we give a summary of the costs using n := maxj nj .

conversion F → RN Rr → F RR → RN

arithmetical cost 0 2rnd for any r, 2Rnd if R > N.

7.6.1 From Full Representation into r-Term Format Assume that v ∈ V is given in full representation, i.e., by all entries v[i1 . . . id ] Qd full (ij ∈ Ij ). The associated memory size is Nmem = j=1 nj . Theoretically, a shortest Pr Nd (j) r-term representation v = ν=1 j=1 vν exists with r := rank(v) and memory P d r-term size Nmem = r j=1 nj . However, its computation is usually far too difficult (cf. Proposition 3.37). On the other hand, we constructively proved in Lemma 3.45 that the tensor rank is always bounded by N in (7.15). Remark 7.19. Given v ∈ V in full representation, the N -term representation with N in (7.15) is realised by ( d (1) (1) XO vi0 ∈ KI1, vi0 [k] := v[k, i02 , . . . , , i0d ] for j = 1, (j) v= vi0 with 0 (j) vi0 = e(j,ij ) ∈ KIj for 2 ≤ j ≤ d, i0 ∈I0 j=1 (7.17) where e(j,i) is the i-th unit vector in KIj , i.e., e(j,i) [k] = δik . Note that no arithmetic operations occur. Unfortunately, conversion to N -term format increases the storage requirement. Even if we do not associate any storage to the unit vectors e(j,i) , j ≥ 2, the vectors (1) vi0 , i0 ∈ I0 (cf. (7.16)), have the same data size as the fully represented tensor v. Because of the large memory cost, the transfer described above, is restricted to d = 3 and moderate nj . Further constructions, leading to smaller ranks than N will be discussed in §7.6.5. Even smaller ranks can be reached if we do not require an exact conversion, but admit an approximation (cf. §9).

7 r-Term Representation

248

7.6.2 From r-Term Format into Full Representation Conversion from r-term format into full representation can be of interest if r > N Pr Nd (j) (j) since then the full format is cheaper. Given v = ν=1 j=1 vν , vν ∈ KIj , Pr Qd (j) the expressions v[i] = ν=1 j=1 vν [ij ] must be evaluated for all i ∈ I. d Q Lemma 7.20. Conversion of an r-term tensor into a full tensor requires 2r nj j=1 operations (plus lower order terms).

Qd (j) [1] Proof. Note that vν [i0 ] := j=2 vν [i0j ] can be obtained for all i0 ∈ I0 (cf. (7.16)) Qd by r(d − 1) j=2 nj operations. This is a lower order term compared with the Pr [1] (1) t u operation count for ν=1 vν [i1 ] · vν [i0 ].

7.6.3 From r -Term into N -Term Format for r > N Pr Nd (j) Here we assume that v = ν=1 j=1 vν ∈ V is given with r > N , where N in (7.15) is the upper bound of the maximal rank (cf. (3.23)). Such a situation may occur after operations, where the number of terms is the product of those of the operands (see, e.g., §13.5). (j)

0

Remark 7.21. Let wi0 = ej,ij for 2 ≤ j ≤ d be the unit vectors in (7.17). Then Pr Nd (j) the tensor v = ν=1 j=1 vν is equal to v=

d XO i0 ∈I0 j=1

(j) wi0

∈ RN

with

(1) wi0

:=

" d r Y X ν=1

# vν(j) [i0j ]

vν(1) .

j=2

The number of terms is #I0 = N . The computational cost is 2 r lower order terms.

Qd

j=1

nj plus

The performed operations are identical to those in §7.6.3, only the interpretation of the result as N -term format is different. Another conversion for the particular case d = 3 is related to [186, Remark 2.7]. Below, e(1,i) (i ∈ I1 ) is again the i-th unit vector in KI1 . Remark 7.22. Without loss of generality, assume that n1 = minj nj . Write v as P [1] [1] [1] (1,i) ⊗ wi with tensors wi ∈ V2 ⊗ V3 defined by wi [i2 , i3 ] := v[i, i2 , i3 ]. i∈I1 e Pr N3 (j) [1] For v = ν=1 j=1 vν , all wi have the form

7.6 Conversions between Formats [1]

wi =

249 r X

vν(1) [i] · vν(2) ⊗ vν(3) .

ν=1 [1]

Evaluation of wi for all i in full format costs 2rn1 n2 n3 + (r − n1 )n2 n3 operations. Note that the leading term is the same as above. The reduced singularPmi [1] value decomposition wi = µ=1 xi,µ ⊗ yi,µ is an mi -term representation, where [1] the (matrix and tensor) rank mi = rank(wi ) is bounded by min{n2 , n3 }. The cost of the SVDs is O(n1 · NSVD (n2 , n3 )) = O(n4 ), where n = maxj nj . Because r > N = O(n2 ), O(n4 ) is a lower order term compared with 2rn1 n2 n3 . Thus, this method needs almost the same computational work, while the resulting N 0 -term representation is X X mi ≤ n1 min{n2 , n3 } = N. e(1,i) ⊗ xi,µ ⊗ yi,µ with N 0 := v= i∈I1

(i,µ)

7.6.4 Sparse-Grid Approach The sparse-grid approach is used to interpolate functions in higher spatial dimensions or it serves as ansatz for discretising partial differential equations. For a review of sparse grids we refer to Bungartz–Griebel [47] and Garcke [110]. Here we only sketch the main line and its relation to tensor representations. Nd To simplify the notation, we assume that the tensor space V = j=1 Vj uses identical spaces V = Vj , which allow a nested sequence of subspaces: V = V(`) ⊃ V(`−1) ⊃ . . . ⊃ V(2) ⊃ V(1) .

(7.18)

Typical examples are finite-element spaces V(`) of functions, say, on the interval [0, 1] using the step size 2−` . For sparse grids in the Fourier space compare Sprengel [268]. While the usual uniform discretisation by V = ⊗d V(`) has a dimension of order 2`d , the sparse-grid approach uses the sum of tensor spaces X

Vsg,` = d P

`j = `+d−1

d O

V(`j ) .

(7.19)

j=1

j=1

These spaces lead to the interpolation error7 by O(2−2` `d−1 ) for suitably regular functions (cf. [47, Theorem 3.8]). This is to be compared with dim(Vsg,` ) ≈ 2` `d−1 (cf. [47, (3.63)]). The basis vectors in Vsg,` are elementary tensors Nd (j) (j) j=1 bk,`j , where `j denotes the level: bk,`j ∈ V(`j ) . Since the number of terms is limited by the dimension of the space Vsg,` , the tensor v ∈ Vsg,` belongs to Rr with r = dim(Vsg,` ) ≈ 2−` `d−1 . 7

Any Lp norm with 2 ≤ p ≤ ∞ can be chosen.

7 r-Term Representation

250

For the practical implementation we use hierarchical bases (cf. [47, §3] and [141, §8.9.11]). In V(1) we choose, e.g., the standard hat function basis b1 := (bi )1≤i≤n1. The basis b2 of V(2) is b1 enriched by n22 hat functions from V(2) . The latter additional basis functions are indexed by n1 + 1 ≤ i ≤ n1 + n22 = n2 . In general, the basis bλ ⊂ V(λ) consists of bλ−1 and additional n2λ hat functions of V(λ) . The index `j corresponds to the dimension nj = 2`j of V(`j ) . The additive side Pd condition j=1 `j ≤ L := ` + d − 1 in (7.19) can be rewritten as d Y

nj ≤ N := 2L .

j=1

It follows from ij ≤ nj that the involved indices of the basis functions bij ∈ V(`j ) Qd satisfy j=1 ij ≤ N . For ease of notation, we replace Vsg,` with Vsg := span

 d O 

d Y

bij :

j=1

ij ≤ N

 

.



j=1

Since Vsg ⊃ Vsg,` , the approximation is not worse, while dim(Vsg ) has the same Qd asymptotic behaviour 2−` `d−1 as dim(Vsg,` ). The inequality j=1 ij ≤ N gives rise to the name ‘hyperbolic cross’. Remark 7.23. The typical hyperbolic cross approach is the approximation of a P Nd function f with the (exact) series expansion f = i∈Nd vi j=1 φij by X

fN := Qd

j=1 ij ≤N

vi

d O

φij .

j=1

The behaviour of the number σd (N ) of tuples i involved in the summation with respect to N is
σd (N ) = O N logd−1 (N ) . (7.20) In the previous example O(N −2 logd−1 (N )) is the accuracy of fN . The following table shows the values of σ2 (N ) and σ10 (N ) for different N , as well as values of σd (10) for increasing d: d=2 d = 10 N = 10

N σd (N ) N σd (N ) d σd (N )

2 3 2 11 2 27

4 8 16 32 64 128 256 8 20 50 119 280 645 1466 4 8 16 32 64 128 256 76 416 2 056 9 533 41 788 172 643 675 355 3 5 10 20 50 100 1000 53 136 571 2 841 29 851 202 201 170 172 001

251

7.6 Conversions between Formats

7.6.5 From Sparse Format into r-Term Format Finally, we consider the sparse format v = ρsparse (˚ I, (vi )i∈˚ I ) in (7.5). By definiP Nd (j) v is an r-term representalatter expression holds. The tion, v = i∈˚ b i j=1 ij I ˚ tion of v with r := #I nonzero terms. The function f from Remark 7.23 is a tensor isomorphically represented by the coefficient v ∈ ⊗d KN , where the tensor space is to equipped with the suitable norm. The tensor vsg corresponding to the approximation fN has sparse format: d ˚ I, (vi )i∈˚ vsg = ρsparse (˚ I ) with I = {i ∈ N :

d Y

ij ≤ N }.

j=1

This ensures an r-term representation with r := #˚ I. The representation rank r can I. Here we follow the idea from the be reduced because of the special structure of ˚ proof of Lemma 3.45: for fixed ij with j ∈ {1, . . . , d}\{k} we can collect all terms for ik ∈ N in ! ! ! d k−1 O O X (j) (j) (k) (7.21) ∈ R1 . bij ⊗ vi bik ⊗ bij j=1

ik

j=k+1

The obvious choice of (i1 , . . . , ik−1 , ik+1 , . . . , id ) are indices such that the sum P ik contains as many nonzero terms as possible. First, we discuss the situation for d = 2. Figure 7.1 shows the pairs (i1 , i2 ) ∈ ˚ I with 16 Qd For the first choice N i ≤ = 16. j j=1 k = 1 and i2 = 1, the indices i involved in (7.21) are contained in the first column of height 16. The second choice k = 2 and i1 = 1 leads to the lower row. Here the sum 8 in (7.21) ranges from 2 to 16 since v[1, 1] belongs to the previous column. Similarly, 5 4 two additional columns and rows correspond 3 to i2 = 2, 3 and i1 = 2, 3. Then we are 21 16 1 2 3 4 5 8 left with a single index i = (4, 4) so that we I into seven groups. Each Fig. 7.1 Sparse-grid indices have decomposed ˚ group gives rise to one elementary tensor (7.21). This finishes the construction of a 7-term representation of vsg . Obviously, for general N we can construct a representation in Rr with √ √ r ≤ 2b N c ≤ 2 N √ √ and even r ≤ 2 N − 1 if N ∈ N.

7 r-Term Representation

252

For general d, the decomposition of ˚ I can be achieved as follows. Let d−1 n o Y d−1 tj ≤ N T := (t1 , . . . , td−1 ) ∈ Nd−1 : max{tj } · j=1

j=1

be a set of (d−1)-tuples. For each t := (t1 , . . . , td−1 ) ∈ T and 1 ≤ k ≤ d define   d−1 ˚ It,k := (t1 , . . . , tk−1 , ik , tk , . . . , td−1 ) ∈ ˚ I with ik ≥ max{tj } . j=1

Sd S I and I. For a proof, take any i = (i1 , . . . , id ) ∈ ˚ We claim that t∈T k=1˚ It,k = ˚ let k and m be indices of the largest and second largest ij , i.e., ik ≥ im ≥ ij

for all j ∈ {1, . . . , d}\{k, m} with k 6= m.

Qd Q Inequality im ≤ ik and i ∈ ˚ I imply that im · j6=k ij ≤ j=1 ij ≤ N . Therefore the tuple t := (i1 , . . . , ik−1 , ik+1 , . . . , id ) belongs to T and shows that i ∈ ˚ It,k . Sd S I ⊂ t∈T k=1˚ This proves ˚ It,k , while direction ‘⊃’ follows by the definition of ˚ It,k . We conclude that {˚ It,k : t ∈ T, 1 ≤ k ≤ d} is a (not necessarily disjoint) I, whose cardinality is denoted by τd (N ). Each8 set ˚ It,k gives decomposition of ˚ rise to an elementary tensor (7.21) and proves vsg ∈ Rr , where9 r := τd (N ) ≤ d · #T. It remains to estimate #T . ForQt := (t1 , . . . , td−1 ) ∈ T let m be an index√with 2 tm = maxd−1 N. j=1 {tj }. From tm j6=m tj ≤ N we conclude that 1 ≤ tm ≤ In the following considerations, we distinguish the cases tm ≤ N 1/d

and N 1/d < tm ≤ N 1/2 .

Inequality tm ≤ N 1/d implies tj ≤ N 1/d , and the condition d−1

max{tj } · j=1

d−1 Y

tj ≤ N.

j=1

The number of tuples t ∈ T with maxj {tj } ≤ N 1/d is bounded by N (d−1)/d . Now we consider the case N 1/d < tm ≤ N 1/2 . The remaining components tj Q (j 6= k), satisfy j6=m tj ≤ N/t2m . We ignore the condition tj ≤ tm , and ask Q for all (d − 2)-tuples (tj : j ∈ {1, . . . , d − 1}\{m}) with j6=m tj ≤ N/t2m .
Its number is σd−2 (N/t2m ) = O tN2 logd−3 ( tN2 ) (cf. (7.20)). It remains to bound m

m

Since the sets ˚ It,k may overlap, one must take care that each vi is associated to only one ˚ It,k . τd (N ) < d · #T may occur, since ˚ It,k = ˚ It,k0 may hold for k 6= k0 . An example is the decomposition from Figure 7.1, where τ2 (16) = 7. 8 9

7.7 Representation of (Anti-)Symmetric Tensors

the sum Z

N 1/2

N 1/d

N N 1/d
P

253

logd−3 ( N t2 ). Instead, we consider the integral

N  √ N logd−3 2 dx = N 2 x x

1

Z

1

N 2−d

logd−3 (y)dy = O N

d−1 d


logd−3 (N ) .

1

Therefore any tensor v = ρsparse (˚ I, (vi )i∈˚ be written as v ∈ Rr with I can
d−3 (d−1)/d representation rank r ≤ O N log (N ) . This proves that although #˚ I is not bounded by O(N ), the rank is strictly better than O(N ). Qd Proposition 7.24. For an index set ˚ I ⊂ {i ∈ Nd : j=1 ij ≤ N }, any tensor v = ρsparse (˚ I, (vi )i∈˚ I ) can be explicitly converted into v ∈ Rr with a representation rank
r = τd (N ) ≤ O N (d−1)/d logd−3 (N ) . Numerical tests show that the asymptotic behaviour sets in rather late. The fitted exponent γd (N ) := log2 (τd (N )/τd (N/2)) for N = 2n . should converge to (d − 1)/d. n 3 6 9 12 15 18 21 24 27 30 γ3 (2n ) 1.0 0.88 0.78 0.73 0.71 0.693 0.684 0.679 0.675 0.672 → γ4 (2n ) 1.4 1.15 0.96 0.89 0.85 0.825 0.806 0.794 0.784 0.777 →

2 3 3 4

Finally, we compare the quantities σd (N ) (cardinality of the sparse grid) and τd (N ) for d = 3 and d = 6: N 10 100 1 000 10 000 100 000 1 000 000 σ3 (N ) 53 1 471 29 425 496 623 7 518 850 106 030 594 τ3 (N ) 12 102 606 3265 16542 81 050 σ6 (N ) 195 14 393 584 325 17 769 991 439 766 262 τ6 (N ) 51 2 047 36 018 502 669 5 812 401 59 730 405

7.7 Representation of (Anti-)Symmetric Tensors Symmetric and antisymmetric tensors belong to V = ⊗d V. As any tensor from V, also (anti-)symmetric tensors can be represented by the general r-term format ρr-term in (7.7a). Here we look for representations such that only (anti-)symmetric tensors are involved, i.e., the corresponding ρ maps the parameters onto S(V ) or A(V ), respectively (cf. Definition 3.74). More details are in Hackbusch [143].

7 r-Term Representation

254

7.7.1 Sums of Symmetric Rank-1 Tensors Pr Any symmetric tensor s ∈ Sd (V ) can be written as a sum s = i=1 ⊗d vi with vi ∈ V. The smallest r is the symmetric rank from Definition 3.83. We modify Definition 7.3 as follows. Definition 7.25. For variable r ∈ N0 , the r-term representation is explained by the mapping r   X symR1 ρr-term ⊗d vν ∈ Sd (V ) with r ∈ N0 , vν ∈ V. r, (vν )1≤ν≤r := ν=1 symR1 Obviously, the representation rank r for s = ρr-term (r, (vν )1≤ν≤r ) satisfies r ≥ ranksym (s). The instability described in §9.4.3 also holds for the subset Sd (V ) ⊂ ⊗d V. d The typical example is s := dλ ⊗d (s + λt)|λ=0 for linearly independent s, t ∈ V. s is the symmetrisation of t ⊗ (⊗d−1 s) which has  rank d (cf. Lemma  3.46), while s is also the limit of the symmetric rank-2 tensors ⊗d (s + εt)−⊗d s /ε as ε → 0.

7.7.2 Indirect Representation An (anti-)symmetric tensor v ∈ Sd (V ) or v ∈ Ad (V ) may be represented by a general tensor w ∈ V := ⊗d V with the property PS (w) = v or PA (w) = v, where PS (PA ) is the symmetriser (alternator) defined in Proposition 3.76. The Pr Nd (j) representation of w ∈ V uses the r-term format: w = i=1 j=1 wi . For instance, v = a ⊗ a ⊗ b + a ⊗ b ⊗ a + b ⊗ a ⊗ a ∈ Sd (V ) is represented by w = 3a ⊗ a ⊗ b. This example indicates that w may be of a simpler form than the symmetric tensor v = PS (w). Definition 7.26. In the symmetric case, the indirect representation uses the r-term format in Definition 7.3 via      sym(ind) (j) (j) ρr-term r, (vν )1≤j≤d = PS ρr-term r, (vν )1≤j≤d . 1≤ν≤r

1≤ν≤r

Antisymmetric tensors are represented by anti(ind) ρr-term



r, (vν(j) )1≤j≤d 1≤ν≤r



 = PA



ρr-term r, (vν(j) )1≤j≤d 1≤ν≤r



.

This approach is proposed by Beylkin–Mohlenkamp–P´erez [32]. The cost (storage size) of the representation is only one aspect. Another question concerns the tensor operations. The addition is easy to perform since, by linearity of PS , the sum of v0 = PS (w0 ) and v00 = PS (w00 ) is represented by w0 + w00 . Similar in the antisymmetric case. The computation of the scalar product hv0 , v00 i of v0 , v00

7.7 Representation of (Anti-)Symmetric Tensors

255

in Sd (V ) or Ad (V ) is discussed in §4.7.1. In the antisymmetric case the L¨owdin rule can be applied involving determinants (cf. (4.90)). If the respective representation ranks of v0 and v00 are r0 and r00 , the cost amounts to O(r0 r00 d3 ). In the symmetric case, NP-hard permanents are needed. The summation within the r-term format does not require computational work, but increases the representation rank r. The required truncation is the (difficult) truncation discussed in §9.6. Let V = KI×I be the matrix space. Then Sd (V ) contains the symmetric Kronecker products.10 The multiplication of A ∈ Sd (V ) by a tensor v ∈ Sd (V ) [v ∈ Ad (V )] yields Av ∈ Sd (V ) [Av ∈ Ad (V )]. Given an Rk representation of A and the indirect representation v = PS (w) [v = PA (w)], the product Av has the indirect representation Av = PS (Aw) [Av = PA (Aw)]. A problem arises if also A ∈ Sd (V ) is indirectly represented: A = PS (B). Unfortunately, Av is not the (anti)symmetrisation of Bw. The computation of Av requires that either the symmetric tensor A or the (anti)symmetric tensor11 v must be constructed explicitly, which contradicts the intention of the indirect representation. The Hadamard product v0 v00 and the convolution v0 ? v00 lead to the same difficulties. Let v ∈ Sd (V ) be a tensor with rv = ranksym (v). The indirect representation uses the rw -term representation of some w ∈ V. The gain is characterised by the ratio rv /rw , where rw is the smallest possible rank. rw is known as the Chow rank (cf. Torrance [282]): rankChow (v) = min {rank(w) : w ∈ V, v = S(w)} . The larger rv /rw is the more effective is the use of the indirect representation. The Nd maximal value12 rv /rw = 2d−1 is obtained for v = S( j=1 w(j) ) with linearly independent w(j) (cf. Ranestad–Schreyer [248]). The minimal value rv /rw = 1 occurs for v = w = ⊗d v. For K = C, almost all tensors have a generic rank (cf. §3.2.6.5). Therefore we may compare the generic ranks. Abo–Vannieuwenhoven [2] state that &
' &
' dim(V )−1+d dim(V )−1+d rv (dim(V ) − 1)d + 1 d d = / ≈ rw dim(V ) (dim(V ) − 1)d + 1 dim(V ) is the generic ratio. For large dimensions of V the ratio approaches d. In the case of antisymmetric tensors the corresponding ratio is & '

' & dim(V ) dim(V ) (dim(V ) − d)d + 1 rv d d = / ≈ →d rw dim(V ) (dim(V ) − d)d + 1 dim(V ) (cf. Abo–Ottaviani–Peterson [1]). 10 11 12

The symmetry refers to the tensor (e.g., A ⊗ B + B ⊗ A), not to the (Kronecker) matrix. In that case A v = PS (B v) [A v = PA (B v)] can be applied. The author thanks Anna Seigal for providing the results and literature about the ratio rv /rw .

7 r-Term Representation

256

7.8 Modifications (d)

(1)

So far, the terms vν ⊗ . . . ⊗ vν of the r-term representation has been completely unrelated. Instead, one can impose some orthogonality conditions. Kolda [196] introduces orthogonal, strongly orthogonal, and completely orthogonal r-term representations, which give rise to three new kinds of rank. A related property is used in Sørensen et al. [266]. Here we shall not make use of these constructions. Restricting
Vj to certain subsets Aj ⊂ Vj , we can define a modified r-term format Rr (Aj )dj=1 by ( Rr (Aj )dj=1




:=

r X

) vν(1)

⊗ ... ⊗

vν(d)

:

vν(j)

∈ Aj

(r ∈ N0 ).

ν=1

Examples of Aj are • Vj = Rnj , Aj = {v ∈ Vj : vi ≥ 0}. Hence Aj contains the nonnegative vectors (see Vavasis [293] and Arora et al. [9] for d = 2). • Vj = Lp (Ij ), then Aj = {v ∈ Vj : v(x) ≥ 0 for x ∈ Ij } describes the nonnegative functions. • Vj = Cnj ×nj , Aj = {M ∈ Vj : M Hermitian matrix}. • Vj = Cnj ×nj , Aj = {M ∈ Vj : M positive semidefinite} We shall refer to these modifications in §9.7, §9.8.1, and §9.8.2.5.1.

Chapter 8

Tensor Subspace Representation

Abstract This chapter is devoted to the tensor subspace representation which is We use the term ‘tensor subspace’1 for the also called the Tucker representation. Nd tensor product U := a j=1 Uj of subspaces Uj ⊂ Vj . Obviously, U is a subNd space of V := a j=1 Vj . For d = 2, r-term and tensor subspace representations are identical. Therefore, both approaches can be viewed as extensions of the concept of rank-r matrices to the tensor case d ≥ 3. The resulting set Tr introduced in Section 8.1 will be characterised by a vector-valued rank r = (r1 , . . . , rd ). Since by definition, tensors v ∈ Tr are closely related to subspaces, their descriptions by means of frames or bases is of interest (see Section 8.2). Differently from the r-term format, algebraic tools like the singular-value decomposition can be applied and lead to a higher order singular-value decomposition (HOSVD), which is a quite important feature of the tensor subspace representation (cf. Section 8.3). Moreover, HOSVD yields a connection to the minimal subspaces from Chapter 6. In Section 8.5 we compare the formats discussed so far and describe conversions between the formats. In a natural way, a hybrid format appears using the r-term format for the coefficient tensor of the tensor subspace representation (cf. §8.2.6). Section 8.6 deals with the problem of joining two representation systems, as it is needed when we add two tensors involving different tensor subspaces.

8.1 The Set Tr The tensor subspace representation can be equivalently described by subspaces or by concrete coefficients and bases. We start with the former description. The latter one will follow in (8.5b). Throughout this chapter, v ∈ V is a fixed algebraic tensor in d O Vj . V= a j=1 1

‘Tensor subspace’ is to be understood as ‘subspace and tensor space’.

© Springer Nature Switzerland AG 2019 W. Hackbusch, Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics 56, https://doi.org/10.1007/978-3-030-35554-8_8

257

258

8 Tensor Subspace Representation

In the following we make use of possibly lower dimensional subspaces Uj ⊂ Vj such that d O (8.1) with dim(Uj ) ≤ rj v ∈ U := a Uj j=1

holds (cf. (6.1)). In §8.2 we shall represent v by suitable frames or bases of Uj . The optimal choice for Uj ⊂ Vj are the minimal subspaces Ujmin (v) described in §6. Note that in this chapter only exact representations are considered. Approximations in possibly even smaller subspaces will be discussed in Chapter 10. Above, for a fixed v, we were looking for a subspace family {Uj } with property (8.1). Now we reverse the viewpoint and fix the dimension bounds rj of Uj , in r := (r1 , . . . , rd ) ∈ N0d , and ask for all v satisfying (8.1) for subspaces Uj with dim(Uj ) ≤ rj . Note that the subspaces Uj may depend on v. The dimensions rj play a similar role to the parameter r of the r-term representation. Therefore we refer to r as the tensor subspace representation rank. Since it is vector-valued, it cannot be confounded with the (scalar) tensor rank in §3.2.6. Definition 8.1 ( Tr ). Let V :=

a

Nd

j=1

Vj , fix2 r := (r1 , . . . , rd ) ∈ N0d , and set

Tr := Tr (V) := {v ∈ V : rankj (v) ≤ rj for 1 ≤ j ≤ d} .

(8.2)

For v ∈ Tr we say that v possesses an (r1 , ..., rd )-tensor subspace representation.3 The symbol Tr is used if the reference to the underlying tensor space V is obvious; otherwise Tr (V) is preferred. An equivalent definition is d [ O Uj . Tr = Uj ⊂Vj subspaces with dim(Uj )≤rj (1≤j≤d)

j=1

The letter T may also be read as ‘Tucker format’ (cf. Tucker [284, formula (17)]). The set Tr corresponds to Rr in (3.18). The parameters r1 , . . . , rd are not necessarily the optimal ones; i.e., the bound rj may be larger than rankj (v). Exercise 8.2. (a) If rj = 0 for some j, then Tr = R0 = {0} is the trivial subspace. (b) Tr = R1 holds for all 0 < r ∈ N0d with rj = 1 for at least d − 1 directions j. (c) If d = 2, Tr = Rr holds for pairs r = (r1 , r2 ) with r := min{r1 , r2 } ∈ N0 . Q (d) Tr = Rr = V holds if and only if rj ≥ min{dim(Vj ), k6=j dim(Vk )} and r ≥ rmax (cf. §3.2.6.5). Exercise 8.2 describes the only cases for which the sets Tr and Rr coincide. Relations and conversions between both formats will be discussed in §8.5. 2 3

Obviously, only integer rj ≤ dim(Vj ) are of interest. By formal reasons, we admit any rj ∈ N0 . Accordingly, this format is also called the multilinear rank-(r1 , . . . , rd ) decomposition.

8.1 The Set Tr

259

Tr satisfies similar properties to Rr in (3.19a):4 {0} = Tr Tr ⊂ Ts Tr + Ts ⊂ Tr+s

if rj = 0 for at least one j, for r ≤ s, for all r, s ∈ Nd0 .

(8.3)

Note that Tr is not a subspace! Two different Nd tensors v, w ∈NTdr may belong to different systems of subspaces: v ∈ j=1 Wj . In the j=1 Uj and w ∈ worst case, Uj ∩ Wj = {0} holds and the sum v + w requires the subspace Uj + Wj with dim(Uj + Wj ) = dim(Uj ) + dim(Wj ) for its tensor subspace representation, proving the last line of (8.3). Differently from (3.19b), Tr + Ts is in general a proper subset of Tr+s . Exercise 8.3. Prove that Tr = Ts holds for rQ≤ s if and only if each j ∈ {1, . . . , d} satisfies either rj = sj or min dim(Vj ), k6=j min {sk , dim(Vk )} ≤ rj < sj . Summarising the results of §6.3, we can state: Remark 8.4 (Tucker rank). Given v ∈ V, there is a minimal representation rank r = rmin (v) ∈ N0d with v ∈ Tr . This rmin (v) has the components
rj = rankj (v) = dim Ujmin (v) (cf. (5.6b)). The corresponding subspaces in (8.1) are Uj := Ujmin (v). The tuple rmin (v) is called the tensor subspace rank, multilinear rank, or Tucker rank of v (although this rank was introduced by Hitchcock [164] much earlier). Example 8.5. (a) Let Ppj ⊂ Vj := L2 ([0, 1]) be the subspace of polynomials of Nd degree at most pj . All multivariate polynomials f (x1 , . . . , xd ) ∈ V := a j=1 Vj Nd with polynomial degree ≤ pj with respect to xj belong to U := j=1 Ppj ⊂ Tr with rj = pj + 1, i.e., Uj = Ppj holds. (b) The particular polynomial f (x, y, z) = xz + x2 y belongs to T(2,2,2) , involving the subspaces U1 := span{x, x2 }, U2 := span{1, y}, U3 := span{1, z}. The next property of Tr will become important for approximation problems. Nd Lemma 8.6. Let V = k·k j=1 Vj be a Banach tensor space with the property (6.14). Then the subset Tr ⊂ V is weakly closed. The same statement holds under the condition (6.24) of Theorem 6.44. Proof. Let vn ∈ Tr be a weakly convergent sequence with vn * v ∈ V. From vn ∈ Tr we infer that Ujmin (vn ) has a dimension not exceeding rj . By Theorem t u 6.29, dim(Ujmin (v)) ≤ rj follows, implying v ∈ Tr (cf. Remark 6.36). Q Below we use the abbreviations nj := dim(Vj ) ∈ N∪{∞} and n[j] := k6=j nk Nd for V = a j=1 Vj 6= {0}. 4

Inequalities s ≥ r for vectors from Nd 0 are understood componentwise: sj ≥ rj for all 1 ≤ j ≤ d.

260

8 Tensor Subspace Representation

Lemma 8.7. (a) Tr (V) = V holds if and only if 5 rj ≥ min{nj , n[j] } < ∞ for all 1 ≤ j ≤ d. (b) Assume Tr (V) 6= V. Then the interior of Tr (V) is empty; i.e., each neighbourhood of v ∈ Tr contains tensors not belonging to Tr . Proof. (a) There are tensors with rankj (v) = min{nj , n[j]} for all j (cf. Remark 5.8). (b) According to Part (a), there is one direction j with rj < min{nj , n[j] }. Choose any v ∈ Tr . We have to demonstrate that any neighbourhood contains elements outside of Tr . Set r := rankj (v) and consider the matricisation ∼ Mj (v) ∈ Vj ⊗ V[j] . Note that r ≤ rj < min{nj , n[j] }. There are linearly v = [j] (j) independent {vν : 1 ≤ ν ≤ r} and {vν : 1 ≤ ν ≤ r} such that v=

r X

vν(j) ⊗ vν[j] .

ν=1

m := 1+rj −r satisfies m ≥ 1 and r+m ≤ min{nj , n[j] }. From r+m ≤ n[j] = [j]

dim(V[j] ) we conclude that there are m linearly independent tensors wµ ∈ V[j] [j]

[j]

[j]

0 and m linearly independent functionals Ψρ ∈ V[j] with Ψρ (vν ) = 0 (1 ≤ ρ ≤ m, [j] [j] 1 ≤ ν ≤ r) but Ψρ (wµ ) 6= 0 (1 ≤ ρ, µ ≤ m). Because r + m ≤ nj , there are (j) m linearly independent vectors wµ ∈ Vj \Ujmin (v). Set

v(λ) := v + λw

with w :=

m X

wµ(j) ⊗ wµ[j] for λ > 0.

µ=1 [j]

(j)

0 in (6.7b), we obtain Ujmin (v(λ)) ⊂ span{wµ : 1 ≤ µ ≤ m}. Using Ψρ ∈ V[j] [j] [j] [j] 0 with Φν (vκ ) = δνκ (1 ≤ ν, κ ≤ r) On the other hand, there are Φν ∈ V[j] (j) [j] [j] but Φν (wµ ) = 0. This proves Ujmin (v(λ)) ⊂ span{vν : 1 ≤ ν ≤ r}. Together, we obtain that

rankj (v(λ)) = r + m = rj + 1. Hence v(λ) ∈ / Tr holds for all λ > 0, proving the second assertion.

t u

Now we are in position to prove Proposition 4.36 which states that the algebraic tensor space is a set of first category (cf. Uschmajew [289]). Proof of Proposition 4.36. By assumption, at least two spaces of V1 , . . . , Vd are infinite dimensional, which excludes the case (a) of Lemma 8.7 and part (b) applies showing that all sets Tr have no interior point. Furthermore, the algebraic tensor space Valg is the union of the countably many sets Tr , r ∈ Nd . As stated in Lemma 8.6, Tr is weakly closed, which implies that Tr is closed. Hence Valg is t u of first category. 5

This includes the case that nj = ∞ holds for at most one index j.

8.2 Tensor Subspace Formats

261

8.2 Tensor Subspace Formats Nd The previous characterisation of a tensor by subspaces Uj ⊂ Vj with v ∈ j=1 Uj corresponds to a more theoretical level of linear algebra. The practical realisation requires a description of the subspaces Uj by bases or, more general, by frames6 . Even when a basis (in contrast to a frame) is the desired choice, there are intermediate situations in which frames cannot be avoided (cf. §8.6). Concerning the choice of the basis (frame) of Uj , we distinguish three levels: 1. Use of a general frame or basis (cf. §8.2.1). 2. An orthonormal basis is always a good starting point for stable numerical computations (cf. §8.2.4). 3. A special orthonormal basis is the HOSVD basis discussed in §8.3. After fixing a basis, we obtain the explicit representation (8.5b) which often is given as the primal definition of the Tucker format. To change a basis into another one, we study transformations in §8.2.2. Concerning the (non)uniqueness of the tensor subspace representation we refer to §8.4.1.

8.2.1 General Frame or Basis In this section, the underlying spaces Vj are (abstract) vector spaces. For instance, Vj = L2 (Ωj ) may be a function space, while Uj ⊂ Vj is a finite-dimensional subspace spanned by concrete functions. The case Vj = KIj will be discussed in §8.2.3. Then the quantities Bj , B, Bα introduced below become (Kronecker) matrices.

8.2.1.1 Notation By Lemma 6.1 we can suppose that dim(Uj ) < ∞. We represent the subspace by (j)

Uj = span{bi (j)

The bi 6

: 1 ≤ i ≤ rj }.

are called the frame vectors or generating vectors. They form the rj -tuple

The frame is a system of vectors generating the subspace without assuming linear independence. When the term ‘frame’ is used, this does not exclude the special case of a basis; otherwise, we use (j) the term ‘proper frame’. Note that a frame cannot be described by a set {bν : 1 ≤ ν ≤ rj }, (j) (j) since bν = bµ may hold for ν 6= µ.

262

8 Tensor Subspace Representation

i h (j) (j) ∈ (Vj )rj . Bj := b1 , b2 , . . . , b(j) rj Set with Jj = {1 ≤ i ≤ rj }

J = J1 × . . . × Jd

for 1 ≤ j ≤ d.

Bj ∈ (Vj )rj can be considered as an element in L(KJj , Vj ). Its action on a ∈ KJj is X (j) ai bi . Bj a = i∈Jj

The frame data will be abbreviated by Bj ∈ (Vj )rj for 1 ≤ j ≤ d with rj = #Jj . These quantities define the elementary Kronecker product B :=

d O

Bj ∈ L(KJ , V).

(8.4)

j=1

The column of B corresponding to a multi-index i ∈ J is bi = all columns of B form the frame [or basis] of U ⊂ V.

(j) j=1 bij .

Nd

Hence

Using the data Bj ∈ (Vj )rj

frame or basis of Uj for 1 ≤ j ≤ d, ,

Jj := {1, . . . , rj }

for 1 ≤ j ≤ d,

a∈

d N

KJj = KJ

(8.5a)

for J = J1 × . . . × Jd ,

j=1

we define v = Ba =

X

ai

i∈J

=

r2 r1 X X i1 =1 i2 =1

···

d O

j=1 rd X

(j)

bij

(8.5b) (1)

(2)

(d)

a[i1 , i2 , · · · , id ] bi1 ⊗ bi2 ⊗ . . . ⊗ bid .

id =1

Note that rj ≥ dim(Uj ). Equality rj = dim(Uj ) holds if and only if Bj is a basis. According to §7.1, the representation (8.5b) is given by the mapping d O X
(j) ai bij = Ba. ρTS a, (Bj )dj=1 := i∈J

(8.5c)

j=1

The coefficient tensor a is also called the core tensor (in [284, page 287] Tucker used the term ‘core matrix’). Formally, representation (8.5c) looks very similar to the full representation (7.3). However, there are two important differences. First, the index set J is hopefully

8.2 Tensor Subspace Formats

263 (j)

much smaller than the original index set I. Second, the vectors {bi } are of different nature. In the case of the full representation (7.3), Bj is a fixed basis. (j) For instance, for the space V of multivariate polynomials, the vectors bi = xji (j) I represent the monomials, or for V = K the basis vectors bi are the unit vectors e(i) ∈ KIj (cf. (2.2)). Because of the fixed (symbolic) meaning, these basis vectors need not be stored. The opposite is true for the representation (8.5c). Here Uj and (j) its frame depend on the tensor v. Therefore the frame vectors bi must be stored explicitly.

8.2.1.2 Data Size and Further Properties Remark 8.8 (general tensor subspace representation). (a) The storage require(j) ments of the vectors bi depend on the nature of Uj (cf. §7.5). Denoting the storage of each frame vector by size(Uj ), the basis data require the memory size d
X d TSR Nmem = (Bj )j=1 rj · size(Uj ).

(8.6a)

j=1

(b) The coefficient tensor a ∈ KJ is given by its full representation (cf. §7.2) and requires a storage of the size TSR Nmem (a) =

d Y

rj .

(8.6b)

j=1

(c) For the optimal choice Uj = Ujmin (v) together with bases Bj of Uj , the numbers rj are given by rj = rankj (v) (cf. Remark 8.4). (d) If, at least for one j, Bj is not a basis, the coefficient tensor a ∈ KJ is not uniquely defined. The counterpart of Remark 7.11 reads as follows. Proposition 6.46 directly applies to Uk = Ukmin (v). Remark 8.9. Suppose a representation of v with rj = rankj (v) for 1 ≤ j ≤ d. (k) (a) If v satisfies a linear constraint ϕk (as defined in §6.8), then ϕk (bi ) = 0 (k) holds for all basis vectors bi from Bk . (b) Let V be the intersection Banach space in (4.58b). Then algebraic tensors (N ) (j) (j) v ∈ V imply that bi ∈ Vj j for all bi from Bj (1 ≤ j ≤ d). Set n := maxj size(Uj ) and r := maxj rj . Then the memory costs (8.6a,b) sum to rdn + rd . How rdn and rd compare, depends on the sizes of r and d . If r is small compared with n and if d is small (say d = 3), rd < rdn may hold. For medium sized d, the term rd becomes easily larger than rdn. For really large d, this term makes the representation infeasible.

264

8 Tensor Subspace Representation

8.2.2 Transformations i h i h (j) (j) (j) (j) Let a basis Bj = b1 , . . . , brj and a new basis Bjnew = b1,new , . . . , brj ,new be given. The transformation is described by an rj ×rj matrix T (j) : (j)

Bj = Bjnew T (j) , i.e., bi

=

rj X

(j) (j)

Tki bk,new

for 1 ≤ i ≤ rj ,

(8.7a)

for 1 ≤ k ≤ rj .

(8.7b)

k=1

The reverse direction is given by S (j) = (T (j) )−1 : (j)

Bjnew = Bj S (j) , i.e., bk,new =

rj X

(j) (j)

Sik bi

i=1

Nd The Kronecker matrix T = j=1 T (j) describes the simultaneous transformation in all directions 1 ≤ j ≤ d (with T (j) = I if Bj = Bjnew remains unchained). Nd Then B in (8.4) is transformed into Bnew = j=1 Bjnew via B = Bnew T ,

Bnew = B S,

S = T−1 =

d O

S (j) .

(8.7c)

j=1

Nd Lemma 8.10 (basis transformation). Let v ∈ U = j=1 Uj be described by (8.5a,b) and consider the transformation of Bjnew to the bases Bj by (8.7a) with matrices T (j) , i.e., B = Bnew T. The corresponding transformation of the coefficient tensor is d O anew := T a (8.7d) with T = T (j) . j=1

Then

d d . = ρTS anew , (Bjnew )j=1 ρTS a, (Bj )j=1

(8.7e)

Proof. Ba = (Bnew T)a = Bnew (Ta) = Bnew anew proves (8.7d).

The elementwise formulation of (8.7d) reads as anew [i1 , i2 , · · · , id ] rd r2 r1 X X X = ··· T (1) [i1 , k1 ] T (2) [i2 , k2 ] · · · T (d) [id , kd ] a[k1 , k2 , · · · , kd ]. k1 =1 k2 =1

kd =1

In the case of bases we have rj = dim(Uj ). This may be different for frames.

t u

8.2 Tensor Subspace Formats

265

Remark 8.11 (frame transformation). Let Bj and Bjnew be two frames of size rj and rjnew , respectively. (a) Assume that the frames Bj and Bjnew span the same space Uj . Then there are T (j) and S (j) satisfying (8.7a) and (8.7b) with rjnew instead of rj as upper bound of (j) k in bk,new . The rjnew ×rj matrix T (j) and the rj ×rjnew matrix S (j) are, in general, not uniquely determined. Although Bj = Bj S (j) T (j) and Bjnew = Bjnew T (j) S (j) , the products S (j) T (j) and T (j) S (j) need not be the identity matrix. (8.7d) and (8.7e) are still valid. (b) Let Uj = range(Bj ), while Ujnew = range(Bjnew ) is smaller and satisfies Uj % Ujnew ⊃ Ujmin (v). This includes the interesting case of Ujnew = Ujmin (v). Then S (j) with (8.7b) exists, but there is no T (j) fulfilling (8.7a) and therefore (8.7d) does not make sense. Here we have to proceed from the coefficients to the frame. Assume that the coefficient tensor a allows the formulation a = S anew ,

S=

d O

S (j) .

j=1

Then the new frames Bjnew = Bj S (j) satisfy Bnew := B S and (8.7e). Exercise 8.12. Discuss the case of range(Bj ) $ range(Bjnew ).

8.2.3 Tensors in KI Nd We now consider the tensor space V = KI = j=1 KIj with I = I1 × . . . ×Id and Nd U = j=1 Uj with subspaces Uj ⊂ KIj . The quantity Bj representing a frame or basis is the matrix i h (j) (j) Bj := b1 , b2 , . . . , b(j) ∈ KIj ×Jj (1 ≤ j ≤ d) , rj where the index sets Jj := {1, . . . , rj } form the product J = J1 × . . . ×Jd . Note that rj = dim(Uj ) in the case of a basis; otherwise, rj > dim(Uj ). For the sake of simplicity, we shall speak about ‘the frame Bj or basis Bj ’, although Bj is a matrix and only the collection of its columns form the frame or basis. The matrices Bj generate the Kronecker matrix B :=

d O

Bj ∈ KI×J

j=1

(cf. (8.4)). We repeat the formats (8.5a–c) and (8.10a,b) for Vj = KIj with the modification that the frames are expressed by matrices Bj .

266

8 Tensor Subspace Representation

Lemma 8.13 (general tensor subspace representation). (a) The coefficient tensor a∈

d O

KJj = KJ

for J = J1 × . . . × Jd ,

(8.8a)

j=1

and the tuple (Bj )1≤j≤d of frames represent the tensor v = Ba with the entries v[i1 · · · id ] = =

r1 X r2 X

···

rd X

k1 =1 k2 =1 r1 X r2 X

kd =1 rd X

k1 =1 k2 =1

kd =1

···

B1 [i1 , k1 ]B2 [i2 , k2 ] · · · Bd [id , kd ] a[k1 , k2 ,· · ·, kd ] (1)

(2)

(d)

bk1 [i1 ]bk2 [i2 ]· · ·bkd [id ] a[k1 , k2 ,· · ·, kd ],

(8.8b)

(j)

using the columns bk = Bj [•, k] of Bj . Equation (8.8b) is equivalent to v = Ba. The representation by ρframe (a, (Bj )dj=1 ) = Ba (8.8c) is identical to (8.5c), but now the data Bj are stored as (fully populated) matrices. (b) The storage cost required by B and a is d
X d Nmem (B) = Nmem (Bj )j=1 = rj · #Ij ,

Nmem (a) =

d Y

rj .

(8.8d)

j=1

j=1

8.2.4 Orthonormal Basis 8.2.4.1 Definitions and Transformations Nd 7 Let V = j=1 Vj be a [pre-]Hilbert space with scalar product h·, ·i induced by the scalar products h·, ·ij of Vj . Consider again the representation (8.5a,b) of v ∈ U with a basis (Bj )1≤j≤d . Let G(j) ∈ KJj ×Jj be the Gram matrix (cf. (2.13)) with the entries

(j) (j)  (j) Gik := bk , bi for 1 ≤ i, k ≤ rj , 1 ≤ j ≤ d. (8.9)

 P P (j) (j) Note that hBj v, Bj wi = h i vi bi , k wk bk i = G(j) v, w 2 (v, w ∈ KJj ), where h·, ·i2 is the Euclidean scalar product in KJj . This proves that the adjoint map Bj∗ ∈ L(Vj , KJj ) of Bj ∈ L(KJj , Vj ) leads to the products Bj∗ Bj = G(j) ∈ KJj ×Jj

and

B∗ B = G :=

d O

G(j) .

j=1 7

The assumption of a [pre-]Hilbert space V Ncan be relaxed. Even if V possesses no topology, one can define a Hilbert structure on U = d j=1 Uj . For a fixed basis Bj of Uj , one defines (j) (j) the scalar product on Uj by hbi , bk i := δik . With the induced scalar product, U becomes a Hilbert space. If V is equipped with another norm, the V norm and the newly defined U norm are equivalent on U since dim(U) < ∞.

8.2 Tensor Subspace Formats

267

In the Hilbert space setting an orthonormal basis is obviously the desirable choice. An orthonormal basis is characterised by Bj∗ Bj = I for 1 ≤ j ≤ d or, equivalently, by B∗ B = I ∈ KJ×J . In the matrix case Bj∗ and B∗ are written as BjH and BH . This setting yields the next representation. Definition 8.14 (orthonormal tensor subspace representation). (a)N If the bases d Bj of Uj are orthonormal, the representation (8.5a,b) of v ∈ U = j=1 Uj is called an orthonormal tensor subspace representation in U. (b) The detailed parameters of the representation are rj := dim(Uj ) rj

Bj ∈ (Vj )

with

for 1 ≤ j ≤ d, orthonormal basis of Uj ,

Jj := {1 ≤ i ≤ rj } for 1 ≤ j ≤ d, Nd J Jj a ∈ j=1 K = K for J = J1 × . . . × Jd X Od
(j) d ρorth a, (Bj )j=1 bij = Ba. ai := i∈J

j=1

(8.10a)

(8.10b)

(c) If V = KI (cf. §8.2.3), Bj ∈ (Vj )rj in (8.10a) becomes the orthogonal matrix8 Bj ∈ KIj ×Jj , and B ∈ KI×J is an orthogonal Kronecker matrix: BH B = I. Remark 8.15. The statements in Lemma 8.10 and Remark 8.11b are still valid. However, proper frames are excluded since Bj and Bjnew are bases. Moreover, the transformations must be unitary, i.e., (T (j) )∗ T (j) = I, (S (j) )∗ S (j) = I, T∗ T = I, S∗ S = I. If, starting from general frames Bj , we want to obtain orthonormal bases, we must find transformations such that Bjnew is an orthogonal matrix. For this purpose, two standard approaches can be applied. We recall Exercise 4.163: a QR decomposition of B or a Cholesky decomposition of B∗ B are equivalent to the respective decompositions of Bj or BjH Bj . The following lemma follows from Lemma 2.20. For the cost see §8.2.4.2. Lemma 8.16. Let v = Ba be given. (a) Assume that B is a basis. Then B∗ B ∈ KJ×J is a positive-definite matrix. Its Cholesky decomposition B∗ B = LLH defines the transformation v = (B L−H ) anew

with anew := LH a.

B L−H represents an orthogonal basis. (b) In the matrix case B ∈ KI×J , the QR decomposition9 B = QR yields v = Q anew

with anew := R a.

By definition, Q is an orthogonal matrix representing an orthonormal basis. 8

‘Orthonormal basis Bj ’ and ‘orthogonal matrix Bj ’ are equivalent expressions (cf. (2.3)). Also here, Vj may be general Hilbert spaces. In this case Qj is a tuple of orthonormal bases replacing the suitably ordered tuple Bj , while Rj in Bj = Qj Rj is a usual matrix. Their tensor product yields B = QR. 9

268

8 Tensor Subspace Representation

Remark 8.11b can be supplemented with orthogonality conditions.
d be given. Assume a = S anew with Corollary 8.17. Let v = ρorth a, (Bj )j=1 an orthogonal matrix S, i.e., SH S = I. Then the new tensor subspace representation is also orthonormal:

d d = ρorth anew , (Bjnew )j=1 with Bnew := B S. ρorth a, (Bj )j=1 Proof. Use B∗new Bnew = SH B∗ BS = SH S = I.

u t

In the case of Corollary 8.17, range(Bjnew ) ⊂ range(Bj ) holds. If both orthonormal bases span the same subspace, transformations must be unitary. Given unitary transformations Q(j) of Bj into Bjnew , i.e., (j)

bi =

rj X

(j) (j)

(j)

Qki bk,new , bk,new =

rj X

(j) (j)

Qik bi

for 1 ≤ i ≤ rj , 1 ≤ j ≤ d,

i=1

k=1

Nd the Kronecker product Q := j=1 Q(j) is also unitary and the coefficients transform according to anew = Qa, i.e.,

ρorth a, (Bj )dj=1 = ρorth anew , (Bjnew )dj=1 ; cf. (8.7c,d) with T = Q and (Q(j) )−1 = Q(j)H . Above, the new coefficient tensor anew is obtained from a by some transformation Ta. Alternatively, the coefficient tensor can be obtained directly from v via projection. Lemma 8.18. (a) Let v ∈ U and orthonormal bases Bj (1 ≤ j ≤ d) be given: Nd v = Ba with B := j=1 Bj . Then the coefficient tensor a of v has the entries * + d O (j) ai := v, bij , i.e., a = B∗ v. (8.11) j=1

(b) For a general the coefficient tensor in v = Ba is equal to a = G−1 b

Nbasis, Nd (j)  d with bk := v, j=1 bkj and the Gram matrix G = j=1 G(j) in (8.9). Exercise 8.19. (a) Prove that the orthonormal tensor subspace representation v = P Nd (j) i∈J ai j=1 bij (cf. (8.10b)) implies that kvk = kak2 , where k·k : V → R is the norm associated with the induced scalar product of V, while k·k2 is the Euclidean norm of KJ (cf. Example 4.149). P Nd (j) (b) If a second tensor w = i∈J ci j=1 bij uses the same bases, the scalar products — h·, ·i in V, h·, ·i2 in KJ — coincide: hv, wi = ha, ci2 .

8.2 Tensor Subspace Formats

269

In the case of Vj = KIj , the orthonormal tensor subspace representation reads as ! d O
d (8.12) Bj a with BjH Bj = I. v = ρorth a, (Bj )j=1 = j=1

The required memory size is the same as in (8.8d). According to (8.11), the coefficient tensor a can be obtained from v by a = BH v.

(8.13)

8.2.4.2 Orthonormalisation and Computational Cost ˆj )d ) is given with a proper We now assume10 that a tensor v = ρframe (ˆ a, (B j=1 nj ׈ rj ˆ frame or non-orthonormal basis Bj ∈ K . Lemma 8.16 proposes two methods for generating orthonormal bases Bj . Another possibility is computing the HOSVD bases (cf. §8.3 and §8.3.3). The latter approach is more expensive, but it allows us to determine orthonormal bases of the minimal subspaces Ujmin (v), whereas ˆj ). the following methods yield bases of possibly larger subspaces Uj := range(B d ˆj ) , procedure We start with the QR decomposition. Given frames or bases (B j=1 RQR(nj , rˆj , rj , Bj , Qj , Rj ) in (2.26) yields the decomposition ˆj = Qj Rj B

ˆj ∈ Knj ׈rj , Qj ∈ Knj ×rj , Rj ∈ Krj ׈rj ) (B

ˆj , Qj , and Rj . Defining with orthogonal matrices Qj , where rj is the rank of B the Kronecker matrices ˆ := B

d O

ˆj , B

Q :=

d O

Qj ,

j=1

j=1

R :=

d O

Rj ,

j=1

ˆ a = QRˆ a (cf. (8.5c)). Besides the exact operation count, we give a we get v = Bˆ bound in terms of rˆ := maxj rˆj

and

n := maxj nj .

(8.14)

Remark 8.20. Use the notations from above. The computational cost of all QR ˆj = Qj Rj and the cost of the product a := Rˆ decompositions B a add to " # j d d X Y Y 2 d+1 NQR (nj , rˆj ) + rk · rˆk ≤ 2dnˆ r + dˆ r . (8.15) j=1

k=1

k=j

Proof. The second term describes the cost of Rˆ a considered in (13.27a). Because of the triangular structure, a factor of two can be saved. t u 10

Also tensors represented in the r-term format are related to subspaces Uj , for which orthonormal basis can be determined (cf. Lemma 6.1). We can convert such tensors into the format ρTS according to §8.5.2.2 without arithmetical cost and apply the present algorithms.

270

8 Tensor Subspace Representation

The first approach in Lemma 8.16 is based on the Cholesky decomposition, proˆj )d ˆj = rj vided that (B j=1 represents bases. Because of the latter assumption, r holds. Note that in particular for the case r  n, the resulting cost in (8.16) is almost the same as in (8.15). Remark 8.21. With the notations from above, the computational cost of the Cholesky approach in Lemma 8.16a is " # d d  X Y 1 3 r¯  2 2 2nj rj + rj + rj rk ≤ d 2n + r¯ + d¯ rd+1 . (8.16) 3 3 j=1 k=1

ˆ HB ˆj takes 1 (2nj − 1)rj (rj + 1) ≈ nj r2 operations. The Proof. The product B j j 2 1 3 Cholesky decomposition into Lj LH requires r operations (cf. Remark 2.19). j 3 j ˆj L−H . The Further nj rj2 operations are needed to build the new basis B := B j j   P Q d d transformation a = LH ˆ a costs t j=1 rj · j=1 rj operations (cf. Remark 2.19).u For larger d, the major part of the computational cost in Remark 8.20 is d¯ rd+1 , J which is caused by the fact that ˆ a is organised as full tensor in K . Instead, the hybrid format discussed in §8.2.6 uses the r-term format for ˆ a. The resulting cost of Rˆ a (R and ˆ a as in Remark 8.20) described in (13.28b) is given in the following corollary. Corollary 8.22. Let the coefficient tensor ˆ a ∈ Rr be given in r-term format. The following transformations yield the new coefficient tensor a in the same format. (a) Using the QR decompositions from Remark 8.20, the cost of a := Rˆ a is r

d X

2

rj (2ˆ rj − rj ) . drˆ r ,

j=1

while the QR cost 2 by d(2n + r)ˆ r .

Pd

ˆj ) j=1 NQR (nj , r

does not change. The total cost is bounded

(b) In the Cholesky approach from Remark 8.21 the coefficient tensor a := LH ˆ a Pd r¯ 2 can be obtained by r j=1 rj operations, yielding the total bound d(2n+ 3 +r)¯ r2 .

8.2.5 Summary of the Formats The representation schemes ρTS , ρframe , ρorth are mentioned in (8.5c), (8.8c), (8.10b). The format ρHOSVD will be discussed in §8.3 (cf. (8.25)). ρTS is the most general case. If Vj = KIj , it becomes ρframe . If the bases are orthonormal, ρframe turns into ρorth . If the orthonormal bases are the particular HOSVD bases in Definition 8.24, ρorth becomes ρHOSVD .

8.2 Tensor Subspace Formats

271

Finally, we mention a generalisation. The mentioned representations can be used to represent several tensors simultaneously in the same tensor subspace: Od Uj . v(1) , . . . , v(m) ∈ U = j=1

In this case the data (Bj )1≤j≤d need to be stored only once. Each tensor v(µ) requires a coefficient tensor a(µ) (1 ≤ µ ≤ m). The required data size is r¯dn + m¯ rd , where n := maxj #Ij and r¯ := maxj rj .

8.2.6 Hybrid Format P Nd (j) Let v = i∈J ai j=1 bij be the standard tensor subspace representation ρTS or ρorth . An essential drawback of this format is the fact that the coefficient tensor a ∈ KJ is still represented in full format. Although J = J1 × . . . × Jd might be of much smaller size than I = I1 × . . . × Id , the exponential increase of #J with respect to d proves disadvantageous. An obvious idea is to represent a itself by one of the tensor formats described so far. However, using again a tensor subspace representation for a does not yield a new format as seen in Remark 8.23. An interesting approach is to use the r-term representation for the coefficient tensor a. Often, such an approach goes together with an approximation, but here we consider an exact representation of a by a=

d r O X

J a(j) ν ∈K

Jj with a(j) ν ∈K .

(8.17)

ν=1 j=1 d ) combined with the r-term repThe tensor subspace format v = ρTS(a, (Bj )j=1 (j) resentation a = ρr-term(r, (aν )1≤j≤d, 1≤ν≤r ) in (8.17) yields the hybrid format, which may be interpreted in two different ways. The first interpretation views v as a particular tensor from Tr (with rj = #Jj ) described by the iterated representation11       d (j) (j) d ρhybr r, (aν )1≤j≤d , (Bj )j=1 := ρTS ρr-term r, (aν )1≤j≤d , (Bj )j=1 1≤ν≤r

1≤ν≤r

=

r Y d XX i∈J

ν=1 j=1

O d (j) (j) aν [ij ] bi j

(8.18)

j=1

with ρTS from (8.5c) and a = ρr-term (. . .) from (7.7a). Similarly, we may define    
d (j) d (j) ρhybr (a (B , (8.19) r, := ρ ) ), (a r, ) (B ) ρ j j=1 r-term j j=1 , orth ν ν orth provided that Bj describes an orthonormal basis. 11

Carroll–Pruzansky–Kruskal [49] call this format CANDELINC. The last four letters abbreviate (j) ‘linear constraints’ which means that the vectors aν belong to certain subspaces.

272

8 Tensor Subspace Representation

The second interpretation views v as a particular tensor from Rr :   X r O d X d  r XO X (j) (j) (j) ] b v= [i] a(j) b [i . a = j ij ν ν i ν=1 j=1

ν=1 i∈J j=1

The right-hand side may be seen as v =

Pr

ν=1

(j) vν

modification (7.14), is described by the basis the matrix Bj . The format is abbreviated by ρhybr r-term



r, J, (aν(j) )1≤j≤d , (Bj )dj=1 1≤ν≤r



=

i∈Jj

(j) j=1 vν , where, (j) {bi : i ∈ Jj },

Nd

according to which yields !

d r O X

X

ν=1 j=1

i∈Jj

(j) a(j) ν [i] bi

.

(8.20)

Note that the formats (8.18) and (8.20) are equivalent in the sense that they use the same data representing the same tensor. Another characterisation of a tensor v in the hybrid format is v ∈ Rr ∩ Tr with r, r = (r1 , . . . , rd ), rj = #Jj in (8.18) and (8.20). The hybrid format is intensively used in Espig [87, Satz 2.2.4] (cf. §9.6.1) and in Khoromskij–Khoromskaia [186]. Finally, we discuss what happens when we again represent the coefficient tensor a in tensor subspace format. Remark 8.23. (a) Consider the following nested tensor subspace formats: v=

X i∈J

ai

d O

(j)

bi j ,

j=1

X

a=

k∈K

ck

d O

(j)

βk j ,

(8.21a)

j=1

(j)

(j)

where v ∈ V = KI with I = I1 ×. . .×Id , bij ∈ Vj = KIj , a ∈ KJ , βkj ∈ KJj , c ∈ KK with K = K1 × . . . × Kd . Then v has the standard tensor subspace representation ( d (j) X O with ˆbk ∈ KIj defined by (j) ˆb (8.21b) v= ck (j) (j) k ˆb(j) := P (k ∈ Kj ). i∈Jj βk [i] bi k j=1 k∈K Nd (j) (j) (j) (j) (b) Using Bj := [b1 · · · brj ] (rj := #Jj ), B := j=1 B, β j := [β1 · · · βsj ] Nd ˆj := [ˆb(j) · · · ˆbs(j) (sj := #Kj ), β := j=1 β j , and B j ], we rewrite (8.21a,b) as 1 v = Ba,

a = βc,

ˆ := Bβ. ˆ v = Bc with B

(8.21c)

The equations in (8.21c) can be interpreted as transformation: set anew = c, S = β, ˆ , i.e., of all products ˆ in Remark 8.11b. The computation of B and Bnew = B Pd ˆ Bj = Bj βj , requires 2 j=1 nj rj sj operations, where nj := #Ij . (c) Orthonormal tensor subspace representations for v and a in (8.21a) again yield an orthonormal tensor subspace representation in (8.21b).

8.3 Higher-Order Singular-Value Decomposition (HOSVD)

273

8.3 Higher-Order Singular-Value Decomposition (HOSVD) The typical symbols for the matrices appearing in the singular-value decomposition are U, V or even Uj , Vj (cf. (2.16b)). These matrices are to be distinguished from the (sub)spaces Uj , Ujmin , Vj with similar or even equal names. As stated in Remark 3.50a, there is no true generalisation of the singular-value decomposition (SVD) for d ≥ 3. However, it is possible to extend parts of the SVD structure to higher dimensions as sketched below. Considering a (reduced) singular-value decomposition of a matrix, we observe the following properties: Pr (a1 ) M = UΣ V T = i=1 σi ui viT can be exploited, e.g., for rank truncation. Ps (a2 ) In fact, Ms := i=1 σi ui viT is the best approximation of rank s. (b1 ) We may use ui and vi as new basis vectors. (b2 ) The basis transformation from (b1 ) maps M into diagonal form. HOSVD12 will also be helpful for truncation (as in (a1 )), and, in fact, this property will be a very important tool in practice. However, the result of truncation is not necessarily optimal; i.e., (a2 ) does not extend to d ≥ 3. As in (b1 ), {ui } will provide a new basis, while {vi } is not used. The tensor expressed with respect to these bases is by no means diagonal, not even sparse; i.e., (b2 ) has no tensor counterpart.

8.3.1 Definitions Nd We start with the tensor space V = j=1 KIj . Given v ∈ V, we consider the matricisation M := Mj (v) which is a matrix of the size Ij×I[j] with I[j] = ×k6=j Ik . Its reduced singular-value decomposition is M = UΣ V

T

=

rj X

σi ui viT ∈ KIj ×I[j] ,

(8.22)

i=1

where ui and vi are the columns of the respective orthogonal matrices U ∈ KIj ×rj and V ∈ KI[j] ×rj , σ1 ≥ σ2 ≥ . . . > 0 are the singular values, and rj = rank(M ) = rankj (v) (cf. (5.6b)). While U may be of reasonable size, V ∈ KI[j] ×rj is expected to have a huge number of rows, which we do not want to compute. Fortunately, it turns out that the matrix V is not needed. We recall the ‘left-sided singular-value decomposition’: as mentioned in Remark 2.28b, we may ask only for U and Σ in the singular-value decomposition M = UΣ V T , and the computation of U and Σ may be based on M M H = U Σ 2 U H . The diagonal matrix Σ controls the truncation procedure (see item (a1 ) from above), while U defines an orthonormal basis (see item (b1 )). We remark that range(U ) = range(M ) = Ujmin (v) (cf. Remark 8.4). 12

Also the term multilinear singular-value decomposition (MLSVD) is used; cf. De Lathauwer– De Moor–Vandevalle [70].

274

8 Tensor Subspace Representation

Unlike the case d = 2, we have d different matricisations Mj (v), leading to a tuple of d different decompositions (8.22), called the higher-order singularvalue decomposition (HOSVD) by De Lathauwer–De Moor–Vandewalle [70]. To distinguish the matricisations, we ornament the quantities of (8.22) with the index j referring to Mj (v) ∈ Vj ⊗ V[j] . In the next N definition, V is a general Hilbert tensor space. This space, as well as all V[j] = k6=j Vk , is equipped with the corresponding induced scalar product. Nd Nd Definition 8.24 (HOSVD basis). Let v ∈ k·k j=1 Ujmin (v) ⊂ k·k j=1 Vj . (j) (j) An orthonormal basis Bj = (bi , . . . , brj } of Ujmin (v) is called the j-th HOSVD basis for v if the following (singular-value) decomposition is valid: Mj (v) =

rj P i=1

(j)

(j) (j)

[j]

σi bi ⊗ mi

with

(j)

σ1 ≥ σ2 ≥ . . . > 0

(8.23)

and

[j]

orthonormal {mi : 1 ≤ i ≤ rj } ⊂ V[j] :=

k·k

N

k6=j

Vk .

(j)

The σi are called the singular values of the j-th matricisation. For infinitedimensional Hilbert spaces Vj and topological tensors, rj = ∞ may occur. (α)

α of Similarly, for a subset ∅ = 6 α $ {1, . . . , d}, an orthonormal basis (bi )ri=1 is called an α-HOSVD basis for v if

min (v) Uα

Mα (v) =

rα P i=1

(α)

σ1

(α)

≥ σ2

(α) (α)

σi bi

≥ ... > 0 (αc )

orthonormal {mi

(αc )

⊗ mi

with and

(8.24)

: 1 ≤ i ≤ rj } ⊂ Vαc .

Definition 8.25 (HOSVD representation). A tensor subspace representation v = ρorth (a, (Bj )1≤j≤d ) is a higher-order singular-value decomposition (HOSVD) (or HOSVD tensor subspace representation or for short HOSVD representation) of v if all bases Bj (1 ≤ j ≤ d) are HOSVD bases for v.13 For Bj satisfying these conditions, we write
v = ρHOSVD a, (Bj )1≤j≤d . (8.25) The storage requirements of HOSVD are those for the general case described in Lemma 8.13b. The next statement follows from Lemma 5.6. It allows us to replace the decomposition of v by the cheaper decomposition of the coefficient tensor a . 13

Because of the orthogonality property (8.23) for all 1 ≤ j ≤ d, De Lathauwer et al. [69], [173] call such a tensor representation all-orthogonal.

275

8.3 Higher-Order Singular-Value Decomposition (HOSVD)

Lemma 8.26. (a) A tensor v = Ba with B = T Mj (v) = Bj Mj (a) B[j]

Nd

j=1

Bj yields

with B[j] =

O k6=j

Bk .

If, at least for k 6= j, the bases Bk are orthonormal, the matricisations of v and a are related by   Mj (v) Mj (v)H = Bj Mj (a) Mj (a)H BjH . (b) If Bj also describes an orthonormal basis, a diagonalisation of the matrix ˆH ˆj Σ 2 U Mj (a) Mj (a)H = U j j yields the left-sided singular-value decomposition ˆj . with Uj := Bj U

Mj (v) Mj (v)H = Uj Σj2 UjH

As a consequence, HOSVD representations of v and a are closely connected. Nd Corollary 8.27. Let v ∈ j=1 KIj be given by an orthonormal tensor subspace representation (8.12): v = ρorth (a, (Bj )1≤j≤d ) with BjH Bj = I. Then (Bj )1≤j≤d describes the j-th HOSVD basis of v if and only if Mj (a) Mj (a)H = Σj2 Σj =

(j) (j) diag{σ1 , σ2 , . . .}

with (j)

(j)

and σ1 ≥ σ2 ≥ . . . > 0.

8.3.2 Examples We give two examples of the HOSVD for the simple case d = 3 and the symmetric situation r1 = r2 = r3 = 2, V1 = V2 = V3 =: V . Example 8.28. Let x, y ∈ V be two orthonormal vectors and set14 v := x ⊗ x ⊗ x + σy ⊗ y ⊗ y ∈ V := ⊗3 V.

(8.26)

(8.26) is already the HOSVD representation of v. For all 1 ≤ j ≤ 3, (8.23) holds with (j)

(j)

(j)

(j)

(j)

(j)

rj = 2, σ1 = 1, σ2 = σ, b1 = x, b2 = y, m1 = x ⊗ x, m2 = y ⊗ y. While v in (8.26) has tensor rank 2, the next tensor has rank 3. Example 8.29. Let x, y ∈ V be two orthonormal vectors and set v = αx ⊗ x ⊗ x + βx ⊗ x ⊗ y + βx ⊗ y ⊗ x + βy ⊗ x ⊗ x ∈ V := ⊗3 V. For the choice α := 14

q √ 1 − 32 2 σ + σ 2

q and β :=

√ σ/ 2,

(8.27a)

The coefficient tensor a has the entries a[1, 1, 1] = 1, a[2, 2, 2] = σ, and zero, otherwise.

276

8 Tensor Subspace Representation (j)

(j)

the singular values are again σ1 = 1, σ2 = σ. The HOSVD basis is given by r  r    q q σ 1 1 − √2 x + σ √2 − σ y σ √12 − σ x − 1 − √σ2 y (j) (j) p p b1 = , b2 = . (1 + σ) (1 − σ) (1 + σ) (1 − σ) (8.27b) In principle, the HOSVD can also be performed in Hilbert tensor spaces V = Nd case, the HOSVD bases k·k j=1 Vj with induced scalar product. In the generalN d are infinite (rj = ∞, cf. Theorem 4.137). If v := a j=1 Vj is an algebraic tensor, finite bases are ensured as in the next example referring to the polynomial from Example 8.5b. Note that here Vj is the function space L2 ([0, 1]) with the corresponding scalar product. Example 8.30. The HOSVD bases and the corresponding singular values15 of the Nd polynomial f (x, y, z) = xz + x2 y ∈ V := a j=1 Vj , Vj = L2 ([0, 1]), are (1)

(1)

q

(1)

q

109 720

(2)

q

109 720

(2)

q

b1 = 0.99953x + 0.96327x2 , σ1 = (1)

b2 = 6.8557x − 8.8922x2 , (2)

b1 = 0.58909 + 0.77158y,

σ2 = σ1 =

(2)

σ2 =

(3)

σ1 =

(3)

σ2 =

b2 = 1.9113 − 3.3771y, b1 = 0.44547 + 1.0203z, b2 = 1.9498 − 3.3104z,

(3) (3)

109 720

109 720 (2) σ1 , (2) σ2 .

+ −

1 45 1 45

√

46

≈ 0.54964,

√

46 ≈ 0.025893, √ 1 + 360 2899 ≈ 0.54859, √ 1 − 360 2899 ≈ 0.042741,

Proof. The matricisations Mj (f ) define integral operators Kj := Mj (f )M∗j (f ) ∈ R1 L (L2 ([0, 1]), L2 ([0, 1])) of the form (Kj (g)) (ξ) = 0 kj (ξ, ξ 0 ) g(ξ 0 ) dξ 0 (cf. Example 5.17). The involved kernel functions are Z 1Z

1

1 1 1 1 f (x, y, z)f (x0 , y, z)dydz = xx0 + x2 x0 + xx02 + x2 x02 , 3 4 4 3 0 0 1 1 1 1 1 1 1 1 k2 (y, y 0 ) = + y + y 0 + yy 0 , k3 (z, z 0 ) = + z + z 0 + zz 0 . 9 8 8 5 15 8 8 3 √ √ The eigenfunctions of K1 are √ x − 16 ( 46 + 1)x2 and x + 16 ( 46 − 1)x2 with the 109 1 eigenvalues λ1,2 = 720 ± 45 46. Normalising the eigenfunctions and extracting (1) (1) the square root of λ1,2 , we obtain the orthonormal basis functions bi and σi (i = 1, 2) from above. 0

k1 (x, x ) =

Similarly, the eigenfunctions 1 + yield the indicated results. 15

Since

P2

i=1

(j)
2

σ1

=

109 720

√ − 2899±8 y 35

of K2 and 1 +

√ 8± 2899 z 27

of K3 t u

for all 1 ≤ j ≤ 3, the values pass the test by Remark 5.13b.

277

8.3 Higher-Order Singular-Value Decomposition (HOSVD)

8.3.3 Computation and Computational Cost Nd Let V = j=1 Vj with Vj = KIj and nj := #I. For subsets ∅ 6= α $ D of N D := {1, . . . , d}, we use the notations αc := D\α and Vα = a k∈α Vk . The usual choice is α = {j}. We introduce the mapping (Bα , Σα ) := HOSVDα (v), where the left-hand side is the pair of the matrices Bα , Σα in the left-sided singular-value decomposition Mα (v)Mα (v)H = Bα Σα2 BαH , Bα ∈ Knα ×rα, 0 ≤ Σα ∈ Krα ×rα, BαH Bα = I, min (v)). Since the singular-value decomposition where rα := rankα (v) = dim(Uα is not always unique (cf. Corollary 2.24b), the map HOSVDα is not well-defined in all cases. If multiple solutions exist, we may pick a suitable one.

Performing HOSVDj (v) for all 1 ≤ j ≤ d, we obtain the complete higher-order singular-value decomposition (B1 , Σ1 , B2 , Σ2 , . . . , Bd , Σd ) := HOSVD(v). The computational realisation of HOSVDj depends on the used format. We discuss the following cases: (A) Tensor v given in full format.   (j) (B) v given in r-term format ρr-term r, (vν )1≤j≤d,1≤ν≤r .
(C) v given in the orthonormal tensor subspace format ρorth a, (Bj )dj=1 , where the coefficient tensor a may have various formats.
(D) v given in the general tensor subspace format ρframe a, (Bj )dj=1 , which means that Bj is not necessarily orthogonal. As review we list the cost (up to lower order terms) for the various cases:

format of v full r-term ρorth ρhybr orth

computational cost for HOSVD nd+1  d 2nr min(n, r)+nr2 +2n¯ r2 +2r2 r¯+3r¯ r2 + 83 r¯3 d+1 2 3d rˆ + 2d rˆ (n + 43 rˆ) 2 3 2dnr rˆ+(d+2) r2 rˆ+2dr rˆ min(ˆ r, r)+3r rˆ + 14 ˆ 3 dr

details in Remark 8.31 Remark 8.32 (8.33b) (8.34)

278

8 Tensor Subspace Representation

8.3.3.1 Case A: Full Format Set nj := #Ij , I[j] = ×k6=j Ik , n := maxj nj . The data HOSVDj (v) = (Bj , Σj ) can be determined by procedure LSVD(#Ij , #I[j] , rj , Mj (v), Bj , Σj ) in (2.29), where rj = dim(Ujmin (v)) describes the size of Bj ∈ KIj ×rj . The cost NLSVD (nj , #I[j] ) summed over all 1 ≤ j ≤ d yields " #  2 Y  d d d X X
nj 8n3j 8nj 8 2#I[j] −1 (nj + 1)+ ≈ nj + nk ≤ dnd+1 + dn3 . 2 3 3 3 j=1 j=1 k=1

For d ≥ 3, the dominant part of the cost is dnd+1 arising from the evaluation of the matrix entries of Mj := Mj (v)Mj (v)H : X Mj [ν, µ] = v[i1 , · · ·, ij−1 , ν, ij+1 , · · ·, id ] v[i1 , · · ·, ij−1 , µ, ij+1 , · · ·, id ]. i∈I[j]

If the HOSVD tensor subspace format of v is desired, we have to determine the coefficient tensor a. (8.13) implies that H

a=B v

with B :=

d O

Bj .

(8.29)

j=1

P P i.e., a[k1 , . . . , kd ] = i1 ∈I1 · · · id ∈Id B1 [k1 , i1 ] · · · Bd [kd , id ] v[i1 i2 · · · id ]. The cost for evaluating a is d X j=1

(2nj − 1) ·

j Y k=1

rk ·

d Y

nk . 2r1 nd .

k=j+1

In this estimate we assume that rj  nj so that the terms for j > 1 containing 2r1 r2 nd−1 , 2r1 r2 r3 nd−2 , . . . are much smaller than the first term. Obviously, the summation should be started with j ∗ = argmin{rj : 1 ≤ j ≤ d}. Above, we first determined the HOSVD bases and afterwards performed the projection (8.29). In fact, it is advantageous to apply the projection by Bj BjH immediately after computing Bj since the projection reduces the size of the tensor: v0 := v for j := 1 to d do begin (Bj , Σj ) := HOSVDj (vj−1 );
vj := I ⊗ . . . ⊗ I ⊗ BjH ⊗ I ⊗ . . . ⊗ I vj−1 end; {I is the identity matrix of varying size} return: a := vd start: loop:

(8.30)

279

8.3 Higher-Order Singular-Value Decomposition (HOSVD)

Set B(1,d) := B1H ⊗

Nd

j=2 I (1,d)

and B(1,d−1) := B1H ⊗

Nd

j=3

I. Lemma 5.6 implies

) . Since (B(1,d−1) )H B(1,d−1) that M2 (v1 ) = M2 (B v) = M2 (v)(B Nd min is the projection onto the subspace U1 (v) ⊗ j=2 Vj containing v, we have (1,d−1) T

(B(1,d−1) )H B(1,d−1) v = v. This proves  H H M2 (v1 )M2 (v1 ) = M2 (v) M2 (v) (B(1,d−1) )T B(1,d−1) 
 H H = M2 (v) M2 (B(1,d−1) )H B(1,d−1) v = M2 (v) M2 (v) . Similarly, we prove the identity Mj (vj−1 ) Mj (vj−1 )H = Mj (v) Mj (v) implying (Bj , Σj ) = HOSVDj (vj−1 ) = HOSVDj (v). Remark 8.31. The cost of algorithm (8.30) is " # j−1 d d X Y Y 8 3 (nj + 2rj ) · rk · nk + nj . 3 j=1 k=1

H

(8.31)

k=j

Under the assumptions rj  nj and d ≥ 3, the dominant part is n1

Qd

k=1

nk .

8.3.3.2 Case B: r-Term Format In §8.5 we shall discuss conversions between formats. The present case is already such a conversion from r-term format into HOSVD tensor subspace representation (other variants will be discussed in §8.5.2). Pr Nd (j) (j) (j) Let v = ν=1 j=1 vν ∈ Rr be given. First, all scalar products hvν , vµ i for 1 ≤ j ≤ d, 1 ≤ ν, µ ≤ r are to be computed. We discuss in detail the computation of (B1 , Σ1 ) = HOSVD1 (v). The first matricisation is given by M1 (v) =

r X

vν(1) ⊗ vν[1]

with vν[1] =

ν=1

d O

vν(j) .

j=2

By definition, B1 and Σ1 from HOSVD1 (v) results from the diagonalisation of M1 := M1 (v)M1 (v)H = B1 Σ12 B1H . We exploit the special structure of M1 (v): M1 =

r X r D r X r Y d D E E X X vν[1] , vµ[1] vν(1) (vµ(1) )H = vν(j) , vµ(j) vν(1) (vµ(1) )H . ν=1 µ=1

M1 has the form

ν=1 µ=1

j=2

280

8 Tensor Subspace Representation

M1 = A1 C1 AH 1

  A1 := [v1(1) v2(1) · · · vr(1) ] and Kd
r with  C1 := Gj with Gj := hvν(j) , vµ(j) i ν,µ=1 j=2

Jd (here j=2 denotes the multiple Hadamard product; cf. §4.6.4). As explained in Remark 2.39, the diagonalisation M1 = B1 Σ12 B1H is not performed directly. H H Instead, we use the QR decomposition in M1 = A1 C1 AH 1 = Q1 R1 C1 R1 Q1 and H diagonalise R1 C1 R1 . In the following algorithm, the index j varies from 1 to d: (j) (j)
r form Gram matrices Gj := hvν , vµ i ν,µ=1 ; J compute Hadamard products Cj := k6=j Gk ; (j) (j) [v1 · · · vr ] = Qj Rj (Qj ∈ Knj×rj, Rj ∈ Krj×r ); form products Aj := Rj Cj RjH ; diagonalise Aj = Uj Λj UjH ; 1/2 return Σj := Λj and Bj := Qj Uj ;

Gj ∈ Kr×r Cj ∈ Kr×r rj = rank(Qj Rj ) Aj ∈ Krj ×rj Uj , Λj ∈ Krj ×rj Bj ∈ Knj ×rj

1 2 3 4 5

(8.32a)

6

In line 3, rank rj = dim(Ujmin (v)) is determined. Therefore r = (r1 , . . . , rd ) is the Tucker rank which should be distinguished from the representation rank r of v ∈ Rr . Line 6 delivers the values (Bj , Σj ) = HOSVDj (v). Nd It remains to determine the coefficient tensor a ∈ j=1 Krj of v. As known from Theorem 8.44, a also possesses an r-term representation. Indeed, a=

r O d X

u(j) ν


(j) (j) H with u(j) ν [i] = hvν , bi i = Rj Uj [ν, i]

(8.32b)

ν=1 j=1

holds. Hence we obtain the hybrid format (8.18) of v. If wanted, we may convert a into full representation. This would yield the standard tensor subspace format of v. Remark 8.32. The computational cost of (8.32a,b), up to lower order terms, is r

d d X   X nj (r + 2 min(nj , r)) + rj (2r + 3rj ) + j=1

8 3 rj


+ 2nj rj2

j=1 2

2

2

≤ d 2nr min(n, r) + nr + 2n¯ r + 2r r¯ + 3r¯ r2 + 83 r¯3




with n := maxj nj and r¯ := maxj rj . Pd Proof. The cost of each line in (8.32a) is r(r+1) (2nj − 1) (line 1), 2 Pd Pd j=1 (d − 1) r(r + 1) (line 2), N (n , r) = 2r n j=1 QR j j=1 j min(nj , r) (line 3), d d d P P P r +1 (2r − 1) rj (r+ j2 ) (line 4), 83 rj3 (line 5), and rj (1 + nj (2rj − 1)) j=1 j=1 Pdj=1 (line 6), while (8.32b) requires r j=1 rj (2rj − 1) operations. t u

281

8.3 Higher-Order Singular-Value Decomposition (HOSVD)

This approach is in particular favourable if r, rj  nj since nj appears only linearly, whereas squares of r and third powers of rj are present. For later purposes we mention an approximate variant which can be used for large r. The next remark is formulated for the first step j = 1 in (8.32a). The other steps are analogous. Pr Nd (j) Remark 8.33. Assume that the norm16 of v = ν=1 j=1 vν ∈ Rr is known, (j) say, kvk ≈ 1. Normalise the vectors by kvν k = 1 for 2 ≤ j ≤ d. Instead (1) (1) of the exact QR decomposition [v1 · · · vr ] = Q1 R1 , apply the algorithm from Corollary 2.42 and Remark 2.43. According to Corollary 2.42, we may omit sufficiently small terms and reduce r to r0 (called m0 in Corollary 2.42). Alternatively or additionally, we may use the approximate scalar product h·, ·ip from Remark 2.43 for the cheap approximate computation of Gj .

8.3.3.3 Case C: Orthonormal Tensor Subspace Format ˆj )1≤j≤d ), i.e., v = Bˆ ˆ a. According a, (B Let v ∈ V be represented by v = ρorth (ˆ to Lemma 8.26b, the HOSVD bases of v can be derived from the HOSVD bases of ˆj , Σj ) := HOSVDj (ˆ a), we obtain a. Having determined (U the coefficient tensor ˆ ˆ ˆ (Bj , Σj ) := HOSVDj (v) by Bj := Bj Uj . ˆ ˆ = ×dj=1 Jˆj , a ∈ KJ is given in full format with J Case C1: Assume that ˆ ˆj , Σj ) := HOSVDj (ˆ a) together with the Jˆj = {1, . . . , rˆj }. The computation of (U evaluation of the coefficient tensor a ∈ KJ with J = N ˆ ˆ := d U ˆ U a = Ua, and the property ˆ j=1 j , requires d  X

rj + 2rj ) · (ˆ

j=1

j−1 Y

rk ·

k=1

d Y k=j

×dj=1 Jj , Jj = {1, . . . , rj }

8 rˆk + rˆj3 3



operations (cf. (8.31)), where rj = dim(Ujmin (ˆ a)) = dim(Ujmin (v)). From rj ≤ rˆj  Pd  Qd d+1 3 we get the estimate by rj · k=1 rˆk + 83 rˆj3 ≤ 3 d rˆ + 83 d rˆ with j=1 3ˆ rˆ := maxj rˆj . ˆj U ˆj for all 1 ≤ j ≤ d is The cost of Bj := B d X

(2ˆ rj − 1) nj rj .

(8.33a)

j=1

In total, the computational work is estimated by d+1

3dˆ r 16

 4  2 + 2dˆ r n + rˆ 3

with n, rˆ in (8.14).

(8.33b)

Because of the instability discussed later, the norm of v may be much smaller than the sum of the norms of all terms (cf. Definition 9.20).

282

8 Tensor Subspace Representation

Pr Nd ˆ (j) a = a ∈ KJ be given in r-term format ˆ Case C2: Let ˆ ν=1 aν j=1 vν . ˆj , Σj ) := HOSVDj (ˆ By Remark 8.32 (with nj replaced by rˆj ), the cost of (U a) ˆ including the computation of a with ˆ a = Ua amounts to r

 d d  X   X 8 rj , r)) + rj (2r + 3rj ) + rˆj (r + 2 min(ˆ rj + 2ˆ rj rj2 3 j=1 j=1 2

≤ (d + 2) r2 rˆ + 2 d r rˆ min(ˆ r, r) + 3 r rˆ +

14 3 d rˆ . 3

ˆj U ˆj , we obtain the following operation count. Adding the cost (8.33a) of Bj := B ˆj )1≤j≤d ) is repreRemark 8.34. If the coefficient tensor ˆ a in v = ρorth (ˆ a, ( B (j) sented as ˆ a = ρr-term (r, (vν )1≤j≤d,1≤ν≤r ), computing the HOSVD bases Bj and the coefficient tensor a ∈ Rr in v = ρHOSVD (a, (Bj )1≤j≤d ) requires    d  X 8 rˆ rj (r + 2 min(ˆ rj , r)) + rrj (2r + 3rj ) + rj + 2ˆ rj rj2 + 2nj rˆj rj 3 j=1 2

≤ 2dnrˆ r + (d + 2) r2 rˆ + 2dr rˆ min(ˆ r, r) + 3r rˆ +

14 3 d rˆ 3

(8.34)

operations, where rˆ := maxj rˆj and n := maxj nj as in (8.14).

8.3.3.4 Case D: General and Hybrid Tensor Subspace Format ˆj )d ) with non-orthonormal bases, the simplest In the case of v = ρframe (ˆ a, (B j=1 approach combines the following steps: Step 1: convert the representation into orthonormal tensor subspace format v = ρorth (a0 , (Bj0 )dj=1 ) by one of the methods described in §8.2.4.2. Step 2: apply the methods in §8.3.3.3 to obtain v = ρHOSVD (a, (Bj )dj=1 ). Alternatively, we may determine ρHOSVD (a, (Bj )dj=1 ) directly from the full ˆj )d ˆj )d ). However, this tensor ˆ a and and the bases (B a, (B j=1 in v = ρframe (ˆ j=1 17 approach turns out to be more costly. The situation differs, at least in case of r < nj , for the hybrid format when the Pr Nd (j) coefficient tensor ˆ a is given in r-term format: ˆ a = ν=1 aν j=1 vν ∈ Rr . First, the Gram matrices G(j) :=



rˆj ˆb(j) , ˆb(j) ν µ

ν,µ=1

∈ Krˆj ׈rj

(1 ≤ j ≤ d)

(8.35)

N Starting from (8.35) and Kronecker products G[j] := G(k) , one has to determine

[j]  N k6=j rˆ j (cf. §4.5.5). matrices G ˆ a, ˆ a [j] using the partial scalar product in k6=j K 17

8.4 Tangent Space and Sensitivity

283

P are generated (cost: j nj rˆj2 ). A different kind of Gram matrices are Gj ∈ Kr×r Pd (j) (j) rj2 + r2 rˆj ). with Gj [ν, µ] := hG(j) vν , vµ i, whose computation costs j=1 (2rˆ ˆ If Bj represents a basis, a modification is possible: compute the Cholesky decom(j) (j) L(j) L(j)H and use Gj [ν, µ] = hL(j)H vν , L(j)H vµ i. Then the positions G(j) =P d 1 3 2 2 operation count ˆj + rˆj r + r rˆj ) is reduced because of the triangular j=1 ( 3 r shape of L(j)H . The matrices Gj correspond to the equally named matrices in the first line of (8.32a). Since the additional steps are identical to those in (8.32a), we 2 2 2 3 2 must compare the costs d(n rˆ + 2r rˆ + r2 rˆ) or d(n rˆ + 13 rˆ + rˆ r + r2 rˆ) from 2 above with the sum of d rˆ (2n + r) from Corollary 8.22 plus dr2 n for (8.32a1 ) (n, rˆ in (8.14)). Unless r  n, the direct approach is cheaper. We summarise the results below. For the sake of simplicity, we compare the upper bounds. ˆj )d ) with B ˆj ∈ Knj ׈rj and assume that Remark 8.35. Let v = ρframe (ˆ a, ( B j=1 ˆ a has an r-term representation. Then the direct method is advantageous if n > r. 
 2 Its cost is by d (n − r) rˆ + n − rˆ r2 cheaper than the combination of Step 1 and 2 from above. The Cholesky modification mentioned above is even cheaper by 
2
 d n − rˆ/3 rˆ + n − rˆ r2 .

8.4 Tangent Space and Sensitivity 8.4.1 Uniqueness First we consider the subspace notation v ∈

Nd

j=1

Uj .

Nd Remark 8.36. (a) The spaces Ujmin (v) in v ∈ j=1 Ujmin (v) are unique. Nd (b) If v ∈ j=1 Uj , there is a direct sum Uj = Ujmin (v) ⊕ Wj . The space Wj with Wj ∩ Ujmin (v) = {0} can be chosen arbitrarily, but is superfluous. The representation v = ρTS (a, (Bj )) = Ba requires fixing a generating system Bj of Uj . Remark 8.37. (a) If Bj is a proper frame, the kernel ker(B) is nontrivial and v = B(a + b) holds for all b ∈ ker(B). (b) If Bj (1 ≤ j ≤ d) are fixed bases of Uj , the coefficient tensor a of v = Ba is unique. For fixed subspaces Uj , all possible representations are of the form Nd v = B0 a0 with B0 = BT, a0 = T−1 a, where T = j=1 Tj can be any isomorphisms (cf. Lemma 8.10). If Uj % Ujmin (v) for at least one j, the coefficient tensor
a cannot take all values, but must belong to the kernel of PWj ⊗ id[j] B, where

284

8 Tensor Subspace Representation

PWj is the projection18 corresponding to the direct sum Uj = Ujmin (v) ⊕ Wj (cf. Remark 8.36b). (c) In the case of v = ρorth (a, (Bj )), part (b) holds for all unitary Tj . (d) The representation v = ρHOSVD (a, (Bj )) is unique if the singular values (j) {σi : 1 ≤ i ≤ rj } are simple for all 1 ≤ j ≤ d.

8.4.2 Tangent Space Given v = ρTS (a, (Bj )) = Ba with a ∈ we set

Nd

j=1

KJj , Bj ∈ (Vj )rj , B =

v(t) := ρTS (a + tb, (Bj + tCj )) for t ∈ R, b ∈

d O

Nd

KJj , Cj ∈ (Vj )rj

j=1

Bj

(8.36)

j=1

Again the derivative dv(t) at t = 0 for all b and Cj spans the tangent space dt TTS (a, (Bj )) (cf. Definition 7.12). Remark 8.38. (a) The tangent space TTS (a, (Bj )) is the sum of TTS,a (a, (Bj )) Pd and j=1 TTS,Bj (a, (Bj )): TTS,a (a, (Bj )) := U :=

d O

Uj

(Uj := span{Bj }),

j=1

TTS,Bj (a, (Bj )) :=




B[j] ⊗ Cj a : Cj ∈ (Vj )rj ,

where B[j] ⊗Cj = B1 ⊗. . .⊗Bj−1 ⊗Cj ⊗Bj+1 . . .⊗Bd . In the case of the hybrid format (cf. §8.2.6), TTS,a (a, (Bj )) = {Bb : b ∈ Tr-term (v)}, where Tr-term (v) is the tangent space of a = ρr-term (r, v). (j)

(j)

(b) If Bj = (b1 , . . . , brj ) represents a basis with Uj = span{Bj } ⊂ Vj , then 
TTS,Bj (a, (Bj )) = id[j] ⊗ Lj v : Lj ∈ L(Uj , Vj ) . Proof. (a) The chain rule applied to the first argument a + tb yields Bb. Since b Nd Nd Jj varies in the whole space j=1 K , all u ∈ j=1 Uj belong to TTS,a (a, (Bj )). (j)

(j)

(b) Each tuple Cj = (c1 , . . . , crj ) ∈ (Vj )rj uniquely defines a map Lj ∈
(j) (j) L(Uj , Vj ) with Lj bi = ci (1 ≤ i ≤ rj ). Since id[j] ⊗ Lj B = B[j] ⊗ Cj and v = Ba, the statement follows. t u 18

For u = v + w ∈ Uj with v ∈ Ujmin (v), w ∈ Wj the projection is defined by PWj u = w.

8.4 Tangent Space and Sensitivity

285

Remark 8.39. (a) span{v} ⊂ TTS,a (a, (Bj )) and span{v} ⊂ TTS,Bj (a, (Bj )) and therefore also v ∈ TTS (a, (Bj )). (b) Assume that Bj (1 ≤ j ≤ d) represent bases. Let v = Ba = B0 a0 be two Nd representations with B0 = BT and a0 = T−1 a with T = j=1 Tj according to Remark 8.37b. Then TTS (a, (Bj )) = TTS (a0 , (Bj0 )) is independent of the choice of theN bases, but dependent on Uj = span{Bj }. TTS,Bj (a, (Bj )) is contained in Vj ⊗ k6=j Uk . (c) If, for all 1 ≤ j ≤ d, Bj is a basis of Ujmin (v), the tangent space only depends on the tensor v = ρTS (a, (Bj )). Proof. (a) Choose b and Cj as multiples of a and Bj . (b) Uj = span{Bj } = span{Bj0 } = span{Bj Tj } holds, since Tj is an isomorphism. This proves TTS,a (a, (Bj )) = TTS,a (a0 , (Bj0 )). Similarly,  

0 B[j] ⊗ Cj a = B[j] ⊗ Cj TT−1 a = B[j] ⊗ Cj Tj a0 proves TTS,B (a, (Bj )) = TTS,B (a0 , (Bj0 )). (c) Apply Remark 8.37b and part (b).

t u

For orthonormal bases, i.e., v = ρorth (a, (Bj )), we have replace Bj + tCj in (8.36) by a path Bj (t) of orthonormal bases with Bj (0) = Bj and Cj = d H H H dt Bj (t)|t=0 . From Bj (t) Bj (t) = I one infers Cj Bj + Bj Cj = 0. Exercise 8.40. Let B ∈ KI×J (#I ≥ #J) satisfy B H B = I. Then the subspace ˆ : S ∈ KJ×J skewof all C ∈ KI×J with C H B + B H C = 0 is {C = BS + B I×J ˆ ˆ symmetric, B ∈ K with range(B) ⊥ range(B)}. Remark 8.41. In the case of v = ρorth (a, (Bj )), Remarks 8.38 and 8.39 hold with Cj restricted by the side condition CjH Bj + BjH Cj = 0.

8.4.3 Sensitivity Now we consider tb and tCj in (8.36) as perturbations and write instead δa and (j) (δi )i=1,...,rj . The perturbation ˜ a := a + δa P Nd (j) leads to a perturbation δv = i δai j=1 bij of v. In the case of a general crossnorm and a general basis, we have kδvk ≤

X i

|δai |

d Y j=1

(j)

kbij k.

286

8 Tensor Subspace Representation

For an orthonormal basis and Hilbert norm, we can use the fact that the products Nd (j) j=1 bij are pairwise orthonormal, and get kδvk ≤

sX

2

|δai | =: kδak ,

i

where the norm on the right-hand side is the Euclidean norm. For perturbations of the basis vectors we give only a differential analysis; i.e., we only consider a small perturbation in a single component. Furthermore, we assume (j) that the vectors bi form orthonormal bases. Without loss of generality we may (1) (1) (1) suppose that b1 is perturbed into b1 + δ1 . Then (1)

X

v ˜=v+

(2)

(3)

(d)

a[1, i2 , . . . , id ] δ1 ⊗ bi2 ⊗ bi3 ⊗ · · · ⊗ bid ,

i2 ,...,id

i.e., δv =

(1)

P

i2 ···id

a[1, i2 , . . . , id ] δ1 ⊗

are orthogonal. Therefore

d N

(j)

j=2

bij . Terms with different (i2 , . . . , id )

s X (1) a[1, i2 , . . . , id ] 2 . kδvk = kδ1 k i2 ,...,id

(j)

As a consequence, for small perturbations δi approximation is kδvk ≈

d X X j=1

≤ kvk

s

(j) kδ` k

(j)

of all vectors bi , the first order

a[i1 , . . . , ij−1 , `, ij+1 , . . . , id ] 2

X i1 ,...,ij−1 ,ij+1 ,...,id

` d rX X `

j=1

(j)

kδ` k2 ≤ kvk

√ d

rX

(j)

j,`

kδ` k2 .

(8.37)

Here we have used the Schwarz inequality X

s

(j) kδ` k

i1 ,...,ij−1 ,ij+1 ,...,id

`

r ≤

a[i1 , . . . , ij−1 , `, ij+1 , . . . , id ] 2

X

X

(j) kδ` k2 `

r

X i

2

|ai |

P 2 2 together with i |ai | = kvk (cf. Exercise 8.19). The last inequality in (8.37) is again Schwarz’ inequality.

8.5 Conversions between Different Formats

287

8.5 Conversions between Different Formats So far, three formats (full representation, Rr , Tr ) have been introduced; in addition, there is the hybrid format Rr ∩ Tr . A natural question is how to convert one format into another one. It will turn out that conversions between Rr and Tr lead to the hybrid format introduced in §8.2.6. The conversions Rr → Tr and Tr → Rr are described in §8.5.2 and §8.5.3. The mapping from Rr into the HOSVD representation has already been mentioned in §8.3.3.2. For completeness, the full format is considered in §8.5.1 (see also §7.6.1).

8.5.1 Conversion from Full Representation into Tensor Subspace Format Nd nj Assume that a tensor v ∈ V := is given in full representation. The j=1 K P Nd (j) translation into tensor subspace format is v = j=1 bij with the unit i∈J vi (j) basis vectors bi := e(i) ∈ Knj in (2.2). Here the tensor v and its coefficient tensor are identical. The memory cost of the tensor subspace format is even larger because of the additional basis vectors. To reduce the memory, we may determine the minimal subspaces Ujmin (v) ⊂ Knj , e.g., by the HOSVD representation. Qd Qd If rj = dim(Ujmin (v)) < nj , the memory cost j=1 nj is reduced to j=1 rj .

8.5.2 Conversion from Rr to Tr The letter ‘r’ is the standard variable name for all kinds of ranks. Here we have to distinguish the tensor rank or representation rank r in v ∈ Rr from the vectorvalued tensor subspace rank r with components rj .

8.5.2.1 Theoretical Statements First, we recall the special situation of d = 2 (matrix case). The matrix rank is equal = rank1 = rank2 . to all ranks introduced for tensors: matrix-rank P=r tensor-rank (2) (1) Therefore there is an r-term representation v = i=1 vi ⊗vi with r = rank(v). (2) (1) Since {vi : 1 ≤ i ≤ r} and {vi : 1 ≤ i ≤ r} are sets of linearly independent vectors, they can be used as bases and yield a tensor subspace representation (8.5b) for r = (r, r) with the coefficients aij = Prδij . The singular-value decomposition yields another r-term representation v = i=1 σi ui ⊗ vi, which is an orthonormal tensor subspace representation (8.12) (with orthonormal bases {ui }, {vi }, and coefficients aij = δij σi ). The demonstrated identity Rr = T(r,r) of the formats does not extend to d ≥ 3.

288

8 Tensor Subspace Representation

Given an r-term representation v=

d r O X

(j)

vi ,

(8.38)

i=1 j=1

Lemma 6.1 states that v ∈ This proves v ∈ Tr

a

Nd

j=1 Uj

(j)

(j)

(cf. (6.1)) with Uj := span{v1 , . . . , vd }.

for r = (r1 , . . . , rd ), rj := dim(Uj ).

We recall the relations between the different ranks. Theorem 8.42. (a) The minimal tensor subspace rank r = (r1 , . . . , rd ) of v ∈ V is given by rj = rankj (v) (cf. (5.6b)). The tensor rank r = rank(v) (cf. Definition 3.35) satisfies r ≥ rj . The tensor rank may depend on the field: rR ≥ rC ≥ rj (cf. Proposition 3.44), while rj = rankj (v) is independent of the field. (b) The inequalities of Part (a) are also valid for the border rank defined in (9.11) instead of r, rR , rC . Proof. Part (a) follows from Remark 6.24. In the case of (b), v is the limit of a sequence vn ∈ Rr . There are minimal subspaces Ujmin (vn ) (1 ≤ j ≤ d, n ∈ N) of dimension rj,n satisfying r := rank(v) ≥ rj,n . By Theorem 6.29 the minimal subspace Ujmin (v) has dimension rj := dim(Ujmin (v)) ≤ lim inf rj,n ≤ r. n→∞

Hence r ≥ rj is proved.

t u (j)

(j)

Using a basis Bj of Uj = span{v1 , . .
. , vd }, we shall construct a tensor subspace representation v = ρTS a, (Bj )dj=1 in §8.5.2.3. In general, these subspaces Uj may be larger than necessary; however, under the assumptions of Proposition 7.10, Uj = Ujmin (v) are the minimal subspaces. This proves the following statement. Remark 8.43. If v is given by the r-term representation (8.38) with r = rank(v), (j) (j) constructions based19 on Uj := span{v1 , . . . , vd } yield
v = ρTS a, (Bj )dj=1 with rj = rank(v). j


Having converted v = ρr-term (. . .) into v = ρTS a, (Bj )dj=1 , the next statement describes the r-term structure of the coefficient tensor a. This helps, in particular, to obtain the hybrid format in §8.2.6. 19

Representations with rj = rankj (v) can be obtained anyway by HOSVD. These decompositions, however, cannot be obtained from Uj alone.

8.5 Conversions between Different Formats

289

d ) be any tensor subspace representation Theorem 8.44. Let v = ρTS (a, (Bj )j=1 J with the coefficient tensor a ∈ K and bases20 Bj . Then the ranks of v and a coincide in two different meanings. First, the true tensor ranks satisfy

rank(a) = rank(v). Second, given (8.38) with representation rank r, the r-term representation of a (with same number r) can be obtained constructively as detailed in §8.5.2.3. The resulting tensor subspace representation of v is the hybrid format (8.18). Nd (j) ∼ (j) Proof. Consider v as an element of j=1 Uj with Uj = span{v1 , . . . , vd } = KJj . By Lemma 3.39a, the rank is invariant. This proves rank(a) = rank(v). The second part of the statement follows from the constructions in §8.5.2.3. t u

8.5.2.2 Conversion into General Tensor Subspace Format Pr Nd (j) ∈ Rr into v ∈ Tr with A rather trivial translation of v = j=1 vi i=1 r = (r, . . . , r) can be obtained without any arithmetic cost by choosing the frames (j)

Bj = [ v1 , . . . , vr(j) ] and the diagonal coefficient tensor a [ i, . . . , i ] = 1 for 1 ≤ i ≤ r,

and a[i] = 0 otherwise.

(8.39)

(j)

d Obviously, v = ρr-term (r, (vi )) = ρTS (a, (Bj )j=1 ) holds. Note that there is no guarantee that the frames are bases; i.e., the subspaces Uj = range(Bj ) in (6.2b) may have a dimension smaller than r (cf. Remark 3.42).

The diagonal tensor a is a particular case of a sparse tensor (cf. (7.5)).

8.5.2.3 Conversion into Orthonormal Hybrid Tensor Subspace Format Let Vj = KIj . An orthonormal basis of the subspace Uj from above can be ob(j) (j) tained by a QR decomposition of the matrix Aj := [v1 · · · vr ] ∈ KIj ×r . Procedure RQR in (2.26) yields Aj = Bj Rj with an orthogonal matrix Bj ∈ KIj ×rj , where rj = rank(Aj ) = dim(Uj ). The second matrix Rj allows the representaPrj (j) (j) (j) ri,k bi . From this we derive tions vk = Bj Rj [•, k] = i=1 20

For general frames Bj the coefficient tensor is not uniquely defined and, in fact, different (equivalent) coefficient tensors may have different ranks. However, the minimum of rank(a) over all equivalent a coincides with rank(v).

290

8 Tensor Subspace Representation

v=

d r O X

(j) vk

=

k=1 j=1

=

r1 X

···

i1 =1

rd  X r Y d X

r O d X




(j)

rij ,k

O d

k=1 j=1

|

a=

(j)

(j)

rij ,k bij

(8.40a)

k=1 j=1 ij =1

id =1

proving v = ρorth a, (Bj )dj=1 r-term format

rj r O d X X

(j)

bij ,

j=1

{z } a[i1 · · · id ]

with the coefficient tensor a described in the

(j)

rk ,

 r (j) (j) j rk := ri,k ∈ Krj . i=1

k=1 j=1

(8.40b)


(j) d This yields the orthonormal hybrid format v = ρhybr orth r, (rk ), (Bj )j=1 (cf. (8.19)). Pn The conversion cost caused by the QR decomposition is j=1 NQR (nj , r) with nj := #Ij . If r ≤ n := maxj n, the cost is bounded by 2nr2 . The construction from above can equivalently be obtained by the following two steps: (i) apply the approach of §8.5.2.2 and (ii) perform an orthonormalisation of the frame as described in §8.2.4. Note that the transformation to new orthonormal bases destroys the sparsity of the coefficient tensor (8.39).

8.5.2.4 Case of Large r As seen from (8.40a,b), the representation rank r of v ∈ Rr (KI ) is inherited by the coefficient tensor a ∈ Rr (KJ ). In §7.6.2 the conversion of v ∈ Rr into full
Qd format is proposed, provided that r > N := n / max 1≤i≤d ni . Now the j=1 j same consideration can be applied to a ∈ Rr , provided that r > R :=

d Y j=1

 rj / max ri , 1≤i≤d

(8.41)

where rj is obtained in §8.5.2.3 as size of the basis Bj ∈ KIj ×Jj , Jj = {1, . . . , rj }. Lemma 7.20 and Remark 7.21 show that a conversion of a ∈ Rr (KJ ) into full Qd format or R-term format a ∈ RR (KJ ) requires 2r j=1 rj operations. The cost Pn Pn to obtain a ∈ Rr is j=1 NQR (nj , r) ≤ 2r j=1 nj min{r, nj } (cf. §8.5.2.3). This yields the following result. (j)
Lemma 8.45. Assume v = ρr-term r, (vν
) with r satisfying (8.41). Then v can be converted into v = ρorth a, (Bj )dj=1 or a hybrid format with a ∈ RR (KJ )
Qd Pn requiring 2r j=1 rj + j=1 nj min{r, nj } operations.

8.5 Conversions between Different Formats

291

8.5.3 Conversion from Tr to Rr The tensor subspace representation v=

r2 r1 X X

rd X

···

k1 =1 k2 =1

(2)

(1)

(d)

a[k1 k2 · · · kd ] bk1 ⊗ bk2 ⊗ · · · ⊗ bkd

kd =1

Qd from (8.5b) or (8.12) is an r-term representation of v with r := j=1 rj terms. To P Nd (j) (1) reach the format (7.7a): v = k j=1 vk , the vectors vk for j = 1 could be (1)

(j)

(j)

defined by a[k1 k2 · · · kd ]bk1 , while vk := bkj for j > 1. Following the proof of Lemma 3.45, an improvement is possible. Choose the largest r` . Without loss of generality, assume r1 ≥ rj for all j. Rewrite v as v=

r2 X k2 =1

···

rd  X r1 X

(1) a[k1 k2 · · · kd ] bk1



(2)

(d)

⊗ bk2 ⊗ · · · ⊗ bkd .

(8.42)

k1 =1

kd =1

|

{z (1) ˆ =: b [k2 · · · kd ]

}

Qd This is an r-term representation of v with r := j=2 rj terms. Because of the presumably very large number r of terms, the storage requirement Pd r j=1 size(Uj ) of the r-term representation of v seems huge. An improvement Pr Nd (j) (j) can be based on the fact that the r factors vν (1 ≤ ν ≤ r) in v = ν=1 j=1 vν for j ≥ 2 are r not necessarily linearly independent vectors but can be expressed (j) rj by the basis (bi )i=1 , which is already stored (cf. (7.14)). This again leads to the hybrid format, now considered as a particular case of the r-term format (cf. (8.20)). Next, we assume that v ∈ Tr is given in the hybrid format in (8.18): v=

r1 X i1 =1

···

rd  X r Y d X id =1

O d

a(j) ν [ij ]

ν=1 j=1

(j)

bi j .

j=1

Using the reformulation (8.42), we have to compute the vectors ! r1 r Y d X X (1) (1) (j) ˆb [i2 · · · id ] := a [ij ] b ∈ V1 = Kn1 . ν

i1 =1

i1

ν=1 j=1

As above, the direction k = 1 is chosen because of the assumption r1 ≥ rj for all Qd j so that the resulting representation rank N = j=2 nj is minimal. Remark 8.46. Let v be given in the hybrid Qdformat in (8.18) with rj , r as above.
Conversion into N -term format with N = j=2 nj requires N (d − 1)r + 2n1 r1 operations.

292

8 Tensor Subspace Representation

8.5.4 A Comparison of Both Representations We summarise the results from above. Remark 8.47. (a) If v ∈ Rr , then v ∈ Tr with r = (r, . . . , r). (b) If v ∈ Rr , a hybrid format Rr ∩ Tr can be constructed with r = (r1 , . . . , rd ), rj = rankj (v). (c) If v ∈ Tr with r = (r1 , . . . , rd ), then v ∈ Rr with r :=

Qd

j=1 rj max1≤j≤d rj .

For simplification, we now assume that Vj = Kn (i.e., the dimension n is independent of j). The transfer between both representations is quite nonsymmetric. According to Remark 8.47a, v ∈ Rr yields v ∈ Tr with r = (r, r, . . . , r). Note that vectors of a similar size must be stored: r-term TSR Nmem = r · d · n = Nmem ((Bj )1≤j≤d )

(cf. (7.8c) and (8.6a)).

Additionally, the tensor subspace representation needs storage for the coefficient tensor a: TSR Nmem (a) = rd

(cf. Remark 8.8b). This large additional memory cost makes the tensor subspace representation clearly less advantageous. The hybrid format from Remark 8.47b needs the storage d X hybr Nmem = (n + r) rj , j=1 r-term which may be smaller than Nmem if rj < r < n.

On the other hand, if a tensor v ∈ Tr with r = (r, · · · , r) is converted into the rd−1 -term representation from (8.42), the latter format requires storage of size N -term Nmem = rd−1 · d · n.

Since r ≤ n (and usually r  n), the inequality TSR r-term Nmem = r · d · n + rd  rd−1 · n < rd−1 · d · n = Nmem

indicates that the tensor subspace representation is far better. The previous examples underline that none of the formats Rr or Tr are, in general, better than the other. It depends on the nature of the tensor what format is preferable. Often, the hybrid format is the best compromise.

8.6 Joining two Tensor Subspace Representation Systems

293

8.5.5 r-Term Format for Large r > N In §7.6.3 we discussed the case of r > N , where N is the bound of the maximal rank in (8.43). In particular in the case of d = 3, we may use an intermediate tensor subspace representation (and, possibly, approximation tools for this format: see §10). Here we assume that a tensor is represented in the r-term format, v=

r X

(1)

vi

(2)

⊗ vi

(3)

⊗ vi

∈ Kn1 ×n2 ×n3 ,

i=1

with rather large r. The term ‘rather large’ may mean r > N := min{n1 n2 , n1 n3 , n2 n3 }.

(8.43)

Instead of a large r, we may also assume N to be rather small. In this case the following procedure yields an exact N 0 -term representation with N 0 ≤ N < r. Step 1. Convert v from r-term format into tensor subspace format (hybrid variant in §8.2.6). Step 2. Convert v back into N 0 -term format with N 0 ≤ N (concerning N in (8.43), see §8.5.3).

8.6 Joining two Tensor Subspace Representation Systems d d ) and v00 = ρTS (a00 , (Bj00 )j=1 Let v0 = ρTS (a0 , (Bj0 )j=1 ) be two tensors from Nd 0 00 V = j=1 Vj , involving different subspaces Uj and Uj . Obviously, the sum v0 + v00 requires the spaces Uj defined by

Uj := Uj0 + Uj00

for 1 ≤ j ≤ d.

A common systems Bj spanning Uj has to be constructed. Then the coefficient tensors a0 of v0 and a00 of v00 have to be transformed according to the new bases.

8.6.1 Trivial Joining of Frames The least requirement is that 0(j)

Bj0 = [b1 Bj00

=

0(j)

0(j)

, b2 , . . . , br0 ] ∈ Uj0 j

00(j) 00(j) 00(j) [b1 , b2 , . . . , br0 ] j

∈


rj0

and


r00 Uj00 j

are frames spanning the respective subspaces Uj0 and Uj00 . The respective index

294

8 Tensor Subspace Representation d

d

sets are J0 = ×j=1 Jj0 and J00 = ×j=1 Jj00 with Jj0 = {1, . . . , rj0 } and Jj00 = {1, . . . , rj00 }. Since no linear independence is required, the simple definition i   h 0(j) 0(j) 00(j) 0(j) 00(j) 00(j) Bj := Bj0 Bj00 = b1 , b2 , . . . , br0 , b1 , b2 , . . . , br0 , j

j

Jj := {1, . . . , rj } with rj := rj0 + rj00 , yields a frame associated with the subspace Uj := Uj0 + Uj00 . The representation rank rj is the sum of the previous ones, even when the subspaces Uj0 and Uj00 overlap. An advantage is the easy construction of the coefficients. The columns of Bj = 00(j) 0(j) (j) (j) are bi := bi for 1 ≤ i ≤ rj0 and bi+r0 := bi for 1 ≤ i ≤ rj00 . j 0 The coefficient a of

(j) (j) [b1 , . . . , brj ]

v0 =

X

a0i0

i0 ∈J0

X

0(j)

bi0

j

∈

d O

ai

i∈J

d O j=1

Uj0

(8.44a)

Uj

(8.44b)

j=1

j=1

becomes v0 =

d O

(j)

bij ∈

d O j=1

with ai := ai0 for i ∈ J0 and ai := 0 for i ∈ J\J0 . Analogously, a coefficient P Nd 00(j) 0 becomes a i+r0 := ai00 for i ∈ J00 with a00 of v00 = i00 ∈J00 ai0 j=1 bij00 r0 = (r10 , . . . , rd0 ), and ai := 0 otherwise. Remark 8.48. The joining of frames requires only a rearrangement of data but no arithmetic operations.

8.6.2 Common Bases Assume that Bj0 , Bj00 are the respective bases of Uj0 , Uj00 . We want to construct a new, common basis Bj for the sum Uj = Uj0 + Uj00 . Applying the procedure JoinBases(Bj0 , Bj00 , rj , Bj , Tj0 , Tj00 ) (j)

(8.45a)

(j)

in (2.32), we produce a common basis Bj = [ b1 , . . . , brj ] of dimension rj and transformation matrices Tj0 and Tj00 satisfying (8.45b) (cf. (2.31)): Bj0 = Bj Tj0

and Bj00 = Bj Tj00

(8.45b)

8.6 Joining two Tensor Subspace Representation Systems

295

0

Lemma 8.49. Let a0 ∈ KJ be the coefficient tensor of v0 in (8.44a) with respect to the bases Bj0 . The coefficient tensor a0new ∈ KJ of v0 with respect to the bases Bj satisfying (8.45b) is given by 0 anew =

d O

! Tj0 a0

j=1

Nd 00 (cf. (8.44b)). Similarly, the coefficient tensor a00 ∈ KJ representing v00 ∈ j=1 Uj00 N d 00 = ( j=1 Tj00 )a00 ∈ KJ with respect to the bases Bj . A transforms into anew JoinBases is to take Bj0 as the first part of Bj , possible option in procedure   I which leads to Tj0 = 0 . Remark 8.50. Assume Vj = Knj . The cost of (8.45a) is NQR (nj , rj0 + rj00 ). 0 is without arithmetic cost under the option mentioned The coefficient tensor anew  Pd  Qj Qd in Lemma 8.49, while a00new requires 2 j=1 ( `=1 r`00 )( `=j r` ) operations. If nj ≤ n and rj ≤ r, the total cost can be estimated by 8dnr2 + 2drd+1 . The cost for the basis transformation is much smaller if v0 and v00 are given in hybrid format. We use that   d d d r 00 O r 00 O O X X 00(j) 00 00(j) 00 00 00 Tj  a = and anew =  aν,new aν a = ν=1 j=1 00(j)

00(j)

with aν,new = Tj00 aν

j=1

cost 2r00

Pd

00 j=1 rj rj

ν=1 j=1

operations.

Remark 8.51. In the case of hybrid tensors v0 and v00 , computing common bases Pd and transforming the coefficient tensors cost NQR (nj , rj0 + rj00 ) + 2r00 j=1 rj rj00 . If all ranks are bounded by r and nj ≤ n, the total cost can be estimated by 8dnr2 + 2dr3 . In the case of a tensor subspace representation with orthonormal bases, procedure JoinBases has to be replaced with JoinONB.

Chapter 9

r-Term Approximation

Abstract In general, one tries to approximate a tensor v by another tensor u requiring less data. The reason is twofold: the memory size should decrease and, hopefully, operations involving u should require less computational work. In fact, u ∈ Rr leads to decreasing cost for storage and operations as r decreases. However, the other side of the coin is an increasing approximation error. Correspondingly, in Section 9.1 two approximation strategies are presented, where either the representation rank r of u or the accuracy is prescribed. Before we study the approximation problem in general, two particular situations are discussed. Section 9.2 is devoted to r = 1, i.e., u ∈ R1 is an elementary tensor. The matrix case d = 2 is recalled in Section 9.3. The properties obse