A THEORETICAL INTRODUCTION TO NUMERICAL ANALYSIS
Victor S. Rvaben'kii Semvon V. Tsvnkov
Boca Raton
London
New York
Chapman &; Hall/CRC is an imprint of the
Taylor &; Francis Group, an informa business
& Hall!CRC & Francis Group
Chapman Taylor
6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742
© 2007 by Taylor & Francis Group, LLC & Hall!CRC is an imprint of Taylor & Francis Group, an Informa business
Chapman
No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 65 4 3 2 1 International Standard Book Number-10: 1-58488-607-2 (Hardcover) International Standard Book Number-13: 978-1-58488-607-5 (Hardcover) This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the conse quences of their use. No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www. copyright.com (http://www.copyright.coml) or contact the Copyright Clearance Center, Inc. (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the
CRC Press Web site at
http://www.crcpress.com
Contents
Preface Acknowledgments
1
Introduction
1.1
1.2 1.3
1 .4
Discretization Exercises Conditioning . . Exercises Error 1.3.1 Unavoidable Error 1.3.2 Error of the Method 1 .3.3 Round-off Error . . Exercises . . . . . . On Methods of Computation 1 .4. 1 Accuracy . . . . 1 .4.2 Operation Count . . 1 .4.3 Stability . . . . . . . 1.4.4 Loss of Significant Digits 1.4.5 Convergence . . . 1.4.6 General Comments Exercises . . . . .
I
Interpolation of Functions. Quadratures
2
Algebraic Interpolation
2. 1
2.2
Existence and Uniqueness of Interpolating Polynomial . . . . . . . 2. 1.1 The Lagrange Form of Interpolating Polynomial . . . . . . 2. 1.2 The Newton Form of Interpolating Polynomial. Divided Differences . . . . . . . . . . . . . . . . . . . . . . 2. 1.3 Comparison of the Lagrange and Newton Forms . . . . . . 2. 1.4 Conditioning of the Interpolating Polynomial . . . . . . . . 2. 1.5 On Poor Convergence of Interpolation with Equidistant Nodes . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . Classical Piecewise Polynomial Interpolation . . . . . 2.2. 1 Definition of Piecewise Polynomial Interpolation
xi xiii
1
4 5 6 7 7
8 10 10 11 12 13 14 14 15 18 18 19 21
25
25 25
26 31 32 33 34 35 35 iii
iv Formula for the Interpolation Error . . . . . . . . . . . . . Approximation of Derivatives for a Grid Function . . . . . . Estimate of the Unavoidable Error and the Choice of Degree for Piecewise Polynomial Interpolation . . . . . 2.2.5 Saturation of Piecewise Polynomial Interpolation Exercises . . . . . . . . . . . . . . . . . . . . . Smooth Piecewise Polynomial Interpolation (Splines) 2.3. 1 Local Interpolation of Smoothness s and Its Properties 2.3.2 Nonlocal Smooth Piecewise Polynomial Interpolation 2.3.3 Proof of Theorem 2. 1 1 . . . . . . . Exercises . . . . . . . . . . . . . . Interpolation of Functions of Two Variables 2.4. 1 Structured Grids . 2.4.2 Unstructured Grids Exercises . . . . 2.2.2 2.2.3 2.2.4
2.3
2.4
3
Trigonometric Interpolation
3. 1
3.2
Interpolation of Periodic Functions . . . . . . . . . . . . . . . 3 . 1 . 1 An Important Particular Choice of Interpolation Nodes . . . 3 . 1 .2 Sensitivity of the Interpolating Polynomial to Perturbations of the Function Values . . . . . . . . . . . . . 3 . 1 .3 Estimate of Interpolation Error . . . . . . . . . . . . . . . . 3 . 1 .4 An Alternative Choice of Interpolation Nodes . . . . . . . . Interpolation of Functions on an Interval. Relation between Algebraic and Trigonometric Interpolation 3.2. 1 Periodization . . . . . . . . . . . 3.2.2 Trigonometric Interpolation . . . 3.2.3 Chebyshev Polynomials. Relation between Algebraic and Trigonometric Interpolation . . . . . . . . . . . . . . . . . 3.2.4 Properties of Algebraic Interpolation with Roots of the Chebyshev Polynomial Tn+! (x) as Nodes . . . . . . . . . 3.2.5 An Algorithm for Evaluating the Interpolating Polynomial . 3.2.6 Algebraic Interpolation with Extrema of the Chebyshev Polynomial Tn (x) as Nodes . . . . . . . . . . . . . . . . . . 3.2.7 More on the Lebesgue Constants and Convergence of Interpolants . Exercises . . . . . . . . . . . . . . . .
.
.
4
Computation of Definite Integrals. Quadratures
4. 1
4.2
Trapezoidal Rule, Simpson's Formula, and the Like 4. 1 . 1 General Construction of Quadrature Formulae . 4. 1 .2 Trapezoidal Rule . . 4. 1 .3 Simpson's Formula . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . Quadrature Formulae with No Saturation. Gaussian Quadratures
35 38 40 42 42 43 43 48 53 56 57 57 59 60 61
62 62
67 68 72 73 73 75 75 77 78 79 80 89 91
91 92 93 98 102 1 02
v 4.3 4.4
II
5
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . Improper Integrals. Combination of Numerical and Analytical Methods . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . Multiple Integrals . . . . . . . . . . . . . . . . . . . 4.4. 1 Repeated Integrals and Quadrature Formulae 4.4.2 The Use of Coordinate Transformations 4.4.3 The Notion of Monte Carlo Methods .
Systems of Scalar Equations Systems of Linear Algebraic Equations: Direct Methods
5.1
5.2
5.3
5.4
5.5 5.6
Different Forms of Consistent Linear Systems 5. 1 . 1 Canonical Form of a Linear System . . . . . . 5 . 1 .2 Operator Form . . . . . . . . . . . . . . . . . 5 . 1 . 3 Finite-Difference Dirichlet Problem for the Poisson Equation . . . . . . . . . . . . . . Exercises . . . . . . . . . . . Linear Spaces, Norms, and Operators 5.2. 1 Normed Spaces . . . . . . . 5.2.2 Norm of a Linear Operator . Exercises . . . . . . . . Conditioning of Linear Systems . . 5.3. 1 Condition Number . . . . . 5.3.2 Characterization of a Linear System by Means of Its Condition Number . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . Gaussian Elimination and Its Tri-Diagonal Version 5 .4. 1 Standard Gaussian Elimination . . 5.4.2 Tri-Diagonal Elimination . . . . . . . . . . 5.4.3 Cyclic Tri-Diagonal Elimination . . . . . . 5.4.4 Matrix Interpretation of the Gaussian Elimination. LU Factorization . . . . . . . . . . . . . . 5.4.5 Cholesky Factorization . . . . . . . . . . . . . . 5 .4.6 Gaussian Elimination with Pivoting . . . . . . . 5 .4.7 An Algorithm with a Guaranteed Error Estimate Exercises . . . . . . . . . . . . . . . . . . . . . Minimization of Quadratic Functions and Its Relation to Linear Systerns . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . The Method of Conjugate Gradients 5.6. 1 Construction of the Method 5 .6.2 Flexibility in Specifying the Operator A 5.6.3 Computational Complexity . Exercises . . . . . . . . . . . . . . . .
107 108 1 10 1 10 111 1 12 1 13 115 119
1 20 1 20 121
121 1 24 1 24 1 26 1 29 131 133 1 34 1 36 139 1 40 141 145 148 149 153 1 54 155 156 1 57 1 59 1 59 1 59 1 63 1 63 1 63
vi 5.7
6
Finite Fourier Series . . . . . . . . . . . . . . . . . . . . . 5.7. 1 Fourier Series for Grid Functions . . . . . . . . . . 5.7.2 Representation of Solution as a Finite Fourier Series 5.7.3 Fast Fourier Transform . Exercises . . . . . . . . . . . . . .
Iterative Methods for Solving Linear Systems
6. 1
Richardson Iterations and the Like . . . . . 6. 1 . 1 General Iteration Scheme . . . . . . 6. 1 .2 A Necessary and Sufficient Condition for Convergence . 6. 1 .3 The Richardson Method for A = A * > 0 . 6. 1 .4 Preconditioning . 6. 1 . 5 Scaling . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . 6.2 Chebyshev Iterations and Conjugate Gradients 6.2.1 Chebyshev Iterations 6.2.2 Conjugate Gradients Exercises . . . . . . 6.3 Krylov Subspace Iterations . 6.3.1 Definition of Krylov Subspaces 6.3.2 GMRES . . Exercises . . . . 6.4 Multigrid Iterations . . . 6.4. 1 Idea of the Method 6.4.2 Description of the Algorithm . 6.4.3 Bibliography Comments Exercises . . . . . . . . . . . .
.
7
Overdetermined Linear Systems. The Method of Least Squares
7. 1
Examples of Problems that Result in Overdetermined Systems 7. 1 . 1 Processing of Experimental Data. Empirical Formulae 7. 1 .2 Improving the Accuracy of Experimental Results by Increasing the Number of Measurements . . . . . . . . 7.2 Weak Solutions of Full Rank Systems. QR Factorization 7.2. 1 Existence and Uniqueness o f Weak Solutions . . . 7.2.2 Computation of Weak Solutions. QR Factorization 7.2.3 Geometric Interpretation of the Method of Least Squares 7.2.4 Overdetermined Systems in the Operator Form . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Rank Deficient Systems. Singular Value Decomposition . . . . 7.3.1 Singular Value Decomposition and Moore-Penrose Pseudoinverse . . . . . . . . . . . . 7.3.2 Minimum Norm Weak Solution Exercises . . . . . . . . . . . . .
1 64 1 65 168 1 69 171 173
174 1 74 1 78 181 1 88 1 92 1 93 1 94 1 94 1 96 1 97 1 98 1 99 20 1 204 204 205 208 210 210 211
21 1 211
213 214 214 217 220 221 222 225 225 227 229
vii 8
8. 1
8.2
8.3
III
9
231
Numerical Solution of Nonlinear Equations and Systems
Commonly U sed Methods of Rootfinding 8. 1 . 1 The Bisection Method 8. 1 .2 The Chord Method . 8 . 1 .3 The Secant Method . 8. 1 .4 Newton's Method . . Fixed Point Iterations . . . . 8.2. 1 The Case of One Scalar Equation 8.2.2 The Case of a System of Equations Exercises . . . . . . . . . . . . . . Newton's Method . . . . . . . . . . . . . . 8.3. 1 Newton's Linearization for One Scalar Equation 8.3.2 Newton's Linearization for Systems 8.3.3 Modified Newton's Methods Exercises . . . . . . . . . . . . . .
233 233 234 235 236 237 237 240 242 242 242 244 246 247
The Method of Finite Differences for the Numerical Solution of Differential Equations 249
253
Numerical Solution of Ordinary Differential Equations
9. 1
9.2
9.3
Examples of Finite-Difference Schemes. Convergence 253 9. 1 . 1 Examples of Difference Schemes . . . . . . . 254 9. 1 .2 Convergent Difference Schemes . . . . . . . . 256 259 9. 1 .3 Verification of Convergence for a Difference Scheme Approximation of Continuous Problem by a Difference Scheme. 260 Consistency . . . . . . . . . . . . . . . . . . . . 9.2. 1 Truncation Error 8/(11) . . . . . . . . . . 261 262 9.2.2 Evaluation of the Truncation Error 8/(11) 264 9.2.3 Accuracy of Order hk . . . . . . . . . . . 265 9.2.4 Examples . . . . . . . . . . . . . . . . . 9.2.5 Replacement of Derivatives by Difference Quotients 269 269 9.2.6 Other Approaches to Constructing Difference Schemes . Exercises . . . . . . . . . . . 27 1 Stability of Finite-Difference Schemes . . . . . . . . . . . . . . 27 1 9.3. 1 Definition of Stability . . . . . . . . . . . . . . . . 272 9.3.2 The Relation between Consistency, Stability, and Conver273 gence . . . . . . . . . . . . . . . . . . . . . 277 9.3.3 Convergent Scheme for an Integral Equation 9.3.4 The Effect of Rounding . . . . 278 280 9.3.5 General Comments. A-stability 283 Exercises . . . . . . . . . The Runge-Kutta Methods . . . . 284 9.4 . 1 The Runge-Kutta Schemes 284 9.4.2 Extension to Systems . 286 Exercises . . . . . . . . . 288 .
9.4
.
viii 9.5
9.6 9.7
Solution of Boundary Value Problems 9.5 . 1 The Shooting Method . . 9.5.2 Tri-Diagonal Elimination . 9.5.3 Newton's Method . . . . . Exercises . . . . . . . . . Saturation of Finite-Difference Methods by Smoothness Exercises . . . . . . . . The Notion of Spectral Methods Exercises . . . . . . . .
10 Finite-Difference Schemes for Partial Differential Equations
1 0 . 1 Key Definitions and Illustrating Examples 1 0. 1 . 1 Definition of Convergence 10. 1 .2 Definition of Consistency . . . . 10. 1.3 Definition of Stability . . . . . . 10. 1 .4 The Courant, Friedrichs, and Lewy Condition 10. 1 .5 The Mechanism of lnstability . . . . . . . . 10. 1 .6 The Kantorovich Theorem . . . . . . . . . . 10. 1 .7 On the Efficacy of Finite-Difference Schemes . 1 0 . 1 . 8 Bibliography Comments . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . 1 0.2 Construction of Consistent Difference Schemes . . . . 1 0.2. 1 Replacement of Derivatives by Difference Quotients 10.2.2 The Method of Undetermined Coefficients 10.2.3 Other Methods. Phase Error . 10.2.4 Predictor-Corrector Schemes . . . . . . . . Exercises . . . . . . . . . . . . . . . . . 10.3 Spectral Stability Criterion for Finite-Difference Cauchy Problems 10.3. 1 Stability with Respect to Initial Data . . . . . 10.3.2 A Necessary Spectral Condition for Stability 10.3.3 Examples . . . . . . . . . . . . . . . . . . . 10.3.4 Stability in C . . . . . . . . . . . . . . . . . 10.3.5 Sufficiency of the Spectral Stability Condition in 12 1 0.3.6 Scalar Equations vs. Systems . . . . . . . Exercises . . . . . . . . . . . . . . . . . 1 0.4 Stability for Problems with Variable Coefficients 10.4. 1 The Principle of Frozen Coefficients . . . 10.4.2 Dissipation of Finite-Difference Schemes Exercises . . . . . . . . . . . . . . . 1 0.5 Stability for Initial Boundary Value Problems . . 10.5. 1 The Babenko-Gelfand Criterion . . . . . 10.5.2 Spectra of the Families of Operators. The GodunovRyaben'kii Criterion 10.5.3 The Energy Method . . . . . . . . . . . .
288 289 29 1 29 1 292 293 300 301 306 307
307 307 309 312 317 319 320 322 323 324 327 327 333 340 344 345 349 349 350 352 362 362 365 367 369 369 372 377 377 377
385 402
IX
1 0.5.4 A Necessary and Sufficient Condition of Stability. Kreiss Criterion . . . . . . . . . . Exercises . . . . . . . . . . . . . 10.6 Maximum Principle for the Heat Equation 1 0.6. 1 An Explicit Scheme 10.6.2 An Implicit Scheme Exercises . . . . . .
The
11 Discontinuous Solutions and Methods of Their Computation
1 1. 1 Differential Form of an Integral Conservation Law . . . . 1 1 . 1 . 1 Differential Equation in the Case of Smooth Solutions 1 1 . 1 .2 The Mechanism of Formation of Discontinuities 1 1 . 1 .3 Condition at the Discontinuity . . . . . . . . . 1 1 . 1 .4 Generalized Solution of a Differential Problem 1 1 . 1 .5 The Riemann Problem . . . Exercises . . . . . . . . . . 1 1 .2 Construction of Difference Schemes 1 1 .2. 1 Artificial Viscosity . . . . . 1 1 .2.2 The Method of Characteristics 1 1 .2.3 Conservative Schemes. The Godunov Scheme Exercises . . . . . . . . . .
12 Discrete Methods for Elliptic Problems
1 2. 1 A Simple Finite-Difference Scheme. The Maximum Principle 1 2. 1 . 1 Consistency . . . . . . . . . . . . 1 2. 1 .2 Maximum Principle and Stability 12. 1 .3 Variable Coefficients . . . . . . . Exercises . . . . . . . . . . . . . 1 2.2 The Notion of Finite Elements. Ritz and Galerkin Approximations 1 2.2. 1 Variational Problem . 1 2.2.2 The Ritz Method . . . . . . . . . . . . . . . 12.2.3 The Galerkin Method . . . . . . . . . . . . . 12.2.4 An Example of Finite Element Discretization 1 2.2.5 Convergence of Finite Element Approximations . Exercises . . . . . . . . . . . . . . . . . . . . .
IV
409 418 422 422 425 426 427
428 428 429 43 1 433 434 436 436 437 438 439 444 445
446 447 448 45 1 452 453 454 458 460 464 466 469
The Methods of Boundary Equations for the Numerical 471 Solution of Boundary Value Problems
13 Boundary Integral Equations and the Method of Boundary Elements
1 3 . 1 Reduction of Boundary Value Problems to Integral Equations 1 3.2 Discretization of Integral Equations and Boundary Elements 1 3.3 The Range of Applicability for Boundary Elements . . . . . .
475
475 479 480
x
14 Boundary Equations with Projections and the Method of Difference Po483 tentials
14. 1 Formulation of Model Problems . . . . . 14. 1. 1 Interior Boundary Value Problem 1 4. 1 .2 Exterior Boundary Value Problem 1 4. 1 .3 Problem of Artificial Boundary Conditions 14.1.4 Problem of Two Subdomains . 14. 1 .5 Problem of Active Shielding . 1 4.2 Difference Potentials . . . . . . . . . 1 4.2. 1 Auxiliary Difference Problem 1 4.2.2 The Potential u+ = P+vy . . . 14.2.3 Difference Potential u - = P-vy 14.2.4 Cauchy Type Difference Potential w± = P±vy . 14.2.5 Analogy with Classical Cauchy Type Integral 1 4.3 Solution of Model Problems . . . . . . . 14.3.1 Interior Boundary Value Problem . . . . . 1 4.3.2 Exterior Boundary Value Problem . . . . . 14.3.3 Problem of Artificial Boundary Conditions 14.3.4 Problem of Two Subdomains . 14.3.5 Problem of Active Shielding 1 4.4 General Remarks . . . . 1 4.5 Bibliography Comments . . . . . .
List of Figures Referenced Books Referenced Journal Articles Index
484 485 485 485 486 487 488 488 489 492 493 497 498 498 500 50 1 501 503 505 506
507
509
517 521
Preface
This book introduces the key ideas and concepts of numerical analysis. The dis cussion focuses on how one can represent different mathematical models in a form that enables their efficient study by means of a computer. The material learned from this book can be applied in various contexts that require the use of numerical meth ods. The general methodology and principles of numerical analysis are illustrated by specific examples of the methods for real analysis, linear algebra, and differential equations. The reason for this particular selection of subjects is that these methods are proven, provide a number of well-known efficient algorithms, and are used for solving different applied problems that are often quite distinct from one another. The contemplated readership of this book consists of beginning graduate and se nior undergraduate students in mathematics, science and engineering. It may also be of interest to working scientists and engineers. The book offers a first mathematical course on the subject of numerical analysis. It is carefully structured and can be read in its entirety, as well as by selected parts. The portions of the text considered more difficult are clearly identified; they can be skipped during the first reading without creating any substantial gaps in the material studied otherwise. In particular, more difficult subjects are discussed in Sections 2.3 . 1 and 2.3.3, Sections 3. 1 .3 and 3.2.7, parts of Sections 4.2 and 9.7, Section 10.5, Section 12.2, and Chapter 14. Hereafter, numerical analysis is interpreted as a mathematical discipline. The ba sic concepts, such as discretization, error, efficiency, complexity, numerical stability, consistency, convergence, and others, are explained and illustrated in different parts of the book with varying levels of depth using different subject material. Moreover, some ideas and views that are addressed, or at least touched upon in the text, may also draw the attention of more advanced readers. First and foremost, this applies to the key notion of the saturation of numerical methods by smoothness. A given method of approximation is said to be saturated by smoothness if, because of its design, it may stop short of reaching the intrinsic accuracy limit (unavoidable error) determined by the smoothness of the approximated solution and by the discretization parameters. If, conversely, the accuracy of approximation self-adjusts to the smoothness, then the method does not saturate. Examples include algebraic vs. trigonometric interpola tion, Newton-Cotes vs. Gaussian quadratures, finite-difference vs. spectral methods for differential equations, etc. Another advanced subject is an introduction to the method of difference potentials in Chapter 14. This is the first account of difference potentials in the educational literature. The method employs discrete analogues of modified Calderon's potentials and boundary projection operators. It has been successfully applied to solving a variety of direct and inverse problems in fluids, acoustics, and electromagnetism. This book covers three semesters of instruction in the framework of a commonly Xl
xii used curriculum with three credit hours per semester. Three semester-long courses can be designed based on Parts I, II, and III of the book, respectively. Part I includes interpolation of functions and numerical evaluation of definite integrals. Part II cov ers direct and iterative solution of consistent linear systems, solution of overdeter mined linear systems, and solution of nonlinear equations and systems. Part III discusses finite-difference methods for differential equations. The first chapter in this part, Chapter 9, is devoted to ordinary differential equations and serves an intro ductory purpose. Chapters 10, 1 1 , and 12 cover different aspects of finite-difference approximation for both steady-state and evolution partial differential equations, in cluding rigorous analysis of stability for initial boundary value problems and ap proximation of the weak solutions for nonlinear conservation laws. Alternatively, for the curricula that introduce numerical differentiation right after the interpolation of functions and quadratures, the material from Chapter 9 can be added to a course based predominantly on Part I of the book. A rigorous mathematical style is maintained throughout the book, yet very little use is made of the apparatus of functional analysis. This approach makes the book accessible to a much broader audience than only mathematicians and mathematics majors, while not compromising any fundamentals in the field. A thorough expla nation of the key ideas in the simplest possible setting is always prioritized over various technicalities and generalizations. All important mathematical results are accompanied by proofs. At the same time, a large number of examples are provided that illustrate how those results apply to the analysis of individual problems. This book has no objective whatsoever of describing as many different methods and techniques as possible. On the contrary, it treats only a limited number of well known methodologies, and only for the purpose of exemplifying the most fundamen tal concepts that unite different branches of the discipline. A number of important results are given as exercises for independent study. Altogether, many exercises supplement the core material; they range from elementary to quite challenging. Some exercises require computer implementation of the corresponding tech niques. However, no substantial emphasis is put on issues related to programming. In other words, any computer implementation serves only as an illustration of the rel evant mathematical concepts and does not carry an independent learning objective. For example, it may be useful to have different iteration schemes implemented for a system of linear algebraic equations. By comparing how their convergence rates depend on the condition number, one can subsequently judge the efficiency from a mathematical standpoint. However, other efficiency issues, e.g., runtime efficiency determined by the software and/or computer platform, are not addressed as there is no direct relation between them and the mathematical analysis of numerical methods. Likewise, no substantial emphasis is put on any specific applications. Indeed, the goal is to clearly and concisely present the key mathematical concepts pertinent to the analysis of numerical methods. This provides a foundation for the subsequent specialized training. Subjects such as computational fluid dynamics, computational acoustics, computational electromagnetism, etc., are very well addressed in the lit erature. Most corresponding books require some numerical background from the reader, the background of precisely the kind that the current text offers.
Acknowledgments
This book has a Russian language prototype [RyaOO] that withstood two editions: in 1 994 and in 2000. It serves as the main numerical analysis text at Moscow Institute for Physics and Technology. The authors are most grateful to the rector of the Insti tute at the time, Academician O. M. Belotserkovskii, who has influenced the original concept of this textbook. Compared to [RyaOO] , the current book is completely rewritten. It accommodates the differences that exist between the Russian language culture and the English lan guage culture of mathematics education. Moreover, the current textbook includes a very considerable amount of additional material. When writing Part III of the book, we exploited the ideas and methods previously developed in [GR64] and [GR87]. When writing Chapter 14, we used the approach of [Rya02, Introduction]. We are indebted to all our colleagues and friends with whom we discussed the subject of teaching the numerical analysis. The book has greatly benefited from all those discussions. In particular, we would like to thank S. Abarbanel, K. Brush linskii, V. Demchenko, A. Chertock, L. Choudov, L. Demkowicz, A. Ditkowski, R. Fedorenko, G. Fibich, P. Gremaud, T. Hagstrom, V. Ivanov, C. Kelley, D. Keyes, A. Kholodov, V. Kosarev, A. Kurganov, C. Meyer, N. Onofrieva, I. Petrov, V. Pirogov, L. Strygina, E. Tadmor, E. Turkel, S. Utyuzhnikov, and A. Zabrodin. We also remem ber the late K. Babenko, O. Lokutsievskii, and Yu. Radvogin. We would like to specially thank Alexandre Chorin of UC Berkeley and David Gottlieb of Brown University who read the manuscript prior to publication. A crucial and painstaking task of proofreading the manuscript was performed by the students who took classes on the subject of this book when it was in prepara tion. We are most grateful to L. Bilbro, A. Constantinescu, S. Ernsberger, S. Grove, A. Peterson, H. Qasimov, A. Sampat, and W. Weiselquist. All the imperfections still remaining are a sole responsibility of the authors. It is also a pleasure for the second author to thank Arje Nachman and Richard Albanese of the US Air Force for their consistent support of the second author's research work during and beyond the period of time when the book was written. And last, but not least, we are very grateful to the CRC Press Editor, Sunil Nair, as well as to the company staff in London and in Florida, for their advice and assistance. Finally, our deepest thanks go to our families for their patience and understanding without which this book project would have never been completed. V.
Ryaben'kii, Moscow, Russia S. Tsynkov, Raleigh, USA
August 2006
xiii
Chapter 1 Introduction
Modern numerical mathematics provides a theoretical foundation behind the use of electronic computers for solving applied problems. A mathematical approach to any such problem typically begins with building a model for the phenomenon of interest (situation, process, object, device, laboratory/experimental setting, etc.). Classical examples of mathematical models include definite integrals, equation of a pendulum, the heat equation, equations of elasticity, equations of electromagnetic waves, and many other equations of mathematical physics. For comparison, we should also mention here a model used in formal logics - the Boolean algebra. Analytical methods have always been considered a fundamental means for study ing the mathematical models. In particular, these methods allow one to obtain closed form exact solutions for some special cases (for example, tabular integrals). There are also classes of problems for which one can obtain a solution in the form of a power series, Fourier series, or some other expansion. In addition, a certain role has always been played by approximate computations. For example, quadrature formu lae are used for the evaluation of definite integrals. The advent of computers in the middle of the twentieth century has drastically increased our capability of performing approximate computations. Computers have essentially transformed approximate computations into a dominant tool for the anal ysis of mathematical models. Analytical methods have not lost their importance, and have even gained some additional "functionality" as components of combined ana lytical/computational techniques and as verification tools. Yet sophisticated math ematical models are analyzed nowadays mostly with the help of computers. Com puters have dramatically broadened the applicability range of mathematical meth ods in many traditional areas, such as mechanics, physics, and engineering. They have also facilitated a rapid expansion of the mathematical methods into various non-traditional fields, such as management, economics, finance, chemistry, biology, psychology, linguistics, ecology, and others. Computers provide a capability of storing large (but still finite) arrays of numbers, and performing arithmetic operations with these numbers according to a given pro gram that would run with a fast (but still finite) execution speed. Therefore, comput ers may only be appropriate for studying those particular models that are described by finite sets of numbers and require no more than finite sequences of arithmetic operations to be performed. Besides the arithmetic operations per se, a computer model can also contain comparisons between numbers that are typically needed for the automated control of subsequent computations. In the traditional fields, one frequently employs such mathematical models as
2
A Theoretical Introduction to Numerical Analysis
functions, derivatives, integrals, and differential equations. To enable the use of com puters, these original models must therefore be (approximately) replaced by the new models that would only be based on finite arrays of numbers supplemented by finite sequences of arithmetic operations for their processing (i.e., finite algorithms). For example, a function can be replaced by a table of its numerical values; the derivative
df =lim f(x+h) -f(x) dx h->O h can be replaced by an approximate formula, such as
j'(x) � f(x+h) - f(x) ,
h
where h is fixed (and small); a definite integral can be replaced by its integral sum; a boundary value problem for the differential equation can be replaced by the problem of finding its solution at the discrete nodes of some grid, so that by taking a suitable (i.e., sufficiently small) grid size an arbitrary desired accuracy can be achieved. In so doing, among the two methods that could seem equivalent at first glance, one may produce good results while the other may turn out completely inapplicable. The reason can be that the approximate solution it generates would not approach the exact solution as the grid size decreases, or that the approximate solution would turn out overly sensitive to the small round-off errors. The subject of numerical analysis is precisely the theory of those models and al gorithms that are applicable, i.e., that can be efficiently implemented on comput ers. This theory is intimately connected with many other branches of mathematics: Approximation theory and interpolation of functions, ordinary and partial differen tial equations, integral equations, complexity theory for functional classes and algo rithms, etc., as well as with the theory and practice of programming languages. In general, both the exploratory capacity and the methodological advantages that com puters deliver to numerous applied areas are truly unparalleled. Modern numerical methods allow, for example, the computation of the flow of fluid around a given aerodynamic configuration, e.g., an airplane, which in most cases would present an insurmountable task for analytical methods (like a non-tabular integral). Moreover, the use of computers has enabled an entirely new scientific methodol ogy known as computational experiment, i.e., computations aimed at verifying the hypotheses, as well as at monitoring the behavior of the model, when it is not known ahead of time what may interest the researcher. In fact, computational experiment may provide a sufficient level of feedback for the original formulation of the problem to be noticeably refined. In other words, numerical computations help accumulate the vital information that eventually allows one to identify the most interesting cases and results in a given area of study. Many remarkable observations, and even dis coveries, have been made along this route that empowered the development of the theory and have found important practical applications as well. Computers have also facilitated the application of mathematical methods to non traditional areas, for which few or no "compact" mathematical models, such as dif ferential equations, are readily available. However, other models can be built that
Introduction
3
lend themselves to the analysis by means of a computer. A model of this kind can often be interpreted as a direct numerical counterpart (such as encoding) of the ob ject of interest and of the pertinent relations between its elements (e.g., a language or its abridged subset and the corresponding words and phrases). The very possi bility of studying such models on a computer prompts their construction, which, in tum, requires that the rules and guiding principles that govern the original object be clearly and unambiguously identified. On the other hand, the results of computer simulations, e.g., a machine translation of the simplified text from one language to another, provide a practical criterion for assessing the adequacy of the theories that constitute the foundation of the corresponding mathematical model (e.g., linguistic theories). Furthermore, computers have made it possible to analyze probabilistic models that require large amounts of test computations, as well as the so-called imitation models that describe the object or phenomenon of interest without simplifications (e.g., functional properties of a telephone network). The variety of problems that can benefit from the use of computers is huge. For solving a given problem, one would obviously need to know enough specific detail. Clearly, this knowledge cannot be obtained ahead of time for all possible scenarios. Therefore, the purpose of this book is rather to provide a systematic perspective on those fundamental ideas and concepts that span across different applied disciplines and can be considered established in the field of numerical analysis. Having mastered the material of this book, one should encounter little or no difficulties when receiving subsequent specialized training required for the successful work in a given research or industrial field. The general methodology and principles of numerical analysis are illustrated in the book by "sampling" the methods designed for mathematical analy sis, linear algebra, and differential equations. The reason for this particular selection is that the aforementioned methods are most mature, lead to a number of well-known, efficient algorithms, and are extensively used for solving various applied problems that are often quite distant from one another. Let us mention here some of the general ideas and concepts that require the most thorough attention in every particular setting. These general ideas acquire a concrete interpretation and meaning in the context of each specific problem that needs to be solved on a computer. They are the discretization of the problem, conditioning of the problem, numerical error, and computational stability of a given algorithm. In addition, comparison of the algorithms along different lines obviously plays a central role when selecting a specific method. The key criteria for comparison are accuracy, storage, and operation count requirements, as well as the efficiency of utilization of the input information. On top of that, different algorithms may vary in how amenable they are to parallelization - a technique that allows one to conduct computations simultaneously on multi-processor computer platforms. In the rest of the Introduction, we provide a brief overview of the foregoing no tions and concepts. It helps create a general perspective on the subject of numerical mathematics, and establishes a foundation for studying the subsequent material.
4
A Theoretical Introduction to Numerical Analysis
1.1
Discretization
Let f(x) be a function of the continuous argument x E [0, 1]. Assume that this function provides (some of) the required input data for a given problem that needs to be approximately solved on a computer. The value of the function f at every given x can be either measured or obtained numerically. Then, to store this function in the memory of a computer, one may need to approximately characterize it with a table of values at a finite set of points: Xl , x2 , ' " ,xn . This is an elementary example of discretization: The problem of storing the function defined on the interval [0, I], which is a continuum o f points, is replaced b y the problem o f storing a table o f its discrete values at the subset of points X l , x2 , ' " ,X/l that all belong to this interval. Let now f(x) be sufficiently smooth, and assume that we need to calculate its derivative at a given point x. The problem of exactly evaluating the expression
f(x + h) - f(x) j' (x) = hlim -->O h that contains a limit can be replaced by the problem of computing an approximate value of this expression using one of the following formulae:
j' (x) ;:::; f(x + h)h - f(x) ,
(1.1)
- h) , (1.2) j' (x);:::; f(x) - f(x h - f(x - h) . (1 .3 ) j' (x) ;:::; f(x + h) 2h Similarly, the second derivative f" (x) can be replaced by the finite formula: (1.4) f" (X) ;:::; f(x + h) - 2fh x) + f(x - h) . One can show that all these formulae become more and more accurate as h becomes smaller; this is the subject of Exercise I , and the details of the analysis can be found in Section 9.2. 1. Moreover, for every fixed h, each formula (1.1)-(1.4) will only require a finite set of values of f and a finite number of arithmetic operations. These formulae are examples of discretization for the derivatives l' (x) and f" (x).
�
Let us now consider a boundary value problem:
d2y 2 = cosx, (1 .5) dx2 - x Y y(O) = 2, y( l ) = 3 , where the unknown function y = y(x) is defined on the interval 0 � x � I. To con struct a discrete approximation of problem (1.5), let us first partition the interval
5
Introduction
into N equal sub-intervals of size h = N- 1 • Instead of the continuous func tion y(x), we will be looking for a finite set of its values Yo, Y I , ... ,YN on the grid Xk = kh, k = 0, 1 , . . . , N. At the interior nodes of this grid: Xb k = 1 ,2 , . . . ,N - 1 , we can approximately replace the second derivative Y" (x) by expression ( 1.4). After substituting into the differential equation of ( 1 .5) this yields:
[0, 1 ]
Yk+l -
�k + Yk-l
_
(kh ?Yk = cos (kh ) ,
k=1,2,. . . ,N -
1.
( 1 . 6)
Furthermore, the boundary conditions at x = 0 and at x = 1 from ( 1 .5) translate into: Yo = 2 ,
YN
=
3.
( 1 .7)
The system of N + 1 linear algebraic equations ( 1 .6), ( 1 .7) contains exactly as many unknowns Yo, YI , . . . ,Y N, and renders a discrete counterpart of the boundary value problem (1 .5). One can, in fact, show that the finer the grid, i.e., the larger the N, the more accurate will the approximation be that the discrete solution of problem (1.6), ( 1 .7) provides for the continuous solution of problem (1.5). Later, this fact will be formulated and proven rigorously. Let us denote the continuous boundary value problem (1.5) by M� , and the discrete boundary value problem ( 1 .6), ( 1 .7) by MN. By taking N = 2,3,. . . , we associate an infinite sequence of discrete problems {MN} with the continuous problem M� . When computing the solution to a given problem MN for any fixed N, we only have to work with a finite array of numbers that specify the input data, and with a finite set of unknown quantities YO,YI, Y2 , . . . , YN. It is, however, the entire infinite sequence of finite discrete models {MN} that plays the central role from the standpoint of numer ical mathematics. Indeed, as those models happen to be more and more accurate, we can always choose a sufficiently large N that would guarantee any desired accuracy of approximation. In general, there are many different ways of transitioning from a given continuous problem M� to the sequence {MN} of its discrete counterparts. In other words, the approximation ( 1 . 6 ), ( 1 .7) of the boundary value problem (1.5) is by no means the only one possible. Let {MN} and {M�} be two sequences of approximations, and let us also assume that the computational costs of obtaining the discrete solutions of MN and M� are the same. Then, a better method of discretization would be the one that provides the same accuracy of approximation with a smaller value of N. Let us also note that for two seemingly equivalent discretization methods MN and M�, it may happen that one will approximate the continuous solution of problem M� with an increasingly high accuracy as N increases, whereas the other will yield "an approximate solution" that would bear less and less resemblance to the continuous solution of M�. We will encounter situations like this in Part III of the book, where we also discuss how the corresponding difficulties can be partially or fully overcome. Exercises
1. LetI(x) have as many bounded derivatives as needed. Show that the approximation error of formulae (1.1), (1 .2), (1.3), and (1 .4), is 0(11), O(h), 0(h2), and 0(h2).
6
1 .2
A Theoretical Introduction to Numerical Analysis
Conditioning
Speaking in most general terms, for any given problem one can basically identify the input data and the output result(s), i.e., the solution, so that the former determine the latter. In this book, we will mostly analyze problems for which the solution exists and is unique. If, in addition, the solution depends continuously on the data, i.e., if for a vanishing perturbation of the data the corresponding perturbation of the solution will also be vanishing, then the problem is said to be well-posed. A somewhat more subtle characterization of the problem, on top of its well posedness, is known as the conditioning. It has to do with quantifying the sensitivity of the solution, or some of its key characteristics, to perturbations of the input data. This sensitivity may vary strongly for different problems that could otherwise look very similar. If it is "low" (weak), then the problem is said to be well conditioned; if, conversely, the sensitivity is "high" then the problem is ill conditioned. The notions of low and high are, of course, problem-specific. We emphasize that the concept of conditioning pertains to both continuous and discrete problems. Typically, not only do ill conditioned problems require excessively accurate definition of the input data, but also appear more difficult for computations. Consider, for example, the quadratic equation x2 - 2ax + 1 = 0 for lal > 1 . It has two real roots that can be expressed as functions of the argument a: XI,2 a ± v'a2 - 1 . We will interpret a as the datum in the problem, and Xl = XI ( a) and X2 = X2 (a) as the corresponding solution. Clearly, the sensitivity of the solution to the perturbations of a can be characterized by the magnitude of the derivatives = 1 ± I . Indeed, LlxI ,2 � �a. We can easily see that the derivatives =
d;�2 v':Zd;�2 d;�2 are small for large ial, but they become large when a approaches I. We can
therefore conclude that the problem of finding the roots of x2 - 2ax + 1 = 0 is well conditioned when lal » 1 , and ill conditioned when lal = 0'( 1 ) . We should also note that conditioning can be improved if, instead of the original quadratic equation, we consider its equivalent x2 - J X + I = 0, where f3 = a + v'a2 - 1 . In this case, XI = f3 and X2 = f3 -I; the two roots coincide for 1f31 = 1 , or equivalently, I a I = 1 . However, the problem of evaluating f3 = f3 (a) is still ill conditioned near I a I = 1 . Our next example involves a simple ordinary differential equation. Let y = y(t) be the concentration of some substance at the time t, and assume that it satisfies:
�p2
dy - l Oy = O. dt Let us take an arbitrary to , 0 :s; to :s; 1 , and perform an approximate measurement of the actual concentration yO = y(to ) at this moment of time, thus obtaining: yl t =tQ = Yo· Our overall task will be to determine the concentration y y(t) at all other moments of time t from the interval [0, 1 ] . =
Introduction
7
If we knew the quantity Yo = y(to ) exactly, then we could have used the exact fonnula available for the concentration: ( 1 .8) y(t) yoelO(t-to). We, however, only know the approximate value Yo � Yo of the unknown quantity Yo. =
Therefore, instead of ( 1 .8), the next best thing is to employ the approximate fonnula:
y* (t) yoeIO(t-to). =
( 1 .9)
Clearly, the error y* -y of the approximate fonnula ( 1 .9) is given by:
y* (t) -y(t) (Y o - yo)elO(t-to), 0 � t � 1. Assume now that we need to measure Yo to the the accuracy 8, jyo - Yo j < 8, that =
would be sufficient to guarantee an initially prescribed tolerance
f
for determining
y(t) everywhere on the interval 0 � t � 1 , i.e., would guarantee the error estimate: jy* (t) -y(t) j < f, O � t � l . It is easy to see that max jy*(t) - y(t) j = jy* ( I) - y( l ) j jyo _Yo j elO(l-to). This 09<; 1 =
yields the following constraint that the accuracy 8 of measuring Yo must satisfy: 8 � fe-lO(l-to).
(LlO)
Let Yo be measured at the moment of time to = O. Then, inequality ( 1 . 10) would imply that this measurement has to be elO times, i.e., thousands of times, more ac curate than the required guaranteed accuracy of the result f. In other words, the answer y(t) appears quite sensitive to the error in specifying the input data Yo, and the problem is ill conditioned. On the other hand, ifYo were to be measured at to = 1 , then 8 f, and it would be sufficient to conduct the measurement with a considerably lower accuracy than the one required in the case of to O. This problem is well conditioned. =
=
Exercises
1. Consider the problem of computing y(x) = l�:� as a function ofx, forx E (1/2, 1) and als o for x E (-1,0). On w hich of the tw o intervals is this problem better conditioned with res pect to the perturbations ofx? 2. Lety = v'2 - 1. Equivalently, one can w ritey = (v'2 + 1) - I . Which of the tw o formu lae is more s ensitive tothe error w hen v'2 is approximated by a finite decimal fraction? Hint. Compare abs olute values of derivatives forthe functions (x- 1 ) and (x+ 1) -I.
1.3
Error
In any computational problem, one needs to find the solution given some appropri ate input data. If the solution can be obtained with an ideal accuracy, then there is no
A Theoretical Introduction to Numerical Analysis
8
error. Typically, however, there is a certain error content in every feasible numerical solution. This error may be attributed to (at least) three different mechanisms. First, the input data are often specified with some degree of uncertainty that, in turn, will generate uncertainty in the corresponding output. Then, the solution to the problem of interest may only be obtained with an error called unavoidable error. Second, even if we eliminate the foregoing uncertainty by fixing the input data, and subsequently compute the solution using an approximate method, then we still won't find the solution that would exactly correspond to the specified data. There will be error due to the choice of an approximate computational procedure. Third, the chosen approximate method is not implemented exactly either, because of round-offerrors that arise when performing computations on a real machine. Therefore, the overall error in the solution consists of unavoidable error, the error of the method, and round-off error. We will now illustrate these concepts.
1 .3.1
U navoidable Error
Assume that we need to find the value y of some function y f(x) for a given x = xo. The quantity Xo , as well as the relation f itself that associates the value of the =
function with every given value of its argument, are considered the input data of the problem, whereas the quantity y y(xo) will be its solution. Now let the function f(x) be known approximately rather than exactly, say, f(x) � sinx, and suppose that f (x) may differ from sinx by no more than a specified e > 0: =
sin x - e5:.f(x) 5:. sinx + e.
( 1 . 1 1)
xo - 85:.xo 5:.xo + 8,
( 1 . 1 2)
Let the value of the argument x = Xo be also measured approximately: x = xo, so that regarding the actual Xo we can only say that
where 8 > 0 characterizes the accuracy of the measurement. One can easily see from Figure 1 . 1 that any point on the interval [a, b] of yj sinx+£ variable y, where a sin (xo - 8) - e I sin x and b = sin (xo + 8) + e, can serve in I sinx-€ bL- ---------------- : the capacity of y = f(xo ). Clearly, by taking an arbitrary y* E [a, b] as the ap proximate value of y f(xo), one can always guarantee the error estimate: =
=
ly - y*1 5:. I b - al · FIGURE 1 . 1 : Unavoidable error.
( 1 . 1 3)
For the given uncertainty in the input data, see formulae ( 1 . 1 1 ) and 0 . 1 2), this estimate cannot be considerably im proved. In fact, the best error estimate
Introduction
9
that one can guarantee is obtained by choosing y* exactly in the middle of the interval
[a, b]:
y* = y�Pt = (a + b) 12.
From Figure 1 . 1 we then conclude that
Iy - Y* I :::: I b - a 1 /2.
( 1 . 1 4)
This inequality transforms into an exact equality when y (xo ) = a or when y (xo ) = b. As such, the quantity Ib - a l /2 is precisely the unavoidable (or irreducible) error, Le., the minimum error content that will always be present in the solution and that cannot be "dodged" no matter how the approximation y* is actually chosen, simply because of the uncertainty that exists in the input data. For the optimal choice of the approximate solution y;pt the smallest error ( 1 . 1 4) can be guaranteed; otherwise, the appropriate error estimate is ( 1 . 1 3). We see, however, that the optimal error estimate ( 1 . 1 4) is not that much better than the general estimate ( 1 . 1 3). We will therefore stay within reason if we interpret any arbitrary point y* E [a, b], rather than only y;pl' as an approximate solution for y (xo ) obtained within the limits of the unavoidable error. In so doing, the quantity Ib - a l shall replace I b - a l /2 of ( 1 . 1 4) as the estimate of the unavoidable error. Along with the simplest illustrative example of Figure 1 . 1 , let us consider another example that would be a little more realistic and would involve one of the most common problem formulations in numerical analysis, namely, that of reconstructing a function of continuous argument given its tabulated values at some discrete set of points. More precisely, let the values f(Xk ) of the function f = f(x) be known at the equidistant grid nodes Xk = kh, h > 0, k = 0, ± 1 , ±2, . . . Let us also assume that the first derivative of f(x) is bounded everywhere: 1J' (x) I :::: 1 , and that together with f(xd, this is basically all the information that we have about f(x). We need to be able to obtain the (approximate) value of f(x) at an arbitrary "intermediate" point x that does not necessarily coincide with any of the nodes Xk. A large variety of methods have been developed in the literature for solving this problem. Later, we will consider interpolation by means of algebraic (Chapter 2) and trigonometric (Chapter 3) polynomials. There are other ways of building the approximating polynomials, e.g., the least squares fit, and there are other types of functions that can be used as a basis for the approximation, e.g., wavelets. Each specific method will obviously have its own accuracy. We, however, are going to show that irrespective ofany particular technique used for reconstructing f(x), there will always be error due to incomplete specification of the input data. This error merely reflects the uncertainty in the formulation; it is unavoidable and cannot be suppressed by any "smart" choice of the reconstruction procedure. Consider the simplest case f(Xk ) = 0 for all k = 0, ± I , ±2, . . . . Clearly, the func tion fl (x) == 0 has the required trivial table of values, and also I ff (x) I :::: 1 . Along with fl (x), it is easy to find another function that would satisfy the same constraints, e.g., 12 (x) � sin (�X ) . Indeed, 12 (Xk) 0 , and If�(x) 1 = I cos (�X ) I :::: 1 . We there fore see that there are at least two different functions that cannot be told apart based on the available information. Consequently, the error max I fI (x) - 12 (x) I = tJ(h) is .
=
=
x
10
A Theoretical Introduction to Numerical Analysis
unavoidable when reconstructing the function f(x), given its tabulated values f(xk) and the fact that its first derivative is bounded, no matter what specific reconstruction methodology may be employed. For more on the notion of the unavoidable error in the context of reconstructing continuous functions from their discrete values see Section 2.2.4 of Chapter 2. 1.3.2
Error o f the Method
Let y* = sinxo. The number y* belongs to the interval [a, b]; it can be considered a non-improvable approximate solution of the first problem analyzed in Section 1 .3.1. For this solution, the error satisfies estimate (1. 13) and i s unavoidable. The point y* = sin xo has been selected among other points of the interval [a, b] only because it is given by the formula convenient for subsequent analysis. To evaluate the quantity y' = sin xo on a computer, let us use Taylor's expansion for the function sinx: sinx
=x
-
x3
x5
3! 5! . . . .
-+-
-
Thus, for computing y' one can take one of the following approximate expressions:
y* � Yi = xo , (xO ) 3 Y* � Y2* = XO* - � '
( 1 . 15)
n )2k - 1 (x--,,-o* ,--_ l )k� l -c'.,. - ... * = * ( � Y Yn � (2k - 1 ) ! k By choosing a specific formulae (1.15) for the approximate evaluation o f y* , we select our method of computation. The quantity Iy* - y;' 1 is then known as the error ofthe computational method. In fact, we are considering a family of methods param eterized by the integer n. The larger the n the smaller the error, see (1. 15); and by taking a sufficiently large n we can always make sure that the associated error will be smaller than any initially prescribed threshold. It, however, does not make sense to drive the computational error much further down than the level of the unavoidable error. Therefore, the number n does not need to be taken excessively large. On the other hand, if n is taken too small so that the error of the method appears much larger than the unavoidable error, then one can say that the chosen method does not fully utilize the information about the solution that is contained in the input data, or equivalently, loses a part of this information. 1 .3.3
Round-off Error
Assume that we have fixed the computational method by selecting a particular n in ( 1 . 15), i.e., by setting y* � y�. When calculating this y:' on an actual computer, we will, generally speaking, obtain a different value Y:. due to rounding. Rounding
11
Introduction
is an intrinsic feature of the floating-point arithmetic on computers, as they only operate with numbers that can be represented as finite binary fractions of a given fixed length. As such, all other real numbers (e.g., infinite fractions) may only be stored approximately in the computer memory, and the corresponding approximation procedure is known as rounding. The error I Y� - Y� I is called the round- off error. This error shall not noticeably exceed the error of the computational method. Oth erwise, a loss of the overall accuracy will be incurred due to the round-off error. Exercises I
1. A s s ume that w e need to cal cul ate the val uey = J(x) of s ome functionJ(x), w hil e there is an uncertainty i n the i nput datax*: x* 8 ::; x ::; x* + o. H ow does the corres pondi ng unav oi dabl e error depend onx* and on0 for thefoll ow ing functi ons: -
a) b)
J(x) = s inx; J(x) = l nx, w herex > O?
F or w hat val ues ofx* , obtai ned by approxi matel y meas uri ng the"l oos e" quantity x wi th the accuracy 8, can one g uarantee onl y a one- s ided upper bound for l nx i n probl emb)? F ind thi s upper bound. 2. L et the functi on J(x) be defi ned by i ts val ues s ampl ed on the g ri d Xk = kh, w here h = 1/N and k = 0, ± 1, ±2, . . . . I n addi ti on to thes e di s crete val ues, as s ume that
max \J"(x) j ::; 1 . x
P rove that as the avail abl e i nput data are i ncompl ete, they do not, g eneralI y s peaki ng, all ow one to recons truct the functi on at an arbitrary giv en poi ntx wi th accuracy better than the unavoi dabl e errorE (h) = h2 / n2. Hint. S how that al ong wi th the function J(x) == 0, w hich obviousl y has all its g ri d val ues equal to z ero, another functi on, cp(x) = (h2 / n2) sin(Nnx), al s o has all /l i ts g ri d val ues equal to z ero, and s ati sfi es the condi ti on max j cp (x) j ::; 1 , w hil e max IJ(x) - cp(x) I x
=
x
h2 /n2.
3. L et J = J(x) be a functi on, s uch that the abs ol ute val ue of i ts s econd deri vati ve does not ex ceed 1 . S how that the approxi mation error for the formul a:
J(x + h) - J(x) 1' (x) ;:::; h w il l not ex ceed h.
4. L et J = J(x) be a functi on that has bounded s econd deri vati ve: \:Ix : IJ"(x) I ::; 1. F or any x, the val ue of the functi on J(x) is meas ured and comes out to be equal to s ome j* (x); i n s o doi ng w e as s ume that the accuracy of the meas urement g uarantees the foll owi ng es timate: IJ(x) - J* (x) 1 ::; E , E > 0. S uppos e now that w e need to approxi matel y evaluate the fi rs t derivati ve l' (x).
I Hereafter,
we will be using the symbol * to indicate the increased level of difficulty for a given problem.
12
A Theoretical Introduction to Numerical Analysis a) H ow s hall one choos e the p arameter h s o that to minimiz e the guar anteed error es timate of the app rox imate formu la:
f (x + h) - f* (x) . J' (x) � * h b) S how that g iven the ex is ting u ncertainty in the inpu t data, the u navoidable error of evalu ating J' (x) is at leas t 6 ( J£) , no matter w hat sp ecifi c method is u s ed. Cons ider tw o fu nctions, f(x) == 0 and f* (x) = c s in(xl J£). Clearly, the abs olu te valu e of the s econd derivativefor either of thes e tw ofu nctions does not ex ceed 1 . M oreover, max If(x) - * (x) I S; c. A t the s ame time,
Hint.
x
*
I d/dx
f
/
d l I
x
I
--= 6 (J£) . dx = J£ cos J£
B y compar ing the s olu tions of su b-p roblems a) and b), verif y that the sp ecifi c app rox i matefor mu la forJ' (x) g iven in a) y ields the er ror of the ir redu cible order 6 ( J£) ; and als o s how that the u navoidable error is, in fact, ex actly of order 6 ( J£).
5. F or s toring the information abou t a linear fu nction f(x) = kx + b, a S; x S; f3, that s atisfi es the inequ alities: 0 S; f(x) S; 1, w eu s e a tablew ith s ix available cells, s uch that one of the ten dig its: 0, 1 , 2, . . , 9, can be w ritten into each cell. What is the u navoidable err or of recons tru cting the fu nction, if the foreg oing s ix cells of the table are fi lled according to one of the follow ing recip es ? .
a) T hefi rs t three cells contain thefi rs t three dig its that app ear rig ht after the decimal p ointw hen the nu mberf( a) is rep res ented as a normaliz ed decimalfraction; and the remaining three cells contain the fi rs t three dig its after the decimal p oint in the normaliz ed decimal fractionforf(f3 ).
b) Let a = 0 and f3 = 10-2 . T he fi rs t three cells contain the fi rs t three dig its in the normaliz ed decimalfractionfork, thefou rth cell contai ns either0 or 1 dep ending on the s ig n of k, and the remaining tw o cells contain the fi rs t tw o dig its after the decimal p oint in the normaliz ed deci mal fraction for b. c)* S how that irresp ective of any sp ecifi c s trateg y for fi lling out the aforementioned s ix- cell table, the u navoidable er ror of recons tru cting the linear fu nction f(x) = 3 kx + b is alw ay s at leas t 10- . 6 Hint. Bu ild 10 diff erentfunctions from theforeg oing clas s, su ch that the max i mu m modu lu s of the diff erence betw een any tw o of them w ill be at leas t 10-3 .
1.4
On Methods of Computation
Suppose that a mathematical model is constructed for studying a given object or phenomenon, and subsequently this model is analyzed using mathematical and com-
Introduction
13
putational means. For example, under certain assumptions the following problem:
I
( 1 . 16)
yeO) = 0, ddtY 1=0 = 1 '
can provide an adequate mathematical model for small oscillations of a pendulum, where yet) is the pendulum displacement from its equilibrium at the time t. A study of harmonic oscillations based on this mathematical model, i.e., on the Cauchy problem ( 1 . 1 6), can benefit from a priori knowledge about the physical na ture of the object of study. In particular, one can predict, based on physical reasoning, that the motion of the pendulum will be periodic. However, once the mathematical model ( 1 . 1 6) has been built, it becomes a separate and independent object that can be investigated using any available mathematical tools, including those that have little or no relation to the physical origins of the problem. For example, the numerical value of the solution y sin t to problem ( 1 . 1 6) at any given moment of time t = Z can be obtained by expanding sinz into the Taylor series:
=
Z3 Z5 . Slll Z = Z - - + - - . . . 3! 5! , and subsequently taking its appropriate partial sum. In so doing, representation of the function sin t as a power series hardly admits any tangible physical interpretation. In general, when solving a given problem on the computer, many different meth ods, or different algorithms, can be used. Some of them may prove far superior to others. In subsequent parts of the book, we are going to describe a number of established, robust and efficient, algorithms for frequently encountered classes of problems in numerical analysis. In the meantime, let us briefly explain how the algorithms may differ. Assume that for computing the solution y to a given problem we can employ two algorithms, and that yield the approximate solutions yj (X) and Yi = (X) , respectively, where X denotes the entire required set of the input data. In so doing, a variety of situations may occur.
= AI
A I A2 ,
A2
1 .4.1
Accuracy
The algorithm
A2 may be more accurate than the algorithm A [y -yj [ » [y - yi [.
x 1 x2k-1 y� = tJ(-I)k-1 (2k - l ) ! '
I,
that is:
For example, let us approximately evaluate y = sin ! x=o using the expansion: . n
( 1 . 17)
14
A Theoretical Introduction to Numerical Analysis
The algorithmAI will correspond to taking n = 1 in formula (1. 17), and the algorithm A2 will correspond to taking n = 2 in formula (1. 17). Then, obviously, i sin O. I - yii » i sin O. l - yi i · 1 .4 . 2
Operation Count
Both algorithms may provide the same accuracy, but the computation of yj = A I (X) may require many more arithmetic operations than the computation of yi = A 2 (X), Suppose, for example, that we need to find the value of 1 x \ o24 clearly, y
=
(
I -x
--:-
)
for x = 0.99. Let AI b e the algorithm that would perform the computations directly using the given formula, i.e., by raising 0.99 to the powers 1 , 2, . . . , 1023 one after another, and subsequently adding the results. Let A2 be the algorithm that would perform the computations according to the formula:
1 024 y = 1 -1 -0.99 0.99 The accuracy of these two algorithms is the same - both are absolutely accurate provided that there are no round-off errors. However, the first algorithm requires considerably more arithmetic operations, i.e., it is computationally more expensive. Namely, for successively computing
x,
� = x · x,
...,
one will have to perform 1022 multiplications. On the other hand, to compute 0.99\ 024 one only needs 10 multiplications: 0.992 0.99 · 0.99, 0.994 = 0.992 . 0.992 , . . . ,
=
1 .4 . 3
Stability
The algorithms, again, may yield the same accuracy, but Al (X) may be computa tionally unstable, whereas A 2 (X) may be stable. For example, to evaluate y = sinx with the prescribed tolerance £ 10-3 , i.e., to guarantee iy - y* i S 10 - 3 , let us employ the same finite Taylor expansion as in formula (1. 17):
=
yi = y'i (x) =
� ( _ I )k- I (2kx2k-- I ) ! ' n
1
where n = n(£ ) is to be chosen to ensure that the inequality
iy - yi i S 10 - 3
( 1 . 1 8)
Introduction
15
will hold. The first algorithm A J will compute the result directly according to (1.18). If I x l :; then by noticing that the following inequality holds already for n = 5:
n/2,
( n ) 2n- 1 -< 10 - 3 ,
1 (2n - 1 ) ! -2 we can reduce the sum (1.18) to
:;
Clearly, the computations by this formula will only be weakly sensitive to round off errors when evaluating each term on the right-hand side. Moreover, as for I x l those terms rapidly decay when the power grows, there is no room for the cancellation of significant digits, and the algorithm A I will be computationally stable. Consider now Ix l » 1 ; for example, x = Then, for achieving the prescribed accuracy of e = 1 3 , the number n should satisfy the inequality:
n/2,
0-
100.
1002n- 1
< 1 -3 (2n - 1 ) ! - 0 ,
...,-_ ---:-
which yields an obvious conservative lower bound for n: n > 49. This implies that the terms in sum (1. 18) become small only for sufficiently large n. At the same time, the first few leading terms in this sum will be very large. A small relative error committed when computing those terms will result in a large absolute error; and since taking a difference of large quantities to evaluate a small quantity sinx ( I sinx l 1) is prone to the loss of significant digits (see Section 1 .4.4), the algorithm A I in this case will be computationally unstable. On the other hand, in the case of large x a stable algorithm A2 for evaluating sinx is also easy to build. Let us represent a given x in the form x = + z, where Izi and is integer. Then,
:;
In
I
Y2* = A2 (X)
sinx = ( - l / sin z, Z3 Z5 = ( _ 1 )i z - - + - - - .
(
Z7)
3 ! 5 ! 7!
This algorithm has the same stability properties as the algorithm A I for Ixl 1 .4.4
:; n /2
:; n/2.
Loss of S ignificant Digits
Most typically, numerical instability manifests itself through a strong amplifica tion of the small round-off errors in the course of computations. A key mechanism for the amplification is the loss of significant digits, which is a purely computer related phenomenon that only occurs because the numbers inside a computer are represented as finite (binary) fractions (see Section 1 .3.3). If computers could oper ate with infinite fractions (no rounding), then this phenomenon would not take place.
16
A Theoretical Introduction to Numerical Analysis
Consider two real numbers a and b represented in a computer by finite fractions with m significant digits after the decimal point:
a = O.a l a2a3 " . am , b = O.blb2b3 ' " bll/ ' We are assuming that both numbers are normalized and that they have the same exponent that we are leaving out for simplicity. Suppose that these two numbers are close to one another, i.e., that the first k out of the total of m digits coincide:
Then the difference a - b will only have m - k < m significant digits (provided that ak+ I > bk+ I , which we, again, assume for simplicity):
a - b = O. "-v--' O . . . OCk+ I · · · Cm· k The reason for this reduction from m to m - k, which is called the loss of significant digits, is obvious. Even though the actual numbers a and b may be represented by the fractions much longer than m digits, or by infinite fractions, the computer simply has no information about anything beyond digit number m. Even if the result a - b is subsequently normalized:
Cm+ I . . . Cm+k · f3 - k , a - b = O.Ck+ I . . . Cm � artifacts
where f3 is the radix, or base (f3 = 2 for all computers), then the digits from cm + I through Cm +k will still be completely artificial and will have nothing to do with the true representation of a - b. It is clear that the loss of significant digits may lead to a very considerable degra dation of the overall accuracy. The error once committed at an intermediate stage of the computation will not disappear and will rather "propagate" further and con taminate the subsequent results. Therefore, when organizing the computations, it is not advisable to compute small numbers as differences of large numbers. For example, suppose that we need to evaluate the function = 1 - cosx for x which is close to 1. Then cosx will also be close to and significant digits could be lost when computing A l (x) 1 - cosx. Of course, there is an easy fix for this difficulty. Instead of the original formula we should use = A2 (x) = 2 sin2 � . The loss o f significant digits may cause an instability even i f the original continu ous problem is well conditioned. Indeed, assume that we need to compute the value of the function = Vi - v:x=1. Conditioning of this problem can be judged by evaluating the maximum ratio of the resulting relative error in the solution over the eliciting relative error in the input data:
1,
=
sup fu;
f(x) I1LVru:I1//llfxll 1!'(x) I BIfI �
f(x)
f(x)
1 1_ 2 Vi
=�
_
1_
_
1
Ixl
v:x=1 I Vi - v:x=11
=
Ixl
2 Viv:x=1 '
x
Introduction
17
For large the previous quantity is approximately equal to !, which means that the problem is perfectly well conditioned. Yet we can expect to incur a loss of signifi cant digits when 1 . Consider, for example, 12345, and assume that we are operating in a six-digit decimal arithmetic. Then:
x»
x=
=
.JX=1 1 1 1 . 10355529865 . . . ::::: v'x = 1 1 1 . 10805551 354 . . . :::::
1 1 1 . 1 04, 1 1 1 . 1 08,
= A (x) =(xv'x) =- Jx f = t2 , = v'x t2 t2 af . hl -- _t2_ l at2 l IfI Itt - t2 1' and we conclude that this number is large when t2 is close to tl , which is precisely the case for large x. In other words, although the entire function is well conditioned, there is an intermediate stage that is ill conditioned, and it gives rise to large errors in
and consequently, l I ::::: 1 1 1 . 108 - 1 1 1 . 104 0.004. At the same time, the true answer is 0.004500214891 . . ., which implies that our approx imate computation carries an error of roughly 1 1 %. To understand where this error is corning from, consider f as a function of two arguments: f = f(tl , t2) t) where t l and = JX=1 Conditioning with respect to the second argument can be estimated as follows: -
.
the course of computation. This example illustrates why it is practically impossible to design a stable numerical procedure for an ill conditioned continuous problem. A remedy to overcome the previous hurdle is quite easy to find:
1 1 f(x) = A2 (x) = v'x+ JX=1 ::::: 1 1 1 . 1 04 + 1 1 1 . 108 = 0.00450020701 . . . .
2 I 2ax + = a XI , (a) = a 2 a X = a a2 a » 2 dX2 (a) . M = a2 --t l as a --t +oo. I da I IX2 1 Ja - 1 Nevertheless, the computation by the formula X2 (a) = a - Ja2 - will obviously digits for large a. A cure may be to compute 2 ofandsignificant then X2 = 1 /XI. Note that even for the equation x2 XIbe2ax(a)prone= ato0,+theforJalosswhich both roots XI.2 (a ) = a Ja2 + 1 are well conditioned for = all a, the computation of X2 (a) = a � Ja2 + 1 is still prone to the loss of significant digits and as such, to instability. This is a considerably more accurate answer. Yet another example is given by the same quadratic equation x2 1 0 as we considered in Section 1.2. The roots ± J - have been found to be ill conditioned for close to 1. However, for 1 both roots are clearly well conditioned. In particular, for J I we have: -
-
'
I
-
-
I
I
±
18
A Theoretical Introduction to Numerical Analysis
1 .4 . 5
Convergence
Finally, the algorithm may be either convergent or divergent. Suppose we need to compute the value of y = In ( 1 +x). Let us employ the power series: Y=
.xl
x3 x4
In ( 1 + x) = x - "2 + "3 - 4 + . . .
and set
n
(1.19)
J!'
( 1.20) y* (x) ::::; y;' = I, (- l ) k+ l _ . k k= l In doing so, we will obtain a method of approximately evaluating y = In ( 1 + x) that will depend on n as a parameter. If I x l q < 1, then lim� y;' (x) y (x) , i.e., the error committed when computing
=
n->
=
y (x) according to formula (1 .20) will be vanishing as n increases. If, however, x > 1, then lim y;' (x) = 00 , because the convergence radius for the series (1.19) is r = 1. In this case the algorithm based on formula (1 .20) diverges, and cannot be used for ll---+oo
computations. 1 .4 . 6
General Comments
Basically, the properties of continuous well-posedness and numerical stability, as well as those of ill and well conditioning, are independent. There are, however, certain relations between these concepts. •
First of all, it is clear that no numerical method can ever fix a continuous ill posedness. 2
•
For a well-posed continuous problem there may be stable and unstable dis cretizations.
•
Even for a well conditioned continuous problem one can still obtain both stable and unstable discretizations.
•
For an iII-conditioned continuous problem a discretization will typically be unstable.
Altogether, we can say that numerical methods cannot improve things in the per spective of well-posedness and conditioning.
In the book, we are going to discuss some other characteristics of numerical algo rithms as well. We will see the algorithms that admit easy parallelization, and those that are limited to sequential computations; algorithms that automatically adapt to specific characteristics of the input data, such as their smoothness, and those that only partially take it into account; algorithms that have a straightforward logical structure, as well as the more elaborate ones. 2 The opposite of well-posedness, when there is no continuous dependence of the solution on the data.
Introduction
19
Exercises
1. Propose an algorithm for evaluating y = In( I
2.
+x) that would also apply to X > 1 . Show that the intermediate stages of the algorithm A2 from page 17 are well condi
tioned, and there is no danger of losing significant digits when computing:
f(x) = A2 (X) =
1
x- I Vxx + v'x-=T
3. Consider the problem of evaluating the sequence of numbers Xo , X I , . . . , XN that satisfy the difference equations:
and the additional condition: ( 1 .2 1 ) We introduce two algorithms for computing XII . First, let
Xn = un + cvn' n = O, I , . . . ,N.
( 1 .22)
Then, in the algorithm AI we define Un, n = 0, 1 , . . . ,N, as solution of the system:
2un - lIn+l = I + nz /Nz, subject to the initial condition: Consequently, the sequence
Vn,
/1
= 0, 1 , . . . ,N - I,
( 1 .23)
Uo = 0. n = 0, 1 , . . . ,N, is defined by the equalities:
( 1 .24)
2Vn - Vn+ 1 = 0, n = O, I, . . . ,N - l, Vo = 1 ,
( 1 .25) ( 1 .26)
and the constant C of ( 1 . 22) is obtained from the condition ( 1 . 2 1 ). In so doing, the actual values of U n and VII are computed consecutively using the formulae:
Un + 1 = 2un - ( I + n2 /Nz), n = O, I , . . . , vn+l = 2"+I , n = O, I , . . . . In the algorithm A , lin , n = 0, I , . . . ,N, is still defined as solution to system ( 1 .23), z but instead of the condition ( 1 .24) an alternative condition UN = 0 is employed. The sequence Vn , n = 0, 1 , . . . ,N, is again defined as a solution to system ( 1 .25), but instead of the condition ( 1 . 26) we use VN = 1 . a) Verify that the second algorithm, lently") unstable.
A z , is stable while the first one, A
I,
is ("vio
b) Implement both algorithms on the computer and try to compare their perfor mance for N = 1 0 and for N = 100.
Part I Interpolation of Functions. Quadratures
21
One of the key concepts in mathematics is that of a function. In the simplest case, the function y = j(x), a S; x S; b, can be specified in the closed form, i.e., defined by means of a finite formula, say, y = x2 . This formula can subsequently be transformed into a computer code that will calculate the value of y = xl for every given x. In real life settings, however, the functions of interest are rarely available in the closed form. Instead, a finite array of numbers, commonly referred to as the table, would often be associated with the function y = j(x). By processing the numbers from the table in a particular prescribed way, one should be able to obtain an approximate value of the function j(x) at any point x. For instance, a table can contain several leading coefficients of a power series for j(x). In this case, processing the table would mean calculating the corresponding partial sum of the series. Let us, for example, take the function
y = ex, O S; x S; 1 , for which the power series converges for all x, and consider the table
1 1 1, 1 .1 ' 2 .1
- "
"
'
1 1 n.
o f its n + 1 leading Taylor coefficients, where n > 0 i s given. The larger the n , the more accurately can one reconstruct the function j(x) = � from this table. In so doing, the formula
x x-2 . . . ;il eX ::::: 1 + + + +1 ! 2! n!
i s used for processing the table. In most cases, however, the table that is supposed to characterize the function y = j(x) would not contain its Taylor coefficients, and would rather be obtained by sampling the values of this function at some finite set of points XO , X I , . . . , Xn E [a, b]. In practice, sampling can be rendered by either measurements or computations. This naturally gives rise to the problem of reconstructing (e.g., interpolating) the function j(x) at the "intermediate" locations x that do not necessarily coincide with any of the nodes XO , XI , ' " ,xn. The two most widely used and most efficient interpolation techniques are alge braic interpolation and trigonometric interpolation. We are going to analyze both of them. In addition, in the current Part I of the book we will also consider the problem of evaluating definite integrals of a given function when the latter, again, is specified by a finite table of its numerical values. The motivation behind considering this problem along with interpolation is that the main approaches to approximate evaluation of definite integrals, i.e., to obtaining the so-called quadrature formulae, are very closely related to the interpolation techniques. Before proceeding further, let us also mention several books for additional reading on the subject: [Hen64, IK66, CdB80, Atk89, PT96, QSSOO, Sch02, DB03].
23
Chapter 2 Algebraic Interp ol a tion
Let XO,XI , . . . ,Xn be a given set of points, and let f(XO),j(XI ), . . . ,j(xn) be values of the function f(x) at these points (assumed known). The one-to-one correspondence Xo
I
Xl
I
...
I
Xn
will be called a table of values of the function f(x) at the nodes XO,XI , . . . ,Xn. We need to realize, of course, that for actual computer implementations one may only use the numbers that can be represented as finite binary fractions (Section 1 .3.3 of the Introduction), whereas the values f(x do not necessarily have to belong to this class (e.g., vi:3). Therefore, the foregoing table may, in fact, contain rounded rather than true values of the function f(x). A polynomial P,, (x) == Pn (x,j,XO,XI , . . . ,xll) of degree no greater than n that has the form
j)
P,, (X) = Co + CIX+ . . . + cllx" and coincides with f(XO),j(XI ) , . . . ,j(xn) at the nodes XO,XI , . . . ,Xn, respectively, is called the algebraic interpolating polynomial.
2.1
2.1.1
Existence and Uniqueness of Interpolating Polyno mial The Lagrange Form of Interpolating Polynomial
THEOREM 2. 1 Let XO,XI , . . . ,XII be a given set of distinct interpolation nodes, and let the val'ues f(XO),j(XI ), . . . ,j(xn) of the function f(x) be kno'Wn at these nodes. There is one and only one algebraic polynomial Pn(x) == Pn(x,j,XO ,Xl , . . . ,XII) of degree no greater than n that 'Would coincide 'With the given f(Xk ) at the nodes Xk ,
k = O, l , . . . , n. PRO OF
We will first show that there may be no more than one interpo-
25
A Theoretical Introduction to Numerical Analysis
26
1 P� ) ( x ) RII(x) (x)
lating polynomial, and will subsequently construct it explicitly. Assume that there are two algebraic interpolating polynomials, and I Then, the difference between these two polynomials, = P,) ) p'\2) is also a polynomial of degree no greater than n that vanishes at the n + 1 points However, for any polynomial that is not identically equal to zero, the number of roots (counting their multiplicities) is equal to the degree. Therefore, = 0, i.e., which proves uniqueness. Let us now introduce the auxiliary polynomials
pP) ((xx)).,
XO,Xl , . . ,Xn. RIl (x)
p�J ) (x) = p,Fl (x),
It is clear that each is a polynomial of degree no greater than n, and that the following equalities hold:
lk(x)
j = O, l , . . . , n.
(x) Pn(X) f(Pn(xxo), jlo,(XxO,) XI f(, . x. ,xn)(x) f(xn)ln(x) f(xj)lj(x) Pn(Xj) f(xj) . Pn(x,j,XO,XI , . . ,xn).
Then, the polynomial P,,
� =
given by the equality
+
d ll
+ ...+
(2. 1 )
is precisely the interpolating polynomial that we are seeking. Indeed, its degree is no greater than n, because each term is a polynomial of degree no greater than n. Moreover, it is clear that this polynomial satisfies the equalities = for all j = 0, I , . . , n. 0 Let us emphasize that not only have we proven Theorem 2. 1 , but we have also written the interpolating polynomial explicitly using formula (2. 1 ). This formula is known as the Lagrange form of the interpolating polynomial. There are other convenient forms of the unique interpolating polynomial The Newton form is used particularly often. 2.1.2
The Newton Form o f Interpolating Polynomial. vided D ifferences
Xa,f(xf(Xba)x,,)xcf(,XXdb),, j(Xc),jXk(Xd), f(xd = f(f(xd,xk,xm) Xk, Xm (Xk Xm
Let nodes function point:
Di
f(x) f(Xk)
etc., be values of the function at the given etc. A Newton's divided difference of order zero of the at the point is defined as simply the value of the function at this
A divided difference of order one arbitrary pair of points and
k = a,b,c, d, . . . .
f(x)
of the function is defined for an do not have to be neighbors, and we allow
Algebraic Interpolation
27
Xk Z xm) through the previously introduced divided differences of order zero:
In general, a divided difference of order n: f(XO ,X I , . . . ,XII) for the function f(x) is defined through the preceding divided differences of order n - 1 as follows: (2.2) Note that all the points XO ,XJ , . . . ,Xn in formula (2.2) have to be distinct, but they do not have to be arranged in any particular way, say, from the smallest to the largest value of Xj or vice versa. Having defined the Newton divided differences according to (2.2), we can now represent the interpolating polynomial Pn (x, f,xo ,X I , . . . ,xn ) in the following Newton
form
I
:
Pn (X,j,XO ,X I , . . . ,xn ) = f(xo) + (x - xo)f(xo ,x l ) + . . . + (x - xO ) (X - X I ) . . . (x - XII - I )J(XO ,X I , . . . ,xn )'
(2.3)
Formula (2.3) itself will be proven later. In the meantime, we will rather establish several useful corollaries that it implies. COROLLARY 2. 1 The following equality holds:
PI1 (x,f,xo,x J , . . . ,XII) = Pn - I (x,f,XO ,X I , ' " ,Xn - I ) + (x - xo) (x - x I } . . . (x - xn - d f(xo,X I , . . . ,xn ) ' PRO OF
Immediately follows from formula (2.3) .
(2.4)
o
COROLLARY 2.2 The divided difference f(XO ,X I , . . . ,xn ) of order n is equal to the coefficient Cll in front of the term XIl in the interpolating polynomial
In other words, the following equality holds: (2.5) 1 A more detailed account of divided differences and their role in building the interpolating polynomials can be found, e.g., in [CdB801.
28
A Theoretical Introduction to Numerical Analysis
PROO F It is clear that the monomial X' on the right-hand side of expression (2.3) is multiplied by the coefficient j(xo,x " . . . ,XII )' 0
COROLLARY 2.3 The divided difference j(XO ,X l , . . . ,xn ) may be equal to zero if and only if the quantities j(xo), j(X l ) , . . . ,j(xn) are nodal values of some polynomial Qm (x) of degree m that is strictly less than n (m < n) .
If j(XO , X I , . . . ,XII) = 0 , then formula (2.3) implies that the degree of the interpolating polynomial PII(x,j,XO ,X I , . . . ,XII) is less than n, because ac cording to equality (2.5) the coefficient Cn in front of X' is equal to zero. As the nodal values of this interpolating polynomial are equal to j(Xj), j = 0 , 1 , . . . , n, we can simply set Qm (x) = P,, (X), Conversely, as the interpolating polyno mial of degree no greater than n is unique ( Theorem 2 . 1 ) , the polynomial Qm (x) with nodal values j(xo), J(x, ), . . . ,j(xn ) must coincide with the inter polating polynomial P', (x,j,XO ,X I , . . . ,Xn) = cnx" + clI _ I x, - 1 + . + co . As m < n, equality Qm (x) = Pn(x) implies that Cll = 0 Then, according to formula (2.5), 0 j(XO ,XI , . . . ,XII) = O. PROO F
.
. .
COROLLARY 2.4 The divided difference j(xo,X\ , . . . ,xn) remains unchanged under any arbitrary permutation of its arguments XO ,X I , · · · ,Xn . PROO F Due to its uniqueness, the interpolating polynomial Pn (x) will not be affected by the order of the interpolation nodes. Let Xo ,x'" . . . ,x" be a per mutation of XO ,X I , . . . ,XI!; then, \Ix : Pn (x,j,xO ,X l , . . . ,XII) = P', (x,j,Xo ,x; , . . . ,x� ). Consequently, along with formula (2.3) one can write
P', (X,j,XO ,X " . . . ,XII ) = j(x� ) + (x - x�)j(x� ,x; ) + . . . + (x - Xo ) (x - x; ) . . . (x - X;, - l )j(x� ,X; , . . . ,x;, ) . According to Corollary 2.2, one can therefore conclude that
(2.6) By comparing formulae (2.5) and (2.6) , one can see that j(Xo ,X; , . ,x,, ) . . .
j(XO ,XI , . . . ,xn )
=
0
COROLLARY 2.5 The following equality holds:
- Pn - 1 (xn,j,XO ,X l , ' " ,XII- I ) j(XO ,Xl " " ,XIl ) - j(xn) (XIl - Xo ) (XII - X I ) . . . (Xn - Xn - I ) . _
(2.7)
Algebraic Interpolation
29
Let us set x = Xn in equality (2.4) ; then its left-hand side becomes j(xn ) , and formula (2.7) follows. 0
PRO O F
equal to
THEOREM 2.2 The interpolating polynomial P',(x,j,XO , XI , . . . ,XII) can be represented in the Newton form, i. e . , equality {2. 3} does hold. PRO O F We will nse induction with respect to n. For n = ° (and n = 1 ) formula (2.3 ) obviously holds. Assume now that it has already been justified for n = 1 , 2 , . . . , k, and let us show that it will also hold for n = k + 1 . In other words, let us prove the following equality:
=
Pk+ 1 (x, j,XQ,XI , ' " ,Xk ,Xk+ l ) Pk (x,j, xo ,X I , . . . ,Xk ) + j(XQ ,X I , ' " ,Xk ,Xk+ 1 ) (x -xo)(x - x I ) . . . (x - Xk ) '
(2.8)
Notice that due to the assumption of the induction, formula ( 2.3 ) is valid for ::::: k. Consequently, the proofs of Corollaries 2 . 1 through 2.5 that we have carried out on the basis of formula ( 2.3 ) will also remain valid for n ::::: k. To prove equality (2.8), we will first demonstrate that the polynomial Pk + 1 (x,j,XO , XI , ' " , Xk , Xk+I ) can be represented in the form: n
(2.9) Indeed, it is clear that on the right-hand side of formula ( 2.9 ) we have a polynomial of degree no greater than k + 1 that is equal to j(Xj ) at all nodes xj , j = 0, 1 , . . . , k + 1 . Therefore, the expression on the the right-hand side of ( 2.9 ) is actually the interpolating polynomial which proves that ( 2.9 ) is a true equality. Next, by comparing formulae ( 2.8 ) and ( 2.9) we see that in order to justify (2.8) we need to establish the equality: (2. 10) Using the same argument as in the proof of Corollary 2.4, and also employing Corollary 2. 1 , we can write:
Pk (X,j,Xo ,XI , ' " ,Xk ) = Pk (X,j,X I , X2 , · · · , Xk , XO ) = Pk - I (X,j,x I , X2 , ' " ,Xk ) + j(XJ ,X2 , ' " ,xk ,xo) (x - x d (x - X2 ) . . . (X - Xk ) '
(2. 1 1 )
A Theoretical Introduction to Numerical Analysis
30
Then, by substituting x = Xk+ I into (2. 1 1 ) , we can transform the right-hand side of equality (2. 10) into:
j(Xk+ l ) - Pk (Xk+ I ,f,XO ,X l , . · . , Xk ) (Xk+ I - XO ) (Xk+ I - x I ) . . . (Xk+ I - Xk ) 1 j(Xk+ I ) - Pk- l (Xk+ l , f, xI , · · · , Xk ) j(X I ,X2 , · · · ,Xk ,XO ) Xk+ 1 -Xo xk+ I - XO (Xk+ l -X I ) . . . (Xk+ I - Xk )
(2. 1 2)
By virtue of Corollary 2.5, the minuend on the right-hand side of equality (2.12) is equal to: 1
j(X l ,X2 , . . . ,Xk ,Xk+ I ), x--k+ 1 - Xo
whereas in the subtrahend, according to Corollary 2.4, one can change the order of the arguments so that it would coincide with
j(XO ,X I , . . . ,Xk ) Xk+ I - Xo Consequently, the right-hand side of equality (2.12) is equal to
j(X I ,X2 , · · · ,xk+ d - j(XO ,X l , . . . ,Xk ) � j(X ,X , ,X ) - O I · · · k+ l · Xk+ I - Xo In other words, equality ( 2.12 ) coincides with equality ( 2.10 ) that we need to establish in order to justify formula (2.8 ) . This completes the proof.
0
THEOREM 2.3 Let Xo < X I < . . . < Xn; assume also that the function j(x) is defined on the interval Xo ::; x ::; Xn , and is n times differentiable on this interval. Then,
l
l1 n.I j(XO ,X I , . . . ,Xn ) _ - ddx"j x=� -= j ( I1 ) ( ':>;: ) ,
where � is some point from the interval PRO OF
(2. 13)
[xo,xn] .
Consider an auxiliary function (2. 14)
defined on [xo,xn ] ; it obviously has a minimum of n + 1 zeros on this interval located at the nodes XO ,X I , . . . ,Xn . Then, according to the Rolle (mean value) theorem, its first derivative vanishes at least at one point in between every two neighboring zeros of lP(x). Therefore, the function lP'(x) will have a minimum of n zeros on the interval [xo,xn] . Similarly, the function lP"(x) vanishes at least at one point in between every two neighboring zeros of lP'(x) , and will therefore have a minimum of n - 1 zeros on [xo,xn].
Algebraic Interpolation
31
By continuing this line of argument, we conclude that the n-th derivative cp( n) (x) will have at least one zero on the interval [xo,xn ] . Let us denote this zero by � , so that cp( n ) ( � ) = 0 . Next, we differentiate identity (2. 14) exactly n times and subsequently substitute x = � , which yields: 0 = cp( n) (� )
=
d" i") (� ) - -Pn(x,j,X O ,X[ , . . . , x n ) 1x=� . dx"
(2. 15)
On the other hand, according to Corollary 2.2, the divided difference
j(xo ,X[ , . . . , x n ) is equal to the leading coefficient of the interpolating polynomial Pn , i.e., Pn (x,j,XO ,X[ , . . . ,x n ) = j(xo,X[ , . . . ,xn )x" + C n - [x" - [ + . . . + co . Consequently, f; Pn(x,j,XO ,X[ , . . . ,x n ) = n!j(xo,X[ , . . . , x n ) , and therefore, equality
D
(2.15) implies (2.13).
THEOREM 2.4 The values j(xo), j(xt }, . . . , j(x n ) of the function j(x) are expressed through the divided differences j(xo), j(xo,xt } , . . . , j(XO ,XI , . . . , x n ) by the formulae:
j(Xj ) = j(xo) + (Xj - xo)j(xo,x t ) + (Xj - xO ) (Xj - x d j(XO ,X I ,Xl ) j = O, I , . . . , n , + (Xj - xO ) (Xj - xd . . . (Xj - x,, - L )j(xo,X[ , . . . , x n ) ,
i. e. , by linear combinations of the type:
j(Xj ) = ajoj(xo) + aj J /(xo, X[ ) + . . . + aj"j(xo,x l , . . . ,xn ) ,
PROOF
j = O, l, . . . ,n.
(2. 16)
The result follows immediately from formula (2.3) and equalities D
j(Xj ) = P(x,j,XO ,X [ , . . . , xn) ! x=Xj for j = O, I , . . . , n. 2.1.3
Comparison of the Lagrange and Newton Forms
To evaluate the function j(x) at a point x that is not one of the interpolation nodes, one can approximately set: j(x) � PIl (x,j,XO ,X I , . . . ,x,,). Assume that the polynomial PIl (x,j,XO ,XI , . . . ,x,,) has already been built, but in order to try and improve the accuracy we incorporate an additional interpolation node Xn+ I and the corresponding function value j(xn+ I ). Then, to construct the interpolating polynomial Pn+ [ (x,j,xo ,X I , . . . ,x,,+ L ) using the Lagrange formula (2. 1 ) one basically needs to start from the scratch. At the same time, to use the Newton formula (2.3 ), see also Corollary 2. 1 :
P,,+ I (x,j,xO ,Xj , . . . ,X,,+ I ) = PIl (x,j,xO ,X I , . . . ,xn ) + (x - xo) (x - xL) . . . (x - xll )f(xo,Xj , . . . ,xn + d
one only needs to obtain the correction
(x - xo) (x - x L ) . . . (x - xll )f(xo,X [ , . . . ,xll+ L ). Moreover, one will immediately b e able to see how large this correction is.
A Theoretical Introduction to Numerical Analysis
32 2 . 1 .4
Conditioning of the Interpolating Polynomial
Let all the interpolation nodes belong to some interval a :s: :s: b. Let also the values of the function at these nodes be given. Hereafter, we will be using a shortened notation for the interpolating polynomial = Let us now perturb the values j = 0 , I , . . . , no j by some quantities Then, the interpolating polynomial will change and become + One can clearly see from the Lagrange formula (2. 1) that = + + Therefore, the corresponding perturbation of the interpolating polynomial, i.e., its response to will be For a given fixed set of this perturbation depends only on and not on itself. As such, one can introduce the minimum number such that the following inequality would hold for any
f(xo), f(XI ), .XO,., Xf(I ,x., ). ,Xn P,1(X,f(j)x) x Pn(x) Pn(x,j,XO,XI ,f(' " x,x)n). P,,(x,j) P,1 (Xof(, jxj)of), P,,(xP,,j1(X,jof)) . Pn (x, of). of, P,,(x, of). XO,XI , . " ,Xn, of f L" of: Ln Ln(XO,XI , f(xj) f(x) x X XI . LI Xo Xo XI LI �-=-a Xl Xo Lt . f(x).
(2. 17)
The numbers . . . ,x/I ,a,b) are called the Lebesgue constants. 2 They provide a natural measure for the sensitivity of the interpolating polynomial to the perturbations 0 of the interpolated function at the nodes j. The Lebesgue constants are known to grow as n increases. Their specific behavior strongly depends on how the interpolation nodes j , j = 0 , I , . , n, are located on the interval [a, b] . If, for example, n = 1, = a, = b, then = 1. If, however, -::f. a and/or -::f. b, then 2 2 1 ro l ' i.e., if and are sufficiently close to one another, then the interpolation may appear arbitrarily sensitive to the perturbations of The reader can easily verify the foregoing statements regarding In the case of equally spaced interpolation nodes: =
.
Xj = a + j · h, j = O, I , . . . , n, h = b -n a ,
one can show that
2/1 Ln >
1
> 2,, -2__ . Vn
1_ .
(2. 1 8)
_ _
n - I /2
In other words, the sensitivity of the interpolant to any errors committed when spec ifying the values of will grow rapidly (exponentially) as n increases. Note that in practice it is impossible to specify the values of j) without any error, no matter how these values are actually obtained, i.e., whether they are measured (with inevitable experimental inaccuracies) or computed (subject to rounding errors). For a rigorous proof of inequalities (2. 18) we refer the reader to the literature on the theory of approximation, in particular, the monographs and texts cited in Section 3.2.7 of Chapter 3. However, an elementary treatment can also be given, and one can easily provide a qualitative argument of why the Lebesgue constants for equidistant nodes grow exponentially as the grid dimension n increases. From the
f(Xi)
f(x
2 Note that the Lebesgue constant Ln corresponds to interpolation on " + 1 nodes:
Xo ,
. . . , Xn .
33
Algebraic Interpolation Lagrange form of the interpolating polynomial (2. 1 ) and definition that:
(2. 1 7) it is clear (2. 19)
(later, see Section 3.2.7 of Chapter 3, we will prove an even more precise statement). Take k ::::; n l2 and x very close to one of the edges a or b, say, x a = 11 « h. Then, -
The foregoing estimate for I lk (x) l , along with the previous formula (2. 1 9), do imply the exponential growth of the Lebesgue constants on uniform (equally spaced) inter polation grids. Let now a = - 1 , b = I , and let the interpolation nodes on [a, b] be rather given by the formula:
(2j + 1 )n j = O, I , . . . , n. (2.20) 2(n + 1) , It is possible to show that placing the nodes according to (2.20) guarantees a much better estimate for the Lebesgue constants (again, see Section 3.2.7): Xj = - cos
2 Ln :::::: - In(n +
n
I ) + 1.
(2.21)
We therefore conclude that in contradistinction to the previous case (2. 1 8), the Lebesgue constants may, in fact, grow slowly rather than rapidly, as they do on the non-equally spaced nodes (2.20). As such, even the high-degree interpolating polynomials in this case will not be overly sensitive to perturbations of the input data. Interpolation nodes (2.20) are known as the Chebyshev nodes. They will be discussed in detail in Chapter 3. 2.1.5
On Poor Convergence of Interpolation with Equidis tant Nodes
One should not think that for any continuous function j(x), x E [a, b], the algebraic interpolating polynomials P,, (x,j) built on the equidistant nodes xj = a + j . h, Xo = a, Xn = b, will converge to j(x) as n increases, i.e., that the deviation of Pn (x,j) from j(x) will decrease. For example, as has been shown by Bernstein, the sequence
34
A Theoretical Introduction to Numerical Analysis
=
of interpolating polynomials obtained for the function J(x) = Ix l on equally spaced nodes diverges at every point of the interval [a , b] [- 1 , 1] except at { - 1 , 0 , I } . The next example is attributed to Runge. Consider the function J(x) = xZ;I/4 on the same interval [a , b] = [- 1, 1]; not only is this function continuous, but also has continuous derivatives of all orders. It is, however, possible to show that for the sequence of interpolating polynomials with equally spaced nodes the maximum difference max I J(x) - Pn (x,j) I will not approach zero as n increases. - 1 :::;x:::; 1 Moreover, by working on Exercise 4 below, one will be able to see that the areas of no convergence for this function are located next to the endpoints of the interval [ - 1 , 1]. For larger intervals the situation may even deteriorate and the sequence of interpolating polynomials P,, (x,j) may diverge. In other words, the quantity max I J(x) - P" (x,j) I may become arbitrarily large for large n 's (see, e.g., [IK66]). a:::;x:::; b
Altogether, these convergence difficulties can be accounted for by the fact that on the complex plane the function J(z) = z2+ /4 is not an entire function of its argument z, and has singularities at z = ± i/2. On the other hand, if, instead of the equidistant nodes, we use Chebyshev nodes (2.20) to interpolate either the Bernstein function J(x) = Ixl or the Runge function J(x) = x2 ;1/4 ' then in both cases the sequence of interpolating polynomials Pn (x,j) converges to J(x) uniformly as n increases (see Exercise 5).
\
Exercises
I . Evaluate f( I . 14) by means of linear, quadratic, and cubic interpolation using the fol lowing table of values:
1.31 1 .284
x J(x
Implement the interpolating polynomials in both the Lagrange and Newton form.
2. Let xj = j . h, j = 0, ± 1 , ±2, . . . , be equidistant nodes with spacing h. Verify that the following equality holds:
f(Xk-I , Xk, Xk+1 ) -
f(Xk+ l ) - 2f(Xk ) + f(Xk- l ) 2!h2
3. Let a = XC, a < Xl < b, X2 = b. Find the value of the Lebesgue constant L2 when Xl is the midpoint of [a, b] : XI = (a + b) /2. Show that if, conversely, Xl ---> a or XI ---> b, then the Lebesgue constant L2 = L2 (XO,XI , X2 , a, b) grows with no bound. 4. Plot the graphs of f(x) = X';1 /4 and Pn(x,j) from Section 2. 1 .5 (Runge example) on the computer and thus corroborate experimentally that there is no convergence of the interpolating polynomial on equally spaced nodes when n increases. 5. Use Chebyshev nodes (2.20) to interpolate f(x) Ixi and f(x) = X';1 /4 on the interval [-1, 1]' plot the graphs of each f(x) and the corresponding Pn (x, j) for n = 10,20,40, and 80, evaluate numerically the error max [f(x) - P (x, j) I, and show that it de =
creases as n increases.
-1<;-,<; 1
n
35
Algebraic Interpolation
2.2
Classical P iecewise Polynomial Interpolation
High sensitivity of algebraic interpolating polynomials to the errors in the tabu lated values of f(x), as well as the "iffy" convergence of the sequence Pn (x,j) on uniform grids, prompt the use of piecewise polynomial interpolation. 2.2.1
Definition of P iecewise Polynomial Interpolation
Let the function f(x), x E [a, b] , be defined by the table {f(XO ),j(X I ), . . . ,j(xn) } of its numerical values at the nodes { a = Xo < X I < X2 < . . . < Xn b} . To reconstruct this function in between the nodesxO ,x l , ' " ,Xn, one can use an auxiliary function that would coincide with a polynomial of a given low degree (say, the first, the second, the third, etc.) between every two neighboring nodes of the interpolation grid. This approach is known as piecewise polynomial interpolation; in particular, it may be =
piecewise linear, piecewise quadratic, piecewise cubic, etc.
In the case of piecewise linear interpolation on the interval Xk :::; X :::; Xk+ I , one uses the linear interpolating polynomial PI (x,j,xk ,Xk+ d to approximate the func tion f(x). In the case of piecewise quadratic interpolation on the interval Xk :::; x :::; Xk+ I , one can use either of the two polynomials: P2 (X,j,Xk ,Xk+ ] ,Xk+2 ) or
P2 (X,j,Xk - 1 ,Xk ,Xk+l ) .
Piecewise polynomial interpolation of an arbitrary degree s is obtained similarly. There is always some flexibility in constructing the interpolant, and to approximate the function f(x) on the interval Xk :::; x :::; Xk+ l one can basically use any of the poly nomials Ps(X,j,Xk_ j ,Xk - H I , . . . ,xk- Hs), where j is one of the integers 0, 1 , . . . , s - 1. It is, however, desirable that the smaller interval [xk ,xk+ d b e located maximally close to the middle of the larger interval [Xk- j ,Xk - j +s] (see Section 2. 1.4). For equidistant nodes, the latter requirement translates into choosing j maximally close to s/2. In general, once the strategy for selecting j has been adopted, one can reconstruct f(x) on [a, b] in the form of a piecewise polynomial of degree s. It will be composed of the individual interpolating polynomials that correspond to different intervals [Xk ,Xk+ d , k 0, 1 , . . . , n - 1. For simplicity, we will hereafter denote the piecewise polynomial as follows: =
2.2.2
Formula for the Interpolation Error
Let us estimate the error
(2.22) that arises when the function f(x) is approximately replaced by the polynomial Ps(X,jkj ). To do so, we will need to exploit the following general theorem:
36
A Theoretical Introduction to Numerical Analysis
THEOREM 2.5 Let the function f = f(t) be defined on a � t � {3 , with a continuous derivative of order s + 1 on this interval. Let to, tl , . . . , ts be an arbitrary set of distinct points that all belong to [ a , {31 , and let f(to ) ,j(tJ ), . . . ,j(ts ) be the values of the function f(t) at these points. Finally, let Ps (t) == Ps (t ,j,to , t l , ' " , ts ) be the algebraic interpolating polynomial of degree no greater than s built for these given points and function values. Then, the interpolation error R.,. (t) = f(t) - P.� (t) can be represented on [ a, {3 l as follows:
fS+ I ) Rs (t) = (s 1(s) 1. ) (t - to ) (t - t d · · · (t - ts ), + where S = S (t) is some point from the interval ( a , {3 )
(2.23)
.
PRO OF We first notice that formula (2.23) does hold for all nodes tj , j = O, l , . . . , s, themselves, because on one hand \/tj : f(tj ) - PAtj ) = 0, and on the other hand, Rs(tj) = 0, where R.,. (t) is defined by formula (2.23). Let us now take an arbitrary 1 E [ a, {3l that does not coincide with any of to,t l , ' " , ts . To prove formula (2.23) for t = 1, we introduce an auxiliary function: cp(t) = f(t) - Ps (t) - k(t - to ) (t - tJ ) . . . (t ( ) (2.24) and choose the parameter k so that cp(l) = 0, which obviously implies f(l) - Ps (l) (2.25) k= (t - to)(t - t l ) . . . (t - (I) The numerator in formula (2.25) coincides with the value of the error Rs (l), -
I
_
therefore, this formula yields:
(2.26) The auxiliary function cp of (2.24) clearly has a minimum of s + 2 zeros on the interval [ a , {31 10cated at the points l, to, t l , . . . , ts' Then, its first derivative cp'(t) will have a minimum of s + 1 zeros on the ( open) interval ( a, {3) , because according to the Rolle ( mean value ) theorem, the derivative cp' (t) has to van ish at least once in between every two neighboring points where cp(t) itself vanishes. Similarly, cp//(t) will have at least s zeros on ( a, {3 ) , cp (3) (t) will have at least s - 1 zeros, etc., so that finally the derivative cp (s+ l ) (t) will have to have a minimum of one zero on the interval ( a, {3 ) . Let us denote this zero by S E ( a , {3) , so that cp (.,+ I ) (S ) 0. Next, we note that
=
ds+ 1 ,+ 1 = (s I ) ! _-t + dts+ 1 and that (t - to) (t - tJ ) . . . (t - ts ) t�+ 1 + Qs (t), where Qs (t) is a polynomial of degree no greater than s. We also note that ds+ 1 ds+ 1 Q,. (t) 0. == PAt) dt s+ 1 dt s+ 1 . == =
37
Algebraic Interpolation Using the previous two expressions, we differentiate the function by formula (2.24) s + times and obtain:
1
=
cp (s+ l l (t) = j!S + l l (t) - k( s + I ) !.
Substituting t � into the last equality, and recalling that arrive at the following expression for k:
k=
cp(t) defined
cp (s+ I ) (� ) = 0, we
j( s+ l l (t) (s + I) ! . =---_--'--C.
Finally, by substituting k into equality (2.26) we obtain a formula for Rs (t) that would actually coincide with formula (2.23) because t E [a,J3] has been chosen arbitrarily. 0 THEOREM 2. 6 Under the assumptions of the previous theorem, the following estimate holds:
max [Rs(t)[ a9::;�
max [j!s+ l l (t) [ (J3 - ay + l . ::; _ (s + I ) ! a9::;� 1_
(2.27)
PRO OF We first note that \It E [a , 13] the absolute value of each expression t - to, t - tl , ... , t - (, will not exceed 13 - a. Then, we use formula (2.23) :
_1_) ::; _1 a 9::;/3
!s+ I ) t [Rs(t) [ = (s + 1 ! [j (� ) (t - O )(t - t l ) . . . (t - ts ) [ _ max [j!s+l l (t) [ (J3 ay+ l . (s + I ) ! _
(2.28)
As t E [a,J3] on the left-hand side of formula ( 2.28) is arbitrary, the required e�timate (2.27) follows. 0 Let us emphasize that we have proven inequality (2.27) for an arbitrary distribution of the (distinct) interpolation nodes to, t l , . . . , ts on the interval [a, 13]. For a given fixed distribution of nodes, estimate (2.27) can often be improved. For example, consider a piecewise linear interpolation and assume that the nodes to and t l coincide with the endpoints a and 13, respectively, of the interval a ::; t ::; 13. Then,
1 ����
1
/ [R I (t) [ = ( ! (t - a)(t - J3 ) max [jl/(t) [ max [ (t - a)(t - 13 ) [ = � max [jl/ (t) [ (J3 - a) 2 , ::; �2 a9::;� 8 a9 ::;/3 a 9 ::;/3 which yields max [R I (t) [ a 9::;/3
1/
1 max [ j (t) [ (J3 - a) 2 , ::; -8 a9::;�
(2.29)
38
A Theoretical Introduction to Numerical Analysis
1
whereas estimate (2.27) for s = transforms into
We will now use Theorems 2.5 and 2.6 to estimate the error (2.22) of piecewise polynomial interpolation of the function on the interval First, let
f(x)
Xk ::::; x ::::; Xk+ I .
Then, it is clear that
and according to (2.27) we obtain
IRs (X,Jkj)I ( +1 1 ) , max lis+ l ) (X) I (Xk-j+s - Xk_jy+ l . (2.30) If the quantity l is+ I )(x)1 undergoes strong variations on the interval [a, b], then, in order for the estimate (2.30) to guarantee some prescribed accuracy, it will be advantageous to have the grid size (distance between the neighboring nodes) and the value of Xk-j+s -Xk -j smaller in those parts of [a, b] where If (s+ I)(x)1 is larger. In the case of equidistant nodes XO,XI, ... ,XIl, estimate (2.30) implies ss+I max If(s+ I) (x)W+ 1 , (2.3 1 ) max IRs (x,JdI (s + I ) ! where h (b - a)jn =Xk + l -Xk is the size of the interpolation grid. Inequality (2.31) can be recast as ::::;
Xk :O;x:O;xk+ 1
max
=
S
-
Xk :O;x:O;xk+ 1
.
xk_ j :O;x:O;xk -j+s
::::;
--
xk- j :O;X:O; Xk_ j+s
(2.32) where the key consideration is that the constant on the right-hand side of (2.32) does not depend on the grid size h. To conclude this section, let us specifically mention the case of piecewise linear interpolation: s = a and f3 Then, according to estimate (2.29), we have:
1 , = Xb
2.2.3
= Xk+ I.
Approximation o f Derivatives for a Grid Function
THEOREM 2. 7 Let the function be defined on the interval [a, f3J, and let it have a continuous derivative of order s + on this interval. Let
f = f(x)
1
Xk-j,Xk-j+ I, · ·· , Xk-j+s
Algebraic Interpolation
39
= Xk-j Xk-j+ 1
be a set of interpolation nodes, such that a Then, to approximately evaluate the derivatives
<
< ... <
xk-j+s = /3 .
dqj(x) , q = 1,2, ... d� of the function j(x) on the interval Xk ::;; x ::;; xk+ l , one can employ the inter polating polynomial Ps(X, jkj) and set (2.34) , s,
In so doing, the approximation error will satisfy the estimate: max
dqj(X) dq -- - -Ps(X,jkj) d� d� I I I 1 (2.35) ::;; ( -q + I ) ! Xk-j:O;X:O;Xk-J+s IjlH ) (X)!(Xk-j+s -xk-y-q+ l . Consider an auxiliary function cp(x) � j(x) -P.� (X,jkj); it obvi ously vanishes at all + 1 interpolation nodes Xk-j,Xk-j+1 ,··· ,Xk-j+s . There fore, its first derivative cp' (x) will have a minimum of zeros on the interval Xk-j ::;; x::;; Xk-j+S, because according to the Rolle (mean value) theorem, there is a zero of the function cp'(x) in between any two neighboring zeros of cp(x). Similarly, the function d::Jx) will have at least - q + 1 zeros on the interval Xk-j ::;; X ::;; Xk-j+s . This implies that the derivative dYJx) and the polynomial 1; P;. (X,jkj) of degree no greater than - q coincide at - q + 1 distinct points. In other words, the polynomial pSq) (X,jkj) can be interpreted as an interpolat ing polynomial of degree no greater than -q for the function jlq) (x) on the interval Xk-j ::;; X ::;; Xk-j+s , built on some set of s -q + 1 interpolation nodes. Moreover, the function jlq) (x) has a continuous derivative of order s - q + 1 on [a, /31: s+1 j(x) . + 1 (q) (x) -d ds--q.j dx' q+ I - dx'+ I Consequently, one can use Theorem 2.6 and, by setting a =Xk- j, /3 =Xk-j+s, Xk :O;X:O;Xk+ I
max
s
PROOF
s
s
s
s
s
s
obtain the following estimate [ef. formula (2.27)]:
Ijlq) (x) - pSq) (X,jkj) I 1 _ (s -q+ 1 ) ! Xk-j:O;X:O;Xk-j+s ! j(H l )(x)!(Xk-j+s -Xk-j ) s- q+1 . As a = Xk-j ::;; Xk Xk+ 1 ::;; Xk-j+s = /3, it immediately yields (2.35). Xk- jf.J�;k-j+S <
<
max
0
40 2 . 2 .4
A Theoretical Introduction to Numerical Analysis Estimate of the Unavoidable Error and the Choice of Degree for Piecewise Polynomial Interpolation
Let the function f = f(x) be defined on the interval [O , nj , and let its values be known at the nodes of the uniform grid: Xk = kn / n == kh, k = 0, 1 , . . . , n. Using only the tabulated values of the function f(xo), f(XI ) , . . . , f(xn ), one cannot, even in principle, obtain an exact reconstruction of f(x) in between the nodes, because different functions may have identical tables, i.e., may coincide at the nodes Xk . k = 0 , 1 , . . . , n, and at the same time be different elsewhere. If, for example, in addition to the table of values nothing is known about the function f(x) except that it is simply continuous, then one cannot guarantee any accuracy at all when reconstructing f(x) at x =I=- Xk . k = 0, 1 , . . . , n. Assume now that f(x) has a bounded derivative of the maximum order s + I : max If(s+ 1 l (x) 1 x
::; Ms = const.
(2.36)
It is easy to find two different functions from the class characterized by Ms = I :
=
sinnx h(x) - ----:n" + 1 - ' that would deviate from one another by the value of order hs+ l : sin nx
fl (x) = nS+ 1
-
and
(2,37) and for which the tables would nonetheless fully coincide (both will be trivial): k = O, I , . . . ,n. We therefore conclude that given the tabulated values o f the function f(x), and only estimate (2.36) in addition to that, one cannot, even in theory, reconstruct the function f(x) on the interval ° ::; x ::; n with the accuracy better than t7(h"'+ 1 ). In other words, the error t7(hs+ l ) is unavoidable when reconstructing the function f(x), ° ::; x ::; n, using its table of values on a uniform grid with size h. It is also clear that O .<:;x.<:; n:
max
I dqdf�l (x)
_
1
dqh(x) = 2 -1- _2_ hS- q+ 1 ' ns-q+ 1 = ns-q+ 1 d�
(2.38)
which means that the unavoidable error when reconstructing the derivative d:fXl is at least t7( hs-q+ 1 ). By comparing equalities (2.37) and (2.38) with estimates of the error obtained in Sections 2.2.2 and 2.2.3 for the piecewise polynomial interpolation of the function f(x) and its derivatives, we conclude that the interpolation error and the unavoidable error have the same asymptotic order (of smallness) with respect to the grid size h. If, under the condition (2.36), one still chooses to use interpolating polynomials of
Algebraic Interpolation
41
degree r < s, then the interpolation error (for the function itself) will be 6(hr+ I ). In other words, there will be an additional loss of the order of accuracy, on top of the uncertainty-based unavoidable error 6(hs+ 1 ) that is due to the specification of through its discrete table of values. On the other hand, the use of interpolation of a higher degree r > s cannot increase the order of accuracy beyond the threshold set by the unavoidable error 6(hs+ I ) , and therefore cannot speed u p the convergence as h ---t 0. As such, the degree s of piecewise polynomial interpolation is optimal for the functions that satisfy
f(x)
(2.36).
REMARK 2.1 The considerations of the current section pertain primarily to the asymptotic behavior of the error as h ---t 0. For a given fixed h > 0 , interpolation o f some degree r < s may, i n fact, appear more accurate than the interpolation of degree s. Besides, in practice the tabulated values k = 0 , 1 , . . . , n, may only b e specified approximately, rather than exactly, with a finite fixed number of decimal ( or binary ) digits. In this case, the loss of interpolation accuracy due to rounding is going to increase as s increases, because of the growth of the Lebesgue constants ( defined by formula (2. 18) of Section 2 . 1 .4 ) . Therefore, the piecewise polynomial interpolation of high degree ( higher than the third ) is not used routinely. 0
f(Xk),
Error estimate (2.32 ) does, in fact, imply uniform conver ( a piecewise polynomial) to the interpolated function with the rate 6(hs+ I ) as the grid is refined, i.e., as h ---t 0. Estimate (2.33) , in particular, indicates that piecewise linear interpolation converges uniformly with the rate 6(h2 ) . Likewise, estimate (2.35) in the case of a uniform grid with size h will imply uniform convergence of the q-th derivative of the interpolant f�q) to the q-th derivative of the interpo lated function fq) with the rate 6(hs-q+ ! ) as h ---t 0 . 0 REMARK 2 . 2
gence of the interpolant
f(x)
f�(X,jkj )
(X,jkj )
(x)
REMARK 2 . 3 The notion of unavoidable error as presented in this sec tion ( see also Section 1.3 ) illustrates the concept of Kolmogorov diameters for compact sets of functions (see Section 1 2.2.5 for more detail ) . Let W be a linear normed space, and let U c W. Introduce also an N-dimensional linear N N manifold W (N) C W, for example, W ( N) = span { w i ) , w� ) , . . . , w}:) } , where the functions w}[") E W, n = . . . , N, are given. The N-dimensional Kolmogorov diameter of the set U with respect to the space W is defined as:
1,2,
J(N (U, W) = inf W(N) C W
uEU wEW(N)
sup
inf
Ilw - ul lw .
(2.39)
This quantity tells us how accurately we can approximate an arbitrary u from a given set U C W by selecting the optimal approximating subspace W (N) whose dimension N is fixed. The Kolmogorov diameter and related concepts play a fundamental role in the modern theory of approximation; in particular, for the analysis of the so-called best approximations, for the analysis of saturation of numerical methods by smoothness ( Section 2.2.5 ) , as well as in the theory of £-entropy and related theory of transmission and processing of information. 0
A Theoretical Introduction to Numerical Analysis
42 2 .2 .5
Saturation of Piecewise Polynomial Interpolation
As we have seen previously (Sections 1.3.1 and 2.2.4), reconstruction of continu ous functions from discrete data is normally accompanied by the unavoidable error. This error is caused by the loss of information which inevitably occurs in the course of discretization. The unavoidable error is typically determined by the regularity of the continuous function and by the discretization parameters. Hence, it is not related to any particular reconstruction technique and rather presents a common intrinsic accuracy limit for all such techniques, in particular, for algebraic interpolation. Therefore, an important question regarding every specific reconstruction method is whether or not it can reach the aforementioned intrinsic accuracy limit. If the accuracy of a given method is limited by its own design and does not, generally speaking, reach the level of the unavoidable error determined by the smoothness of the approximated function, then the method is said to be saturated by smoothness. Otherwise, if the accuracy of the method automatically adjusts to the smoothness of the approximated function, then the method does not get saturated. Let f(x) be defined on the interval [a, bl, and let its table of values f(xd be known for the equally spaced nodes Xk = a + kh, k = 0, 1 , . . . , n, h = (b a) / n. In Sec tion 2.2.2, we saw that the error of piecewise polynomial interpolation of degree s is of order d(hs+ 1 ) , provided that the polynomial P.�(X, jkj) is used to approximate f(x) on the interval Xk :::::: X :::::: Xk+l , and that the derivative fs+! ) (x) exists and is bounded. Assume now that the only thing we know about f(x) besides the table of values is that it has a bounded derivative of the maximum order q + 1 . If q < s, then the unavoidable error of reconstructing f(x) from its tabulated values is d(hq+ I ). This is not as good as the d(hs+ I ) error that the method can potentially achieve, and the reason for deterioration is obviously the lack of smoothness. On the other hand, if q > s, then the accuracy of interpolation still remains of order d(hs+! ) and does not reach the intrinsic limit d(hq+! ) . In other words, the order of interpolation error does not react in any way to the additional smoothness of the function f(x), beyond the required s + 1 derivatives. This is a manifestation of susceptibility of the algebraic piecewise polynomial interpolation to the saturation by smoothness. In Chapter 3, we discuss an alternative interpolation strategy based on the use of trigonometric polynomials. That type of interpolation appears to be not susceptible to the saturation by smoothness. -
Exercises
1. What size of the grid h guarantees that the error of piecewise linear interpolation for the function f(x) = sinx will never exceed 1O -6 ?
2. What size of the grid h guarantees that the error of piecewise quadratic interpolation for the function f(x) = sin x will never exceed 1O-6 ? 3. The values of f(x) can be measured at any given point x with the accuracy 1 8fl ::; 10 -4 . What is the optimal grid size for tabulating f(x), if the function is to be subsequently reconstructed by means of a piecewise linear interpolation?
Algebraic Interpolation
43
Hint. Choosing h excessively small can make the interpolation error smaller than the perturbation of the interpolating polynomial due to the perturbations in the data, see Section 2. 1.4.
4.* The same question as in problem 3, but for piecewise quadratic interpolation. 5. Consider two approximation formulae for the first derivative l' (x):
and let 1 J"(x) I
1' (x)
�
1' (x)
�
f(x + h) - f(x) , h f(x + h ) - f(x - h ) , 2h
(2.40) (2.41 )
::::: 1 and I f'" (x) I ::::: 1 .
a ) Find h such that the error of either formula will not exceed
10 -3.
b)* Assume that the function f itself is only known with the error o. What is the best accuracy that one can achieve using formulae (2.40) and (2.41), and how one should properly choose h ?
c)* Show that the asymptotic order of the error with respect to 0, obtained by formula (2.41) with the optimal h, cannot be improved.
2.3
P iecewise
Smooth ( S plines)
Polynomial
Interpolation
A classical piecewise polynomial interpolation of any degree s, e.g., piecewise linear, piecewise quadratic, etc., see Section 2.2, yields the interpolant that, gener ally speaking, is not differentiable even once at the interpolation nodes. There are, however, two alternative types of piecewise polynomial interpolants - local and nonlocal splines - that do have a given number of continuous derivatives every where, including the interpolation nodes. 2.3.1
Local Interpolation of Smoothness
s and Its Properties
Assume that the interpolation nodes Xk and the function values f (xd are given. Let us then specify a positive integer number s and also fix another positive integer j: 0 :<:::; j :<:::; s We will associate the interpolating polyno mial Ps (X, jkj ) == P.�(X, j, Xk- j ,Xk-J+ I , . . . ,Xk- j+s ) with every point Xk ; this poly nomial is built on the nodes Xk -j, Xk-j+ I , . . . ,Xk-J+s using the function values f (Xk - j ) , f (Xk -j + I ) ' . . , f (Xk-j +s ). A piecewise polynomial local spline q> (x, s ) that has continuous derivatives up to the order s is defined individually for each segment [Xk , Xk+ I J by means of the following equalities:
1.
·
(2.42)
44
A Theoretical Introduction to Numerical Analysis
where each Q2s+ I (x, k) is a polynomial of degree no greater than 2s + 1 that satisfies the relations: dmQ2s+1 (x, k) m = O,I,2, . . . , s, (2.43) dxm
I
X=Xk
m = O, I,2, . . . , s.
(2.44)
THEOREM 2.8 The polynomial (x, k) of degree no greater than 2s + 1 defined by means of equalities (2.43) and (2.44) exists and is unique.
Q2s+ I
Q2s+ 1
C2s+ I ,kX2s+ l. CO,k, C I ,k? ... , C2s+ I ,k. Q2s+ 1 Xk+I . Q2s+ 1 CO,kQ2s+1 = CI,k . =
= CO,k C I ,kX
Let Then, relations (x,k) +...+ + ( 2.43 ) and ( 2.44 ) together can be considered a system of 2s + 2 linear algebraic equations with respect to the 2s + 2 unknown coefficients This system becomes homogeneous if we replace all the right-hand sides of equalities ( 2.43 ) and ( 2.44 ) by zeros. Having this system as homogeneous means that the corresponding polynomial (x,k) would have a root of multiplicity s + 1 at x and another root of multiplicity s + I at x = In other words, Q2s+ 1 (X, k) would have a total of 2s + 2 roots counting their multiplicities. This is only possible if (x, k) is a (x, k) == 0, because polynomial of degree no greater than 2s + 1 . Consequently, = .. 0, and we conclude that the homogeneous counterpart of the linear algebraic system (2.43 ) , (2.44) may only have a trivial solution. As such, the original inhomogeneous system (2.43), (2.44) itself will have a unique solution for any given choice of its right-hand sides. 0 PRO OF
= Xk
C2s+ I ,k =
THEOREM 2. 9 Let f(x) be a polynomial of degree no greater than
s.
Then, the interpolant
PRO OF We will prove the identity
k k+ Q2s+ 1 Ps(X,jk Ps (x, jk+ l .j )
THEOREM 2. 1 0 The piecewise polynomial interpolating function
= O , ±l, ...
Xk,
45
Algebraic Interpolation
PRO O F According to equalities ( 2.43) and (2.44) , at any given node Xk the two functions: Q2s+ I (X, k - 1 ) and Q2s+ I (X, k) , have derivatives of orders3 m = 0, 1 , 2, . . . , s that coincide with the corresponding derivatives of one and the same interpolating polynomial Ps (X, ]kj ). By virtue of equalities (2.42), this proves the theorem. 0
Let us now recast the polynomial Q2s+ I (x, k) as
(2.45) where R2s+ 1 (x, k) denotes a correction to the classical interpolating polynomial P' (X, ]kj ). Then, the following theorem holds. THEOREM 2. 1 1 The correction R2s+ I (x, k) defined b y (2.4 5) can be written i n the form:
R2s+I (X, k) = (Xk+ 1 - xky + I !(Xk - j,Xk - j+ I " " ,Xk-Hs+ I ) Q 2s+ 1
(
)
X - Xk ,k , Xk+ I - Xk
(2.46)
where !(Xk - j,Xk- H I , . . . ,Xk-Hs+ I ) is a divided difference of order s + 1 , and
Q2s+ I (X , k) -
S {[S (
_
L
r=O
(
Xk-j+s+ I - Xk-j xk+ 1 - Xk
x j+ - Xk II X - ;_ � k+1 xk 1=1
)
x
)](r) }
X - Xk X = --- , Xk+1 - Xk
Ir(X) ,
X= I
(2.47)
(2.48) In fonnula (2.47), expression [ . . . J r� 1 denotes a derivative of order r with respect to X evaluated for X = 1 . REMARK 2.4 Representation (2.45 ) o f the local piecewise polynomial interpolant with s continuous derivatives can be thought of as Newton's form of the interpolating polynomial />.\,+ 1 (x, !kj ) :
Ps+1 (X, ]kJ = Ps (X, ]kj ) + !(Xk- j,Xk-H I , · . . , xk-Hs+J ) l/>s+I (x, k) , I/>s+ 1 (x, k) = (x - Xk- j ) (X - xk- H d . . (x - xk-j+s ) , ·
3A
derivative of order zero shall naturally be interpreted as the function itself.
46
A Theoretical Introduction to Numerical Analysis
)
in which the polynomial (/)s+ 1 (x, k) has been replaced by another polynomial:
(Xk+ 1 -Xk ) s+ 1 Q2s+ 1
(
X - Xk ,k . Xk+ 1 - Xk
o
To illustrate Theorem 2. 1 1 , we first note that according to formulae (2.45)-(2.48), in the case s = 0, j = 0, the local spline cp (s , k) renders a piecewise linear interpo lation .4 As far as the most interesting case from the standpoint of applications, it is, perhaps, s = 2, j = 1 , and xk+ 1 - Xk = h = const for all k. Then, we have:
(2.49) The proof of Theorem 2. 1 1 is given in Section 2.3.3; it concludes our discussion of smooth piecewise polynomial interpolation. THEOREM 2. 12 Let the function ] (x) be defined everywhere on the interval [Xk- j , Xk-j+s+ d and let its derivative fs+ (X) be bounded on this interval. Then, the approximate equalities
1)
](m) (x)
�
dm cp (x s ) dxm
,
m = O, l , . . . , s,
,
(2.50)
hold for all x E [Xk, Xk+l ] , and the following error estimates are g'uaranteed: max
Xk :SX:SXk+ J
PROOF
I
fm) (x)
_
dm cp (x s ) dxm
,
I
:::;
const · (Xk_j+s - Xk_ j y+ l - m
Xk- j :SX:SXk_ j+s+ J
max
[](s + l ) (x) [ .
(2.51)
First, using formulae ( 2.42) and ( 2.45 ) , we can obtain:
(2.52) Next, we recall estimate ( 2.35 ) established in Theorem 2.7:
const 1 . (Xk-j+s - Xk - j ) s+l- m
max
Xk_ j:SX:SXk_J+S
[](s+ I ) (x) [ .
(2.5 3)
4 Note that we have originally required that s > 0 and 0 "" j "" s - 1. However, a fonnal substitution of s = 0, j = 0 into (2.45)-(2.48) does yield a piecewise linear interpolant.
47
Algebraic Interpolation
m I m dmm where account th at ddA-'"- Th en, tak·mg mto · xk + l -xk ) dX , using formulae ( 2.46) and (2.47) we obtain:
dmR2H I (x, k) I dxm I
I
(
X-
�, Xk + l -Xk
and
(xk+l - xk ) s+ l -m [ j (Xk -j,Xk- HI , · · · ,Xk -Hs+I ) [ x
=
Xk -HS+ 1 - Xk - j xk+ 1 - Xk
I - { [IT (X ±
r
0
-
-_ I
_
1-
xk-Hi - Xk xk+1 - Xk
) ] (r) } dmdxlr(mX)
(2.54)
X=I
Finally, according to Theorem 2.3 we can write:
(s + 1 ) ! j(Xk -j,Xk-H I , . . . ,xk -j+s+d � E [Xk-j,Xk-Hs+I ] .
dmR2s+ 1 (x, k) - const2 · (Xk+ 1 - Xk)s+l -m I dXm I
=
I
H ) (� ),
ji
(2.5 5)
Then, substitution of (2.55) into (2.54) yields the estimate: <
Xk-j'SX'SXk_j+s+ 1
max
where const2 depends only on the quantities
Xk- j+H 1 - Xk- j Xk+ 1 - Xk
Xk-Hi - Xk ---"---, Xk+1 - Xk
and
dHI j(x) I d +I I '
i = 1 , 2, . .
.
x'
(2.56)
,s
(2.57)
that appear in formula (2.54). Clearly, const2 will remain bounded when the grid is refined as long as all ratios (2.57) remain bounded. Then, the required estimate (2.51) follows from (2.52), (2.53 ) , and (2.56) , max because max [jiS+I ) (X) [ s: [jis+l ) (x) [ and also (Xk+ 1 - Xk ) s: Xk-j 'SX'SXk_ j+s
(Xk-Hs - Xk - j ) .
Xk_j 'SX'SXk_ )+s+ l
0
REMARK 2 . 5 On an equally spaced grid with size h , error estimate (2.51) means uniform convergence of the local spline q>(x, s) and its derivatives of orders m = 1 , 2, . , s, to the interpolated function j(x) and its respective derivatives, with the rate O' (hs+l - m ) , as the grid is refined, i.e., as h ---7 0. 0
..
Next, we will discuss optimality and saturation of local splines (2.42).
1
,1
REMARK 2.6 Assuming that s + is the maximum order of a bounded derivative of the function j(x) , one cannot reconstruct J im) (x) , m = 0 , 2 , . . . ,s, from the tabulated values j(xd , k 0, ± 1 , . . . , with an accuracy better than 6'(hs+ l - m ) . This is an intrinsic constraint, no matter what specific reconstruc tion methodology is used. It is due to the lack of information contained in the values j(Xk ) (see Section 2.2.4) . At the same time, the best allowable accuracy 6'(hH I - m ) is achieved by estimate (2.5 1 ) . =
A Theoretical Introduction to Numerical Analysis
48
j(x)
However, if the function has a bounded derivative of the maximum order q + I , where q > s, then the accuracy of obtaining f m ) (x) by means of local spline (2.42) still remains tJ(hs+ l -m ) and does not reach 8(hQ+ I -m ) , which is the limit given by the unavoidable error. This is a manifestation of susceptibility of local splines to the saturation by smoothness. 0 REMARK 2 . 7 To evaluate cp(x,s) for every given x, one needs no more than s + 2 interpolation nodes. This characteristic reflects on the local nature of formula (2.42) , and it cannot be improved in the following sense. If we were to require that no more than s + 1 interpolation nodes be used, then in order to achieve convergence of cp(m) (x) to (x) with the rate of tJ (hs+ 1 -m ) we would have had to relinquish the condition of continuity even for the first derivative of cp(x,s), and rather resort back to the classical piecewise polynomial interpolation of degree no greater than s. 0
j(m)
We should also note that previously in our analysis we used the interpolation grid with no endpoints, i.e., the grid that formally contained infinitely many nodes k = 0, ± 1 , . . .. If we were to use a finite interpolation grid instead, with a total of n + I nodes: located on some interval [a, b] so that a = < < . . . < = b, then we would have had to modify formula (2.42) near the endpoints of the interval [a, b] and define the new interpolant cp(x) == cp(x, s, a,b) by the following equalities:
XO,XI , . . ,Xn, Xo Xl Xn {ps(X,jj ) , Xo k kXj, Ps j-s,j) , X j-s Xn· Ps(X,jkj)
cp(x,s,a,b) =
Q2s+1 (x,k), (x, jn +
if if if
"5.
Xh
x "5. x E [x x + d and k = j, j + 1 , . . . ,n + j - s - l , + "5. x "5. n
,
(2.58) Recall that the notion of was introduced in the end of Section 2.2. 1 . The interpolant cp(x,s,a, b) defined on the interval [a, b] has continuous derivatives up to the order s everywhere on this interval. For this new interpolant, the results similar to those of Theorems 2.8-2. 1 2 can be established. REMARK 2 . 8 Local smooth piecewise polynomial interpolation was introduced by Ryaben'kii in 1952 for the functions specified on multi dimensional rectangular grids with constant size; the formulae presented in this section were obtained in 1974. The multi-dimensional results and the bibliography can be found in [ Rya02, Section 1 . 1 ] and in [Rya75] . 0
2 .3 . 2
Nonlocal Smooth P iecewise Polynomial Interpolation
Local splines with s continuous derivatives that we have constructed in Sec tion 2.3. 1 are realized using polynomials of degree 2s + For example, a twice continuously differentiable local spline is built as a piecewise polynomial function
I.
49
Algebraic Interpolation
of degree five. A natural question then arises of whether or not the same level of regularity (Le., smoothness) can be achieved using polynomials of a lower degree. The answer to this question is, generally speaking, affirmative. Splines s) of minimal degree m that would guarantee the desired extent of smoothness s were constructed by Schoenberg for all positive integers s. For example, if s = 2, then it turns out that the minimal necessary degree is m = i.e., the spline is realized by a piecewise cubic polynomial. In the literature, such splines are referred to as Schoenberg's cubic splines or simply cubic splines. The key distinction between Schoenberg's splines and local splines of Section 2.3. 1 is that the lower degree of the constituent polynomials for Schoenberg's splines is basically obtained at the expense of losing their local nature. In other words, Schoenberg's splines become nonlocal, so that the coefficients of each of their polynomial pieces depend on the function values on the entire interpolation grid, rather than only at a few neighboring nodes. Assume that a total of n + 1 interpolation nodes: are given on the interval b] so that a = < < . . . < = b, and assume also that the function = 0, 1 , . . . ,n, at these nodes are known. The Schoenberg cubic spline values 2) = is formally defined on b] as a function composed of individual third-degree polynomials specified on each partition interval � � i.e., =for x E I l , = 0 , I , . . . , n - I . The spline must coincide with the original function at the interpolation nodes: for all = = 0 , 1 , . . . , n, and its first and second derivatives must be continuous everywhere. As we have = = is a piecewise polynomial function, i.e., for E
ljI(x,
3,
[f(aX,k), k Xo XI Xn XO,XI , ' " ,Xn, ljI(x, ljI(x) P3 (x, k) [a, Xk x Xk+I , ljI(x) P3 (x, k) f[Xk,Xk+ k IjI IjI(Xk) f(Xk) k IjI x [xk,xk+d ljI(x) P3 (x, k[a), 3 akl (x - Xk)l, 1jI'(Xk 1jI'(Xk 1jI"(Xk - 1jI1jI""(x(X)k. k [a x [Xk,xk+d 1jI"(Xk+ ) -1jI"(Xk) 1fI"(x) 1fI"(Xk) x - Xk l hk Xk+ (X--Xkx ) 1fI" (Xk+ hk + 1fI" (Xk) /lk , hk Xk+1 - Xb k P3(x, k)- ak,2 = 1jI"(Xk+ + 2ak+(x1.2-x. d3 (Xk+1 _ x)3 X-Xk Xk+1 (-Xkx+1 IjI(x) = ak+L2 3hk + ak,2 3hk +Ck,l + Ck,2 hk ' IjI(Xk) f(Xk) IjI(Xk+l ) f(cXkk.1+I ), Ck.2 hZ ak,2 "3 Ck,2 f(Xk) . I.
1=0
the condition of continuity for its derivatives everywhere on
b]
reduces to the condition of their continuity at the interpolation nodes: - 0) = + 0), 0) = + 0), = I , . . . , n 1 . Consider the function It is piecewise linear and continuous on the interval we can write: , b] . Then, for E =
+
=
d
( 2 .59 )
where we denote � = 0, 1 , . . . , n 1 , hereafter. Moreover, according to the previously adopted notations for we have: 1fI" (xd/2. Then, since the second derivative of the spline is continuous, we obtain: 1fI" 0) = 0 ) = Next, by integrating equality (2.59) twice we get: I
�
where the integration constants = = and
are to be obtained from the conditions and respectively. This yields: and
+
=
A Theoretical Introduction to Numerical Analysis
50
Therefore, the cubic spline for E
k , k I ] is given by the following expression:
x [ X X + lJI(x) (x - xd (Xk+1 - x)3 f(( Xk+ - 1 ,23) X-Xk (f(Xk) ) Xk+I-X lJI(xd f(xd, x [XXk- ,Xk], lJI'(Xk x Xk (Xk + lJI'(Xk - (Xk + + ak + I , 2 3 f(xk+J ) -f(xd f(xd -f(Xk-l) = ak + ] , 2
3hk + ak,2 3hk + h� h� - ak ,2 3 ---;:;;- + J ) a k+ hk ' When deriving formula (2.60) we have already used the condition
(2.60)
=
as well as the condition of continuity of the second derivatives of the spline. We have not used the continuity of the first derivatives yet. To actually do so, we first differentiate formula (2.60) and substitute = thus obtaining + 0). Then we write down a similar fonnula for the interval k I differentiate it and substitute = thus obtaining If/' - 0). By requiring that 0) = If/' 0), we get:
hk - I a 2hk_ 1 2hk ak- I , 2 3 3-+ k, 2
hk
(2.61)
-
Equations (2.61 ) need to b e considered for k = 1 , 2, . . . , n I , which yields a to tal of n I linear algebraic equations with respect to n + I unknown coefficients a , , a] an ,2 . We therefore conclude that the cubic spline is not yet defined unambiguously by the conditions specified previously, and two additional conditions must be provided. For example, we may set a , = all,2 = 0, which corresponds to = = 0 and = = O. Then, system (2.61) can be written in the matrix fonn as follows:
-
O 2 ,2, ' " lJI(x) If/" (xo) lJI"(a) lJI"(xn) If/"(b) O 2 . a2 , 2 , . n- 2 V [ g l g . . , g n-I V , , 2 n+ [f(XO) , J(XI ) , . . ,j(Xn)]T 2"0+2"1 2"1+3 2h2 h2 33 2"2+3 2h3 2", -2+3 2hn_1 h n 2 3 +
(2.62)
Aa = g.
In system (2.62), a = unknowns, g = of dimension
,a I , is an n I -dimensional vector of is the right-hand side: g = Bj, where j = is a given vector of dimension I , A is a square matrix
[a l ,2 ,
(n - l) x (n - l): -r '3 0 A=
!!J.
�
0
'3
• • •
0 0 and B is a matrix of dimension (n - 1 ) x (n 1 ) : I I 0 h(j hi I 1 0 hi h2 I 0 B= 0
0 0 0
0 0 0
0
(--10 - � )
0
0
(- � - t) h2
(- t - �)
0
0
0 0 0
0 0 0
-( h"�2 -hn l ) hn_1 �
1
Algebraic Interpolation
51
Both matrices A and B are tri-diagonal. The matrix A is strictly diagonally dominant by rows, as defined in Chapter 5 (this matrix is also symmetric). Therefore, A is non-singular, and as such, system (2.62) always has one and only one solution a for any right-hand side g. This means that the spline exists and is unique; it can be obtained from the tabulated function values k = 0, I , . . . , n, using formula (2.60). Numerically, the matrix A can be inverted by a very efficient algorithm known as the tri-diagonal elimination. This algorithm is stable, and has linear computational complexity with respect to the dimension of the matrix (see Section 5 .4.2). Let us also note that once the solution a = of system (2.62) has been obtained, the spline can already be built. If necessary, however, the remaining
lfI(x) f(Xk) ,
[a l ,2 , a2,2 , ... ,an_ I ,2]T lfI(x) 3 three coefficients ak ,O , ak , l , and ak ,3 of the polynomial P3 (x, k) = L akl (x - xd for 1=0 every k = 0, 1 , . . . , n - 1 , can also be found explicitly, by merely regrouping the terms in formula (2.60). As has been shown, the Schoenberg cubic spline basically requires two additional conditions for uniqueness. Of course, there are many other choices for these con ditions, besides the previously used (b ) = O. For example, the spline = may have a non-zero curvature at the endpoints a and b; this yields inhomogeneous conditions for its second derivatives instead of the homogeneous ones:
1fI" (a) 1fI"
' - 2an,2 - IfIn(2) , .J 'Y ( b ) =
(2 .J' 'Y (a ) = 2aO,2 - lfIo ) ' _
_
_
1fI�2)
1fI�2)
(2.63)
where and are given constants. The new boundary conditions (2.63) prompt a minor change in system (2.62); namely, the first and last components of its right-hand side g will now be defined as follows:
f(X2 ) -f(Xl) f(xl) - f(xo ) (2) h0 6 ho hI -f(Xn-l) f(Xn-l) -f(Xn-2) IfIn(2) hn-I g, - l f(xn) hn-I 6 hn-2 gl
_
\flo
Alternatively, one may specify the slope of the spline rather than its curvature at the endpoints a and b , which yields two additional conditions for the first derivatives:
f(xl) -f(xo) - a0'2 2ho - a 1 '2 ho = l"1'0Ii l ) ' 3 3 ho (2.64) 2 -f(x f(x _1 hn n) hn-l -d , (I .'YJ (b ) = ) fI , hn-I +a",2 3 +a,,- 1 ,2 3 - l n 1 where, again, 1fI� 1 ) and 1fI� ) are some constants. Relations (2.64) can be added to sys tem (2.61 ) as its new first and new last equation, thus making the overall dimension of the system (n + 1) (n + 1). This modification will not alter the consistency of .J 'Y (a)
==
_
x
the system, and will not disrupt its solvability by the tri-diagonal elimination either. Finally, we may consider periodic boundary conditions for the spline: =
=
If/' (a) v/ (b) , v/' (a) v/' (b) .
(2.65)
A Theoretical Introduction to Numerical Analysis
52
Then, system (2.6 1 ) gets supplemented by one additional equation of the same struc ture (only one is needed because the other one is simply an ,2 = aO ,2 ):
ll --I + a O, 2hll - I + 2ho + a ho = all - I,2 hl,2 3 2 3 3 f(x d - f(xo) f(xo) - f(xll - d ho
and while its resulting matrix form can still be symbolically written as (2.62), the vector of unknowns will now be n-dimensional: a = [ao ,2 , a u , . . . , all- I,2V, so will be the vector of the right-hand sides g = B/; the matrix A will have dimension n x n: 2h" - 1 + 2ho ho
3
"3 0
A=
ho
2ho+2h l
"3
h,, _ 1
-3-
0
/l j
h"
0 0 0
� 3
2/lj"3+2h2
0
0
-3-
0
0
hi
0
3
...
0 0
-
' h" _ 1
,
7iO 0
B=
- do)
, �
I
7iO
(-� t) -
, hi o
(2.66)
h,,- 2 2h,,_ 2 +2h,,_ 1
3
and the matrix B will be a square n x n matrix as well:
(
_I
-3-
I
(-t - t) ···
I
hn- I
0
0
0
. (2.67)
o
In so doing, the vector/ whose components are given tabulated values of the function
f will be n-dimensional: / = [.t(xo) , f (X I ), . . . , j(xn - d V, because f(xn ) = f(xo).
Notice that both the matrix A of (2.66) and the matrix B of (2.67) are symmetric and almost tri-diagonal, except for the non-zero entries in the upper-right and lower-left corners of each matrix. The matrix A of (2.66) can be efficiently inverted by a special cyclic version of the tri-diagonal elimination (see Section 5.4.3). Other types of boundary conditions for the spline l/f(x) at the endpoints a and b can be used as well. Let us now assume that the function f(x) has a bounded derivative of order s + I on the interval [a, b]. Then, one can set the required additional conditions so that the nonlocal spline ljI(x,s) with s continuous derivatives will keep the same optimal approximation properties that characterize both the classical piecewise polynomial interpolation of Section 2.2 and the local smooth piecewise polynomial interpolation of Section 2.3. 1 . Namely, estimates of type (2.5 1 ) will hold for m = 0, 1 , . . . , s:
,)
m (X f(m) (x) d dx IjI ln ,5 o'Oox'Ooh max
I
-
I
:::; const '
( ) k
max hk
s+
l -m
o'Oox'Ooh
max
1 J'(s+ I ) (x) j.
53 Algebraic Interpolation In particular, provided that f(x) is twice continuously differentiable on [a, b], the cu bic spline If/(x, 2) If/(x) , along with its derivatives of orders = 1, 2, will converge to the function f(x) and its respective derivatives with the rate of tJ (hs+ ) On the other hand, the undesirable property of saturation by smoothness, which is m
==
I -m .
inherent for both the classical piecewise polynomial interpolation and local splines, is shared by the Schoenberg splines as well, notwithstanding the loss of their local nature. Namely, the nonlocal splines of smoothness s on a uniform interpolation grid with size will guarantee the order of error ) for the functions that have a maximum of s + 1 bounded derivatives, and they will not provide accuracy higher that have more than s + 1 derivatives. ) even for the functions than Even though the coefficients of a nonlocal spline on a given interval depend on the function values on the entire grid, it is known that in practice the influence of the remote nodes is rather weak. Nonetheless, to actually evaluate the coeffic ients, one needs to solve the full system (2.62). Therefore, a natural task of improving the accuracy of a Schoenberg's spline by adding a few interpolation nodes in a particular local region basically implies starting from the very beginning, i.e., writing down and then solving a new system of type (2.62). In contradistinction to that, the splines of Section 2.3 . 1 are particularly well suited for such local grid refinements. In doing so, the additional computational effort is merely proportional to the number of new nodes. For further detail on the subject of splines we refer the reader to [dBO I ] .
h tJ(hs+ I
2.3.3
tJ(hS+
f(x)
I
[Xk , Xk+ t l
Proof of Theorem 2 . 1 1
This section can be skipped during the first reading. The coefficients of the polynomial
Q2s+1 (x, k) = CO,k + CI,kX + " , + C2s+I,kX2s+ 1
(2.68)
are determined by solving the linear algebraic system (2.43), (2.44). The right-hand sides of the equations that compose sub-system (2.43) have the form f
f
f
aom)Jk-j +a lm)Jk-j+1 + . . . + as Jk-j+s,
(m) ( bo )Jk-j+1 + b(lm)Jk-j+2 + ... + bs Jk-j+S+I, ai(m) bi(lIl) fk-j' fk-j+1, ... , fk-j+s+I' (
m
while those that pertain to sub-system (2.44) have the form
(m
f
f
(Ill) f
= O, m
I , . . . , s,
= O , l , . . . , s,
. and where I - 0 , 1 , . . . , s, m - 0 , 1 , . . . , s, are some numbers that d0 not depend ons Consequently, one can say that the given quantities I, . . determine the solution of system (2.43), (2.44) through the for mulae of type '
fk-j ' fk-j+ . , fk-j+s+1 CO,b C I ,b ... , C2s+I ,k Cr,k = aor Jk-j + al Jk-j+I + ... + as+I(r) Jk-j+s+I, r = 0, 1 , . , 2s + 1 , (2.69) ()
f
r
( )f
5 Hereafter in this section, we will use the notation fl
f
=
f(x/) for all nodes I.
..
A Theoretical Introduction to Numerical Analysis
54 =
I 0, 1 , . . . , s + 1 , are, again, numbers that do not depend on fk- j, fk-HI , substituting expressions (2.69) into formula (2.68) and group k-j+s+ I · allNext,the byterms .ing. . , ftogether that contain each particular fk-HI , I = 0 , 1 , + 1 , we can recast the polynomial (2.68) as follows: s+1 (2.70) Q2s+ 1 (X,k) = "L A-j+IP (x, I) , where p(x, I), 0, 1 , . . . , s + 1 , are some polynomials of the argument x. Let us now replace the quantities fk-j+1 in formula (2.70) using formulae (2. 1 6), see Theo rem 2.4. Then, the polynomial Q2s+1 (x,k) becomes: s Q2s+I (X,k) "L f(Xk-j,Xk-j+ l , ... , Xk-HI)ql (x,k)+ (2.7 1 ) s+ ) (Xk+1 -Xk ) ( l f(Xk-j,Xk-j+I,'" , Xk-j+s+1 )(12s+ 1 (x, k), where q (x,k), 0 , 1 , . . . , s, and i'hs+1 (x,k ) are some polynomials independent of this independence for the purpose of fk-I , Al-j+ 1 > , A- j+s+ I · We will exploit finding the polynomials q l (x,k ), 1 = 0 , 1 , . . . , s. Let us specify A-I , A-j+ 1> . . . , A-j+s arbitrarily, and let us specify fk- Hs+ 1 as follows: where at) ,
. . . ,s
1=0
I =
=
1=0
1 =
•••
According to Theorem 2.9, we then have
while Corollary 2.3 from the Newton's formula (see Section 2. 1 ) implies that the di vided difference of order s 1 vanishes. Then, formula (2.7 1 ) transforms into:
+ f(Xk-j,Xk-j+I, ' " ,Xk-j+s+l) Ps (X,!kj ) = "Ls f(Xk- j , Xk-HI, . .. ,Xk-j+l )ql (x,k).
(2.72)
1=0
By comparing formula (2.72) with formula (2.3), we conclude that as the values
fk- I , fk- HI , . . . , A-Hs have been chosen arbitrarily, then ql (x,k) = (X-Xk_ j)(X ) ... (x - xk-j+I ), 0, 1 , . . . , s. Equality (2.72) also implies that formula Xk(2.7- HI1 ) can be recast as follows: Q2s+1 (x,k) Ps(X,!kj) + (Xk+1 -Xk) (s+ l )f(Xk-j,Xk-HI,··· , xk-j+s+r )<12s+ (x, k). (2.73) To obtain the polynomial <12s+1 (x,k) from (2.73), we specify (2.74) fk-j = fk-j+1 . . . = fk-j+s 0 , and fk-j+s+1 1 . I =
=
1
=
=
=
Algebraic Interpolation
55
Then, equality (2.73) becomes:
Q2s+1 (Xk+ 1 - Xk)(s+ l)f(Xk-j,Xk-HI , · · ,xk-Hs+1 )(12s+I Ps+I (X,j,Xk-j,Xk-HI , . . ,Xk-Hs+l ) s Xk-Hs+I -Xk-j+1 f(Xk-j,Xk-HI , · · ,xk-Hs+I ) s -xk-HI). -I f(Xk-j,Xk-HI , · · ,xk-j+s+I ) [fI(Xk-j+s+1 -xk-HI)] (12s+1 (Xk+ 1 - Xk)-s- I Q2s+1 s (Xk-Hs+I - xk-HI) , (x,k) =
(x,k).
(2.75)
Note that under conditions (2.74), the Lagrange form of the interpolating polynomial, see formula (2. 1), yields
x - xk- HI
= II
'
1=0
(2.76)
while according to its Newton's form (2.3), we have:
II (x
(2.77)
1=0
By comparing (2.76) and (2.77), we conclude that once conditions (2.74) hold: =
1=0
Then, from formula (2.75) we find that
(x, k) II
(x, k) =
(2.78)
1=0
where the expressions on the right-hand side of (2.78) are evaluated, again, under conditions (2.74). Let us now transform formula (2.78) to the required form (2.47), (2.48), see Theo rem 2. 1 1 . For that purpose, we will need to obtain an explicit expression for the poly nomial (x, k) driven by the function values of (2.74). Note that due to condition (2.43), along with (2.74), the value x = appears to be a root of multiplicity s + 1 for the polynomial (x, k)/(x (x,k), so that the function (x i s a polynomial of degree no greater than s . We can then expand with respect to the powers of (x
Q2s+1
Q2s+ 1 Xk Q ) Q2s+1Q(x) - xd+1 - Xk+I ) : .1"+1 Q2s+1 ( ) (X -Xk) Q( ) (X -Xk)s+1 )Q2(X -s+Xk1 )s+1 (X-Xk)s+1 [ Q2s+1k)s+1 ] X - Xk+I )r' [ Q2S-+Xk1 )s+ 1 ] I -Xk)-'5-1 ] (r) =
x,k
x
=
�
1 £.., ;:! r=O
(x, k)
=
(x - x
(x,k)
(r
_
=
(2.79)
(
X-Xk+!
Next, we can use equality (2.44) and write:
� dxr (x
(x, k)
X=Xk+!
(2.80)
[(x
X=Xk+ ! '
56
A Theoretical Introduction to Numerical Analysis
=
where G) l!(:�!)! are the binomial coefficients. However, conditions (2.74) along with the Lagrange formula (2. 1 ) yield [cf. formula (2.76)]:
(2.81)
] \ = [IIS [ � (;) � [!]
Substituting (2.8 1) into (2.80), we obtain:
d' d
' x
Q2s+ I (X, k) (x - xk ) s+ l
x =xk+!
1= 1
.
.
[(X - Xk ) -'- l l
]- 1 C:+
(Xk-j+s+ 1 - Xk-j+i)
" +l
(X - Xk_J+;)
1
X
(2.82)
Finally, we substitute (2.82) into (2.79), and subsequently (2.79) into (2.78), thus obtaining an explicit expression for Q2s+ 1 (X, k) as a double sum with respect to I and r. This sum would depend on x only as a composite function Q2s+ (X , k ) of the argument X (X - Xk)/ (Xk+ l - Xk ) :
=
Q2s+ 1 (x, k) = Q2s+ 1 (X , k) .
1
Then, changing the summation order in the resulting formula for Q2s+ 1 (X , k) ob tained after the last substitution into (2.78), we arrive at expressions (2.47), (2.48). Exercises
1. Let xk+ 1 - Xk = h = const, and let qJ(x,s) be a local spline of given smoothness s defined by formula (2.45) for Xk S; x S; Xk+ I . Explicitly write down the polynomials h (x, s) that would enable representation of qJ(x, s) in the form [ef. the Lagrange formula (2. 1 )]: qJ(x, s)
=
I, f (xk)h(x,s) . k
Analyze the cases: s = 1, j = ° and s = 2, j = 1 .
max 2. Let II f[[ �f [ fl [ , and let the interpolation grid be uniform. Estimate the k- j�lg-j+s+ I value of the Lebesgue constant
11/11 = 1 Xk�X�Xk+ 1
Ls = max
max
[Q2s+ ! (x, k) [
for the local spline with s = 1 and j 0. 3.* Delineate the concluding part of the proof of Theorem 2. 1 1 , given in the last paragraph of Section 2.3.3. 4.* Prove that the nonlocal cubic spline 1f!(x) of (2.60) that satisfies the homogeneous boundary conditions (2.63): vI' (a) = vI'(b) = 0, delivers minimum to the functional =
cfJ(u)
=
f [ull(x)j2dX
.
on the class of all interpolants u = II (X) , a ::; x S; b, u(xk l have a square integrable second derivative: II E Wi [a, bj.
=
f(xk) , k = 0, 1 , . . . , n, that
57
Algebraic Interpolation
5. Let [a, b] = [ -n,n], 11 (x) = [.J [ , h (x) = sinx, and let the interpolation grid be uniform with size h. For both functions II and h implement on the computer the local spline (2.42), (2.58) with s = 2, j 1 , and the nonlocal cubic spline (2.60) with any type of boundary conditions: (2.63), (2.64), or (2.65). Demonstrate that in either case the convergence rate is (j(Jz2 ). =
2 .4
Interpolation of Functions of Two Variables
The problem of reconstructing a function of continuous argument from its discrete table of values can be formulated in the multi-dimensional case as well, for exam ple, when j = j(x,y), i.e., when there are two independent variables. The principal objective remains the same as in the case of one dimension, namely, to build a proce dure for (approximately) evaluating the function in between the given interpolation nodes. However, in the case of two variables one can consider a much wider variety of interpolation grids. All these grids basically fall into one of the two categories: structured or unstructured. 2 .4 . 1
Struct ured Grids
Typically, structured grids on the (x,y) plane are composed of rectangular cells. In other words, the nodes (Xk,Y/) , k = 0, ± 1 , . . . , I = 0, ± I , . . , are obtained as inter sections of the two families of straight lines: the vertical lines x = Xk> k = 0, ± I . . . , and the horizontal lines Y = Y t, I = 0, ± 1 , . . .. In so doing, we always assume that Vk : Xk+ 1 > Xk> and VI : YI+ I > y/. In the literature, such grids are also referred to as rectangular or Cartesian. The grid sizes h x) = Xk+ ! - Xk and h iy) = Yt+ ! - Yt may but do not have to be constant. In the case of constant size(s), the grid is called uniform or equally spaced (in the corresponding direction). The simplest example of a uniform two-dimensional grid is a grid with square cells: hix) = h iy) = const. To approximately compute the value of the function j at the point (x,y) that does not coincide with any of the nodes (Xk,Yt ) of a structured rectangular grid, one can, in fact, use the apparatus of piecewise polynomial interpolation for the functions of one variable. To do so, we first select the parameters s (degree of interpolation) and j, as in Section 2.2. We also need to determine which cell of the grid contains the point of interest. Let us assume that Xk < i < Xk+ l and Yt < Y < Yt+ 1 for some particular values of k and I. Then, we interpolate along the horizontal grid lines: .
,
1
and obtain the intermediate values ]. Having done that, we interpolate along the vertical grid lines and obtain the approximate value j:
58
A Theoretical Introduction to Numerical Analysis
Clearly, the foregoing formulae can be used to approximate the function f at any point (i,ji) inside the rectangular grid cell { (x,y) [ Xk < x < Xk+ l ,Yl < Y < Yl+d. For example, if we choose piecewise linear interpolation along x and y, i.e., s = 1 , then
Note, however, that in general the degree of interpolation does not necessarily have to be the same for both dimensions. Also note that the procedure is obviously sym metric. In other words, it does not matter whether we first interpolate along x and then along y, as shown above, or the other way around. The piecewise polynomial interpolation on the plane, built dimension-by dimension on a rectangular grid as explained above, inherits the key properties of the one-dimensional interpolation. For example, if the function f = f(x,y) is twice differentiable, with bounded second partial derivatives, then the interpolation error of formula (2.83) on a square-cell grid with size h will be d(h2 ). For piecewise polynomial interpolation of a higher degree, the rate of convergence will accord ingly be faster, provided that the interpolated function is sufficiently smooth. On the other hand, similarly to the one-dimensional case, the two-dimensional piecewise polynomial interpolation is also prone to the saturation by smoothness. Again, similarly to the one-dimensional case, one can also construct a smooth piecewise polynomial interpolation in two dimensions. As before, this interpolation may be either local or nonlocal. Local splines that extend the methodology of Sec tion 2.3. 1 can be built on the plane dimension-by-dimension, in much the same way as the conventional piecewise polynomials outlined previously. Their key properties will be preserved from the one-dimensional case, specifically, the relation between their degree and smoothness, the minimum number of grid nodes in each direction, the convergence rate, and susceptibility to saturation (see [Rya75] for detail). The construction of nonlocal cubic splines can also be extended to two dimen sions; in this case the splines are called bi-cubic. On a domain of rectangular shape, they can be obtained by solving multiple tri-diagonal linear systems of type (2.62) along the x and Y coordinate lines of the Cartesian grid. The approximation properties of bi-cubic splines remain the same as those of the one-dimensional cubic splines. Similar constructions, standard piecewise polynomials, local splines, and nonlo cal splines, are also available for the interpolation of multivariable functions (more than two arguments). We should emphasize, however, that in general the size of the tables that would guarantee a given accuracy of interpolation for a function of certain smoothness rapidly grows as the number of arguments of the function increases. The corresponding interpolation algorithms also become more cumbersome.
Algebraic Interpolation 2 .4 . 2
59
Unstructured Grids
Unstructured grids on the (x,y) plane are typically composed of triangular cells. In so doing, it is required that any two triangles have either a common side or a common vertex, or alternatively, do not intersect at all. Unstructured grids offer a lot more flexibility in accommodating irregular geometric shapes, see Figure 2. 1 . In addition, they may also be more convenient when the interpolated function undergoes a strong variation in some local area. Let, for example, the function 1 = l(x,y) be defined on the domain schematically shown in Figure 2. 1 . Let us additionally assume that it varies rapidly inside the "bottleneck." In this case, a rectangular grid will not be par ticularly well suited for tabulating the function l(x,y). One reason is that such a grid obviously cannot fit the curvilin ear boundary of the domain. Another reason is that a rectangular grid cannot be refined locally, i.e., only in the bottleFIGURE 2. 1 : Unstructured grid. neck area, where most of the variation of the function l(x,y) supposedly occurs. On the other hand, the unstructured trian gular grid shown in Figure 2. 1 is obviously free of these disadvantages. Therefore, the vertices of the triangles can be used in the capacity of interpolation nodes. However, a key difficulty in using triangular grids for interpolation is that the previously analyzed one-dimensional methods will not directly extend to this case in a dimension-by-dimension fashion. Instead, triangular grids require genuinely two-dimensional approaches to building the interpolating polynomials. Let, for example, 11 , 12, and 13 be the values of the function l(x,y) at the vertices (Xl , Y I ), (X2 , Y2 ), and (X3 , Y3 ), respectively, of some triangle. Then, inside the triangle the unknown function l(x,y) can be approximated by the linear function (a first degree polynomial of two variables):
l(x,y) � ax + by + c, where the coefficients a, b, and c must satisfy the equalities: ax, + by, + c = 11 , ax2 + bY2 + c = 12 , ax3 + bY3 + c = 13,
(2.84)
in order for the interpolant to coincide exactly with the function values at the interpo lation nodes. Geometrically, the linear function (2.84) is a plane that crosses through the three points: (X l , Y I ,1, ), (X2 , Y2 ,12 ), and (x3 , Y3 ,13) in the three-dimensional space. For a twice differentiable function with bounded second derivatives, the in terpolation error guaranteed by this approach will be tJ(h2 ), where h is the largest
A Theoretical Introduction to Numerical Analysis
60
side of the triangle. If the linear interpolant (2.84) is built for every triangle of the grid, see Figure 2. 1 , then the overall interpolating function will be piecewise linear and continuous on the entire domain. We note that linear interpolants on triangular grids are very helpful for building the finite-element approximations of partial differential equations, see Section 1 2.2 of Chapter 1 2. We also note that piecewise polynomial interpolants of higher degrees can be obtained on triangular grids as well. Exercises
1. Prove the 2.
fJ (h(X) 2 h(y) 2 ) error estimate for the interpolation (2.83), where h(x) and +
h(Y) are the grid sizes (constant in each direction). Prove the fJ(h2) error estimate for the interpolation (2.84), where h is the largest side
of the triangle. Assume that all second order partial derivatives of f(x,y) are bounded.
3. Let f f(x,y) be twice differentiable, and let all of its second partial derivatives be bounded by 1. How can one economically choose the size(s) of the rectangular interpo lation grid on the plane to be able to reconstruct the function f(x,y) everywhere inside the square I xl ::; I , Iyl ::; 1 with the error not exceeding 1 0- 3 ? =
4. Let f = f(x,y) be twice differentiable, let all of its second partial derivatives be bounded by 1 , and let also the circles x2 + l r2 be level lines of the function f = f(x,y) for every r. How can one economically place the interpolation nodes on the plane to achieve the same quality of the reconstructed function as in Exercise 3? =
x2
2
5.* Answer the question of Exercise 4, but instead of having the circles + l r as level lines of f, assume that the function f = f(x,y) can be represented as a product of two single-variable functions: f cP (x) IfI(Y) , where cp (x) =1= 0 and IfI(Y) =1= O. =
=
Chapter 3 Trigonometric Interpolation
Along with the algebraic interpolation described in Chapter 2, one also uses interpo lation by means of trigonometric polynomials of the type:
(
11 ) = L11 ak cos --X+ 2 rr:k 2 rr:k L hk sin --x,
2 rr: 2 rr: Q cos -x, sin -x L L
k=O
L
k= !
L
(3.1 )
where n is a positive integer, L > 0, and ak & bk are the coefficients. A trigonometric interpolating polynomial Q (cos 2Z x, sin 2Z x) that would coincide with the given L periodic function f(x), f(x + L) = f(x), at the equidistant interpolation nodes: m = 0, 1 , . . . , N - 1 ,
Xo = const,
(3.2)
can be chosen such that it will have some important advantages compared to the algebraic interpolating polynomial built on the same grid (3.2). First, the error of the trigonometric interpolation (3.3) converges to zero uniformly with respect to x as N -----> 00 already if the second deriva tive of f(x) is piecewise continuous. ! Moreover, the rate of this convergence, i.e., the rate of decay of the error (3.3) as N -----> DO, automatically takes into account the smoothness of f(x), i.e., increases for those functions f(x) that have more deriva tives. Specifically, we will prove that
Second, it turns out that the sensitivity of the trigonometric interpolating polyno mial (3. 1 ) to the errors committed when specifying the function values fm = f(xm ) on the grid (3.2) remains "practically flat" (i.e., grows slowly) as N increases. The foregoing two properties - automatic improvement of accuracy for smoother functions, and slow growth of the Lebesgue constants that translates into numerical stability - are distinctly different from the properties of algebraic interpolation on l In fact, even less regularity may be required off (x) , see Section 3.2.7.
61
A Theoretical Introduction to Numerical Analysis
62
uniform grids, see Chapter 2. It, however, turns out that algebraic interpolation of functions on an interval can also possess the same remarkable qualities. To achieve that, one shall choose the Chebyshev interpolation nodes and use Chebyshev inter polating polynomials. When reading the book for the first time, the reader can restrict him/herself by what has been said about the contents of Chapter 3, and proceed directly to Chapter 4.
3.1
Interpolation of Periodic Functions
Let f
=
f(x) be an L-periodic function: 'tIX : f(x + L) = f(x) ,
L > O,
(3.4)
,
(3.5)
defined on the grid:
m = 0, ± 1 , ±2, . . .
Xo = const,
where N is a positive integer. Introducing the notation fm virtue of (3.4) and (3.5):
'tim : 3.1 . 1
An Important Nodes
==
f(xm ) , we obtain by
fm+N = fm .
Particular
Choice
of
Interpolation
THEOREM 3. 1 Let Xo = L/ (2N), N = 2( n + I ) , n being a positive integer. Let f be an L-periodic junction, and let fm be its values on the grid (3. 5). For a given arbitmry set oj fm , there is one and only one trigonometric interpolating polynomial:
(
Qn cos
2n
T
x, sin
) T, 2n
x f
=
Il
+!
n 2nk 2nk ak cos T x + L h sin T x, L k= ! k=O
that satisfies the equalities:
m
=
0, ± 1 , ±2, . . . .
(3.6)
63
Trigonometric Interpolation The coefficients of this polynomial are given by the formulae:
NI ao = N1 L- j,n ,
(3.7)
m=O
k = 1 , 2, . . . ,n,
(3.8)
k = I , 2, . . . ,n,
(3.9) (3. 10)
PROOF
Let us consider a set of all real valued periodic discrete functions:
fm+N =fm, m = 0, ±I , ±2, . . . , (3. 1 1 ) defined on the grid Xm = � m + 2�' We will only b e considering these functions
on the grid interval m = 0, 1 , . . . , N 1 , because for all other m's they can be unambiguously reconstructed by virtue of periodicity (3. 1 1 ) . The entire set o f these functions, supplemented by the conventional opera tions of addition and multiplication by real scalars, form a linear space that The dimension of this space is equal to N, because the we will denote system of N linearly independent functions ( vectors ) iJP) E k 1 , 2, . . . , N: -
FN.
FN , =
{ o,
lJIm(k) = 1 , ifif mm l-= kk -- I1 ,, provides a basis in the space FN. Indeed, any function f E FN, f = {fm 1 m = 0, 1 , . . . ,N} always admits a unique representation as a linear combination of k the basis functions iji(k) : f = 2..�= I ik-I iji( ) . Let us now introduce a Euclidean dot ( i.e., inner ) product in the space FN: N-I (3.12) (1,g ) = N1 L fmgm , m=O k and show that the system of 2(n + 1 ) functions: � (k) = g� ) }, k = 0, 1 , . . . , n, and 71 (k) = {71,\;') }, k = I , 2, . . . , n + 1 , where �m = cos (0 · Xm) = 1 , ��k) = V2 cos ( 2�k xm ) , k = 1 , 2, . . . , n, (3. 1 3) 2nk ) , k = I , 2, . . . , n, 71m(k) = v 2sIn ( T Xm 71m(11+ 1 ) - Sin ( 2n (nL+ 1 ) Xm) _- ( l )m , _
def
): (0 )
_
r;;; .
_
.
-
A Theoretical Introduction to Numerical Analysis
64
forms an orthonormal basis in the space FN (recall, n = N/2 - 1 ) . The overall number of functions defined by formulae (3.13) is equal to N. Therefore, it only remains to prove the equalities:
) (� (k , � (k) ) = I , ) ( 1') (k , 1') (k) ) = 1 , (� (r) , � (s) ) = 0, ) ( 1') ( r , 1') (s) ) =0, (� (r) , 1') ( ) ) =0, s
(3. 14 ) (3. 1 5) r I- s, r,s = O, I , . . . , n, (3. 1 6) (3. 17) r l- s, r,s = 1 , 2, . . . , n + l , (3. J 8) r = O, I , . . . , n" s = 1 , 2 , . . . , n + 1 . (3. 14) - (3.18) , we first notice that for any N and y the k = 0, I , . . . , n, k = 1 , 2, . . . , n + 1 ,
To prove equalities following relations hold:
1 N- l 1 = 1 ' 111=0 2 m + y = 0, - � N L.
(: ) � (2 ) L sin l : m + y = 0,
N-l
I O N- l
cos I
111=0 Indeed, (3.19) O, I , . . . , N - l :
i s obviously true. To verify
(3. 19) 1 = 1 , 2, . . . , N - l ,
(3. 20)
1 = 1 , 2, . . . ,N - 1 .
(3.21)
(3.20) ,
we can write for any
II� cosl ( 2: m + y) 1 N-l ( { ( 21nm )} + exp { -i ( ---21nm )}] = "2 111� exp i -----;::;- + ly -;::;- + ly
1=
N- l
( { })
( {
})
1 . N- I 21 n In I - .I Y N- I 21n In + "2 e t L exp -i N = "2 etlY L exp i N 111=0 111=0 1 exp{ 21n} exp{ -i21n} 1 i = �2 eilY l - exp{i21n/N} + �2 e -ilY l - exp{-i21n/N} = ° + ° = °,
(3. 22 )
where we have used the formula for the sum of a geometric sequence. Equality (3.21) is proven similarly. We are now ready to show that the basis (3. 13) is indeed orthonormal, i.e., that equalities (3.14) -(3. 18) do hold. Equality (3. 14) for k = 0, as well as equality (3.15) for k = n + 1 , coincide with (3.19). For k = 1 , 2, . . . , n both equalities (3. 14) and (3.15) hold by virtue of (3.20) and (3.21):
(� (k) , � (k) )
=
I cos2 ( 2nk m + nkN ) I N N �
II1=o
Trigonometric Interpolation
I 2nk )] 4nk m + N = N1 Nm=O L- [ l + cos ( N I N- I = N1 Nm=O L- 1 + m=O L cos 2k ( 2: m + � )
6S
=
1 +0= I,
I 2nk -nk ) ( 'f (k) " f (k) ) N2 Nm=O L- . ( -m+ N N 11
11
_ _ -
SIll
2
2nk ) ] = 1. m + = N1 N1L1=-0I [ 1 - cos ( 4nk N N To prove equality (3.16) , we first notice that for r, s = 0, 1,
always have I ::;
and r i= s we I r ± sl ::; N - 1 , and then use formula (3.20) to obtain: . . . , il
2nl1l + Nn ) } 2nm + Nn ) } . cos { s ( N (� (r) , � ( ) ) = N2 11'=-0 cos { r ( N s
=
N I
I N�- 1 [cos { (r + s) ( N 2nl1l + Nn ) } + cos { (r - s) ( N 2nl1l + Nn ) } ] = 0.
N
/I O
Equality ( 3.17 ) is proven similarly, except that instead of the trigonometric identity co s a cos f3 = cos (a + f3 ) + cos(a - f3 ) that has been used when prov ing ( 3.16) , one rather needs to employ another identity: 2sin a sin f3 cos( a + f3 ) - cos(a - f3 ) . Finally, yet another trigonometric identity: 2 sin a cosf3 = sin( a + f3 ) + sin( a - f3 ) is to be used for proving formula (3.1 8 ) . Altogether, we have established that (3.13) is an orthonormal basis in the space FN . Therefore, every function f U;n } E FN can be represented as a linear combination of the basis functions ( 3.13 ) :
2
=
=
n k=O
2n
n+ 1 k= 1
2n
kXII1 + L bk sin -k Xm· fin = L ak cos L L Calculating the dot products of both the left-hand side and the right-hand side of the previous equality with all the basis functions (r) and (s) , r 0, 1, . . . , n, s . . . , n + 1 , we arrive at the equalities:
= 1 , 2,
�
aO = (j, � (O) ), ak = V2(j,�(k)),
bk = V2(j, T/ (k) ), bll+ 1 = (j, T/ (n+I) ) ,
T/
=
k = I,2, ,n, k = 1,2, . . . ,n, . . .
that, according to definitions ( 3.13 ) , coincide with formulae
(3.7) - (3.lO).
0
A Theoretical Introduction to Numerical Analysis
66
The grid Xm = � m + 2� ' m = 0, ± 1 , ±2, . . . , that has been used for specifying the discrete functions f E FN in Theorem 3.1 is symmetric with respect to the origin x = 0 2 so that along with the point x = Xm it also contains the point X = -Xm = X- (m+ l)' Therefore, one can consider even and odd functions o n this grid. The grid function fm == f (xm ) is called even if ,
fm = f- (m+l ),
m = 0, ± 1 , ±2, . . . .
(3.23)
Similarly the grid function fm is called odd if ,
fm = f- (m+ I ) ,
m = 0, ± 1 , ±2, . . . .
-
(3.24)
THEOREM 3.2 Let fm be an even N-periodic grid function specified at the nodes Xm = � m + 2� ' m = 0, ± 1 , ±2, . . . , N 2(n + 1 ) . Then, the trigonometric interpolating polynomial (3. 6) becomes: =
n
Qn = L ak c os 2rrk L x, k=O
(3.25)
-
where
(3.26) k = I , 2, . . . , n.
(3.27)
For an even function 1,,, [see formula (3.23)] expressions (3.7) and translate into (3.26) and (3.27), respectively, whereas expressions (3.9) and (3.1 0) imply that bk == O. 0 PROO F
(3.8)
THEOREM 3.3 Let fm be an odd N -periodic grid function specified at the nodes Xm = � m + 2� ' m = 0, ± 1 , ±2, . . . , N = 2(n + l) . Then, the trigonometric interpolating polynomial (3. 6) becomes:
(3.28) 2 Note that the origin itself is not a node of this grid.
Trigonometric Interpolation
67
where
k = I , 2 , . . . , n,
(3.29) (3.30)
PRO O F For an odd function fm [see formula (3.24)] formulae (3.7) and (3.8) imply that the coefficients ak , k 0, 1 , . . . , n, are equal to zero, expressions (3.9) and (3.10) then transform into (3.29) and (3.30), respectively, and the polynomial (3.6), accordingly, reduces to (3.28). 0 =
3.1.2
Sensitivity o f the Interpolating Polynomial to Pertur bations of the Function Values
Let us estimate how sensitive the trigonometric interpolating polynomial (3.6) is to the errors committed when specifying the values of the grid function fm. Assume that instead of f = {fm } we actually have a perturbed function f + 8f = {fm + 8fm}. Then, instead o f the polynomial (3.6) w e obtain a new polynomial:
(
) (
)
Qn cos LX, SIll LX, f + of = Qn cos L X, SIll LX, f + 8 Qn .
2n
. 2n
2n
. 2n
From formulae (3.6) and (3.7)-(3. 10) one can easily see that the corresponding error is given by:
(
)
O Qn = QIl cos LX, SIll LX, 8f .
2n
. 2n
Therefore, similarly to how it has been done in Section introduce the Lebesgue constants:
2. 1 .4 of Chapter 2, we can
n = 1 , 2, . . . ,
(3.3 1 )
that would naturally quantify the sensitivity of the trigonometric interpolating poly nomial (3.6) to the perturbations of its input data. Obviously,
(3.32) THEOREM 3.4 The Lebesgue constants of the trigonometric interpolating polynomials (3.6) satisfy the estimates Ln � 4 (n + l ) . (3.33)
A Theoretical Introduction to Numerical Analysis
68 PRO O F
Formulae (3.7 ) -(3 . 10 ) imply that
n l ak l :::; 2 max m [!t l ,
(3.34) (3.35)
Then, from formula (3.6 ) and inequalities (3.34 ) , (3.35 ) applied to the pertur bation 8] one can easily see that
I
(
2n
2n L
)I
Qn cos -x, sin -x, 8]
L
n n+ I :::; L. [ a k [ + L. [bk [ :::; 4( n + l ) max [ 8!tn [ . k=O k= 1 1/1
Estimate (3.36 ) holds for any 8] E FN and any N. This implies (3.33 ) .
(3.36)
o
Let us emphasize that the growth rate of the Lebesgue constants in the case of trigonometric interpolation on equally spaced nodes, see estimate (3.33), is con siderably slower than in the case of algebraic interpolation, see estimates (2. 1 8) in Section 2. 1 .4 of Chapter 2. 3.1.3
Estimate o f Interpolation Error
THEOREM 3.5 Let ] = ](x) be an L-periodic function that has continuous L-periodic deriva tives up to the order r > ° and a square integrable derivative of order r + 1 :
foL [/r+l) (x)f dx < 00
.
Let ] E FN , ] = {fm } , be a table of values of this function sampled at the equally spaced grid nodes
m = 0 , ± 1 , ±2, . . . ,
N = 2(n + I ) ,
and accordingly, let Qn (cos 2t x, sin 2tx, ]) be the trigonometric interpolating polynomial of type (3.6). Then, for the error of the trigonometric interpola tion: 2n . 2n def Rn (x) = ](x) - Qn cos LX, Sill X, ] ,
(
L
)
the following estimate holds:
��
[Rn (X) [ :::; r /2 ' where 'n = o ( l ) as n P RO O F
n ----'> cx>.
Let us represent ](x) as the sum of its Fourier series: ](x)
= Sn(x) + 8Sn (x) ,
(3.37)
Trigonometric Interpolation where
69
0'0
II 2nh 2nh n+ ! - + I. Clk COS -- + I. f3k sin -2 k =! L L k= ! is a partial sum of order n and
Sn (x)
=
(3.38)
(3.39) is the corresponding remainder. The coefficients Clk and series for j(x) are given by the formulae:
Clk =
2 L
L
J o
2nh j(x) cos T dx,
2
fL
13k of the Fourier
. 2nkx
13k = - j(x) sm --dx. L. L
(3.40)
o
Let us first prove the following estimate for the remainder 8Sn of
(3.39) : (3.4 1 )
where �n is a numerical sequence such that �Il = o( I ) , i.e., limn --->= �n = o. Estimate (3.41 ) is an estimate of the convergence rate of the Fourier series for j(x) . Denote by Ak and Bk the Fourier coefficients of the square integrable L-periodic function fr+ ! ) (x):
Bk = -L 2
L
J
o
( . 2nh j r+ ! ) (x) sm --dx. L
Then, we can integrate the previous equalities by parts r + 1 times, employ the periodicity, and use definitions (3.40) to obtain Ao = 0 and either: (3.42) or (3.43) for k = 1 , 2, . . . , depending on the particular value of r. Moreover, according to the Bessel inequality ( see, e.g., [KF75, Section 16] ) we can write: (3.44)
A Theoretical Introduction to Numerical Analysis
70
Let us then define the sequence
I. III
implies that the series equalities
k=1
V
+1 A� + B�. Inequality (3.44)
11k �f u;. r
converges. Now assume for definiteness that
(3.42) hold; then we have:
L ( ) r+ 1 [Ak [ :::; ( L ) r+ 1 VA� + B� 11k = [ak[ kr+1 = kr+1 ' kr+l 2,.
and similarly
2,.
r+1 B L ( ) [ k[ :::; ( L ) r+1 VA� + B� = 11k . = [�k[ kr+1 kr+! kr+l 2,.
2,.
The same estimates can obviously be obtained when relations (3.43) hold instead of (3.42). We can therefore say that the Fourier coefficients (3.40) satisfy the following inequalities:
< 11k [ ak [ kr+! ' where the sequence
(3.45)
11k = 0 ( 1 ) is such that the series I. III converges. k=1
Then, for the remainder
(3.39) of the Fourier series we can write: �
:::; L [ak[ + L [�k [ :::; L [ak [ + [�k[ k=n +!
11k < < 2 k=�n+ 1 kr+1 -
LJ
-
2
(3.46)
k=n+1
k=n+2
�
LJ
k=n+ 1
11k2
�
LJ
1
) k=n+ I k2 (r+I '
where the last estimate in (3.46) is obtained using the classical Holder in equality ( see [KF75, Section 5 ] ) :
which is valid if
i UkVk :::; Ji u� Jk� v� ,
I. uz < 00 and I. vz < 00
.
k=1 Next, we note that for � E [k - l , k] the following inequality always holds: k 1 1 1 :::; :::; ' Therefore, k2 (r+ ! ) � 2 ( r+I ) ' and consequently, k2 (r+ I ) � 2 (r+ I ) k-I k 1 = (2r + l1 n2r+1 = 2d� L k2 (r+ I ) :::; L ) ) ( ( +l r+l r 2 ) k=Il+lk_ 1 � k=n+ 1 Il � �
k=1
�
J d� J
�
J d�
.
Trigonometric InterpoLation Substituting the latter estimate into inequality
71
(3.46), we obtain: (3.47)
It only remains to notice that
I. converges. k=1 III
'/l � ,) 2+ 1
2r
k=I.n+l III
= 0( 1 ) ,
because the series
(3.47) does imply (3.41) . Having justified estimate (3.41 ), we next notice that the partial sum S/l(x) is, in fact, a trigonometric polynomial of type (3.6). Due to the uniqueness of the trigonometric interpolating polynomial, see Theorem 3.1, we then have Then, estimate
(
2n
. 2n
)
Q/l cos L X, sm L X, Sn = S/l(x) . Moreover, estimates
(3.48)
(3.32), (3.33), and (3.41) together yield: (3.49)
where
'n = 0( 1 )
as
n ---+ 00.
Finally, by combining
(3.41) , (3.48), and (3.49) Rn(x) :
we obtain the following estimate for the interpolation error
:x, sin 2:x, j) I 2 2 2 2 = J(X) - Q/l cos :x, sin : x, S/l ) - Q/l (cos : x, sin : x, 8S/l ) I 2 2 = U (x) - S/l(x)) - Q/l (cos : x, sin :x, 8S/l ) I 2 2 = 8 Sn (x) - Q/l (cos : x, sin : x, 8Sn ) I 2 2 ::; 1 8S/l(x) I + I QIl (cos : x, sin :x, 8SIl ) I
j j I 1
( (
I R/l (X) I = J(X) - Q/l cos
2
::; 1 8SIl (x) I + LII I 8SIl (x) I ::;
'n
1 + 4 -.f;- ::; const . -.f;- ,
n r+ !
n r- !
nr- ! which is obviously equivalent to the required estimate (3.37).
D
We emphasize that the rate of convergence of the trigonometric interpolating poly nomials established by estimate (3.37) automatically becomes faster for smoother interpolated functions j(x). In this sense, trigonometric interpolation of periodic functions appears to be not susceptibLe to the saturation by smoothness. This is a remarkable difference compared to the case of algebraic interpolation (Chapter 2), when the convergence rate is limited by the degree of the polynomial.
A Theoretical Introduction to Numerical Analysis
72
It shall be noted though, that there is a gap between the convergence estimate
(3.37) and the estimate of the unavoidable error for reconstructing a function with
r + 1 derivatives on the grid with size '" 1 / n, which is 6' (n - (r+ I ) ) . This gap, however, is of purely technical nature. Estimate (3.33) for the Lebesgue constants that we have used when proving Theorem 3.5 can, in fact, be further improved (Section 3.2.7). Accordingly, estimate (3.37) for the interpolation error can be improved as well. An additional improvement is brought along by the Jackson inequality (Theorem 3.8). Altogether, it is shown (Section 3.2.7, page 87) that the difference between the actual interpolation error and its theoretical limit (the unavoidable error) can be driven down and made as small as a slow logarithmic factor. 3 . 1 .4
An Alternative Choice of Interpolation Nodes
Another type of trigonometric interpolation is also important for applications. THEOREM 3. 6 Let 1 be an L-peTiodic flLnction, and let 1m be its values on the gTid:
m = 0, ± l , ±2, . . . ,
N = 2n.
FOT a given aTbitmTY set of 1m , theTe is one and only one tTigonometTic in teTpolating polynomial:
(
)
n _ 2n 2n 2nk n-I _ 2nk Qn cos -x, sin -x, 1 = L ak cos -x + L bk sin x, L L L L k =1 k=O -
(3.50)
that satisfies the equalities:
m = O, ± I , ±2, . . . . The coefficients of this polynomial aTe given by the fOTm71lae: 1
N- I
(3.5 1 )
ao = N L 111l , m=O
2 N- I
2nkm
ak = N L 1m cos ------;;;- ' IIl=O 1 N-I all = N L fin ( - I ) m , m=O
_
2 N- I
2nkm
b k = N L fin sin ------;;;- ' m=O PRO O F
k = 1 , 2, . . . , n - l
,
(3.52) (3.53)
k = l , 2, . . . , n - 1 .
(3.54)
It is very similar to that of Theorem 3 . 1 , and we omit it here. 0
Trigonometric Interpolation
73
For an even grid function, fm = f- m , formulae (3.5 1)-(3.54) transform into: 0.0
II - I
= N1 (jo + fll ) + N2 L 1,n, m=1
2
4
II - I
2nkm , ak = N (jo + (- 1 /1,,) + N L 1,n cos -;:;-
t1n = � (jo + (- l r1,l)'
m= 1
k = 1 , 2, . , n - 1 , . .
k = 1 , 2, . , n - l , .
.
and the polynomial (3.50) reduces to:
_ (cos -x, 2n sin -x, 2n f) = " COS -x L t1k 2nk .
Q"
L
L
k=O
L
Note that the arguments which are very similar to those used when proving the key properties of the trigonometric interpolating polynomial QII (cos 2;: x, sin 2;: x, f) in Theorems 3.4 and 3.5, also apply to the polynomial On (cos 2;: X, sin 2;: x, f) defined by formulae (3.50)-(3.54). Namely, this polynomial has slowly growing Lebesgue constants and as such, is basically stable with respect to the perturbations of the grid function fm . Moreover, it converges to the interpolated function f(x) as n -----> co with the rate determined by the smoothness of f (x), i.e., there is no saturation. REMARK 3 . 1 If the interpolated function f (x) has derivatives of all orders, then the rate of convergence of the trigonometric interpolating poly nomials to f (x) will be faster than any inverse power of n. In the literature, this type of convergence is often referred to as spectral. 0
3.2
Interpolation o f Functions o n an Interval. Relation between Algebraic and Trigonometric Interpolation
Let f = f (x) be defined on the interval - 1 ::::: x ::::: 1 , and let it have there a bounded derivative of order r + We have chosen this specific interval - 1 ::::: x ::::: 1 as the domain of f (x) , rather than an arbitrary interval a ::::: x ::::: b, for the only reason of simplicity and convenience. Indeed, the transformation x = a!b + t b;a renders a transition from the function f (x) defined on an arbitrary interval a ::::: x ::::: b to the function F (t ) == f ( a!b + t b;a ) defined on the interval - 1 ::::: t ::::: 1 .
I.
3.2.1
Periodization
According to Theorem 3.5 of Section 3.1, trigonometric interpolation is only suit able for the reconstruction of smooth periodic functions from their tables of values.
A Theoretical Introduction to Numerical Analysis
74
Therefore, to be able to apply it to the function J(x) given on - 1 :::; x :::; 1 , one should first equivalently replace J(x) by some smooth periodic function. However, a straightforward extension of the function J(x) from its domain - 1 :::; x :::; 1 to the entire real axis may, generally speaking, yield a discontinuous periodic function with the period L = 2, see Figure 3. 1 .
y
x
FIGURE 3. I : Straightforward periodization.
Therefore, instead of the function J(x), - I :::; x :::; 1, let us consider a new function
y
-1
F ( q» = J (cos q» ,
1 x
FIGURE 3.2: Periodization according to formula (3.55).
x = cos q>. (3.55)
It will be convenient to think that the function F ( q» of (3.55) is defined on the unit circle as a function of the polar an gle q>. The value of F ( q» is obtained by merely translating the value of J(x) from the point x E [- 1 , 1] to the correspond ing point q> E [0, trJ on the unit circle, see Figure 3.2. In so doing, one can inter pret the resulting function F ( q» as even, F ( - q» = F ( q> ) , 2tr-periodic function of its argument q>. Moreover, it is easy to see from definition (3.55) that the deriva+ ! F(rp) tive dr � exists and is bounded.
75
Trigonometric Interpolation 3.2.2
Trigonometric Interpolation
Let us choose the following interpolation nodes:
N = 2(n + I ) .
m = O, ± I , . . . , ±n, - (n + 1),
(3.56)
According to (3.55), the values Fm = F ( CPm ) of the function F( cp ) at the nodes CPm of (3.56) coincide with the values fm = f(xm) of the original function f(x) at the points Xm = cos CPm . To interpolate a 21r-periodic even function F ( cp ) using its tabulated values at the nodes (3.56), one can employ formula (3.25) of Section 3.1: Qn (cos cp, sin qJ , F )
II
= � >k coskcp. k =O
(3.57)
As Fm = fm for all m, the coefficients a k of the trigonometric interpolating polyno mial (3.57) are given by formulae (3.26), (3.27) of Section 3.1:
k= 1
3.2.3
,
2, . . . , n.
(3.58)
Chebyshev Polynomials . Relation between Algebraic and Trigonometric Interpolation
Let us use the equality cos cP = x and introduce the functions:
Tk (X) = coskcp
=
cos (k arccosx) ,
k = 0, I , 2, . . . .
(3.59)
THEOREM 3. 7 The functions n (x) defined by formula {3. 59} are polynomials of degree k = 0, 1 ,2, . . . . Specifically, To(x) = 1 , TJ (x) = x, and all other polynomials: T2 (X) , T3 (X), etc., can be obtained consecutively using the recursion formula
(3.60) PRO O F It is clear that To (x) = cosO = 1 and T] (x) = cos arccos x = x. Then, we employ a well-known trigonometric identity
cos (k + l ) cp = 2 cos qJ coskcp - cos (k -
1 ) cp, k = 1 , 2, . . . , which immediately yields formula (3.60) when qJ = arccosx. It only remains to prove that Tk (X) is a polynomial of degree k; we will use induction with respect to k to do that. For k = 0 and k = 1 it has been proven directly. Let us fix some k > 1 and assume that for all j = 0, 1 , . . . , k we have already shown
76
A Theoretical Introduction to Numerical Analysis
that Tj (x) are polynomials of degree j. Then, the expression on the right-hand side of (3.60) , and as such, Tk+ 1 (x) , is a polynomial of degree k + I . 0 The polynomials Tk (x) were first introduced and studied by Chebyshev. We pro vide here the formulae for a first few Chebyshev polynomials, along with their graphs, see Figure 3.3:
TO (x) = 1 ,
�
�
�
�
�
0 X
�
M
M
M
_1 �__L-�L-�L-�����__���__�__�
FIGURE 3.3: Chebyshev polynomials. Next, by substituting qJ = arccosx into the right-hand side of formula (3.57), we can recast it as a function of x, thus obtaining:
Qn ( cos qJ, sin qJ , where Pn (x,j) = and
F)
11
== P,, x, j ,
(
L ak Tdx),
k=O
)
(3.61) (3.62)
(3.63)
Trigonometric Interpolation Therefore, we can conclude using for mulae (3.6 1 ), (3.62) and Theorem 3.7, that P,,(x,j) is an algebraic polynomial of degree no greater than n that coincides with the given function values f(xm) = j,n at the interpolation nodes x", = cos CPI/1 ' In accordance with (3.56), these interpola tion nodes can be defined by the formula:
n(2m + 1) , 2(n + l ) (3.64) m = O, I , . . . , n,
--1l\ -
77
'
xm = cos cpm = cos
Xn-I
o.
��-
XI
Xo '
X
that basically coincides with fonnula (2.20) of Chapter 2. We are schematically FIGURE 3.4: Chebyshev interpolashowing the nodes (3.64) in Figure 3.4. tion nodes. Note that the points CPm = n�1 m + m = 0, 1 , . . . , n, defined by formula (3.56) are actually zeros of the function 2 (,,: 1) ' cos( n + 1) cp. Accordingly, the points Xm = cos CPm defined by formula (3.64) are roots of the Chebyshev polynomial T,,+ l (x) = cos(n + 1 )cp. In the literature, this particular choice of nodes for the Chebyshev grid, see Figure 3.4, is often referred to as the Chebyshev-Gauss or simply Gauss nodes (an alternative choice of nodes is discussed in Section 3.2.6). In other words, the polynomial Pn (x, j) specified by fonnulae (3.62), (3.63) ren ders algebraic interpolation of the function f(x) based on its values fm sampled at the roots Xm, see (3.64), of the Chebyshev polynomial Tn + l (x) . 3.2.4
Properties of Algebraic Interpolation with Roots o f the Chebyshev Polynomial n l (x) a s Nodes
T+
Equality Qn (cos cP, sin cP, F) = Pn (x,j) implies that the properties of the trigono metric interpolating polynomial Qn (cos cP, sin cP, F) established by Theorems 3.4 and 3.5 of Section 3.1 do carry over to the algebraic interpolating polynomial Pn (x,j) defined by formula (3.62). In particular, the Lebesgue constants Ln that characterize the sensitivity of the polynomial Pn (x,j) to the perturbations of j,n , satisfy estimate (3.33) from Section 3. 1:
LI1 :::; 4(n + l ) ,
and the interpolation error
uniformly converges to zero as n -----> 00 with the rate automatically determined by the number of derivatives r + 1 that the function f(x) has: max IR,, (x) l :::;
- l �x � l
S';/2 '
nr-
where
SIl = o(I),
n -----> oo .
(3.65)
78
A Theoretical Introduction to Numerical Analysis
In other words, similarly to the trigonometric interpolation (see Section 3.1.3), alge braic interpolation on the Chebyshev nodes does not get saturated by smoothness. In other words, the interpolation error self-adjusts to the regularity of the interpolated function without having to change anything in the construction of the method. REMARK 3 . 2 Estimate (3.33) for the Lebesgue constants that we proved in Theorem 3.4 of Section 3.1 can be substantially improved. In fact, the following equality holds [see the bibliography quoted in Section 3.2.7, and cf. formula (2.21) of Section 2.1, Chapter 2] :
Ln
=
-2 7r
In (n + 1 ) + 1 - en,
(3.66)
Accordingly, the estimate for the interpolation error can also be improved, and instead of (3.65) we will obtain: - 1 <::x<:: 1
max JRn (x) J
I n ) = 0 (nr n
+ 1 /2
as
n
---)
00,
(3.67)
where r+ 1 is the maximum number of derivatives that the function f(x) has. Further improvements of estimate (3.67) can be obtained with the help of the Jackson inequality, see Section 3.2.7. D In contradistinction to the Chebyshev nodes (3.64), when a uniform grid is used for interpolation, the Lebesgue constants rapidly grow as n increases, see inequalities (2. 18) of Chapter 2, and convergence of the interpolating polynomial to the function f(x) may break down even for infinitely smooth functions, see Section 2. 1 .5. These are precisely the considerations that make the algebraic interpolation of high degree inappropriate, and prompt the use of piecewise polynomial or spline interpolation on uniform or arbitrary non-uniform grids (see Chapter 2). 3.2.5
A n Algorithm for Evaluating the Interpolating Poly nomial
To obtain the coefficients ak of the polynomial Pn (x,j) of (3.62) with the help of formulae (3.63), as well as to actually evaluate this polynomial itself at a given x E [- 1 , 1], one needs to be able to compute the values of the polynomials Tk (X) , k = 0 1 , 2 . . . , for - 1 ::; x ::; 1. We will show that it is appropriate to use formula (3.60) for this purpose. This formula is obviously easy to use and its computational efficacy is also apparent. We only need to demonstrate that the computations according to this formula are stable with respect to the round-off errors. Consider a difference equation of the type:
, ,
Yk+ 1
=
(3.68)
2.xyk - Yk - h
=
where Yk is the unknown sequence parameterized by the integer quantity k. We will be looking for a solution of equation (3.68) in the form Yk l, where q is a fixed
Trigonometric Interpolation number. Substituting the latter expression into the difference equation obtain the following algebraic equation for q:
79 (3.68), we
It is often called the characteristic equation, and it has two roots:
Due to the linearity of equation (3.68), its general solution can be written in the form
(3.69) where c, and C2 are arbitrary constants. Let us choose these constants C I and C2 so that to satisfy the following conditions:
Yo = To (x) = I , or equivalently,
C] + C2 = I , This implies C I = C2 = equation (3.68):
1 /2, and then formula (3.69) yields the following solution of (3.70)
According to formula (3.60), Tdx) defined by (3.70) shall be interpreted for a given fixed x as a (discrete) function of k that solves equation (3.68). Next, note that when [xl < I the roots q l ,2 = x ± v'x2 - 1 of the characteristic equation are complex conjugate and have unit moduli. Consequently, the quantities It and q� will remain equal to one by their absolute value as k increases. An error committed for some k = ko would cause a perturbation in the values of C I and C2 that enter into formula (3.69) for k > ko . However, due to the equalities [qt[ = [q� [ = 1 , k = 1 , 2, . . . , this error will not get amplified as k increases. This implies numerical stability of the computations according to formula (3.60) for I xl < 1 . 3.2.6
Algebraic Interpolation with Extrema of the Cheby shev Polynomial as Nodes
Tn (x)
To interpolate the function F ( qJ) = f( cos qJ), let us now use the nodes: _
1C
qJm = -m,
n
m = O, I , . . . , n.
A Theoretical Introduction to Numerical Analysis
80
In accordance with Theorem 3.6 of Section 3. 1 , and the discussion on page 72 that follows this theorem, we obtain the trigonometric interpolating polynomial:
Qn (COS cp, sin cp , F ) =
1
1
(fo + 1,, ) + ao = 211 11
n- l
L L
n
L ak cos kcp k=O
=
an 1,n , m= 1 n- I 1 ak = n- (fo + (- 1 /1,,) + -n2 1,11 coskcpm , ,n= 1
,
1 (fo + ( - l Y1,, ) , 211
k = 1 , 2, . . . , n - 1 .
Changing the variable to x = cos cp and denoting Qn (cos cp, sin cp,F) = Pn (x,j) , we have:
Pn (x,j) = ao =
i n- I
l
n
L akTk (X) , k=O
fm , - (fo + 1,1) + 211 111=1
=
an nL k ak = -n (fo + ( - l ) fn ) + -n L fm Tk (Xm ) , m 1
2 "- 1
=1
1
2
n (fo + ( - 1 rfn ) , k = 1 , 2, . . . , n - 1 .
Similarly to the polynomial Pn (x, f) of (3.62), the algebraic interpolating polynomial
Pn (x, j) built on the grid: _
_
7r
Xm = cos CPm = cos -m,
n
m = 0, 1 , . . . , n,
(3. 7 1 )
also inherits the two foremost advantageous properties from the trigonometric inter polating polynomial Qn (cos cp, sin cp, F). They are the slow growth of the Lebesgue constants as n increases (that translates into the numerical stability with respect to the perturbations of 1,n ), as well convergence with the rate that automatically takes into account the smoothness of f(x) , i.e., no susceptibility to saturation. Finally, we notice that the Chebyshev polynomial Tn (x) reaches its extreme values on the interval - 1 :s; x :s; 1 precisely at the interpolation nodes xm of (3.7 1 ) : Tn (xm ) cos trm = (_1 )111, m = 0, 1 , . . . ,n. In the literature, the grid nodes xm of (3.7 1 ) are known as the Chebyshev-Gauss-Lobatto nodes or simply the Gauss-Lobatto nodes. =
3.2.7
More o n t h e Lebesgue Constants and Convergence of Interpolants
In this section, we discuss the problem of interpolation from the general per spective of approximation of functions by polynomials. Our considerations, in a substantially abridged form, follow those of [LG95] , see also [Bab86]. We quote many of the fundamental results without a proof (the theorems of Jack son, Weierstrass, Faber-Bernstein, and Bernstein). The justification of these re sults, along with a broader and more comprehensive account of the subject, can
Trigonometric Interpolation
81
be found in the literature on the classical theory of approximation, see, e.g., [Jac94, Ber52, Ber54, Ach92, Nat64, Nat65a, Nat65b, Lor86, Che66, Riv74]. In the numerical analysis literature, some of these issues are addressed in [Wen66] . In these books, the reader will also find references to research articles. The material of this section is more advanced, and can be skipped during the first reading. The meaning of Lebesgue's constants LI1 introduced in Chapter 2 as minimum numbers that for each n guarantee the estimate [see formula (2. 1 7)]:
is basically that of an operator norm. Indeed, interpolation by means of the polyno mial P,,(x,f) can be interpreted as a linear operator that maps the finite-dimensional space of vectors [JO ,fI , . . . ,fn ] into the space qa, b] of all continuous functions I(x) defined on [a, b]. The space qa, b] is equipped with the maximum norm 1 1 I11 = max I /(x) I· Likewise, the space of vectors J = [JO ,fl , . . . ,1,,] can also be equipped a<.5x5,b
with a maximum norm, but discrete rather than continuous:
max I h l . Then, IIJII = O�J� n
Ln of (2. 17) appears to be the induced norm of the foregoing linear operator: L" =
sup
1 71 = 1
II Pn (x,f) II ·
(3.72)
However, for subsequent analysis it will be more convenient to use a slightly different definition of Ln as norm of an operator that would rather map qa, b] onto itself. -
DEFINITION 3.1 The operator 9n = 911 (XO ,X I , . . . ,xn ) : qa, b] {Pn(x)} C qa, b] takes a function I E qa, b] , samples its values at a given set of nodes {XO ,X I , . . . ,XI1 } E [a, b] thus creating the table {/o ,fI , . . . ,fn }, and subsequently builds the polynomial Pn (x,f,xO ,X I , . . . ,xl1 ) E qa, b] . f----7
LEMMA 3. 1 The operator- fYJl1 introduced by Dejinit'i on 3. 1 is linear and continuous. PRO OF The linearity of 9n is obvious. To show the continuity, we use the Lagrange formula (2. 1 ) of Section 2 . 1 , Chapter 2 , and obtain:
11 I fYJl1 [J] (x) I = 1P',(x,f,xo,x I , . . . ,x,,) 1 :s; L I /k l l lk (x) 1 :s; 11 I 11 L I ldx) l · k=O k=O n
Next, we introduce a new quantity (3.73)
82
A Theoretical Introduction to Numerical Analysis
where the second equality in (3.73) holds because
n
L [h (x) 1 is a continuous function.
k =O
[a, b] is a compact
set, and
Then clearly,
Consequently, 9n is a bounded operator, 9n it is continuous. Moreover, [ [ 9n [[ :::; An .
:
C[a, b]
f-'>
C[a, b] , and therefore,
0
DEFINITION 3.2 The norm of the operator 9n introduced by Defini tion 3. 1 is called the Lebesgue constant of the polynomial interpolation based on the nodes Xo ,xI , . . . , Xn :
(3.74)
Recall that the operator norm on the right-hand side of formula (3.74) is given by:
1[ 9n [ [ = sup II 9n [f] (X) II 11/11=[
=
sup l i Pn (x, J) I[ .
11/11 = [
(3.75)
We have therefore formulated two alternative definitions of the Lebesgue constants - by means of formula (3.72) and by means of formulae (3.74), (3.75). We will now prove that these definitions are, in fact, equivalent. In other words, we will show that the right-hand side of formula (3.75) coincides with the right-hand side of formula (3.72). The difference between these right-hand sides is that in (3.75) the smallest upper bound is taken across the unit sphere in the space C[a, b] that has infinite dimension, whereas in (3.72) it is taken across the unit sphere in the n + 1dimensional space of vectors } = [fo ,J[ , . . . ,In ] . LEMMA 3.2 The Lebesgue constant defined by formulae (3. 74), (3. 75) is the same as the Lebesgue constant defined by formula (3. 72). PRO OF For every vector } = [fo ,J[ , . . . , In ] , consider a piecewise linear function defined as:
=
=
Clearly, f(x) E C[a, b] , and also if II }II 1 then Il f ll 1 . In other words, every unit vector } [fo , iI , . . . , fn ] gives rise to a continuous (piecewise linear) function that belongs to the unit sphere in C[a, b] . Therefore, one can say that the smallest upper bound on the right-hand side of (3.75) is taken across a wider set than that on the right-hand side of (3.72 ) . Consequently,
=
sup li Pn (x,J) II ;::: sup I I Pn (x, J) I I ·
11/11= 1
11111 = 1
(3.76)
Trigonometric Interpolation
83
On the other hand, let f (x) E C[a, bJ be a particular function that realizes the smallest upper bound on the right-hand side of (3.75). By construction, Il f ll 1. Let us sample the values of f (x) at the node� Xo ,X" . . . ,XIl . This yields the table { fo,jj , · . . , jll}' or equivalently, the vector f = [fo ,j" . . . , j;I] . Assume that 11111 < 1 . Then, denote a = 1111 1 - ' > 1 and stretch the vector ]: 1 1-+ al = [afo , af, , · · · , aj;l ] so that II alII = 1 . As the interpolation by means of the polynomials Pn is a linear operator, we obviously have Pn (x, af) = aPIl (x,j) , and consequently, Il Pn (x, af) I I > I I Pn (x, j) II· We have therefore found a unit vector al, for which the norm of the corresponding interpolating polynomial will be greater than the left-hand side of (3.76). The contradiction proves that the two definitions of the Lebesgue constants are indeed equivalent. 0 =
LEMMA 3.3 The Lebesgue constant of (3. 74), (3. 75) is equal to
(3.77) where the q11antity An is defined by formula (3. 73). PRO O F When proving Lemma 3.1, we have seen that 1 I .9'n ll :::; An . We therefore need to show that An :::; 11 9n I I . As has been mentioned, the function ljf(x) Ih(x) 1 is continuous on
cg k=IO
[a, b]. Consequently, 3x* E [a, bj : ljf(x* ) = All. Let us now consider a function E C[a, b] such that fO (Xk) = sign 1k (x* ) , k = 0 , l , 2, . . . , n, and also IIfo l l = I .
fo
For this function we have:
On the other hand, which implies All :::; 11 9n ll . It only remains to construct a specific example of fo E C[a, b]. This can be done easily by taking fo (x) as a piecewise linear function with the values sign h (x* ) at the points Xk J k = 0 , 1 , 2, . . . , n. 0 We can therefore conclude that
Ln
=
max
II
L I h (x) l ·
a'50x'50h k=o
We have used a somewhat weaker form of this result in Section 2. 1 04 of Chapter 2. The Lebesgue constants of Definition 3.2 play a fundamental role when studying the convergence of interpolating polynomials. To actually see that, we will first need to introduce another key new concept and formulate some important results.
A Theoretical Introduction to Numerical Analysis
84
DEFINITION 3.3
The quantity
c(j,P,,) = min max !P,, (x) - j(x) ! p,, ) a�x�b
(3.78)
(x
is called the best appmximation of a given function j(x) by polynomials of degree no greater than n on the interval [a, bj . Note that the minimum in formula (3.78) is taken with respect to all algebraic poly nomials of degree no greater than n on the interval [a, bj , not only the interpolating polynomials. In other words, the polynomials in (3.78) do not, generally speaking, have to coincide with j(x) at any given point of [a, b] . It is possible to show existence of a particular polynomial that realizes the best approximation (3.78). In most cases, however, this polynomial is difficult to obtain constructively. In general, polynomials of the best approximation can only be built using sophisticated iterative algorithms of non-smooth optimization. On the other hand, their theoretical properties are well studied. Perhaps the most fundamental property is given by the Jackson inequality. THEOREM 3.8 (Jackson inequality) Let j = j(x) be defined on the interval [a, b] , let it be r - 1 times continuously differentiable, and let the derivative j(r - I ) (x) be Lipshitz-continuous:
M > O. Then, for any n :2:
r the following inequality holds: b-a r M c(j, P,, ) < Cr 2nr '
(
)
(3.79)
where C,. (�r � are universal constants that depend neither on j, nor on nor on M.
n,
=
The Jackson inequality [Jac94] reinforces, for sufficiently smooth functions, the result of the following classical theorem established in real analysis (see, e.g., [Rud87]): THEOREM 3. 9 (Weierstrass) Let j E C [a, bj . Then, for any c > 0 there is an algebraic polynomial such that '\Ix E [a, bj : !j(x) - Pf (x) ! :s; c.
Pf (x)
A classical proof of Theorem 3.9 is based on periodization of j that preserves its continuity (the period should obviously be larger than [a, bj ) and then on the approximation by partial sums of the Taylor series that converges uniformly. The Weierstrass theorem implies that for j E C [a, bJ the best approximation defined by (3.78) converges to zero: c(j, Pn) � 0 when n � 00. This is basically as much
Trigonometric Interpolation
85
as one can tell regarding the behavior of E(j, Pn ) if nothing else is known about f (x) except that it is continuous. On the other hand, the Jackson inequality specifies the rate of decay for the best approximation as a particular inverse power of n, see formula (3.79), provided that f (x) is smooth. Let us also note that the value of � that enters the expression for Cr = (T) r � in the Jackson inequality (3.79) may, in fact, be replaced by smaller values: 11:
Ko = 1 , Kj = 2 ' known as the Favard constants. The Favard constants can be obtained explicitly for all r = 0 , 1 , 2, . . . , and it is possible to show that all Kr < �. The key considera tion regarding the Favard constants is that substituting them into (3.79) makes this inequality sharp. The main result that connects the properties of the best approximation (Defini tion 3.3) and the quality of interpolation by means of algebraic polynomials is given by the following THEOREM 3. 1 0 (Lebesgue inequality) Let f E C [a, bj and let {xo ,X j , . . . , xn } be an arbitrary set of distinct interpola tion nodes on [a, b] . Then,
(3.80) Note that according to Definition 3. 1 , the operator 9/1 in formula (3.80) generally speaking depends on the choice of the interpolation nodes. PRO O F
It is obvious that we only need to prove the second inequality in
( 3.80 ) , i.e., the upper bound. Consider an arbitrary polynomial Q (x) E {�l (X) } of degree no greater than n. As the algebraic interpolating polynomial is unique ( Theorem 2.1 of Chapter 2), we obviously have 9n[Q] = Q . Next, I I f - 911 [1] [[
=
I I f - Q + 9n [Q] - 911[f] [I :; I I f - QII + lI &n[Q] - 911[1] 1 1 = I I f - Q [[ + 11 911[ Q - f] [[ :; I I f - QII + I I 9"n ll I I f - Q [ I = ( 1 + Ln ) lI f - QII ·
Let us now introduce 0 > 0 and denote by Q.s (x) E {Pn (x)} a polynomial for which I I f - Q.s [[ < E(j' �l) + O. Then,
[[ f - 9n [1] I I :; ( 1 + Ln) lI f - Q.s 11 < ( 1 + Ln ) (E(j' �I ) + O). Finally, by taking the limit
0
-----;
0, we obtain the desired inequality ( 3.80 ) . 0
The Lebesgue inequality (3.80) essentially provides an upper bound for the inter polation error I I f - 911 [1] [ 1 in terms of a product of the best approximation (3.78)
86
A Theoretical Introduction to Numerical Analysis
times the Lebesgue constant (3.74). Often, this estimate allows one to judge the convergence of algebraic interpolating polynomials as n increases. It is therefore clear that the behavior of the Lebesgue constants is of central importance for the convergence study. THEOREM 3. 1 1 (Faber-Bernstein) For any choice of interpolation nodes XO , X l , following inequality holds:
Ln >
. • •
, XI! on the interval
1 8yr.;1r ln (n + l ) .
[a, b],
the
(3.81)
Theorem 3. 1 1 shows that the Lebesgue constants always grow as the grid dimen sion n increases. As such, the best one can generally hope for is to be able to place the interpolation nodes in such a way that this growth will be optimal, i.e., logarithmic. As far as the problem of interpolation is concerned, if, for example, nothing is known about the function f E qa, bj except that it is continuous, then nothing can be said about the behavior of the error beyond the estimate given by the Lebesgue inequality (3.80). The Weierstrass theorem (Theorem 3.9) indicates that £(j, Pn ) ----t o as n ----t 00 , and the Faber-Bernstein theorem (Theorem 3.11) says that Ln ----t 00 as n ----t 00. We therefore have the uncertainty 0 . 00 on the right-hand side of the Lebesgue inequality; and the behavior of this right-hand side is determined by which of the two processes dominates - the decay of the best approximations or the growth of the Lebesgue constants. In particular, if limn --->� Ln £(j, Pn) = 0, then the interpolating polynomials uniformly converge to f(x). If the function f(x) is sufficiently smooth (as formulated in Theorem 3.8), then combining the Lebesgue inequality (3.80) and the Jackson inequality (3.79) we ob tain the following error estimate: 3
( )
b-a 'M Il f - 8¢'1![JJ II < (Ln + I )C, -2- n"
(3.82)
which implies that the convergence rate (if there is convergence) will depend on the behavior of Ln when n increases. If the interpolation grid is uniform (equidistant nodes), then the Lebesgue constants grow exponentially as n increases, see inequal ities (2. 18) of Section 2. 1 , Chapter 2. In this case, the limit (as n ----t 00) on the right-hand side of (3.82) is infinite for any finite value of r. This does not necessarily mean that the sequence of interpolating polynomials 8¢'I! [J] (x) diverges, because in equality (3.82) only provides an upper bound for the error. It does mean though that in this case convergence of the interpolating polynomials simply cannot be judged using the arguments based on the inequalities of Lebesgue and Jackson. 3 Note that in estimate (3. 82) the function fix) is assumed to have a maximum of , - I derivatives, and the derivative f( r- I ) (x) is required to be Lipshitz-continuous (Theorem 3.8), which basically makes fix) "almost" r times differentiable. Previously, we have used a slightly different notation and in the error estimate (3.67) the function was assumed to have a maximum of r + I derivatives.
Trigonometric Interpolation
87
On the other hand, for the Chebyshev interpolation grid (3.64) the following the orem asserts that the asymptotic behavior of the Lebesgue constants is optimal: THEOREM 3.12 (Bernstein) Let the interpolation nodes XO, X l , . . . ,xn on the interval [- 1 , 1] be given by roots of the Chebyshev polynomial Tn + l (x) . Then,
4
1C
Ln < 8 + - 1n(n + 1 ) .
(3.83)
Therefore, according to (3.82) and (3.83), if the derivative fr- l ) (x) of the function f(x) is Lipshitz-continuous, then the sequence of algebraic interpolating polynomials built on the Chebyshev nodes converges uniformly to f(x) with the rate 6'(n -r ln(n + 1 ) )
as
n --+ oo•
Let us now recall that a Lipshitz-continuous function f(x) on the interval [- 1 , 1]: I f(x' ) - f(x") I ::; constlx' - x" l ,
x', x" E [- 1 , 1] ,
i s absolutely continuous o n [- 1 , 1], and as such, according t o the Lebesgue theorem [KF75], its derivative is integrable, i.e., exists in Ld- 1 , 1 ] . In this sense, we can say that Lipshitz-continuity is "not very far" from differentiability, although this is, of course, not sufficient to claim that the derivative fr) (x) is bounded. On the other hand, if f(x) is r times differentiable on [- 1 , 1] and f(r) (x) is bounded, then fr- l ) (x) is Lipshitz-continuous. Therefore, for a function with its r-th derivative bounded, the rate of convergence of Chebyshev interpolating polynomials is at least as fast as the inverse of the grid dimension raised to the power r (smoothness of the interpolated function), times an additional slowly increasing factor '" Inn. At the same time, recall that the unavoidable error of reconstructing a function with r derivatives on a uniform grid with n nodes is 6' (n - r ) . This is also true for the Chebyshev grid, because Chebyshev grid on the diameter is equivalent to a uniform grid on the circle, see Figure 3.4. Consequently, accuracy of the Chebyshev interpolating polynomials appears to be only a logarithmic factor away from the level of the unavoidable error. As such, we have shown that Chebyshev interpolation is not saturated by smoothness and practically reaches the intrinsic accuracy limit. Altogether, we see that the type of interpolation grid may indeed have a drastic effect on convergence, which corroborates our previous observations. For the Bern stein example f(x) = lxi, - 1 ::; x ::; 1 (Section 2. 1 .5 of Chapter 2), the sequence of interpolating polynomials constructed on a uniform grid diverges. On the Chebyshev grid we have seen experimentally that it converges. Now, using estimates (3.82) and (3.83) we can say that the rate of this convergence is at least 6'(n- l ln(n + 1 ) ) . To conclude, let u s also note that strictly speaking the behavior o fLn o n the Cheby shev grid is only asymptotically optimal rather than optimal, because the constants in the lower bound (3.8 1 ) and in the upper bound (3.83) are different. Better values of these constants than those guaranteed by the Bernstein theorem (Theorem 3 . 1 2)
88
A Theoretical Introduction to Numerical Analysis
have been obtained more recently, see inequality (3.66). However, there is still a gap between (3.8 1 ) and (3.66). REMARK 3.3 Formula (3.78) that introduces the best approximation according to Definition 3.3 can obviously be recast as
£ (j, Pn)
=
min II P,/ (x) - f(x) l i e,
Pn (x)
where the norm on the right-hand side is taken in the sense of the space qa, b] . In general, the notion of the best approximation admits a much broader in terpretation, when both the norm (currently, I I . li e ) and the class of approxi mating functions (currently, polynomials p,/ (x)) may be different. In fact, one can consider the problem of approximating a given element of the linear space by linear combinations of a pre-defined set of elements from the same space in the sense of a selected norm. This is the same concept as exploited when introducing the Kolmogorov diameters, see Remark 2.3 on page 4 l . For example, consider the space L2 [a , b] of all square integrable functions (x) f , x E [a , b] , equipped with the norm:
I I f l 12 �
[1 f (X)dX] 2 b
1
This space is known to be a Hilbert space. Let us take an arbitrary f E L2 [a, b] and consider a set of all trigonometric polynomials Qn (x) of type (3.6), where L b - a is the length of the interval. Similarly to the algebraic polynomi als P,/ (x) employed in Definition 3.3, the trigonometric polynomials Qn (x) do not have to be interpolating polynomials. Then, it is known that the best approximation in the sense of L2 : =
£2(j, Qn ) = min I l f (x) - Qn (X) 11 2 Qn(x) is, in fact, realized by the partial sum Sn (x) , see formula (3.38), of the Fourier series for f (x) with the coefficients defined by ( 3.40 ) . An upper bound for the actual magnitude of the L2 best approximation is then given by estimate ( 3.41 ) for the remainder 8Sn(x) of the series, see formula (3.39 ) :
£2(j, QII )
:::;
-.f;-,
nr+ z
where Sn
= 0( 1 ), n ---> 00,
where r + 1 is the maximum smoothness of f(x). Having identified what the best approximation in the sense of L2 is, we can easily see now that both the Lebesgue inequality (Theorem 3.10 ) and the error estimate for trigono metric interpolation (Theorem 3.5 ) are, in fact, justified using the same argu ment. It employs uniqueness of the corresponding interpolating polynomial, the estimate for the best approximation, and the estimate of sensitivity to perturbations given by the Lebesgue constants. 0
Trigonometric Interpolation
89
Exercises
1.
Let the function f f(x) be defined on an arbitrary interval [a, b], rather than on [- 1 , 1]. Construct the Chebyshev interpolation nodes for [a, b], and write down the in terpolating polynomials P" (x,f) and Pn (x,f) similar to those obtained in Sections 3.2.3 and 3.2.6, respectively. =
2. For the function f(x) x2�1/4 ' - 1 ::; x ::; 1, construct the algebraic interpolating poly nomial PIl (x,f) using roots of the Chebyshev polynomial T,,+ I (x), see (3. 64) , as inter polation nodes. Plot the graphs of f(x) and Pn (x,f) for n 5 , 10, 20, 30, and 40. Do the same for the interpolating polynomial P,,(x,f) built on the equally spaced nodes Xk = - 1 + 2k/n, k 0, 1 , 2 , . . . , n (Runge example of Section 2. 1 .5, Chapter 2). Explain the observable qualitative difference between the two interpolation techniques. =
=
=
3. Introduce the normalized Chebyshev polynomial
2 1 -1I T,I (x).
tn (x) of degree n by setting 7;, (x) =
a) Show that the coefficient in front of x' in the polynomial
7;, (x) is equal to one. b) Show that the deviation max 17;, (x) I of the polynomial tn (x) from zero on the interval
-
1 S;xS; 1
- 1 ::; x ::; 1 is equal to 2 1 -11 .
c)* Show that among all the polynomials of degree n with the leading coefficient (i.e., the coefficient in front of Xl) equal to one, the normalized Chebyshev polynomial tn (x) has the smallest deviation from zero on the interval - I ::; x ::; 1 .
d) How can one choose the interpolation nodes to , tl , . . . , til on the interval [- 1 , 1], so that the polynomial (t - to ) (t - t I l . . . (t - til ) , which is a part of the formula for the interpolation error (2.23), Chapter 2, would have the smallest possible deviation from zero on the interval [- 1 , I]?
4. Find a set of interpolation nodes for an even 2n-periodic function F ( cp), see formula (3.55), for which the Lebesgue constants would coincide with the Lebesgue constants of algebraic interpolation on equidistant nodes.
Chapter 4 Compu ta tion of Definite Integrals. Qu adr a tures
A definite integral
lb j(x)dx can only be evaluated exactly if the primitive (an
tiderivative) of the function is available in the closed form: = e.g., as a tabular integral. Then we can use the fundamental theorem of calculus (the Newton-Leibniz formula) and obtain:
j(x)
F(x) f j(x)dx,
lb j(x)dx = F(b ) -F(a). It is an even more rare occasion that a multiple integral, say,
fLj(x, y)dXdy, where
Q is a given domain, can be evaluated exactly. Therefore, numerical methods play an important role for the approximate computation of definite integrals.
In this chapter, we will introduce several numerical methods for computing defi nite integrals. We will identify the methods that automatically take into account the regularity of the integrand, i.e., do not get saturated by smoothness (Section 4.2), as opposed to those methods that are prone to saturation (Section 4. 1). We will also discuss the difficulties associated with the increase of dimension for multiple integrals (Section 4.4) and outline some combined analytical/numerical approaches that can be used for computing improper integrals (Section 4.3). Formulae that are used for the approximate evaluation of definite integrals given a table of values of the integrand are called quadrature formulae.
4.1
Trapezoidal Rule, Simpson's Formula, and the Like
Recall that the value of the definite integral
lb j(x)dx can be interpreted geomet
j j(x).
rically as the area under the graph of =
91
A Theoretical Introduction to Numerical Analysis
92 4.1.1
General Construction of Quadrature Formulae
To compute an approximate value of the foregoing area under the graph, we first introduce a grid of distinct nodes and thus partition the interval [a, b] into n subinter vals: a = Xo < XI < . . . < XII = b. Then, we employ a piecewise polynomial interpolation of some degree s, and replace the integrand j = j(x) by the piecewise polynomial function:
In other words, on every subinterval [Xk ,Xk+ l ] , the function Ps (x) is defined as a polynomial of degree no greater than s built on the s + 1 consecutive grid nodes: Xk-j ,Xk - j+ I , ' " ,Xk-j+s, where j may also depend on k, see Section 2.2. Finally, we use the following approximation of the integral:
J j(x)dx J Ps(x)dx = J P,, (x,jOj )dx+ J P,, (x,jl j )dx + . . J P,, (x,jn - I ,j )dx. b
b
�
a
a
X2
XI
b
. + xn-!
XI
a
(4.1)
Each term o n the right-hand side o f this formula is the integral of a polynomial of degree no greater than s. For every k = 0, 1 , . . . , n - 1, the polynomial is given by P,,(X,jkj ), and the corresponding integral over [xk ,xk+ d can be evaluated explicitly using the fundamental theorem of calculus. The resulting expression will only con tain the locations of the grid nodes Xk - j ,Xk- j + 1 , . . . ,Xk -j +s and the function values j(Xk -j ),j(Xk - j+ J ) , . . . ,j(Xk- j+s). As such, formula (4. 1) is a quadrature formula. It is clear that the following error estimate holds for the quadrature formula (4. 1):
I J j(x)dx - J p,\ (X)dX I � J ij(x) - P,, (x) idx b
b
a
b
a
a
(4.2)
THEOREM 4.1 Let the function j(x) have a continuous derivative of order s + 1 on [a, b], and max iXk+ I - Xk i = 11. Then, denote max iis + I ) (x) i = Ms+ I . Let also
k=O,1 , ... ,1/ - 1
aSxSb
I J j(x)dx - J P,,(X)dX I b
a
b
II
� const
· ib - a i Ms+ 1 hs+ I .
(4.3)
93
Computation of Definite Integrals. Quadratures
According to Theorem 4. 1 , if the integrand f(x) is s + 1 times continuously differ entiable, then the quadrature formula (4. 1) has order of accuracy s + I with respect to the grid size h. PRO O F In Section 2.2.2, we obtained the following error estimate for the piecewise polynomial algebraic interpolation, see formula (2.32) :
Xk::;max X::;Xk+l If(x) - Ps(X,jkj) ! :::; const · Xk- max
j ::;X::;Xk _ j+s
Consequently, we can write: max
a::;x::;b
I lS+ I ) (x) l hs+ l •
� s
I f(x) - f� (x) 1 :::; const · M. + I h + I .
Combining this estimate with inequality (4.2) , we immediately arrive at the desired result (4.3). 0 Quadrature formulae of type (4.1) are most commonly used when the underlying interpolation is piecewise linear (the trapezoidal rule, see Section 4.1 .2) or piecewise quadratic (Simpson's formula, see Section 4. 1.3). 4.1.2
Trapezoidal Rule
Let s = I , and let the grid be uniform: k = 0, 1, . . . , n,
Xk = a + k · h,
-
b - a. h- n
tk+1 PI (x)dx lxk+1 PI (x,jkO )dx is equal to the area of the .!tk trapezoid under the graph of the linear function PI (X,jkO ) = f(Xk ) (Xk+ I - x)/h + ==
Then the integral
xk
f(Xk+ I ) (x - Xk ) /h on the interval [XbXk+ d : b - a fk + fk+1 . PI (x)dx = -n 2 Adding these equalities together for all k = 0, 1 , . . . , n I , we transform the general formula (4. 1) to:
lXk+l Xk
J f(x)dx J PI (x)dx b
a
b
�
it
=
(
-
b - a fo f 1,, n- 2 + l + . . . + 1,,- 1 + 2
).
(4.4)
Quadrature formula (4.4) is called the trapezoidal rule. Error estimate (4.3) for the general quadrature formula (4. 1) can be refined in the case of the trapezoidal rule (4.4). Recall that in Section 2.2.2 we obtained the fol lowing error estimate for the piecewise linear interpolation itself, see formula (2.33):
Xk ::;X::;Xk+ max
I
h2 I f(x) _ PI (x) l :::; 8
Xk ::;max X::;Xk+ !J" (x) l . I
94
A Theoretical Introduction to Numerical Analysis
Consequently,
h2 l s (x) P lf(x) I 8 I +l � Xkg�. a���h lf /(x) l,
and as the right-hand side of this inequality does not depend on what particular subin terval [Xk , Xk+ I ] the point x belongs to, the same estimate holds for all x E [a, b]:
h2 max I fl/ (x) I (x) If(x) P 1 S · I a 9Sb 8 a9Sb According to formula (4.2), the error of the trapezoidal rule built on equidistant nodes -
max
is then estimated as:
i jab f(X)dx - jab PI (X)dXi
s
b-a 8
max I fl/(x) l h2 . aSxS h
(4.5)
This estimate guarantees that the integration error will decay no slower than tJ(h2 ) as h --.., O. Alternatively, we say that the order of accuracy of the trapezoidal rule is second with respect to the grid size h. Furthermore, estimate (4.5) is in fact sharp in the sense that even for infinitely differentiable functions [as opposed to only twice differentiable functions that enable estimate (4.5)] the error will only decrease as tJ(h2 ) and not faster. This is demonstrated by the following example. Example
10 1
[a,b] = [0, 1]. Then, x2dx = 31 o the trapezoidal rule (4.4) we can write: Let f(x) = x2 and let
-.
On the other hand, using
Therefore, the integration error always remains equal to h2 /6 even though the in tegrand f(x) = � has continuous derivatives of all orders rather than only up to the second order. In other words, the error is insensitive to the smoothness of f(x) [beyond fl/ (x)], which means that generally speaking the trapezoidal quadrature for mula is prone to the saturation by smoothness. There is, however, one special case when the trapezoidal rule does not get satu rated and rather adjusts its accuracy automatically to the smoothness of the integrand. This is the case of a smooth and periodic function f = f(x) that has period L = b - a. For such a function, the order accuracy with respect to the grid size h will no longer
Computation ofDefinite Integrals. Quadratures
95
+
be limited by s 1 = 2 and will rather increase as the regularity of f(x) increases. More precisely, the following theorem holds. THEOREM 4.2 Let f = f(x) be an L-periodic function that has continuous L-periodic deriva tives up to the order r > 0 and a square integrable derivative of order r 1 :
+
+ J [ir+ l) (x) ] 2 dx < 00.
aL a
Then the error of the trapezoidal quadrature rule (4.4) can be estimated as:
a+L a+L f(x)dx - PI (X)dX a a where 'n 0 as n --+ 00.
/J
a+L ' f(X)dX - h 1 a
/ /J
J
=:
--+
�0 (� + fl; 1 ) / I
::; L ·
:� nr /2 ' (4.6)
The result of Theorem 4.2 is considered well-known; a proof, for instance can be found in [DR84, Section 2.9]. Our proof below is based on a simpler argument than the one exploited in [DR84]. PROOF We first note that since the function f is integrated over a full period, we may assume without loss of generality that a = O. Let us represent f(x) as the sum of its Fourier series (cf. Section 3.1 .3):
(4.7)
where
5n - 1 (x) = is a partial sum of order
ao
II
n- I 2nkx 2nkx + L ak cos L + L 13k sin -L 2 k=1 k=1
(4.8)
+
(4.9)
--
-
n 1 and -
�
.
�
2nkx
2nkx 05n_ I (X) = L ak cos T L 13k sm T k=n k=n+ 1
is the corresponding remainder. The coefficients ak and 13k of the Fourier series for f(x) are given by the formulae: ak
=
2 L
L
J o
2nkx f(x) cos T dx,
According to ( 4.7) , we can write:
L
L
13k
=
2 L
L
L
2nkx J f(x) sin T dx.
o
J f(x)dx = J 5n - 1 (x)dx + J 05n- l (x)dx. 0
0
0
(4. 10)
A Theoretical Introduction to Numerical Analysis
96
Let us first apply the trapezoidal quadrature formula to the first integral on the right-hand side of (4.10). This can be done individually for each term in the sum 4 . 8 . First of all, for the constant component 0 we immediately derive:
( )
k=
L L (10/2 + -(10/2 ) = L- (10 = f Sn_l(x)dx = f f(x)dx. h Il-I,I ( --
2 2 2 For all other terms k= 1 , 2 , . . . , n - l , we exploit periodicity with the period L, usc the definition of the grid h = L/n , and obtain: 2nklh 2nklh + 1 cos 2nk(l + l )h ) h '�I cos -h �I,L., ( -2 cos -L L L 2 ( n - -2h I, (e +e ) - -h2 11 --ee1-r + 1 -- ee-1-y- - 0 Analogously, � sin 2nk(l + 1 ) h ) = h 'f ( �2 Sin 2nklh + L 2 L Altogether we conclude that the trapezoidal rule integrates the partial sum Sn-I (x) given by formula (4.8) exactly: L h I ( Sn-,2(Xd + Sn- l 2(xl+d ) = ! Sn_ I (X)dx = �(10. (4.1 1 ) 2 We also note that choosing the partial sum of order n - 1 , where the number of grid cells is n, is not accidental. From the previous derivation it is easy to see that equality (4.11) would no longer hold if we were to take Sn (x) instead of Sn-I (X). Next, we need to apply the trapezoidal rule to the remainder of the series 8Sn- l(x) given by formula (4.9). Recall that the magnitude of this remainder, or equivalently, the rate of convergence of the Fourier series, is determined by the smoothness of the function f(x). More precisely, as a part of the proof of Theorem 3.5 ( page 68), we have shown that for the function f(x) that has a square integrable derivative of order r+ 1 , the following estimate holds, see formula (3.41): (4. 1 2) sup [8Sn - 1 (x)[ :::; +S': /2 ' nr where Sn o( 1 ) when n Therefore, ( 8Sn;I (X1) 8Sn-�xl+ d ) �� ( [8Sn-I (XI )[ + [ 8Sn-I (XI+d l ) :::; -h2 211 sup [8Sn-1 (x)[ = nr� +Sn /2 · 1-0
1=0
_
0
0
1
1
1 =0
-
i 2Trklh L
_i 2"klh
-r
=
i 2Trknh
-r
_
· 2"kh
1=0
I
I/�
1
i 2Trkllh -r
· 2Trkh
o
1 =0
=
,L.,
/=0
)
.
0
O$x$L
+
--+ 00.
1
:::;
O $x$L
[.
_
.
Computation ofDefinite Integrals. Quadratures
97
o
which completes the proof of the theorem.
Thus, if the integrand is periodic, then the trapezoidal quadrature rule does not get saturated by smoothness, because according to (4.6) the integration error will self adjust to the regularity of the integrand without us having to change anything in the quadrature formula. Moreover, if the function f(x) is infinitely differentiable, then Theorem 4.2 implies spectral convergence. In other words, the rate of decay of the error will be faster than any polynomial, i.e., faster than tJ(n- r ) for any r > O. REMARK 4 . 1 The result of Theorem 4.2 can, in fact, be extended to a large family of quadrature formulae (4.1 ) called the Newton-Cotes formulae. Namely, let n = M · s, where M and s are positive integers, and partition the original grid of n cells into M clusters of s h-size cells:
Xm = m · sh, m = 0, I , . . . , M. Clearly, the relation between the original grid nodes Xk , k = 0, 1, . . . ,n, and the coarsened grid nodes Xm , m = 0, 1 , . . . , M, is given by: xm = Xm . s . On each inter val [xm ,xm+ !], m = 0, 1 , . . . , M - 1 , we interpolate the integrand by an algebraic polynomial of degree no greater than
s:
f(x) Ps(x,f,xmos , Xm.s+ ! , . . . ,X(m+ I ) 's), Xm == xm.s ::; X ::; X(m+ ! ) os == Xm+ I . ;:::0
This polynomial iH obviously unique. In other words, we can say that if
x E [Xk ' Xk+ I ] , k = 0, 1 , . . . , n - 1 , then f(x) p'\, (X,J,Xk- j ,Xk- j+ I , ' " ,Xk -j+s) , ;:::0
where j = k - [kls] · s,
and [ . ] denotes the integer part. The quadrature formula is subsequently built by integrating the resulting piecewise polynomial function p.\,(x); it clearly belongs to the class (4. 1). In particular, the SimpHon formula of Section 4.1.3 is obtained by taking n even, setting s = 2 and M = n 1 2. For periodic integrands, quadrature formulae of this type do not get Haturated by smoothness, see Exercise 2. Quadrature formulae of a completely different type that do not saturate for non-periodic integrands either are discussed in Section 4.2. 0
98
A Theoretical Introduction to Numerical Analysis
4.1.3
S impson's Formula
f(x) f(x),
This quadrature formula is obtained by replacing the integrand with a piece wise quadratic interpolant. For sufficiently smooth functions the error of the Simpson formula decays with the rate t!J(h4 ) when h ------> 0, i.e., faster than the error of the trapezoidal rule. Assume that n is even, n = 2 , and that the grid nodes are equidistant: = h = (b a)/n, k = 0, 1 , . . . , n - 1 . On every interval m = O, I , by a quadratic interpolating we replace the integrand polynomial which is defined == uniquely and can be written, e.g., using the Lagrange formula (see Section 2. 1. 1):
M [X2m , X2m+2 ], - l, f(x) . .. , M(X,j,X P2 2m , X2m+ l , X2m+2 ) P2 (x, 12m ,12m+ 1 , 12m+2 ), 2m+ I ) (x -X2m+2 ) f(x) ';::j P2 (X,j2m, j2m+ 1 , 12m+2 ) = 12m (x2m(x -X -X2m+ I )(x2m -X2m+2 ) 2m ) (x -X2m+2) +12m+2 (x -X2m) (x - X2m+ I ) + 2m+I (X2m+I(x -X (X2m+2 - x2m )(X2m+2 -X2m+I ) x - 2m )(X2m+1 -X2m+2 ) Equivalently, we can say that for x E [Xk, Xk , k = 0, 1 , . . . , n - 1, we approximate f(x) by means of the quadratic polynomial P+!l2 (X,jkj) P2 (X,j, Xk-j , Xk-j+I , Xk -j+2 ), where j = ° i f k i s even, and j = 1 i f k i s odd. The polynomial P (X, " 12m+1 , 12m+2 ) can be easily integrated over the interval [X2m , X2m+2] with the2help12ofn the fundamental theorem of calculus, which yields:
xk+ l -Xk
1:
==
By adding these equalities together for all m = 0, 1 , . . . b
, M - 1, we obtain:
b-a = - (fo +4fl +212 +413 + ... + 4fn-I + fn). ( f(x)dx f) ';::j (In 3n i def
(4. 13)
a
Formula (4. 13) for the approximate computation of the definite integral is known as the Simpson quadrature formula.
l f(x)dx b
THEOREM 4.3 Let the third derivative interval [a b and let max ,
]
,
f(3 ) (x)(3 ) of the integrand f(x) be continuous on the If (x) I = M3 . Then the error of the Simpson
a�x�b
formula satisfies the following estimate:
l if(x)dx- (In (f) I ::::: b
·a
(b-V3a)M3 h3 . 9
(4. 14)
Computation of Definite Integrals. Quadratures
99
PRO O F To estimate the integration error for the Simpson formula, we first need to recall the error estimate for the interpolation of the integrand f (x) by piecewise quadratic polynomials. Let us denote the interpolation error by R2 (X) � f (x) - P2 (X) , where the function P2 (X) coincides with the quadratic polynomial P2 (X'/2m ' /2m+ l , /2m+2) if x E [X2m,X2m+2] . Then we can use Theorem 2.5 [formula (2.23) on page 36] and write for x E [X2m ,X2m+2] :
3 f( ) ( � ) R2 (X) = -(X - X2m) (X - X2m+ J ) (X - X2m+2) , 3!
� E (X2m, X2m+2 ) .
Consequently,
The previous inequality holds for x E [X2m,X2m+2] , where m can be any number: m = 0, . . . , M Hence it holds for any x E [a , ] and we therefore obtain:
1,
- 1.
b M3 3 h 9vM3 .
max IR2 (X) I <S;
aSxSb
For the integration error, we thus have: b
If a
f (x) dx - O"n U)
=
b
If a
h
1 = If a
f (x) dx -
u(X) - P2 (X)) dX
b
f a
P2 (x) dx b
I If =
a
l
R2 (X) dX
b
I f <S;
a
I R2 (x) l dx
o
which completes the proof of the theorem.
Theorem 4.3 implies that for the function f (x) that is three times continuously differentiable on [a, the error of the Simpson formula will decrease cubically with as h ------7 0, see estimate (4. l 4). For respect to the grid size i.e., with the rate a smoother integrand f (x), one can, in fact, obtain an even faster convergence.
b]
tJ(h3 )
h,
THEOREM 4.4 Let the fourth derivative f(4) (x) of the integrand f (x) be continuous on the interval [a , ] , and let max (x) 1 = M . Then the error of the Simpson a<x
l i4 )
b
h
If a
f (x)dx - O"n U)
4
1
<s;
(b - ;)M4 h4 . 1
(4. 15)
A Theoretical Introduction to Numerical Analysis
100
PRO O F We will show that for each equality holds:
m = 0, 1 , . . . ,M - 1
the following in
xJ2m+2 b - a M4 f(x)dx) ( +4hm+, +1 2 m+2 hm ::::; 9 h . � I I X2m
(
5
By adding these inequalities for all m = 0, 1 , . . . , M estimate (4.15):
4.16
)
- 1 , we arrive at the desired
J f(x)dx b�a (12m + 4hm+l + 12m+2 )] I Jb f(x)dx - ern (1) = M-�l [ X2m+2 I I m o X2m I X2J"'+2 b-a (12m +412m+' + 12m+2 ) ::::; 2n M94 h = (b a )M4 h . f(x)dx� ::::; �II 18 I ,�o x2m a
5
-
4
(4. 16), we represent the function f(x) on a neighborhood : X2m+ l X2m ::::; x ::::; X2m+2 , with the help of the Taylor formula: f(x) = Q (x) + R(x),
To prove inequality of the node
where
and Next,
R(x) = j-l44-) (!-S ) (X-X2m+1 )4 , X2m ::::; S ::::; X2m+2'
-'J2m+2 -'2J,"+2 -'J2m+2 f(x)dx- b�a (hm +4hm+l + 12m+2 ) = Q (x)dx+ R(x)dx
b-a [( Q2m +R2m ) +4(Q2m+ 1 +R2m+J ) + (Q2m+2 +R2m+2 )] -� J (x)dx- b�a (Q2m +4Q2m+1 +Q2m+2 ) } { .tX2m+2 Q 2m X2m+2 J + [ R(x)dx- b3�a (R2m +4R2m+1 +R2m+2 )) . X2m One can verify directly that for Q(x) 1 , Q (x) =:ox, Q (x) =:ox2 , and Q (x) =:ox3 the =
=:0
quantity in the curly brackets on the right-hand side of the previous equality
Computation ofDefinite Integrals. Quadratures
101
is equal t o zero. Consequently, this quantity is equal to zero for an arbitrary polynomial as long as its degree does not exceed 3. Therefore,
Q(x)
X2m+2
1 J f(x)dx - b�a (121 1 +412m+l + 121 1+2 ) 1 1 J R(x)dx - b�a (R2m+4R21 1+ 1 +R2m+2) 1 · X2m
=
X2m+2
X2m
[ R(x) [ �4
::::: h4 . Then the 1 = 0 and Let us now take into account that right-hand side of the previous equality can be estimated as follows:
R2m+
X2",+2
J
X2m
-a . 2 m�x [ R(x)[ ::::: m�x[R (x)[ [ / dx+ 2 b---a ] [ R (x)[ dx + b� 3 n . 2 b -a ::::: M 8 b -a M4 . ::::: max[ R (x)[ [2 b -a n + -3 -n ] 4 !4 -3 h4 = -h 9 X2.,,+2
X2m
--
x
--
n
Thus, we have proven inequality (4.16) , and as such, estimate (4. 1 5 ) .
5
0
COROLLARY 4.1 The Simpson quadrature formula on a uniform grid is exact for the integrands that are algebraic polynomials of degree no higher than third.
f(x)
PRO OF
Mzero. 4
=
max
For the cubic and lower degree polynomials, we obviously have = O. Consequently, the right-hand side of estimate (4. 15) is 0
[f4l(x)[
f(x) ( 4)
We would like to emphasize, however, that if the regularity of the integrand increases further, and even if becomes an infinitely differentiable function, the error of the Simpson formula will not, generally speaking, decay faster than 6 h (unless the integrand is periodic, see Remark In other words, similarly to the trapezoidal rule, the Simpson quadrature formula is prone to the saturation by smoothness (Exercise In Section 4.2, we describe alternative strategies for build ing numerical quadratures that will not suffer from saturation. Note also that in practice the dimension of the grid n is typically chosen automat ically in the course of computation. Namely, the computation is started with some particular after which the grid is refined (say, by a factor of two), then maybe refined again, etc., until the answer no longer changes within a prescribed tolerance.
f(x)
4).
no ,
4.1).
102
A Theoretical Introduction to Numerical Analysis
Exercises
1 . Show that the trapezoidal rule (4.4) is exact if the integrand f(x) is a polynomial of degree no higher than first.
2.* Prove the result formulated in Remark 4. l . Show that any quadrature formula from the corresponding family can be repre sented as a linear combination of a number of trapezoidal formulae.
Hint.
- Xk ::; h const, the use of piece wise quadratic interpolation leads to a generalization of the Simpson formula that 3 will only be third order accurate, i.e., will have 6(M3h ) error as h --; 0, where M3 max IfII! (x) I ·
3. Show that for a nonuniform grid that satisfies Xk+ I
=
=
4. Construct an example of an infinitely smooth function f(x), for which the error of the Simpson quadrature formula is still 6(h4 ).
4.2
Quadrature Formulae with No S at uration. Gaussian Quadratures
When we studied the interpolation of functions in Chapter 2, we first discovered that the conventional piecewise polynomial algebraic interpolation normally gets sat urated by smoothness, i.e., exhibits only a fixed rate of convergence as the grid is refined irrespective of how smooth the interpolated function is. In its own turn, this property of interpolation gives rise to the saturation of the quadrature formulae by smoothness I that we described in Section 4. 1 . Later in Chapter 3, we introduced the trigonometric interpolation and showed that it does not get saturated by smooth ness. In other words, it self-adjusts the accuracy of approximation to the regularity of the interpolated function, provided that the latter is periodic. In the non-periodic case, a direct analogue of the trigonometric interpolation is algebraic interpolation on Chebyshev nodes, e.g., on the Chebyshev-Gauss grid or on the Chebyshev-Gauss Lobatto grid, see Section 3.2. Algebraic interpolation of this type does not get satu rated by smoothness, and it is therefore natural to expect that if a quadrature formula is built based on the Chebyshev interpolation, then it will not be prone to saturation either. This expectation does, in fact, prove correct, and the corresponding family of quadrature formulae is commonly referred to as Gaussian quadratures. In this sec tion, we introduce and study some examples of Gaussian quadratures. The material presented hereafter is based on the more advanced considerations of Chapter 3 and can be skipped during the first reading. Chebyshev polynomials Tk (X) � cos(karccosx), k = 0, 1 , 2, . . . , were discussed in Section 3.2 in the context of interpolation. They are algebraic polynomials of degree k defined on the interval - 1 :::; x :::; 1. An arbitrary interval a :::; x :::; b can be mapped 1 With
the exception of the Newton-Cotes formulae applied to periodic functions.
1 03 Computation of Definite Integrals. Quadratures onto [-1, 1] by a simple linear transformation: x a!2 + t b -:t where -1 :s; t :s; 1. Therefore, analyzing both interpolation and quadratures only on the interval [-1, 1 ] (as opposed to [a , b j) does not imply any loss of generality. The n + 1 roots of the Chebyshev polynomial Tn+ (x) on the interval [-1, 1] are given by the formula: 1) , m = 0, 1,2, ... ,n, (4.17) Xm = cos 1!(2m+ 2(n+ 1) and are referred to as the Chebyshev-Gauss nodes. The n + 1 extrema of the Cheby shev polynomial T,,(x) are given by the formula: 1! m = 0, 1,2, ... ,n, (4 . 1 8 ) Xm = cos -m, n =
'
1
and are referred to as the Chebyshev-Gauss-Lobatto nodes. To employ Chebyshev polynomials in the context of numerical quadratures, we will additionally need their orthogonality property given by Lemma
4.1.
LEMMA 4.1 Chebyshev polynomials are orthogonal on the interval -1 :s; weight w(x) = VI - x2 , i. e., the following equalities hold for
1/
PRO O F
x :s; 1 with the
k, I = 0, 1 , 2, ...
:
1! ' k = I = 0, Tk(x)7[(X) (4.19) dx 1!/2, k = 1 -1= 0, J � 1 - x2 -I k -l= l . 0, Let x = cos q> q> = arccosx. Changing the variable in the 1
{
V
¢}
integral, we obtain:
0 k cos (l Tk(x)7[(x) J � dx - J COS ( q» q» ( 1
11
sm q> )d q> - J cos (k q> ) cos (I q> )d q> ' q> [ sm 2 v I x I 0 -I 1!) k = 1 = 0, = � J [COS(k + l)q> + cos (k - l)q>]dq> = 1!/2, k = I -1= 0, o 0, k -l= l , because Chebyshev polynomials are only considered for k :::: ° and I :::: 0. 0 Let the function f = f(x) be defined for - 1 :s; x :S; 1. We will approximate its definite integral over [-1, 1] taken with the weight w(x) = 1 / VI - x2 by integrating the Chebyshev interpolating polynomial P" (x, f) built on the Gauss nodes (4.17): _
_
.
11
{
11
J
1
f(x) dx;:::;:, J P,, (x, j) dx. J V1 - x2 V1 - x2 -I -I
(
4. 20)
A Theoretical Introduction to Numerical Analysis
104
The key point, of course, is to obtain a convenient expression for the integral on the right-hand side of formula (4.20) via the function values m , = 0, sampled on the Gauss grid (4. 17). It is precisely the introduction of the weight = 1 / V I - into the integral (4.20) that enables a particularly straightforward integration of the interpolating polynomial over [- 1, 1 ].
f( ) m x
x2
w(x)
1,2, ... ,m,
�,(X,j)
LEMMA 4.2 Let the function = be its algebraic be defined on [- 1 , I ] , and let interpolating polynomial on the Chebyshev- Gauss grid {4. 1 7} . Then,
Pn (x,j)
f f(x) I
I-.:. v'1Pn (x,-jx2) dx = n� + I mi=O f(xm ).
(4.21)
I
PI (X,j)
PRO O F Recall that is an interpolating polynomial of degree no higher than built for the function on the Gauss grid (4. 17) . The grid has a total of I nodes, and the polynomial is unique. As shown in Section 3.2.3, see formulae (3.62) and (3.63) on page 76, can be represented as:
n n+
f(x)
�,(X,j) Pn (x,j) = k=LO akTk(x), Jl
Tk(X) are Chebyshev polynomials of degree k, and the coefficients ak are n fmT (xm) and 1 2 L f,n Tk (Xm), k = 1 , . . . ,no ao = -ak = -L n + I 111=0 o n + I m=O Accordingly, it will be sufficient to show that equality (4.21) holds for all individual Tk (X), k = 0, 1 , 2 , ... ,n: where given by:
n
(4.22)
(x)
Let k = O, then To == 1 . In this case Lemma 4.1 implies that the left-hand side of ( 4.22 ) is equal to and the right-hand side also appears equal to by direct substitution. For k > 0, orthogonality of the Chebyshev polynomials in the sense of formula (4. 19) means that the left-hand side of (4.22) is equal to zero. Consequently, we need to show that the right-hand side is zero as well. Using trigonometric interpretation, we obtain for all k = 1: n
I�O
Tk (Xm) = =
n,
n
1,2, ... , 2n +
( [ n(2m+ l) ]) 2(11 + 1 ) I�n O ( n(2m+ l ) ) LI cos (k 2(n2nm1 ) + k 2(nn+ ) L cos k 2(n 11
111=0
cos k arccos cos
+ I)
=
111=0
+
I)
= 0.
Computation of Definite Integrals. Quadratures
105
The last equality holds by virtue of formula (3.22) , see the proof of Theo rem 3.1, page 64. Thus, we have established equality (4.21). D
4.2 1 allows us to recast the approximate expression (4.20) for the integral }1 f(x)w(x)dx as follows: n (x f(x) 77: (4.23) dx � 2, J JT=X2 I -x 11 + -0 f m ). Formula (4. 2 3) is known as the Gaussian quadrature formula with the weight w(x) 1 / VI - x2 on the Chebyshev-Gauss grid (4.17). It has a particularly simple structure, which is very convenient for implementation. In practice, if we need to evaluate the 1 integral } f(x)dx for a given f(x) with no weight, we introduce a new function g(x) f(x)Vl - x2 and then rewrite formula (4.23) as: g(x) dx 77: J f(x)dx = J JT=X2 l -x2 n + 1 1112,=0 g(xm ). A key advantage of the Gaussian quadrature (4. 2 3) compared to the quadrature formulae studied previously in Section 4.1 is that the Gaussian quadrature does not get saturated by smoothness. Indeed, according to the following theorem (see also Remark 4.2 right after the proof of Theorem 4.5), the integration error automatically Lemma
-I
1
1
-1
=
111 -
=
-I
1
1
-I
-I
�
11
adjusts to the regularity of the integrand.
THEOREM 4.5 1 Let the function = ; let it have continuous be defined for - 1 :::; derivatives up to the order > and a square integrable derivative of order
r+
1
:
f f(x)
x :::; r 0, J [lr+ I)(x)r dx < 00. 1
-I Then, the error of the Gaussian quadrature (4 .23) can be estimated as: 1
n ) 'n f(x) 77: dx2, I J JT=X2 n + 1 1 1=0f(xm I :::; 77: nr-l/2 ' where 'n = 0 ( 1 ) as n --+ 00. -1
X
(4.24)
PRO O F The proof of inequality (4.24 ) is based on the error estimate (3.65 ) obtained in Section 3.2.4 ( see page 77) for the Chebyshev algebraic in
terpolation. Namely, let
RIl (x) = f(x) -Pn (x,j). Then, under the assumptions
A Theoretical Introduction to Numerical Analysis
106
of the current theorem we have:
max
- 1 �x� 1
IRII(x)l :::; nrS-� /2 '
where Sn = o( I ) ,
n -----> oo.
(4.25)
Consequently,
f(x) PI (X, J)(X) i 1 VIf(x)-x2dx - � n + 1 m=iOf(xm ) I = 1 1 VI -x2 dx _ 1 VIl -x2 dx l SII - -I1 If(x)V1- P,-xl (X,2 J)1 dx - max IRI (x)1 . 1 vIdx-x2 nr-I/2 I
I
-I
-I
I
-I
<
<
I
which yields the desired estimate
- n -- , -
I
-I
- 1 �x� 1
(4.24).
o
f = f(x)
Theorem 4.5 implies, in particular, that if the function i s infinitely dif ferentiable then the Gaussian quadrature (4.23) exhibits a spectral rate of convergence as the dimension of the grid increases. In other words, the integration for any > 0 as error in (4.24) will decay faster than
on [ - 1 , 1],
n 6(n- r)
r
n ----->
00
.
REMARK 4 . 2 Error estimate (4.25) for the algebraic interpolation on Chebyshev grids can, in fact, be improved. According to formula (3.67), see Remark 3.2 on page 78, instead of inequality (4.25) we can write:
max
- 1 �x� 1
I RII(x)1 = 0 ( nrI+n n/2 ) -
1
n -----> 00.
as
Then, the same argument as employed in the proof of Theorem improved convergence result for Gaussian quadratures: I
I1 -I
r+
ln n ) f(x) dx- �n f(xm ) = 0 ( r+I/ 2 n+ 1 I n vr=xz m-O n
n
as
4.5 yields an
n -----> 00, f(x)
where 1 is the maximum number of derivatives that the integrand has on the interval - 1 x 1 . However, even this is not the best estimate yet. As shown in Section 3.2.7, by combining the Jackson inequ ality [Theo rem 3.8, formula (3.79)] , the Lebesgue inequality [Theorem 3.10, formula (3.80)], and estimate (3.83) for the Lebesgue constants given by the Bernstein theorem [Theorem 3.12], one can obtain the following error estimate for the Chebyshev algebraic interpolation:
:::; :::;
max
- 1 �x� 1
1) IRn (x)1 = 6 ( In (n+ nr+1 ) '
as
n ----->
00
,
Computation ofDefinite Integrals. Quadratures 1 07 provided only that the derivative J lr )(x) of the function f(x) is Lipshitz
continuous. Consequently, for the error of the Gaussian quadrature we have: 1 n 1) � L = as ----+ 00 . (4.26) m 1 ' + 1 m=O vi 1 -I Estimate (4.26) implies, in particular, that the Gaussian quadrature (4.23) converges at least for any Lipshit>l-continuous function (page 87), and that 1 the rate of decay of the error is not slower than 1 ) ) as ----+ 00 . Otherwise, for smoother functions the convergence speeds up as the regularity increases, and becomes spectral for infinitely differentiable functions. 0
/J
f(x ) / tJ ( rr ln(nrn++
f(x) dxx2 n -
)
n
tJ(n- In(n +
n
When proving Lemma 4.2, we have, in fact, shown that holds for all Chebyshev polynomials up to the degree k = 2n 1 and not only up to the degree k = This circumstance reflects on another important property of the Gaussian quadrature (4.23) . It happens to be exact, i.e., it generates no error, for any algebraic polynomial of degree no is a higher than + 1 on the interval - 1 :::; :::; l . In other words, if polynomial of degree :::; + 1 , then 1 REMARK 4.3
+
equality
(4.22)
2n
Tk(x)
n.
x 2n rr Q2n+I (X)2 dx = -� J n + 1 m=O Q2n+1 (xm ) · 1 x _1 £..
�
V
Q2n+1 (x)
-
Moreover, it is possible to show that the Gaussian quadrature (4.23) is optimal in the following sense. Consider a family of the quadrature formulae on [- 1 , 1]:
j f(x)w(x)dx
(4.27) ± ajf(xj), where the dimension n is fixed, the weight w(x) is defined as before, w(x) = 1 / vi1 -x2 , and both the nodes xj and the coefficients aj , j = 0, 1 , , n are
-1
�
}=o
. . .
to be determined. Require that this quadrature be exact for the polynomials of the highest degree possible. Then, it turns out that the corresponding degree will be equal to + 1 , while the nodes Xj will coincide with those of the Chebyshev-Gauss grid (4.1 7), and the coefficients aj will all be equal to + 1 ) . Altogether, quadrature (4.27) will coincide with (4.23). 0
rr/(n
2n
Exercises
1. Prove an analogue of Lemma 4.2 for the Gaussian quadrature on the Chebyshev-Gauss Lobatto grid (4. 1 8). Namely, let ?n (x,J) Lk=o ihTk (x) be the corresponding interpo lating polynomial (constructed in Section 3.2.6). Show that =
1
.
_
£..
dx - � � f3 f(xm ) , J v� I - x· n m=o
-I
Pn (x,J)
In
A Theoretical Introduction to Numerical Analysis
108 where f30
4.3
=
13n
=
1/2 and 13m = 1 for In = 1 ,2,
. . . , In
- 1.
Improper Integrals . Combination o f Numerical and Analytical Methods
Even for the simplest improper integrals, a direct application of the quadrature formulae may encounter serious difficulties. For example, the trapezoidal rule (4.4):
b a J f(x) dx : (� b
a
�
+
fl + . . . + fll - I +
a
�)
will fail for the case = 0, b = 1, and f(x) = cosx/ JX, because f(x) has a singularity at x = 0 and consequently, fo = f(O) is not defined. Likewise, the Simpson formula (4. 13) will fail. For the Gaussian quadrature (4.23), the situation may seem a little better at a first glance, because the Chebyshev-Gauss nodes (4. l7) do not include the endpoints of the interval. However, the unboundedness of the function and its derivatives will still prevent one from obtaining any reasonable error estimates. At c x the same time, the integral itself: obviously exists, and procedures for its effi cient numerical approximation need to be developed. To address the difficulties that arise when computing the values of improper in tegrals, it is natural to try and employ a combination of analytical and numerical techniques. The role of the analytical part is to reduce the original problem to a new problem that would only require one to evaluate the integral of a smooth and bounded function. The latter can then be done on a computer with the help of a quadrature formula. In the previous example, we can first use the integration by parts:
10 1 � dx,
jo cO;;dx
=
I�
cosx(2JX) +
j sJXsinxdx, 0
and subsequently approximate the integral on the right-hand side to a desired accu racy using any of the quadrature formulae introduced in Section 4. 1 or 4.2. An alternative approach is based on splitting the original integral into two: 1
J o
cosx JX
c
dx J -
0
c
cosx JX
1
dx, dx+ J cosx JX
(4.28)
c
where > 0 can be considered arbitrary as of yet. To evaluate the first integral on the right-hand side of equality (4.28), we can use the Taylor expansion: x2 x4 x6 cosx = 1 - + - - - + . . . . 21 4! 6!
-
Computation ofDefinite Integrals. Quadratures
109
Then we have:
I l/2 ... 3/2 /. x-dx 7/2 - j X--dx+ cosx dx - j' x-dx jj --dx= + 2! 6! . 4! c
o
c
y'x
0
c
y'x
c
0
c
0
0
,
(4.29)
and every integral on the right-hand side of equality (4.29) can be evaluated explicitly using the fundamental theorem of calculus. Let the tolerance f > 0 be given, and assume that the overall integral needs to be approximated numerically with the error that may not exceed f. It is then suffi cient to approximate each of the two integrals on the right hand side of (4.28) with the accuracy f /2. As far as the first integral, this can be achieved by taking only a few leading terms in the sum on the right-hand side of formula (4.29). The precise number of terms to be taken will, of course, depend on f and on c. In doing so, the value of c should not be taken too large, say, c = I, because in this case even if f is not very small, the number of terms to be evaluated on the right-hand side of formula (4.29) will still be substantial. On the other hand, the value of c should not be taken too small either, say, c « 1 , because c in this case the first integral on the right-hand side of (4.28): can be
foc o;xxdx, evaluated very efficiently, but the integrand in the second integral l' cO;; dx will n
have large derivatives. This will necessitate taking large values of (grid dimension) for evaluating this integral with accuracy f /2, say, using the trapezoidal rule or the Simpson formula. A more universal and generally more efficient technique for the numerical com putation of improper integrals is based on regularization, i.e., on isolation of the singularity. Unlike in the previous approach, it does not require finding a fine balance between the too small and too large values of c. Assume that we need to compute the integral
j1 f(X) dx
o y'x ' where the function is smooth and bounded. Regularization consists of recasting this integral in an equivalent form:
f(x)
j1 f(x) dx j1 f(x) - rp(x) dx + j1 rp(x) dx. 0
y'x
=
0
y'x
rp(x)
0
y'x
(4.30)
is to be chosen so that to remove the singu In doing so, the auxiliary function larity from the first integral on the right-hand side of (4.30) and enable its effi cient and accurate approximation using, e.g., the Simpson formula. The second integral on the right-hand side of (4.30) will still contain a singularity, but we select so that to be able to compute this integral analytically. In our particular case, this goal will be achieved if is taken in the form of around = 0: just two leading terms of the Taylor expansion of
f(x)
rp (x)
rp(x)
x
rp(x) = f(O) +
A Theoretical Introduction to Numerical Analysis
1 10
l' (O )x. Substituting this expression into formula (4.30) we obtain:
1
f(x) dx = 1 f(x) -f(O) - f (O)x dx+ f(O ) 1 dx + 1'(0) v'XdX. J v'X J J v'X J v'X 1
I
0
o
0
Returning to our original example with pression for the integral: 1
0
f(x) = cosx, we arrive at the following ex
1 cosxld dx . x = J v'X x+ J v'X
cosx d
J v'X
0
1
0
0
2,
The second integral on the right-hand side is equal to while for the evaluation of the first integral by the Simpson formula, say, with the accuracy e = 10- 3 , taking a fairly coarse grid 4, i.e., = 1 /4, already appears sufficient. Sometimes we can also use a change of variable for removing the singularity. For instance, setting = in the previous example yields:
n x t2
h
=
1
I
cosx
cost2-2tdt = 2 -
J v'X dx J t 0
=
0
I
2
J cost dt , 0
and the resulting integral no longer contains a singularity. Altogether, a key to success is a sensible and adequate partition of the overall task into the analytical and numerical parts. The analytic derivations and transformations are to be performed by a human researcher, whereas routine number crunching is to be performed by a computer. This statement basically applies to solving any problem with the help of a computer. Exercises
1 . Propose algorithms for the approximate computation of the following improper inte grals:
j Sin3/2Xdx, jIO In(l3/2+x) dX, I
o
4.4
x
o
x
jn/2 o
cosx dx. -y-,=== lr/2 - x
Multiple Integrals
Computation of multiple integrals:
I(m) I(m) (j, Q) = J ... J f(Xl , ... ,Xm )dXl ... dxm =
n
Computation ojDefinite Integrals. Quadratures
111
over a given domain .Q of the space ]Rm of independent variables X I , . . . ,Xm can be reduced to computing integrals of the type g(x)dx using quadrature formulae that we have studied previously. However, the complexity of the corresponding al gorithms and the number of arithmetic operations required for obtaining the answer with the prescribed tolerance e > 0 will rapidly grow as the dimension of the space increases. This is true even in the simplest case when the domain of integration is a and if .Q has a more general cube: .Q = {x = (XI , ,xm ) I l xj l :::; 1 , j = 1 2 shape then the problem gets considerably exacerbated. In the current section, we only provide several simple examples of how the prob lem of numerical computation of multiple integrals can be approached. For more detail, we refer the reader to [DR84, Chapter 5].
lb
, , ... , m},
• • ·
4.4.1
m
Repeated Integrals and Quadrature Formulae
We will illustrate the situation using a two-dimensional setup, = 2. Consider the integral:
1(2 ) =
11 J(x,y)dxdy
m
y
n
over the domain .Q of the Cartesian plane, which is schematically shown in Figure 4. 1 . Assume that this do main lies in between the graphs of two functions: Then, clearly,
1(2 )
=
where
11 J(x,y)dxdy n
lb
=
F(x) =
o
1b F(x)dx,
a
b
FIGURE 4. 1 : Domain of integration.
a
cpz(x)
1 J(x,y)dy, a :::; x :::; b.
IPI (x)
The integral 1(2) = F(x)dx can be approximated with the help of a quadrature formula. For simplicity, let us use the trapezoidal rule:
1b
b : a ( F �o ) +F(xd + . . . +F(xn - d + F�n) ) , b-a Xj a + j . -. . . ,n. n
1(2) = F(x)dx >:::; a
=
,
j = O, I ,
(4.3 1 )
x
A Theoretical Introduction to Numerical Analysis
112
In their own turn, expressions
F(xj)
=
fP2 (Xj)
f f(xj,y)dy,
4'1 (.tj)
j = O, I , . . .
,n
that appear in formula (4.31) can also be computed using the trapezoidal rule:
(4.32)
£ £(n)
In doing so, the approximation error that arises when computing the integral /(2 ) by means of formulae (4.3 1) and (4.32) decreases as the number increases. At the same time, the number of arithmetic operations increases as If the integrand = has continuous partial derivatives up to the second order on and n, and if, in addition, the functions are suffi ciently smooth, then the properties of the one-dimensional trapezoidal rule established in Section 4.1 .2 guarantee convergence of the two-dimensional quadrature (4.3 1 ), (4.32) with second order with respect to = To summarize, the foregoing approach is based on the transition from a multiple integral to the repeated integral. The latter is subsequently computed by the re peated application of a one-dimensional quadrature formula. This technique can be extended to the general case of the integrals 1(111) , > 2, provided that the domain n is specified in a sufficiently convenient way. In doing so, the number of arith metic operations needed for the implementation of the quadrature formulae will be where is the number of nodes in one coordinate direction. If the function has continuous partial derivatives of order two, the boundary of the domain n is sufficiently smooth, and the trapezoidal rule (Section 4. 1 .2) is used in the capacity of the one-dimensional quadrature formula, then the rate of convergence will be quadratic with respect to = In practice, if the tolerance > ° is specified ahead of time, then the dimension is chosen experimentally, by starting the computation with some and then refining the grid, say, by doubling the number of nodes (in each direction) several times until the answer no longer changes within the prescribed tolerance. =
n 0(n2 ).
f f(x,y)
!PI (x) Cj>2(x) n- I : £ (n) 0(n-2 ). m
O(nm ), f(XI , ... ,xm )
n
£
4.4.2
n- ': £(n) eJ(n-2 ).
no
n
T h e U s e o f Coordinate Transformations
Let us analyze an elementary example that shows how a considerable simplifi cation of the original formulation can be achieved by carefully taking into account the specific structure of the problem. Suppose that we need to compute the double where the domain n is a disk: n = xl + l ::; R2 }, integral
fL f(x,y)dxdy,
{(x,y) I
Computation of Definite Integrals. Quadratures
f(x,y)
1 13
x
and the function does not, in fact, depend on the two individual variables Then we can == and but rather depends only on 2 + l = r2 , so that write: R
y,
x
1( 2) =
f(x,y) f(r).
11f(x, y)dxdy 27r 1 f(r)rdr, =
Q
0
and the problem of approximately evaluating a double integral is reduced to the prob lem of evaluating a conventional definite integral. 4.4 . 3
The Notion of Monte C arlo Methods
The foregoing reduction of multiple integrals to repeated integrals may become problematic already for m = 2 if the domain n has a complicated shape. We there fore introduce a completely different approach to the approximate computation of multiple integrals. Assume for simplicity that the domain n lies inside the square D [ O :S 1, 0 :S I } . The function = is originally defined on n, and we extend it = 0 for into D\n and E D\n. Suppose that there is a random number generator that produces the numbers uniformly distributed across the interval [0 1 ] . This means that the probability for a given number to fall into the interval [(X I , (X2 ] , where 0 :S (XI < (X2 :S 1 , is equal to (X2 Using this generator, let us obtain a sequence of random pairs = k d , k = 1 2 . . . , and then construct a numerical sequence: 1 k
:S y
=
f f(x,y) letf(x,y) (x,y) Pk (x y ,
Sk = k
{(x,y) x:S ,
-
(XI .
,
,
j='LI f(xj , Yj).
It seems natural to expect that for a given £ > 0, the probability that the error [/( 2) Sk i is less than £ : will approach one as the number of tests k increases. This statement can, in fact, be put into the framework of rigorous theorems. The meaning of the Monte Carlo method becomes particularly apparent if = I on n. Then the integral 1(2 ) is the area of n, and Sk is the ratio of the number of those points 1 j :S k, that fall into n to the overall number k of these points. It is clear that this ratio is approximately equal to the area of n, because the overall area of the square D is equal to one. The Monte Carlo method is attractive because the corresponding numerical algo rithms have a very simple logical structure. It does not become more complicated when the dimension m of the space increases, or when the shape of the domain n becomes more elaborate. However, in the case of simple domains and special narrow classes of functions (smooth) the Monte Carlo method typically appears less efficient than the methods that take explicit advantage of the specifics of a given problem. The Monte Carlo method has many other applications, see, e.g., [LiuOl , BH02, CH06]; an interesting recent application area is in security pricing [Gla04].
f(x,y)
Pj , :S
Part II Systems of Scalar Equations
1 15
In mathematics, plain numbers are often referred to as scalars; they may be real or complex. Equations or systems of equations with respect to one or several scalar unknowns appear in many branches of mathematics, as well as in applications. Ac cordingly, computational methods for solving these equations and systems are in the core of numerical analysis. Along with the scalar equations, one frequently encoun ters functional equations and systems, in which the unknown quantities are functions rather than plain numbers. A particular class of functional equations are differential equations that contain derivatives of the unknown functions. Numerical methods for solving differential equations are discussed in Part III of the book. In the general category of scalar equations and systems one can identify a very important sub-class of Numerical methods for solving various systems of this type are discussed in Chapters 5, 6, and 7. In Chapter 8, we address numerical solution of the nonlinear equations and systems. Before we actually discuss the solution of linear and nonlinear equations, let us mention here some other books that address similar subjects: [Hen64,IK66, CdB80, Atk89, PT96, QSSOO, Sch02, DB03]. More specific references will be given later.
linear algebraic equations and systems.
1 17
Chapter 5 Sys tems of Line ar Alge br aic Equ a tions: Direct Meth ods
Systems of linear algebraic equations (or simply linear systems) arise in numerous applications; often, these systems appear to have high dimension. A linear system of order (dimension) n can be written as follows:
where ai), i, j = 1 , 2, . . . , n, are the coefficients, Ii, i 1 , 2, . . . , n, are the right-hand sides (both assumed given), and Xj, j = 1 , 2, . . . , n, are the unknown quantities. A more general and often more convenient form of representing a linear system is its operator form, see Section 5. 1 .2. In this chapter, we first outline different forms of consistent linear systems and illustrate those with examples (Section 5. 1), and then provide a concise yet rigorous review of the material from linear algebra (Section 5.2). Next, we introduce the notion of conditioning of a linear operator (matrix) and that of the corresponding system of linear algebraic equations (Section 5.3), and subsequently describe several direct methods for solving linear systems (Sections 5.4, 5.6, and 5.7). We recall that the method is referred to as direct if it produces the exact solution of the system after a finite number of arithmetic operations. 1 (Exactness is interpreted within the machine precision if the method is implemented on a computer.) Also in this chap ter we discuss what types of numerical algorithms (and why) are better suited for solving particular classes of linear systems. Additionally in Section 5.5, we estab lish a relation between the systems of linear algebraic equations and the problem of minimization of multi-variable quadratic functions. Before proceeding any further, let us note that the well-known Cramer's rule is never used for computing solutions of linear systems, even for moderate dimensions n > 5. The reason is that it requires a large amount of arithmetic operations and may also develop computational instabilities. =
1 An
alternative to direct methods are iterative methods analyzed in Chapter 6.
1 19
A Theoretical Introduction to Numerical Analysis
1 20
5.1
Different Forms of Consistent Linear Systems 2
A linear equation with respect to the unknowns ZI , 2
form
,
. . . , 211
is an equation of the
t
where a I , a2 , . . . , all, and are some given numbers (constants). 5.1.1
Canonical Form of a Linear System
n
Let a system of linear algebraic equations be specified with respect to as many unknowns. This system can obviously be written in the following form:
canonical
(5. 1 ) Both the unknowns Xj and the equations in system (5. 1 ) are numbered by the the consecutive integers: 1 , 2, . . . Accordingly, the coefficients in front of the unknowns have double subscripts: j = 1 , 2, where i reflects on the number of the equation and j reflects on the number of the unknown. The right-hand side of equation number i is denoted by Ji for all = 1 , 2, . . From linear algebra we know that system (5. 1 ) is i.e., has a solution, for any choice of the right-hand sides J; if and only if the corresponding homogeneous system2 only has a trivial solution: XI = X2 = . . . = Xn = O. For the homogeneous sys tem to have only the trivial solution, it is necessary and sufficient that the determinant of the system matrix A:
, n.
i,
. .
i
.
aU
, n,
, n. consistent, .
(5.2) differ from zero: detA of- O. The condition detA of- 0 that guarantees consistency of system (5. 1 ) for any right-hand side also implies uniqueness of the solution. Having introduced the matrix A of (5.2), we can rewrite system (5. 1 ) as follows: (5.3) or, using an alternative shorter matrix notation: Ax = J. 2 0btained by setting li
=
0 for all i = 1 , 2, . . . , n .
Systems of Linear Algebraic Equations: Direct Methods 5 . 1 .2
121
Operator Form
Let ]R1I be a linear space of dimension n (see Section 5.2), A be a linear operator, A ]R" f----7 ]RII, and! E ]R11 be a given element of the space ]R". Consider the equation: :
(5.4)
Ax =!
with respect to the unknown x E ]R". It is known that this equation is solvable for an arbitrary! E ]R11 if and only if the only solution of the corresponding homogeneous equation Ax = 0 is trivial: x = 0 E ]R11 . Assume that the space ]R11 consists of the elements: _
z-
[Z�I] :
(5.5)
'
ZI1
where {ZI , ' " , ZII } are ordered sets of numbers. When the elements of the space ]Rn are given in the form (5.5), we say that they are specified by means of their components. From linear algebra we know that a unique matrix of type (5.2) can be associated with any linear operator A in this case, so that for a given x E ]RII the ]R11 will be rendered by the formulay = Ax, i.e., mapping A : ]R11 11
f----7
[YI] ;11
[
][]
a l l " , a l II X l
�:, ; '. : " �;,�
;"
(5.6) •
Then equation (5.4) coincides with the matrix equation (5.3) or, equivalently, with the canonical form (5. 1 ). In this perspective, equation (5.4) appears to be no more than a mere operator interpretation of system (5. 1 ). However, the elements of the space ]R11 do not necessarily have to be specified as ordered sets of numbers (5.5). Likewise, the operator A does not necessarily have to be originally defined in the form (5.6) either. Thus, the operator form (5.4) of a linear system may, generally speaking, differ from the canonical form (5. 1). In the next section, we provide an example of an applied problem that naturally leads to a high-order system of linear algebraic equations in the operator form. This example will be used throughout the book on multiple occasions. 5 . 1 .3
Finite-Difference D irichlet Problem for the Poisson Equation
Let D be a domain of square shape on the Cartesian plane: D = { (x,y) [O < X < Assume that a Dirichlet problem for the Poisson equation:
I , 0 < Y < I }.
(X,y) E D,
(5.7)
A Theoretical Introduction to Numerical Analysis
1 22
needs to be solved on the domain D. In formula (5.7), aD denotes the boundary of D, and f = f(x,y) is a given right-hand side. To solve problem (5.7) numerically, we introduce a positive integer M, specify
h = M- 1 , and construct a uniform Cartesian grid on the square D:
(5.8) Instead of the continuous solution u = u(x,y) to the original problem (5.7), we will rather be looking for its trace, or projection, [ul h onto the grid (5.8). To compute the values of this projection approximately, we replace the derivatives in the Poisson equation by the second order difference quotients at every interior node of the grid (see Section 9.2. 1 for more detail), and thus obtain the following system of difference equations:
uml ,ml
_
A(h)U(h)
I
ml ,mZ + Uml( Uml,/Il++l ,m] -l -2u2/IUmll ,m,2ml +Um1Um,ml -z-I ,m] )l m m ' l , 2 1, 2, . m ,/I l =
� h
_
m] , m2
+
h2
=
. .,M
=
f,
(5.9)
1.
The right-hand side of each equation (5.9) is defined as f l � f(m l h,m2 h). Clearly, the difference equation (5.9) is only valid at the interior nodes of the grid (5.8), see Figure 5 . 1 (a). For every fixed pair of ml and m2 inside the square D, it connects the values of the solution at the five points that are shown in Figure 5 . 1 (b). These points are said to form the stencil of the finite-difference scheme (5.9).
1j.. : ... : .. : ... .... ... ... ...
y
j,-
.
I
I
.
.
I I
I I
, , I
I I I
.
I
. , I I
:
:
I
,
•
..
I I I
I , I
:
I
. I I I
I , I
... �... �. . . �... � .. �.. �... �...
o
. . .. . . . ... ... . .. . . ... . I I
I I
, ,
I I
!
I
I I I
I I
I
I
. I I I
, I
. I
I
(a)
,
•
-
I
I
,
I
.
.
.
.
I I
I
I I
I
, ,
I I
I I
I
I
,
,
I
, I I
interior,
, , ,
,
0
-
I I I
I
- - - - - - - -
- - - - - - - - -
- - - - -
,r ,
- - - - - - - -
- - - - - - - - - r - - - -
I
. , I I
,,r
I I
. . t... t... t... t .. �. . t... t... ... ... ... . . .. ... �... � .. �... � . �... �... �... ... �... ... � . �... �... � �.. ? . + . + . + .. + .. + . + .. + ... .
- - - - -
:
I
.
I I I
I
boundary
x
,,,r ,
- - - -
, ,
(b) Stencil
FIGURE 5. 1 : Finite-difference scheme for the Poisson equation. At the boundary points of the grid (5.8), see Figure 5 . 1 (a), we obviously cannot write the finite-difference equation (5.9). Instead, according to (5.7), we specify the
Systems oj Linear Algebraic Equations: Direct Methods homogeneous Dirichlet boundary conditions: um l ,m2 = 0, for m l = O,M
&
123
=
(5. 10) m2 O,M. Substituting expressions (5. 10) into (5.9), we obtain a system of (M - I? linear alge braic equations with respect to as many unknowns: um " m2 ' m l ,m2 1 , 2, ,M- 1 . This system i s a discretization of problem (5.7). Let u s note that in general there are many different ways of constructing a discrete counterpart to problem (5.7). Its particular form (5.9), (5. 10) introduced in this section has certain merits that we discuss later in Chapters 10 and 12. Here we only mention that in order to solve the Dirichlet problem (5.7) with high accuracy, i.e., in order to make the error between the exact solution [ul h and the approximate solution u (h) small, we should employ a sufficiently fine grid, or in other words, take a sufficiently large M. In this case, the linear system (5.9), (5. 10) will be a system of high dimension. We will now demonstrate how to convert system (5.9), (5. 10) to the operator form (5.4). For that purpose, we introduce a linear space U (h) IRn , n � (M - I ?, that will contain all real-valued functions Zm l ,m2 defined on the grid (m l h, m2 h), m l ,m2 1,2, . . ,M - 1 . This grid is the interior subset of the grid (5.8), see Figure 5.I(a). Let us also introduce the operator A == _/';,(h ) : U (h) U (iz ) that will map